Command line tool

[anadius]

I'd love to see some more command line functionality like listing all tray files (author, name, type, ID, modification date?, etc., possibly in JSON format), exporting them, listing all tray files in archive and importing them.

[deevo]

Hello @anadius

What do you want to achieve with this functionality?

[anadius]

I wanted to share my items automatically. Thought of something like this: my script detects the game was closed, uses Tray Importer to list all tray items, selects the ones with my name, checks modification date, uses Tray Importer to export new/modified items and then uploads them wherever I want.

 

Haven't thought much about importing part.

[deevo]

Will you need the CC tracking part?

This one will be hard to port due to many dependencies.

Not to mention that it could produce very large files without user interaction.

 

As long as you share file access, the game may remain open while reading the contents of the Tray folder.

Importing works too, but requires a restart for the new files to be handled.

 

You may run into problems if the json file is too large, but I think this problem only affects browsers.

[anadius]

I don't think so. If I use CC, I add it to description. Don't plan to export with CC.

The output JSON file should contain as little as possible: ID (the part between "!" and "." perhaps), author name, item name, item description, item type... and I guess that's it.

And then just pass those IDs (comma separated?) and export location (folder or file path) to export function.

 

Good to know!

 

It's cool, I'm writing in Python and there are modules for handling huge JSON files.

 

 

To be hones I would do it myself but I can't find any info on parsing .trayitem files... Your app is a blessing

[deevo]

il y a 33 minutes, anadius a dit :

I don't think so. If I use CC, I add it to description. Don't plan to export with CC.

In that case that fix the issue. 

 

il y a 35 minutes, anadius a dit :

To be hones I would do it myself but I can't find any info on parsing .trayitem files…

If you want to try it yourself, here's some info:

TrayItems are just a Google ProtoBuffer stream wrapped in another stream.

Bytes 0-3 indicate the type of the file. (Must be used on the server side)
Bytes 4-7 indicate the size of the ProtoBuffer stream.
Remaining bytes are the protobuffer stream itself.

Fortunatly you can find the required protobuffer files already converted to python. (look for generated.zip in the game folder)

I converted them back to .proto files since it's easier to see changes and to convert to other languages.

 

 

I'm already working on porting the basic functions to a cmd line tool. 

For now only the -list function will work.

[anadius]

Thank you! But I guess I'll have to decode it anyway because I don't have the same version of python. The problem is there are so many files I don't know which to decompile. And when I tried to decompile them all the tool I used (uncompyle6) failed on the first one (Animation_pb2)...

 

I know that waiting for your tool is easier but I love learning new things.

[deevo]

generated.zip contains all the protobuffer files, so just decompile all of them.

For decompiling try this one unpyc37.

I use Python Script Decompiler (there is better scripts, but it works ^^) to handle unpyc, just update the unpyc3 script with the newer version.

Extract generated.zip in the Python Script Decompiler folder.

Then run the pyDecompiler script to decompile them.

 

 

A python version should be more lighweight than the C# version. 

[deevo]

Here's a first version: s4ticmd_v02.zip

 

S4TI Command Line Tool. © TeameeVo 2018-2019. Version 0.2
usage: s4ticmd -[h|l|a|u] <source> [-ls:[i|c|n|D|t|d|h|f][R]] [-f[I|P|T]:<id|persona|type>] [-o:<destination>] [-d:<dumpfile>] [-dt:[txt|json|xml]]


-l[ist] (default)          List all items of a directory or archive. If '-d' is not specified, output is displayed on screen.
-a[rchive]                 Archive all items to an archive. Require '-o'.
-u[pdate]                  Update an archive with items found in <source>. Require '-o'.
<source>                   Source path. Supports directory and archive.

-ls:[i|c|n|D|t|d|h|f][R]   Define the sort order. Supports long and short forms.
                           Append 'R' to reverse the sort order.
                           options [i[d]|c[reator]|n[ame]|[D|description]|t[ype]|d[ate](default)|h[ashtags]|f[lags]][R]
[-fI:<id>,]                Define the item ids to be filtered. Values must be hex numbers (0x0000000000000000).
[-fP:<persona|id>]         Define the user name to be filtered. Allows string or hex number (0x0000000000000000).
[-fT:<type|[1-4]>]         Define the type of item to be filtered. Allows string or number (1-4).
                           options [[household|hh|1]|[blueprint|bp|lot|2]|[room|ro|2]|[all|4](default)]

-o:<destination>           Path to destination output. Supports folder or archive.
-d:<dumpfile>              Filename to output the dump information. If not set, output is displayed on screen.

-dt:[txt|json|xml]         Define the dump output format. Require '-d'.
                           options [txt(default)|json|xml]

-h[elp]                    Show this help.

sample:

s4ticmd -l "%USERPROFILE/Documents/Electronic Arts/The Sims 4/Tray" -d:myfiles.json -dt:json

 

Sorting:

Use -ls to change the sorted column (default is date), append 'R' at the end to invert the direction.

 

Filtering:

-fI:<id,...> will only include items where the id matches one of the specified ids. Allows multiple ids separated by a comma ','.

-fP:<yourusername|id> will only include the files created by yourusername or id.

-fT:hh will only include household files. 

 

Archive:

Use -a to create a new archive and -u to update an existing archive. Both require -o:<output>.

You can omit -l and it will still output the items copied to the archive by specifying -d:<dumpfile>.

 

archive sample:

s4ticmd -a "%USERPROFILE/Documents/Electronic Arts/The Sims 4/Tray" -o:myarchive.zip

 

For now -dt is mandatory if you want to change the output format since I forgot to add detection by extension. 

[anadius]

Thank you, it works! Python Script Decompiler you sent me didn't, it produced many empty files (later I found out it's encoding problem) so I wrote my own (skipped .zip support):

import os
import glob
import unpyc37

def create_parent_dirs(file_path):
    """create parent dirs for file_path"""
    os.makedirs(os.path.dirname(file_path), exist_ok=True)

for file in glob.glob(os.path.join('**', '*.py[co]'), recursive=True):
    if file.startswith('__pycache__'):
        continue
    decompiled_file = os.path.join('decompiled', file[:-1])
    decompiled_content = unpyc37.decompile(file)
    create_parent_dirs(decompiled_file)
    with open(decompiled_file, 'w', encoding='utf-8') as f:
        f.write(str(decompiled_content))

I had to update to Python 3.7 anyway because unpyc37 uses something that's available only in this version. And replce lines like "import protocolbuffers.ResourceKey_pb2" with just "import ResourceKey_pb2". Then I installed protobuf module, which gave me lots of errors when importing decompiled *_pb2.py files. I had to install old, outdated python3-protobuf module instead. I wonder if converting those *_pb2.py to .proto and then, using up to date protoc.exe, back to *_pb2.py would work with updated protobuf module... How did you get those .proto files anyway?

 

BTW for anyone wondering, TrayMetadata from Exchange_pb2 is what you want to use to read .trayitem files.

 

 

 

Wow! It's going to be much more advanced than I thought. Didn't expect so many options! I'll use it to double check my own code.

 

[deevo]

I will had a new filter that will allow to extract only the desired items.

 

 

Il y a 10 heures, anadius a dit :

Thank you, it works! Python Script Decompiler you sent me didn't, it produced many empty files (later I found out it's encoding problem) so I wrote my own (skipped .zip support):

 Yeah the read from zip never worked for me, so I had to extract them first. I still need to learn python! 

Il y a 10 heures, anadius a dit :

I had to install old, outdated python3-protobuf module instead.

Yes they still use the an older version. 

Il y a 10 heures, anadius a dit :

BTW for anyone wondering, TrayMetadata from Exchange_pb2 is what you want to use to read .trayitem files.

That's the right one! It's easier when you don't have to write everything!

 

Now comes the difficult part, how to gather the required tray files.

Tips:

- For household, Metadata.HhMetadata.SimData give you the id of the required .sgi files, the index+1 is the group followd by 3.

  The hardest part will come from the main thumbs (.hhi) where you must guess the group. If you find the solution, I'll take it! ^^

- For blueprint, Metadata.BpMetadata.NumThumbnails give you the number of thumbnails (.bpi), and the index is the group followed by 03.

- TimeStamp is in Unix format if I remember.

 

Il y a 10 heures, anadius a dit :

I wonder if converting those *_pb2.py to .proto and then, using up to date protoc.exe, back to *_pb2.py would work with updated protobuf module... How did you get those .proto files anyway?

You may need to fix some fields here and there, but it should work since they are supposed to be back/forward compatible. The only problem may come from enums that don't start with 0.

I don't remember if there is a tool to convert back to .proto, I had to create one but never tried to convert them to the new version.

Here they are: protofiles.zip

[anadius]

You. Are. Genius! I've compiled your .proto files and they work flawlessly with protobuf! I compared serialized_pb from decompiled (from .pyc) and compiled (from .proto) versions and they are identical! EA should really just update their tools...

 

Hmm, I might try to dig into that data more but for my needs I think I just need to grab all files with a name like "*!<ID in hex>.*", don't you think?

 

You might consider sharing that _pb2.py to .proto tool, for example on Github. I think many people could find it useful. But of course it's totally understandable if you decide not to.

 

I came here to ask for a tool and not only got what I wanted but also learned a lot! So thanks again!

[deevo]

Indentical is even better! ^^

The update should work as long as you don't start to use enums that wasn't starting with 0, there was a warning about this in the doc.

But for your case I don't think that this will be an issue.

 

Il y a 3 heures, anadius a dit :

Hmm, I might try to dig into that data more but for my needs I think I just need to grab all files with a name like "*!<ID in hex>.*", don't you think?

That was how I handled the files in my first tries, until I encounter missing files.

This method will sure work for Lots and Rooms, but you may run into issue with Households where .sgi may sometime have a different id. 

Using Metadata.HhMetadata.SimData will ensure you get the right ids.

I do extra check just to make sure that the right files are selected and to detect missing files. But you're free to skip it!

 

Il y a 3 heures, anadius a dit :

You might consider sharing that _pb2.py to .proto tool, for example on Github. I think many people could find it useful. But of course it's totally understandable if you decide not to.

Sharing isn't an issue, it's more like letting others see how my coding can be messy!

 

Il y a 3 heures, anadius a dit :

I came here to ask for a tool and not only got what I wanted but also learned a lot! So thanks again!

Well, no one had ever asked before.

 

 

Update: Tool updated with archive support and allows to filter by ids.

Now that you can create one by yourself, I think I'll leave it as is, which means less work for me!

[anadius]

OOOH, now I get it. I think I get how all the files are named now.

1. .trayitem: <item type in hex>!<item ID>.trayitem
2. main file: <0 in hex>!<item ID>.<householdbinary/blueprint/room>
3. two images: <2/3 in hex>!<item ID>.<hhi/bpi/rmi>
4. additional images:

• households: <(index << 4 ) + 3 in hex>!<sim ID>.sgi where index is from 1 to length of Metadata.HhMetadata.SimData
• lots: <(index << 8 ) + 3 in hex>!<item ID>.bpi where index is from 1 to Metadata.BpMetadata.NumThumbnails -1

So it seems there can be two items, different types, with the same ID. No files will collide. Haven't tested how the game behaves in that situation yet but that shouldn't happen, there are 18446744073709551616 possible IDs.

 

Wrote a scrip based on what I wrote above and it seems to be doing fine. Compared files with those listed with your tool and they match.

 

Once again, huge thank you! I wouldn't be able to find out any of it without your help.

[deevo]

That's right! A+

 

The filename is just a ResourceKey <group>!<id>.<type>.

Here's the type ids if you ever want to store them back in a .package:

0x2a8a5e22 .trayitem
0xb3c438f0 .householdbinary
0x3bd45407 .hhi
0x56278554 .sgi
0x3924de26 .blueprint
0x???????? .room (not found yet)
0x00de5ac5 .rmi

 

3. and 4. Only the game use the extra image sizes, so you just get size 2-3.

 

Unless the .trayitem is of type AllTypes(4), the risk of collision is pretty low with such a number. 

 

That's great!

The difficult part was just getting the right .sgi files.

 

You're welcome!