Cleaning PK3s: Using q3map2 to Remove Unused Assets

[2023-06-18 14:10] <Deleted User> anyone know of a tool that can analyze a .pk3 and search for unused textures to remove them?
[2023-06-18 14:10] <Deleted User> for a map specifically
[2023-06-18 15:57] <xycaleth> i thought of making one ages ago. it would have worked the other way around by creating a pk3 based on the map you give it. never got around to doing it though :/
[2023-06-18 16:14] <garuxx> -pk3 of q3map2 does texture part fine for jka afaik
[2023-06-18 16:19] <.mjt> @Deleted User you'd also have to take into account shader remapping that might get executed from script context
[2023-06-18 16:20] <.mjt> or any other esoteric scenario that is technically possible albeit completely implausible
[2023-06-18 16:22] <.mjt> @Deleted User enumerate all paths of files with an image extension - then parse .bsp (both entitiy keys as well as shader lump), .efx, .shader, .menu files and have whitelists for stuff like the levelshots or otherwise to be expected paths wildcards to not be purged
[2023-06-18 16:23] <.mjt> Repacking a new Pk3 with q3map2 for the basics like shaders and textures is a good first start, then just drag in the missing assets like scripts, and other missed folders
[2023-06-18 16:23] <Deleted User> sweet! how does the exclude file creation work? also getting "weird or missing extension in shader image path" a lot, is that a problem?
[2023-06-18 16:24] <Deleted User> thats what im thinking
[2023-06-18 16:24] <Deleted User> but atm it looks like it might be including a lot of stuff that base jka already has, maybe would be nice to exclude
[2023-06-18 16:26] <Deleted User> and, whatss difference between -pk3 and -repack?
[2023-06-18 16:26] <Deleted User> repack seems more powerful with shaders from description?
[2023-06-18 16:27] <garuxx> ye, exclusion for jka is missing / the way i created was -dbg to dump base shader names, pk3 resources copied via total commander
[2023-06-18 16:28] <Deleted User> so, i set it to a vanilla directory and hit -dbg?
[2023-06-18 16:28] <garuxx> -pk3 -dbg
[2023-06-18 16:28] <garuxx> with fresh install
[2023-06-18 16:28] <Deleted User> what about shaderlist, will it be ignored or do i have to include all of the vanilla .shaders? (edited)
[2023-06-18 16:28] <garuxx> -pk3 reads all shaders
[2023-06-18 16:28] <Deleted User> ok excellent
[2023-06-18 16:28] <Deleted User> lemme try
[2023-06-18 16:29] <garuxx> "weird or missing extension in shader image path" likely not problem, normally there is extension, which is not the case in jka i assume
[2023-06-18 16:30] <Deleted User> i think its just ignored by the game mostly so some are wrong
[2023-06-18 16:30] <Deleted User> possibly
[2023-06-18 16:30] <Deleted User> or missing altogether
[2023-06-18 16:30] <Deleted User> -dbg.bsp: No such file or directory
[2023-06-18 16:30] <Deleted User> im prolly stupid
[2023-06-18 16:31] <Deleted User> do i have to specify a bsp file even for -dbg?
[2023-06-18 16:31] <garuxx> yes
[2023-06-18 16:33] <Deleted User> ok not sure what to make of the -dbg output
[2023-06-18 16:33] <Deleted User> it just seems to list all textures?
[2023-06-18 16:34] <garuxx> easy way to get all shader names
[2023-06-18 16:35] <Deleted User> but my bsp has non-vanilla ones too
[2023-06-18 16:35] <Deleted User> i thot it would output all vanilla ones for .exclude file
[2023-06-18 16:35] <garuxx> it does
[2023-06-18 16:36] <Deleted User> but ... it shows texturess like textures/mymapname/whatever which clearly arent vanilla
[2023-06-18 16:37] <Deleted User> despite pointing at vanilla folder
[2023-06-18 16:37] <Deleted User> q3map2 -game ja -fs_basepath "path\GameData\base-extracted" -pk3 -dbg mapname.bsp > test.txt
[2023-06-18 16:38] <garuxx> eh shows in which section (edited)
[2023-06-18 16:39] <Deleted User> [attachment: https://cdn.discordapp.com/attachments/1103343199113711766/1120029966294319215/test.txt?ex=6a02aef1&is=6a015d71&hm=c955d5accc732ba615be6425eeee0d460806d3223b86efdd0ddb1cfcbcfe2f3e&]
[2023-06-18 16:39] <garuxx> no need to extract btw
[2023-06-18 16:39] <Deleted User> ah. i had already did it because its easier for me to work with
[2023-06-18 16:41] <garuxx> entering sections
[2023-06-18 16:41] <garuxx> using simpler bsp would help to not confuse yourself
[2023-06-18 16:42] <garuxx> will need to clean this dump with regex it seems
[2023-06-18 16:42] <Deleted User> oh ok so entering sections shows the shaderss
[2023-06-18 16:43] <Deleted User> what about textures? those are more important to exclude vanilla ones
[2023-06-18 16:43] <Deleted User> i mean the ones without shader
[2023-06-18 16:43] <garuxx> these i did via total commander
[2023-06-18 16:44] <Deleted User> oh right. i can just use dupeguru for that i guess
[2023-06-18 16:44] <Deleted User> or yea that
[2023-06-18 16:44] <Deleted User> total commander has easy way to list files?
[2023-06-18 16:46] <garuxx> yes
[2023-06-18 16:46] <garuxx> i did like search *.jpg *.tga, select all, ctrl+c
[2023-06-18 16:47] <garuxx> then some find/replace stuff
[2023-06-18 16:48] <Deleted User> hmm ok ill try some
[2023-06-18 16:48] <Deleted User> the .exclude file has to include all files/shaders to be excluded, one per line?
[2023-06-18 16:49] <Deleted User> and, is -repack same as -pk3 except that it consolidates shaders?
[2023-06-18 16:49] <garuxx> existing file is a good example
[2023-06-18 16:49] <Deleted User> so if my excluude file has / gfx/2d/cropCircle2 / gfx/2d/cropCircleGlow / gfx/2d/disruptorMask / gfx/2d/droid_view / gfx/2d/insertTick / gfx/2d/jsense / gfx/2d/lagogglesWindow / gfx/2d/lock / gfx/2d/side_bit / gfx/2d/watermark /  / etc thats correct?
[2023-06-18 16:49] <garuxx> yes
[2023-06-18 16:51] <garuxx> rips shaders, also extra exclusion file to exclude what's in current repack, and different logic in some points
[2023-06-18 16:53] <Deleted User> so, if i omit the extra exclude file it should do same thing?
[2023-06-18 16:53] <Deleted User> cuz it sounds nice if it also throws away unused shaders
[2023-06-18 16:54] <garuxx> i don't recall all the details, but you can rip shaders with it
[2023-06-18 16:54] <Deleted User> ok ill test
[2023-06-18 16:54] <Deleted User> ty
[2023-06-18 16:56] <Deleted User> damn i got the list in total commander just no idea how to save it
[2023-06-18 16:56] <Deleted User> 🤣
[2023-06-18 16:58] <Deleted User> ill figure this out.
[2023-06-18 16:58] <garuxx> ctrl+c
[2023-06-18 16:59] <garuxx> or can actually ctrl+b, alt++, f12
[2023-06-18 17:07] <Deleted User> i ended up doing it in some lame other way but ty
[2023-06-18 17:08] <Deleted User> bit dirty (includes all sorts of files under "textures") but seems to work if anyone has use for it / [attachment: https://cdn.discordapp.com/attachments/1103343199113711766/1120037311036539030/ja.exclude?ex=6a02b5c8&is=6a016448&hm=2a93f0e2e311f51868b2cbffbfa94245b33e02b27387904514b444632af6aeb5&]
[2023-06-18 17:08] <Deleted User> has videos sounds etc
[2023-06-18 17:17] <Deleted User> @niger is there a way for it to include qer_editorimage ones too?
[2023-06-18 17:18] <Deleted User> when repacking or pk3ing
[2023-06-18 17:18] <garuxx> no
[2023-06-18 17:18] <Deleted User> what about lightimage?
[2023-06-18 17:19] <Deleted User> havent checked if theyre included
[2023-06-18 17:20] <garuxx> no
[2023-06-18 17:22] <Deleted User> would be nice feature. but already pretty useful
[2023-06-18 17:23] <.mjt> @Deleted User python based JA Assets packer / repacker might be more viable (edited)
[2023-06-18 17:23] <.mjt> for all the other unique filetypes this game has and would require parsing to make sure nothing is missed
[2023-06-18 17:24] <.mjt> adding all of that stuff in a manner that would be extendable for any supported game in NRC might be too much
[2023-06-18 17:24] <garuxx> its intent is creation of release pk3, nothing like caulk shaders, lightimages are possible, but it would be just a bit of what's needed for 'map source bundler'
[2023-06-18 17:26] <garuxx> and the latter should be done the other way around: reading .map, parsing models etc
[2023-06-18 17:26] <Deleted User> mhm
[2023-06-18 17:26] <garuxx> parsing shader remaps too
[2023-06-18 17:30] <garuxx> np to include good modular optional code to handle jka stuff, at least if i imagine those jka formats right / big problem is model format, which is not present in assimp
[2023-06-18 17:30] <.mjt> you mean .glm?
[2023-06-18 17:30] <.mjt> corner cases have glm without .skin files that's true
[2023-06-18 17:30] <.mjt> otherwise you'd get shadernames from .skin file for .glm
[2023-06-18 17:30] <garuxx> glm ye
[2023-06-18 17:31] <.mjt> could hardcode the few cases where no .skin is present - and no one else uses those anymore thankfully (edited)
[2023-06-18 17:31] <.mjt> as in makes assets that way
[2023-06-18 17:47] <Deleted User> well thats funny. ultimately i ended up with a bigger archive than before
[2023-06-18 17:47] <Deleted User> 🤣
[2023-06-18 17:51] <Deleted User> seems it included textures of playermodels that i included as md3 files
[2023-06-18 17:51] <Deleted User> even tho jka has them
[2023-06-18 17:52] <garuxx> dirty exlusion file likely does work dirtily
[2023-06-18 17:53] <Deleted User> everything else worked fine tho, only the files in models/players are messy
[2023-06-18 17:53] <Deleted User> image files
[2023-06-18 17:54] <Deleted User> ah no
[2023-06-18 17:54] <Deleted User> might be the whole models folder (edited)
[2023-06-18 17:54] <Deleted User> oh .... oh no
[2023-06-18 17:54] <Deleted User> hmm
[2023-06-18 17:55] <Deleted User> so in the textures { section
[2023-06-18 17:55] <Deleted User> do i do
[2023-06-18 17:55] <Deleted User> textures\imperial\switch_elevator_grn.jpg / textures\imperial\switch_elevator_grn_glw.jpg / textures\imperial\switch_elevator_red.jpg / textures\imperial\switch_elevator_red_glw.jpg / textures\imperial\switch_hangar_locked.jpg / textures\imperial\switch_hangar_locked_anim.png / textures\imperial\switch_hangar_locked_glw.jpg
[2023-06-18 17:55] <Deleted User> or without the textures\
[2023-06-18 17:55] <Deleted User> or is backslash wrong altogether?
[2023-06-18 17:56] <garuxx> the way it is in working q3.exclude file is a working way
[2023-06-18 18:00] <Deleted User> omg...
[2023-06-18 18:00] <Deleted User> i have to remove .jpg and .png etc 🤣
[2023-06-18 18:00] <Deleted User> nice touch
[2023-06-18 18:03] <Deleted User> fixeroo'd / [attachment: https://cdn.discordapp.com/attachments/1103343199113711766/1120051206002311319/ja.exclude?ex=6a02c2b9&is=6a017139&hm=67bfd52489d68af6813514bb898d7e73efd3218c6b3d33ffb9a89ba2eee6c395&]
[2023-06-18 18:09] <Deleted User> ok around 40 MB saved now. decent.
[2023-06-18 18:14] <Deleted User> i hate python. if i was coding it, i'd use c#
[2023-06-19 03:21] <tayst> I think it still won’t work with some file types but it will work with most
[2023-06-19 03:21] <tayst> I made one of these a while ago too
[2023-06-19 03:21] <tayst> I also made mp_entities.ent but only completed that like 70% and didn’t check it against code so it was kinda meh
[2023-06-19 03:22] <tayst> Would be really cool if someone did a complete .ent file for jk2/a sp/mp
[2023-06-19 03:22] <tayst> With all the right ents, keys and flags, bounding box sizes etc
[2023-06-19 08:13] <.somaz> I hate c#. If I was coding it, I'd use python
[2023-06-19 13:37] <Deleted User> lol cheers
[2023-06-19 13:37] <Deleted User> well judging by the erorr message i got the first time it only supports a number of keys
[2023-06-19 13:38] <Deleted User> like textures,shaders,shaderfiles,sounds,videos or sth like that
[2023-06-19 13:38] <Deleted User> i think i got them all in there now
[2023-06-19 13:38] <Deleted User> altho im not sure if i have to remove the file endings for sounds/videos too for it to recognize them (edited)
[2023-06-19 14:59] <tayst> ye it wont recognize some stuff still tho like rof and icarus
[2023-06-19 14:59] <tayst> thats all i meant
[2023-06-19 15:00] <tayst> [attachment: https://cdn.discordapp.com/attachments/1103343199113711766/1120367499477799082/ja.exclude?ex=6a0297cb&is=6a01464b&hm=da932f2765d9983b2328ae6d296baa2217ae384568723a675ad8a44b859f723a&]
[2023-06-19 15:00] <tayst> this was my version but i havent looked at it in ages so not sure if its correct
[2023-06-19 15:20] <Deleted User> ya its kinda limited in what it can do