Deleted User #135808
What was the worldspawn key again to make tracegrid go faster? Was it something like griscale? Oh, wait, was it gridsize?
Optimizing Lightgrid Compiles and BSP File Limits
A technical discussion on using gridsize and -nogrid to speed up tracegrid during compiles, the limitations of unique lightgrid color indices, and why the engine is hard-capped at 2GB BSP files.
Comms log
[2024-02-07 16:16] <Deleted User> what was the worldspawn key again to make tracegrid go faster
[2024-02-07 16:16] <Deleted User> was something like griscale?
[2024-02-07 16:21] <Deleted User> oh gridsize was it?
[2024-02-07 16:21] <.somaz> @Deleted User / https://github.com/SomaZ/OpenJK/blob/rend2/codemp/rd-rend2/tr_bsp.cpp#L3005
[2024-02-07 16:21] <Deleted User> tyty
[2024-02-07 17:03] <.mjt> -nogrid
[2024-02-07 17:04] <.mjt> If you just want to preview lightmapped/vertlit
[2024-02-07 17:04] <.mjt> But as compile argument during light phase
[2024-02-07 17:04] <Deleted User> gridsize was fine
[2024-02-07 17:04] <.mjt> Yeah but not for final compile
[2024-02-07 17:05] <.mjt> Just to make it fit the array
[2024-02-07 17:05] <Deleted User> idk what it is but you have a way of saying smart things in a way that my brain regularly fails to parse
[2024-02-07 17:05] <Deleted User> i feel like i should understand whaat you just said but i dont
[2024-02-07 17:07] <.somaz> I always hope that @mjt is just already thinking of something else that is related but not nesseccary to know right now, so I ignore most of it. Hoping that its not coming around later 😛
[2024-02-07 17:07] <Deleted User> lol
[2024-02-07 17:09] <Deleted User> yeah its like there is a lot of context to your statements sometimes because you have an elaborate idea in ur head, but you only write a little snippet of it in the chat and i feel lost
[2024-02-07 17:09] <.mjt> If you just need a speed boost during testing, either compile with -nogrid or a large gridsize, but for final compile it should not remain too coarse unless you don't need a lightgrid at all
[2024-02-07 17:10] <.mjt> Sorry
[2024-02-07 17:10] <Deleted User> ah. right on
[2024-02-07 17:10] <Deleted User> no worries, it was meant half as a compliment
[2024-02-07 17:10] <Deleted User> i just feel annoyed at myself for not understanding
[2024-02-07 17:10] <.mjt> Can't fit more than roughly a million grid points before the vanilla games glitch out AFAIK
[2024-02-07 17:10] <Deleted User> ah. well i'll do final compile without that value
[2024-02-07 17:11] <Deleted User> let it hash it out on its own
[2024-02-07 17:11] <.mjt> Don't it's definitely my lacklustre and compressed context lacking way of communicating D:
[2024-02-07 17:11] <.somaz> depends on how many lightgrid points are unique
[2024-02-07 17:11] <.mjt> I'll try to improve
[2024-02-07 17:11] <Deleted User> lol ic
[2024-02-07 17:11] <.mjt> In my bakes, EVERY SINGLE ONE 😄 (edited)
[2024-02-07 17:11] <.mjt> But yeah, that's true - compression and snapping of lightvalues during compile can help bring them down
[2024-02-07 17:11] <.somaz> then its 65535 unique lightgrid points for you 😄 (edited)
[2024-02-07 17:11] <.mjt> as in colors or coordinates?
[2024-02-07 17:12] <.mjt> I was more concerned about the amount of gridpoints, not colors - I though the array that stored the unique colors was restricted to 65535 but not the actual amount of points X * Y * Z
[2024-02-07 17:12] <.somaz> as in colors vector combinations
[2024-02-07 17:12] <.mjt> It's never trivial XD
[2024-02-07 17:13] <.mjt> Been so long SomaZ, I forgot it all, all the good details I once knew when you started with your plugin
[2024-02-07 17:14] <.somaz> there can be a trillion points, but just 65535 color vector variations
[2024-02-07 17:14] <.mjt> So there won't be any renderer glitches when color vector variations stay below that
[2024-02-07 17:14] <.mjt> no black out of lightmap fading due to lumps being too large?
[2024-02-07 17:14] <.mjt> Must've misremembered
[2024-02-07 17:14] <Deleted User> will a trillion points work with x86 jk?
[2024-02-07 17:14] <.mjt> or lower values present in EF1 and I'm mixing things up
[2024-02-07 17:15] <.mjt> Imagine what you could encode in the lightgrid
[2024-02-07 17:15] <.mjt> secret information
[2024-02-07 17:15] <Deleted User> nudes
[2024-02-07 17:15] <.somaz> reason is, lightgrid just stores an index in a short. Index goes to the actual lightdata
[2024-02-07 17:15] <.mjt>
[2024-02-07 17:17] <.somaz> that just happens with too big lightmap lump, because only 255 lightmaps are parsed I thought
[2024-02-07 17:17] <.somaz> But not sure right now
[2024-02-07 17:17] <.mjt> Must've been this, but I thought it also hapened on lightgrid, maybe in EF1
[2024-02-07 17:17] <.mjt> due to it being iBSP still
[2024-02-07 17:17] <.somaz> havent had this issue for a long time now
[2024-02-07 17:17] <.mjt> not comparable to RBSP
[2024-02-07 17:17] <.mjt> how would you after lightmap atlassing - you still test in vanilla? 😄 (edited)
[2024-02-07 17:18] <.somaz> ibsp doesnt index into the lightgrid data, so you arent limited. Except for disc space maybe
[2024-02-07 17:18] <Deleted User> you lied to us. one short index is 2 bytes. lump offsets are addressed as integers. so we caan have a maximum of half of 2,147,483,647 different grid points
[2024-02-07 17:18] <.mjt> wait what, iBSP is less limited than we are in that regard?
[2024-02-07 17:18] <Deleted User> dream of a trillion points destroyed.
[2024-02-07 17:19] <Deleted User> "rend2 can do it" - somaz
[2024-02-07 17:19] <.somaz> noone said anything about lump offsets lol
[2024-02-07 17:19] <.mjt> I menioned lumps when they weren't lumpy enough to fit the topic
[2024-02-07 17:20] <Deleted User> but a bsp needs to store offsets and lump lengths as integers
[2024-02-07 17:20] <.somaz> unsigned short is used as an array index. Better?
[2024-02-07 17:20] <Deleted User> yea but
[2024-02-07 17:20] <Deleted User> how do you store a trillion array indizi in a file that can't exceed 2 GB
[2024-02-07 17:20] <.mjt> why can'T it exceed 2 GB?
[2024-02-07 17:20] <Deleted User> reallymakesyouthink
[2024-02-07 17:20] <Deleted User> because integer file offset.
[2024-02-07 17:20] <Deleted User> and lump length
[2024-02-07 17:21] <.somaz> fat32 filesystem? Lol, you might want to get newer os 😄
[2024-02-07 17:21] <Deleted User> no
[2024-02-07 17:21] <.somaz> integer is 4 bytes 😄
[2024-02-07 17:21] <.mjt> You mean after that offset, you couldn'T fit any larger in it because the offset can'T be adresesd?
[2024-02-07 17:21] <Deleted User> typedef struct { / int fileofs, filelen; / } lump_t;
[2024-02-07 17:21] <Deleted User> correct
[2024-02-07 17:21] <ensiform> Files also can't be larger than 2gb in the engine
[2024-02-07 17:21] <Deleted User> same problem asa old school WAV/AVI files
[2024-02-07 17:22] <Deleted User> header writes the size/offsets as int32, not int64
[2024-02-07 17:22] <Deleted User> "who will ever need 5GB wav files, ridiculous" - engineers 40 years ago
[2024-02-07 17:22] <.mjt> that's what misc_bsp was made for 😄
[2024-02-07 17:22] <Deleted User> thinkwat
[2024-02-07 17:22] <.mjt> nah granted, engine can't handle more than 2 at all, not per bsp
[2024-02-07 17:22] <.mjt> forgot
[2024-02-07 17:22] <.mjt> not even properly with LAA?
[2024-02-07 17:24] <.somaz> well, then I lied. I'm sorry. Its just under two billion points that you can store in a bsp file. Shame on me
[2024-02-07 17:25] <Deleted User> apology accepted
[2024-02-07 17:25] <Deleted User> reluctantly.
[2024-02-07 17:25] <.mjt> @SomaZ did you further the concept of storing the external lightmapped shaders as extra shaders with nopicmip again?
[2024-02-07 17:26] <.mjt> Do you know what the requirements for the bs pstructure would be to make sure they get parsed first?
[2024-02-07 17:26] <.somaz> no, havent had any time the last few weeks. Just had time to setup autobuilds for the bsp edit thing
[2024-02-07 17:26] <.mjt> The last update was pretty cool, I could finally quickly check fog properly applied ❤️
[2024-02-07 17:27] <.somaz> first surfaces need to have the fake shaders assigned, thats all
[2024-02-07 17:27] <.mjt> as in the ones in the list I see with your tool?
[2024-02-07 17:27] <.somaz> jup
[2024-02-07 17:27] <.mjt> as long as they come before the actual ones with multistage?
[2024-02-07 17:27] <.mjt> okay
[2024-02-07 17:27] <.mjt> noice, let me see if I can pull it off manually
[2024-02-07 17:37] <.mjt> Apparently the first drawsurf comes from the first misc_model ❤️
[2024-02-07 17:38] <.mjt> Now gotta check if a broushgroup entity moved before it in .map will destroy that order
[2024-02-07 17:40] <ensiform> 2gb is a limitation also of 32 bit filesystem
[2024-02-07 17:40] <Deleted User> yeah back then
[2024-02-07 17:41] <ensiform> No, the filesystem in use is limited, we don't support large files with that minizip and legacy fopenTransmission Thread
.somaz
Check the rend2 implementation here for context: https://github.com/SomaZ/OpenJK/blob/rend2/codemp/rd-rend2/tr_bsp.cpp#L3005
.mjt
If you just want to preview lightmapped or vertex-lit results, you can use -nogrid as a compile argument during the light phase. gridsize works too, but you shouldn't use a coarse grid for final compiles unless you don't need a lightgrid at all. Just use it to make it fit the array during testing.
I believe the vanilla games start to glitch out if you exceed roughly a million grid points.
Deleted User #135808
I'll do the final compile without that value and let it hash it out on its own. To be honest, mjt, you have a way of saying smart things that my brain regularly fails to parse. I feel like I should understand what you just said, but I don't. It's like there's a lot of context in your head but only a little snippet makes it into the chat!
.somaz
It depends on how many lightgrid points are unique. There can be a trillion points, but only 65,535 unique color vector variations because the lightgrid stores an index in an unsigned short. The index points to the actual light data.
.mjt
Ah, I was more concerned about the total amount of grid points (X * Y * Z), not just the colors. I thought there was a limit that caused blackouts or lightmap fading due to lumps being too large. Maybe I'm mixing it up with EF1 or iBSP limitations.
In my bakes, every single point is unique! But yeah, compression and snapping of light values during compile can help bring that count down.
Deleted User #135808
Wait, you lied to us. One short index is 2 bytes, but lump offsets are addressed as integers. We can have a maximum of half of 2,147,483,647 different grid points. My dream of a trillion points is destroyed. How do you even store a trillion array indices in a file that can't exceed 2GB anyway? The header writes the size/offsets as int32, not int64.
ensiform
Files cannot be larger than 2GB in the engine. It's a limitation of the 32-bit filesystem handling. We don't support large files with the current minizip and legacy fopen implementation.
.mjt
That's what misc_bsp was made for! But yeah, the engine can't handle more than 2GB total, not even properly with LAA.
@SomaZ, did you further the concept of storing external lightmapped shaders as extra shaders with nopicmip? I'm trying to figure out the requirements for the BSP structure to ensure they get parsed first. I noticed the first drawsurf seems to come from the first misc_model.
.somaz
I haven't had time the last few weeks, mostly just setting up autobuilds for the BSP editor. For the shaders, you just need the first surfaces to have the fake shaders assigned. Since an unsigned short is used as the array index for the lightgrid, you're looking at just under two billion points max in a BSP file. My bad on the 'trillion' figure.