Packing Assets in the Order They're Needed

A gzip stream can only be read forward. Reaching a byte halfway in means decompressing everything before it. Reaching a byte already passed means starting over from zero.

That second case is the one that matters, and it's in my engine's streaming loader as an explicit branch:

writeLog("rewinding seek");
...
gzrewind(streamingTextureFileHandle);

An asset requested behind the read head costs a full rewind and a re-decompress of everything up to it. Request assets in a different order from the one they're packed in, and loading degrades from a walk through the file into a series of restarts.

Which makes the order assets sit in the pack a real engineering decision, and one worth a manifest of its own.

This is the ninth post taking one system at a time out of my custom engines.

The Manifest

Next to the game data sits asset-list.txt, 286 entries, one asset per line. It isn't an index of what's in the pack. It's the order the pack is built in.

The sections are these:

Reading down the file is watching the game start up. The ten level definitions and three meshes come first, because the game can't do anything without them. Fonts next, because nothing can be drawn as text before them. Sounds after that.

Then the textures, and their order is the giveaway:

textures\small\manicgames.tga
textures\small\whiteparticle.tga
textures\small\titlegrid.tga
textures\small\newswindow.tga
textures\small\criticalmasslogo.tga
textures\small\loadingbar.tga
textures\small\cursor.tga

The studio logo is first, because it's the first thing on screen. Then the title screen background, the news panel, the game logo, the loading bar, and the cursor. After those come the colored cursor variants, the menu toggles, the social icons, and only then the colored blocks the game is actually played with.

Nobody wrote that order by thinking about it in the abstract. It's a transcription of what the game asks for, in the sequence it asks.

Why the Order Is the Optimization

With the pack built this way, startup is a single forward pass. The read head advances through levels, meshes, fonts, sounds, and into the textures in the order the loader wants them, and never once has to go back.

Get the order wrong and every misordered pair is a rewind. Not a seek, a rewind, decompressing from the beginning of the file. Two assets requested out of order near the end of a large pack means decompressing that pack twice.

That asymmetry is what makes this worth doing at all. In an uncompressed archive, a backward seek is a file pointer moving and costs nothing worth measuring, so pack order is close to irrelevant. Compression buys smaller downloads and a smaller install, and the price is that random access stops being free. The manifest is what pays that price down.

The general shape: when a storage format makes one access pattern cheap and another expensive, the layout of the data becomes part of the performance work, and it's usually a cheaper lever than optimizing the reader.

The Two Tiers Are a Trick

The part I'd point at as genuinely clever is the texture section, which contains every texture twice. One hundred and twenty-seven low-resolution textures, then the same one hundred and twenty-seven at full resolution.

The two blocks are identical sets, in identical order.

That mirroring is what makes both quality settings a forward-only read.

A machine running the low-resolution path reads entries 1 through 159 and stops. Everything it needs is a contiguous prefix of the file, and the second half is never touched.

A machine running full resolution reads 1 through 32, skips the entire low-resolution block in one forward move, and reads 160 through 286 in order. One skip, still monotonic, still no rewind.

Had the tiers been interleaved, with each texture's small and full versions adjacent, both configurations would skip constantly. Had only the differing textures been duplicated, the orders would diverge and one configuration would rewind. Duplicating the whole set costs pack size, which is disk space and download, and buys a forward-only read for every configuration. That's the cost, and it's the right way round for a game people install once and load repeatedly.

Four Manifests, Not One

There are four of these files: the base list at 286 entries, a Mac list at 294, and demo versions of both, at 237 and 245.

That looks like duplication and it's the correct structure. The demo boots into a different flow, shows different screens, and doesn't contain some content at all, so its first-use order genuinely differs. The Mac build loads a different set of platform assets in a different sequence.

One shared list with conditionals would produce an order that's correct for no build in particular. A manifest per product matches the fact that first-use order is a property of the specific thing being shipped.

The Manifest Is Measured, Not Remembered

The order isn't reasoned out. It's recorded. The loader logs every request as it services it:

writeLog("streaming " + filename);

That line is there for this. Play the game through, and the log is the exact sequence in which assets were first asked for, on a real run. Take the log, turn it into the list, rebuild the pack in that order.

The transformation is manual, done in a text editor. That was the right call and I'd make it again. It's 286 entries, once per release, and a log that's already one filename per line in the correct order. A script to do it would have been maybe an hour to write and a thing to maintain forever, against a task that takes a few minutes with a keyboard and happens a handful of times a year.

Knowing which steps are worth automating is its own skill. The threshold is frequency multiplied by the cost of getting it wrong, not how automatable something looks. This one is infrequent and self-correcting: get the order slightly wrong and the next recording fixes it.

The dates say this was the design rather than a later optimisation. Both halves land on 2010-09-24, two months after the project's first commit, and they land together. The first commit that day is called "asset packing and adaptive streaming textures". The second, which finishes the job, is "adaptive streaming done, priority sorted asset file". The loader and the ordering the loader depends on were built as one thing.

That's the only order in which this works. A streaming loader bolted onto an arbitrarily ordered pack would spend its first week rewinding, and the natural fix from inside that position is to make rewinding faster rather than to make it unnecessary. The ordering has to be part of the idea from the start, because once the loader exists and is slow, the pack format looks like a constraint instead of a variable.

What I'd Add

Not automation of the generation. A check.

The weakness isn't that the list is built by hand, it's that nothing verifies the shipped list still matches what the game does. Between one release and the next, a new title-screen texture can appear and land in the middle of the gameplay section, and the only symptom is a slightly slower load that nobody attributes to a text file.

So: run the game as part of the build, capture the log, compare the observed order against the manifest, and fail on divergence. That leaves the manual step exactly where it is, and turns a silent regression into a build error telling someone to redo it.

The cheaper version of the same idea is a counter. The loader already knows when it rewinds, because it logs it. Counting rewinds during a scripted boot and failing on anything non-zero catches a bad ordering immediately, precisely, and without needing to compare two lists at all. One number, and it's zero when the pack is right.

That's what everything else in these engines that eventually improved has in common: leave the judgment with the person, and hand the enforcement to something that can't forget.

What Transfers

Ask what access pattern the storage format makes cheap, and lay the data out to match it. Compressed streams, tape, spinning disks, and network ranges all reward sequential access and punish going backward, and the punishment is usually much larger than the reward for anything else.

Order by first use, not by category. Grouping by type is how a directory is organized and has nothing to do with how the data is read.

Duplicate rather than interleave when two configurations need different subsets. Storage is cheap and a forward-only read for both is worth more than the bytes.

And derive the order from a recording rather than from reasoning. The program already knows what it asked for and when, so the ordering should be a transcript of a real run. Whether the transcript becomes the manifest by script or by text editor is a question about how often it happens, not a question about whether it's rigorous.