Working with Datasets Larger Than Memory

This guide answers one task: compute a correct result over a dataset that does not fit in a browser tab, by turning the computation into bounded passes over chunks rather than one pass over everything.

Prerequisites

  • [ ] A dataset larger than roughly 500 MB, or a tab budget smaller than you assumed.
  • [ ] A module that can process a chunk independently — or a plan to make it one.
  • [ ] OPFS available if you need to spill intermediate results.
  • [ ] A way to measure peak memory; performance.measureUserAgentSpecificMemory() where available, a heap snapshot otherwise.

Know your actual ceiling

There is no single number, and treating a guess as a limit produces code that fails only on the devices you did not test. A 32-bit WebAssembly module addresses at most 4 GB, browsers cap WebAssembly.Memory well below that in practice, and the tab’s own budget is smaller still — commonly 1–4 GB on desktop and several hundred megabytes on mobile.

The practical planning number is the working set: the bytes that must be resident simultaneously for your algorithm to make progress. If that is bounded and small, the dataset size stops mattering. If it is proportional to the dataset, no amount of tuning will save you and the algorithm has to change.

// a crude but honest check before starting a long job
const mem = navigator.deviceMemory || 4;               // GB, coarse and optional
const budgetBytes = Math.min(mem * 0.25, 1.5) * 1e9;   // leave room for everything else
if (estimatedWorkingSet > budgetBytes) useChunkedPath();
Working set, not dataset size Loading everything makes the working set equal to the dataset, which fails past the tab budget. A chunked pass keeps the working set at one chunk plus the accumulated state, which stays flat as the dataset grows. load everything 200 MB — fine 1 GB — slow, risky 4 GB — tab is gone chunked pass chunk chunk chunk accumulated state — small flat, whatever the dataset size The question to ask about any algorithm is whether its state grows with the input. Sums, counts, extrema and histograms do not; sorts and exact distinct counts do.

Streaming aggregation: the easy case

Any computation expressible as an initial state, a per-chunk update and a final result is trivially bounded. Sums, counts, minima, maxima, means, variances, histograms and approximate quantiles all qualify, and together they cover most of what dashboards actually show.

// one chunk at a time through a fixed region of linear memory
const CHUNK = 4 << 20;                                    // 4 MB
const ptr = mod.exports.chunk_ptr();
const view = new Uint8Array(mod.exports.memory.buffer, ptr, CHUNK);

mod.exports.agg_init();
const reader = stream.getReader();
let carry = new Uint8Array(0);
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  for (let off = 0; off < value.length; off += CHUNK) {
    const slice = value.subarray(off, Math.min(off + CHUNK, value.length));
    view.set(slice);
    mod.exports.agg_update(slice.length);                 // state stays inside the module
  }
}
const result = mod.exports.agg_finish();

Peak memory here is the chunk plus the module’s state, regardless of whether the source is 100 MB or 100 GB. Note the record-boundary problem: a chunk rarely ends exactly at a record boundary, so either the module must buffer the partial tail internally, or the caller must carry it into the next chunk. Getting this wrong produces results that are subtly low rather than obviously broken.

Two passes beat one big allocation

Some computations need a value that is only known after seeing all the data — a global mean before computing variance, a maximum before normalising, a dictionary before encoding. The instinct is to hold everything so you can do it in one pass. The alternative is to read the data twice.

Reading twice is usually cheaper than it sounds, especially when the source is local or cached, and it turns an impossible memory requirement into a bounded one. Pass one computes the statistic; pass two uses it. For streamed remote data, cache the bytes in OPFS during the first pass so the second pass reads from disk rather than the network.

Some statistics avoid the second pass entirely with the right algorithm — Welford’s method computes variance in one pass without holding the data, and reservoir sampling produces a uniform sample of unknown-length input with fixed memory. Reaching for the numerically stable streaming algorithm is almost always better than reaching for more memory.

Spilling intermediate results

When per-chunk output must be retained — a sort, a join, a group-by with many keys — write it somewhere other than linear memory. OPFS is the natural destination, and the pattern mirrors an external merge sort: produce sorted runs that each fit in memory, write them out, then merge the runs with one small buffer per run.

const root = await navigator.storage.getDirectory();
async function writeRun(i, bytes) {
  const fh = await root.getFileHandle(`run-${i}.bin`, { create: true });
  const access = await fh.createSyncAccessHandle();       // worker context
  access.write(bytes, { at: 0 });
  access.flush(); access.close();
}

The merge phase holds one buffer per run, so memory is proportional to the number of runs rather than the data. With 4 MB buffers and sixty-four runs that is 256 MB — comfortable, and it sorts a dataset far larger than the tab could ever hold. Delete the runs when the job finishes, including on the error path; orphaned spill files are a common way for an origin to fill its quota.

Sorting more than fits Each pass fills memory, sorts, and writes a sorted run to storage. The merge phase reads a small buffer from each run and emits the global order, so memory depends on the number of runs rather than on the dataset size. phase 1 — produce sorted runs fill · sort · write fill · sort · write fill · sort · write each run fits in memory by construction phase 2 — merge buf 1 buf 2 buf 3 globally ordered output stream Memory is buffers × runs, not dataset size — the same trick databases have used since tape drives, and it still works in a tab.

Keeping the interface responsive

A bounded-memory job is still a long job, and a long job on one thread is a frozen page. Run it in a worker, report progress per chunk, and make cancellation real — a user who started a ten-minute aggregation over the wrong file needs to stop it.

let cancelled = false;
self.onmessage = ({ data }) => { if (data === 'cancel') cancelled = true; };

for (const chunk of chunks) {
  if (cancelled) { cleanupSpillFiles(); self.postMessage({ cancelled: true }); return; }
  processChunk(chunk);
  self.postMessage({ progress: done / total });
}

Cancellation must clean up. Spill files left behind by a cancelled job consume quota indefinitely and will eventually cause a QuotaExceededError in an unrelated part of the application, at which point the connection to the cancelled job is invisible.

Expected output

Instrument peak memory alongside the result, because the whole point is that it stays flat:

chunk 1/240   heap 112 MB   elapsed 0.4 s
chunk 120/240 heap 118 MB   elapsed 47 s
chunk 240/240 heap 119 MB   elapsed 94 s
result: 1,204,558,213 rows, sum = 8.41e12, peak heap 121 MB

A heap number that climbs with chunk index means something is retained per chunk — a growing array of results, an unreleased view, a closure captured in an event handler. Flat is the pass; monotonically increasing is the bug.

Three ways to not hold it all Loading everything fails outright. Paging keeps a working set resident. Streaming aggregation holds only the accumulator, and is the only one whose memory does not depend on the data at all. load everything allocation fails — the dataset is larger than the address space allows page cache fixed working set evicts by least-recently-used; random access stays possible streaming aggregation accumulator memory is constant, but only one pass is available Pick by access pattern: streaming for a single pass, a page cache when the query jumps around. Either way the bound is a decision, not an emergent property — choose the number and enforce it.

Gotchas

  • Records split across chunk boundaries. Buffer the tail explicitly. Undetected, this drops a record per chunk and produces results that are plausibly wrong.
  • Growing memory mid-job. Preallocate the chunk region and never allocate inside the loop, or every view you hold is periodically detached.
  • Progress computed from bytes when work is per record. Variable-length records make byte progress non-linear; report both if the difference is visible.
  • Spill files never deleted. Clean up on success, error and cancellation, and sweep old files at startup.
  • Assuming deviceMemory is accurate. It is a coarse hint, absent on several browsers, and it describes the device rather than what your tab may use. Treat it as a tiebreaker, not a limit.

Performance note

A chunked pass over 40 GB of local Parquet on a laptop sustained roughly 420 MB/s with a 4 MB chunk and a SIMD kernel, with peak heap of 121 MB — bounded by the chunk and the accumulator, exactly as designed. Raising the chunk to 64 MB improved throughput by about 6% and raised peak memory by 60 MB, which is a poor trade. Chunk size mostly affects syscall and boundary overhead, and past a few megabytes there is very little left to win.

Frequently Asked Questions

Is this not what a database is for? Often, yes — an analytical engine already implements streaming aggregation and spilling, and using one is usually less work than writing this yourself. Write it by hand when the computation is not expressible in SQL, or when you need a specific numerical algorithm.

Can I use threads to speed up a chunked pass? Yes, when chunks are independent: give each worker a chunk range and combine the partial states at the end. Combining requires an associative merge, which sums, counts and histograms all have.

What if the algorithm genuinely needs everything at once? Then the honest answers are to sample, to approximate, or to move the computation to a server. Exact distinct counts and full sorts of arbitrary data are the usual examples — and approximate sketches solve the first one well enough for almost every product question.

How do I choose the chunk size? Start at 4 MB and measure. Too small and per-chunk overhead — the boundary crossing, the tail buffering, the progress message — starts to dominate; too large and peak memory rises for no throughput gain. The curve is flat across a wide middle, so anything from 1 MB to 16 MB is usually defensible.

Does memory64 solve this? It raises the address ceiling, not the tab’s budget, so a module can address more than 4 GB while the browser still refuses to give it that much. It helps a specific class of large in-memory workload and changes nothing about the discipline described here.

← Back to Databases & Persistent Storage in Wasm