Databases & Persistent Storage in Wasm

A database compiled to WebAssembly turns the browser into a place where real queries run: joins, indexes, transactions, aggregates over millions of rows, with no network in the loop. That capability changes what a client application can be — offline-first tools, local analytics, instant filtering over data that would be painful to paginate from a server. It also introduces problems that server databases solved decades ago and browsers have only recently made solvable at all: durability, concurrency, and what happens when the storage layer underneath you is a key-value store pretending to be a disk.

Prerequisites

  • [ ] A build of the engine: @sqlite.org/sqlite-wasm, sql.js, or @duckdb/duckdb-wasm.
  • [ ] A secure context — the Origin Private File System and most storage APIs require it.
  • [ ] Cross-origin isolation if you plan to use the synchronous OPFS access handle from a worker in engines that require SharedArrayBuffer.
  • [ ] A realistic dataset. Everything is fast with a thousand rows.

What “a database in the browser” actually is

The engine is ordinary C compiled to WebAssembly. SQLite’s virtual file system layer is the part that makes this work: instead of calling POSIX file operations, the compiled build calls into a VFS implementation you choose, and that implementation decides where pages actually live. Swap the VFS and the same engine goes from in-memory to durable without a line of SQL changing.

That indirection is the whole story of browser databases. An in-memory VFS gives you a fast database that vanishes on reload. A VFS over IndexedDB gives you durability with awkward performance because every page read crosses an asynchronous boundary. A VFS over the Origin Private File System with synchronous access handles gives you something close to real file semantics, which is why current builds recommend it and why it requires a worker.

One engine, three storage backends The compiled SQL engine calls a virtual file system layer rather than the operating system. Choosing memory, IndexedDB or the Origin Private File System changes durability and speed without changing any SQL. SQL engine compiled to Wasm parser · planner · b-tree · pager virtual file system layer memory fastest, no durability good for analysis sessions IndexedDB durable, async, slow pages legacy fallback OPFS access handle durable, synchronous, fast worker only — the default now

Opening a database and running a query

The modern SQLite build wraps all of this behind a small API. The critical decision is made in one line: which VFS the database is opened with.

import sqlite3InitModule from '@sqlite.org/sqlite-wasm';

const sqlite3 = await sqlite3InitModule();
const db = 'opfs' in sqlite3
  ? new sqlite3.oo1.OpfsDb('/app/data.sqlite3')     // durable, worker context
  : new sqlite3.oo1.DB('/data.sqlite3', 'ct');      // memory fallback

db.exec(`
  CREATE TABLE IF NOT EXISTS events (
    id    INTEGER PRIMARY KEY,
    ts    INTEGER NOT NULL,
    kind  TEXT NOT NULL,
    payload TEXT
  );
  CREATE INDEX IF NOT EXISTS events_ts ON events(ts);
`);

db.exec({
  sql: 'INSERT INTO events (ts, kind, payload) VALUES (?, ?, ?)',
  bind: [Date.now(), 'click', JSON.stringify({ x: 12, y: 44 })],
});

Everything after that line is ordinary SQL, which is the point. The engine’s behaviour — transactions, constraints, query planning — is the same code that runs on a server, so knowledge transfers and so do the tools.

Transactions are what make it a database

The reason to use a database rather than a serialised object graph is not query syntax; it is that a transaction either happens completely or not at all. In the browser this matters more than on a server, because a tab can be closed mid-write at any moment and nobody gets a shutdown hook they can trust.

Wrap every logical unit of work in a transaction, and batch inserts aggressively — a thousand individual inserts each in their own implicit transaction can be a hundred times slower than the same inserts inside one explicit transaction, because each commit forces a durable write.

db.transaction(() => {
  const stmt = db.prepare('INSERT INTO events (ts, kind, payload) VALUES (?, ?, ?)');
  try {
    for (const e of batch) { stmt.bind([e.ts, e.kind, e.payload]).stepReset(); }
  } finally { stmt.finalize(); }
});

Finalising statements is not optional hygiene. An unfinalised prepared statement holds a lock and leaks memory inside linear memory, and a page that prepares in a loop without finalising will eventually fail to write with an error about the database being locked.

Schema design for a client database

A schema that works on a server is not automatically right in a tab. Three differences change the design.

The first is that there is exactly one user. Multi-tenant columns, row-level security scaffolding and most of the auditing machinery that server schemas carry are dead weight here, and dropping them makes queries simpler and the file smaller. What replaces them is a notion of local versus synced state: which rows came from the server, which were created locally and not yet acknowledged, and which have been modified since their last sync. That distinction is most cleanly expressed as columns on every synced table rather than as a separate log, because it survives an interrupted session without reconciliation.

The second is that storage is not free and not guaranteed. A server keeps history because disks are cheap; a tab should keep only what the application needs to work, with a retention policy that prunes old rows on startup. A table that accumulates one row per user interaction will reach hundreds of megabytes on an engaged user’s machine, at which point eviction becomes likely and the whole database is at risk, not just the old rows.

The third is that schema migration happens on a machine you cannot inspect. Version the schema with PRAGMA user_version, apply migrations in order inside a transaction on open, and make every migration tolerant of having been partially applied — because a user closing the tab mid-migration is an ordinary event rather than an incident.

const CURRENT = 4;
db.transaction(() => {
  let v = db.selectValue('PRAGMA user_version');
  if (v < 1) { db.exec(MIGRATIONS[1]); v = 1; }
  if (v < 2) { db.exec(MIGRATIONS[2]); v = 2; }
  if (v < 3) { db.exec(MIGRATIONS[3]); v = 3; }
  if (v < 4) { db.exec(MIGRATIONS[4]); v = 4; }
  db.exec(`PRAGMA user_version = ${CURRENT}`);
});

Concurrency: one writer, and browsers make that awkward

SQLite allows many readers and one writer. A browser adds a complication the engine was never designed for: several tabs of the same origin, each with its own instance, each believing it owns the file.

The OPFS synchronous access handle gives exclusive access to a file, which resolves the ambiguity by making the second tab fail to open rather than corrupt anything. That is safe and often surprising — users open second tabs constantly. The robust pattern is to designate one owner: run the database in a shared worker, or elect a leader tab through the Web Locks API and have the others proxy their queries to it.

await navigator.locks.request('db-owner', { mode: 'exclusive' }, async () => {
  await runDatabaseForTheLifetimeOfThisTab();     // other tabs queue behind this
});

Deciding this early is important, because retrofitting a single-owner architecture onto code that assumed direct access touches every call site.

Durability and the storage the browser might reclaim

Browser storage is evictable. Under pressure, a user agent may clear an origin’s data, and the default policy offers no promise. For an application whose data only exists locally, that is a data-loss bug waiting for a busy disk.

if (navigator.storage?.persist) {
  const persisted = await navigator.storage.persist();      // may prompt, may be granted silently
  const { quota, usage } = await navigator.storage.estimate();
  console.log({ persisted, usageMB: (usage / 1e6) | 0, quotaMB: (quota / 1e6) | 0 });
}

Request persistence, check the result, and design for the answer being false. That usually means a sync path to a server, or at minimum an export the user can trigger. Treating a browser database as the sole copy of anything irreplaceable is a product decision, not a technical one, and it should be made deliberately.

Analytical engines are a different shape

SQLite is a transactional row store; DuckDB compiled to WebAssembly is a columnar analytical engine, and the difference shows up immediately in what each is good at. Scanning a million rows to compute a grouped aggregate is a natural fit for the columnar engine and a slog for the row store. Updating one record by primary key is the reverse.

The analytical engine also changes where data comes from. DuckDB can read Parquet files over HTTP with range requests, pulling only the columns and row groups a query touches — which means a 2 GB dataset on a CDN can be queried from a browser that downloads 12 MB. That capability, covered in querying Parquet files with DuckDB-Wasm, is often the reason to choose it.

Rows or columns, depending on the question A row store keeps a record's fields together, which suits point lookups and updates. A columnar store keeps each field's values together, which suits scanning one column across millions of records. row store — fields together id · ts · kind · payload (record 1) id · ts · kind · payload (record 2) one seek returns a whole record — ideal for lookups and updates columnar — values together all ids all ts all kinds an aggregate reads one stripe — the other columns are never touched Compression works far better on a column of similar values, which is why a columnar file is often a quarter the size of the equivalent rows. Use both when it fits: a transactional store for application state, and a columnar file for the analytics view over it. Neither choice is reversible cheaply once a schema has users, so it belongs in the first design conversation.

Memory: the database lives inside linear memory

Every page the engine caches, every result set it materialises and every temporary table it builds lives in linear memory. That has two consequences worth planning for.

The first is that the page cache is a tuning knob that directly trades tab memory for query speed. SQLite’s default cache is small; raising it with PRAGMA cache_size makes repeated queries dramatically faster and makes the tab correspondingly larger. The second is that a query returning a million rows materialises a million rows — inside the module, and again on the JavaScript side if you convert them to objects. Streaming results, or aggregating in SQL rather than in JavaScript, is what keeps that from becoming the dominant cost.

-- do the work where the data already is
SELECT kind, COUNT(*) AS n, AVG(duration) AS avg_ms
FROM events WHERE ts > ?1 GROUP BY kind ORDER BY n DESC LIMIT 20;

Twenty rows crossing the boundary instead of a million is not an optimisation; it is the difference between a feature that works and one that freezes the tab.

Query performance is still query performance

Everything known about indexing applies unchanged, with one amplifier: there is no server to absorb a bad plan, so a missing index shows up as a frozen interface rather than as a slow dashboard.

Use the planner rather than guessing. EXPLAIN QUERY PLAN works exactly as it does elsewhere, and a SCAN where you expected a SEARCH is the same warning sign:

EXPLAIN QUERY PLAN
SELECT kind, COUNT(*) FROM events WHERE ts > ? GROUP BY kind;
-- SEARCH events USING INDEX events_ts (ts>?)      good
-- SCAN events                                      missing index

Two habits matter more in a browser than on a server. Prepare statements once and reuse them — the parse and plan cost is a much larger fraction of a small query here, and re-preparing inside a render loop is a common accidental cost. And keep result sets small by aggregating in SQL, because every row that crosses into JavaScript is converted, allocated and eventually collected.

ANALYZE is worth running after a bulk import, since the planner’s statistics otherwise reflect an empty table and it will happily choose the wrong index for the rest of the session. It takes milliseconds on a client-sized dataset and occasionally halves a query’s cost.

Shipping the engine without hurting first load

The engine is a payload like any other: roughly 700 kB to 1 MB compressed for SQLite with OPFS support, and several megabytes for an analytical engine. It should be fetched lazily, when the user reaches the part of the application that needs data, and cached so subsequent visits skip the download. The compiled module caching techniques apply unchanged.

If the database is central to the product — an editor, an offline tool — loading it during startup is defensible, but it should still be parallel with everything else rather than blocking first paint.

Where the bytes actually live A database compiled to Wasm still needs somewhere to keep its pages. The choice of backing decides durability, speed and how much data fits. in linear memory fastest, capped by the heap, gone on reload IndexedDB durable, asynchronous, and slow for page-sized writes OPFS sync access handles durable and synchronous, worker only HTTP range requests read-only, no local copy, one round trip per page OPFS is the right default for a real database; IndexedDB is the fallback where it is unavailable. Range requests suit a large read-only dataset nobody should have to download in full.

Gotchas and failure modes

  • SQLITE_IOERR or “unable to open database file” in a tab. The OPFS access handle is exclusive and another tab holds it. Elect a single owner rather than retrying.
  • Everything works in development, nothing persists in production. The database was opened with the memory VFS because the OPFS check silently failed — usually an insecure context or the main thread rather than a worker.
  • Writes are extremely slow. Each statement is committing separately. Wrap batches in one explicit transaction.
  • The tab grows without bound. Unfinalised statements, an oversized page cache, or result sets materialised as JavaScript objects and retained.
  • Storage disappeared after a week. Persistence was never requested or was denied. Check navigator.storage.persisted() and design a sync or export path.
  • Queries are fast, the UI still janks. The database is on the main thread. Move it to a worker; this is also required for the fast storage path.

Verifying durability, not just correctness

The test that matters for a browser database is not “does the query return the right rows” but “is the data still there after the browser did something hostile”. Automate the hostile parts:

// in a headless browser test
await page.evaluate(() => insertFixture());
await page.reload();
const rows = await page.evaluate(() => countRows());
expect(rows).toBe(FIXTURE_SIZE);        // survived a reload

await context.close();                   // and a full context teardown

Extend that to a crash simulation by terminating the worker mid-transaction and reopening: a correct setup rolls back cleanly and loses only the uncommitted work. Discovering that it does not is far better in CI than in a support ticket about missing data.

Guides in this topic

Frequently Asked Questions

Is a browser database faster than IndexedDB? For anything involving queries, yes, substantially — IndexedDB has no query planner, no joins and no aggregates, so equivalent work happens in JavaScript over cursors. For simple key-value access IndexedDB is simpler and requires no download.

How much data can I realistically store? Hundreds of megabytes is routine on desktop, with quotas typically a percentage of free disk. Mobile is tighter and more aggressive about eviction. Always read navigator.storage.estimate() rather than assuming a fixed number.

Can several tabs share one database safely? Only through a single owner. Either a shared worker that all tabs message, or leader election with Web Locks. Concurrent direct access to the same OPFS file is not supported and the failure modes are ugly.

What happens to an open transaction if the tab is closed? It is rolled back on the next open, exactly as a server database recovers from a crash — provided the storage layer is durable. With the memory VFS there is nothing to recover.

Should I encrypt the database file? Only if the threat model calls for it, and understand what it buys. Anyone with access to the device can also read the key your JavaScript uses, so encryption at rest protects against casual inspection of the storage directory rather than against a determined local attacker. For genuinely sensitive data, the right answer is usually to keep less of it locally.

How do I ship an initial dataset with the application? Build the database file offline, serve it as a static asset, and import it on first run — either by writing the bytes into OPFS directly or by using the engine’s deserialise API. That is far faster than running thousands of inserts on the user’s machine, and it makes the first-run state identical for everyone.

Can I inspect the database during development? Yes. Export the file from OPFS to a download and open it in any SQLite tool, which is the fastest way to diagnose a schema or data problem. Keeping that export behind a debug flag in production is also a cheap support tool.

← Back to Production Wasm: Workloads & Deployment