s4ql

An SQL database for Go that stores everything in S3. Written by agents, trusted with simulation testing.

s4ql is an embedded database engine in the spirit of SQLite. Your Go application talks to the bucket directly, and there is no server to run. Every line of it is written by AI agents, and no human reviews the code. Trust has to come from verification.

  • Experiment
  • Storage built
  • SQL not built
  • Not ready for use

Why it exists

Two experiments in one repository. One is a database that needs nothing but object storage. The other is whether software that nobody reads can still be trusted.

Nothing to operate
Your Go application talks to S3 directly through the module. There is no server, no cluster, and no replica to look after. The bucket is the only infrastructure and the source of truth.
A database per user
Each database is a key prefix, so one bucket holds as many databases as you like. One database per user costs nothing extra, and write-heavy work spreads across them.
SQLite's SQL, strictly
s4ql speaks a strict subset of SQLite's SQL. Whatever it accepts behaves exactly as in SQLite. Everything else is rejected, never reinterpreted.
Trust without reading
AI agents write and own the code. Markus sets the goals and takes part in design and architecture; he does not review code. The bet is that a database can earn trust when its correctness is checked, not read.

What it does

A database in S3 is a log. Each commit is an immutable object named by the SHA-256 of its content and linked to its parent, as in git. One small file, manifest.json, points at the newest commit. Swapping that pointer is the commit.

manifest.json

{
  "version": 0,
  "branches": {
    "main": {
      "head": "d39febcf6aba…"
    }
  },
  "lease": {
    "holder": "5b0e3a4f9c1d77c3",
    "duration": "30s",
    "renewed": "2026-11-12T12:00:30.123Z",
    "salt": "c41d07a9e2b35f68"
  }
}

A compare-and-swap replaces this file. The transaction is committed when the swap succeeds, and not before.

  1. d39febcf6abaf7ee47aea6f615ae6e8be31d10f1c7ba4f3c8133917f63981295
    insert into t1 (v1) values (2);Parent b2116a98…
  2. b2116a9830d36bf1e6397287f9fab487243dd15484685c6e877f29376fb0e7da
    insert into t1 (v1) values (1);Parent 2f4e6858…
  3. 2f4e68582035cc5f8c0ab2f4d0e39d85a04e143979ac62ab3a2432493a12a1c7
    create table t1 (v1 int primary key) strict, without rowid;no parent

Illustration. Each bar code is the 256 bits of a real SHA-256, taken over sample content. A real commit object records the rows it changed, with the SQL kept beside them as a note.

Transactions
ACID within one database. No transaction spans databases.
Writes
One write transaction at a time per database. A second writer waits until the first has committed or rolled back.
Reads
In parallel, from a local SQLite cache, without contacting S3. Each read sees one consistent state, even while a write is in progress.
Cache
Holds the same tables and rows as the database, is never ahead of S3, and can be rebuilt from the log at any time.
Lease
One instance at a time holds the database's lease, recorded in the manifest. It lasts 30 seconds and is renewed every 20. A spare instance waits and takes over when the lease times out.
Unknown outcome
If the swap ends without an answer, the instance asks S3 until it knows. The transaction is committed if and only if its commit object is in the log.
S3 must offer
Strong consistency, compare-and-swap, and create-if-absent.
Interface
Go's database/sql. Agreed, not built yet.

How it is built

Agents write the code and nobody reviews it, so the tests have to show that it works. The production code runs unchanged inside a simulation where S3, time, and randomness are driven by one seed. The same seed always replays the same run, failures included.

Same seed, same field. Replay it and compare. Open the simulator
Deterministic simulation
S3, clocks, and randomness are reached only through interfaces. One seed controls a whole run, so any failure the simulation finds can be replayed exactly.
Buggify
Marked points in the production code do something legal but unusual in simulation: return an error, add a delay, take a rare branch. Each run switches on a different subset.
SQLite is the oracle
s4ql writes its own parser and engine, and checks them against SQLite. Results and errors are compared with SQLite's, and the parser is run on SQLite's own test statements.
Real S3
A conformance suite and integration tests run against a real S3-compatible bucket on Tigris, to check that the simulated store behaves like the real one.
Tests that test the tests
Property-based tests, fuzzing, and mutation testing. Bugs are planted on purpose to count how many seeds the simulation needs to find each one.
Everything written down
The spec, the research, the builders' diaries, and the conversations behind each decision are in the repository.

Status

As of 16 November 2026. s4ql is an experiment. It is not ready for use, and anything in it may change.

Built
  • The storage core. A database is a log of commits in S3 with a local SQLite cache, and one instance at a time holds a lease on it. Commits, recovery after a crash, and replay of the log work.
  • The tests behind it: a deterministic simulation, and a real S3-compatible bucket.
  • The S3 client, which passes the conformance suite on Tigris.
  • The SQL tokenizer and the SQL parser.
  • The parser agrees with SQLite on what parses and on every syntax error, across 139,234 statements from SQLite's own tests, documentation, and fuzz corpora.
  • The shape of the syntax tree is checked against SQLite as well: the same EXPLAIN output and the same values.
Not built

The pass after the parser, the engine, and the rest of the SQL. There is no engine and no public API yet, so the database cannot be used.

Next

The first slice of SQL: create table, insert, select, and drop table for one integer column, checked against SQLite.

Roadmap

SQL grows in small slices. Nothing is supported until it is listed, all the way down to select 1, and each slice goes through every layer and is checked against SQLite before the next one starts.

  • Built
  • Agreed, not built
  • Open
  1. Storage core

    The log of commit objects, the manifest swap, the lease, crash recovery, and the local cache.

    Built
  2. S3 client

    Uploads are signed with the commit object's SHA-256, so the store refuses content that does not match it.

    Built
  3. Tokenizer

    The first piece of the SQL front end.

    Built
  4. Parser

    s4ql's own parser for all of SQLite's grammar. It agrees with SQLite on which statements parse, on the message of each syntax error, and on the shape of the syntax tree.

    Built
  5. The pass after the parser

    It rejects whatever the roadmap does not list, and it raises the errors that SQLite's grammar actions raise while they parse.

    Agreed, not built
  6. Slice 1: one integer column

    create table, insert, select *, and drop table on a strict table with one int primary key. The answers must survive a restart with an empty cache.

    Agreed, not built
  7. The driver

    A database/sql driver as the main interface.

    Agreed, not built
  8. More SQL, slice by slice

    Each slice is agreed before it is built. Every table is strict and without rowid, and foreign keys are always enforced.

    Open
  9. Storage format version 1

    The formats are drafts until then. From version 1 on, every version of s4ql reads what any older version wrote.

    Open

Inspired by

s4ql builds on ideas from other databases and storage engines. How each idea works in its own project is written up in docs/inspiration.md.

SQLite
The embedded database s4ql is modelled on. s4ql speaks a subset of its SQL, checks its own answers against it, and uses it for the local cache.
FoundationDB
A distributed, transactional key-value store, tested in deterministic simulation. s4ql's simulation and its buggify points follow its example, and so does the rule that an application checks its own data after an unknown commit outcome.
Graft
A storage engine in Rust that replicates SQLite pages to object storage as an append-only log of commits, with no server in between.
SlateDB
An embedded key-value store in Rust that keeps all its files in object storage, with a manifest that lists the current state of the database.
Turbopuffer
A search service where object storage is the only source of truth and each namespace is a prefix in the bucket, with caches in memory and on disk in front of it.
Litestream
A tool in Go that replicates a SQLite database to object storage from its write-ahead log, and a SQLite plugin that queries the copy in object storage directly.

Follow along

The code, the spec, the open questions, and every conversation behind the decisions are public. Watch the repository to see whether a database written by agents can earn trust.

maragudk/s4ql on GitHub