Databases and Data

topic · 132 sources

The database field is converging on a division of labour between a durable, engine-independent data layer — the log, object storage, open columnar formats, and the once-written metric definition — and a thin, swappable, increasingly generated query layer that sits on top of it. "Converged" databases that fold many engines into one product and "composable" stacks that pair a tiny engine with object storage are not rivals but two placements of that same seam; they disagree only about where it should go. What is not converging is the field's newest consumer: AI agents, who on real enterprise data score roughly an order of magnitude below their public-benchmark results, and who are rescued not by handing them more autonomy but by constraining them — restricted query interfaces, deterministic fences, and curated definitions — rather than trusting the model to assemble a correct query.

The Argument

The durable layer is separating from the engine

The intellectual spine here is the log. The Log — Unifying Abstraction for Real-Time Data made the case in 2013: an append-only, totally-ordered log is the natural data structure for moving data between systems — a table is current state, the log is the history from which every past state can be reconstructed, and a central log collapses O(N²) point-to-point integration into O(N) connections. A decade on, The Dataflow Model Revisited — the authors of the 2015 Dataflow Model grading their own work — names what they missed: the streams-and-tables duality, the fact that a stream is a changelog and a table a snapshot, each recoverable from the other. Their sharpest retroactive claim is that the problem they attacked with windows, triggers, and watermarks was really the materialised-view maintenance problem, posed by the database community thirty years earlier; the destination of analytical streaming, they argue, is to disappear into the database.

That destination is now being built from the storage layer up. Lakebase and LTAP describes externalising the write-ahead log into a distributed SafeKeeper service and the data files into a PageServer, making Postgres compute stateless, then transcoding data into open columnar formats (Delta/Iceberg over Parquet) so transactional and analytical engines read the same copy — "no CDC pipeline, no second copy." Postgres to Snowflake Data Mirroring reaches the same place from the opposite direction: rather than external consumers pulling WAL changes, a snowflake_cdc extension pushes transactional batches out of Postgres into Iceberg, "append, never upsert." Aurora DSQL is the purest expression of the trend — compute, commit, and storage split into independently scalable services, with all coordination deferred to commit time for 0-RTT reads — though Aurora DSQL — Murat Demirbas' Insider Review adds the downsides the paper underplays: hot-key write-write conflicts, aborts detected only after paying WAN latency, and limited quantitative evaluation. From the application side, Materialized Views Are Obviously Useful argues the same principle one layer up: derived data should be maintained by the database, not by hand-rolled caching and invalidation code that must have been correct at every point in the past.

Convergence and composability are two placements of the same seam

Given that split, there are two honest strategies, and the field is pursuing both. Converged databases fold engines into one product so a single surface answers every query. AliSQL grafts DuckDB's columnar engine and native HNSW vector search onto MySQL, claiming roughly 200× analytical speedup without asking users to leave their existing tools. The agent-memory convergence runs toward graph-plus-vector-plus-temporal: NornicDB combines Neo4j-compatible Cypher with GPU vector search and an Ebbinghaus-based memory decay that fades unreinforced knowledge; LatticeDB shrinks the same idea to a single embedded Zig file with HNSW, BM25, and a graph changefeed sharing one transaction path; AntFly is the distributed counterpart, a multi-Raft engine doing full-text, vector, and graph search over multimodal data. DocumentDB converges by absorption, making PostgreSQL speak the MongoDB wire protocol; Dolt converges version control and SQL, giving tables Git-style branch-and-merge for agent state.

The composable camp argues the opposite: the smallest tool that will do the job, with object storage as the universal backplane. SQLite Is All You Need benchmarks a single 343 MB file serving a 50,000-user social network and is explicit about the limits — one writer serialises writes, no failover — while SQLite is All You Need for Durable Workflows extends the bet to agent state via Litestream, and Postgres Transactions Are a Distributed Systems Superpower argues the complementary move: put the workflow inside Postgres so a step and its checkpoint commit in one transaction. Graft replicates SQLite to the edge through object storage with no cluster; celld runs Cloudflare Durable Objects on your own machines with the bucket as coordinator; bucketvcs runs a Git server directly on object storage. zvec and Just Brute Force Your Embeddings make the same case for vector search — an in-process library, or a NumPy scan, rather than a system you operate. The extreme of the camp is no database at all: Drilldown Dashboards from a Single Parquet File serves a dashboard from a 40 MB Parquet cube and an 18 KB JavaScript reader, Random Access Parquet (RAP) answers point queries straight from lake files via an external index, and Replace Athena with DuckDB (Lambda) runs DuckDB on Lambda reading only the Parquet byte ranges a query touches. The boundary of the bet is where the environment betrays you: How to Corrupt an SQLite Database catalogues the ways a durability guarantee dies to a rogue process or a lying disk, and Learning a Few Things About Running SQLite records the concurrent-writer timeouts that eventually push a project toward Postgres.

The impedance mismatch is not solved, and agents inherit it

The Vietnam of Computer Science diagnosed the object-relational impedance mismatch in 2006: object systems model identity, state, and encapsulation; relational systems model predicate logic and truth; the two are "too different to bridge silently." Object-Relational Impedance Mismatch catalogues nine fracture points and frames the thing as a clash of worldviews rather than a technical glitch. The new evidence is that agents inherit the mismatch at the same site where ORMs struggled. Constraint Decay measures it: mandating PostgreSQL as a backend drops agent code-generation accuracy by 19.3 percentage points, and data-layer defects drive roughly 45% of logic failures — the agents are not bad at coding, they are bad at holding an object model and a relational schema in context at once. Dapper Performance Trap is the small, repeatable instance: a C# string mapped to nvarchar(4000) against a varchar column triggers an implicit conversion that defeats the index, 176× slower, working correctly all the while. Acadia is one attempt at an exit — a language whose custom types live in the database, with compiler-verified migrations and shared types across client, server, and store, so the mapping never happens because there is nothing to map.

Agents versus enterprise data: the ten-percent problem

The same structural gap shows up, an order of magnitude larger, when agents try to query real data. On BIRD, the leading public text-to-SQL benchmark, LLMs reach roughly 82% execution accuracy. On BEAVER, built from real private warehouses with hundreds of tables and implicit joins, the best agentic setup manages 10.8% — and only 30.1% even when handed oracle hints for all five subtasks, with domain knowledge and query decomposition staying stubbornly hard. Text-to-SQL in the Real World makes the diagnosis from the source: schema rot, bespoke institutional jargon, and 2–3-join real queries that public benchmarks never contained; a pure LLM scores 0%, and agentic scaffolding lifts it only into the low teens. RUBICON is the sharpest counter-proposal: on a benchmark coordinating five heterogeneous sources, every autonomous baseline produces zero correct answers while RUBICON — which subsets SQL to a restricted "baby SQL," integrates over tables with a query processor, and leaves only the predicate in natural language — reaches 100%. DeepSQL lands the same point as engineering practice: LLM-drafted SQL passes through a schema whitelist, an EXPLAIN validator, and a read-only session, so deterministic machinery owns every safety-critical decision. Even Inside OpenAI's In-House Data Agent — the most autonomous system in the evidence, a self-correcting loop over 600 PB — works by burying six layers of context beneath the model and guarding it with a continuously-run eval suite, and its lessons ("less is more," "guide the goal, not the path") read as constraint in disguise.

The human counterpart is what the machine should not be left to decide alone. AI-Assisted Database Work — The Machine Reads, The Human Decides documents nine engagements where AI read 400K lines of PL/SQL or mapped 900 columns, and the human's value was knowing the blast radius the machine could not — a database-wide setting that would pessimise hundreds of queries, a status code that meant Active in one company and Archived in the other. SQL Pagination — Offset vs Seek Method is the granular version of the same gap: agents default to OFFSET, which degrades linearly with page depth, where the seek method needs a deterministic ORDER BY — exactly the kind of non-obvious expertise that produces silently wrong results rather than failures. And SQL Fraud Patterns (Fixel Smith) shows what the working target looks like: fraud detection "is mostly SQL," six composable window-function patterns that collapse fraud rules into three WHERE filters — the same filter-based access agents will need.

Define it once: semantic layers and the source of truth

If agents are the new consumers, the durable layer needs a naming layer above it. The recurring failure is that a metric like "revenue per active user" is defined once in dbt, again in Looker, again in a notebook, and drifts. Metrics SQL answers with a YAML-defined metrics view queried through a SQL dialect that transpiles to the engine; Malloy takes the language-first route, compiling a semantic model to eight dialects with symmetric aggregates that avoid double-counting joined data. Malloyyo turns a Malloy model into an MCP surface with restricted query governance — the agent can only query what the model exposes, and never sees the SQL it ran. KTX Context Layer for Data Agents goes further, ingesting dbt, Looker, Metabase, and live databases into a git-versioned wiki plus an executable semantic layer behind 11 MCP tools. DAB solves the adjacent, lower problem of giving agents governed endpoints over SQL Server, Postgres, MySQL, and Cosmos. Long Live Systems of Record gives the why: when an agent is told to "calculate ARR by segment," something still has to say which canonical table is true, and the place where metric definitions live starts to look like the source of truth. The Oracle Is the Asset generalises it: when compilers become cheap to generate, the scarce asset shifts from implementation to the reference behaviour that defines correctness — you own the spec, and the SQL (or the endpoint, or the compiler) is regenerable.

Data movement is hard to make boring, and it is being absorbed

The pipeline itself is where the seam is most contested. Postgres CDC in ClickHouse, A Year in Review is the field report that replication is operationally hard — long-running transactions, replication-slot backpressure, schema changes discovered only when a customer hits them at midnight — and that making the system feel "boring" takes hundreds or thousands of small edge cases. Streambed is the composable counter-offer: a single Go binary that streams Postgres WAL to Parquet and Iceberg with a DuckDB query server speaking Postgres, validated by Jepsen-style chaos testing — unusual for a solo project. Artie industrialises the same movement as a managed service with exactly-once delivery and schema evolution. DocDB — Stripe's Zero-Downtime Database is the largest claim on the approach: Stripe built data movement into a primitive, so resharding, version upgrades, and tenancy migration all run the same six-phase migration under live traffic at 5 million QPS. And then the unifiers undercut the whole category: Lakebase and LTAP and Postgres to Snowflake Data Mirroring argue the pipeline is a transitional crutch — if the data lives once in open columnar formats, there is "no list of replicated tables, because there is no replication."

Correctness is structural

Correct by Construction is the radical version: decompose data into anchors, attributes, and links, ban NULLs and sentinel values, ban JSON containers — prevent bad data by construction rather than detecting it afterwards. Write Snapshot Isolation applies the same instinct to isolation: snapshot isolation should check for stale reads, not stale writes, and a one-line change yields serializability at the cost of more (correct) aborts. ARIES — Write-Ahead Logging Recovery is the forty-year-old foundation of the whole edifice — repeat history in redo, undo losers with compensation log records — the paper behind every WAL-based recovery in the rest of this wiki. Kale — Transformation-Safe Spreadsheets extends structural safety to the spreadsheet: make unstable range references inexpressible so formulas survive insertion and sort. Fixing Bugs in Event Sourcing is Hard is the operational complement: an append-only event log preserves the inputs a correction needs, where state-based fixes overwrite the evidence the next fix requires. And when correctness fails anyway, Bad Data in Production — Response Playbook argues the fix is process — contain, trace lineage, notify personally, review blamelessly — while Anomaly Detection offers the pragmatic detector for teams who did not build by construction: three statistical checks in constant memory and a single file.

Operating at scale

We Replaced Redis with MySQL for Inventory Reservations is the strongest "your existing database is already enough" data point: Shopify survived Black Friday 2025 at $5.1M/minute with MySQL and SKIP LOCKED, and the real lesson was that the bottleneck was connection pressure from other processes, not the reservations. PgDog argues the database need not converge if the proxy handles pooling, load-balancing, and sharding. DocDB — Stripe's Zero-Downtime Database and ClickBench Playground bracket the reliability problem from opposite ends: Stripe engineered 5.5 nines, while Milovidov's hundred-database playground found most databases cannot run unattended without crashing or OOMing. Why Are Databases So Hard states the physical ceiling behind all of it — a New-York-to-Los-Angeles round trip is roughly 130 ms against a 32 ms speed-of-light floor, so consistency, speed, and geographic availability cannot all be had, only traded. Meerkat — QuePaxa Consensus at Cloudflare is a new point on that curve: a consensus service where any replica can propose, trading latency for throughput under adversarial networks.

The substrate: formats, engines, and data models

Underneath all of this is a quietly standardising substrate. Wes McKinney on Pandas, Arrow, and Data Infrastructure records how Arrow went from "impossible to standardise" to the in-memory format under DuckDB, Polars, and the modern stack — and quotes DuckDB's co-creator on why LLMs cannot yet produce such systems. Apache DataFusion turns Arrow into an extensible query engine for builders; DataFusion for Billion-Scale Graph Algorithms shows it running PageRank on a billion edges from a laptop. Binary Vector Embeddings shows how much retrieval performance survives 32× compression. Tooling to make database work visible follows: Shaper turns SQL into dashboards on DuckDB, sql-crack renders query flow and lineage as diagrams, and SiftRank ranks datasets by relevance to a prompt. Grist, FlareDB, and Chatto each explore a different modelling axis — the relational spreadsheet, the Beam-native streaming store, the event-sourced chat app — while Radicle applies the object-storage-plus-Git pattern to the whole code forge, decentralised.

Where the Sources Disagree

Convergence versus composability. AliSQL, NornicDB, and LatticeDB bet that one database doing everything beats several doing one thing each. SQLite Is All You Need, Graft, and zvec bet the opposite. The evidence does not resolve this; it splits by use case. Converged systems win at operational simplicity under moderate scale; composable systems win on cost, debuggability, and avoiding infrastructure you do not need. Where the crossover lies depends on team size, traffic pattern, and data volume — factors neither camp's benchmarks control for.

Replicate it, or unify the storage layer? Postgres CDC in ClickHouse, A Year in Review, Artie, and Streambed treat change-data capture as a permanent, hard pipeline — the honest reality is hundreds of edge cases before it feels boring. Lakebase and LTAP and Postgres to Snowflake Data Mirroring argue the pipeline is a transitional crutch that storage-layer unification eliminates outright. DocDB — Stripe's Zero-Downtime Database sits between them: data movement is real and difficult, but as a platform primitive it turns replication, resharding, and upgrades into one operation. The disagreement is genuinely about whether CDC is a fact of life or a stopgap.

Structure or autonomy for agents. BEAVER and Text-to-SQL in the Real World show autonomous text-to-SQL failing on real data; RUBICON and DeepSQL conclude the answer is to constrain the interface and let deterministic machinery own correctness. But Inside OpenAI's In-House Data Agent is a heavily autonomous, self-correcting loop that works — at OpenAI, on six layers of context and continuous evals. And Constraint Decay cuts the other way: mandating structure (Postgres, ORMs, Clean Architecture) measurably degrades code generation. So "constraint helps" is true for query generation and false, or at least costly, for code generation — the discipline has not yet said where the line is.

SQLite versus Postgres as the default. SQLite Is All You Need argues SQLite suffices for production workloads far larger than most teams will see, and that premature infrastructure is a tax on traffic you do not have. Learning a Few Things About Running SQLite shows the operational asterisks: concurrent writers cause timeouts, ANALYZE is an undiscoverable fix for slow full-text search, and backup via VACUUM INTO OOM-kills the process. These are not arguments against SQLite — Evans still calls her Postgres-to-SQLite migration a great choice — but they are evidence that "SQLite is all you need" has operational caveats the benchmarks do not surface.

What's Missing

Agent-aware database patterns. Agents generate queries differently from humans — more repetitive, less optimised, prone to the Dapper Performance Trap class of error where everything appears to work but performance silently degrades. Database features aimed at agent-generated workloads — query-plan guards, automatic parameter-type correction, cost-based query rejection — do not exist in the evidence. SQL Pagination — Offset vs Seek Method fills one small piece; a systematic treatment is still a gap.

Data quality for agent-generated data. Correct by Construction addresses human data pipelines. When agents generate and store data at machine speed, without the implicit checks that human data entry provides, the quality problem changes character. There is no "Correct by Construction for agent-generated data" in this wiki.

The migration path between the poles. SQLite Is All You Need names the breakpoints — many writers contending on the same rows, the need for read replicas, real analytics — and Learning a Few Things About Running SQLite shows the concurrent-writer timeout cascade that triggers a migration. But the operational path from SQLite to Postgres, from MySQL to AliSQL, from a single node to a sharded one, is documented only in fragments, not as a discipline.

Reliability beyond SQLite. How to Corrupt an SQLite Database is the only page that systematically catalogues where a database's guarantees end and the environment's betrayal begins, and ClickBench Playground surfaces the same question at fleet scale — most databases cannot run unattended. There is no comparative reliability survey, and no "how to corrupt a Postgres database" companion.

The write path. A master's thesis surveying client-side sync engines finds the read path commoditises while the write path resists generalisation — conflict resolution, business logic, and failure recovery stay application-specific. The wiki's coverage of data serving is rich; its coverage of synchronising writes across replicas is thin.

Also on This Theme

Compiled from 93 sources: The Vietnam of Computer Science, Observability's Three Pillars: A History, RUBICON: Agentic AI for Messy Enterprise Data, Kale: A Transformation-Safe Spreadsheet System, Hitomi — Lightweight Streaming Data Viewer and Analyzer, AliSQL, Anomaly Detection, AntFly, Apache DataFusion — Extensible Query Engine, ARIES: A Transaction Recovery Method Supporting Fine-Granularity Locking and Partial Rollbacks Using Write-Ahead Logging, Artie — Real-Time Database Replication, Aurora DSQL: Scalable, Multi-Region OLTP, Aurora DSQL: Scalable, Multi-Region OLTP, You Just Found Bad Data in Production. Now What?, BEAVER: An Enterprise Benchmark for Text-to-SQL, Binary vector embeddings are so cool, bucketvcs — Git server backed by cloud object storage, celld, celld, Chatto, CipherStash — Summary, ClickBench Playground — How I Host 100 Database Systems for an Interactive Benchmark, Clouded Judgement 12.12.25 - Long Live Systems of Record, Postgres Transactions are a Distributed Systems Superpower, Constraint Decay: The Fragility of LLM Agents in Backend Code Generation, Correct by Construction, Fast drilldown dashboards from a single Parquet file, DAB, How C# Strings Silently Kill Your SQL Server Indexes in Dapper, Data Engineering Central Podcast – Wes McKinney on Pandas, Apache Arrow & Data Infrastructure, Data Engineering for Large Models, Algorithms on billion-scale graph using 10GB RAM: I love DataFusion!, DeepSQL, Stripe's DocDB: How Zero-Downtime Data Movement Powers Trillion-Dollar Payment Processing, DocumentDB — MongoDB-Compatible Open Source Document Database on PostgreSQL, Dolt, Announcing the DuckDB ADBC Extension, Fetching the Next Page — SQL Pagination Patterns, Fixing bugs in Event Sourcing is hard, for real?, FlareDB, Graft, GraphRAG, Grist, grudge — A constant-memory sketch that holds grudges and forgives, How To Corrupt An SQLite Database File, icebug-format, Linked Data Event Streams (LDES) — Consumer Specification v1.0, Indexing the Data Lake for Online Point Queries, Inside OpenAI's in-house data agent, Introducing Metrics SQL: A SQL-based semantic layer for humans and agents, A Cheap Alternative to AWS Athena — Lambda × DuckDB, Just Brute Force Your Embeddings, KTX — Context Layer for Data Agents, From monolith to Lakebase to LTAP: rethinking the database from storage up, LatticeDB, Learning a few things about running SQLite, llm-wiki, Malloy — A Semantic Modeling and Query Language, Malloyyo, Materialized Views Are Obviously Useful, Introducing Meerkat: an experiment in global consensus, Nine Unusual Ways My Clients Use AI with SQL Server, NornicDB, Nubase, Object–relational impedance mismatch, The Log: What every software engineer should know about real-time data's unifying abstraction, OpenSandbox, The Dataflow Model Revisited, PgDog — Scale Postgres n times, Postgres CDC in ClickHouse, A Year in Review, Postgres to Snowflake Replication & Mirroring, Radicle: The Sovereign Code Forge, cuGraph — GPU-Accelerated Graph Analytics, redb 3.3.0: an enterprise .NET stack you actually own — typed store, a homegrown Apache Camel, and a runtime with a dashboard (all free), Rethinking Database Programming, We replaced Redis with MySQL for inventory reservations—and it scaled, shaper, siftrank, sql-crack, Six SQL patterns I use to catch transaction fraud, SQLite Is All You Need, SQLite is All You Need for Durable Workflows, State System — A Model-Mediated Organizational State Layer, Searchable field-level encryption on Supabase with CipherStash, If You Think You Can Do Real-World Text-to-SQL, The Limits of Generalized Sync: A Taxonomy of Architectures, Trade-offs, and Decision Factors, The Oracle Is the Asset, Streambed — Postgres-to-Iceberg CDC Engine, Why are databases so hard?, Write Snapshot Isolation, Xano — The Scalable No-Code Backend Platform, ZIL Lean, zvec Last compiled: 2026-09-12