Managing AI Coding Costs at Scale

Databricks engineers, drawing on their own experience and conversations with Stripe, Coinbase, Uber, and Ramp, present a playbook for taming the exponential growth of AI coding costs without sacrificing the productivity gains that make AI adoption worthwhile. The core argument: companies face a "dual mandate" — provide broad, low-friction access to AI tools while keeping costs inside a predictable per-user envelope — and the earliest large-scale adopters have converged on a set of proven techniques.

The single biggest cost lever is relentlessly chasing the efficiency frontier (best price-per-intelligence) rather than the intelligence frontier. New models are released almost weekly that beat incumbents on intelligence-per-dollar, but public benchmarks do a poor job of indicating real-world coding performance. Companies like Stripe and Databricks have built internal eval suites to test new models against their actual development mix — and frequently produce negative results, declining models that cost more without meaningful quality gains.

Model flexibility requires tooling that doesn't lock you into one model family. The article contrasts two approaches: asking developers to switch between harnesses (high switching cost, de facto lock-in) versus using a meta-harness that provides a unified developer experience while dispatching to underlying harnesses. Databricks open-sourced their meta-harness, Omnigent, as the default mode for their developers.

On budgets: every company surveyed uses hard token caps only as a last resort. The emerging best practice is progressive friction — near-instantaneous spend visibility across all tools, nudges toward cheaper models, and increasing degrees of friction as spend rises, rather than abrupt cutoff.

Context bloat is the hidden cost driver: when a user types a simple request, the agent gathers massive context, invokes many tools, and searches the codebase, so the user's initial prompt accounts for a negligible fraction of the data fed to the LLM. Techniques like context compaction and prompt compression are early but promising. At Databricks, relatively simple tuning of harness and caching settings led to a ~50% reduction in generated tokens with no quality degradation.

The infrastructure layer tying these techniques together is the AI Gateway — a central location for model menu management, unified cost observability, context observability and enforcement, caching configuration, and routing/rate limiting. Databricks released Unity AI Gateway as open source alongside Omnigent.