Hitomi (Data Viewer)

2026-06-12 · page · source ↗ · source summary

Hitomi is a Flutter desktop application for viewing and analyzing structured data — CSV, TSV, and arbitrary custom formats — with zero-wait startup regardless of file size. Its defining characteristic is a streaming ETL architecture: data is parsed, typed, and loaded incrementally as byte chunks arrive, so you can open a 10GB log file and see the first rows instantly. It also handles live data sources (pipes, sockets, subprocess stdout), and includes a custom expression-based filter language (PHLEX) with its own type system and compiler.

Why it matters: it's one of the few open-source desktop data viewers that treats streaming as a first-class architectural concern rather than bolting it on. The streaming parser correctly handles chunk boundaries cutting through multi-byte UTF-8 characters, escaped quotes, and CRLF pairs — edge cases that most "streaming" parsers ignore entirely. The filter engine has a selectivity-based caching strategy that functions as a mini query optimizer.

Architecture

Streaming ETL pipeline into a columnar in-memory store, built in Dart/Flutter.

ByteStream >> Framer >> Scanner >> Columnarizer >> EvIntakeCtor >> EventManifold

Each stage is a Dart extension type with a >> operator, creating a readable pipeline syntax with zero wrapper-object allocation overhead.

Key Techniques

Streaming parser with chunk-boundary recovery

The Quadramaton mixin (lib/infra/quadramaton/Quadramaton.dart) is a state machine that returns one of four states per call: endOfEntry, endOfChunk, incomplete (mid-field at chunk boundary), or indefinite (complete field but row terminator unseen). When incomplete fires, the partial row is discarded and the token stream rewinds to a checkpoint; parsing resumes correctly when the next chunk arrives. The test suite verifies: 'should handle incomplete UTF8 char at end of chunk', 'should handle CRLF split across chunks'.

PHLEX filter language with its own compiler

PHLEX (lib/infra/phlex/) is a full expression language: recursive-descent parser, 9 AST node types with visitor pattern, a type system (ResultType sealed hierarchy), function overload resolution with detailed error messages, and a three-pass compiler (parse → type-check → codegen) that produces row-index → bool closures. 14 built-in functions with 35 overloaded signatures.

Selectivity-based filter caching

FilterSeries (lib/infra/amorphous/FilterSeries.dart) uses a dynamic strategy: if a filter's result is <50% of the original (high selectivity), cache it and run subsequent filters against the cached result — near-free. If >50% (low selectivity), apply against the already-filtered index. Uses O(m+n) two-pointer intersection to combine filter results, enabled by the ordering invariant that all filtered indices are subsequences of the original.

Chunked lazy-allocation column storage

SparseVector<T> (lib/infra/amorphous/SparseVector.dart) uses 64-element chunks with a shared empty-chunk singleton. When all elements in a chunk equal the default value, the chunk reference is set to the shared empty chunk (no allocation). On write, the chunk is materialized. This means sparse columns with many null/default values consume almost no memory.

Extension types for zero-cost pipeline abstraction

Dart 3 extension types let the code wrap configuration data (an int, a Schema, a list) with methods without allocating wrapper objects. Framer(int size), Scanner(Schema schema), Columnarizer._(List), etc. are all compile-time-only wrappers. Combined with >> operator overloads, the pipeline syntax is both readable and allocation-free.

Schema-driven attribute transform chains

Each Attribute<T> (lib/domain/schema/Attribute.dart) carries a format list of FormatSegment instances. A single segment → reshape (regex extraction). Multiple segments → assemble (build composite strings from captures + literal templates). The compiled xformer getter chains through type parsing with null/default fallback.

Design Decisions

Streaming over batch: Zero-wait startup for any file size, at the cost of ~200 lines of parser state-machine complexity vs. ~50 lines for a batch CSV parser. The trade-off is unambiguous for a viewer — you never want to wait for a file to fully load before seeing anything.

Columnar over row-oriented: Column access is O(1) — right for sorting/filtering/plotting by column — but row access requires N independent vector lookups. A data viewer is column-access-heavy, so this is the correct choice.

Custom query language over SQL: PHLEX is simpler than embedding SQLite, operates directly on in-memory vectors without serialization, and gives meaningful type errors. But it's non-standard, lacks JOINs, and users who know SQL must learn a new syntax.

In-memory only: All data lives in SparseVectors in RAM. No spill-to-disk path. Reasonable for a desktop viewer where you're bounded by screen real estate before memory, but a 10GB file won't fit.

Schema-first design: Best results come from providing a schema file declaring attribute types and transforms. The schemaless fallback gives untyped string columns with no analysis capabilities — fine for quick peeks, useless for serious work.

Comparison Notes

Tags: #tool #desktop #database #data-analysis #flutter databases-and-data

Source: https://github.com/Verticalysis/Hitomi — ingested 2026-06-12 via deep code analysis