There is a point in persistence design where good engineering can easily turn into unnecessary architecture.
A database seems obvious because applications need persistence. Once it is there, querying, versioning, and concurrency all get pulled into the same technology by default, whether or not the workload actually needs a database’s transactional guarantees to get them.
While designing the storage layer for RST Platform, I wanted to approach the problem from the opposite direction:
Start with the simplest persistence that naturally fits the data, and introduce database semantics only when the workload demonstrates that it needs them.
The resulting architecture is deliberately small — plain Parquet files as the authoritative record, DuckDB as the query engine over them, and no application database server. The interesting part is not any individual technology, but how storage, querying, and versioning can be pulled apart into separate concerns instead of being bundled into one.
Start With the Requirements
Persistence should follow the dominant complexity of the data, and that complexity is best understood through the underlying stakeholder stories rather than a feature list. Not all stories carry equal weight for storage design — the ones below are significant enough to shape it:
- Collect end-of-day market data
- Collect intraday data for watched instruments
- Calculate technical and analytical indicators from that data
- Screen and query market data
- Maintain portfolios and watchlists
- Keep a log of all trades
The important observation is that this data is primarily written once and read many times, mostly by analytical queries rather than by transactional updates to individual records — and that distinction drives most of the storage design.
Rethinking Persistence and Querying
Separate Persistence From Querying
One of the more counter-intuitive design decisions is to avoid treating storage and querying as the same architectural choice.
A common assumption is:
If I want SQL, I need a SQL database.
That is no longer necessarily true.
There are really two separate questions:
- How should the data be persisted?
- How should the data be queried?
Separating the two is only practical because file formats and query engines have matured to the point where a file can be persisted on its own terms while still being queried with SQL, without a database server sitting in between:
persistence → files
querying → embedded SQL engineThat distinction is central to the design, and the choices behind each side are worth walking through separately.
Why Files?
A database is not required simply because data needs to survive a restart.
The more useful question is:
What persistence semantics does this data actually require?
Market data is unusually well suited to file-based storage: historical prices are largely immutable observations, calculated indicators are reproducible, configuration is small, metadata is naturally document-shaped, and portfolio and watchlist state can initially be represented as simple structured documents. That suggests a file-first architecture. Nor is this a small-scale compromise — the largest data platforms in the world, at petabyte scale, are themselves built on files in object storage rather than a traditional database, so there is a lot of runway before file-based persistence itself becomes the limiting factor.
The important caveat is that file-first should not become file-only ideology. If a part of the system eventually requires transactions, relational integrity, high-frequency concurrent writes, or complex mutable relationships, a database may be the simpler tool — but those semantics should be introduced because the workload requires them, not because applications are assumed to need one.
Why Parquet?
CSV is the obvious plain-file alternative — its main advantage is transparency, since almost anything can open it. But market data is analytical, typed, and columnar, which is exactly where Parquet is strongest.
Parquet provides:
- explicit data types
- efficient compression
- column-oriented storage
- fast analytical scans
- efficient column selection
- broad interoperability
- excellent support in DuckDB, Polars, PyArrow, and Python
For example, a query that only needs date, symbol, close, and volume out of a file whose columns actually look like:
- date
- symbol
- open
- high
- low
- close
- adjusted_close
- volume
- …
does not need to process every other column in the file — and that becomes increasingly useful as datasets grow.
Parquet is binary rather than directly human-readable, but that is a reasonable tradeoff when the data can still be inspected trivially through DuckDB or other tools: the persistence remains portable without requiring the storage format to be readable with a text editor.
DuckDB as a Query Engine Over Files
DuckDB is what makes the file-first approach particularly attractive.
A Parquet file can remain the authoritative persistent artifact:
market/raw/eod/market=XSTO/year=2026/data.parquetwhile DuckDB provides SQL directly over it:
SELECT
symbol,
date,
close,
volume
FROM read_parquet(
'market/raw/eod/market=XSTO/year=2026/*.parquet'
)
WHERE date >= DATE '2026-08-01'
ORDER BY symbol, date;The important point is that DuckDB does not have to become the database — it can simply be the relational query engine over ordinary files, giving the application much of the expressive power of SQL without forcing analytical data into database-managed storage.
What this simple version does not solve: once a derived table gets recalculated and rewritten, a glob like *.parquet has no way to distinguish the new files from whatever a previous run left behind — it just reads everything sitting in the directory. That gap is exactly what a format like DuckLake exists to close, discussed later.
Organizing the Data
Raw Data and Derived Data
Raw market data and calculated values should not be mixed simply because they share the same timestamps: raw data is authoritative, derived data is reproducible. The model is:
raw market data
↓
calculation
↓
derived indicatorsRaw data may contain:
timestamp
symbol
open
high
low
close
adjusted_close
volumewhile derived data contains:
timestamp
symbol
sma_20
sma_50
rsi_14
atr_14
...The architectural invariant is:
Raw data is authoritative. Derived data can always be deleted and regenerated.
That makes experimentation much safer: a bug in an indicator calculation should not contaminate the source data, and a new calculation can be introduced without changing the historical market observations it depends on.
EOD and Intraday Data Are Different Workloads
Market data arrives to serve two distinct needs: broad end-of-day history across many instruments, and a narrower intraday stream for the handful of instruments being watched closely. Those needs don’t have to share a physical layout, because their access patterns are different.
EOD Data
EOD data spans many instruments across multiple markets. Typical operations include:
- screening an entire market
- ranking stocks
- finding candidates matching conditions
- calculating indicators across an exchange
- comparing instruments
That suggests a market-oriented layout:
raw/
eod/
market=XSTO/
year=2025/
data.parquet
year=2026/
data.parquet
market=XNAS/
year=2026/
data.parquetThe key=value naming — market=XSTO, year=2026 — is not just a human-readable label. It is Hive-style partitioning, a convention originating from Apache Hive that DuckDB and most other Parquet-aware query engines recognize automatically. When a query filters on WHERE market = 'XSTO', the engine can skip opening any file under a different market= directory entirely, rather than reading every file and filtering rows afterward — this is called partition pruning. It also means market and year don’t need to be stored as columns inside the Parquet files at all; they’re derived from the path itself, for free.
Each file contains rows for many instruments:
date symbol open high low close volume
2026-08-20 ABB.ST ...
2026-08-20 VOLV-B.ST ...
2026-08-20 INVE-B.ST ...
2026-08-21 ABB.ST ...This is well suited to market-wide analytical queries.
Intraday Data
Intraday data has almost the opposite characteristics. I do not need intraday history for every instrument in every market — I primarily need it for:
- current portfolio holdings
- potential trades being evaluated during the day
The intraday universe is therefore much smaller and changes dynamically, so a symbol-oriented layout makes more sense — and the same Hive-style convention applies here too, since a portfolio-wide intraday scan is still a query across multiple symbols and dates; nesting order does not change what a query engine can prune, since hive-style partitioning is recognized by key name, not position. market still comes first, the same as in the EOD layout: a symbol alone is not guaranteed unique across exchanges, so nesting under market= keeps two different instruments that happen to share a ticker on different markets from ever landing in the same folder:
raw/
intraday/
market=XNAS/
symbol=AAPL/
date=2026-08-21/
data.parquet
symbol=NVDA/
date=2026-08-21/
data.parquet
market=XSTO/
symbol=VOLV-B.ST/
date=2026-08-21/
data.parquetThe physical data organization follows the access pattern instead of forcing artificial consistency.
Keeping symbol above date, rather than the reverse, also matters if a retention policy is ever introduced: a position can stay open for weeks, so deleting by calendar age of the collection date risks deleting data still supporting an open position’s stop-loss or exit decisions — the age of a date folder says nothing about whether the position it belongs to is still held. A policy would instead follow the same boundary as collection itself: once a symbol is sold out of the portfolio or dropped from the watchlist, its entire symbol= folder becomes eligible for deletion as a single unit.
The default here, though, is to not delete anything. Intraday data for a candidate that was evaluated but never entered is exactly the record that makes it possible to go back later and ask why it wasn’t taken, whether it should have been, and what the entry criteria missed — that is analytical value the trade log, which only exists for trades actually made, cannot provide. Deleting it trades a small, ever-cheaper amount of storage for the ability to learn from the decisions that were not made, which is a bad trade. A retention policy stays available as an escape hatch for if storage growth ever becomes a real constraint, not something to reach for by default.
Metadata and Configuration
Different kinds of structured information deserve different formats, but the more important distinction is what they’re for. Configuration controls how the application behaves — it’s written by a person, changed rarely, and mostly read at startup. Metadata describes what the application’s domain actually contains, and is worth keeping separate specifically when it comes from somewhere other than a person: an external reference feed, updated on its own schedule, independent of code deploys.
I would use:
YAML → configuration
JSON → metadata and simple stateConfiguration, metadata, and state all share one property: they represent current facts — a market’s operating settings, an instrument’s latest known attributes, a portfolio’s current holdings — that get revised in place as reality changes. Overwriting a file to update a current fact is the natural operation, but it also erases whatever the fact used to be, and reconstructing an earlier value later without a database means digging through backups or logs that were never meant for that purpose. Writing a new timestamped file on every change instead — with “current” simply meaning whichever file sorts last — keeps that history for free, so this pattern applies uniformly below to configuration, metadata, and state. That sort-based rule depends on timestamps being directly comparable, so filenames use a fixed, UTC-normalized format rather than a local offset — a local time zone or DST transition could otherwise make “sorts last” stop meaning “most recent.” UTC normalization does not, on its own, solve clock skew; if more than one machine could ever write these snapshots, timestamps alone would need a tie-breaker or a coordination mechanism, which this single-writer design does not currently need.
YAML fits configuration well because it handles nested application settings naturally. Markets are few, static, and hand-maintained, so their descriptive facts belong right alongside their operational settings — one file, not two:
# 2026-08-15T10-00-00Z.yaml
markets:
XSTO:
name: Nasdaq Stockholm
country: SE
currency: SEK
timezone: Europe/Stockholm
calendar: XSTO
eod_delay_minutes: 15
intraday:
enabled: true
interval_minutes: 5
XNAS:
name: Nasdaq
country: US
currency: USD
timezone: America/New_York
calendar: XNAS
eod_delay_minutes: 15config/
2026-01-01T00-00-00Z.yaml
2026-08-15T10-00-00Z.yamlInstruments are a different story: there are thousands of them, and their attributes — name, sector, currency, listing status — are too numerous to hand-maintain, so the expected write path is an ingestion job syncing from a market data provider’s reference feed, with a person only ever touching a file to fix an occasional error. YAML’s readability advantage matters when hand-editing is the normal case, as with the market config above; when a program is the primary writer, JSON is what reference feeds and APIs almost universally emit already, so parsing it directly avoids an unnecessary conversion step. That is where metadata as a separate JSON document earns its place, grouped one file per market along the same Hive-style boundary used for the Parquet data — DuckDB’s read_json recognizes hive_partitioning the same way, so a cross-market screen like “every instrument in the technology sector” can filter on market without it being duplicated inside each file:
metadata/
instruments/
market=XSTO/
2026-08-15T10-00-00Z.json
2026-08-22T03-00-00Z.json
market=XNAS/
2026-08-15T10-00-00Z.json
2026-08-22T03-00-00Z.jsonThis also avoids the file-count overhead of one document per instrument, and a re-sync from the reference feed simply adds another timestamped snapshot rather than overwriting the last known-good version.
Globbing that directory naively would return every historical snapshot rather than just the current one, duplicating each instrument once per re-sync — “sorts last” has to actually be applied, not just asserted:
SELECT *
FROM read_json(
'metadata/instruments/market=*/*.json',
hive_partitioning = true,
filename = true
)
QUALIFY filename = max(filename) OVER (PARTITION BY market);row_number() would number individual instrument rows, not files, and silently drop every instrument but one from the newest snapshot; filtering on filename = max(filename) keeps every row belonging to that file. Portfolio and watchlist queries apply the same maximum-filename rule within their respective directories. Configuration uses the equivalent filesystem operation instead of a query: the application opens the lexicographically greatest YAML filename directly.
Portfolio and watchlist state deserve the same JSON treatment, but for a different reason: intraday collection only tracks a small, changing set of instruments, and that set has to come from somewhere. Because intraday data is organized one folder per symbol, it is tempting to treat the presence of a folder as the answer to “what is currently interesting?” That conflates two different things: having collected data for a symbol in the past says nothing about whether it still matters today. It should instead be tracked explicitly, as its own source of truth — and, following the same versioning pattern as configuration and metadata, as a timestamped snapshot rather than a single file overwritten in place:
state/
portfolio/
2026-08-21T09-00-00Z.json
2026-08-21T14-32-00Z.json
watchlist/
2026-08-21T09-00-00Z.jsonBoth directories exist because they support a real-time decision. portfolio/ records current holdings, and intraday data on them supports exit and stop-loss decisions during the trading day. watchlist/ records candidates being evaluated for entry, and intraday data on them supports timing that entry. A longer-term thesis alone does not need intraday granularity — following one only requires EOD data, so it has no place in this state and does not drive the intraday collector.
The active intraday universe — the set of symbols the collector should be watching right now — is then derived from that state rather than inferred from the filesystem:
portfolio
+
watchlist
↓
unique active instruments
↓
intraday collectorIf a stock leaves the watchlist, it simply stops being collected going forward — its historical intraday data does not disappear automatically. Whether that data is later removed is a separate, deliberate retention decision (see above), not a side effect of the collector no longer being told to watch it. What the application currently cares about and what it has previously observed stay cleanly separated.
The Trade Log
Portfolio and watchlist state answer “what matters right now.” A trade answers a different question: once a decision was actually made, what led to it, and did it work? That’s a historical record, not current state, so it lives on its own.
A trade is one JSON document per execution — one buy or one sell, not a round trip. A position can be built or unwound over several buys and sells, so each execution is its own decision with its own record, rather than forced into an entry/exit pair. Because a trade can carry more than one diagram, the unit is a folder, not a single file. Expected trade volume is low, so date plus a running number is enough to keep folders unique; the instrument is just a field, and the trade log is never a deletion candidate — it is the permanent record of what was actually decided:
trades/
date=2026-08-21/
trade=01/
trade.json
chart.png
raw.parquet
derived.parquet
date=2026-08-25/
trade=01/
trade.json
chart.png
raw.parquet
derived.parquet
date=2026-09-02/
trade=01/
trade.json
chart.png
raw.parquet
derived.parquetThat is three independent trades on the same instrument: a buy, a second buy scaling in, and a sell closing it out — three folders, three decisions, nothing forcing them together. trade=01, trade=02 disambiguate same-day trades regardless of instrument. A trade with one diagram names its files plainly; more than one gets chart-1.png, chart-2.png, and so on, never named by role (entry.png) since that classification already lives in label.
{
"market": "XSTO",
"symbol": "VOLV-B.ST",
"side": "buy",
"timestamp": "2026-08-21T09:12:00+02:00",
"price": 284.50,
"quantity": 50,
"trigger": "rsi_14_oversold",
"signal_values": { "rsi_14": 27.8, "sma_20": 291.10, "sma_50": 288.40 },
"rationale": "RSI oversold bounce off rising 50-day average",
"stop_loss": 276.00,
"charts": [
{
"label": "5m",
"image": "chart.png",
"raw": "raw.parquet",
"derived": "derived.parquet",
"annotations": [{ "type": "support", "price": 280.00 }]
}
]
}signal_values, rationale, and the chart’s raw/derived Parquet files are all copies, not references — captured once, at the moment of the trade, and never recomputed. That protects two things: the derived side against indicator logic changing later and quietly altering what a past review would show, and the raw side against that intraday data eventually being deleted under a retention decision. raw and derived stay two separate files rather than one, for the same reason they are separate everywhere else in this architecture: raw is authoritative, derived is reproducible. image is the one thing that cannot be exactly regenerated from data — it could be redrawn approximately, but not with the same styling and annotations it actually had at the time — so it is persisted rather than re-rendered on demand. annotations covers small manual additions, like a marked support level, kept inline in the JSON rather than as their own file.
All paths in a trade record are relative, so its folder is self-contained and portable. A trade is written once and never revised — there is no in-place update or rename discipline to design around here.
The trade log is the authoritative history of what was actually decided; state/portfolio/ is a materialized, operational snapshot of where that history currently leaves things, kept separately because intraday collection needs a fast answer to “what’s open right now” without recomputing it from every trade on every tick. A position is not stored as its own document beyond that snapshot — how it got there is a query over the trade log, scoped to market and symbol, ordered by the actual execution timestamp rather than the folder path, with date and trade as a deterministic tie-breaker for two executions that happen to share a timestamp: a signed running sum of quantity, positive for buy, negative for sell:
SELECT
timestamp,
side,
price,
quantity,
sum(CASE WHEN side = 'buy' THEN quantity ELSE -quantity END)
OVER (
PARTITION BY market, symbol
ORDER BY timestamp, date, trade
) AS position_after
FROM read_json(
'trades/date=*/trade=*/trade.json',
hive_partitioning = true
)
WHERE market = 'XSTO' AND symbol = 'VOLV-B.ST'
ORDER BY timestamp, date, trade;That gives the same view a round-trip-shaped record would have tried to hardcode, without needing to know in advance how many buys or sells a position would end up taking.
Screening trades works the same way as screening any other JSON in this architecture:
SELECT
symbol,
date,
trade,
side,
trigger,
price
FROM read_json(
'trades/date=*/trade=*/trade.json',
hive_partitioning = true
)
WHERE trigger = 'rsi_14_oversold';That turns “did this trigger actually work over time?” into an ordinary query instead of a manual review exercise.
The Data Layout
A complete structure could look approximately like this. Configuration, metadata, and state are all small, application-owned files, distinct from the market data itself, so they sit together under app/, separate from market/. The trade log grows over time but is still application-owned rather than market data, so it sits alongside them:
data/
├── app/
│ ├── config/
│ │ ├── 2026-01-01T00-00-00Z.yaml
│ │ └── 2026-08-15T10-00-00Z.yaml
│ │
│ ├── metadata/
│ │ └── instruments/
│ │ ├── market=XSTO/
│ │ │ └── 2026-08-15T10-00-00Z.json
│ │ ├── market=XNAS/
│ │ │ └── 2026-08-15T10-00-00Z.json
│ │ └── market=XNYS/
│ │ └── 2026-08-15T10-00-00Z.json
│ │
│ ├── state/
│ │ ├── portfolio/
│ │ │ ├── 2026-08-21T09-00-00Z.json
│ │ │ └── 2026-08-21T14-32-00Z.json
│ │ └── watchlist/
│ │ └── 2026-08-21T09-00-00Z.json
│ │
│ └── trades/
│ └── date=2026-08-21/
│ └── trade=01/
│ ├── trade.json
│ ├── chart.png
│ ├── raw.parquet
│ └── derived.parquet
│
└── market/
├── raw/
│ ├── eod/
│ │ ├── market=XSTO/
│ │ │ └── year=2026/
│ │ │ └── data.parquet
│ │ │
│ │ └── market=XNAS/
│ │ └── year=2026/
│ │ └── data.parquet
│ │
│ └── intraday/
│ ├── market=XNAS/
│ │ └── symbol=AAPL/
│ │ └── date=2026-08-21/
│ │ └── data.parquet
│ └── market=XSTO/
│ └── symbol=VOLV-B.ST/
│ └── date=2026-08-21/
│ └── data.parquet
│
└── derived/
├── eod/
│ ├── market=XSTO/
│ │ └── year=2026/
│ │ └── indicators.parquet
│ │
│ └── market=XNAS/
│ └── year=2026/
│ └── indicators.parquet
│
└── intraday/
├── market=XNAS/
│ └── symbol=AAPL/
│ └── date=2026-08-21/
│ └── data.parquet
└── market=XSTO/
└── symbol=VOLV-B.ST/
└── date=2026-08-21/
└── data.parquetThe exact partitioning can evolve with actual data volume — daily data may only need yearly partitions, intraday data may eventually need finer partitioning, and there is no reason to optimize for hypothetical scale before it exists.
Query Raw and Derived Data Together
Physically separating raw and calculated data does not make queries inconvenient — DuckDB can join them at query time:
SELECT
r.date,
r.symbol,
r.close,
i.sma_20,
i.sma_50,
i.rsi_14
FROM read_parquet(
'market/raw/eod/market=XSTO/year=2026/*.parquet'
) r
JOIN read_parquet(
'market/derived/eod/market=XSTO/year=2026/*.parquet'
) i
USING (date, symbol)
WHERE r.date = DATE '2026-08-21'
AND i.rsi_14 < 30
ORDER BY i.rsi_14;This is one of the properties I like most about the architecture: storage can follow durability and lifecycle requirements, queries can follow application requirements, and the two do not need to have the same physical structure.
DuckDB and Polars Have Different Roles
DuckDB and Polars overlap, but I would give them different primary responsibilities.
DuckDB is the query engine:
Parquet
↓
DuckDB
↓
selection / joins / aggregationPolars is the computational dataframe environment:
query result
↓
Polars
↓
transformations / calculations
↓
ParquetA typical data flow might look like:
market provider
↓
Polars
↓
raw Parquet
↓
DuckDB
↓
selected dataset
↓
Polars
↓
calculations
↓
derived ParquetThird-party libraries may require pandas, and there is no need to avoid it artificially — it can be used where needed without becoming the default dataframe abstraction.
A Note on DuckLake
Recall the gap from earlier: once a derived table gets recalculated, a glob has no way to tell the new files from whatever a previous run left behind. DuckLake is the general answer to that — a lakehouse format that layers a transactional catalog on top of plain Parquet files, tracking table versions and file membership explicitly, which gives time travel, schema evolution, and safe concurrent writers. It does not fit this architecture, for two reasons. Its catalog needs a transactional SQL database behind it, reintroducing exactly the stateful component this design set out to avoid. And it moves the rules for what counts as “current” data out of application code and into the extension’s own metadata tables — control I would rather keep. The gap itself is closed more simply here: each partition is one fixed-name file — indicators.parquet, not a numbered series — atomically replaced on every recalculation, so there is never more than one file for a glob to be confused about in the first place.
Making Writes Reliable
File-based persistence becomes much simpler to reason about if writes are safe to repeat and never leave a file half-written. Authoritative files should not be modified in place — a failed process could leave them corrupted — so every write follows the same shape instead:
calculate new dataset
↓
write temporary file
↓
validate
↓
atomic rename over existing fileIf the process fails at any point before the rename, the existing file is untouched and still valid, and simply running the whole thing again produces the same result — retries become normal rather than dangerous, which reduces the need for elaborate job-state infrastructure. That guarantee holds on the intended local POSIX filesystem, where the temporary file is written in the destination directory before being renamed into place; surviving sudden power loss additionally requires fsyncing the temporary file before the rename, and fsyncing the containing directory after it, so both the data and the rename itself are durable.
That guarantee is also per-file, not per-update. Raw and derived Parquet, portfolio state and the trade log, or metadata across markets can each be individually consistent while a reader or a backup catches one file mid-generation and another one file behind it. This design provides per-file atomicity, not transactional consistency across files — acceptable only while consumers tolerate eventual consistency between related datasets, or while reads and backups are avoided during a multi-file publication. Concretely, a recalculation can briefly expose new raw data joined against still-old derived indicators, or vice versa, until the second file’s rename catches up.
Intraday collection is a concrete case of the same pattern. Rather than assuming each run successfully appends exactly one new observation, each run merges the provider’s response into what already exists, deduplicates by timestamp, and sorts, before going through that same write-temp/validate/rename flow:
existing today's data
+
provider response
↓
merge
↓
deduplicate by timestamp
↓
sort
↓
temporary Parquet
↓
atomic rename over existing fileRecalculating and writing a snapshot for a given date and market this way results in the same final state whether it runs once or three times.
Backup Follows the Persistence Model
Backup here works at two layers, because they protect against different failures.
The first is the instance itself: a periodic VM snapshot, restorable as-is for disaster recovery. That is the fastest way back if the machine is simply lost, but it also carries along everything else on that instance — OS state, installed packages, whatever configuration drift has accumulated — which makes it a poor fit for anything beyond recreating the exact same machine.
The second is the data directory on its own, independent of any particular instance. Because persistence is file-based rather than database-based, this is an ordinary file-backup problem rather than a database-specific one: a tool like restic archiving configuration, metadata, state, raw market data, derived market data, and the trade log to any object-storage or SFTP target needs no export step, no vendor-specific dump format, and no database-aware backup tooling to operate. Because that backup is decoupled from the machine it came from, it can be restored onto a different instance just as easily as the original one.
That second layer is what makes a phoenix-server pattern practical for upgrades: instead of patching a running instance in place, a clean instance is provisioned, the data directory is restored onto it from backup, and the old instance is discarded. The data outlives the instance; the instance does not need to.
The Selected Storage Architecture
Putting everything together:
Application Code
│
┌──────────────┴──────────────┐
▼ ▼
YAML DuckDB / Polars
configuration │
(read directly) ┌──────────┴──────────┐
▼ ▼
JSON Parquet
metadata / state raw / derived
/ trades dataApplication code reads YAML configuration directly — there is no reason to route settings a person hand-edits through a query engine. JSON goes through the same path as Parquet instead: metadata, state, and the trade log are queried with DuckDB’s read_json the same way raw and derived market data are queried with read_parquet, even though the two formats suit different shapes of data. Raw versus derived, storage versus query, transactional versus analytical — these boundaries give the architecture its structure without requiring a catalog database or a separate service to express them.
That is what I like most about it: modular without being distributed, SQL without database-managed storage or a database server, valuable datasets persisted without operating one — deliberately boring in some places, unusually capable in others. For this storage layer, that means Python, DuckDB, Polars, Parquet, YAML, JSON, and a filesystem: not a fashionable stack, but one that leaves very little architecture between the data and the code that queries it.
Built a persistence layer that skipped the database entirely, or found the point where files genuinely stopped being enough? Leave a comment below, or reach out through another channel if you would rather keep the conversation private. Follow the RSS feed for more.