Room to Experiment, Confidence to Release

In my current assignment, I am building up a data and analytics team. One of the early questions, which we are working through together with the platform team, is how changes to applications, pipelines, and models should move from an idea to production, and which environments they should pass through on the way.

This post presents a hypothesis for that, not a settled design. Its governing idea is simple:

Give each change a realistic, isolated environment, let people explore in it without disturbing production, and accept the change only once it has been validated against what it will actually join.

The rest of the post follows one application change through that journey, and then shows how model development extends it.

The Problem, Through One Change

Assume a landscape with a few solutions, among them a pipeline, ingest, that loads data from the business systems, and an application, reporting, that presents it. A developer wants reporting to break figures down by customer segment. That means ingest must start capturing the segment, a migration must add and backfill a segment column in a large existing table, and reporting must show the new breakdown.

Four things make this harder than it looks:

• Synthetic data hides what real data does. Whether the backfill finishes in minutes or hours, and what happens to rows with missing or unexpected segment values, depends on production’s volumes and quirks. Synthetic fixtures rarely have either.
• Shared environments let changes interfere. If several people work in one Dev or QAT environment, a colleague’s migration or pipeline run can change the data under someone else’s test, and nobody knows which change caused what.
• A fresh installation is not an upgrade. The migration will meet a table with years of history, and the pipeline a landscape with its own schema history and running versions. Testing on a clean install misses most of what goes wrong when upgrading an existing one.
• A passing test can expire. If someone else merges a change while this one is being tested, the result no longer represents what this change would actually join.

The design below is an answer to those four problems.

Two Clusters, Isolated Environments

Data and analytics gets clusters of its own, separate from the organization’s shared ones. A data and analytics platform concentrates sensitive data from across the business, and a cluster boundary narrows who and what can reach it. Its workloads are also noisy neighbors: transformation jobs, training runs, and heavy queries scale out in bursts that make shared capacity unpredictable for everyone else.

The two clusters separate changes under development and validation from workloads approved to run in production. Both are long-lived and operated by the platform team using the same platform standards.

An environment is a set of namespaces in the QAT cluster, one per solution, together with its own copy of the data. It is created for one pull request and destroyed when that pull request is merged or closed. There is no permanent shared Dev or QAT environment; many isolated environments come and go instead. Each one owns its clones and databases and has its own identities, so a migration in one pull request cannot touch another’s data.

The QAT cluster runs unapproved code, but it holds production-derived data, and that data needs the same protection as in production. Requirements on security and business continuity, from regulation such as NIS2 and from customers who depend on the services, make that non-negotiable. Production-derived data exists only in the two clusters, never on laptops or in loosely controlled environments. Access to it follows each person’s production entitlements.

Unapproved code, in turn, is contained. It can neither reach production nor cause side effects outside its environment: there is no network path from QAT to PRD, production data arrives only as snapshots delivered into QAT’s own storage, egress is blocked by default, and external integrations are replaced by controlled substitutes.

Both clusters share one platform configuration, managed through GitOps, which reduces the configuration differences between an environment and production without making their behavior identical.

One Monorepo, One Candidate

Applications, pipelines, models, shared libraries, deployment definitions, and synthetic test fixtures live in one monorepo. For our change, that means the change in ingest, the migration, and the change in reporting are one branch and one pull request. Reviewers see the whole change, it is tested as a whole, and it is rolled out in a compatible sequence, without coordinating merges across repositories.

A monorepo does not mean everything is built or released together. Components keep their own owners, who approve changes to them, and their own deployment units. Dependency-aware builds in Dagger limit work to the components a change affects. The cost is that everyone with access can read everything, which is normally acceptable within one team.

From Local Development to Realistic Testing

The developer starts locally. Dagger runs the same pipelines on a laptop as in CI, against synthetic fixtures kept next to the code. That is enough to get ingest capturing the segment, the migration running, and reporting showing the breakdown. It is not enough to know how the backfill behaves on production’s volumes, or what the real segment values look like.

So once the change works locally, the developer opens a pull request, as a draft while it is still taking shape. That creates the change’s environment in the QAT cluster, in four steps:

1. A clone is made from the latest production snapshot in QAT’s storage, using copy-on-write so it is cheap.
2. The revisions that were running in production when the snapshot was taken are deployed on top, so data and code match.
3. Accepted changes from the recorded production baseline up to current main are applied, in merge order, including their migrations.
4. The pull request is applied last, including its own migrations.

The result is the upgrade production is expected to take: production as it is, plus what has already been accepted, plus this change. The developer can now test the backfill against realistic volumes and inspect unexpected segment values without affecting production or anyone else’s change. Every push updates the environment.

People can read the environment’s data within their entitlements, but only what would run in production — the deployment, its migrations, and the landscape’s own pipelines and jobs — may change it. Ad hoc queries and notebooks that need to write go to a scratch area that belongs to the environment and is never read by the landscape. That keeps the environment’s data a product of production behavior.

From Acceptance to Production

While our developer has been working, a colleague has merged a change that adds a migration to the same table. It has not been promoted yet. The developer’s environment as it stands no longer represents the candidate that would join the current main.

Acceptance starts when the developer marks the pull request ready for review. Forgejo Actions first builds the affected components with Dagger and runs the fast checks, such as unit tests and static analysis, publishing immutable artifacts. Only if those pass does Forgejo notify the control plane, so the expensive part never starts on a change that does not build. Before validating, the environment is always rebuilt in the same four steps: from a fresh production snapshot, through the current main — now including the colleague’s migration — to the pull request. That removes everything left over from experimenting, and tests the intended upgrade path.

Acceptance then follows a single rule:

A pull request is merged only if it was validated against the current main.

For our change, the checks cover the duration and reversibility of the migration and backfill on production volumes, the quality of the new segment column, ingest capturing the segment on its next run, and reporting showing the breakdown correctly. The control plane reports the result as a status check on the pull request, recording the snapshot, the revision of main, and the artifacts the validation used.

At merge time, the system must atomically verify that both the candidate and the target main still match the successful validation. If either has changed, acceptance runs again, starting from the rebuild. Protected branches and required status checks in Forgejo are the natural place to enforce that.

Acceptance is therefore optimistic: changes are validated in parallel, with no queue, and the price is that a validation is wasted when another change merges first. With a small team, a modest merge rate, and validations measured in tens of minutes, I expect that to be rare and cheap. Making running validations visible lets colleagues hold a merge for a few minutes when it matters.

The diagram is deliberately conceptual. It shows who decides what, and in which order, not how each step is carried out: every rebuild and every deployment, for example, goes through Argo CD reconciling a declaration, as described further down.

Once merged, the change is promoted to PRD. Promotion reuses the validated, immutable artifacts rather than rebuilding them, and changes reach production in merge order, with each validated against its accepted predecessors. Before promoting, it checks that production is still on the validated path: its deployed revisions and schema versions are those recorded at validation, or those of earlier changes in main promoted since. Anything else, such as a schema change from outside this process, means validating again.

Promotion should follow merging as closely as possible, ideally automatically. Every unpromoted change is replayed in every new environment, stacks its migrations onto combinations production has never run, and blocks everything behind it if it fails on the way out. With a gap of zero or one change, an environment is essentially production plus the pull request. That is also where this design meets trunk-based development and continuous deployment, which are topics of their own.

How Model Development Differs

With customer segments in place, the team wants a demand forecast per segment. A model builder changes the forecasting model, demand-forecast, and reporting changes to show the new forecasts. It is the same path as before — one pull request, one environment, the same acceptance rule and promotion — with four differences.

Real data is needed earlier. Whether a forecast per segment is any good cannot be judged on synthetic data. Fixtures still help to get the pipeline mechanics right locally, but the model builder opens a draft pull request on the first day and does most of the work in its environment.

Experiments write to disposable scratch storage. Notebooks and training runs read the environment’s clone and write their results to its scratch area. They run with the entitlements of the person who started them, so a model builder who may see only certain sections of the data in production sees only those sections here, and a model trained on restricted data is restricted in the same way. Long training runs checkpoint, since idle environments are suspended.

Accepted results must become code and durable artifacts. The scratch area disappears with the environment, so whatever an experiment establishes has to be turned into code in the pull request: feature definitions, training and evaluation code, and pipeline steps. Notebooks can be committed, but stripped of their outputs, which are production-derived data.

An experimental model becomes a release artifact only when the controlled training pipeline reproduces it from versioned code and a recorded data snapshot, then publishes it to the protected artifact store. Its version is recorded in the repository. If it cannot be reproduced that way, it is not ready. A retrained model, even without a code change, is a new artifact and goes through a pull request like any other change.

Acceptance includes model evaluation. On top of the usual checks, the model’s predictive performance is compared with the approved version on the same data, and the quality of the data it was trained on is checked.

Tools, Reconciliation, and Capacity

Who Owns Which Decision

The only custom component is a small environment control plane. Everything else is existing, open-source tooling, chosen because portability is a requirement: every component can be self-hosted and runs on any conformant Kubernetes, so the landscape can move between providers without being redesigned. In the terms of Own, Control, Rent, this is a capability to control, even where the infrastructure is rented.

The pieces are connected by reconciliation rather than by a workflow that assumes every step succeeds once. When a pull request is marked ready for review and its build and fast checks pass, for example, the control plane declares a rebuilt environment with a new snapshot and the current main, Argo CD brings the cluster in line with that declaration, and the control plane starts the checks only once it observes the environment ready. If anything restarts or an event is missed, each loop converges again.

How Capacity Is Shared

Environments are admitted when their resource profile fits the budget, with a limit per person, and a running validation takes precedence over environments that are only being explored in. Most environments are idle most of the time, so their compute is suspended while their data clones are kept. Suspension is expressed in the environment’s declaration, never by scaling workloads down behind Argo CD’s back. People usually work on one change at a time, so the team’s size is a useful starting point for the budget, though overlapping pull requests mean the limits still have to be enforced.

By default, the QAT cluster also sleeps outside working hours: environments are suspended, and its worker nodes scale down to zero. The cluster and the storage holding the clones stay in place, so waking it does not require rebuilding the cluster. Each environment’s retained data remains available, and its workloads restart or resume from checkpoints. Anyone who needs it out of hours can wake it themselves, and long training runs can keep it awake. Compute is then paid for mostly when the team is working.

What Makes the Hypothesis Viable

Three things have to hold:

• Production-derived environments can be provisioned quickly and protected properly. That needs copy-on-write clones for both data files and databases, access policies reapplied on every restore, and a maximum clone age, for example a week, so that data erased in production does not live on in copies. Provisioning from snapshots exercises part of the restore path many times a day, but it does not replace testing recovery from backups.
• Small pull requests stay affordable. If an environment takes long to create, people make fewer and bigger pull requests, which stay open longer and lose more races. A prepared base with production revisions already deployed, and practices such as feature flags and expand-and-contract migrations, keep small changes cheap.
• Validated changes reach production promptly, so the chain of pending changes stays short.

Two signals would change the workflow. Frequent lost races would justify serializing validations with a short lease on main. Expensive rebuilds would justify reusing environments that are demonstrably still a valid upgrade, instead of always rebuilding before acceptance.

The aim is to give people room to explore independently and a dependable way to establish that their changes are ready for production.

Further Reading

If you want to dig deeper into the pieces this builds on:

• The NIS2 directive, for the security and continuity requirements behind the boundary
• Kubernetes on multi-tenancy, for what namespaces can and cannot separate
• Forgejo Actions, for the CI triggers and status checks that gate each merge
• Dagger, for pipelines that run the same way locally and in CI
• Argo CD on automated sync, for how environments are kept in line with their declarations