We ship sixteen document templates. Each one is a schema — the fields to extract, their types, the prompt that extracts them — and each one lives as a row in a database table.

The schemas are defined in the repository. They are also, it turned out, defined in production, differently, and had been for months. Half of them disagreed, in both directions at once, and no tool we had could tell us so.

This is the case study: how configuration in a database drifts, why "the repository is the source of truth" quietly stops being true, and the small script that turned an invisible problem into a fifteen-line report.

Two paths into the same rows

The template schemas reach the database two ways.

The seeder defines all sixteen and writes them wholesale. It is the source of truth in the repository sense: read it and you know what a template is supposed to be.

Point migrations fix one field on one template — add a placeholder, correct a required flag, rewrite a prompt.

Each is reasonable. Together they are a trap, and the trap has a specific shape: the seeder cannot run in production. It rewrites every field definition and every prompt of all sixteen rows, which would erase precisely the point fixes the migrations carefully applied.

So the seeder runs locally and in tests, and migrations run in production. Which means a fix that lands only in the seeder never reaches production — and a fix that lands only in a migration is erased the next time someone reseeds a development database.

Drift is not a risk in that arrangement. It is the default.

Both directions at once

When we finally compared, eight of sixteen templates had drifted, and the interesting part is that they had drifted in opposite directions.

Production had edits the repository never saw. The birth certificate template carried four field definitions present in no commit — back_side, father_citizenship, mother_citizenship, seal. Someone had fixed something directly, correctly, and the repository had no idea. Worse, our own tests had recorded those four as accepted "orphans" — unmapped fields we had decided to tolerate. They were not orphans. They were the fix.

The repository had edits production never saw, and this direction was losing data. The Korean export declaration declared thirteen fields in its schema — freight, container seal, licence number, shipment end date and nine more — that the production prompt never asked for. The extractor pins its JSON keys to declared names, but a field the prompt never mentions is one the model has no reason to fill. Those thirteen came back empty in production, on every Korean declaration, for as long as the two sides had been apart.

Nobody had reported it. An empty field looks like a field the document did not have.

The thing we did not have

The reason this ran for months is embarrassingly simple: there was no way to ask.

Comparing them meant reading the seeder, dumping production rows, and diffing sixteen nested JSON structures by eye. Nobody was going to do that on a hunch, and no alert existed because nothing was watching.

So we wrote the smallest possible thing: a script that reads the seeder's definitions, reads the same rows from the database, and prints what differs. Read-only, safe to run against production, about a hundred lines.

The first run produced the eight-template report above. Every finding in this post came from it, on the first execution.

That is the actual lesson, and it is not about templates. Configuration that lives in a database needs a diff, or it will drift and nobody will find out. Code has one built in; git status answers the question continuously and for free. Data has nothing, and the absence is invisible — you do not notice a missing check, you only notice the failures it would have caught, which is to say you do not notice.

Making the same mistake one layer down

Here is the part worth the read, because it happened after we thought the problem was solved.

The diff compared two columns: the field definitions and the extraction prompt. Those are the schema. That felt complete.

Templates also carry routing configuration — the keywords that decide which template a document matches, the negative keywords that exclude it, the match threshold. Same table, same two write paths, same drift exposure. Invisible to a diff that only looked at two columns.

And it had drifted, with a consequence. Our passport template was claiming extracts from the register of individual entrepreneurs: a confident 0.367 against its own 0.3 threshold, on a six-page document containing no passport. Not one of the template's identifying keywords matched. Two generic phrases — the words for "citizen" and "Russian Federation", which appear in essentially every Russian official document — carried it there, because a register extract quotes the entrepreneur's identity document.

We had built the check that makes drift visible, and then scoped it so that the next drift was invisible. Closing one channel while leaving another open beside it is a very easy mistake to make, and it looks exactly like diligence.

The fix was three lines: add the routing columns to the query. The check that mattered was verifying the diff could actually see a routing change — we rolled the fix back, watched the report name the difference, and rolled forward again.

What we would tell someone with config in a database

1. Assume drift, and build the diff before you need it. It is a day of work and it answers a question you otherwise cannot ask at all. Ours found eight problems on its first run.

2. Diff everything in the row, not just the part you think of as "the schema". Whatever column you leave out is where the next surprise lives. Ask what else in this row changes behaviour.

3. Make it safe to run in production. Read-only, no writes, no side effects. A check people are nervous about running is a check that does not run.

4. Treat unexplained production state as a fix, not as noise. Our tests had classified four legitimate production fields as tolerated orphans. A tolerance list is where evidence of drift goes to be forgotten — audit yours.

5. Verify the check detects what it claims to. Break something deliberately and confirm the report names it. A diff nobody has seen fail is not yet a diff.

The general shape

The phrase "the repository is the source of truth" describes an aspiration, not a mechanism. It is true only where something enforces it. For code, the deployment enforces it: what runs is what was committed.

For configuration in a database, nothing enforces it by default. The row is whatever the last writer wrote, and the writers do not agree with each other. "Data is code too" is the slogan; the operational content of it is that data needs the same thing code gets for free — a continuous, cheap, boring comparison between what you meant and what is there.

Ours takes four seconds and prints one line when everything matches. That line — 16 templates match the seeder — is now part of the deployment checklist, and it is the only reason we would notice if this happened again.

Key Takeaways

Two write paths into the same rows guarantee drift, especially when one of them cannot safely run in production. Ours drifted on eight of sixteen templates.
Drift goes both ways. Production carried four fields present in no commit; the repository carried thirteen the production prompt never asked for, silently returning empty on every document.
The reason it lasted months is that nothing could ask the question. A hundred-line read-only diff found everything on its first run.
We then made the same mistake one layer down, scoping the diff to the schema columns and leaving the routing columns invisible — which is how a passport template came to claim a business-registry extract.
A tolerance list is where drift hides. Four fields our tests had accepted as orphans were in fact an undocumented production fix.

FAQ

What is configuration drift in a database?

It is the state where the configuration your code assumes and the configuration actually stored diverge. It happens whenever more than one path writes the same rows — a seeder plus migrations, say — and nothing continuously compares the two.

Why can't the seeder just run in production?

Because it rewrites every row wholesale, which would erase the point fixes that migrations applied. That is exactly why the two paths exist, and exactly why they drift apart: the safe path in production is not the one that defines the truth in the repository.

How do you detect drift?

Write a read-only script that reads the definitions from your repository, reads the same rows from the database, and prints the differences. It is small, safe to run against production, and it is the only thing that turns an invisible problem into a report.

Which columns should the diff cover?

All of them that change behaviour, not just the ones you think of as schema. We scoped ours to field definitions and prompts, and the routing columns — keywords, exclusions, match threshold — drifted invisibly until a passport template started claiming business-registry extracts.

How do you know the diff works?

Break something on purpose. Roll a migration back, run the report, and confirm it names the difference — then roll forward. A check that has never been seen to fail has not been tested, only written.

Conclusion

Nothing in this story required a mistake by anyone. Every migration was correct. Every seeder edit was correct. The seeder was rightly kept away from production. The whole failure came from the absence of a comparison — a thing that costs a day to build and then costs four seconds a run forever.

If your configuration lives in a database, the question worth asking today is not "has it drifted". It is "if it had, how would I find out". If the answer involves reading two things side by side and squinting, you already know what to build.

If you want document translation where template schemas are checked against the repository on every deployment, try KTTC.