Most software testing begins in the present.
We install the current build. We create a new project, document, database, or configuration file. We exercise the new feature. We confirm that the information saves, closes, reopens, and still behaves properly.
Everything passes.
The problem is that our customers don't begin in the present.
They bring the past with them.
They bring databases created by an earlier release. They bring settings written before a new option existed. They bring copied projects, restored backups, documents with older metadata, and files that were perfectly valid when the software created them.
That history is part of the product whether the current source code acknowledges it or not.
Your current code describes the software you have today. Your users' existing data describes the promises you made yesterday.
Both belong in the specification.
A clean installation only proves one thing
There's nothing wrong with testing a clean installation.
We absolutely need to know that the current version can create its own data correctly. A new customer should be able to install the program, create something, save it, and continue working.
But that's only one path through the product.
A clean installation doesn't tell us what happens when the new build encounters:
- A database created six months ago
- A document that doesn't contain newer metadata
- A project copied from another computer
- A backup made before a schema extension was introduced
- Settings that use an older default
- A file that has already passed through several previous upgrades
- A valid older state that the current code no longer knows how to recognize
The fresh-data test tells us whether the software can begin a new history.
It doesn't tell us whether it can continue the history it already has.
For a product with existing users, continuing that history is often the more important test.
The current schema isn't the whole truth
When we look at a database definition, configuration model, or document structure in the current source code, it's easy to treat that as the truth.
These are the tables the application requires.
These are the fields the document contains.
These are the settings the program expects.
These are the values that must exist.
That's true for data created today.
It may not be true for data the product created last year.
An earlier version may have created fewer tables. A newer feature may have added optional metadata. A later build may have extended an existing schema without changing every older file already carrying that schema number.
The current definition tells us what the software creates now. It doesn't automatically tell us every valid state the software may encounter.
That's where compatibility failures hide.
The new code says:
Every valid database contains this table.
The product's history says:
That wasn't true when some of these valid databases were created.
Both statements can make sense within their own timeframe. The failure comes from pretending only one timeframe exists.
AI naturally sees the product as it exists now
This problem becomes especially important when we're programming with AI.
When we give an AI the current repository and ask it to add a feature, the AI normally reasons from what it can see:
- The current schema
- The current initialization code
- The current data structures
- The current tests
- The current documentation
- The current expected behavior
That's reasonable.
The AI isn't automatically aware of every state produced by every earlier build. It may see a table being created during current database initialization and conclude that the table will always be there.
It may update the code that creates new databases without asking what happens when an existing database is opened.
It may write a test that creates a new file, exercises the new feature, and proves that the current code works perfectly with data created by the current code.
Again, that's not the AI being stupid.
It's the AI working from the specification we gave it.
If the repository describes the present but doesn't preserve enough information about the past, the AI can't be expected to reconstruct that history by instinct.
The programmer has to add the time dimension.
We have to tell the AI:
- Existing customer data is authoritative.
- Older valid files must continue to open.
- This table or field may not exist in earlier data.
- The upgrade must preserve all existing content.
- The repair may run more than once.
- Save As, backup, restore, and duplication must be tested too.
- A successful clean-file test is not enough.
Once that context is available, AI can be extremely useful. It can inspect schema history, compare initialization paths, locate assumptions, write migration logic, and help build a meaningful compatibility test matrix.
But it can't protect a history we've failed to include in the assignment.
Valid old data isn't corrupt data
There's an important distinction here.
Sometimes a file really is damaged. A write may have been interrupted. A user may have manually modified it. Storage may have failed. A third-party tool may have changed something it didn't understand.
That's corruption.
But a file isn't corrupt merely because it was created before the latest feature existed.
If an earlier released version created the file correctly, and the user has continued using it normally, that file is part of the supported product history.
Calling it obsolete or malformed doesn't erase the responsibility.
The current version needs one of three honest policies:
- Open and use the older data directly.
- Upgrade it safely.
- Clearly declare that compatibility has ended and provide a deliberate transition path.
What it shouldn't do is crash because the code silently assumed that every user started with today's build.
Test through four doors
A useful compatibility test doesn't need to become an enormous archaeological project.
For most persistent data changes, I want to know what happens through at least four doors.
Door one: Fresh creation
Create the project, document, database, or settings file using the current build.
Then exercise the feature, save it, close it, reopen it, and verify the result.
This proves that the current version can create and use its own expected structure.
It's necessary.
It just isn't sufficient.
Door two: Existing valid data
Open a real file created by an earlier version.
Don't recreate what you think that file looked like. Use an actual historical artifact whenever possible.
That matters because our memory of an older schema is often incomplete. A recreated test fixture may contain the fields we remember while missing the odd combination that existed in the real product.
Open the file with the new build.
Verify that it loads without losing information. Use the new feature. Save the result. Close it. Reopen it.
The test isn't complete when the file opens.
The test is complete when the older data survives the journey into the current version.
Door three: Copy, backup, restore, and Save As
This is an easy path to overlook.
Suppose an older database can be opened because some compatibility behavior occurs during one particular workflow. What happens when the user makes a copy?
Does Save As copy the original older structure before the repair is applied?
Does a backup preserve the pre-upgrade state?
Does Restore take a path that bypasses the normal migration?
Does duplicating a project reproduce an old missing field or table?
Copy operations are often treated as simple file movement. But copying historical data also copies historical assumptions.
A successful open test doesn't automatically prove that every derivative copy will be safe.
Door four: Repeated migration
A compatibility repair should usually be safe to run more than once.
The application may check the file every time it opens. A crash may interrupt the first upgrade. A copied database may already contain some—but not all—of the newer structures. A user may open the same file with several builds during testing.
The repair needs to ask what exists and add only what's missing.
It shouldn't assume that migration is a one-time ceremonial event that always runs from a perfectly known starting state.
That's why idempotence matters.
A safe migration moves old data toward the current structure. Running it again should leave already-correct data alone.
The version number may not tell the whole story
Schema and file-format version numbers are useful, but they can create false confidence.
We'd like to believe that every file carrying version 4 has one exact structure, every version 5 file has another, and the upgrade path between them is perfectly defined.
Real products don't always develop that neatly.
A feature may add an extension table without changing the main version. An optional capability may create its supporting data only after the user activates it. A build may add a field while preserving the broader format number for compatibility.
Two files can honestly report the same main version and still reflect different points in the product's history.
That means the migration code sometimes needs to inspect reality instead of trusting one label.
Does the required table exist?
Does the column exist?
Is the metadata present?
Has this particular extension been initialized?
The version number is evidence.
The actual structure is stronger evidence.
Good compatibility repairs are narrow
When older data fails, there's a temptation to solve too much at once.
We may decide that this is the perfect time to redesign the schema, normalize old tables, rename fields, reorganize storage, and remove every historical oddity.
That's usually where a compatibility repair becomes a migration project.
Sometimes a larger migration is justified. But it should be recognized and tested as a larger project.
A repair intended to let valid older files open safely should normally be narrow:
- Identify the exact missing structure
- Add only what's required
- Preserve every existing record
- Avoid rewriting unrelated data
- Make the repair safe to repeat
- Run it at a dependable point in the open process
- Verify both old and current files afterward
The goal isn't to make yesterday's database look as though it was created today.
The goal is to make it safely usable today.
That's a smaller and more responsible assignment.
Put historical states in the test inventory
One practical lesson from this is that development projects need more than sample data.
They need historical samples.
For any product that owns persistent data, it's useful to retain representative artifacts from important points in its life:
- A file from the first released format
- A file from the most recent prior release
- A file created before a major feature was added
- A file containing a retired or unusual state
- A copied or restored file
- A file that has already passed through one migration
- A current file containing realistic user data
These shouldn't be random backups with mysterious origins.
They should be identified test assets.
We should know which version created them, what features they contain, and what the current build is expected to do with them.
That turns product history into something the programmer and the AI can test instead of something they have to remember.
Documentation needs a history lane
The same principle applies to project documentation.
Current documentation should clearly describe current architecture and accepted behavior. The AI shouldn't have to read thousands of lines of old build history before making any change.
But the relevant history can't disappear entirely.
There needs to be a reliable place to answer questions such as:
- When was this table introduced?
- Could an older valid database lack it?
- Did a previous release use a different default?
- Was this field once optional?
- Does this strange-looking branch protect an old file format?
- Which migrations have already shipped?
- What customer data must never be rewritten?
Current authority and historical explanation have different jobs.
Mixing them into one giant document creates clutter.
Discarding the history creates amnesia.
The right structure keeps the present easy to find while preserving the past where it can be consulted when a change crosses that boundary.
Compatibility is a product promise
Backward compatibility isn't always permanent.
There are times when an old format genuinely has to be retired. Security requirements may change. An old platform may no longer be supportable. The cost of carrying a decades-old structure may exceed its value.
But that should be a product decision.
It shouldn't happen accidentally because nobody tested an old file.
When software accepts a user's work and saves it, the user has a reasonable expectation that the software will continue treating that work with care.
The program may need to upgrade it.
It may eventually need to export it into a newer form.
It may need to warn the user that a transition is required.
But persistent data is not disposable implementation detail.
It belongs to the user.
That makes compatibility more than a technical convenience. It's part of the trust relationship between the product and the person using it.
The programmer still owns the timeline
AI can write migration code faster than I can.
It can inspect every place a table is referenced. It can compare schema initialization with open-time behavior. It can identify assumptions and generate tests. It can often diagnose the exact failure as soon as it's given the old file and the error.
That's valuable.
But the programmer still has to ask the question the repository may not ask on its own:
What valid history can arrive at this line of code?
We know which versions were released.
We know which experiments reached customers and which stayed internal.
We know whether an odd data state is corruption, a supported older format, or the residue of a failed development build.
We know what promises the product has made.
The AI can help us carry those promises forward.
It can't decide which promises existed unless we tell it.
Add time to the specification
When a feature touches persistent data, the specification shouldn't only describe the desired final structure.
It should also describe the journey.
What can exist before the change?
Which older states are valid?
How will each one move forward?
What must remain untouched?
What happens when the migration runs twice?
What happens when the user copies the data before or after the repair?
How do we prove that current files still work and old files haven't lost anything?
Those aren't secondary cleanup questions.
They're part of the feature.
The current source code is part of the specification.
The current schema is part of the specification.
The new behavior is part of the specification.
And for any product that already has users, the past is part of the specification too.
Related field note
For the ProHelp Viewer build that forced me to confront this directly, read The Build That Worked Until I Opened Yesterday's Database.