The AI returned the repair.
The filename said it was fixed.
The report said it was fixed.
The SHA-256 said otherwise.
Not almost the same.
Not a repair that failed one test.
Not a build that contained the wrong version.
The supposedly repaired ZIP was byte-for-byte identical to the original unfixed return.
That ended the investigation rather quickly.
This started as a very ordinary repair
I was working on the administration side of the new ClarionDesktop.com, which we're building as a global resource and front door for all things Clarion.
An AI worker had completed a bounded assignment and returned a deployment package.
Most of the actual feature work looked good, but there was a problem with the validation package.
Some of the validation code contained hardcoded paths from the worker's temporary environment.
That meant the validation worked where the AI had built it, but failed when the package was extracted somewhere else.
This wasn't a mysterious architectural problem.
It was a straightforward packaging defect.
So we commissioned a bounded repair.
Remove the hardcoded validation paths.
Make the validation run correctly from a clean extraction.
Return a corrected package.
Nothing else needed to change.
The worker came back with the repaired return.
Or so it appeared.
The package looked right
This is exactly the kind of situation where AI can make you waste time if you aren't careful.
The returned filename identified it as the fixed package.
The response described the repair.
Everything about the handoff suggested that the next step should be qualification.
We could have extracted it.
We could have run the validation.
When that failed, we could have started examining the scripts.
Maybe the relative-path repair was wrong.
Maybe the working directory was different.
Maybe something hadn't been included in the ZIP.
Maybe we needed another patch.
That would all have been reasonable debugging if the package had actually changed.
But before doing any of that, we checked the SHA-256.
The returned ZIP was:
459d61167204cd062da2c66118a7c8efbbe0ed71aebbe66ab16ccfd20947368c
So was the original unfixed return.
There was nothing new to debug.
They were the same file.
That one fact changed the question
Before checking the hash, the question appeared to be:
Why doesn't the repair work?
After checking the hash, the question became:
Why did the worker return the old artifact?
Those are completely different problems.
The first could send us into the source code.
The second tells us not to touch the source code at all.
Maybe the AI repaired another copy and packaged the wrong one.
Maybe it wrote the fix in its workspace but never refreshed the return ZIP.
Maybe it copied the original package over the repaired one.
Maybe something else happened.
At that point, I didn't particularly care.
The important fact was already established.
The artifact delivered to me had not changed.
This is why mechanical evidence matters
One of the things AI is extremely good at doing is constructing a plausible explanation.
That is usually helpful.
But when something goes wrong, the explanation can cause us to spend time reasoning about a state that doesn't actually exist.
If I had accepted the worker report as the starting point, I would have assumed that I was debugging a failed repair.
The hash prevented that assumption.
It didn't tell me what the worker did internally.
It didn't tell me why the wrong ZIP came back.
It didn't tell me whether a correct repair existed somewhere in the worker's environment.
It answered a much narrower question:
Is the returned ZIP different from the ZIP that needed repair?
No.
That was enough.
Same bytes means stop
I like tests that let you stop.
A lot of debugging evidence merely gives you another direction to explore.
A matching hash can sometimes do something better.
It can eliminate the entire branch.
There is no reason to inspect whether the new validation script is correct because there is no new validation script in the artifact.
There is no reason to compare expected repair behavior because no repaired artifact was delivered.
There is no reason to wonder whether some subtle regression was introduced.
The package is the same package.
Stop.
Send it back.
That is a much better use of time than asking another AI to reason about why an imaginary repair doesn't work.
The corrected instruction became much simpler
Once we knew what had happened, the return instruction could be extremely specific.
The worker didn't need a long explanation of the validation defect again.
It needed to know that the file it had returned was byte-for-byte identical to the original and therefore could not be accepted as the repaired package.
It also needed an acceptance condition:
The new return ZIP must not have the old SHA-256.
That doesn't prove the next ZIP is correct.
But it proves one very useful thing before any further testing begins:
At least you gave me a different file.
Then we can validate whether that different file contains the actual repair.
AI development needs more stop conditions
I think this is something we're going to need more of as AI development becomes more agentic.
AI is very good at continuing.
If something fails, it will inspect.
Then patch.
Then inspect again.
Then try another explanation.
That persistence is useful until the work is moving down the wrong branch.
Mechanical evidence gives us stop conditions.
Same hash? Stop.
Forbidden dependency found? Stop.
Clean extraction fails? Stop.
Expected file missing? Stop.
Regression test fails? Stop.
You don't need more creativity yet.
You need the previous stage corrected.
That distinction can save enormous amounts of AI time and human time.
The funny part is that the workflow worked
It would be easy to tell this story as an example of AI failing.
I don't think that's the interesting part.
Of course the AI failed.
Software work fails all the time.
People package the wrong build.
People forget to copy files.
People validate against their own machine and discover that the package doesn't work anywhere else.
What mattered was that the workflow caught the failure immediately.
The worker didn't get to declare its own return authoritative simply because it said the job was complete.
The artifact crossed a boundary.
At that boundary, it was independently checked.
One hash was enough to reject it.
That is what a good AI workflow should do.
We shouldn't be designing systems around the assumption that AI workers will never make mistakes.
We should be designing them so that mistakes become cheap to detect.
The filename doesn't get a vote
There is something almost comical about a file whose name ends in FIXED while its bytes are exactly the same as the broken file.
But it makes the lesson easy to remember.
A filename is a claim.
A worker report is a claim.
A completion message is a claim.
The artifact is evidence.
When the evidence disagrees with the claim, trust the evidence.
In this case, 64 hexadecimal characters saved us from debugging work that had never actually been delivered.
I'll take that kind of test every time.
Related guide
This incident is one example of a larger rule for AI worker handoffs: qualify what was actually delivered rather than relying on the worker's description of what it intended to deliver.