Software development has always involved a certain amount of trust.
We trust programming languages and compilers. We trust operating systems, libraries, frameworks, and the countless components that make modern applications possible.
We also trust other programmers.
Nobody expects the developer of a modern business application to personally inspect every line of code in the operating system, database engine, and supporting libraries.
But artificial intelligence is introducing a new dimension to that familiar problem.
A single developer, working with AI, can now produce an astonishing amount of software in a relatively short period of time.
Entire applications. Complex user interfaces. Supporting libraries. Automated tests. Documentation. Integration code.
The problem is that the developer's ability to read, understand, and verify all that code hasn't increased at the same rate.
That raises a question we need to take seriously.
When AI can produce more software than any individual can realistically review, how do we know whether the software is trustworthy?
This Problem Isn't Entirely New
Large software projects have been dealing with this problem for decades.
No single programmer understands every line of code in a modern operating system. Major commercial applications may contain millions of lines of code distributed across hundreds or thousands of components.
Those organizations learned to divide responsibility among teams, establish architectural boundaries, maintain development standards, and use various forms of automated verification.
They didn't solve the problem by expecting one extraordinary programmer to understand everything.
They created processes for managing complexity.
AI-assisted development is making that same challenge relevant to much smaller teams.
A project that once might have required a dozen programmers can increasingly be approached by a handful of experienced developers coordinating AI agents.
That's a remarkable opportunity.
But it also means that a small team may suddenly find itself responsible for the kind of software complexity that previously belonged to a much larger organization.
And the traditional answer, "I'll review the code before I release it," may no longer be sufficient.
Reading Code Is Not the Same as Understanding a System
Suppose an AI agent produces a new feature containing several thousand lines of code.
You could read every line.
You might identify obvious mistakes, unnecessary complexity, or suspicious logic.
But would you necessarily know whether the feature behaves correctly in every situation?
Would you know whether it interacts properly with the rest of the application?
Would you recognize that the agent has duplicated functionality already available elsewhere?
Would you know whether it introduced a security problem or violated an architectural decision made months earlier?
Reading code can help answer those questions.
It cannot necessarily answer all of them.
And the problem becomes more difficult as the application grows.
A developer may thoroughly understand an individual component while overlooking the consequences of the way that component interacts with another part of the system.
A perfectly reasonable change in one place can produce an unexpected failure somewhere else.
That's why reviewing code and verifying software are not the same activity.
Code review is one source of evidence. It is not a guarantee of correctness.
A Successful Test Doesn't Settle the Question
AI coding agents have become increasingly capable of writing code, executing tests, investigating failures, and correcting problems.
That's enormously useful.
But consider what happens when the same AI system writes both the implementation and the tests used to validate it.
The agent interprets a requirement.
It produces code based on that interpretation.
It writes tests based on the same interpretation.
The tests pass.
Does that prove the feature is correct?
Not necessarily.
It may prove that the implementation behaves consistently with the agent's understanding of the requirement.
But what if that understanding was wrong?
Suppose a customer management application is supposed to prevent unauthorized users from seeing confidential account information.
The AI implements a new customer search feature.
It also creates tests demonstrating that the search returns the expected customers.
Every test passes.
Unfortunately, the tests never check whether a user should have permission to see those customers in the first place.
The software works exactly as the tests expect.
And it is still wrong.
The problem wasn't necessarily poor programming.
It was incomplete verification.
This is why passing tests should be treated as evidence of specific behaviors, not as a universal declaration that the application is correct.
We need to ask what the tests actually demonstrate, what they don't cover, and whether the expectations were established independently of the implementation.
Architecture Becomes a Form of Quality Control
There's another kind of problem that automated tests may not detect.
Imagine an application that uses a shared library for document processing.
The architecture requires every document-related feature to use that library so that formatting, validation, security, and output behavior remain consistent.
Now suppose an AI agent is asked to add a document export feature.
Rather than using the approved library, it writes a completely new export implementation.
The new feature works.
The output looks correct.
The tests pass.
From the user's perspective, everything may appear satisfactory.
But the implementation has violated an important architectural requirement.
The application now has two different implementations of something that was supposed to have one.
Future corrections may be applied to one implementation but not the other. Security improvements may be missed. Behavior may gradually diverge.
Nothing necessarily fails today.
The problem may not become apparent until much later.
This illustrates why architecture cannot be treated as an optional suggestion that an AI agent is expected to remember.
Important constraints need to be explicit.
Where appropriate, they also need to be enforceable.
If a component must come from an approved library, the build process should be capable of detecting an unauthorized replacement or dependency.
If two modules must communicate through a particular interface, the architecture should make violations difficult and detectable.
If certain operations are forbidden, automated checks should help identify them before release.
The goal is not to replace good architectural judgment with a collection of mechanical rules.
It's to turn important architectural decisions into constraints the development process can actually enforce.
Trust Requires Evidence
This leads to what I believe is the central issue.
The objective is not to personally inspect every line of generated code.
The objective is to establish sufficient, credible evidence that the software satisfies its requirements and respects its constraints.
That evidence can come from several places.
Automated tests can demonstrate expected behavior.
Integration tests can establish that components work together.
Security reviews can identify vulnerabilities and unsafe assumptions.
Build rules can enforce architectural boundaries.
Static analysis can detect certain classes of programming errors.
Human review can examine high-risk decisions and determine whether the system actually solves the intended problem.
Independent evaluation can identify mistakes that the implementation team, human or AI, failed to recognize.
None of these techniques is perfect.
And they are not interchangeable.
A successful build doesn't prove the application behaves correctly.
A successful unit test doesn't prove the architecture is appropriate.
A security scan doesn't prove the system is secure.
A beautiful user interface doesn't prove the underlying data is being handled correctly.
What matters is combining the right forms of evidence and understanding what each one actually establishes.
An additional consideration is independence.
If the same agent designs the feature, writes the implementation, creates the tests, and summarizes the results, those activities may all reflect the same mistaken assumption.
Using another reviewer or independently established acceptance criteria can help expose that weakness.
But even an independent AI reviewer is not automatically correct.
It must be given a meaningful question to answer and evidence it can actually examine.
Not Every Line Deserves the Same Attention
There is also a practical reality.
Some mistakes matter much more than others.
A minor formatting inconsistency in a report is not equivalent to an authorization failure that exposes private customer records.
A problem in a cosmetic user-interface component is not equivalent to an error in a financial transaction or database migration.
Trying to apply exactly the same level of human scrutiny to everything can become impractical.
Instead, developers need to make deliberate decisions about risk.
Security-sensitive operations deserve particularly careful review.
Changes that could destroy or corrupt data need stronger safeguards and recovery tests.
Architectural changes that affect multiple components deserve broader evaluation.
Low-risk presentation changes may be adequately covered by automated checks and focused visual inspection.
This isn't permission to ignore the rest of the application.
It's a recognition that responsible engineering requires allocating verification effort where failures have the greatest consequences.
The more software AI can produce, the more important those decisions become.
Five Questions Worth Asking Before Calling It Finished
When evaluating an AI-assisted implementation, I think five questions provide a useful starting point.
- What was the software actually supposed to do? Are the requirements and acceptance criteria clear enough to distinguish a correct result from a plausible-looking one?
- What changed, and where? Do we understand the affected components, dependencies, interfaces, and architectural boundaries?
- What evidence demonstrates that it works? Have the expected behaviors been tested, including important failures and edge cases?
- What could still go wrong? Have we examined security, data integrity, integration behavior, and the consequences of assumptions that may be incorrect?
- Who is responsible for accepting the result? Has someone with appropriate knowledge examined the evidence and made an informed decision about the remaining risks?
These questions don't eliminate uncertainty.
No practical software development process does.
But they move the discussion away from a simple declaration that the AI completed its assignment.
They require us to examine what completion actually means.
Responsibility Doesn't Disappear When AI Writes the Code
There is a tempting argument that if AI produced the software, AI should be responsible for its mistakes.
But an AI system is not the organization deploying the application.
It's not the developer promising customers that their information will be handled correctly.
It's not the person deciding whether a system is ready to enter production.
Those responsibilities still belong to people and organizations.
AI can assist with implementation, testing, documentation, and review.
It can help us gather the evidence needed to make good decisions.
But it cannot relieve us of the obligation to make those decisions responsibly.
And perhaps that's the most important shift in how we should think about AI-assisted development.
Our responsibility is no longer adequately described by how much code we personally write or read.
It increasingly involves establishing the conditions under which software can be developed, evaluated, and trusted.
The Question Worth Asking
I don't believe the solution is to slow AI development down until every developer can personally inspect every line of code it produces.
That would discard much of what makes these tools valuable.
I also don't believe we should accept a system simply because an AI agent reports that the project builds successfully and all its tests passed.
The answer lies somewhere more demanding.
We need better specifications, stronger architectural boundaries, more meaningful tests, appropriate independent verification, and clearer evidence about what has actually been accomplished.
We need development processes that scale with the amount of software being produced.
And we need to recognize that confidence in a system is something we establish through evidence, not something we inherit from the confidence of the tool that created it.
The real question is no longer whether one programmer can read every line of code.
It's whether we have built a development process that gives us good reasons to trust the software even when no individual can understand every part of it.
That is a harder question.
But it's the one we need to be asking.
Related Field Note: The Project That Outgrew Its Reviewer