Breaking: Traditional Testing Methods Fail Against AI-Generated Code
The rise of large language model (LLM)-driven software agents is breaking established testing paradigms, according to Fitz Nowlan, VP of AI and Architecture at SmartBear. These agents introduce non-deterministic behavior into code, making it impossible to rely on traditional deterministic testing approaches.
Speaking on a recent podcast, Nowlan highlighted that the old assumption of knowing exactly what code is doing—and what it contains—no longer holds. “We’re moving away from a world where we can fully understand the internals of a system,” he said. “Testing now requires a fundamental shift in how we think about verification.”
The challenge is most acute with MCP (Model Context Protocol) servers, which act as interfaces between LLMs and external tools. Because LLMs generate responses probabilistically, the same input can produce different code paths each time. This non-determinism breaks unit tests, integration tests, and even static analysis tools built for predictable systems.
Background: The Rise of AI-Generated Code
Software development has long relied on deterministic logic: given an input, a fixed output. But LLM-based agents generate code on the fly, often with no two runs producing identical results. This has profound implications for testing, which assumes repeatability.
Nowlan explained that developers must abandon the traditional “white-box” testing mindset. Instead, they need to focus on behavior at the boundaries of the system—what comes in and what goes out—rather than internal state. That shift elevates the importance of data locality and data construction.
When source code is trivial to generate—simply ask an LLM—the real value moves to the data that feeds into and comes out of these systems. “Data becomes the asset, not the code,” Nowlan said. “How do you construct test data that captures the range of possible behaviors without understanding the internals?”
Non-Determinism: The Root Cause
Non-determinism means that identical inputs can lead to different outputs because the LLM’s responses are sampled from probability distributions. This randomness makes traditional assertions—like “this function returns 42”—invalid.
Testing MCP servers is especially tricky because they orchestrate calls to external APIs and databases, compounding the unpredictability. “You’re testing a system that doesn’t have a fixed specification,” Nowlan noted. “You have to test for ranges of acceptable behavior, not exact matches.”
- Old approach: Unit tests with expected outputs.
- New approach: Property-based tests that verify invariants (e.g., “the response is always a valid JSON”).
Data Locality and Data Construction Rise in Value
When code is cheap to generate, the bottleneck shifts to test data. Developers must create high-quality, diverse datasets that exercise the many possible paths an LLM might take. Data locality—keeping test data close to where it’s used—reduces latency and improves reproducibility.
- Curated datasets become critical for validation.
- Synthetic data generation tools are gaining traction to produce edge-case inputs.
- Observability (e.g., logging all interactions) helps reconstruct failures when testing is insufficient.
Nowlan emphasized that teams should invest in data infrastructure rather than trying to lock down unpredictable code. “Stop trying to make the code deterministic. Instead, build confidence by testing the data and the observable outcomes,” he said.
What This Means for Development Teams
Organizations must rethink their quality assurance processes. Traditional CI/CD pipelines that rely on deterministic tests will need to incorporate probabilistic testing strategies.
Key takeaways for practitioners:
- Adopt property-based testing frameworks like QuickCheck or Hypothesis.
- Use canary releases and shadow deployments to measure real-world behavior.
- Invest in data pipelines that generate and manage test datasets automatically.
The shift is urgent. As LLM agents become embedded in production systems—customer support bots, code review tools, autonomous workflows—the cost of failure increases. “We cannot afford to wait for perfect testing,” Nowlan warned. “We need to develop new heuristics and trust mechanisms now.”
Industry observers expect adoption of behaviour-driven development (BDD) and contract testing to accelerate, as these methods focus on external contracts rather than internal code. The era of “unknown code” is here, and testing must evolve or become irrelevant.