Mocked ≠ real: the bug my unit tests were hiding

The library is content-credentials: a PHP client for C2PA Content Credentials, built to mark AI-generated media the way the EU AI Act, Article 50 requires. It builds a manifest, hands it to a small Node signing service that holds the key, and reads credentials back out of an asset. Framework-agnostic core, optional Laravel layer, PHPStan at max, Deptrac enforcing the boundary. Spec-driven throughout: nothing gets built before a spec is approved, and every acceptance criterion maps to a test.

None of that caught it.

The contract

Reading an asset with no C2PA data has to report absence, not failure. A caller checks whether an image is marked before deciding what to do with it; most images aren't. If that path throws, every consumer has to wrap every read in a try/catch for the ordinary case.

SPEC-003 says so, and ManifestReport implements it. There's a unit test pinning it:

// tests/Unit/Reading/SigningServiceReaderTest.php
$client->addResponse(readStoreResponse([]));   // HTTP 200, empty manifest store

$report = $reader->read(new Asset($bytes, MediaType::Png));

expect($report->hasManifest())->toBeFalse();
expect($report->isAiGenerated())->toBeFalse();

That test is correct. It has been green since the day I wrote it. It tests the PHP client's handling of an empty manifest store, and the PHP client handles an empty manifest store correctly.

It just never asked whether the service actually sends one.

What the property test did differently

I'd been adding property-based tests with Eris — not to replace the example tests, but to quantify over the input space instead of picking points in it. Most of them are stateless: for every media type and every non-blank agent name, whatever the builder marks as AI-generated, the reader reads back as AI-generated.

The interesting one is stateful, and it runs against the real service. C2PA manifests stack: sign an already-signed asset and the new manifest goes on top, the old one becomes an ingredient. That's genuine state, so the test generates sequences of commands rather than single calls:

function pbtProvenanceCommand(): Eris\Generator
{
    return Generators::associative([
        'op'      => Generators::elements(['sign', 'sign', 'read']),
        'agent'   => Generators::elements(['ACME GenAI Image Model', 'agent-2', 'x']),
        'version' => Generators::elements([null, '1.0.0', '3.1.0']),
    ]);
}

A shadow model tracks what should be true — how many signings have happened, therefore whether a manifest should be present — and after every command the test compares the real report against it. The invariant I cared about was that the Article 50 marking survives an arbitrary chain of re-signings.

That's not the invariant that broke.

The sequence that failed

Generated sequences start wherever the generator says. Some of them start with read.

Reading the unsigned fixture returned HTTP 500:

{"error":"Cannot read properties of null (reading 'json')"}

Which the PHP client faithfully turned into a ReadFailedException, exactly as designed. The client was fine. The service was not:

// service/server.js — POST /v1/read
const reader = await Reader.fromAsset({ buffer: fileBuffer, mimeType: mime_type });
const json   = reader.json();

Reader.fromAsset() resolves to null when the asset carries no C2PA data. Not an empty reader — null. So reader.json() throws, the handler 500s, and the contract SPEC-003 spells out is violated by the one component the unit tests never touch.

The fix is three lines:

if (!reader) {
    return res.json({});   // no C2PA data → empty manifest store, HTTP 200
}

{} because that's precisely what SigningServiceReader::parse() already treats as "no active manifest" — the same shape the unit test mocks with readStoreResponse([]). Service and client now agree on the wire format instead of each being separately correct about a different one.

Why the mock hid it

Here's the part worth generalising, and it isn't really about property testing.

A mock encodes your assumption about how the other side behaves. When you assert against it, you're testing that assumption, not the world. My mock returned a well-formed empty store for a manifest-less asset, because that's what I believed c2pa-node would produce. It produces null. The mock was more forgiving than the real thing, so the suite stayed green while the path stayed broken.

That's a quiet failure mode. Nothing warns you. Coverage looks fine — the line is exercised. The test name even reads correctly: reads an asset with no manifest as an empty report. It does. Against a fiction.

Property testing found it not because properties are magic, but because the stateful test ran against the real service and the generator had no idea which orderings were "normal". I wouldn't have written read before sign by hand. Why would I? You sign, then you read. The generator doesn't know that's the story, so it tried the other order, and the other order was the bug.

What I changed, beyond the fix

SPEC-010 now covers it, with the regression guard and the error path alongside — a manifest-less read must be 200, a signed read must still return its marking, and malformed input must still be 400 and not accidentally swallowed by the new empty path. There's an integration test per criterion.

I also added an example test for the specific case. That's standard practice when a property test fires: the property prevents the whole class from coming back, the example documents what actually went wrong and reads well in review. Keep both.

The structural change I'm working on is less obvious. The read contract shouldn't be verified twice, separately, in two places that can drift. It should be one set of properties driven against both implementations of ReaderInterface — the mock in the unit group, the real service in the integration group. If they ever diverge again, something goes red the day the drift appears rather than months later. Same properties, two drivers.

The part about AI-generated code

This library was built spec-driven with heavy assistant involvement, and I'll be straight about what that changes here: not much, and that's the point.

The code an assistant writes is plausible. That's its defining property. reader.json() is exactly what you write if you assume fromAsset returns a reader, and nothing about the line looks wrong. It reviews clean. It passes a mocked test, because the mock was written under the same assumption as the code.

So the leverage isn't in reviewing generated code harder. It's in writing the specification separately from the implementation, and then checking the implementation against it with something that doesn't share its assumptions. Properties written against what the spec says — absence is an empty report, never an error — rather than against what the code happens to do. That's a discipline, not a tool, and it survives whoever or whatever writes the code.

If you want to try this

Eris is the property-based testing library for PHP. It's stateless-only, so the command-sequence pattern above is hand-rolled on top of its sequence generator and a small shadow model. Pest needs uses(\Eris\TestTrait::class) to make $this->forAll() available inside the closures. Integration properties are slow — every command is an HTTP round-trip — so cap the sequence length, cap the case count, and keep them out of the default suite.

Start with a round-trip invariant if you have two halves that must agree. Mine is: whatever the builder marks as AI-generated, the reader must read back as AI-generated. It's one property, it covers the whole input space, and it's the test that will fail loudest when someone changes the marking on one side only.

The manifest-less read is now green for the right reason.

Update — 1 August 2026

Two things in this article have moved since it was written. Rather than quietly rewriting the text, here is what changed.

The structural change still is not built

The article ends on a plan: one set of properties driven against both implementations of ReaderInterface, so a mock and the real service cannot drift apart unnoticed. It is not in the repo, and saying so is more useful than leaving the sentence in the future tense.

The reason is worth stating, because it is the same reason the drift happened in the first place. Nothing forces it. The mocked unit tests are green, the integration tests are green, and the two sets agree today — so there is never a day on which building the shared harness is the most urgent thing. It only pays off on the day they diverge, and that day announces itself by a production bug rather than by a red test.

That is the honest state: a known, unmitigated gap, kept visible here instead of in a private list. It remains the right fix, and I would still recommend it to anyone who verifies one contract in two places.

Eris is no longer what I would reach for

The closing section says Eris is the property-based testing library for PHP, that it is stateless-only, and that the command-sequence pattern has to be hand-rolled on top of it. That was accurate in July. It is no longer what I would advise.

Hand-rolling that pattern twice made the gap obvious: PHP had no model-based (stateful) property testing at all. Erlang, Clojure, Python and TypeScript have had it for years; the technique generates sequences of operations, runs them against a system and a shadow model in lockstep, and — the part that actually matters — reduces a failing twelve-command sequence to the two commands that carry the bug.

So I wrote one, for my own use and not yet public. It owns its generation (PHP 8.2's Random extension, seeded and reproducible across processes), has no runtime dependencies, and does real sequence shrinking — including the values inside a command, so a counterexample reports deposit(52) rather than the deposit(9999) that happened to be drawn.

The limits are as interesting as the capability: no parallel execution or race detection (PHP is share-nothing and request-scoped), no simplification of which command was chosen, and a local minimum rather than a global one — no single further reduction fails, which is a weaker claim than it sounds and worth stating plainly.

None of that helps you today. What does: the hand-rolled suite described in this article still exists and still passes, and the pattern is portable — a command alphabet, a shadow model, a transition, a postcondition. That is fifty lines on top of any value generator. The hard part is not the loop; it is reducing a failing twelve-command sequence to the two that matter, and that is what nobody in PHP has yet.

What has not changed

The bug, the fix, and the reason the mock hid it. A mock encodes your assumption about the other side; asserting against it tests the assumption, not the world. That was the point of the article and it has not aged.