In my company-brain implementation, review caught a failure that was easy to miss: an alert could be blocked by a permissions check and still be marked delivered.

The access check did its job. The status told a different story.

That distinction matters to anyone relying on the alert. A manager might believe a team had been notified when nothing had been sent. This was a defect found during local verification, not a reported production incident.

The system needs evidence for its own status

The repair made a permission block return no delivery result. The calling path kept the alert undelivered, recorded the blocked state, and kept it out of the ordinary published view. Related review also caught a manual-approval path that could leave an alert marked approved despite blocked publication.

Approval and delivery are different events. So are proposing an action, attempting it, committing it, and telling the user it happened. Those distinctions have to survive in the records people use to understand the system.

An evaluation that checked only whether the permissions gate fired would have missed the misleading delivery state. The assertion needed to cover the consequence: after a blocked attempt, the alert must still be undelivered.

A successful transaction can still be an unfinished product

The clinic phone-agent work revealed another boundary. The latest test record contains two bounded synthetic calls that each committed exactly one reschedule, avoided an unwanted callback or handoff, and reached the end of the interaction.

That supports a specific claim about those calls. It does not establish that the full voice release is ready.

The same record still notes spoken planning filler and uncertainty about the opening practice name: it was not established whether the name was inaudible or missed by the transcript. Audio continuity and latency acceptance remained open. The release was kept not fully verified.

There are at least three things to inspect here: what changed in the booking system, whether the authorized action happened once, and what the caller experienced. A good transcript cannot establish the database state. A correct database write cannot establish a good conversation.

Write the assertions before judging the demo

For a rescheduling task, I would start with these assertions:

  • The intended appointment moved to the authorized slot.
  • The action happened once; retries did not create duplicate effects.
  • A different patient could not inherit a previous verification.
  • The confirmation matched the committed state.
  • An interruption or withdrawal produced the correct next step for the state the system was already in.

The last item is particularly important. Backing out before a write and changing your mind after a write are different requests. A conversation that treats them identically can mislead someone about what has already happened.

This is a proposed evaluation checklist, not a claim that every case has passed live acceptance. The implementation records contain narrower local checks and bounded call evidence.

Test the grader as well as the agent

A test that only runs the successful path can give false comfort. Keep a reproduced failure when you can, show that the assertion detects it, and then show the repaired behavior under the same relevant conditions. Bind the result to the version actually tested.

Use application records for state changes and delivery receipts. Use listening or user observation for the experience. Use local substitutes to exercise controlled failure paths, while keeping their results separate from actual provider behavior.

Anthropic’s agent-evaluation guidance draws the same useful distinction between the transcript and the final environment state. It also explains why different parts of an evaluation may need different kinds of graders.

Charge the failures to the task

For the business, the useful denominator is completed work. Include retries, human checks, and repairs in the cost of reaching that result. Otherwise an agent that makes cheap attempts and leaves expensive cleanup can look better than it is.

Before accepting “done,” decide what evidence would make that statement true. Then make the software preserve it.

Based on the June 23, 2026 company-brain verification packet and the clinic-agent Round21 release receipt reviewed October 16. Company-brain evidence is local backend/API verification; provider OAuth validation was outside that scope. Clinic calls used synthetic test data, not real patients, and the full voice release remained unverified. Company-brain project · Clinic-agent project.