The AI fixed Save. Can we put it live?
A customer needs an answer. An invoice needs checking. Your team wants to try a new tool. Start with a familiar problem, see what is happening, and choose what you would do. No technical background needed.
- Meet the problem
- See what is happening
- Make a choice
- Find out why
- Try a small change
All decisions are free without an account or API key. Drafts stay on this device only, not in account sync. Follow-up lessons retain their existing sign-in and plan requirements.
These are made-up (synthetic) stories with feedback written in advance, not live AI results or real customer outcomes. They help you practise, but do not certify your skills.
On your team's website, the Save button does not keep a person's display name. An AI coding helper offers a fix and says its checks pass. The same website also holds passwords and payment details.
The problem: Get Save working without accidentally changing who can sign in or see someone else's account.
AI-illustrated teaching scene. Follow the arrows from 1 to 3; they show how to read the story, not actions taken by an AI.
- Name did not save
A person changes their display name, but the old name returns.
- An AI offers a fix
The helper has changed some files and reports passing checks.
- Someone must approve it
Your team has not yet reviewed the changes.
What would you do?
What would you ask for before putting the fix live?
- Have the maintainer review and try the small fix
Why this choice?
Show that the Save problem happens before the change and is gone afterward. The maintainer should inspect the changed files and check that one person still cannot edit another's account. The AI should work on test accounts without live passwords or payment details.
- Publish it because the AI's checks passed
Why this choice?
Those checks are useful, but they may miss the account rule the team cares about. A maintainer still needs to review what changed and approve the release.
- Ask the AI to tidy all account code at the same time
Why this choice?
A cleanup could be useful later, but it makes this fix harder to review. First find out whether the small Save change works without changing account access.
What to take away
Ask for a small change you can check against the original problem. Passing checks help a reviewer; they do not replace the person responsible for the release.
What if something changes?
The maintainer finds that two family members can share a display name. The proposed fix would combine their stored accounts. The helper offers a newer AI model with a higher coding score, but the team has not tried this change on separate family accounts.
What needs to change about the plan?
- Use the newer AI and release after its checks pass
Why this choice?
A higher coding score does not tell you whether these family accounts stay separate. The team needs checks for this website and a reviewed way to recover changed account data.
- Keep reviewing only whether the Save button works
Why this choice?
The job now affects stored accounts, not just a button. A successful name change could still merge two people's data, and reverting the button code would not necessarily separate it again.
- Pause release and involve the account maintainer
Why this choice?
The account maintainer should review how existing data changes, test separate family accounts and check that access stays separate. Rehearse recovery on test data before approving release. Keep the work focused; a newer model does not make this a smaller change.
What to take away
When a small fix turns into a change to people's stored accounts, widen the review to match the risk. A better coding score does not shrink that responsibility.
The idea behind this story
Check what the change does to people, not just whether the button works.
If Save worked but two people's accounts were mixed together, which check would have been missing?
The technical connection: Change review and regression tests
Change review means a responsible person checks the actual edits. Regression tests check that a fix does not break behavior that already worked. Changes to stored data also need a tested recovery plan.
Advanced: Passing tests are evidence, not merge authority
Review an agent-authored repository change and decide what a better coding benchmark should actually change.
Software engineering · 8 minutes
The problem
Synthetic practice scenario: all quantities and evaluation outcomes are illustrative, not reported research. A team asks a coding agent to fix an account-settings defect in a repository that also contains billing and authentication code.
Your initial decision
Which implementation and review path fits this bug?
- The requirement is narrow: changing a display preference must not affect account permissions.
- Repository text and downloaded dependencies are untrusted inputs; production secrets must not enter the execution environment.
- A maintainer owns merge approval and can review a focused diff, but not an unrelated repository-wide refactor.
- Tests generated by the author are useful but cannot alone establish the intended access behavior.
Compare the alternatives
- Review a focused agent-authored patch
The agent reproduces the bug and runs permitted tests while a maintainer checks requirements, access boundaries and the final diff, adding review effort.
- Have a maintainer implement the fix directly
A small human patch avoids agent setup and trust overhead but uses maintainer implementation time.
- Refactor the settings layer before fixing the defect
A reviewed refactor could remove duplicated state handling and make later fixes easier, but expands the diff and regression surface beyond the requested behavior.
Mechanism, trade-offs, and evidence
- State the invariant before implementation: preferences must not alter permissions; derive an independent negative test from it.
- Keep the workspace and credentials narrow because repository instructions can be malicious or simply inappropriate for the task.
- Review the diff as executable behavior, including dependency and configuration changes, rather than relying on the agent summary.
- Save a reproducible test result and a reversible patch; a tool-use transcript alone does not show that the user-visible defect is fixed.
Reconsider after a change
A new model advertises a higher coding score, while investigation reveals that the fix also requires an authentication migration.
Should the team relax review for the new model?
- The advertised benchmark improvement is not a result on this repository and provides no local migration evidence.
- In this illustrative migration, 3 permission states must preserve access correctly, including suspended accounts.
- A rollback must account for stored data changes; reverting code alone may not restore the previous state.
- The maintainer can stage the migration and arrange a security-owner review before release.
Compare the alternatives
- Combine migration with a reviewed settings refactor
The stronger coding model could consolidate state and permission handling together, but combining structural and data changes makes regressions harder to isolate and reverse.
- Keep the patch and migration narrowly reviewed
Split schema and behavior changes, test all permission states and rehearse rollback with the responsible owners, accepting a slower release.
- Have the maintainer implement the migration directly
A maintainer who knows the data invariants can avoid reviewing unfamiliar generated migration logic, but implementation still competes with their limited review and operations time.
Mechanism, trade-offs, and evidence
- Reconsideration does not require changing the answer; identify which assumptions changed and whether they weaken or strengthen the original boundary.
- Use the new model as a candidate implementation tool while holding acceptance tests and ownership constant.
- Test migration and rollback on representative synthetic account states, including partial failure, before exposing real accounts.
- The outcome sought is correct user access with manageable recovery, not the fastest generated patch.
Related technical concepts
Follow-up lessons retain their existing sign-in and plan requirements.
- client coding agents
- agent authored change review
- spec driven development
- agent execution sandboxing
- prompt injection defense
- agent evaluation
- agent skills
Technical sources
- Demystifying evals for AI agents — Anthropic Engineering
Explains tasks, graders, repeated trials and outcome evaluation for agents; use task-specific tests rather than importing reported scores.
- MCP Security Best Practices — Model Context Protocol
Describes consent and authorization boundaries for tool integrations; adopting a protocol does not grant access to customer data or actions.
- Building effective agents — Anthropic Engineering
Distinguishes fixed workflows from model-directed agents and recommends testing simpler solutions first; its tooling examples are historical.
Learning together? Pick a story, compare your choices, and talk about what might change your mind. There are no verified team scores or completion records here.
Self-checks and completion records help you reflect; they are not independently verified credentials. Credential availability, assessment, and eligibility are separate.
AI Agents, Explained — free, no login
Explore the cases
- Can I return these shoes?
- The refund screen froze. Did the money move?
- The AI fixed Save. Can we put it live?
- The shop website will not take orders
- We both said approved supplier. Did we mean the same thing?
- It gets the same ten questions right. Is it ready?
- How do I book an appointment on a noisy bus?
- We can improve one task. Which one is worth our time?
- It is your first day at the garden. Where do you start?
- Can the helper pay this repair bill?
- Can we sign the room booking agreement today?
- The caterers sent their prices. Are we comparing the same thing?
- A customer wants a cake by Saturday. Can we promise it?
- Should our shopping helper reorder the household supplies?
- Can we put the neighbors' feedback into an AI tool?
- A volunteer needs the event plans. Which folders should they see?
- These club files are old. Can we delete them?
- There is a new AI helper. Should the club switch?
- Our online helper is down. How can we help visitors?
- The AI writes faster. Why do handovers take longer?