Build a QA Agent That Tests Your App

A QA agent is a model that follows a written test plan against a running app, drives a browser or API, and reports bugs with steps and evidence, without editing your code. In Claude Code you build one from a plan file, a read-only subagent with the Playwright MCP server, a bug schema and a headless run. Facts checked 31 October 2026.

What you will have at the end

• A qa/test-plan.md in a format an agent can follow and a person can review
• A QA subagent with a browser and no edit tools
• A bug report schema so findings are comparable between runs
• A headless run that works on a schedule or in CI
• Rules that stop it inventing bugs or hiding real ones

The browser setup is covered in depth in our Claude Code and Playwright stack. This guide is about what to build on top of it. If the agent idea is new, start with how to build an AI agent.

What QA agents are good at

An agent explores, reads error text, and notices that a page looks wrong. A script does none of that. A script, though, is exact and cheap to run a thousand times. Use each for what it does well.

The last row is the one to remember. The agent finds and describes, and a person decides. The sensible loop is to let the agent explore, then turn each real bug it finds into a plain Playwright test so the next run is cheap and exact.

Step 1: Write the test plan as a file

A plan the agent can follow beats a prompt typed each time. Keep one in the repo, such as qa/test-plan.md. Each case has a goal, steps, and an observable pass condition.

# Test plan

Base URL: http://localhost:3000
Seed users: qa-member@example.test (member), qa-admin@example.test (admin). Password in .env.qa.

## T1 Sign in
Steps: open /login, enter the member credentials, submit.
Pass: the page shows the member's name in the header and the URL is /dashboard.

## T2 Create a project
Steps: from /dashboard click New project, enter "QA project", save.
Pass: the project appears in the list and its page loads with the same name.

## T3 Empty and long input
Steps: try to save a project with an empty name, then with a 300-character name.
Pass: the empty name shows an error message; the long name is accepted or rejected with a clear message and never a blank page.

## T4 Permission check
Steps: signed in as member, open /admin.
Pass: access is refused with a visible message and no admin data is shown.

The word "pass" matters in every case. Without an observable pass condition the agent decides for itself, and its decisions drift between runs.

Write cases for flows a customer would care about. Ten good ones are worth more than fifty vague ones. Keep credentials for seed users on a test database only. Never put real customer logins in the plan.

Step 2: Add a browser

The standard way to give Claude Code a browser is the Playwright MCP server from Microsoft.

claude mcp add --transport stdio playwright -- npx @playwright/mcp@latest --headless --isolated

Per the project's README as summarized on our stack page, --headless hides the window (the default is headed), and --isolated keeps the browser profile in memory, so no cookies or logins persist between runs. Playwright MCP needs Node.js 18 or newer. Check the server with claude mcp list or /mcp. Our MCP install guide covers scopes and troubleshooting.

Scope matters here. Add the server in user or local scope for experiments. Once the team uses it, add it with --scope project so .mcp.json carries it. Or give it to the QA subagent alone, as in the next step, so its many browser tools stay out of your normal sessions.

Step 3: Create the QA subagent

Save this as .claude/agents/qa-tester.md. It has no Edit or Write tool, so it cannot change the code it is testing.

---
name: qa-tester
description: Runs the test plan in qa/test-plan.md against the running app and reports bugs. Use after UI changes or before a release.
tools: Read, Grep, Glob
mcpServers:
  - playwright:
      type: stdio
      command: npx
      args: ["@playwright/mcp@latest", "--headless", "--isolated"]
model: inherit
---

You are a QA tester. You did not write this app.

1. Read qa/test-plan.md.
2. For each case, follow the steps exactly in the browser.
3. Compare what you see with the case's Pass line.
4. For a failure, record: case id, steps you took, expected result, actual result,
   and the exact visible text or error you saw.
5. Do not guess at causes in the code. Report what a user sees.
6. If the app is not reachable, stop and say so. Do not report bugs you could not observe.
7. A case passes only when its Pass line is true on screen.

Never edit files. Report only what you observed in this run.

Two lines do most of the work. Line 5 stops the agent from blaming a file it never opened, and line 6 stops it from fabricating results when the server is down. Inline servers in mcpServers connect when the subagent starts and are subject to the trust rule for the agent file's folder.

Claude Code subagents run in the background by default, and a background subagent has a narrower tool set. If the browser tools do not appear, ask Claude to run it in the foreground, or set the subagent up as a forked skill with background: false.

Step 4: Fix the report format

Free-form reports are hard to compare. Ask for one shape every time. In a conversation, put the format in the subagent prompt. For headless runs, use a JSON schema.

claude --bare -p "Use the qa-tester subagent to run qa/test-plan.md against http://localhost:3000 and report every case." \
  --agents ./qa-agents.json \
  --permission-mode dontAsk \
  --output-format json \
  --json-schema '{
    "type":"object",
    "properties":{
      "cases":{"type":"array","items":{
        "type":"object",
        "properties":{
          "id":{"type":"string"},
          "result":{"type":"string","enum":["pass","fail","blocked"]},
          "steps":{"type":"string"},
          "expected":{"type":"string"},
          "actual":{"type":"string"}
        },
        "required":["id","result"]
      }}
    },
    "required":["cases"]
  }' > qa-report.json

The --agents flag takes agent definitions as JSON, and in non-interactive mode it also accepts a path to a JSON file. That is how a --bare run, which skips auto-discovery, loads a custom agent. Each key is an agent name, with description, prompt, tools and model fields, as in this qa-agents.json:

{
  "qa-tester": {
    "description": "Runs qa/test-plan.md against the running app and reports bugs.",
    "prompt": "You are a QA tester. Follow qa/test-plan.md exactly. Report only what you observe. Never edit files.",
    "tools": ["Read", "Grep", "Glob"]
  }
}

The browser tools still need the Playwright server, so pass it with --mcp-config in a bare run. blocked is its own result, so "could not run" never looks like "passed" in the report.

As with any headless run, --bare does not use your subscription login, so set ANTHROPIC_API_KEY. The JSON response carries total_cost_usd, so you can log what each QA pass costs. A response without structured_output means the run did not produce a valid report, so check for the field before you act on it.

Step 5: Run it against something safe

Point the agent at a local or staging copy with seeded data, never at production. A QA agent clicks things. On a live system, that includes the buttons that send email, take payment or delete records.

Start the app and the database in a known state before each run: reseed, then run. That makes failures reproducible, since a failure that depends on leftover data from the last run is hard to chase. Playwright's isolated profile keeps browser state clean, and a reset script keeps the data state clean.

Step 6: Turn real bugs into tests

When the agent reports a real failure, a person confirms it by hand, fixes it, and then writes a plain test so it cannot come back. The agent can draft that test. Ask it, in a normal session, to write a Playwright test for the confirmed steps, then run it, watch it fail on the old code and pass on the fixed code.

This habit is the real payoff. Each confirmed bug becomes a cheap exact check, so the agent keeps spending its time on exploration rather than rechecking old ground. After a few weeks, the plan file holds the unscripted exploratory cases and the test suite holds the regressions.

A sample report, so you know what good looks like

This is an example of one failed case in the shape the schema asks for. The content is illustrative.

A person can read that in ten seconds, reproduce it in thirty, and decide what to do. Compare it with a vague report such as "form validation seems off". The first is a bug report and the second is a mood.

Choosing what to put in the plan first

When you have a blank file, rank flows by what hurts if they break.

1. Money and sign-up. Anything that takes payment, creates an account or sends a confirmation.
2. The first five minutes. What a new user sees after sign-up. If this breaks, you lose people quietly.
3. Permissions. Pages that one role must not reach. These bugs are rare and costly.
4. Edge input. Empty, very long and unusual characters in every form that saves data.
5. Recent changes. Whatever shipped in the last release.

Revisit the plan after every release. A plan that never changes describes last year's app.

When the agent gets stuck

Browser agents fail in recognizable ways. Plan for them in the prompt and the plan file.

• Waiting for a slow page. Tell it in the plan how long a page may take and what to do after: report blocked with the last visible text.
• Cookie banners and pop-ups. Add a step zero to the plan that dismisses known banners, or seed the test environment so they do not appear.
• Captchas and email links. The agent cannot pass a captcha and should not try. Turn captchas off in the test environment, and use a test mailbox or a log file for emailed links.
• Moving targets. If a case clicks a button whose label changes, name the element by what the user sees and add a fallback in the plan text.
• Endless exploration. Set maxTurns in the subagent frontmatter. When the limit is reached, the output is marked partial, and Claude can resume it.

A capped run that says "partial" is better than an uncapped run that burns tokens on a loop. Log the partial runs, because a case that is often partial usually needs splitting.

Keep it honest

A QA agent has two failure modes: it reports bugs that are not there, and it passes things that are broken. Guard against both.

• Require evidence. Every failure needs the visible text, URL and steps. A screenshot path is better still.
• Allow `blocked`. If the app was down or a step could not be done, the agent must say so.
• Re-run failures once. A failure that does not repeat is probably a timing issue. Report it as flaky, not as a bug.
• Spot-check passes. Each week, a person runs two passing cases by hand. If the agent said pass on something broken, the pass lines are too loose.
• Never let it close bugs. It can report. People triage and decide.
• Keep the plan in version control, so changes to what counts as a pass go through review.

The plan's Pass lines are the real test. If you cannot say how a person would check one, the agent cannot either.

Add a Stop check for local QA

Claude Code hooks can enforce part of this. A Stop hook can refuse to let a session end until a script confirms a report file exists and parses.

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          { "type": "command", "command": "jq -e '.cases | length > 0' qa-report.json > /dev/null 2>&1 || { echo 'qa-report.json is missing or empty' >&2; exit 2; }" }
        ]
      }
    ]
  }
}

Read stop_hook_active in real use, so the hook cannot loop, and remember Claude Code overrides a block after eight in a row. See the hooks guide.

Scale and schedule

For a large plan, split it. Run one subagent per area, such as sign-in, billing and admin, since subagents run in parallel and each has its own context. Mind the default cap of 20 running subagents in a session. Keep each plan file short enough that one agent can finish it without its context filling up.

For a schedule, run the headless command from your CI scheduler or a cron job against staging each night. Post the report, not the transcript. Track three numbers over time: cases run, cases failed and cases blocked. A rising blocked count usually means the environment is flaky, not the app.

Costs and limits

Each run pays for model tokens, and browser work is verbose: every page read becomes text the model must process. Keep plans focused, avoid asking it to explore the whole app, and log total_cost_usd per run. A heavier model is rarely needed for following a written plan, so you can set the subagent's model lower and keep inherit only for the cases that need judgment. See our Claude models page for the tiers.

Our take

The valuable artifact here is the test plan, not the agent. A plan with clear pass lines makes any agent useful and gives your team a document that explains what the product is supposed to do. We would write ten cases for the flows that earn money or lose customers when broken, run them weekly, and convert each real bug into an ordinary test.

Be wary of agents that promise to test everything. Exploration is a good use of a model, and a verdict on shippability is not. Keep a person at the decision.

Checklist

• qa/test-plan.md has an observable Pass line per case
• The subagent has no Edit or Write tool
• The browser runs headless and isolated
• Tests run on local or staging with seeded data
• Reports have pass, fail and blocked
• Failures are rerun once before they count
• Each confirmed bug becomes a regular test
• A person spot-checks passes weekly

Related pages

• Claude Code and Playwright for browser QA
• Playwright MCP
• Build a code review agent
• How to write a subagent
• How to build an AI agent