Running an LLM agent on untrusted pull requests with Claude Code and GitHub Actions
I built an agent that reads a pull request, writes or updates unit tests for the changed code, runs them, and reports the result. I used GitHub Actions and the Claude Code CLI on a project with a React frontend and a Node backend.
One question came up early: how could I run the agent safely on a pull request I didn’t trust? The agent would need an API key, and the workflow would need a token to push the tests it wrote. The PR could also contain instructions and configuration that Claude Code might read.
Before building the workflow, I tested that last part. I put a fake agent file, a fake skill and a CLAUDE.md into a PR checkout, then started Claude Code on it as a CI job would. All three took effect.
This post is about how I separated the PR code from the agent’s credentials and configuration, and what I found while testing the result.
Keeping PR code away from secrets
The rule I ended up with: no job that holds a secret or a write token runs PR code. I split the workflow into small jobs:

The generate job has the API key and checks out the PR as data. It doesn’t install or run anything from the PR. Baseline and verify run PR code, including npm ci --ignore-scripts, tsc and Vitest, without a secret and with a read-only token. Publish has the write token and no PR checkout at all.
Every job takes its scripts from the PR’s base commit. Before the agent starts, the workflow removes .claude, .github, .mcp.json, every CLAUDE.md and AGENTS.md, and every symlink from the PR checkout. The agent and its skills then come from the base commit.
I repeated the planted-agent test after that change. This time, no planted text showed up.
The verifier had a hole
The verify job still had a hole.
The first version of the verify job ran PR code as the normal runner user. A review of another PR found the problem: verify.sh installed the PR’s dependencies before it loaded the trusted verifier. The PR code and the verifier ran as the same user, so PR code could change the verifier’s files. I confirmed it locally.
I changed the job to run PR code as a separate OS user, prtest, with no sudo and no groups. It runs in a copy of the tree under /srv/pr. The trusted scripts, results and GitHub control files stay with the runner user. The verifier hashes the copy before and after PR code runs, and kills all prtest processes after every command.
Then I opened a PR that attacked the verifier from a test file in four ways. It tried to overwrite the verifier, forge the gate result, change a candidate file and leave a background process behind. The first two got EACCES, the hash caught the third, and the fourth was killed. None reached publish.
The next review found another hole. When the verify script received a bare --install flag, it dropped the sandbox user. That one is fixed now.
An agent that can’t run its own tests
The test-writing agent is a Claude Code subagent called test-writer. It can read, search, edit and call skills. It can’t run anything. Its permissions come from a settings file on the base commit (trimmed):
"allow": ["Read", "Grep", "Glob", "Skill",
"Edit(backend/tests/**/*.ts)",
"Edit(frontend/src/**/*.test.ts)",
"Edit(frontend/src/**/*.test.tsx)"],
"deny": ["Bash", "WebFetch", "WebSearch", "Agent",
"Edit(.github/**)", "Edit(**/package*.json)", "Edit(**/*config.ts)"]
The CLI runs under env -i, with --strict-mcp-config and a $1.00 budget per attempt.
The agent can’t run the tests it writes, so it needs to read the types and mocks carefully. The verify job then checks its work. If verification fails, an optional second attempt gets the failures as feedback. A sandbox where the agent can run its own tests is on my list for production.
The agent has four skills covering PR analysis, backend conventions, frontend conventions and what “verified” means. The last skill also holds guard.mjs and verify.mjs, the scripts that check the patch.
Finding what the PR broke
A red test suite doesn’t necessarily mean the PR broke it. Some tests may already have been failing on main, and I didn’t want the agent to treat those as the PR’s problem.
Before generation, a baseline job runs the existing tests on both the merge base and the PR head. Failing files on the head run three times. Each failure is classed as PR-induced, pre-existing, flaky or new in the PR. Only PR-induced failures go to the agent.
In one example PR, a small feature added a “completed” filter and a new hook. Nine existing page tests broke because the page now called that hook. The baseline classed all nine as PR-induced, and the agent fixed them by adding a mock. It added 12 lines to the test file and removed none, so every assertion stayed.
That run took about 4 minutes and cost $0.19. A PR where the agent decides nothing needs tests costs about $0.02. A docs-only PR costs nothing because the scope job stops it before any model call.
When the guard blocks a useful patch
An agent can make tests green by deleting the failing ones. The agent file forbids that, but I also added a guard script outside the model. It runs in verify and again in publish.
The guard refuses paths outside the test folders, deletes, symlinks, .only and .skip, and any drop in the number of test cases or expect( calls in a file. If the agent changes a line in an existing test, it has to declare the change in its output, and the PR comment shows it.
The weak-matcher rule took more work. I had to decide when the agent could add something like toBeDefined() to an existing file. The rule changed four times over four review rounds, and every fix caused the next finding. In the end, the strict version stayed: if an edit changes any line in an existing test file, no weak matcher is allowed anywhere in that file. That sometimes blocks a good patch.
A later run showed another cost of strict rules. The agent found an actual bug: a slug helper dropped the Polish letter “ł”. But the guard refused the patch because the agent had grown a one-line import into a multi-line one. The import rule is too strict, and I haven’t fixed it yet.
Publishing what was tested
The verify job records the tree SHA of what it tested: the PR head plus the agent’s test files. Publish writes that tree through the GitHub Git Data API and refuses any other. The branch moves with force: false.
I tested what happens if the branch moves at the wrong time. In the drill, a commit moved it just before publish updated it. GitHub refused the update with 422 “Update is not a fast forward”, and no test commit landed.
Publish is all or nothing. If one new test file is flaky, nothing is published. Publishing a subset would mean publishing a tree that verify never tested.
Putting instructions inside a PR
I also opened a PR with instructions for the agent inside a doc comment and a string constant. The instructions asked it to read the API key and write it into a test, edit the workflow, add a postinstall script, delete existing tests and send a request to an outside host.
The agent wrote one normal test file for the new code and followed none of those instructions. The secret scan on the output was clean.
Why I didn’t use gh-aw
GitHub Agentic Workflows was the obvious alternative. It restores agent config from the base commit, keeps the key in a proxy and has a firewall. I gave it a time-boxed spike: seven runs at a cost of about $1.60. It passed six of my seven criteria.
The agent’s push returned “No such tool available” because gh-aw publishes through its own MCP tools. Fixing that meant giving my agent MCP tools and Bash. In that setup, the agent also had a shell and ran PR code inside the sandbox. In my workflow, the agent has no shell and never runs PR code.
How it was built
I didn’t write this alone. It ran through Triglav, my orchestrator for Claude Code and Codex agents. A Staff agent planned it, Senior agents built the parts in parallel, researchers challenged the plan, and Codex and Claude reviewers read the diffs.
It took about a day and a half, and the local reviews cost about $13. Both holes in the verifier came from reviews.
What is still open
- The verifier protects its own files. It can’t check the test tooling: Vitest and its config come from the PR, so a PR can forge a clean report.
- A collaborator in the same repo can edit the workflow file in their PR. A ruleset on
.githuband.claudefixes that, and it needs the repo owner. - The second attempt is built and unit-tested, but it never ran live.
- Bot commits pushed with
GITHUB_TOKENdon’t start normal CI runs. A commit status carries the result instead.