Agent evaluations for Next.js coding tasks, powered by @vercel/agent-eval.

npm install
cp .env.local .env   # requires VERCEL_OIDC_TOKEN and AI_GATEWAY_API_KEY

Runs agent evaluations with memoization. Only runs (model, eval) pairs that haven't been completed yet.

npm run eval              # Run only missing pairs
npm run eval:dry          # Preview what would run
npm run eval -- --force   # Re-run everything
npm run eval:smoke        # Run 1 eval per experiment (sanity check)

The runner automatically detects:

• New model added → runs all evals for that model
• New eval added → runs that eval for all models
• Already completed → skips

Exports clean results to agent-results.json. Non-model failures (infra/timeout) are automatically deleted during eval runs, so only valid model results are exported.

Each eval is a self-contained Next.js project in evals/:

evals/agent-031-proxy-middleware/
├── PROMPT.md        # task given to the agent
├── EVAL.ts          # vitest assertions (withheld from the agent)
├── package.json     # Next.js project manifest
├── tsconfig.json
├── next.config.ts
└── app/
    ├── layout.tsx
    └── page.tsx

1. Create a directory under evals/ (e.g., evals/agent-040-my-eval/)
2. Add PROMPT.md with the task description
3. Add EVAL.ts with vitest assertions
4. Add package.json with "type": "module" and "build": "next build"
5. Add the Next.js source files the agent starts with
6. Run npm run eval — it will automatically run the new eval for all models

1. Create a config in experiments/ (e.g., experiments/gpt-5.ts)
2. Add the display name to MODEL_NAMES in scripts/export-results.ts
3. Run npm run eval — it will automatically run all evals for the new model

After running evals:

1. Export results: npm run export-results
2. Copy to front repo:

   cp agent-results.json <path-to-front>/apps/next-site/app/\(next-site\)/evals/agent-results.json

3. Commit and deploy the front repo

See LICENSE.