AI · Evals
Vetto AI
Where frontier AI benchmarks get built.
- Role
- Full-Stack Engineer
- Period
- 2026
- Links
- vetto.ai ↗
Outcomes
The problem
Frontier labs need benchmarks that make top models genuinely struggle on reasoning and agentic tasks: work that demands reading the full context, exploring the environment, and never assuming or hallucinating what is not there. Authoring such tasks is a craft. Each one must be provably solvable, must leak no part of the solution in its instructions, and must be calibrated so that even the strongest models pass only partially. Contributors need serious tooling to do that at scale.
The approach
Built Coding Arena as a full-stack engineer on a Next.js and Django platform with an agentic pipeline architecture. A contributor picks a task category (software engineering, machine learning, computer vision, networking, and more), describes the task and its environment in plain language, and the system generates the task files, the oracle, and test cases, then runs frontier models against it inside a sandboxed environment the agent cannot bypass. Every task is executed across repeated runs per model, with log analysis to confirm the model follows the criteria and the instructions expose nothing. Tasks that models pass fully are sent back for refinement with harder reasoning challenges until they land in the target difficulty band. The VS Code integration gives contributors a sidebar with live logs, per-model pass rates, refinement history, and per-task budget tracking. I also authored eval tasks and the AI benchmark engineering handbook used across the talent pool.
Stack
- Next.js
- Django
- TypeScript
- Agentic Pipelines
- VS Code Integration
- Sandboxed Execution
- LLM Evals
- Benchmark Design
