Research Engineer - Evals
- Location
- San Francisco, CA · On-site
- Job type
- Full-time
- Level
- Mid-level
- Listed
- 23 days ago
Skills & highlights for this role
- evaluation design
- AI model evaluation
- benchmark design
Also useful
- on-device performance analysis
About the role
THINK DIFFERENT. BUILD THE FUTURE.
Tasks You Will Own
- The eval suites that gate every model and agent release - capability, behavior, regressions, and human-rated rubrics that catch what automated evals miss
- The dashboards and tooling that make researcher experiment loops fast and leadership decisions easy
- The bar - what counts as ready to ship, and how we know
Areas Where You Will Assist
- Research, by making sure what we measure is what we want
- Product engineers, by instrumenting real-user behavior on real devices
- Partnerships, by translating "did it get better" into language an OEM partner can hold us to
Skills You'll Be Expected to Teach
- How to measure non-deterministic systems - agent eval, tool use, long-horizon tasks, multilingual behavior
- How to push back on a metric that's being gamed without breaking the team
Skills You'll Be Expected to Learn
- On-device perf trade-offs and how they show up in real-user evals
- What QA-ing AI at OEM scale actually looks like
- The realities of shipping consumer agents to production partners
Timeline of Success
After 30 days - You've audited every eval we run today and produced a sharp doc on what's good, what's noise, and what's missing. You've fixed the most embarrassing gap.
After 60 days - You've stood up a new eval surface - agentic, on-device, or behavioral - and the team is making real decisions on its output. Researchers come to you before launching a run, not after.
After 90 days - Releases now ship against your eval bar, not a vibe-check. You've caught a regression that would have shipped, and cleared a launch the team was nervous about. You're shaping the research roadmap by surfacing where we're flat, where we're climbing, and where we're lying to ourselves.
Compensation
Competitive cash and meaningful equity. Top-tier relocation and immigration support. SF, in person.
How to Apply
Send a link to an eval, benchmark, or measurement system you built - and one paragraph on what decision it changed. Plus your resume or LinkedIn. Every exceptional candidate hears back within 48 hours.
Our Mission
Build everyday AGI. Trustworthy, consumer-grade agents that redefine human-AI collaboration for millions. Software shouldn’t wait for commands; it should partner with you, amplifying what you can do every single day.
WHY AGI, INC.
We’re a stealth team of elite founders and AI researchers, with backgrounds spanning Stanford, OpenAI, and DeepMind. We’re industry leaders in mobile and computer-use agents, bringing these capabilities to consumer scale.
Grounded in years of agent research, our AI is designed with trustworthiness and reliability as core pillars, not afterthoughts.
We are supported by tier-1 investors who funded the first generation of AI giants; now they’re backing us to build the next: everyday AGI. (Watch the demo)
If you see possibility where others see limits, read on.
You decide what "better" means.
Models, agents, and product features all ship behind one question: did this actually get better? Without a strong evals function, the lab ships vibes. With one, every training run, every prompt change, every agent capability moves a number we trust - and the team makes decisions on real signal, not the loudest opinion in the room.
You'll build the eval harness for AGI - across model capability, agentic behavior, on-device performance, and end-user experience. You'll set the bar for what counts as "shipped" and protect it from the gravity of product deadlines.
From AGI, Inc.'s careers page, first listed on Rolev.ai on October 17, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Research Engineer - Evals role at AGI, Inc., it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Similar roles
All Software Developer jobs- Software DeveloperAccenture Federal Services · Washington, DC · $97.5k – $115k/yr
- Software Developer 803Freedom Technology Solutions Group · Annapolis Junction, MD · $150k – $200k/yr
- Software DeveloperPurvis Systems · Newport, RI
- NASA Human Spaceflight Artificial Intelligence (AI) Software Engineer and Data ScientistCACI · Houston, TX
- Senior Manager Software EngineeringNasdaq · Atlanta, GA
- Software Engineering Supervisor - Test SystemsMoog · Torrance, CA
Common questions
Where is the Research Engineer - Evals job at AGI, Inc. located?
This Research Engineer - Evals position at AGI, Inc. is located in San Francisco, CA (On-site). Always check AGI, Inc.'s application page for specific state eligibility or relocation policies.
What qualifications are required for Research Engineer - Evals at AGI, Inc.?
For the Research Engineer - Evals role, AGI, Inc. highlights experience with evaluation design, AI model evaluation, benchmark design. Rolev.ai can score your existing resume against these requirements and show any skill gaps before you apply.
How do I apply for the Research Engineer - Evals position at AGI, Inc.?
This listing links directly to AGI, Inc.'s Ashby application page. You can apply directly on AGI, Inc.'s site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the Ashby application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.