Hiring | AI Engineer - Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote
- Location
- Los Angeles, CA (Remote)
- Listed
- 4 days ago
Role
- AI Engineer - Agent Evaluation Specialist
- Location: Los Angeles, CA (Hybrid)
- Duration: 3+ Months (Possible Extension)
- Primary Focus: Hands-on Agent Evaluation
Specialisation: Agentic AI, AWS Bedrock AgentCore, Claude Agent SDK, Langfuse, AI Observability & Evaluation
We are looking for an AI Engineer with strong, practical experience in evaluating LLM-powered agents, rather than someone focused primarily on building or developing agents.
Key Requirements (Must-Have)
- LLM-as-a-Judge: Hands-on experience designing, implementing, and running LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
- Golden Dataset Development: Experience collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets that serve as benchmarks for agent evaluation.
- Langfuse: Practical experience instrumenting AI agents using Langfuse, including tracing agent execution, capturing evaluation scores, monitoring performance, and troubleshooting failures.
- Agent Evaluation: Strong understanding of end-to-end agent evaluation, including testing agent behavior, tool usage, reasoning workflows, and response quality.
Critical Screening Criteria
- Candidates must demonstrate actual implementation experience with agent evaluation frameworks, not just theoretical knowledge or general GenAI/LLM development.
- Agent evaluation is the core requirement and will be the primary selection criterion.
Additional Information
All your information will be kept confidential according to EEO guidelines.
From TMS LLC's careers page, first listed on Rolev.ai on November 5, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Hiring | AI Engineer - Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote role at TMS LLC, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Common questions
Where is the Hiring | AI Engineer - Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote job at TMS LLC located?
Yes, this Hiring | AI Engineer - Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote opening at TMS LLC is listed as Los Angeles, CA (Remote). Always check TMS LLC's application page for specific state eligibility or relocation policies.
How do I apply for the Hiring | AI Engineer - Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote position at TMS LLC?
This listing links directly to TMS LLC's SmartRecruiters application page. You can apply directly on TMS LLC's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the SmartRecruiters application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.