Model Behavior Engineer
- Pay
- $160,000 – $190,000 per year
- Location
- New York City, NY · On-site
- Job type
- Full-time
- Level
- Senior
- Experience
- 3+ years
- Listed
- Today
Skills & highlights for this role
- LLM evaluation
- LLM-as-judge calibration
- prompt engineering
- rubric design
Also useful
- Braintrust
- human labeling programs
- conversational AI
- voice AI
Stated benefits & perks
- Medical
- Dental
- Vision
- 401k
- Pto
How well does your resume fit this job?
Upload your resume to see your score for this Overtone role, what matches, what's missing, and your best other matches.
Scored on our server and not saved there. The file stays in this browser so you don't have to upload it again after you sign up.
Your Role
Your job is to work with our relationship scientist and eval engineer to build and execute an eval strategy that keeps our LLM content high quality at scale.
- Our relationship scientist defines what good looks like across our content surfaces and drafts golden examples. They also label for LLM judges that need subject matter expertise or calls on tone and voice.
- Our eval engineer builds the infrastructure to power our content monitoring and iteration: human labeling tools, automated judge calibration tools, Braintrust experiments, and production monitoring dashboards.
- You are the hands-on operator of this system: the eval engineer builds the tools, the relationship scientist defines goldens, and you use both every day to find and fix problems in our content.
The team is early, so you'll have real influence over how this workflow gets built.
What You’ll Do
- Prompt iteration and engineering. Fix straightforward errors yourself: when our judges flag a problem in production content, diagnose the failure, change the prompt, and confirm the fix with evals before it ships. When a fix depends on subject matter expertise, partner with our relationship scientist. You’ll draft prompts for our content surfaces and our judge rubrics.
- Eval + content engineering strategy. Design our infrastructure for evals and content iteration, including labeling, judge calibration and production quality monitoring and keep evolving it as the product grows.
- Define what we measure. Work with our relationship scientist to decide which content features to measure, from proactive baseline quality checks to reactive checks driven by errors we find. Set up automated error analysis so new failure patterns surface on their own, not only when someone spots them.
- Labeling strategy and tools. Own how we label and calibrate our judges, including the tools and workflows that scale our relationship scientist's judgment as efficiently as possible. Make sure they label only where their expertise is needed. Everywhere else, label production outputs and match assessments yourself or through outsourced labelers, so we can check that our judges agree with human judgment.
- Synthetic testing. Design synthetic tests that tell us whether a new prompt is working before it reaches human testing, and review the results.
- Production review and prioritization. Read real production outputs across our core surfaces and spot quality issues. Prioritize what to fix first across everything our judges, automated error analysis and manual reviews surface.
Who You Are
We're open to a very product-minded engineer, or a highly technical product person:
- 3+ years in product management or engineering, including shipping LLM-powered features to real users
- Hands-on experience with evals: writing rubrics, building or running LLM-as-judge setups, and aligning judges against human labels
- Hands-on prompt engineering experience: you can diagnose why an output went wrong, rewrite the prompt, and show with evals that the fix worked
- Comfortable writing code when the work calls for it, like scripts for error analysis or changes to our eval pipeline
- A detail-oriented, editorial eye for tone, structure and voice in AI-written content
- Eagerness to dive in and read lots of AI outputs by hand
Great to have
- Experience with Braintrust or similar eval tooling
- Experience running human labeling programs
- Experience with conversational or voice AI products
- Interest in relationships, psychology or matchmaking
Compensation & Benefits
- Salary of $160k to $190k.
- The offer within that range depends on experience and skills.
- This position will receive meaningful stock options with considerable upside potential.
We provide standard benefits such as
- Medical, dental and vision insurance
- 401(k)
- Commuter benefits, daily lunch and other perks
- Paid time off, holidays and sick leave
Our Core Qualities
- We look for these qualities in everyone we hire at Overtone.
- Curiosity: You seek to understand before you decide. You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges.
- Craft: You care deeply about quality and coherence. You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made.
- Collaboration: You engage with others honestly and generatively. You challenge ideas with respect, listen deeply, and help synthesize perspectives into clearer, better outcomes.
- Care: You take responsibility for people and outcomes. You act with integrity, follow through on commitments, and protect trust-with users, teammates, and the work itself.
About Overtone
Overtone is a modern matchmaking company, from the creator of Hinge. We’re building a voice- and audio-forward service, enabled by AI, grounded in relationship science. There’s no profiles or swiping. We get to know people in their own voice, then make thoughtful introductions, one at a time, to people we think are worth meeting. The service is in beta-testing and not yet open to the public. We’re building a small team of people passionate about relationships and human connection, to work together in person in our NYC office (5 days a week in Flatiron).
Equal opportunity and legal notices
Overtone is an equal opportunity employer. We consider all qualified applicants without regard to race, color, religion, creed, sex, pregnancy, sexual orientation, gender identity or expression, national origin, ancestry, age, disability, genetic information, marital or partnership status, caregiver status, veteran or military status, or any other status protected by law.
We will consider qualified applicants with arrest and conviction records in a manner consistent with the New York City Fair Chance Act and other applicable law. If you need an accommodation at any point in the application or interview process, email and we'll work with you.
From Overtone's careers page, first listed on Rolev.ai on November 20, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Model Behavior Engineer role at Overtone, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Common questions
What is the salary for the Model Behavior Engineer role at Overtone?
The Model Behavior Engineer posting at Overtone states compensation of $160,000 – $190,000 per year. It is listed as a full-time position. Final pay may depend on location, experience, and interview evaluation on Overtone's official career site.
Where is the Model Behavior Engineer job at Overtone located?
This Model Behavior Engineer position at Overtone is located in New York City, NY (On-site). Always check Overtone's application page for specific state eligibility or relocation policies.
What qualifications are required for Model Behavior Engineer at Overtone?
For the Model Behavior Engineer role, Overtone highlights 3+ years of relevant experience; experience with LLM evaluation, LLM-as-judge calibration, prompt engineering, rubric design. Rolev.ai can score your existing resume against these requirements and show any skill gaps before you apply.
How do I apply for the Model Behavior Engineer position at Overtone?
This listing links directly to Overtone's Ashby application page. You can apply directly on Overtone's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the Ashby application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.