Member of Technical Staff, Evals and Post-Training

What you’ll do

• Build Profound’s eval system: tasks, rubrics, LLM judges, programmatic verifiers, regression alerts, and self-improvement loops where we look at the traces when something is wrong and come up with the fix
• Define what a great AI Marketer run looks like, and analyze trajectories across multi-step tool use, analytical correctness, and content quality
• Work with marketing experts and data labeling companies to define tasks, metrics, rubrics, and other grading mechanisms
• Close the loop from usage: input use cases, cluster failure modes, and turn these into reliable evals
• Turn eval results and expert feedback into training signal: rewards, preference data, and training environments
• Build tooling for debugging model and agent behavior
• Make evals part of how we decide things: wired into CI, launch reviews, and model selection

Who you are

• 2+ years building evaluation or measurement systems, whether AI evals, experimentation, ranking, or search quality
• Strong Python and data skills, with a track record of shipping production systems
• Deep familiarity with LLM evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and understanding how to integrate them into professional settings
• Sound experimental design, and the instinct to separate signal from noise in non-deterministic systems
• Familiarity with modern post-training methods: SFT, preference optimization, and RL
• Strong opinions about how models and agents should behave, and keeping up with current research
• Comfortable operating in a fast-paced startup environment with rapid iteration cycles

Location

This is an on-site role based in our NYC or SF office, designed for builders who thrive on speed, iteration, and meaningful impact.

Pay and benefits

For this role, the expected base salary range is $230,000 to $325,000. Profound’s total compensation package is designed to be competitive and includes base salary, equity, and a full range of benefits and perks. Final compensation will depend on factors such as your skills, experience, qualifications, and location, and will be determined during the interview process. Our recruiting team will share more details about the full compensation package and benefits as you move through hiring.

#LI-PRO

Note: All official communication from Profound will come from a @tryprofound.com email address. If you're contacted by anyone using a different domain, please disregard and report it as spam.