Senior Manager, Evaluation, Services Special Projects

Summary

We are looking for a hands-on Evaluation leader to build and lead a team of applied scientists and machine learning engineers who measure the quality of a new, multi-year Apple initiative. You will join at the start as a key contributor to the evaluation function, working with other eval leaders to establish how the team knows the system is correct, relevant, and trustworthy before and after it ships. You will own specific components of the evaluation plan, including methods, datasets, and automated grading systems that turn product behavior into clear quality signals for engineering and executive leadership.

About the role

You will lead a team that designs and runs evaluation for a conversational experience that understands what people ask for, finds and ranks the right results, and allows them to complete tasks. Quality in this product spans several layers: interpreting a request correctly, retrieving and ranking the right results, composing a grounded and accurate response, and executing actions correctly through integrations with external partners. You and your team will be key contributors to the overall evaluation plan for this service and will own the evaluation methods for the layers in your area.

Within your components, you will set the technical direction for large language model (LLM) based autograders and rubrics, including rubric design, evaluation set development, validation, and calibration. You will pair these with logging and instrumentation and online experimentation, working alongside the other eval leaders who own different parts of the plan. You will build the roadmap for your components, translate the broader evaluation strategy into concrete methods and deliverables, and make sure your results turn into measurable quality gains and clear inputs to release decisions.

This is a hands-on role. You will review designs and experiments, stay close to the data, and mentor a team of scientists and engineers while you recruit and grow it. You will partner with engineering, applied machine learning, quality assurance, and program management teams, and you will present quality status, risks, and trade-offs for your areas to senior leadership. The product, its architecture, and its quality bar are still taking shape, so you will operate with significant latitude and help shape what "good" and "shippable" mean as the work evolves.

We're looking for someone with a proven track record in

• Building automated grading systems that scale to large, frequently changing datasets and using them to drive improvements in models and product features.
• Defining metrics and evaluation sets that reflect real user behavior and predict real-world quality.
• Turning evaluation findings into action by partnering with engineering and applied machine learning teams to diagnose failures and ship fixes.
• Contributing evaluation results to release processes as explicit quality gates.
• Growing senior individual contributors and technical leads from within a team, and recruiting strong talent in competitive hiring markets

Minimum Qualifications

• 6+ years of experience in machine learning, data science, or machine learning evaluation, including 3+ years of direct people management of scientists or engineers.
• Track record of hiring, developing, and leading teams of applied scientists, data scientists, or machine learning engineers.
• Hands-on experience building and deploying LLM-based autograders and rubrics, including rubric design, evaluation set development, validation, and calibration.
• Experience evaluating conversational AI, search, recommendation, or ranking systems, using both offline metrics and online experimentation.
• Strong command of statistics and experimental design, including A/B testing and interpreting noisy metrics.
• Exceptional written and verbal communication skills, with a track record of influencing engineers and senior leaders

Preferred Qualifications

• MS or PhD in Computer Science, Machine Learning, Statistics, or a related quantitative field, or equivalent experience.
• Experience evaluating agentic or tool-use systems that take real-world actions, including multi-turn behavior, planning, and task completion.
• Experience with relevance and ranking evaluation, including graded relevance judgments and ranking metrics.
• Experience with groundedness and factuality evaluation of generated responses.
• Experience with logging infrastructure and instrumentation for AI product quality measurement.
• Experience generating and validating synthetic evaluation data.
• Experience evaluating systems that integrate with third-party partner APIs.
• Experience with privacy-preserving evaluation practices.