Principal Software Engineer for Copilot Evals
- Pay
- $142,800 – $274,800 per year
- Location
- Redmond, WA · On-site
- Job type
- Full-time
- Level
- Director
- Listed
- Today
Skills & highlights for this role
- software architecture
- distributed systems
- data pipelines
- evaluation infrastructure
- data contracts and schema design
- observability
Also useful
- AI agents
- machine learning evaluation
- clustering and classification
- quality attribution
- cross-team technical leadership
Overview
CADET (Customer and Analytics Driven Evals Team) is building a customer-grounded quality system for Copilot. Our mission is to rapidly identify the customer scenarios that matter most, represent them faithfully in evaluation and learning assets, run quality gates continuously, and turn every important failure into reusable product and model improvements.
We bring together DSAT and other product signals, deep customer engagements to create representative eval sets. Operating in a fast-paced environment, we connect customer grounded quality issues with quality teams to advance Copilot quality and product innovation.
We are looking for a Principal Software Engineer to architect and build the production system behind CADET's (Customer and Analytics Driven Evals Team) customer-grounded quality loop, partnering with our platform teams.
This is a high impact role that will set technical direction, drive rapid delivery, and align engineering, science, product, and platform teams as we shift from manually assembled signals and one-time evaluations to an agentic, automated system.
You will connect heterogeneous data sources, AI agents, evaluation infrastructure, RLE platforms, model and product telemetry, human review, and quality reporting into a reliable and inspectable end-to-end system. You will set architecture across organizational boundaries, establish engineering standards, align partner teams, and drive the most technically demanding parts of the system.
The system will continuously capture customer evidence, prioritize and triage it, create and run evaluation assets, attribute quality losses, report outcomes, and drive measurable customer impact. This role requires both systems depth and product judgment. The ideal candidate builds for rapid innovation and reuse without sacrificing interoperability, traceability, privacy, scientific rigor, or human control at consequential decision points.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.
Responsibilities
- Own and drive the end-to-end CADET architecture, integration contracts, and technical execution for the customer-signal and evaluation system, spanning capture, enrichment, prioritization, triage, asset generation, execution, diagnosis, reporting, and retention. Resolve architectural ambiguity, accelerate delivery, and create reusable capabilities across partner teams while retaining accountability for CADET-specific workflows, interfaces, and system behavior and reusing shared platform capabilities.
- Create scalable services and pipelines that reproduce customer pain points across scenarios, business segments, and interventions through automated grounding-data generation, intent classification, deduplication, clustering, taxonomy assignment, coverage analysis, loss-pattern detection, and work-item routing.
- Automate recurring evaluation execution and quality gates for top intents and customers, including scheduling, versioning, reproducibility, failure handling, and result publication.
- Establish data contracts and extensible schemas for customer signals, workflows, evaluation cases, rubrics, graders, runs, results, attribution, and regression assets.
- Develop quality-attribution workflows that combine automated diagnosis with human review; route losses to the appropriate product, data, model, tool, orchestration, or evaluation owner; track remediation; add regression protection; and verify improvement in customer outcomes.
- Create observability and benchmarks for pipeline health, agent behavior, data quality, grader health, coverage, cost, latency, operational reliability, and regression detection across CADET workflows and tooling.
- Design safe human-in-the-loop gates for ambiguous data, sensitive customer evidence, taxonomy changes, evaluation publication, and consequential quality decisions.
- Mentor engineers across the team and raise engineering quality through clear interfaces, testing, security, operational discipline, and technical documentation.
Required Qualifications
- Bachelor's Degree in Computer Science or related technical field AND 6+ years technical engineering experience with coding in languages including, but not limited to, C, C++, C#, Java, JavaScript, or Python
- OR equivalent experience.
Preferred Qualifications
- Extensive experience designing and delivering large-scale distributed systems, data platforms, developer platforms, or AI infrastructure in production, with practical exposure to large language model technologies, agent systems, or AI/ML platforms.
- Demonstrated principal-level technical leadership across teams and organizational boundaries.
- Solid understanding of security, privacy, data governance, and responsible engineering for sensitive data.
- Excellent communication and collaboration skills with the ability to influence senior engineering, science, and product leaders.
- Experience building AI/ML platforms, LLM or agent systems, evaluation infrastructure, experimentation systems, or model pipelines.
- Experience with retrieval, tool-using agents, prompt and rubric management, human-in-the-loop systems, or automated diagnosis.
- Experience building data and workflow platforms that combine batch, streaming, unstructured, and human-generated inputs.
- Experience with RLEs, synthetic environments, post-training pipelines, or large-scale evaluation execution.
- Experience integrating customer feedback, product telemetry, and engineering work-management systems.
- Experience operating high-availability services and driving platform adoption across multiple product teams, with knowledge of competing AI models, platforms, and developer ecosystems and the ability to apply relevant external advances to Microsoft’s evaluation and quality systems.
#cadets
Pay and benefits
Software Engineering IC5 - The typical base pay range for this role across the U.S. is USD $142,800 - $274,800 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $188,000 - $304,200 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Equal opportunity and legal notices
Microsoft is an equal opportunity employer.
All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances.
If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
From Microsoft's careers page, first listed on Rolev.ai on November 15, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Principal Software Engineer for Copilot Evals role at Microsoft, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Common questions
What is the salary for the Principal Software Engineer for Copilot Evals role at Microsoft?
The Principal Software Engineer for Copilot Evals posting at Microsoft states compensation of $142,800 – $274,800 per year. It is listed as a full-time position. Final pay may depend on location, experience, and interview evaluation on Microsoft's official career site.
Where is the Principal Software Engineer for Copilot Evals job at Microsoft located?
This Principal Software Engineer for Copilot Evals position at Microsoft is located in Redmond, WA (On-site). Always check Microsoft's application page for specific state eligibility or relocation policies.
What qualifications are required for Principal Software Engineer for Copilot Evals at Microsoft?
For the Principal Software Engineer for Copilot Evals role, Microsoft highlights experience with software architecture, distributed systems, data pipelines, evaluation infrastructure, data contracts and schema design, observability. Rolev.ai can score your existing resume against these requirements and show any skill gaps before you apply.
How do I apply for the Principal Software Engineer for Copilot Evals position at Microsoft?
This listing links directly to Microsoft's application page. You can apply directly on Microsoft's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.