Principal Applied Scientist - CoreAI Foundry Observability
- Pay
- $165,600 – $296,400 per year
- Location
- Redmond | United States, WA
- Job type
- Full-time
- Level
- Director
- Experience
- 8+ years
- Education
- Bachelor's degree
- Listed
- 3 days ago
Skills & highlights for this role
- offline evaluation
- production telemetry
- LLM fine-tuning
- distributed systems
Also useful
- supervised fine-tuning
- policy optimization
- distillation
- transformer models
How well does your resume fit this job?
Upload your resume to see your score for this Microsoft role, what matches, what's missing, and your best other matches.
Scored on our server and not saved there. The file stays in this browser so you don't have to upload it again after you sign up.
Overview
The CoreAI organization at Microsoft builds the end-to-end Artificial Intelligence (AI) stack and is core to Azure AI innovation and differentiation, as well as all of Microsoft’s flagship products, from Foundry, GitHub, to Office, Teams, and Xbox. This role is for a Principal Applied Scientist in the Foundry observability group, focused on scientific innovations for AI agents and agent-based systems.
The work centers on defining how we measure, diagnose, and improve agent quality across offline evaluation, online signals, and production telemetry, and then turning that science into practical capabilities for Foundry customers.
In this role, you will lead the science for agent and multi-agent systems, including offline, online, and continuous evaluation and agent improvement methods tied to real product behavior. You will build measurement frameworks that help explain regressions, detect failure modes, and support higher-confidence product and release decisions.
Microsoft’s mission is to empower every person and every organization on the planet to achieve more, and we’re dedicated to this mission across every aspect of our company. Our culture is centered on embracing a growth mindset and encouraging teams and leaders to bring their best each day. Join us and help shape the future of the world.
Responsibilities
- Develop scientific frameworks for agent lifecycle spanning quality, safety, reliability, cost, and behavioral consistency.
- Define scientifically grounded quality signals and benchmarks for agent systems, including task success, tool-use effectiveness, plan quality, failure modes, coordination quality, and user-perceived outcomes.
- Advance observability for AI systems through new approaches to trace analysis, agent health modeling, behavioral clustering, anomaly detection, and multi-agent coordination analysis.
- Partner with engineering teams to operationalize evaluation and observability methods in production systems, enabling safe iteration through staged rollouts, experimentation, A/B testing, and automated regression detection.
- Contribute to instrumentation and semantic standards for agent observability, helping make agent execution more explainable, diagnosable, and comparable across systems.
- Collaborate deeply with product and platform teams across Foundry, Azure Monitor, and agent runtimes to shape end-to-end experiences for evaluation, benchmarking, monitoring, and investigation.
- Act as a technical leader by setting scientific direction, driving research-informed product decisions, mentoring others, and raising the technical bar across the organization.
Required Qualifications
- Bachelor's Degree in Computer Science, Computational Linguistics, or related field AND 8+ years related experience (e.g., statistics, predictive analytics, research)
- OR equivalent experience.
- OR Master's Degree in Computer Science, Computational Linguistics or related field AND 6+ years related experience (e.g., statistics, predictive analytics, research)
- OR Doctorate in Computer Science, Computational Linguistics, or related field AND 5+ years related experience (e.g., statistics, predictive analytics, research)
Other Requirements
Ability to meet Microsoft, customer and/or government security screening requirements are required for this role. These requirements include but are not limited to the following specialized security screenings:
- Microsoft Cloud Background Check: This position will be required to pass the Microsoft Cloud background check upon hire/transfer and every two years thereafter.
Preferred Qualifications
- Experience and familiarity with large language models (LLMs).
- Experience with and a solid foundation in large distributed systems, algorithms, and software engineering principles.
- PhD in Computer Science, Machine Learning, Data Science, or a related field.
- 5+ years of experience in a research/ML engineering or an applied research scientist position, ideally with a focus on AI safety.
- Hands-on experience with deep learning and transformer-based models
- Excel at problem-solving and analytics, with a proactive approach to challenges
- Thrive in fast-moving environments where priorities shift and definitions evolve.
- Enjoy taking ownership end-to-end and learning whatever is necessary to get results.
- Comfortable working independently while thriving in cross-team collaborations.
- Understand methods for training and fine-tuning LLMs, including distillation, supervised fine-tuning, and policy optimization.
- #coreai
- #foundry
Pay and benefits
Applied Sciences IC6 - The typical base pay range for this role across the U.S. is USD $165,600 - $296,400 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $220,800 - $331,200 per year.
Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here:
This position will be open for a minimum of 5 days, with applications accepted on an ongoing basis until the position is filled.
Equal opportunity and legal notices
Microsoft is an equal opportunity employer.
All qualified applicants will receive consideration for employment without regard to age, ancestry, citizenship, color, family or medical care leave, gender identity or expression, genetic information, immigration status, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran or military status, race, ethnicity, religion, sex (including pregnancy), sexual orientation, or any other characteristic protected by applicable local laws, regulations and ordinances.
If you need assistance with religious accommodations and/or a reasonable accommodation due to a disability during the application process, read more about requesting accommodations.
From Microsoft's careers page, first listed on Rolev.ai on November 16, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Principal Applied Scientist - CoreAI Foundry Observability role at Microsoft, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Similar roles
All Research Scientist jobs- Member of Technical Staff - Open EndednessVmax · San Francisco, CA
- Principal Automation and AI Operations EngineerAST SpaceMobile · Dallas, TX
- Member of the Technical Staff, Computer Vision Model TrainingConquest Tech Solutions, Inc. · Palo Alto, CA
- Sr. Applied Scientist, Amazon Connect TalentAmazon Development Center U.S., Inc. · New York, NY
- Lead Research Engineer, Large-Scale Agentic RL TrainingThomson Reuters · United States
- AI InternCamping World · Chicago, IL
Common questions
What is the salary for the Principal Applied Scientist - CoreAI Foundry Observability role at Microsoft?
The Principal Applied Scientist - CoreAI Foundry Observability posting at Microsoft states compensation of $165,600 – $296,400 per year. It is listed as a full-time position. Final pay may depend on location, experience, and interview evaluation on Microsoft's official career site.
Where is the Principal Applied Scientist - CoreAI Foundry Observability job at Microsoft located?
This Principal Applied Scientist - CoreAI Foundry Observability position at Microsoft is located in Redmond | United States, WA. Always check Microsoft's application page for specific state eligibility or relocation policies.
What qualifications are required for Principal Applied Scientist - CoreAI Foundry Observability at Microsoft?
For the Principal Applied Scientist - CoreAI Foundry Observability role, Microsoft highlights 8+ years of relevant experience; a bachelor's degree; experience with offline evaluation, production telemetry, LLM fine-tuning, distributed systems. Rolev.ai can score your existing resume against these requirements and show any skill gaps before you apply.
How do I apply for the Principal Applied Scientist - CoreAI Foundry Observability position at Microsoft?
This listing links directly to Microsoft's application page. You can apply directly on Microsoft's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.