Peraton
Direct employer career site

Ai Ml Operations Assurance Engineer Chantilly Virginia

Location
United States
Posted
33 days ago
Free · No account needed

How well does your resume fit this job?

Upload your resume to see your score for this Peraton role, what matches, what's missing, and your best other matches.

Scored on our server and not saved there. The file stays in this browser so you don't have to upload it again after you sign up.

Required Qualifications

  • Active TS/SCI clearance with CI Polygraph.
  • BS/BA and 5 years of experience, an additional 4 years of experience may be considered in lieu of a degree.
  • Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support.
  • Experience operating and troubleshooting production software applications or cloud services.
  • Experience with monitoring, logging, metrics, incident response, and configuration or change management.
  • Experience evaluating software behavior through repeatable test cases, regression testing, quality measures, or operational performance analysis.
  • Working knowledge of foundation-model or LLM-based applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications.
  • Experience with AWS or comparable cloud platforms.
  • Strong analytical, troubleshooting, documentation, and communication skills.

Desired Qualifications

  • Experience with AWS Bedrock or comparable managed foundation-model services.
  • Familiarity with LLM or agent evaluation frameworks, automated evaluation harnesses, or AI application testing.
  • Experience with OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, or comparable observability technologies.
  • Familiarity with ServiceNow, IT service management, and incident-management processes.
  • Experience with GitLab, CI/CD pipelines, and controlled configuration promotion.
  • Familiarity with Kubernetes and containerized application environments.
  • Experience monitoring cloud utilization, model consumption, and service costs.
  • Knowledge of risk management frameworks, continuous monitoring, or security evidence collection.
  • Experience supporting classified, restricted, or highly regulated environments.

Additional information

We are seeking an AI/ML Operations & Assurance Engineer to monitor, evaluate, and maintain production AI-enabled applications, foundation-model integrations, and agentic workflows. This role focuses on operational reliability, workflow performance, evaluation and regression testing, configuration control, usage and cost monitoring, and the safe introduction of AI-related changes into production environments.

The ideal candidate combines cloud and application operations experience with an understanding of large language model (LLM) applications, agent behavior, and AI-specific operational risks. The engineer will collaborate with AI/ML developers, site reliability engineers, cybersecurity teams, and platform operations personnel to maintain reliable, auditable, and secure AI-enabled services.

ResponsibilitiesProduction Monitoring and Operations

  • Monitor availability, latency, error rates, model and token usage, service costs, tool-call failures, and workflow completion metrics for production AI services.
  • Integrate AI service health and agent-workflow telemetry into observability dashboards, monitoring systems, and alerting processes.
  • Investigate failed, stalled, repetitive, or abnormal AI workflows and coordinate remediation with software, platform, and AI engineering teams.
  • Maintain operational runbooks, escalation procedures, troubleshooting guidance, and recovery processes for AI-enabled applications.
  • Distinguish infrastructure and application failures from issues associated with model behavior, prompts, retrieval sources, or agent orchestration.

AI Evaluation and Operational Assurance

  • Develop and maintain repeatable evaluation scenarios for critical AI workflows, developer assistants, and model-service updates.
  • Assess task completion, output quality, error patterns, inappropriate tool use, and other indicators of operational suitability.
  • Coordinate regression testing before changes to models, prompts, tools, workflow logic, or retrieval sources are deployed to production.
  • Analyze user feedback and operational results to identify where workflows require refinement, additional controls, or restrictions on automated execution.
  • Document evaluation results, limitations, and recommendations to support evidence-based deployment decisions.

Change and Configuration Management

  • Maintain version control and change records for prompts, agent instructions, tool definitions, model selections, evaluation datasets, and workflow configurations.
  • Coordinate model and service updates with AI engineering, cybersecurity, platform operations, and architecture stakeholders.
  • Support rollback to validated configurations when changes negatively affect reliability, performance, or workflow quality.
  • Ensure AI-related changes follow established review, testing, approval, and deployment procedures.
  • Monitor configuration drift and verify that production services remain aligned with approved configurations.

Security, Audit, and Compliance Support

  • Coordinate with cybersecurity and identity teams to verify access controls, tool permissions, logging, and auditability for production AI services.
  • Support investigations involving anomalous AI behavior, unauthorized actions, or potential data-handling concerns.
  • Collect and maintain operational evidence supporting security assessments, continuous monitoring, and service reviews.
  • Verify that production workflows use approved models, endpoints, tools, and data sources in accordance with established policies.
  • Support the implementation and validation of operational safeguards for AI-enabled functions.

Operational Improvement and Reporting

  • Identify repetitive operational tasks and support activities suitable for controlled AI-assisted automation.
  • Measure the impact of deployed workflows on processing time, support workload, service quality, and operational efficiency.
  • Collaborate with service delivery, Tier 2 support, SRE, and platform engineering teams to improve workflows based on operational demand.
  • Report trends in service health, model usage, operating costs, evaluation results, and unresolved issues.
  • Recommend improvements to monitoring, testing, configuration control, and operational procedures.

Location: This position is fully onsite in Chantilly, VA.

Read the original posting

From Peraton's careers page, posted on October 19, 2026, first listed on Rolev.ai on November 22, 2026. Rolev.ai is not the employer.

How Rolev.ai helps with this application

Rolev.ai is a job search and application tool for US job seekers. For this Ai Ml Operations Assurance Engineer Chantilly Virginia role at Peraton, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.

  • See how you fit

    Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.

  • Tailor without inventing

    Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.

  • Draft the full kit

    Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.

  • 1-Click Extension Autofill

    The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.

Common questions

Where is the Ai Ml Operations Assurance Engineer Chantilly Virginia job at Peraton located?

This Ai Ml Operations Assurance Engineer Chantilly Virginia position at Peraton is located in United States. Always check Peraton's application page for specific state eligibility or relocation policies.

How do I apply for the Ai Ml Operations Assurance Engineer Chantilly Virginia position at Peraton?

This listing links directly to Peraton's application page. You can apply directly on Peraton's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.