Site Reliability Engineer

About the role

• Job Posting Title
• Site Reliability Engineer
• Agency
• 090 OFFICE OF MANAGEMENT AND ENTERPRISE SERV
• Supervisory Organization
• IS-CS
• Job Posting End Date

Refer to the date listed at the top of this posting, if available. Continuous if date is blank.

Note: Applications will be accepted until 11:59 PM on the day prior to the posting end date above.

• Estimated Appointment End Date (Continuous if Blank)
• Full/Part-Time
• Full time
• Job Type
• Regular

Job Details

• Full-time 40-hour work weeks.
• Supports the Information Services Division.
• Salary is up to $110,000 based on education and experience.
• Position reports onsite in Oklahoma City.
• Sponsorship: At this time, we cannot offer sponsorship now or in the future.

About the role

As a Site Reliability Engineer with OMES you will enjoy

• Generous leave including 15 days of vacation, 15 days of sick leave and 11 paid holidays annually.
• A comprehensive Benefit Package with a generous benefit allowance to offset the cost of insurance premiums for employees and their eligible dependents.

Job Details

• Full-time 40-hour work weeks

Responsibilities

Incident Escalation & Remediation (approx. 40%)

• Serve as the first technical escalation point for P1/P2 incidents affecting infrastructure, cloud, and application environments.
• Lead hands-on troubleshooting across Windows and Linux servers, virtualization, Azure and hybrid cloud, networking (DNS, load balancing, firewalls), storage, databases, and application middleware.
• Act as technical lead within the incident command structure, advising the Incident Commander on impact, options, and risk of remediation actions.
• Engage and coordinate subject-matter experts from network, security, database, application, vendor, and agency teams; drive cross-domain remediation to closure.
• Execute and document emergency changes in accordance with change management and audit requirements.

Observability Partnership (approx. 15%)

• Collaborate with Observability engineers to define service-level indicators (SLIs), dashboards, and actionable alerts across Zabbix, Grafana, Azure Monitor, and Datadog.
• Identify telemetry gaps surfaced during incidents and drive instrumentation improvements.
• Reduce alert noise and false positives; tune thresholds to reflect true user and service impact.

Reliability Engineering & Problem Management (approx. 25%)

• Lead blameless post-incident reviews; produce root cause analyses with corrective and preventive actions tracked to completion.
• Define and maintain service-level objectives (SLOs) and error budgets with service owners and agency stakeholders.
• Perform capacity planning, performance analysis, and resilience testing, including disaster recovery and failover validation.
• Identify single points of failure and architectural risks; recommend and implement remediation.

Automation & Toil Reduction (approx. 15%)

• Develop automated remediation, self-healing routines, and runbooks using PowerShell, Python, Bash, and infrastructure-as-code tooling (Terraform, Ansible, ARM/Bicep).
• Measure and reduce operational toil; target a sustained ratio of engineering work to reactive work.
• Contribute reliability guardrails to CI/CD pipelines in partnership with DevSecOps.

Governance, Security & Compliance (approx. 5%)

• Ensure remediation activities comply with NIST SP 800-53 controls, CJIS Security Policy, IRS Publication 1075, HIPAA, and state security standards as applicable.
• Preserve evidence and audit trails for incidents with security, privacy, or regulatory implications; coordinate with the security operations team and Operational Risk & Controls.
• Support internal and external audits by providing incident, change, and availability documentation.

Physical Demands and Work Environment

• This position works in a comfortable office setting with a computer and telephone for a large percentage of the workday.
• This position requires long periods, up to eight hours per day, of sitting and working at a desk. The noise level in the work environment is usually mild.
• Participation in a rotating on-call schedule, including occasional after-hours, weekend, and holiday response for critical incidents.
• Work supports essential public services; may be designated essential personnel during declared emergencies or continuity-of-operations events.
• Occasional travel to state data centers or agency sites.

Minimum Qualifications

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field; equivalent professional experience may substitute year-for-year.
• 5+ years in infrastructure, systems, or application operations, including 2+ years in an SRE, production engineering, or senior escalation role.
• Demonstrated ability to troubleshoot complex, multi-layer incidents in enterprise or hybrid cloud environments under time pressure.
• Strong working knowledge of Windows Server, Linux, virtualization (VMware/Hyper-V), Microsoft Azure, and core networking (TCP/IP, DNS, DHCP, load balancing, firewalls).
• Hands-on experience with at least two observability platforms (e.g., Zabbix, Grafana, Azure Monitor, Datadog, Splunk).
• Scripting proficiency in PowerShell and/or Python; working knowledge of infrastructure as code.
• Working understanding of ITIL incident, problem, and change practices.
• Ability to pass fingerprint-based background checks and meet CJIS and IRS 1075 personnel security requirements.

Preference will be given to candidates who possess

• Experience in state, local, or federal government IT, or another regulated industry (insurance, financial services, healthcare).
• Experience defining and operating SLOs and error budgets for production services.
• Familiarity with containers and orchestration (Docker, Kubernetes/AKS) and with APM and distributed tracing.
• Experience with ServiceNow ITSM and incident command frameworks.
• Certifications such as Azure Administrator/Solutions Architect, AWS SysOps, CKA, ITIL 4, or Security+.
• Experience working within agile delivery (Kanban, Agile Release Train / PI planning).