Site Reliability Engineer, Production

About the Role

We're looking for a Site Reliability Engineer (SRE) to drive the reliability of Tinker end-to-end. You'll work alongside the engineers building the platform and research teams to make every layer of the system more robust and resilient.

What You’ll Do

• Define and own end-to-end reliability, from CI/CD flows to production observability and incident response.
• Develop appropriate Service Level Objectives for distributed training systems, balancing job completion reliability and scheduling latency with development velocity.
• Design and implement monitoring and observability across the full training path.
• Drive incident response for Tinker platform issues, ensuring rapid recovery, thorough incident reviews, and systematic improvements that prevent recurrence.
• Harden multi-tenant isolation and resource scheduling so that LoRA-based workload co-scheduling maximizes utilization without compromising reliability or data separation
• Collaborate with security teams to address production vulnerabilities

Minimum Qualifications

• Bachelor's degree or equivalent experience in computer science, engineering, or similar.
• Experience in distributed systems, cloud infrastructure, or site reliability engineering.
• Proficiency writing software to solve reliability problems, including building tooling and automation.
• Experience with production incident response, postmortems, and systematic reliability improvement.
• Strong communication skills and track record of coordination across engineering and research teams.

Preferred Qualifications

• Deep experience operating production cloud services at scale (e.g., public cloud platforms, internal cloud services)
• Background in distributed training frameworks and how infrastructure failures surface in training behavior.
• Track record building checkpoint and recovery systems for long-running distributed jobs.
• Expertise in Kubernetes at scale: deploying, operating, debugging, and tuning clusters handling heterogeneous GPU workloads.

Logistics

• Location: This role is based in San Francisco, California.
• Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $300,000- $350,000 USD.
• Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together.
• Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.