Principal Observability Platform Engineer

About the Role

As a Principal/Staff Observability Platform Engineer, you'll own the technical direction of Nscale's observability platform: the systems that give us deep visibility into GPU clusters, AI workloads, and the infrastructure running them. You treat observability as a product and a discipline, not a tooling exercise. You'll set the architectural roadmap, raise the engineering bar across teams, and ensure our platform scales ahead of the business, not behind it.

You understand that complexity is a cost. Solutions that require constant babysitting don't scale, and neither does operational burden. The platforms you build should be simple to operate, easy to understand, and self-evidently correct when something goes wrong.

This isn't a "maintain and operate" role. It's a "define, build, and lead" role.

What You'll Do

• Own the technical strategy and architecture for observability across metrics, logs, traces, and alerting at scale.
• Drive platform decisions that have multi-year impact: tooling, data models, ingestion patterns, retention, cardinality management.
• Identify systemic gaps before they become incidents; design platforms that make failure visible and fast to diagnose.
• Partner with SRE, infrastructure, and AI/ML teams to embed observability natively into how Nscale builds and operates.
• Define standards and patterns that other engineers adopt, not by mandate, but because they're clearly better.
• Mentor and technically grow the observability team; raise the ceiling on what the team can build and own.
• Lead incident postmortems and use them to drive durable platform improvements.
• Evaluate and introduce tooling that meaningfully improves signal quality, operational efficiency, or scalability, and retire what doesn't.

About You

• 8+ years in SRE, infrastructure engineering, platform engineering, or observability-focused roles.
• You've operated observability infrastructure at serious scale. You know what breaks at 10x and you design for it.
• You have a strong bias toward simplicity. You've seen over-engineered observability stacks collapse under their own weight and you build accordingly.
• Deep hands-on experience with a significant subset of: Prometheus, Thanos, VictoriaMetrics, Grafana, Loki, Tempo, OpenTelemetry, ClickHouse, Elastic.
• Strong engineering fundamentals, proficient in Python, Go, or similar; comfortable owning complex systems end to end.
• Experience with Kubernetes at scale; familiarity with GPU infrastructure or HPC environments (Slurm) is a strong plus.
• You can architect systems, write the code, review others' work, and explain the tradeoffs clearly, all in the same week.
• Infrastructure-as-Code is default, not optional (Terraform, Ansible, or equivalent).
• You influence without authority. Teams want your opinion because it makes their work better.

Preferred

• Experience with high-volume streaming pipelines for observability data (Kafka, Vector, Fluent Bit, etc.).
• Background in AI/ML infrastructure observability: GPU utilisation, training job visibility, inference latency.
• Prior experience defining observability strategy at an organisation level.

Salary Range $190,000—$300,000 USD

For information on how Nscale handles candidate personal data, please see our Employee & Candidate Privacy Notice: Here.

Nscale does not accept unsolicited candidate submissions from recruitment agencies.