Senior Hardware Diagnostics Engineer - Infrastructure

About the Role

We are building and operating large-scale infrastructure platforms to support high-performance AI workloads across multiple data centers. Our environment includes GPU-intensive systems, high-throughput networking, and rapidly scaling compute clusters.

We are looking for a Senior Hardware Diagnostics Engineer to own that surface: which versions we run, how we prove a version is safe, and how we push it across a fleet without breaking production. You'll set the bar for what validated hardware means here, build the tooling that enforces it, and be the person we escalate to when a failure doesn't make sense.

Firmware is where a lot of our hardest problems live. A BIOS revision changes how a GPU enumerates. A BMC update fixes one thing and breaks telemetry.

What You’ll Do

• Own the firmware lifecycle across BIOS, BMC, NIC, GPU, and storage controllers: version qualification, approved baselines, staged rollout, and rollback
• Design and build the validation harness that qualifies a firmware release before it reaches production, including regression and performance comparison between versions
• Build automation against Redfish, IPMI, and vendor management APIs so firmware and validation work runs at fleet scale rather than node by node
• Define what passing burn-in means: the test matrix, the thresholds, and the evidence we keep
• Make fleet-wide changes safe and predictable through pre-flight validation, blast radius control, staged waves, and a rollback path that has actually been tested
• Lead root cause analysis on hardware and firmware failures nobody else can reproduce, including PCIe error analysis, sensor and event log forensics, and thermal and power telemetry
• Own technical escalation with OEM, ODM, and silicon vendors on firmware defects, and track fixes to closure
• Prepare validation and bring-up capability for next-generation AMD platforms before the hardware lands
• Mentor the diagnostics engineer on the team, and turn recurring findings into runbooks and tooling
• Participate in incident response and root cause analysis
• Contribute to runbooks, documentation, and operational best practices

Required Qualifications

• 6+ years in infrastructure engineering, hardware validation, systems engineering, or a closely related field
• Deep hands-on experience with BMCs and out-of-band management - Redfish, IPMI, iDRAC, iLO, or equivalent - including scripted interaction, not clicking through a web UI
• Direct firmware experience flashing, qualifying, and troubleshooting BIOS, BMC, and device firmware on enterprise servers, and a clear sense of how badly this can go wrong at scale
• Strong Linux systems skills: boot process, kernel and driver behavior, and hardware diagnostics such as {{dmesg}}, {{lspci}}, {{ipmitool}}, SMART, and vendor CLIs
• Ability to read low-level hardware signals: PCIe topology and error output, SEL and sensor data, thermal and power telemetry
• Python for automation against REST APIs - Redfish, vendor tooling, and internal systems
• A rigorous diagnostic method: you form a hypothesis, design a test that can disprove it, isolate one variable at a time, and document the evidence behind your conclusion
• Experience driving hardware issues with vendors through to resolution
• Clear technical writing for runbooks, RCAs, and vendor escalations

Preferred Qualifications

• AMD GPU and ROCm experience: {{amd-smi}}, {{rocm-smi}}, ROCm Validation Suite
• GPU and HPC validation at scale: stress testing, collective communication testing with RCCL or NCCL, node health frameworks
• Low-level bus and firmware work: I2C, SMBus, BMC firmware behavior, Redfish schema extensions
• Kernel-level debugging, including crash dump analysis
• Ansible for fleet operations, and Git-based workflows with CI/CD
• NetBox or other DCIM and IPAM tooling as a source of truth
• Prior work at an OEM, ODM, or hyperscaler on hardware bring-up or validation

First Six Months

• By 90 days you'll have qualified and rolled out a firmware release using our existing process - and told us what's wrong with it. By six months, a validation step that used to be manual runs automatically, and we can state our approved firmware baseline across the fleet without anyone checking node by node.

What We Offer

• Stock Options
• 100% paid Medical, Dental, and Vision insurance for Employees
• Company Health Savings Account Contributions
• 100% paid Short Term and Long Term Disability Insurance for Employees
• Life and Voluntary Supplemental Insurance Options
• Other Insurance Options, such as Pet & Legal Insurance
• Various Supplementary Health Benefits, such as discounted Virtual Healthcare Appointments and Serious Illness Support
• Flexible Spending Account
• 401(k)
• Employee Assistance Program
• Flexible PTO
• Paid Holidays
• Parental Leave
• Other In-Office Perks