Staff Software Engineer, AI Inference Gateway
- Pay
- $180,000 – $220,000 per year
- Location
- Remote (US)
- Job type
- Full-time
- Level
- Senior
- Experience
- 10+ years
- Listed
- Today
Skills & highlights for this role
- Rust
- LLM inference servers
- distributed systems fundamentals
- OpenTelemetry
Also useful
- Pingora
- Tokio
- CUDA
- ROCm
- GGUF
- AWQ
- GPTQ
- FP8
Stated benefits & perks
- Health
- Dental
- Vision
- 401k
- Unlimited pto
How well does your resume fit this job?
Upload your resume to see your score for this Salad role, what matches, what's missing, and your best other matches.
Scored on our server and not saved there. The file stays in this browser so you don't have to upload it again after you sign up.
About the role
Salad Technologies * Remote * Full-time
The role
SaladCloud runs inference on thousands of geodistributed workstation and consumer GPUs - NVIDIA and AMD - that nobody else can use. Sitting in front of that fleet is a unique AI Gateway written in Rust, built on Pingora and WireGuard, that serves real-time and batch inference and embeddings to customers as a single, reliable API. You'll own it.
This is an infrastructure role with real autonomy. You'll design and ship the systems that decide how requests are fulfilled, when to add or drop capacity, and how to bill for every token - and you'll be the person who knows whether a new model, quantization, or GPU is actually worth running.
What you'll do
- Own the AI Gateway system end to end: request routing, streaming, batch/async job handling, and horizontal scaling across multiple gateway servers
- Build and tune the fleet-efficiency algorithms - autoscaling nodes, scoring performance, and evicting underperformers so the network converges on peak throughput per dollar
- Operate and scale the underlying inference nodes running vLLM, llama.cpp, and similar servers
- Design and maintain OpenTelemetry-based observability across the gateway and the fleet; pay-per-token and subscription billing already rides on this telemetry, so you'll keep that integration accurate
- Spend roughly 25% of your time benchmarking new models, quantizations, and GPU hardware, and work with the product and marketing teams to make the call on what goes into production
- Collaborate on cross-system design and integrations with the rest of Salad engineering; explain trade-offs clearly to both technical and non-technical teammates
The team
You'll join a small team - five engineers plus the CTO, who still writes code, and two deeply technical product managers who ship their own fixes rather than queue them for you. You'll report directly to the CTO. Everyone on the engineering team has 10+ years of experience and has built systems that already run at massive scale - with minimal incidents in a typical year, because we ship things that are production-worthy the first time.
- You'll be the first hire with inference-gateway expertise and you'll be asked to own the whole surface.
- You won't do it alone: your teammates are eager to learn the domain, will pair on design and integrations, and share after-hours support.
- The bar is high; so is the support.
We've listed a lot below. If you're strong in two of the first three - Rust, LLM inference servers, distributed systems - apply, and expect to learn the rest fast with people who'll help.
What we're looking for
- Strong production Rust experience, ideally on high-throughput networked services
- Hands-on experience running LLM inference servers (vLLM, llama.cpp, TGI, TensorRT-LLM, or similar) - you know what a KV cache is and why it fills up
- Distributed systems fundamentals: load balancing, backpressure, failure handling, unreliable nodes
- Comfort operating what you build: metrics, tracing, on-call instincts
- Willing to be on-call for a system you helped make quiet - we treat after-hours pages as a design bug to fix, not a lifestyle
- Clear written and verbal communication; you can run a design without hand-holding and bring others along
Nice to have
- Pingora, Tokio, or proxy/gateway internals
- Experience with heterogeneous or consumer-grade GPU fleets, CUDA, or ROCm
- Quantization formats (GGUF, AWQ, GPTQ, FP8) and their performance characteristics
Benefits
- Unlimited PTO
- 75% of health insurance premiums covered for you and your dependents
- Dental and vision coverage
- 401(k) plan
- Stock options
- Company-provided computer
- $500 WFH budget
- Fully remote, with flexible hours
Compensation $180,000-$220,000/year
Why Salad
You'll work on a hard, unusual problem - coaxing datacenter-grade reliability out of hardware in people's homes - with the freedom to make major decisions and see them hit production quickly.
We're scaling rapidly: demand for inference on SaladCloud currently exceeds the capacity we can bring online, and this role exists to keep up with it. There is no shortage of work and no prospect of it ramping down.
The platform handles workload security and isolation, so you can focus on performance. We don't log or train on our customers' prompts or data.
About the company
Our mission is to democratize cloud computing by utilizing latent consumer resources to power a sustainable, affordable, environmentally friendly cloud for everyone.
salad.com/about
From Salad's careers page, first listed on Rolev.ai on November 18, 2026. Rolev.ai is not the employer.
How Rolev.ai helps with this application
Rolev.ai is a job search and application tool for US job seekers. For this Staff Software Engineer, AI Inference Gateway role at Salad, it shows how your resume fits, tailors your resume to the posting, and prepares the application for you to review.
See how you fit
Openings from employer career sites are ranked against your whole resume—and searchable inside Claude, ChatGPT, and Cursor via Rolev.ai MCP.
Tailor without inventing
Every change to your resume is checked against your original. New experience goes in only with your yes—zero fabricated claims.
Draft the full kit
Tailored and Full Match resumes, role-specific cover letters, and answers to employer screening questions are prepared in seconds.
1-Click Extension Autofill
The Chrome extension fills the employer's Greenhouse, Workday, Lever, or Ashby form in 1 click. You review it and press Submit.
Similar roles
All Software Developer jobs- Technical Lead, Data EngineeringRevance · Nashville, TN
- Senior Product Engineer, McLean, VAAppian Corporation · McLean, VA
- Staff Software Engineer, Tools - Central Technology, Unreal EcosystemRiotgames · Los Angeles, CA
- Software Engineering Intern (Winter '27)Kira · San Francisco, NY
- Fullstack Software Engineer - Embedded PaymentsXplor · Kansas City, MO · $95k – $105k/yr
- Forward Deployed Engineer, PaymentsPolygon Labs · Remote (US)
Common questions
What is the salary for the Staff Software Engineer, AI Inference Gateway role at Salad?
The Staff Software Engineer, AI Inference Gateway posting at Salad states compensation of $180,000 – $220,000 per year. It is listed as a full-time position. Final pay may depend on location, experience, and interview evaluation on Salad's official career site.
Where is the Staff Software Engineer, AI Inference Gateway job at Salad located?
Yes, this Staff Software Engineer, AI Inference Gateway opening at Salad is listed as Remote (US). Always check Salad's application page for specific state eligibility or relocation policies.
What qualifications are required for Staff Software Engineer, AI Inference Gateway at Salad?
For the Staff Software Engineer, AI Inference Gateway role, Salad highlights 10+ years of relevant experience; experience with Rust, LLM inference servers, distributed systems fundamentals, OpenTelemetry. Rolev.ai can score your existing resume against these requirements and show any skill gaps before you apply.
How do I apply for the Staff Software Engineer, AI Inference Gateway position at Salad?
This listing links directly to Salad's Rippling application page. You can apply directly on Salad's site, or use Rolev.ai to check your resume fit, generate a zero-hallucination tailored resume and cover letter for this posting, and autofill the Rippling application form in 1 click with the Rolev.ai Chrome extension. You always review the form and press Submit yourself.