Senior Software Engineer II, Inference

Key Responsibilities

• Lead design reviews and drive architecture within the team; decompose multi-service work into clear milestones.
• Define and own SLIs/SLOs; ensure post-incident actions land and reliability improves release-over-release.
• Implement advanced optimizations (e.g., micro-batch schedulers, speculative decoding, KV-cache reuse) and quantify impact.
• Strengthen incident posture: capacity planning, autoscaling policy, graceful degradation, rollback/traffic-shift strategies.
• Mentor IC1/IC2 engineers; review cross-team designs and elevate coding/testing standards.
• For IC4: own an area spanning multiple services and teams (e.g., request routing & adaptive scheduling, cost-per-token analytics, GPU resource isolation).

Qualifications

• IC3: ~3-5 years; IC4: ~5-8 years industry experience building distributed systems or cloud services.
• Strong coding in Python or Go (C++ a plus) and deep familiarity with networked systems and performance.
• Hands-on experience with Kubernetes at production scale, CI/CD, and observability stacks (Prometheus, Grafana, OpenTelemetry).
• Practical knowledge of inference internals: batching, caching, mixed precision (BF16/FP8), streaming token delivery.
• Proven track record improving tail latency (P95/P99) and service reliability through metrics-driven work.
• Bachelor's or Master's in Computer Science or related field (or equivalent practical experience).

Preferred Skills

• Contributions to inference frameworks (vLLM, Triton, TensorRT-LLM, Ray Serve, TorchServe).
• Experience with CUDA kernels, NCCL/SHARP, RDMA/NUMA, or GPU interconnect topologies.
• Leading multi-team initiatives or partnering with customers on mission-critical launches.

What We Offer

• In addition to a competitive salary, we offer a variety of benefits to support your needs.
• The benefits below reflect our US-based offerings for full-time employees; for roles in other locations, benefits vary and are shared during the hiring process.
• These include:

• Medical, dental, and vision insurance - 100% paid for by CoreWeave
• Company-paid Life Insurance
• Voluntary supplemental life insurance
• Short and long-term disability insurance
• Flexible Spending Account
• Health Savings Account
• Tuition Reimbursement
• Ability to Participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-Forming support provided by Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Flexible PTO
• Catered lunch each day in our office and data center locations
• A casual work environment
• A work culture focused on innovative disruption

• California Applicants
• California Consumer Privacy Act