Member of Technical Staff, Evals Platform

Key Responsibilities

• Scale Eval Infrastructure: Take ownership of our internal eval setup and evolve it for the future. You will design systems to eliminate single-host coordination bottlenecks, resolve log-syncing latency, and build seamless programmatic access.
• Benchmark Obsession & Reproduction: Act as a "metrics obsessive." Track state-of-the-art (SOTA) benchmarks, read the latest research papers, dig deep into data discrepancies, and insist on rigorously reproducing published results internally.
• Pioneer New Benchmarks: Design and build entirely new benchmarks to measure model performance on complex, emerging, or domain-specific use cases.
• Own Fine-Tuning Product Experiences: Build and improve user-facing workflows for post-training, including fine-tuning experiences across SFT, RFT, and related model-improvement capabilities.
• Work Closely With Users: Partner with customers and internal stakeholders to understand evaluation and fine-tuning needs, triage issues, and convert bespoke workflows into productized, reusable solutions.

Minimum Requirements

• Experience: 1-7 years of software engineering or data science experience (We are hiring at multiple levels for this role).
• Strong System Design Skills: You know how to architect scalable, programmatic systems and transition legacy setups into robust infrastructure.
• Sandbox Infrastructure: Hands-on experience building or working with sandbox environments and sandbox infrastructure for secure code execution and testing.
• Analytical & Data Science Mindset: You possess a deep understanding of LLM evaluations, how to design them, and how to use the results to guide model improvement. You are meticulous about data and metrics.
• Understanding of the GenAI Lifecycle: You understand the end-to-end workflow-from prompting a base model to curating a dataset, fine-tuning, and productionizing agents.

Preferred Qualifications

• Experience: 3+ years of software engineering or applied data science experience.
• Frameworks & Orchestration: Experience working with the Harbor framework or similar container registry and orchestration tools.
• Public Writing & Analysis: A strong interest in discovering where different models excel and fall short, with a desire to write up and publish these insights publicly (e.g., technical blogs, whitepapers).
• Inference & Hardware Knowledge: Interest in the hardware side of AI-understanding GPU constraints, inference optimization techniques, and how they relate to model performance.
• Startup DNA: Experience in fast-paced environments where you own features end-to-end.

Why Fireworks?

• Solve Hard Problems: Tackle challenges at the forefront of AI infrastructure, from low-latency inference to scalable model serving.
• Build What’s Next: Work with bleeding-edge technology that impacts how businesses and developers harness AI globally.
• Ownership & Impact: Join a fast-growing, passionate team where your work directly shapes the future of AI-no bureaucracy, just results.
• Learn from the Best: Collaborate with world-class engineers and AI researchers who thrive on curiosity and innovation.

Additional information

Fireworks AI is an equal-opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all innovators.