Senior / Staff Machine Learning Engineer, ACAI Training & Inference Platform

Summary

The ACAI Training & Inference Platform team is seeking a Senior / Staff Software Engineer with strong distributed systems expertise and deep experience in AI training and inference. As part of Apple Cloud AI (ACAI) within Apple’s Data Platform (ADP), you will design and build core capabilities for large-scale model training, post-training, fine-tuning, reinforcement learning, and high-performance inference.

You will develop scalable AI execution frameworks, developer-facing APIs, and runtime optimizations that make it easier to train, adapt, and serve advanced models efficiently across Apple’s AI ecosystem. You will partner closely with AI researchers and engineers to translate evolving model architectures and workload requirements into reliable, high-performance training and inference solutions. This role is ideal for an engineer who combines deep distributed-systems expertise with hands-on knowledge of modern LLM training, post-training, and inference optimization.

About the role

ADP brings together the Apple Cloud Data Platform (ACDP), Apple Cloud AI (ACAI), and Apple Governance Platform (AGP) to provide foundational capabilities for data, analytics, AI, and governance at Apple scale. Within ACAI, the Training & Inference Platform team enables scalable model development and production inference across Apple’s AI ecosystem.

As a Senior / Staff Software Engineer, you will build capabilities spanning the model lifecycle, from distributed training and post-training to optimized inference and serving.

On the training side, you will develop frameworks and APIs for fine-tuning, reinforcement learning, and post-training workflows, enabling AI teams to efficiently adapt foundation models using techniques such as supervised fine-tuning (SFT), preference optimization, and reinforcement learning from feedback or verifiable rewards. You will simplify the execution of complex training pipelines, improve experimentation and iteration speed, and enable reusable, production-ready training workflows.

On the inference side, you will drive deep runtime and LLM-serving optimizations to improve throughput, reduce latency, and lower the cost of serving large models. You will optimize execution across heterogeneous hardware and model architectures, leveraging techniques such as continuous batching, KV-cache optimization, quantization, speculative decoding, disaggregated prefill and decode, and efficient distributed inference.

Across both areas, you will improve the performance, scalability, reliability, and developer experience of AI training and inference workloads. You will work closely with AI practitioners to identify bottlenecks across model execution, scheduling, communication, and resource utilization, translating these insights into end-to-end improvements in model development and serving efficiency.

Responsibilities

Design and build scalable distributed AI execution frameworks for large-scale model training, fine-tuning, post-training, reinforcement learning, and inference using Ray and related technologies. Develop high-level APIs, SDKs, and reusable workflows that simplify model training and adaptation, including supervised fine-tuning, preference optimization, and reinforcement learning workflows.

• Enable end-to-end post-training workflows, from dataset preparation and experiment orchestration through distributed execution, evaluation, and iterative model improvement.
• Optimize LLM inference runtimes for cost, latency, throughput, and resource efficiency across diverse model architectures and hardware configurations.
• Drive deep runtime optimizations spanning GPU execution, memory management, kernel utilization, communication, batching, KV-cache management, and distributed inference.
• Evaluate and integrate modern inference and training engines, including vLLM, SGLang, and other relevant frameworks, to deliver efficient and extensible AI execution capabilities.
• Design and optimize distributed execution strategies for large models, including parallelism, workload placement, resource allocation, and communication efficiency.
• Improve training and inference reliability through fault tolerance, checkpointing, workload recovery, observability, and performance diagnostics.

Partner with AI researchers, model developers, and product teams to translate model and workload requirements into reusable capabilities that accelerate model development and production adoption.Establish performance benchmarks and optimization strategies that quantify improvements in training efficiency, post-training iteration speed, inference latency, throughput, and cost per token.

Contribute architectural improvements to Ray and related open-source AI systems where appropriate.

Minimum Qualifications

• 6+ years of experience building distributed systems, AI platforms, model execution frameworks, or high-performance software systems.
• Strong hands-on experience in at least one of the following areas: large-scale AI training, LLM post-training and reinforcement learning, or LLM inference and serving optimization.
• Proficiency in Python and experience with a systems-level language such as C++, Rust, Go, or Java.
• Experience with PyTorch and modern foundation model architectures, including the execution and performance requirements of large language models.
• Understanding of distributed execution and parallelism strategies for large models, including data, tensor, pipeline, or expert parallelism.
• Experience profiling and optimizing AI workloads across GPUs or other accelerators, with the ability to identify and resolve performance bottlenecks across compute, memory, and communication.
• Strong understanding of at least one of the following: distributed training and post-training workflows, reinforcement learning systems, LLM inference runtimes, or high-performance model serving.
• Experience building APIs, SDKs, or reusable software components that simplify AI model development, training, adaptation, evaluation, or deployment.
• Experience with Ray or comparable distributed execution frameworks, and familiarity with production-scale AI workloads.
• Strong software design, debugging, and performance analysis skills, with the ability to deliver reliable, scalable, production-quality systems.
• B.S., M.S., or Ph.D. in Computer Science, Artificial Intelligence, or a related technical field, or comparable software engineering experience.

Preferred Qualifications

• Experience building end-to-end LLM post-training workflows, including supervised fine-tuning (SFT), preference optimization, RLHF, or reinforcement learning with verifiable rewards (RLVR).
• Experience with distributed training and post-training frameworks such as Ray Train, TRL, VeRL, RLlib, DeepSpeed, FSDP, or equivalent technologies.
• Experience designing developer-facing training APIs or services that support iterative experimentation, checkpointing, evaluation, and scalable model adaptation.
• Deep understanding of LLM inference optimization, including continuous batching, KV-cache management, quantization, speculative decoding, prefix caching, and disaggregated prefill/decode.
• Experience optimizing inference engines such as vLLM, SGLang, TensorRT-LLM, or similar frameworks for latency, throughput, GPU utilization, and cost per token.
• Experience with deep runtime optimization, including GPU kernels, memory allocation, graph execution, communication primitives, scheduling, and accelerator utilization.
• Experience benchmarking and tuning inference workloads across model architectures, hardware configurations, concurrency levels, and latency targets.
• Understanding of LLM serving architectures, including dynamic batching, autoscaling, multi-model serving, heterogeneous hardware, and workload-aware resource allocation.

Additional information

Experience integrating training and inference engines into production AI platforms through stable APIs, reusable execution abstractions, and reliable operational workflows.

Contributions to open-source AI training, post-training, distributed execution, inference, or model-serving projects.

Familiarity with Ray Data, Spark, Flink, or similar data-processing technologies used in AI data preparation and training pipelines.

Experience operating AI workloads in production across cloud or on-premises accelerator environments, including Kubernetes or comparable execution systems.