Senior, ML Engineer - VLM

Meet The Team

Torc is marching toward its AV 3.0 strategy, where end-to-end Vision-Language-Action (VLA) models perceive, reason, and act directly from sensor data. High-quality, semantically rich training data is the single biggest lever for that strategy, and this team owns it.

Sitting within Offline Perception, this team turns petabytes of logged multi-modal fleet data (images, kinematics) into VLM/VLA-ready datasets: geometric annotations, scenario-level semantic descriptions, action- and trajectory-grounded labels, and reasoning traces that explain why a maneuver was taken. We run a continuous data flywheel - mine long-tail and failure cases, auto-label at scale, validate quality, and feed curated datasets directly into Torc's end-to-end VLM/VLA model development. You will own the dataset layer that those models learn from.

What You'll Do

• Own the offline dataset pipeline - design, implement, test, and deploy Cloud-based pipelines that convert logged multi-sensor data into VLM/VLA training datasets, spanning geometric labels (3D/2D detection, tracking, segmentation, depth) through semantic, scenario-level, and action/trajectory-grounded annotations.
• Build VLM-assisted auto-labeling - develop open-vocabulary detection, dense captioning, semantic enrichment, and scene/scenario description generation that move beyond closed-set bounding boxes, using foundation models to scale annotation and cut manual labeling cost.
• Generate reasoning-grounded labels - produce language-grounded reasoning and chain-of-causation style annotations, temporally aligned to ego-motion and trajectories, to support VLA training and explainable driving behavior.
• Mine and curate the long tail - surface rare, difficult, and high-uncertainty scenarios, and build curated datasets that measurably improve downstream VLM/VLA model metrics rather than simply adding volume.
• Close the data flywheel - define dataset schemas, quality metrics, and validation; track auto-labeling quality against model requirements; route model failures back into re-labeling and retraining loops.
• Partner with the end-to-end model team - co-define dataset specifications with VLM/VLA model developers, own the quality bar and delivery cadence, and operationalize a continuous dataset delivery loop into their training pipelines.
• Scale on cloud infrastructure - build distributed, reproducible pipelines using columnar data formats and distributed compute, with disciplined software practices, version control, and documentation.
• Lead and mentor - serve as project lead, guide less-experienced engineers, run design reviews, set coding and annotation standards, and drive alignment across team interfaces to the rest of the organization.
• Stay current - track the latest advances in multimodal models, auto-labeling, and end-to-end autonomous driving, and translate relevant research into production data systems.

What You'll Need to Succeed

• Considered highly skilled and proficient in discipline; conducts complex, important work under minimal supervision and with wide latitude for independent judgment.
• Scope of Influence: Expected to drive alignment across team interfaces to the rest of the organization. Designs, maintains, and owns team technical solutions and drives consensus. Mentors and guides engineers within the group.
• Bachelor's Degree in Computer Science, Robotics, Electrical Engineering, or related technical field plus competences typically acquired through 6+ years of experience; OR Master's Degree in a related technical field plus competences typically acquired through 3+ years of experience.

Required Qualifications

• Computer Vision & Deep Learning - model training and at least two of: 2D/3D Object Detection, Tracking, Sensor Fusion, Semantic Segmentation, BEV, Depth Estimation.
• Multimodal / VLM experience - hands-on work with vision-language models, open-vocabulary or zero-shot recognition, dense captioning, or semantic embeddings / search applied to perception data.
• Model Data Curation - building targeted datasets that measurably improve downstream model performance; large-scale Parquet data processing (Databricks, Daft, Pandas, etc.).
• Distributed ML & data frameworks - PyTorch, Lightning, Ray, Spark, or equivalent for training and large-scale data processing.
• Scaled MLOps & Tooling - experiment tracking, model registry, MLflow / Weights & Biases, and ML metrics, evaluation, and quality.
• Development Tools & Eco-System (at scale) - strong Python software development, VDI and cloud-based development environments, CI systems (GitHub Actions), and Docker.

Bonus Points!

• End-to-end / VLA driving - familiarity with VLM/VLA or end-to-end driving models, trajectory and action grounding, or chain-of-causation / reasoning-trace datasets.
• Auto-labeling foundation models - experience with segmentation, open-vocabulary detectors, or VLM/LLM-driven data engines for annotation and verification.
• High-throughput model serving - vLLM, SGLang, or similar for batch auto-labeling and inference at scale.
• Semantic inference & retrieval - attribute mapping, semantic search, and vector databases (e.g., LanceDB) for automotive data.
• AV data standards & tooling - scenario-description standards such as Pegasus layers; parsing robotics formats (ROS bags, MCAP) and optimizing columnar storage (Parquet, Arrow).
• Cloud development & orchestration - Terraform and AWS managed services (S3, ECS, Lambda, DynamoDB, Step Functions, Athena); AWS HyperPod / Anyscale; inference orchestration.
• Data visualization - Foxglove, FiftyOne (51), three.js, OpenGL, or similar for dataset inspection and accessibility.
• Evaluation & research - closed-loop / open-loop evaluation frameworks (e.g., NavSim-style planning metrics); publications in top-tier CV/AI/Robotics venues (CVPR/ECCV/ICCV, NeurIPS/ICLR/ICML, CoRL).

Perks of Being a Full-time Torc'r

• Torc cares about our team members and we strive to provide benefits and resources to support their health, work/life balance, and future.
• Our culture is collaborative, energetic, and team focused.
• Torc offers:

• A competitive compensation package that includes a bonus component and stock options
• 100% paid medical, dental, and vision premiums for full-time employees
• 401K plan with a 6% employer match
• Flexibility in schedule and generous paid vacation (available immediately after start date)
• Company-wide holiday office closures
• AD+D and Life Insurance

Additional Information

At Torc, we're committed to building a diverse and inclusive workplace. Even if you don't meet 100% of the qualifications listed for this opportunity, we encourage you to apply.

Our compensation reflects the cost of labor across several geographic markets. Pay is based on a number of factors and may vary depending on job-related knowledge, skills, and experience. Torc's total compensation package will also include our corporate bonus and stock option plan. Dependent on the position offered, sign-on payments, relocation, and other forms of compensation may be provided as part of a total compensation package, in addition to a full range of medical, financial, and/or other benefits.

• Job ID: R-102744
• Hiring Range for Job Opening
• US Pay Range $177,300—$212,800 USD