JobCubby JobCubby
Sign up
T

Machine Learning Infrastructure Engineer

True Talent Solutions Partners San Francisco, California, United States · $200K–$400K/yr

Yesterday
machine-learning Senior (5-10 yrs) Full-time United States
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.

About the role

You will design, deploy, and maintain distributed machine learning training and inference clusters for large-scale models. Additionally, you will build scalable data pipelines and optimize low-level GPU operations to improve performance.

What they look for

Machine learning infrastructure Distributed training GPU performance debugging Python C++ CUDA JAX Kubernetes Docker FSDP DeepSpeed Cloud platforms Linux Data infrastructure Inference systems

Requirements

Candidates must have two to ten years of experience building large-scale machine learning infrastructure and proficiency with distributed training frameworks. Strong technical skills in Python, C++, CUDA, and cloud platforms are required, along with experience in GPU performance debugging.

Benefits

Relocation support

Full description

About the Role

Join an AI research team building large-scale models for physical systems, with weather as an initial application. You will develop the infrastructure for distributed model training and inference, helping turn large volumes of observational data into useful predictions and insights.

What You'll Do

  • Design, deploy, and maintain distributed machine learning training and inference clusters.
  • Build scalable pipelines for petabyte-scale datasets and manage model training across the machine learning lifecycle.
  • Test training approaches, including parallelization methods and numerical precision trade-offs across model sizes.
  • Profile and debug low-level GPU operations to improve performance.
  • Evaluate research and apply relevant techniques to infrastructure and training workflows.
  • Analyze training experiments, including results from ablation studies, to inform model development.

What We're Looking For

  • Two to ten years of experience building large-scale machine learning infrastructure for pretrained or foundation models.
  • Hands-on experience with compute clusters, data infrastructure, inference systems, and distributed training.
  • Strong skills in optimizing large-scale training, data, or inference workloads, including GPU performance debugging.
  • Proficiency with distributed training frameworks such as FSDP or DeepSpeed, and experience with CUDA and JAX.
  • Experience with Python, C++, Linux, NVIDIA GPUs, and cloud platforms such as GCP, AWS, or Azure.
  • Familiarity with Docker, Kubernetes, distributed task management, model serving, monitoring, logging, and observability.
  • Background in physical AI or AI for science, such as robotics, self-driving, or biology, and interest in mission-driven work.
  • A generalist approach to the machine learning lifecycle and experience delivering in fast-paced, time-sensitive environments.

Compensation & Benefits

Base salary is $200,000 to $400,000 USD annually. Visa sponsorship is not available. Relocation support is provided.

Location

On-site in San Francisco, California, United States, five days per week.

Similar roles