Senior Software Engineer - Data Infrastructure

About us:

Calder is solving the data bottleneck for embodied AI and world models. Today's AI models lack the real-world, 4D data needed to reason on spatial problems and operate in physical environments. We capture, process, and deliver high-fidelity human-generated spatial data at scale, providing the substrate that trains the next generation of generalizable models for manipulation, navigation, and interaction.

About the role

We're looking for a Senior Data Infrastructure Engineer to help build and run the petabyte-scale data platform behind Calder's spatial data products. Every capture — robot rigs, egocentric headsets, multi-camera video, depth, IMU — and everything derived from it — reconstructions, Gaussian Splats, labels, training datasets — lands in this platform, gets processed and versioned, and is served to customers and training clusters. You'll work on storage layout, table formats, versioning, ingest, and the distributed processing that turns raw sensor data into training-ready datasets, and help the platform hold up as volume grows by orders of magnitude. This is a hands-on role: you'll write, deploy, and operate core parts of the platform alongside the team. We're looking for someone who has personally built and run a PB-level platform in production, not only someone who has used one.

• Build and scale Calder's PB-scale data lake / lakehouse for multimodal spatial data — object storage layout, partitioning, open table formats (Iceberg or equivalent), catalogs, and metadata

• Work on data versioning and lineage (LakeFS or equivalent): branches, commits, reproducible dataset snapshots, and provenance from raw capture to customer delivery

• Build high-throughput, fault-tolerant ingest for field and lab captures — upload, validation, checksumming, content-addressed deduplication, and safe promotion through bronze → silver → gold

• Build and operate large-scale distributed processing across CPU and GPU fleets — video decode and frame extraction, sync and calibration application, SLAM / reconstruction / 3DGS batch jobs, and model-based auto-labeling (pose, depth, detection)

• Build and maintain data pipelines at scale: idempotent jobs, retries, backfills, partitioned reprocessing, and clean recovery when a multi-day run fails halfway

• Build the dataset curation and query layer — metadata indexing, SQL over the lake (Trino / DuckDB / Spark SQL), and search so researchers and customers can slice data without copying it

• Keep GPUs fed: sharded training formats (WebDataset, Mosaic Streaming, or similar), caching, prefetching, and efficient cross-region movement between capture sites and training regions

• Work on platform reliability, observability, and cost on AWS — S3 tiering and lifecycle, egress, spot and right-sizing, capacity planning, and alerting

• Contribute to infrastructure-as-code (Terraform) and CI/CD for the data platform; write design docs and runbooks so others can operate it without tribal knowledge

• Work with capture, calibration, SLAM, 3DGS, labeling, and QA teams to turn their workload requirements into platform capabilities

About you

• B.S. or M.S. in Computer Science, Software Engineering, or a related field, or equivalent practical experience

• 6+ years in data infrastructure, data platform, or distributed systems engineering

• Hands-on experience setting up and operating a data platform at petabyte scale in production — you can speak concretely to the volumes, throughput, object counts, and failure modes you personally handled

• Deep object storage experience (S3 or equivalent): key/prefix design, request-rate and throughput limits, the small-file problem, lifecycle and storage-class tiering, and cost control

• Production experience with open table formats and lakehouse components — Iceberg, Delta Lake, or Hudi; catalogs (Glue, Polaris, Nessie, Unity, or similar); compaction and snapshot management

• Strong experience with distributed processing — Spark, Ray, Flink, or Dask — tuned for large production jobs, not only used through notebooks

• Experience with workflow orchestration (Airflow, Argo, Prefect, or similar) and the discipline that goes with it: idempotency, retries, backfills, and data contracts

• Strong programming in Python plus at least one of Go, Rust, C++, Java, or Scala

• Solid AWS infrastructure skills — EC2, EKS/ECS, S3, IAM, VPC, networking — and Terraform or equivalent IaC

• Production operations experience: you've been on call for a data platform and care about its uptime, correctness, and cost

• Clear written communication: design docs and runbooks other engineers can follow

Nice to Have

• Multimodal or unstructured data at scale — video, images, LiDAR / point clouds, sensor logs, or 3D assets — ideally from autonomous driving, robotics, mapping, or simulation

• Data versioning systems (LakeFS, DVC, Nessie) and content-addressed storage / deduplication design

• Feeding large-scale ML training: WebDataset, Mosaic Streaming, Petastorm, Ray Data, or custom loaders for multi-node GPU training

• GPU-accelerated data processing (NVIDIA DALI, RAPIDS, NVDEC / FFmpeg pipelines)

• Query engines over lake data (Trino, DuckDB, ClickHouse) and embedding / metadata search for dataset curation

• Multi-region or hybrid cloud / on-prem architectures, including large-scale data transfer and egress optimization

• Observability stacks (Datadog, Prometheus / Grafana, OpenTelemetry)

• Robotics log formats (ROS bags, MCAP) and time-synchronized multi-sensor data

• Contributions to open-source data infrastructure projects

Calder is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status. Learn more here.

By applying for this position, you consent to the collection, storage, and processing of your personal data by Calder for recruitment purposes. We will handle your information securely and in accordance with local privacy policies. Your data may be shared with authorized third-party service providers aiding in the hiring process.