Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.
About the role
Design and operate high-volume analytical data systems end to end using Apache Spark. Build and maintain high-scale batch and near-real-time data pipelines on self-managed infrastructure.
What they look for
Requirements
Requires 7+ years of experience in data engineering and software development with deep production expertise in Apache Spark. Candidates must be proficient in Java, Scala, or Python and have experience operating Spark on self-managed clusters.
Full description
Building high-scale batch and near-real-time data pipelines deployed on infrastructure we run ourselves (on-prem), not managed cloud services. You will design and operate high-volume analytical data systems end to end, with Apache Spark as the core processing engine for both batch and streaming workloads.
Requirements
- 7+ years
of experience in data engineering and software development
- Ability to
write high-quality code in Java/Scala, Python, or equivalent languages
- Deep,
hands-on production experience with Apache Spark — batch and Spark Structured Streaming (core requirement)
- Demonstrated
Spark performance tuning: partitioning, caching and persistence, broadcast joins, shuffle reduction, data-skew handling, and Adaptive Query Execution
- Experience
operating Spark on self-managed clusters (YARN, Kubernetes, or standalone) — executor sizing, resource allocation, and multi-tenant workloads
- Practical
experience with Kafka (or equivalent messaging systems) as a Spark source and sink for high-volume workloads, including offset and checkpoint management
- Practical
experience with distributed query engines (e.g., Trino/Presto or similar)
- Practical
experience with ETL / data integration tools, commercial or open-source (e.g., Datastage, Informatica, Apache NiFi, or similar)
- Practical
experience with SQL-based transformation frameworks (e.g., dbt or others)
- Strong SQL
skills and understanding of data modeling and data warehousing for analytical workloads
- Hands-on
experience with real-time / low-latency analytical stores (columnar or OLAP engines, e.g., Apache Pinot/ClickHouse or similar)
- Practical
experience with big-data platforms and distributions (e.g., Cloudera, Hadoop ecosystem, Databricks, or similar)
- Practical
experience containerizing and operating data workloads (Docker; Kubernetes a plus)
- Experience
with workflow orchestration tools (e.g., Airflow or similar)
- Familiarity
with data lake table formats (e.g., Apache Iceberg, Delta Lake, or similar), including schema evolution and compaction
- Familiarity
with data governance / cataloging tools (e.g., DataHub or similar)
- Familiarity
with lakehouse management systems (e.g., Apache Amoro or similar)
- Familiarity
using AI tools for development and debugging (Claude, Cursor, Codex)
Similar roles
-
Senior Data Engineer II
Oscar Health New York, New York, United States · $181K–$237K/yr
-
Data Engineer
AcuityMD $113K–$159K/yr
-
Staff Data Engineer, Analytics
Afresh San Francisco, California, United States · $168K–$250K/yr
-
Data Engineer - Agentic AI
Defense Unicorns Springfield, Virginia, United States · $149K–$201K/yr
-
Senior Data Engineer
Southend Pharmacy Houston, Texas, United States · €45K–€65K/yr
-
Senior Data Engineer (Talent Pool)
ioet San José de Alluriquín, Santo Domingo de los Tsáchilas, Ecuador