Senior Software Engineer, Apple Data Platform
Apple Cupertino, California, United States
Computers and Electronics Manufacturing · 10,001+ employees
Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.
About the role
You will design and implement operators and expressions to accelerate Apache Spark using Apache DataFusion and Apache Arrow. Additionally, you will profile and benchmark real workloads to optimize memory, shuffle, and I/O performance across data centers and cloud services.
What they look for
Requirements
Candidates must have at least 3 years of professional software engineering experience and a proven track record in building distributed systems. A degree in Computer Science or a related field is required, along with experience contributing to Apache Spark or related open-source projects.
Full description
We are seeking extraordinary software engineers to work on Apache DataFusion Comet, the open-source accelerator for Apache Spark that Apple created and donated to the Apache Software Foundation. You are passionate about pushing distributed query engines to their limits and getting every ounce of performance out of them, while matching Spark’s results exactly. We are looking for engineers with in-depth knowledge of Spark internals and of related technologies such as Apache Iceberg, Parquet, Arrow, and DataFusion. You are also excited about working in the open, collaborating with teams across Apple that run Spark at scale and contributing upstream to these open-source communities.
Description
As an engineer on our team, you will advance Apple Data Platform by making Apache Spark faster and more efficient with Apache DataFusion Comet. You will design and implement operators and expressions in a native language on Apache DataFusion and Apache Arrow, extend Comet’s Spark integration across Spark versions, and accelerate reads and writes of Apache Iceberg and Apache Parquet data, all while keeping results identical to Spark’s. You’ll collaborate closely with a dedicated team of open-source data experts, including committers to Apache Spark, DataFusion, and Iceberg, among others, and contribute that work upstream. You will also profile and benchmark real workloads and tune memory, shuffle, and I/O to roll Comet out reliably across both data centers and cloud services. Your work will directly contribute to the resilience, scalability, and performance of systems that support millions of users worldwide.
Minimum Qualifications
3+ years of professional software engineering experience. A proven track record of building distributed systems. Contributor or committer to Apache Spark and/or Apache DataFusion, Comet. BS, MS, or PhD in Computer Science or a related field, or equivalent practical experience.
Preferred Qualifications
Experience with columnar or vectorized query execution, or with performance profiling and optimization of query engines. Good knowledge of Apache Iceberg and Apache Parquet internals. Experience working on or building Spark connectors to external data sources. Knowledge of Kubernetes, or other compute platforms.
