Lead Data Engineer - Data Engineering, Databricks, Model Serving, ML Ops, Data Architecture and modeling
JPMorgan Chase & Co. Mumbai, Maharashtra, India
Financial Services · 10,001+ employees
Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.
About the role
The Lead Data Engineer will architect and maintain large-scale data pipelines and machine learning infrastructure within the Databricks environment. They will collaborate with cross-functional teams to deliver secure, scalable data solutions while ensuring high standards of data quality and governance.
What they look for
Requirements
Candidates must have 5+ years of applied experience in data engineering with strong proficiency in Apache Spark and distributed computing frameworks. Practical experience with Databricks, SQL, and production-level ML model serving infrastructure is required.
Full description
Join us as we embark on a journey of collaboration and innovation, where your unique skills and talents will be valued and celebrated. Together we will create a brighter future and make a meaningful difference.
As a Lead Data Engineer at JPMorganChase within the Consumer and Community Banking , you are an integral part of an agile team that works to enhance, build, and deliver data collection, storage, access, and analytics solutions in a secure, stable, and scalable way. As a core technical contributor, you are responsible for maintaining critical data pipelines and architectures across multiple technical areas within various business functions in support of the firm’s business objectives.
Job responsibilities
- Architects and maintains large-scale data pipelines using Apache Spark and related distributed computing frameworks, ensuring efficient processing of structured and unstructured data.
- Builds and optimizes both batch and streaming data workloads to support analytical and operational use cases.
- Develops, optimizes, and productionizes notebooks, jobs, and workflows within the Databricks environment to support analytical and machine learning use cases.
- Designs and implements data architecture and data modeling strategies, including dimensional models, lakehouse structures, and schema design that balance performance, scalability, and governance requirements.
- Writes and optimizes complex SQL queries for data transformation, validation, and performance tuning across large datasets.
- Supports ML model serving infrastructure, ensuring trained models are deployed, monitored, and scaled reliably in production environments.
- Collaborates closely with data scientists and ML engineers on MLOps practices, including model versioning, CI/CD pipelines for ML artifacts, monitoring, and automated retraining workflows.
- Partners with cross-functional stakeholders — including analytics, risk, and platform engineering teams — to translate business requirements into technical data solutions.
- Maintains strong data quality, lineage, and security standards consistent with data governance and compliance expectations.
- Leverages enterprise-authorized AI coding assist tools within the work environment to improve code quality, delivery speed, and productivity across complex deliverables (e.g., code generation/refactoring, unit test creation, documentation), while validating outputs through peer review, automated testing, and secure coding standards; contributes learnings and reusable patterns to improve broader team effectiveness.
- Applies knowledge of tools within the Software Development Life Cycle toolchain, including enterprise-authorized AI-assisted development and automation capabilities, to improve the value realized by automation
Required qualifications, capabilities, and skills
- Formal training or certification on data engineering concepts and 5+ years applied experience
- Demonstrates strong, hands-on proficiency with Apache Spark or comparable distributed compute frameworks, including experience optimizing jobs for performance and cost at scale.
- Demonstrates hands-on exposure to distributed computing, cloud data platforms, and production ML infrastructure.
- Holds direct, production-level experience with Databricks, including notebooks, Delta Lake, job orchestration, and cluster management.
- Demonstrates solid command of SQL, with the ability to write, debug, and optimize complex queries against large and varied datasets.
- Brings practical experience supporting ML model serving infrastructure in production environments and shows familiarity with MLOps tooling and practices, such as model monitoring, versioning, and deployment automation.
- Possesses a strong foundation in data architecture and data modeling, covering both operational and analytical data store design.
- Has working knowledge of cloud data platforms such as AWS, Azure, or GCP.
- Maintains proficiency with version control systems such as Git.
- Hands-on experience using enterprise-authorized AI-assisted software development tools within the work environment (e.g., for coding, test creation, troubleshooting, or documentation) with demonstrated ability to critically evaluate, validate, and refine AI-generated outputs for correctness, performance, and security.
- Understanding of responsible AI use in engineering workflows, including data sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectations; ability to guide peers on safe and effective usage within team practices
Preferred qualifications, capabilities, and skills
- Experience with Airflow or Databricks Workflows, Docker and Kubernetes, and CI/CD pipelines for data and machine-learning workflows.
- Familiarity with feature stores and model registries such as MLflow, and experience with real-time streaming architectures such as Kafka and Spark Structured Streaming.
- Prior experience in financial services or another highly regulated industry, with an understanding of data governance and compliance requirements.
Similar roles
-
(Junior) Data Engineer | Knowledge Graphs & Pharma Data (f/m/d)
Accurids Cologne, North Rhine-Westphalia, Germany · €50K–€80K/yr
-
Staff, Data Engineer, Enablement
Walmart Global Marketplace | Europe Shenzhen, Guangdong Province, China
-
Senior, Data Engineer, Enablement
Walmart Global Marketplace | Europe Shenzhen, Guangdong Province, China
-
Data Engineer [The Division 2]
Ubisoft Bucharest, Romania
-
Data Engineer, Elgato
CORSAIR Munich, Bavaria, Germany
-
Senior Data Engineer
Madlan Tel Aviv, Tel-Aviv District, Israel