JobCubby JobCubby
Sign up
Photon

Lead Data Engineer - Chennai / Bengaluru

Photon India

IT Services and IT Consulting · 5,001-10,000 employees

6 h ago
data-engineer Senior (5-10 yrs) Full-time India
Log in to apply, save this posting, or score it against your profile with AI.

About the role

Develop and maintain data pipelines, ELT processes, and workflow orchestration to ensure efficient data delivery. Design and implement custom connectors and CI/CD pipelines to facilitate automated data integration and transformation.

What they look for

Apache Airflow Python PySpark Hive Trino SQL Snowflake Data Pipelines ELT Processes Workflow Orchestration Data Ingestion CI/CD Pipelines Data Integration Data Validation Data Governance

Requirements

Candidates must possess strong skills in Apache Airflow, Python, PySpark, Hive, Trino, SQL, and Snowflake. The role requires experience in designing data pipelines and collaborating with cross-functional teams to translate business needs into technical solutions.

Full description

Must Needed Skills: Apache Airflow, Python, PySpark ,Hive(Trino), SQL and Snowflake

  • Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python, PySpark ,Hive(Trino) and Snowflake to ensure the efficient and reliable delivery of data.
  • Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .
  • Collaborate closely with cross-functional teams to gather requirements, understand data needs, and translate them into technical solutions.
  • Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.
  • Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading.
  • Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data.
  • Stay up-to-date with emerging technologies and best practices in data engineering.
  • Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

Similar roles