Data Engineer

Job Summary

We are looking for an experienced Data Engineer to design, build, and maintain a scalable, metadata-driven data ingestion and transformation framework on Google Cloud Platform (GCP).

The role will be responsible for developing configuration-driven ingestion pipelines, onboarding multiple source systems, and implementing Raw, Bronze, Silver, and Gold data layers. The Data Engineer will use technologies including BigQuery, Cloud Composer/Apache Airflow, Python, Dataform, Datastream, Pub/Sub, Dataflow, Cloud Run, and Cloud Storage to deliver reliable and reusable data solutions.

The ideal candidate will have strong experience in metadata-driven pipeline development, CDC, advanced SQL, Python, data quality, reconciliation, automated testing, and CI/CD. The engineer will work closely with Data Governance and other platform stakeholders to ensure that data pipelines are scalable, governed, auditable, testable, and production-ready.

Responsibilities

• Design and build a metadata-driven and configuration-driven ingestion framework using reusable pipeline templates.

• Design, develop, and maintain the metadata configuration model covering source systems, source objects, load configurations, execution parameters, run logs, audit information, and reprocessing.

• Build dynamic Cloud Composer / Apache Airflow DAGs that generate and execute ingestion workflows based on metadata configurations.

• Develop reusable ingestion patterns and pipeline generators instead of building separate pipelines for individual source tables.

• Onboard multiple source systems, including relational databases, SAP, REST APIs, files, streaming sources, and other enterprise platforms.

• Implement data ingestion into Raw and Bronze layers following defined lakehouse architecture and engineering standards.

• Implement source-to-target reconciliation to ensure completeness and accuracy of ingested data.

• Implement robust operational controls, including error handling, retries, quarantine processing, alerting, monitoring, and audit logging.

• Develop replay-by-batch and reprocessing capabilities to support failed loads, historical reloads, and controlled data recovery.

• Support batch, incremental, streaming, and Change Data Capture (CDC) ingestion patterns.

• Implement Bronze-layer processing, including data typing, cleansing, schema validation, and schema enforcement.

• Implement data de-duplication using appropriate source keys, primary keys, or business keys.

• Implement soft-delete representation and appropriate handling of deleted source records.

• Implement CDC change-history materialization and maintain historical changes where required.

• Develop appropriate BigQuery partitioning, clustering, MERGE, and incremental processing strategies.

• Perform data compaction and other performance optimization activities where required.

• Design and develop Silver and Gold transformation models using Dataform.

• Develop reusable transformation components, macros, dependencies, and incremental transformation strategies.

• Implement assertions and automated tests for transformation models and ensure no untested transformation reaches production.

• Implement in-pipeline data quality checks, validation rules, and automated promotion gates.

• Work closely with the Data Governance Consultant to incorporate data quality, governance, lineage, audit, and control requirements into the engineering framework.

• Prevent data that fails critical quality requirements from being promoted to downstream layers.

• Implement geospatial data ingestion, including GeoJSON processing and conversion to BigQuery GEOGRAPHY.

• Ensure appropriate preservation and handling of Spatial Reference System (SRS) information for geospatial datasets.

• Implement ingestion and management of semi-structured and unstructured data using Google Cloud Storage and BigQuery object tables.

• Follow Git-based development practices, including branching, pull/merge requests, peer reviews, and code reviews.

• Develop automated tests as a standard part of pipeline and transformation development.

• Integrate data pipelines and Dataform transformations with CI/CD processes.

• Document each source-system onboarding, including configuration, mappings, dependencies, reconciliation, operational procedures, and troubleshooting guidance.

• Develop and maintain operational runbooks and onboarding documentation that enable ESNAD teams to independently onboard additional data sources.

Must-Have Skills

• Minimum 5+ years of professional Data Engineering experience.

• Minimum 2+ years of hands-on experience building data pipelines on Google Cloud Platform (GCP).

• Strong hands-on experience with Google Cloud Platform data engineering services.

• Expert-level SQL skills, including complex analytical SQL development.

• Strong hands-on experience with Google BigQuery, including data modeling and large-volume data processing.

• Strong Python programming skills for data engineering, automation, API integration, validation, and framework development.

• Strong hands-on experience with Cloud Composer / Apache Airflow.

• Proven experience developing dynamic Airflow DAGs.

• Experience developing metadata/configuration-driven DAG generation.

• Experience with Airflow sensors, custom operators, task dependencies, scheduling, retries, and failure management.

• Proven experience designing and implementing metadata-driven or configuration-driven ingestion frameworks.

• Experience developing pipeline generators and reusable ingestion templates, rather than only building individual pipelines for individual tables.

• Experience designing metadata configurations for source systems, source objects, load parameters, runtime configurations, audit logs, and reprocessing.

• Hands-on experience with Google Cloud Datastream or an equivalent log-based CDC technology.

• Strong understanding of CDC patterns, including inserts, updates, deletes, soft deletes, incremental loads, and historical change management.

• Experience implementing CDC recovery, reconciliation, replay, and reprocessing patterns.

• Hands-on experience with Dataform or dbt.

• Strong understanding of transformation model structures, macros, reusable logic, assertions, automated tests, dependencies, and incremental strategies.

• Experience developing Raw, Bronze, Silver, and Gold data layers or equivalent medallion/lakehouse architecture.

• Experience implementing source-to-target data reconciliation and validation.

• Experience implementing exception handling, quarantine processes, retry mechanisms, operational monitoring, and alerting.

• Working knowledge of Google Cloud Pub/Sub and event-driven or streaming ingestion patterns.

• Working knowledge of Dataflow / Apache Beam for data processing.

• Working knowledge of Cloud Run and/or Cloud Data Fusion.

• Strong understanding of relational database extraction and database-engine-specific CDC constraints.

• Working knowledge of SAP data extraction patterns, including ODP, SLT, or certified SAP connectors.

• Experience with REST API ingestion, including authentication, pagination, error handling, and incremental extraction.

• Experience with file-based data ingestion and file manifest validation.

• Experience implementing automated data quality checks, validation rules, and pipeline quality gates.

• Strong knowledge of BigQuery partitioning, clustering, MERGE operations, de-duplication, schema enforcement, and incremental processing.

• Strong understanding of data pipeline monitoring, auditability, traceability, and operational support.

• Strong experience with Git-based development and version control.

• Experience working with peer reviews, code reviews, and controlled development workflows.

• Experience writing unit, integration, data quality, or pipeline tests as a standard engineering practice.

• Experience implementing or working with CI/CD pipelines for data engineering workloads.

• Ability to produce high-quality technical documentation, source onboarding documentation, and operational runbooks.

Good-to-Have Skills

• Google Cloud Professional Data Engineer certification.

• Google Cloud Associate Cloud Engineer certification.

• dbt Analytics Engineering Certification.

• Advanced hands-on experience with Google Cloud Datastream and enterprise-scale CDC implementations.

• Advanced experience with Dataflow / Apache Beam for batch and streaming workloads.

• Advanced experience with Pub/Sub and event-driven data architectures.

• Advanced BigQuery performance optimization and cost-optimization experience.

• Experience designing and implementing enterprise-scale lakehouse or medallion architectures.

• Experience designing reusable enterprise data ingestion frameworks and platform accelerators.

• Advanced hands-on experience with Dataform, including reusable components, assertions, incremental models, and CI/CD.

• Experience designing enterprise-level automated data quality frameworks.

• Experience with data governance, metadata management, data lineage, data cataloging, auditability, and traceability.

• Strong hands-on experience with SAP ODP, SAP SLT, or certified SAP extraction connectors.

• Experience implementing SAP CDC and incremental extraction patterns.

• Experience processing GeoJSON and BigQuery GEOGRAPHY data.

• Knowledge of geospatial data engineering and Spatial Reference Systems (SRS).

• Experience handling semi-structured and unstructured data at enterprise scale.

• Experience with Google Cloud Storage and BigQuery object tables.

• Experience implementing CI/CD pipelines for Dataform, dbt, Airflow, and other data engineering workloads.

• Experience with automated deployments, environment promotion, testing gates, and release-management practices.

• Knowledge of cloud security, IAM, service accounts, secrets management, and secure data pipeline design.

• Experience implementing monitoring, observability, logging, alerting, and operational dashboards for enterprise data pipelines.

• Experience working in large enterprise data transformation or cloud migration programs.

• Experience working closely with Data Governance, Data Architecture, Security, DevOps, and Business teams.

• Experience mentoring junior and mid-level Data Engineers and establishing engineering standards.

• For a Senior Data Engineer / Technical Lead, 8+ years of overall Data Engineering experience is preferred, along with demonstrated technical leadership and architecture ownership.