InFynd

Senior Data Engineer

InFynd Coimbatore North, Tamil Nadu, India

Information Services · 51-200 employees

4 d ago
data-engineer Mid (2-5 yrs) Full-time India
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

About the role

Design, develop, and maintain scalable ETL/ELT data pipelines using Azure technologies and distributed processing frameworks. Manage data ingestion, transformation, and workflow orchestration while ensuring optimal performance and production support.

What they look for

Azure Data Factory Azure Databricks PySpark Python SQL Apache Spark Data Engineering ETL/ELT Data Modeling Azure Synapse Analytics Data Warehousing CI/CD Azure DevOps Git Apache Airflow Change Data Capture

Requirements

Requires 4-5 years of relevant experience in data engineering with strong proficiency in Python, SQL, and PySpark. Candidates must have hands-on experience with Azure cloud services and data warehousing methodologies.

Full description

  • Design,

develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.

  • Develop

data ingestion and transformation workflows using Python, SQL, and PySpark.

  • Work

with Apache Spark and PySpark for large-scale distributed data processing.

  • Implement

batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms.

  • Integrate

data from multiple sources, including relational databases, APIs, and cloud storage.

  • Work

with Azure Data Lake Storage Gen2 and Azure Blob Storage.

  • Develop

and optimize data models, fact tables, dimension tables, and data warehouse structures.

  • Perform

data cleansing, validation, deduplication, and aggregation.

  • Implement

and manage workflows using Azure Data Factory and Apache Airflow.

  • Optimize

SQL queries, Spark jobs, and data pipelines for better performance.

  • Work

with Azure Synapse Analytics and relational databases.

  • Implement

error handling, logging, monitoring, and alerting for production pipelines.

  • Manage

version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.

  • Troubleshoot

production issues, perform root cause analysis, and provide production support.

  • Maintain

technical documentation and collaborate with cross-functional teams.

Requirements

Programming Languages

  • Python,

SQL, PySpark.

  • Pandas,

NumPy, OOP concepts, exception handling, and API integration.

Big Data Technologies

  • Apache

Spark and PySpark.

  • Spark

DataFrames, Spark SQL, transformations, actions, joins, aggregations, and window functions.

  • Partitioning,

caching, and performance optimization.

Azure Cloud Technologies

  • Azure

Data Factory (ADF).

  • Azure

Databricks.

  • Azure

Data Lake Storage Gen2 (ADLS Gen2).

  • Azure

Blob Storage.

  • Azure

Synapse Analytics.

  • Azure

Key Vault and Azure Monitor.

ETL & Data Engineering

  • ETL/ELT

pipeline development.

  • Data

ingestion, transformation, and migration.

  • Change

Data Capture (CDC).

  • Incremental

loading and full loading.

  • Batch

and distributed processing.

Database Technologies

  • Microsoft

SQL Server, PostgreSQL, MySQL, and Oracle.

  • Complex

SQL queries, indexing, stored procedures, and query optimization.

Data Modeling & Warehousing

  • Dimensional

and relational data modeling.

  • Fact

and dimension tables.

  • Data

normalization and denormalization.

  • Schema

design and data warehouse development.

Workflow Orchestration & DevOps

  • Azure

Data Factory and Apache Airflow.

  • Git,

GitHub, Azure DevOps, and CI/CD pipelines.

File Formats

  • CSV,

JSON, XML, Parquet, and Delta Lake.

  • 4–5

years of relevant experience in Data Engineering.

  • Strong

hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL.

  • Experience

developing and optimizing scalable data pipelines.

  • Good

understanding of cloud-based data engineering and data warehousing concepts.

  • Strong

analytical, debugging, and problem-solving skills.

  • Good

communication and collaboration skills.

Similar roles