Senior Data Engineer
InFynd Coimbatore North, Tamil Nadu, India
Information Services · 51-200 employees
Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.
About the role
Design, develop, and maintain scalable ETL/ELT data pipelines using Azure technologies and distributed processing frameworks. Manage data ingestion, transformation, and workflow orchestration while ensuring optimal performance and production support.
What they look for
Requirements
Requires 4-5 years of relevant experience in data engineering with strong proficiency in Python, SQL, and PySpark. Candidates must have hands-on experience with Azure cloud services and data warehousing methodologies.
Full description
- Design,
develop, and maintain scalable ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
- Develop
data ingestion and transformation workflows using Python, SQL, and PySpark.
- Work
with Apache Spark and PySpark for large-scale distributed data processing.
- Implement
batch processing, incremental loading, full loading, and Change Data Capture (CDC) mechanisms.
- Integrate
data from multiple sources, including relational databases, APIs, and cloud storage.
- Work
with Azure Data Lake Storage Gen2 and Azure Blob Storage.
- Develop
and optimize data models, fact tables, dimension tables, and data warehouse structures.
- Perform
data cleansing, validation, deduplication, and aggregation.
- Implement
and manage workflows using Azure Data Factory and Apache Airflow.
- Optimize
SQL queries, Spark jobs, and data pipelines for better performance.
- Work
with Azure Synapse Analytics and relational databases.
- Implement
error handling, logging, monitoring, and alerting for production pipelines.
- Manage
version control and CI/CD pipelines using Git, GitHub, and Azure DevOps.
- Troubleshoot
production issues, perform root cause analysis, and provide production support.
- Maintain
technical documentation and collaborate with cross-functional teams.
Requirements
Programming Languages
- Python,
SQL, PySpark.
- Pandas,
NumPy, OOP concepts, exception handling, and API integration.
Big Data Technologies
- Apache
Spark and PySpark.
- Spark
DataFrames, Spark SQL, transformations, actions, joins, aggregations, and window functions.
- Partitioning,
caching, and performance optimization.
Azure Cloud Technologies
- Azure
Data Factory (ADF).
- Azure
Databricks.
- Azure
Data Lake Storage Gen2 (ADLS Gen2).
- Azure
Blob Storage.
- Azure
Synapse Analytics.
- Azure
Key Vault and Azure Monitor.
ETL & Data Engineering
- ETL/ELT
pipeline development.
- Data
ingestion, transformation, and migration.
- Change
Data Capture (CDC).
- Incremental
loading and full loading.
- Batch
and distributed processing.
Database Technologies
- Microsoft
SQL Server, PostgreSQL, MySQL, and Oracle.
- Complex
SQL queries, indexing, stored procedures, and query optimization.
Data Modeling & Warehousing
- Dimensional
and relational data modeling.
- Fact
and dimension tables.
- Data
normalization and denormalization.
- Schema
design and data warehouse development.
Workflow Orchestration & DevOps
- Azure
Data Factory and Apache Airflow.
- Git,
GitHub, Azure DevOps, and CI/CD pipelines.
File Formats
- CSV,
JSON, XML, Parquet, and Delta Lake.
- 4–5
years of relevant experience in Data Engineering.
- Strong
hands-on experience with Azure Data Factory, Azure Databricks, PySpark, Python, and SQL.
- Experience
developing and optimizing scalable data pipelines.
- Good
understanding of cloud-based data engineering and data warehousing concepts.
- Strong
analytical, debugging, and problem-solving skills.
- Good
communication and collaboration skills.
Similar roles
-
Senior Data Engineer II
Oscar Health New York, New York, United States · $181K–$237K/yr
-
Data Engineer
AcuityMD $113K–$159K/yr
-
Staff Data Engineer, Analytics
Afresh San Francisco, California, United States · $168K–$250K/yr
-
Data Engineer - Agentic AI
Defense Unicorns Springfield, Virginia, United States · $149K–$201K/yr
-
Senior Data Engineer
Southend Pharmacy Houston, Texas, United States · €45K–€65K/yr
-
Senior Data Engineer (Talent Pool)
ioet San José de Alluriquín, Santo Domingo de los Tsáchilas, Ecuador