Data Bricks Engineer

This role is for one of the Weekday's clients

Salary range: Rs 1500000 - Rs 2700000 (ie INR 15- 27 LPA)

Min Experience: 5+ years

Location: Pune JobType: full-time

We are looking for an experienced Data Bricks Engineer with 5–13 years of experience in designing, building, and optimizing scalable data engineering solutions. The ideal candidate will have strong expertise in Azure Databricks and ETL development, with hands-on experience in building high-performance data pipelines that enable advanced analytics and business intelligence.

In this role, you will collaborate closely with data architects, analysts, software engineers, and business stakeholders to develop robust data platforms capable of processing large-scale structured and unstructured datasets. You will be responsible for ensuring data quality, reliability, scalability, and performance while implementing modern cloud-based data engineering best practices.

Key Responsibilities• Design, develop, and maintain scalable data pipelines using Azure Databricks.

• Build efficient ETL workflows for extracting, transforming, and loading data from multiple enterprise data sources.
• Develop batch and incremental data processing solutions to support reporting, analytics, and machine learning workloads.
• Optimize Databricks jobs, notebooks, and Spark applications for maximum performance and cost efficiency.
• Implement data validation, cleansing, and transformation processes to ensure high-quality datasets.
• Work with business stakeholders to understand data requirements and translate them into technical solutions.
• Integrate data from databases, APIs, cloud storage, and third-party applications into centralized data platforms.
• Monitor data pipelines, troubleshoot failures, and proactively resolve performance bottlenecks.
• Collaborate with cross-functional teams to implement secure, scalable, and reliable cloud data architectures.
• Participate in code reviews, technical discussions, and knowledge-sharing sessions to promote engineering best practices.
• Ensure adherence to governance, security, and compliance standards across data engineering processes.
• Maintain technical documentation for data pipelines, workflows, and system architecture.

Required SkillsMust-Have Skills• Strong hands-on experience with Azure Databricks for building and managing scalable data engineering solutions.

• Extensive experience designing, developing, and optimizing ETL pipelines for large-scale enterprise data environments.
• Proficiency in Apache Spark, Spark SQL, and distributed data processing concepts.
• Strong programming experience in Python or Scala for data engineering tasks.
• Experience working with SQL and relational databases.
• Knowledge of data modeling, data warehousing, and dimensional modeling concepts.
• Familiarity with Azure Data Lake Storage, Azure Blob Storage, or similar cloud storage solutions.
• Experience handling large datasets while ensuring performance optimization and reliability.
• Strong understanding of data quality, validation, monitoring, and pipeline orchestration.

Good to Have• Experience with Azure Data Factory or other orchestration tools.

• Exposure to Delta Lake, Lakehouse architecture, and modern cloud data platforms.
• Knowledge of CI/CD practices and version control systems such as Git.
• Familiarity with Power BI or other business intelligence platforms.
• Experience supporting machine learning or advanced analytics workloads.
• Exposure to DevOps practices and Infrastructure as Code.

Qualifications• Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related discipline.

• 5–13 years of professional experience in data engineering or cloud-based data platform development.
• Proven expertise in Azure cloud technologies with a strong focus on Azure Databricks and ETL development.
• Excellent analytical, troubleshooting, and problem-solving abilities.
• Strong communication skills with the ability to collaborate effectively across technical and business teams.