JobCubby JobCubby
Sign up
Unison Group

Sr Data Engineer (Databricks)

Unison Group Bengaluru, Karnataka, India

Business Consulting and Services · 11-50 employees

Oct 02
data-engineer Senior (5-10 yrs) Full-time India
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.

About the role

Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for complex data and AI initiatives. Implement end-to-end ETL/ELT processes, ensure data quality, and mentor team members on best practices for high-volume, petabyte-scale workloads.

What they look for

Azure Databricks PySpark Python SQL Scala Delta Lake Azure DevOps CI/CD Data Engineering Distributed Systems Data Architecture Unity Catalog Structured Streaming Bicep Cloud Computing

Requirements

Requires 7-8+ years of experience in data engineering with deep expertise in Azure Databricks and distributed systems. Proficiency in Python, PySpark, SQL, and Scala is essential, along with a proven track record in cloud-native solutions and CI/CD practices.

Full description

Position Overview:

  • Lead the design, architecture, and optimization of enterprise-scale data pipelines using Azure Databricks for ACDP projects, driving innovation in DATA & AI initiatives.
  • Leverage 7-8 years of expertise to mentor teams, ensure scalability, and deliver mission-critical data solutions.

Key Responsibilities:

  • Architect and implement end-to-end data pipelines with Azure Databricks (notebooks, jobs, clusters) for ingestion, ETL/ELT, and complex transformations, incorporating Medallion Architecture (Bronze, Silver, Gold) using Delta Lake for ACID compliance, versioning, schema evolution, and data quality.
  • Provide hands-on leadership in Databricks Spark (PySpark, Scala, SQL), Delta Live Tables, Structured Streaming, and Unity Catalog for advanced governance, security, and multi-workspace management.
  • Integrate with Azure ecosystem including ADLS Gen2, Azure SQL Database, Serverless/Dedicated SQL Pools; orchestrate via Databricks workflows and hybrid ADF triggers for high-volume ACDP workloads.
  • Design and deploy CI/CD pipelines using Azure DevOps, Bicep for infrastructure-as code, and advanced automation for Databricks workspaces, emphasizing zero downtime deployments and cost optimization.
  • Champion data quality, validation, cleansing with Databricks expectations/constraints, and compliance via RBAC, encryption, and Purview; mentor on best practices for Photon engine, auto-scaling, spot instances, and performance tuning
  • Collaborate with senior analysts, data scientists, and executives to translate complex business requirements into reusable Databricks components, fostering  AI/ML integration.
  • Drive monitoring, troubleshooting, and optimization for petabyte-scale processing,  including unstructured data handling, data modeling, and streaming from Kafka/Event Hubs

Required Qualifications:

  • 7-8+ years in data engineering, with deep Azure Databricks expertise for big data processing in high-demand environments
  • Advanced proficiency in Python/PySpark, SQL, Scala; strong data architecture and distributed systems experience
  • Proven track record in cloud-native solutions (AZ-900+ preferred), CI/CD, and mentoring junior engineers
  • Excellent communication for stakeholder alignment and documentation

Preferred: Power BI, Power Automate, Azure ML, enterprise governance tools.

Similar roles