Senior Technical Lead – Azure Data Engineer

About USEREADY

USEReady helps enterprises apply AI and agentic intelligence to improve decisions, automate operations, and build smarter, more autonomous business systems.

For more than a decade, we have built the foundations that make this possible by modernizing BI environments, migrating legacy platforms, improving data quality, and enabling governed, cloud-first architectures. These foundations now support the next step: AI-driven insights, automated intelligence, and agent-powered decision support that reduce complexity and accelerate outcomes.

We work closely with technology leaders such as AWS, Elementum, Snowflake, Tableau, Databricks, and others to help organizations modernize analytics, strengthen governance, and deploy agentic automation with confidence. We founded in 2011 and Headquartered in New York City with 450+ experts across the United States, Canada, India, and Singapore, we serve industries including financial services, healthcare, manufacturing, government, education, and retail. Our deep expertise, player-coach delivery model, and focus on fast, measurable results make us a trusted partner for building an AI-ready enterprise.

Job Summary

We are looking for a Senior Technical Lead – Azure Data Engineer with 8–10 years of experience and strong hands-on expertise in Microsoft Azure, Databricks, ADF, Python, PySpark, and SQL.

The ideal candidate will have strong experience designing and implementing scalable data engineering solutions, architecting Databricks environments, optimizing data pipelines and workloads, and implementing data governance and security using Unity Catalog.

The candidate should be comfortable working with large-scale distributed data platforms, mentoring engineering teams, and collaborating with business and technical stakeholders to deliver high-quality data solutions.

Key Responsibilities

• Architect and design scalable Azure Databricks environments and data engineering solutions.

• Design and implement robust, scalable, and optimized data pipelines using Databricks and Azure Data Factory (ADF).

• Develop Databricks notebooks using Python, PySpark, and SQL.

• Implement and manage Unity Catalog for data governance, security, access control, and data discovery.

• Design data models and define schema strategies, data structures, and pipeline organization.

• Build and maintain Delta Lake tables and data models supporting analytics and BI workloads.

• Optimize Databricks jobs, Spark workloads, SQL queries, and data pipelines for performance, scalability, and cost efficiency.

• Apply knowledge of Apache Spark internals to troubleshoot and tune distributed data processing workloads.

• Develop complex and optimized SQL queries, including efficient joins and processing of large datasets.

• Implement data quality, monitoring, validation, and reconciliation mechanisms.

• Work with Azure data services including Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL Database.

• Implement data governance, security, access controls, and data lineage.

• Troubleshoot pipeline failures and performance issues and ensure reliability and uptime.

• Collaborate with stakeholders to understand business requirements and translate them into scalable data solutions.

• Identify opportunities to automate manual processes and improve overall data platform efficiency.

• Mentor junior and mid-level engineers and establish best practices across Azure and Databricks development.

Mandatory Skills

• 8–10 years of overall experience in Data Engineering, with strong focus on Azure.

• Strong hands-on experience with Azure Databricks.

• Excellent experience with Python and PySpark.

• Strong knowledge of Apache Spark architecture, internals, and performance tuning.

• Strong SQL skills, including complex queries, query optimization, and large-scale data processing.

• Strong experience with Azure Data Factory (ADF).

• Hands-on experience with Databricks Notebooks using Python and SQL.

• Experience with Unity Catalog, data governance, security, and access control.

• Strong understanding of Delta Lake.

• Experience with Azure Data Lake Storage (ADLS).

• Good understanding of data warehousing and data modeling, including Star and Snowflake schemas.

• Experience working with large-scale distributed data processing environments.

• Strong troubleshooting and performance optimization skills.

Good to Have

• Experience with Azure Synapse Analytics.

• Experience with Azure SQL Database.

• Databricks or Azure Data Engineering certifications.

• Experience implementing enterprise-level data governance and data lineage.

• Experience mentoring technical teams and driving engineering best practices.

• Strong stakeholder management and communication skills.