Principal Data Engineer

Role: Principal Data Engineer

Location: London Onsite Full Time

Start Date: ASAP

End Date: 12 months

Daily Rate Inside IR35

Payroll provider – Rockford Payroll Info for Contingent Workers – Rockford Pay

Assignment Overview

Deloitte sets the requirements for the assignment. Rockford Resourcing acts as the agency on behalf of Deloitte and is responsible for sourcing candidates to fulfil the assignment. Successful candidates will be employed by Rockford Payroll. Rockford Payroll is responsible for the Associate’s employment, pay and applicable employment-related processes.

The Role

We are seeking a visionary Principal Data Engineer to lead the delivery of innovative data analytics solutions, with a focus on technical excellence, governance, and operational performance in a high-volume data environment. This role involves strategic and technical leadership in building Azure-based data lake solutions, driving maturity in data management, and engaging with cross-functional teams to leverage data for organizational success.

Key Responsibilities

• Architect, design, and implement

scalable Azure-based data lake solutions.

• Establish roadmaps for

capability-building across technologies such as Qlik Replicate, Databricks Delta Live Tables, and Unity Catalog.

• Provide technical leadership,

ensuring adherence to best practices and quality standards.

• Lead the development of data

transformation pipelines using Databricks Delta Live Tables and Structured Streaming.

• Optimize Databricks resources for

high-volume data processing without exceeding budget.

• Oversee the use of Databricks Unity

Catalog for data governance, lineage, and metadata management.

• Develop and enforce robust data

governance policies, ensuring compliance, security, and data quality.

• Monitor data pipeline performance and

proactively resolve production issues.

• Collaborate with cross-functional

teams to align project objectives with business goals.

• Translate technical challenges into

actionable insights for executive teams.

• Mentor and develop a high-performing

team skilled in Azure, Databricks, and data streaming technologies.

• Facilitate knowledge transfer

sessions to empower internal teams with cutting-edge methodologies.

Experience & Skills Required

• Bachelor’s or Master’s degree in IT

or a related discipline.

• 15–17 years of experience in the IT

industry with expertise in delivering cost-effective cloud-based data analytics solutions.

• Proven track record in building data

lake architectures and optimizing performance in big data environments.

• Certification in Azure Data

Engineering or Databricks preferred.

• Strong knowledge of Azure

technologies, including ADF, ADLS, Databricks, Stream Analytics, Delta Live Tables, and Synapse Analytics.

• Advanced proficiency in SQL and

programming languages such as Python, Scala, or Spark-SQL.

• Experience in designing and

developing scalable ETL pipelines and optimizing data workflows.

• Deep understanding of data migration

from on-premises to cloud and cloud to cloud.

• Proven ability to lead diverse teams

and manage end-to-end analytics projects.

• Exceptional communication and

collaboration skills for engaging executives and stakeholders.

• Strong sense of ownership,

accountability, adaptability, and commitment to results.

• Expertise in identifying risks and

implementing effective mitigation strategies.

Nice to Have

• Familiarity with big data tools like

Hadoop, Kafka, and Spark.

• Experience with visualization tools

such as PowerBI or QlikSense.

• Knowledge of AI/ML, NLP, and GenAI

concepts.

• Familiarity with AGILE and KANBAN

methodologies.