Principal Data Engineer (Databricks)
Sabenza IT & Recruitment City of Cape Town, Western Cape, South Africa
IT Services and IT Consulting · 11-50 employees
Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.
About the role
The Principal Data Engineer will design and build production-grade data pipelines on Databricks while optimizing platform performance and costs. They will also provide technical leadership, mentor junior engineers, and act as a key interface between business stakeholders and technical teams.
What they look for
Requirements
Candidates must have at least three years of hands-on Databricks experience and strong proficiency in Python, SQL, and Spark. A relevant tertiary qualification in Computer Science, Information Systems, or Engineering is required, along with a track record of managing stakeholders and coaching team members.
Full description
Reporting to the Data & Analytics Lead, the Principal Data Engineer is a hands-on role, combining the building and optimization of production data pipelines with the technical leadership needed to raise engineering standards and develop the people around them.
The role functions as a key interface between business stakeholders and department heads, the wider data and analytics team, and the owners of source systems and data in Finance and IT.
Requirements
KEY RESPONSIBILITIES:
- Design and build robust, production-grade data pipelines on Databricks, making appropriate use of current platform capabilities such as Spark Declarative Pipelines (SDP).
- Work “under the hood” to optimise the platform for performance and cost tuning compute, jobs and queries,managing storage and table layout, and keeping platform spend under control.
- Define and maintain consistent business metrics and a semantic layer (for example, using Metric Views) so that reporting is built on trusted, reusable definitions.
- Integrate data from core business systems, including SAP S/4HANA, and other sources, using tools such as Fivetran (both SaaS connectors and HVR).
- Set and uphold engineering standards across the team, including code quality, testing, documentation, CI/CD and data governance.
- Coach and mentor junior and mid-level engineers, review their work and help them develop.
- Partner with business stakeholders to understand their needs, shape practical solutions and ensure the platform delivers genuine value.
- Engage data owners in Finance and IT to ensure data is well understood, valid and fit for purpose, and initiate data quality improvements where required.
REQUIRED SKILLS AND EXPERIENCE:
- At least three years of hands-on, day-to-day Databricks experience, covering both pipeline development and under-the-hood performance and cost optimisation.
- Up to date with recent platform developments, such as Spark Declarative Pipelines (SDP) and Metric Views.
- Strong proficiency in programming languages commonly used in data engineering, such as Python, SQL and Spark.
- Advanced experience with data manipulation, data modelling, database design and query optimisation.
- Experience managing or coaching junior developers.
- A track record of managing and influencing business stakeholders.
KEY COMPETENCIES:
- Combining deep, hands-on engineering skill with sound judgement about cost, performance and long-term maintainability.
- Coaching, mentoring and raising the capability of less experienced engineers.
- Collaborating, communicating confidently and influencing business stakeholders.
- Breaking down complex technical concepts and explaining them simply to non-technical audiences.
- Staying current with a fast-moving platform and bringing new capabilities into everyday practice.
- Taking ownership and driving work independently, from concept through to production.
NICE TO HAVE:
- Experience working with SAP S/4HANA data sets.
- Experience with Fivetran, including both SaaS connectors and HVR.
- Power BI semantic modelling experience.
QUALIFICATIONS:
- A relevant tertiary qualification in Computer Science, Information Systems, Engineering or a related field, or equivalent practical experience.
Similar roles
-
Experienced - Data Engineer DATABRICKS - NextHub Bari
Deloitte Italia Bari, Apulia, Italy · €28K–€44K/yr
-
Senior Data Engineer - Databricks Specialist
University of Otago Otago, New Zealand
-
Senior Data Engineer
UBDS Group Manchester, England, United Kingdom
-
Data Engineer
UBDS Group London, England, United Kingdom · £40K–£48K/yr
-
Data Engineer II, AWS Analytics Engineering
Amazon Seattle, Washington, United States · $132K–$179K/yr
-
Data Engineer Intern
Munich Re Singapore, Singapore, Singapore