Sabenza IT & Recruitment

Principal Data Engineer (Databricks)

Sabenza IT & Recruitment City of Cape Town, Western Cape, South Africa

IT Services and IT Consulting · 11-50 employees

Nov 10
data-engineer Principal (10+ yrs) Full-time South Africa
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

About the role

The Principal Data Engineer will design and build production-grade data pipelines on Databricks while optimizing platform performance and costs. They will also provide technical leadership, mentor junior engineers, and act as a key interface between business stakeholders and technical teams.

What they look for

Databricks Python SQL Spark Data Engineering Data Modeling Database Design Query Optimization CI/CD Data Governance Mentoring Stakeholder Management SAP S/4HANA Fivetran Power BI Spark Declarative Pipelines

Requirements

Candidates must have at least three years of hands-on Databricks experience and strong proficiency in Python, SQL, and Spark. A relevant tertiary qualification in Computer Science, Information Systems, or Engineering is required, along with a track record of managing stakeholders and coaching team members.

Full description

Reporting to the Data & Analytics Lead, the Principal Data Engineer is a hands-on role, combining the building and optimization of production data pipelines with the technical leadership needed to raise engineering standards and develop the people around them.

The role functions as a key interface between business stakeholders and department heads, the wider data and analytics team, and the owners of source systems and data in Finance and IT.

Requirements

KEY RESPONSIBILITIES:

  • Design and build robust, production-grade data pipelines on Databricks, making appropriate use of current platform capabilities such as Spark Declarative Pipelines (SDP).
  • Work “under the hood” to optimise the platform for performance and cost tuning compute, jobs and queries,managing storage and table layout, and keeping platform spend under control.
  • Define and maintain consistent business metrics and a semantic layer (for example, using Metric Views) so that reporting is built on trusted, reusable definitions.
  • Integrate data from core business systems, including SAP S/4HANA, and other sources, using tools such as Fivetran (both SaaS connectors and HVR).
  • Set and uphold engineering standards across the team, including code quality, testing, documentation, CI/CD and data governance.
  • Coach and mentor junior and mid-level engineers, review their work and help them develop.
  • Partner with business stakeholders to understand their needs, shape practical solutions and ensure the platform delivers genuine value.
  • Engage data owners in Finance and IT to ensure data is well understood, valid and fit for purpose, and initiate data quality improvements where required.

REQUIRED SKILLS AND EXPERIENCE:

  • At least three years of hands-on, day-to-day Databricks experience, covering both pipeline development and under-the-hood performance and cost optimisation.
  • Up to date with recent platform developments, such as Spark Declarative Pipelines (SDP) and Metric Views.
  • Strong proficiency in programming languages commonly used in data engineering, such as Python, SQL and Spark.
  • Advanced experience with data manipulation, data modelling, database design and query optimisation.
  • Experience managing or coaching junior developers.
  • A track record of managing and influencing business stakeholders.

KEY COMPETENCIES:

  • Combining deep, hands-on engineering skill with sound judgement about cost, performance and long-term maintainability.
  • Coaching, mentoring and raising the capability of less experienced engineers.
  • Collaborating, communicating confidently and influencing business stakeholders.
  • Breaking down complex technical concepts and explaining them simply to non-technical audiences.
  • Staying current with a fast-moving platform and bringing new capabilities into everyday practice.
  • Taking ownership and driving work independently, from concept through to production.

NICE TO HAVE:

  • Experience working with SAP S/4HANA data sets.
  • Experience with Fivetran, including both SaaS connectors and HVR.
  • Power BI semantic modelling experience.

QUALIFICATIONS:

  • A relevant tertiary qualification in Computer Science, Information Systems, Engineering or a related field, or equivalent practical experience.

Similar roles