M24 - Senior Data Engineer

Overview

As a Senior Data Engineer, you own complex assignments that support the different stages of the data lifecycle. You design moderately complex data systems and platforms, develop and implement data products and components, set up their supporting infrastructure, and integrate systems. You analyse and break down complex problems into actionable tasks for team members, and coordinate efforts across multiple teams.

You are deeply proficient in the skills required to develop data products and components supporting different stages of the data lifecycle and their underlying infrastructure. You solve complex tasks independently with minimal guidance from more senior engineers, and drive high-quality delivery and scalability by championing DataOps best practices. You guide junior engineers, mentor less experienced peers, and influence team processes to enhance productivity, quality, and reliability. 

How We Work

We are a team of passionate individuals committed to solving challenges and driving meaningful change. We embrace diversity of thought, encourage experimentation, and foster a culture of continuous learning. Three principles guide how we operate:

• Decision-making: We identify overarching goals, weigh options, assess impact, and clearly communicate the risks and trade-offs of our decisions.

• Ownership: We have opinions on what is being built and ideas on what should come next. Building something you believe in is the best way to build something good.

• Continuous Learning: Working on new ideas often means operating at the edge of our knowledge. Learning new architectures, frameworks, technologies, and languages is not just encouraged — it is essential.

Responsibilities

Craft and Execution

• Design the architecture of moderately complex data systems and platforms, with deliberate consideration of scalability, reliability, and long-term maintainability.

• Proactively refactor and reduce technical debt to ensure long-term system stability.

• Implement advanced DataOps practices including automated pipeline deployments, workflow monitoring, and data observability.

• Apply advanced ETL/ELT techniques to make data useful for complex use cases.

• Utilise advanced data modelling techniques to accurately represent complex business processes.

• Apply advanced data integration techniques such as data streaming, CDC, and message queues when building data pipelines and components.

• Navigate data privacy issues, governance, and regulatory requirements effectively, ensuring system design remains compliant with policies and standards.

• Develop comprehensive plans to achieve key milestones, effectively breaking down, defining, and prioritising tasks.

• Communicate and collaborate effectively with immediate team members and stakeholders to ensure aligned and coordinated delivery.

Ownership

• Take full ownership of broad, ambiguously-scoped projects and drive them to successful outcomes.

• Take calculated risks, treating both successes and failures as opportunities for learning and growth.

• Mentor and support junior team members through knowledge sharing, problem decomposition, and constructive feedback.

• Actively raise team productivity and capability through hands-on guidance and support.

Strategic Alignment

• Translate team goals into actionable plans, breaking down work and prioritising tasks effectively.

• Proactively identify opportunities, drive workstreams, and synthesise data into clear recommendations connected to organisational impact.

• Identify and mitigate risks at the project level, anticipating potential challenges before they materialise.

Culture and Organisational Influence

• Coordinate cross-functional collaboration and guide team members to ensure projects are delivered effectively.

• Navigate and help resolve disagreements constructively, facilitating alignment among team members and stakeholders.

• Constructively challenge existing processes to drive continuous improvement and change initiatives at the team and division level.

• Actively share successes and failures, offering recommendations that enhance team performance and foster a culture of learning.

Requirements

Experience and Education

• Bachelor's degree or higher in Data Science, Computer Science, Statistics, Applied Mathematics, or a related quantitative discipline.

• At least 5 years of experience in data engineering.

• Proven experience delivering data platforms or products in large-scale, enterprise environments.

Technical Skills

• Strong expertise in data modelling, including OLTP, OLAP, and dimensional modelling.

• Experience with cloud data platforms and tools such as AWS, Azure Synapse/Microsoft Fabric, Snowflake, Databricks, Redshift, and Data Lakes.

• Proficiency in big data technologies such as Hadoop, Spark, Kafka, and Flink.

• Strong programming skills in Python, Scala, or Java.

• Experience with CI/CD and DataOps practices, including SHIP-HATS or equivalent.

• Expertise in data integration techniques including ETL/ELT, streaming, and APIs.

AI and Machine Learning (Advantageous)

• Understanding of machine learning and LLM concepts, including prompt engineering and AI application development.

• Familiarity with MLOps practices and cloud-based ML deployment.

• Awareness of responsible AI principles covering fairness, robustness, and safety.

Core Competencies

• Strong analytical and problem-solving skills, with the ability to translate business problems into data solutions.

• Ability to design scalable, reliable, and maintainable data systems.

• Effective communication and stakeholder management skills.

• Self-driven, adaptable, and able to operate in ambiguity.

• Strong sense of ownership and commitment to delivery excellence.

• Passion for leveraging data and technology to deliver public good.