Lead Data Engineer AI

Description & Requirements

On behalf of the Cabinet Office, we are looking for a Lead Data Engineer AI (Inside IR35) for a 6 Months contract based REMOTELY.

The Cabinet Office supports the Prime Minister and ensures the effective running of government. The Cabinet Office is also the corporate headquarters for the government, in partnership with HM Treasury, and takes the lead in certain critical policy areas. More information about the Cabinet Office can be found on its website at: www.gov.uk/government/organisations/cabinet-office

As a Lead Data Engineer AI, your main responsibilities will be:

• AI Data Integration & Pipeline Design: Establish Cabinet Office data integration standards and lead the engineering of automated data pipelines, ETL/ELT workflows, and API connections connecting enterprise SaaS and internal databases to AI applications whilst ensuring alignment with existing pipelines and standards.
• AI Data Security & Compliance: Implement identity and access controls, including SSO, conduct data quality, sensitivity and access checks before indexing, and ensure AI applications respect source-system permissions to prevent data leakage.
• AI Data Innovation & Diagnostics: Conduct early-stage data quality diagnostics to test dataset viability for AI automation, configure synthetic test datasets within isolated sandboxes, and structure domain-specific knowledge engines.
• Data Development Process: Establish repeatable, secure, and maintainable data solutions, managing metadata repositories and configuring observability pipelines across Cabinet Office AI workflows, reusing existing repositories and observability pipelines where possible.
• Technical & Stakeholder Communication: Act as the key bridge between technical engineering partners and Cabinet Office business leads, interpreting data requirements and translating complex data risks into clear choices for stakeholders

Essential:

• Technical Mastery & Engineering Practice: Hands-on expertise in building scalable ETL/ELT pipelines, designing robust API integrations, and configuring vector indices or retrieval systems for large-scale enterprise AI architectures.
• Data Governance, Security & Compliance: Expert knowledge of Identity and Access Management, identity federation, data classification standards, and UK GDPR/DPA compliance frameworks within AI environments.
• Analytical Proficiency & Data Diagnostics: Substantial experience conducting quantitative data quality assessments, structuring synthetic testing environments, and
• Evaluating dataset readiness for complex automation or ML models.
• Languages & Software Engineering Standards: Advanced proficiency in AWS, Python, SQL, and query optimisation, alongside demonstrated experience implementing CI/CD pipelines using Terraform, automated testing, and metadata management best practices.
• Strategic Communication & Leadership: Proven ability to translate complex technical data risks into strategic guidance for senior leadership, alongside a track record of leading cross-functional engineering teams in agile environments

Please be aware that this role can only be worked within the UK and not Overseas.

Disability Confident

As a Disability Confident employer, the Cabinet Office encourage applications from disabled and neurodivergent candidates. Applications from candidates who identify as having a disability and meet the essential criteria for the role will be prioritised for review. Where application volumes are high, additional role-specific and desirable criteria may be applied as part of the shortlisting process which may include holding active security clearance.

Armed Forces Covenant

As a signatory of the Armed Forces Covenant, the Cabinet Office welcome applications from veterans, service leavers, reservists and military spouses or partners. Applications from eligible candidates who meet the essential criteria for the role will be prioritised for review. Where application volumes are high, additional role-specific and desirable criteria may be applied as part of the shortlisting process which may include holding active security clearance.

In applying for this role, you acknowledge the following "this role falls in scope of the Off Payroll Working in the Public Sector legislation. Any rates of payment quoted will reflect the gross rate per day for the assignment and will be subject to appropriate taxes and statutory costs. As such the payment to the intermediary and your income resulting from this contract will be different".

General information

Posting ID

PSR1JP00106828

Type of employment

Contract

Organisation

Cabinet Office

Contract duration

6 months

Location

London

Workplace Type

Remote