Senior Data Engineer- Databricks

Overview

The Senior Data Engineer owns substantial components of the Databricks platform, including full pipelines from ingestion through Gold-layer modeling. This role is expected to take on tasks with unresolved ambiguity, resolve open questions efficiently, and deliver without close supervision. The Senior Data Engineer identifies data and design issues before they affect downstream work and supports less experienced Data Engineers on the pod in developing the same judgment.

Responsibilities

• Design and build Databricks pipelines (Spark/PySpark, Delta Lake, Unity Catalog) across Bronze, Silver, and Gold layers for financial and operational data sets.

• Own a defined workstream (e.g., a source system or a domain such as project financials) from intake through delivery, including testing and documentation.

• Participate actively in sprint planning and daily standups, surfacing risks and clarifying requirements rather than waiting for direction.

• Review code and output from Data Engineers on the pod and provide direct, constructive feedback.

• Implement and validate row-level and column-level security controls on owned data sets.

• Support Boomi-based integration build and validation work (e.g., JDE-Concur, JDE-UKG) as the pod assumes ownership from the onshore Integration & Middleware team.

• Partner with the Lead Engineer to break down larger backlog items into task-ready work.

• Identify and raise technical debt, data quality issues, or design gaps proactively, rather than deferring to QA.

Qualifications

Required Qualifications

• 5+ years of data engineering experience, including at least 2–3 years of hands-on Databricks experience (Spark/PySpark, Delta Lake, Unity Catalog).

• Experience building pipelines from financial or ERP source systems — JD Edwards (JDE) directly preferred, with EcoSys or Microsoft Dynamics also relevant — including general ledger, cost, or project financial data.

• Strong SQL and Python skills, with the ability to own a pipeline end-to-end.

• Experience working in agile sprints with daily standups and sprint-based delivery.

• Strong written and verbal English communication skills.

Preferred Qualifications

• Databricks Certified Data Engineer Associate or Professional.

• Experience in a construction, EPC, energy, or other asset-heavy industry, particularly with cost-to-complete or earned value data.

• Experience implementing RLS/CLS security models in Unity Catalog.

• Familiarity with Databricks Genie (AI/BI) or Power BI reporting on governed Gold-layer data.

• Prior experience in a hybrid onshore/offshore delivery model.

• Hands-on experience with Boomi or a comparable iPaaS platform (pipeline build, deployment, monitoring).

• Familiarity with REST/webhook API integration patterns and file-based integration (EDI, flat file, CSV).

Professional Attributes:

• Defaults to asking questions when requirements are unclear rather than proceeding on assumptions.

• Prioritizes efficient resolution of ambiguity over continuing without clarity.

• Takes end-to-end ownership of assigned work and maintains a high quality bar without close supervision.

• Communicates clearly and directly with the Lead Engineer and, when required, the onshore team.