JobCubby JobCubby
Sign up
CTI Consulting

Data Engineer III | 1177

CTI Consulting Quincy, Massachusetts, United States

Public Safety · 11-50 employees

Yesterday
data-engineer Senior (5-10 yrs) Contractor United States
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.

About the role

Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark to support enterprise data needs. Collaborate with an Agile team to develop scalable data solutions and implement robust data quality rules.

What they look for

Databricks PySpark SQL Azure Delta Lake Kafka ETL ELT Change Data Capture Data Modeling CI/CD GitHub Python Unity Catalog REST APIs Data Engineering

Requirements

Requires extensive hands-on experience with Databricks, PySpark, and SQL in an Azure environment. Candidates must have proven experience building end-to-end data pipelines with Change Data Capture and data modeling techniques.

Full description

About This Opportunity

CTI Staffing is partnering with a leading retail organization to find a Senior Databricks Data Engineer for their enterprise data team based in the Greater Boston, MA area.

The team is building out a modern Azure data platform that brings together data from many retail systems. This role matters because reliable, incremental pipelines are what let the business trust its data and act on it faster.

What You’ll Do

  • Design, build, and optimize ETL/ELT pipelines using Databricks and PySpark.
  • Build Change Data Capture pipelines for incremental and near real-time processing.
  • Develop scalable data solutions on Azure using Delta Lake and Kafka streaming.
  • Improve existing pipelines by quickly learning current codebases and delivering targeted upgrades.
  • Ingest and process data from flat files, XML, JSON, SQL Server, JDBC sources, APIs, message queues, and Kafka topics.
  • Implement data quality rules and quarantine error records.
  • Apply data modeling techniques including SCD Type 1 and Type 2.
  • Automate workflows with Databricks Workflows and GitHub-based CI/CD, and work with developers, architects, and product owners in an Agile team.

Requirements

What You Bring

Must-Have:

  • Extensive hands-on Databricks experience, able to contribute from day one.
  • Proven experience building end-to-end data pipelines with CDC.
  • Strong PySpark and SQL skills.
  • Experience working in Microsoft Azure.
  • Working knowledge of Delta Lake and distributed data processing.
  • Experience with Kafka or comparable streaming technologies.
  • Hands-on experience with SCD Type 1 and Type 2 data modeling.
  • Familiarity with GitHub and CI/CD for version control and deployment.

Nice-to-Have:

  • Python experience beyond PySpark.
  • Databricks Workflows and Unity Catalog experience.
  • Experience with message queues and API-based ingestion.
  • Retail, CPG, or supply chain data background.
  • Databricks certification.

Technical Environment: Azure, Databricks, PySpark, SQL, Delta Lake, Kafka, Unity Catalog, GitHub, CI/CD, SQL Server, JSON, XML, REST APIs.

Similar roles