Securiport

Senior Databricks Data Engineer

Securiport Fairfax County, Virginia, United States · $170K–$210K/yr

Security Systems Services · 201-500 employees

Nov 06
data-engineer Senior (5-10 yrs) Full-time United States
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

About the role

The Senior Databricks Data Engineer will lead the design, development, and optimization of data pipelines and platforms within a Databricks-on-AWS environment. This role involves building scalable, secure, and governed data solutions to support analytics, reporting, and AI initiatives.

What they look for

Databricks AWS PySpark Spark SQL Delta Lake Auto Loader Medallion Architecture Unity Catalog Data Engineering Python SQL CI/CD Git Data Governance Data Pipelines Cloud Architecture

Requirements

Candidates must have 5+ years of experience in Data Engineering with at least 2 years of hands-on Databricks production experience. Proficiency in PySpark, Delta Lake, Unity Catalog, and AWS infrastructure is required, along with a bachelor's degree in a relevant field.

Benefits

Competitive compensation Comprehensive benefits

Full description

KIBO is a registered United States company dedicated to providing governments worldwide its services of inspection, analysis, certification, and compliance. The Senior Databricks Data Engineer will lead the design, development, and optimization of data pipelines and data platforms within a Databricks-on-AWS environment.

Working closely with the Principal AI Architect, this role will build scalable, secure, and governed data solutions that support analytics, reporting, and AI initiatives. The ideal candidate brings deep expertise in Databricks, cloud-based data engineering, and modern data architecture. 

Key Responsibilities

  • Data platform development. Configure, manage, and optimize Databricks workspaces and supporting AWS infrastructure.
  • Data pipeline engineering. Design and build scalable batch and streaming data pipelines using PySpark, Spark SQL, Delta Lake, and Auto Loader.
  • Data architecture. Implement and maintain Medallion Architecture (Bronze, Silver, Gold) and analytics-ready data models.
  • Data governance & security. Configure Unity Catalog, permissions, row-level security, column masking, and data-access controls.
  • Data integration. Ingest data from third-party APIs, file transfers, and operational systems using modern Databricks patterns.
  • CI/CD & quality. Implement Git-based development workflows, automated testing, and deployment processes across environments.
  • Agile delivery. Collaborate with cross-functional teams in an Agile environment to deliver reliable, high-quality data solutions

Required Qualifications

  • 5+ years of experience in Data Engineering, including 2+ years of hands-on Databricks experience in production.
  • Strong experience with Databricks Lakehouse Platform, Delta Lake, Unity Catalog, and PySpark.
  • Advanced SQL and Python development skills.
  • Experience implementing Medallion Architecture and data models for analytics and reporting.
  • Experience with Delta Live Tables, change data capture (CDC), incremental processing, and data-quality controls.
  • Working knowledge of AWS services including S3, IAM, storage credentials, and external locations.
  • Experience with Git, CI/CD pipelines, and automated testing practices.
  • Bachelor's degree in Computer Science, Information Systems, Engineering, or equivalent experience.

Preferred Qualifications

  • Databricks Certified Data Engineer Associate or Professional certification.
  • Experience with Spark performance tuning and cost optimization.
  • Software engineering background with API development experience.
  • Experience developing and testing locally using PySpark, Delta Lake, and pytest.
  • Experience supporting AI/ML, financial, trade, customs, or other regulated-data environments.

What Success Looks Like in Your First Few Months

  • Establish and secure the Databricks environment and governance framework.
  • Deliver initial ingestion pipelines and curated Bronze, Silver, and Gold data layers.
  • Implement CI/CD processes and data-quality monitoring standards.
  • Support the transition from batch processing to near real-time incremental data ingestion.

Reporting and Working Relationships Reports to the Principal AI Architect and works closely with data, AI, analytics, and engineering stakeholders across the organization.

What KIBO Offers

  • Competitive compensation and comprehensive benefits.
  • Opportunity to build a modern Databricks data platform from the ground up.
  • Exposure to cutting-edge data engineering and AI initiatives.
  • A collaborative, mission-driven environment with significant ownership and impact.

Join Us If you are passionate about building scalable cloud data platforms and want to help shape the future of data and AI solutions, we encourage you to apply.

KIBO Laboratories LLC is an equal-opportunity employer and welcomes qualified applicants of all backgrounds. Offers are contingent upon background and right-to-work verification where applicable.

Similar roles