Sr. Databricks Data Engineer

Halvik Corp delivers a wide range of services to 13 executive agencies and 15 independent agencies. Halvik is a highly successful WOB business with more than 50 prime contracts and 500+ professionals delivering Digital Services, Advanced Analytics, Artificial Intelligence/Machine Learning, Cyber Security and Cutting-Edge Technology across the US Government. Be a part of something special!

Position Overview

The Senior Databricks Data Engineer / Technical Lead will support the U.S. Department of Transportation (DOT) data modernization effort by designing, building, migrating, and operating secure, scalable data pipelines in a Databricks Lakehouse. This is a highly hands-on role focused on legacy and external-source migration, metadata-driven ingestion, Medallion Architecture, data quality and reconciliation, production operations, and transition to the AWS/OneDOT environment.

Core Responsibilities

• Strategic Execution: Lead the design and implementation of reusable ingestion and transformation patterns for legacy and external data sources into Databricks, including support for migration to the AWS/OneDOT environment.

• Operational Oversight: Build and operate Bronze, Silver, and Gold pipelines; manage batch and incremental loads, orchestration, dependencies, retries/recovery, monitoring, alerting, and production support.

• Collaboration: Partner with the Technical Project Manager, Data Architects, business/data SMEs, BI developers, Cloud/DevSecOps, security, and governance teams to translate requirements into implementable data solutions.

• Technical Performance: Develop and optimize Python, PySpark, and SQL workloads using Databricks, Delta Lake, Unity Catalog, Auto Loader and/or Lakeflow capabilities, and Databricks Jobs/Workflows with a focus on reliability, performance, and cost.

• Data Quality & Governance: Implement source-to-target reconciliation, data validation, schema evolution, lineage, access controls, documentation, and reusable engineering standards for production data pipelines.

Minimum Requirements

• Education: Bachelor's degree in computer science, Data Engineering, Information Technology, Engineering, or a related field; equivalent relevant experience may be considered.

• Experience: Minimum of 7 years of professional data engineering experience, including significant hands-on experience with Databricks/Spark and production data pipelines.

• Technical Proficiency: Advanced Python, PySpark, and SQL skills; strong experience with Databricks, Delta Lake, AWS/S3, batch/file ingestion (CSV/JSON), REST APIs, Git/CI/CD, and data pipeline orchestration.

• Compliance: Must meet applicable DOT contract security and suitability requirements and demonstrate secure handling of PII and other sensitive data, including access controls and secrets management.

• Certifications: Relevant Databricks and/or AWS data engineering certification preferred; equivalent demonstrated hands-on experience will be considered.

Preferred Expertise

• Demonstrated success modernizing legacy data platforms (Sybase/SAP IQ or similar) into Databricks, including source-to-target reconciliation, cutover, and production stabilization.

• Experience with Unity Catalog, metadata/configuration-driven ingestion, Auto Loader/Lakeflow, Databricks Jobs/Workflows, lineage/cataloging tools, and scalable multi-source ingestion frameworks.

• Federal/DOT delivery experience; familiarity with Agile/Kanban, production O&M, cloud cost optimization, and data governance. Streaming and AI/ML data pipeline exposure is a plus.

Halvik offers a competitive full benefits package including:

Company-supported medical, dental, vision, life, STD, and LTD insurance

Benefits include 11 federal holidays and PTO

Eligible employees may receive performance-based incentives in recognition of individual and/or team achievements.

401(k) with company matching

Flexible Spending Accounts for commuter, medical, and dependent care expenses

Tuition Assistance

Charitable Contribution matching

Halvik Corp is an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or veteran status.

Halvik's pay range for this job level is a general guideline only and not a guarantee of compensation or salary. Additional factors considered in extending an offer include (but are not limited to) responsibilities of the job, education, experience, knowledge, skills, and abilities, as well as internal equity, alignment with market data, applicable bargaining agreement (if any), or other law.