Sr Data Engineer

PLEASE NOTE:

• IT IS 100 % ON SITE POSITION IN Mc lean VA

KEY REQUIRED SKILLS:

• PySpark & Python for data pipeline development, Snowflake & AWS

DESCRIPITION:

We are seeking a hands-on, delivery-focused Senior Data Engineer to help build and scale our cloud data platform. In this role, you will design and develop modern data pipelines using PySpark, Snowflake, and AWS to optimize cloud data workloads. The ideal candidate combines strong engineering fundamentals with cloud-native data expertise and is capable of translating complex business needs into robust, performant, and well-documented data solutions. Experience within Fannie Mae, Freddie Mac, or equivalent GSE/mortgage enterprise environments is highly valued.

RESPONSIBILITES:

• Scalable Architecture: Design and build scalable batch and streaming data pipelines using PySpark for large-scale data processing.

• Modernization: Migrate legacy, on-premises ETL workloads (e.g., IBM DataStage, Informatica) to high-performing PySpark and Snowflake cloud pipelines.

• Data Transformation: Write production-grade PySpark code to read from Amazon S3 (Parquet/Delta files), execute complex transformations, and process massive datasets efficiently.

• Deduplication: Design and implement robust deduplication strategies for high-volume datasets using PySpark.

Platform Engineering: Build and manage Snowflake warehouses, schemas, and data models optimized for enterprise analytics and business intelligence reporting.

• Iceberg Tables: Design and implement Apache Iceberg tables in Snowflake to support open lakehouse architectures and data interoperability.

Incremental Processing: Build and maintain Snowflake Dynamic Tables and Materialized Views to enable near real-time analytics and query acceleration.

• PySpark Tuning: Optimize distributed Spark jobs by leveraging partitioning, caching, broadcast joins, and shuffle optimization.

Snowflake Optimization: Tune Snowflake workloads using clustering keys, micro-partition pruning, query profiling, precise warehouse sizing, and strategic result caching.

• Cost Management: Continuously monitor and optimize Spark jobs, Snowflake queries, and AWS infrastructure to balance speed and cloud expenditure.

• Data Quality: Implement data validation, lineage tracking, and monitoring solutions across all pipeline stages to ensure high data integrity.

Cross-Functional Collaboration: Partner closely with data architects, business analysts, Technical Program Managers (TPMs), and corporate stakeholders to deliver dependable data products.

• Technical Documentation: Author comprehensive technical designs, data schemas, and operational runbooks to ensure every pipeline is maintainable and audit-ready.

Requirements

REQUIRED QUALIFICATION:

• Experience: 6+ years of hands-on data engineering experience in large-scale enterprise environments.

• PySpark Expertise: Deep proficiency in building distributed data processing pipelines, handling S3 Parquet/Delta files, and implementing complex transformations and deduplication logic.

• Snowflake Proficiency: Strong hands-on experience with SnowSQL, Snowpipe, Streams, Tasks, and Role-Based Access Control (RBAC). Proven track record establishing Iceberg tables, Dynamic Tables, and Materialized Views.

• AWS Cloud Ecosystem: Robust working knowledge of AWS services, including S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, and Redshift.

• Advanced SQL & Python: Mastery of advanced SQL techniques (window functions, CTEs, complex joins) alongside strong Python programming skills for automation, scripting, and orchestration utilities.

• Orchestration & Architecture: Solid understanding of data warehousing, ELT/ETL patterns, data lakes, and lakehouse architectures using tools like Airflow or AWS Step Functions.

• Communication: Strong verbal and written communication skills with the ability to articulate technical decisions clearly to both technical peers and business leaders.

PREFERRED QUALIFICATION:

• Industry Experience: Prior experience working within heavily regulated environments such as financial services, mortgage banking, or GSE programs (Fannie Mae / Freddie Mac).

• ETL Migration: Hands-on experience with legacy ETL frameworks (e.g., IBM DataStage) to support modernization initiatives.

• DevOps & CI/CD: Familiarity with continuous integration and continuous deployment pipelines for data infrastructure (Git, Jenkins, GitHub Actions, Terraform).

• Data Quality Frameworks: Exposure to automated data quality and validation frameworks (e.g., Great Expectations, dbt testing suites).

• Streaming Analytics: Knowledge of real-time streaming platforms like Apache Kafka or AWS Kinesis.

• Professional Certifications: AWS Certified Data Analytics, AWS Certified Solutions Architect, or SnowPro Core/Advanced certifications.