Lead Data Engineer

Job Title: Lead Data Engineer

Experience: 5+ Years Location: Remote (Anywhere in India) Employment Type: Full-time

About the Role

We are seeking a Lead Data Engineer with 5+ years of hands-on experience designing and building scalable, reliable data platforms and pipelines. The ideal candidate will have strong technical expertise across the modern data stack, experience owning data engineering initiatives end-to-end, and the ability to translate business and analytical requirements into robust data solutions.

You will be responsible for architecting, designing, and delivering data engineering solutions, while also mentoring engineers and driving best practices across data architecture, data quality, governance, and performance.

This is a hands-on technical leadership role where you will work closely with Product, Engineering, Analytics, and Business stakeholders to build a scalable data foundation that supports reporting, analytics, and self-service BI.

Key Responsibilities

• Design, develop, and manage scalable ETL/ELT pipelines for structured and unstructured data using AWS services such as AWS Glue, Redshift, S3, Athena, Lambda, and Kinesis/Kafka.

• Architect and implement data lake and data warehouse solutions, following best practices for data modeling, governance, scalability, and performance.

• Own the technical design and end-to-end delivery of data engineering initiatives.

• Develop robust data models and pipelines that support analytics, reporting, and business intelligence use cases.

• Drive data pipeline automation, orchestration, monitoring, and observability using tools such as Apache Airflow and AWS Step Functions.

• Enable self-service analytics by designing and curating data models and semantic layers that empower product owners and business users to build their own visualizations using Generative BI tools such as ThoughtSpot, QuickSight Q, or similar platforms.

• Optimize data pipelines, queries, and warehouse performance to ensure scalability, reliability, and cost efficiency.

• Establish and maintain standards for data quality, integrity, security, governance, and compliance across the data lifecycle.

• Collaborate with stakeholders across Product, Engineering, Analytics, and Business to translate requirements into scalable and maintainable data solutions.

• Identify opportunities to improve existing data architecture, processes, and infrastructure through automation and engineering best practices.

• Mentor junior and mid-level engineers and contribute to building a high-performing data engineering team.

• Stay current with emerging technologies and best practices in data engineering, cloud platforms, data architecture, and analytics.

Required Skills & Qualifications

• 5+ years of professional experience in Data Engineering, with demonstrated ownership of complex data engineering projects.

• Strong proficiency in Python and SQL.

• Hands-on experience with ETL/ELT frameworks and orchestration tools such as AWS Glue, Apache Airflow, and AWS Step Functions.

• Strong experience with the AWS data stack, including S3, Glue, Redshift, Athena, Lambda, and Kinesis or Kafka.

• Proven experience designing and implementing data lakes, data warehouses, and scalable data pipelines.

• Strong understanding of data modeling, dimensional modeling, data warehousing, and database design.

• Experience with query optimization, performance tuning, and cost optimization across data platforms.

• Experience with Spark/PySpark and/or EMR for large-scale data processing.

• Experience enabling self-service analytics through well-designed data models and semantic layers.

• Hands-on experience with BI/analytics platforms such as ThoughtSpot, Amazon QuickSight, or similar tools.

• Strong understanding of data quality, governance, security, and data lifecycle management.

• Excellent problem-solving, communication, stakeholder management, and technical leadership skills.

Preferred Qualifications

• Experience leading a technical team or mentoring data engineers.

• Experience owning data engineering projects from architecture and design through implementation and production.

• Experience working in a SaaS or product-based environment.

• Familiarity with modern data architecture patterns, including data lakes, lakehouses, streaming pipelines, and event-driven architectures.

• Experience with Kafka/Kinesis and real-time or near-real-time data pipelines.

• AWS certifications in Data Engineering, Solutions Architecture, or related areas.

• Contributions to open-source projects or demonstrated involvement in the broader data engineering community.