Senior / Lead Data Engineer – Service Layer / QTC Integration Platform

Senior / Lead Data Engineer – Service Layer / QTC Integration Platform

Role Details

• Role: Senior / Lead Data Engineer

• Primary Skills: Go / Golang, Python, AWS, Kafka / MSK, Microservices, Distributed Systems, Data Engineering

• Platform Exposure: AWS Serverless, Kafka, Snowflake, PostgreSQL, ClickHouse, Event-Driven Architecture

• Domain Exposure: Quote-to-Cash, CRM, ERP, Customer Lifecycle, Enterprise Data preferred

• Experience: 8+ years

• Location/ Mode: Bengaluru/Hybrid

Role Overview

We are looking for a strong, hands-on, and highly autonomous Senior / Lead Data Engineer to design, build, and operate the Service Layer platform powering enterprise Quote-to-Cash (QTC), customer lifecycle, finance, and enterprise application integrations.

This role requires deep expertise in distributed systems, event-driven architecture, microservices, data engineering, cloud-native development, and platform reliability. The candidate will also provide technical leadership to a team of engineers, driving solution design, engineering standards, delivery execution, and operational support across multiple integration initiatives.

The platform is built using AWS, Kafka / AWS MSK, Go, Python, canonical data models, asynchronous messaging, and cloud-native architecture patterns, processing high-volume, business-critical events with strong focus on reliability, security, scalability, observability, and operational readiness.

Key Responsibilities

Technical Delivery & Engineering Leadership

• Own end-to-end technical delivery of key initiatives, ensuring solutions are delivered with quality, scalability, security, and operational readiness.

• Lead and mentor a team of engineers through task planning, technical guidance, code reviews, design reviews, and delivery management.

• Drive engineering best practices across software development, CI/CD, observability, resiliency, security, and cloud-native architecture.

• Manage multiple concurrent initiatives while balancing business priorities, technical debt, and platform evolution.

Architecture & Platform Design

• Drive architecture and design decisions for event-driven integrations, canonical data models, orchestration frameworks, and platform capabilities.

• Define and enforce engineering standards, secure-by-design principles, cloud best practices, data governance, and non-functional requirements.

• Design and build scalable Service Layer foundation capabilities including messaging, orchestration, observability, reconciliation, data migration, operational tooling, and canonical data management.

• Ensure platform design supports resilience, scalability, fault tolerance, performance, and supportability.

Event-Driven Integration & Microservices

• Design, develop, and operate real-time event-driven integration services using Go / Golang and Python.

• Build and enhance the Service Layer platform to orchestrate Quote-to-Cash workflows across enterprise applications.

• Design and implement scalable Kafka / AWS MSK-based event processing with support for: • Retries

• Idempotency

• Sequencing

• Message replay

• Fault recovery

• Dead-letter queues

• Eventual consistency

• Build secure and scalable RESTful APIs, integration services, and enterprise system interfaces.

Data Engineering, Migration & Reconciliation

• Design and implement large-scale ETL, migration, reconciliation, and data-quality solutions using: • AWS Glue

• Step Functions

• Snowflake

• PostgreSQL

• ClickHouse

• Build and enforce canonical data models, data contracts, transformation layers, and validation frameworks.

• Support enterprise integrations by standardizing communication across multiple business systems.

• Ensure data accuracy, traceability, consistency, and operational reliability across the platform.

AWS Cloud-Native Engineering

• Design and build cloud-native applications on AWS using: • AWS Lambda

• API Gateway

• S3

• RDS / Aurora PostgreSQL

• DynamoDB

• SQS

• SNS

• EventBridge

• IAM

• CloudWatch

• VPC

• Build high-availability, business-critical applications with strong focus on performance, security, and operational supportability.

Observability, Monitoring & Operations

• Design and implement observability, monitoring, alerting, and operational dashboards for end-to-end workflow visibility.

• Work with tools such as: • CloudWatch

• Grafana

• Datadog

• Prometheus

• OpenTelemetry

• Support production operations including incident management, root-cause analysis, troubleshooting, and performance optimization.

• Drive metrics-driven improvements for reliability, scalability, and platform performance.

Collaboration & Stakeholder Management

• Partner with Product Managers, Architects, Enterprise Data teams, Engineering, DevOps, and business stakeholders.

• Define technical roadmaps, execution plans, dependencies, and delivery milestones.

• Communicate technical risks, trade-offs, design decisions, and mitigation plans clearly to stakeholders.

• Collaborate across multiple teams to ensure integration initiatives are delivered efficiently.

Must-Have Skills

Leadership & Delivery

• Proven experience leading and mentoring engineering teams.

• Strong experience in task planning, technical guidance, code reviews, and delivery management.

• Ability to drive architecture decisions for large-scale distributed systems and enterprise integration platforms.

• Experience managing multiple parallel technical initiatives.

Programming & Backend Engineering

• Strong hands-on experience in: • Go / Golang

• Python

• Experience building scalable backend services, event-driven integrations, orchestration workflows, and data processing solutions.

• Strong understanding of REST API development and secure API design.

AWS Cloud & Serverless

• Strong experience designing and building cloud-native applications on AWS.

• Hands-on experience with: • Lambda

• API Gateway

• S3

• RDS / Aurora PostgreSQL

• DynamoDB

• SQS / SNS

• EventBridge

• IAM

• CloudWatch

• VPC

• Strong understanding of AWS serverless architecture patterns.

Kafka / Event-Driven Systems

• Strong experience designing and operating event-driven and asynchronous systems using: • Kafka

• AWS MSK

• RabbitMQ or equivalent messaging technologies

• Strong understanding of: • Fault tolerance

• Idempotency

• Retry patterns

• Dead-letter queues

• Message replay

• Eventual consistency

• Sequencing

• Event orchestration

Databases & Data Engineering

• Strong experience with: • PostgreSQL

• ClickHouse

• Relational databases

• DynamoDB / NoSQL databases

• Strong knowledge of: • Data modelling

• Query optimization

• Performance tuning

• Data validation

• Reconciliation frameworks

• Experience with AWS Glue, Step Functions, Snowflake, and large-scale data workflows.

CI/CD, Security & Observability

• Experience with CI/CD pipelines, automated testing, code quality, and secure delivery using tools such as: • GitLab CI

• Jenkins

• SonarQube

• Trivy

• Semgrep

• Strong understanding of cloud security and secure-by-design principles including: • IAM

• OAuth2

• JWT authentication

• Encryption at rest and in transit

• Secrets Manager

• Vulnerability management

• Audit logging

• Compliance controls

Required Qualifications

• Bachelor’s degree in Computer Science, Engineering, Information Technology, or related discipline.

• 8+ years of software engineering experience.

• At least 4+ years focused on cloud-native, data engineering, integration, or distributed platforms.

• 4+ years building cloud-native solutions on AWS.

• 4+ years working with Go / Golang, Python, AWS services, and Kafka in production environments.

• Proven experience designing, building, and operating highly available enterprise integration and event-driven platforms.

• Strong experience in production support, incident management, root-cause analysis, and performance optimization.

Nice to Have

• AWS certifications such as: • AWS Certified Data Engineer – Associate

• AWS Certified Solutions Architect – Associate

• AWS Certified Developer – Associate

• Equivalent AWS certifications

• Experience in: • Quote-to-Cash

• CRM

• ERP

• Customer Lifecycle Management

• Enterprise Data domains

• Exposure to large-scale finance, billing, or enterprise integration transformation programmes.

• Experience building operational dashboards and platform-level reporting for engineering and business teams.