Data Engineer II

Job Requirements

About the Role

As a Data Engineer II in the Data & Analytics function at IDFC FIRST Bank, you will be responsible for developing and maintaining scalable, high-performance data pipelines that support the bank’s analytics and reporting needs. You will work closely with business stakeholders and analytics teams to build efficient data processing systems and ensure the availability of clean, reliable data for decision-making. This role requires strong technical expertise in big data technologies, cloud platforms, and data architecture.

Key Responsibilities

Primary Responsibilities

• Design, develop, and maintain optimized and highly available data pipelines across the full data lifecycle—from ingestion and transformation to consumption.
• Work with large-scale data volumes (TBs) and ensure efficient processing using Spark, SQL, and Hadoop ecosystem tools.
• Collaborate with business stakeholders to identify and document high-impact business problems and translate them into scalable data solutions.
• Implement and manage Data Lake/Lakehouse architectures using platforms like Cloudera, Hortonworks, and AWS.
• Utilize APIs for seamless data integration and usability across systems.
• Apply big data infrastructure tools such as Spark Streaming, Hive, MapReduce, HDFS, YARN, HBase, and Oozie.
• Debug and resolve technical issues in data pipelines; manage version control using Git.
• Create and maintain technical design documentation (HLD/LLD) for data projects and pipelines.

Secondary Responsibilities

• Work independently and manage your own development efforts while collaborating with senior engineers as needed.
• Learn and apply internally available analytics technologies.
• Identify key performance indicators and contribute to strategies for achieving analytical goals.
• Perform data mining and analysis using foundational data engineering techniques.
• Collaborate with BI analysts and engineers to develop prototypes and present solutions.

What We Are Looking For

Education

• Bachelor’s Degreein: Science (B.Sc) or Technology (B.Tech) or Computer Applications (BCA)
• Postgraduate Degree(preferred) in Science (M.Sc) or Technology (M.Tech) or Computer Applications (MCA) or MA

Experience

• 2to6 years of relevant experiencein data engineering or related roles, with hands-on experience in large-scale data environments.

Skills and Attributes

• Proficiency in SQL, Spark, Hive, and Hadoop ecosystem tools.
• Strong understanding of Data Lake/Lakehouse architecture and cloud platforms (AWS preferred).
• Familiarity with big data infrastructure and relational databases.
• Strong debugging and problem-solving skills.
• Experience with Git for version control.
• Excellent communication and documentation skills.
• Ability to work independently and deliver high-quality, error-free solutions.

Key Success Metrics

• Timely and error-free delivery of data engineering tasks.
• High-quality code and documentation.
• Effective collaboration with stakeholders and team members.
• Identification and resolution of data issues.
• Technical ownership of assigned projects and pipelines.