Python QA - (Pyspark with Databricks)
TestYantra Software Solutions · Bengaluru, Karnataka, India
Software Development · 501-1,000 employees
About the role
Design and maintain automated test frameworks to validate large-scale ETL pipelines and data transformations using Python, PySpark, and Databricks. Collaborate with cross-functional teams to perform data reconciliation, quality checks, and integrate automated tests into CI/CD pipelines.
What they look for
Requirements
Requires 8+ years of experience in software testing or automation with strong proficiency in Python, PySpark, and Databricks. Candidates must have solid expertise in SQL, ETL validation, and experience with data quality frameworks in an Agile environment.
Full description
About the Role
We are looking for an experienced QA Automation Engineer with strong expertise in Python, PySpark, and Databricks to support enterprise data engineering and analytics initiatives. The ideal candidate should have hands-on experience in validating large-scale data pipelines, ETL processes, and data platforms while building robust automation frameworks to ensure data quality and reliability.
Key Responsibilities
- Design, develop, and maintain automated test frameworks using Python.
- Validate ETL pipelines, data transformations, and data ingestion processes built using PySpark and Databricks.
- Perform data validation, reconciliation, and quality checks across structured and semi-structured datasets.
- Develop automated test scripts for functional, integration, regression, and end-to-end testing of data platforms.
- Validate large datasets using SQL and PySpark.
- Build and maintain automated data quality checks and monitoring processes.
- Collaborate with Data Engineers, Developers, and Business Analysts to understand data requirements and ensure high-quality deliveries.
- Analyse defects, perform root cause analysis, and support issue resolution.
- Integrate automated tests into CI/CD pipelines and support Agile delivery.
- Contribute to QA best practices, automation strategy, and continuous improvement initiatives.
Skills & Qualifications
- 8+ years of experience in Software Testing or Test Automation.
- Strong hands-on experience with Python.
- Solid expertise in PySpark and Databricks.
- Strong experience in SQL and ETL/data validation.
- Experience testing large-scale data pipelines and data engineering workflows.
- Hands-on experience with data quality frameworks or libraries such as Great Expectations, Soda, Deequ, dbt Tests, or similar.
- Strong understanding of data validation, reconciliation, and quality assurance methodologies.
- Experience working in Agile/Scrum environments.
- Excellent analytical, troubleshooting, and problem-solving skills.
