About the role

The QA Engineer will develop and maintain automated test frameworks for AI solutions, including model validation and data quality assurance. They will collaborate with developers to implement continuous testing practices and define quality metrics for AI products.

What they look for

Python Test Automation Pytest Selenium API Testing CI/CD Git SQL Docker Kubernetes AWS Machine Learning Data Pipelines Performance Testing LLM Testing Cloud Platforms

Requirements

Candidates must have a bachelor's degree in Computer Science or a related field and 3-5 years of experience in software quality assurance. Proficiency in Python, automation frameworks, and experience testing machine learning models or data-intensive applications is required.

Full description

We're seeking an innovative QA Engineer to join our team. This role goes beyond traditional testing - you'll be at the forefront of using automation frameworks and AI-powered tools to ensure the quality, reliability, and performance of our AI solutions. You'll work closely with AI Integration engineers and software developers to build robust testing strategies that handle the unique challenges of AI systems, including model validation, data quality, and non-deterministic outputs.

Key Responsibilities

Test Automation Development

Design, develop, and maintain automated test frameworks using Python and modern testing tools

Build end-to-end test suites for APIs, data pipelines, and AI model/MCP endpoints

Implement continuous testing practices integrated with CI/CD pipelines

Create reusable test libraries and utilities to accelerate testing across projects

AI-Assisted Testing

Leverage AI tools (like GitHub Copilot, ChatGPT, or dedicated test generation tools) to accelerate test case creation and code reviews

Explore and implement AI-powered testing solutions for test data generation, visual testing, and anomaly detection

Experiment with automated test scenario generation using LLMs

Use AI tools for log analysis and defect pattern recognition

AI-Specific Testing

Develop testing strategies for LLMs including performance, accuracy, and bias detection

Test data quality, feature engineering pipelines, and model training workflows

Validate model output and monitor for model drift in production

Create test datasets that cover edge cases and ensure model robustness

Quality Strategy & Collaboration

Define and implement QA best practices and quality metrics for AI products

Collaborate with developers to identify testability requirements early in the development cycle

Perform code reviews focused on test coverage and quality

Document test strategies, test cases, and quality reports

Performance & Security Testing

Conduct load and performance testing for AI inference endpoints

Identify bottlenecks in data processing and model serving infrastructure

Participate in security testing activities focusing on data privacy and model security

Required Qualifications

Education & Experience

Bachelor's degree in Computer Science, Engineering, or related field

3-5 years of experience in software quality assurance and test automation

Proven experience building and maintaining automation frameworks from scratch

Experience testing cloud-based applications and APIs

Experience testing machine learning models, agentic data flows and data-intensive applications

Technical Skills

Strong proficiency in Python for test automation

Hands-on experience with testing frameworks: pytest, unittest, Selenium, or similar

Experience with API testing tools: Postman, REST Assured, or Python requests

Knowledge of CI/CD tools: GitLab CI, GitHub Actions

Familiarity with version control systems (Git)

Understanding of SQL and database testing (incl. Data Lake architectures)

Experience with containerization (Docker) and orchestration tools

Experience with AI-powered testing tools or test generation platforms

Experience with cloud platforms: AWS, Azure, or GCP

Understanding of microservices architecture and distributed systems

Experience with workflow orchestration tools (Airflow, Temporal, Prefect, n8n)

Experience mentoring junior QA engineers

What You'll Work With:

Languages: Python, SQL, JavaScript/TypeScript, YAML Data Tools: Airflow, dbt, Spark, Polars, pyArrow APIs & Integration: FastAPI, GraphQL, Kafka, Redis, Javascript/Typescript, WASM AI/ML: OpenAI API, Pydantic AI, Anthropic Claude, LangChain, vector databases, MCP protocol Cloud: AWS (Lambda, S3, RDS, Bedrock, SageMaker) or equivalent in GCP/Azure Infrastructure: Docker, Kubernetes, Terraform, GitHub Action

As part of our commitment to quality and excellence, Medlytix will continue to maintain a safe and healthy environment for you by requiring all applicants to submit to a criminal history check and those tentatively selected for a position to submit to screening for illegal drug use prior to appointment for a job. In addition, applicants may be screened for ability to perform essential functions of some positions.