AI QA Engineer

There are no limitations, no barriers when the right people are on your way — apply for the vacancy and succeed with us!

Innovecs is an equal opportunity employer. All hiring decisions are based on professional qualifications, skills, and experience. We are committed to a transparent, merit-based recruitment process that prevents discrimination and ensures equal opportunities for all candidates. Reasonable accommodations are available upon request throughout the recruitment process to support accessibility and inclusion.

Project description

Our client is building a next-generation Warehouse Management System (WMS) that combines traditional supply chain execution with advanced Artificial Intelligence and Agentic AI capabilities. The platform is evolving into an intelligent operational assistant capable of decision support, warehouse optimisation, exception management, inventory analysis, workforce productivity recommendations, and conversational interactions.

The solution leverages LLMs, AI agents, and enterprise integrations to automate warehouse operations and improve decision-making across fulfillment, inventory, labor management, and logistics processes.

As part of this transformation, quality assurance plays a critical role in validating both traditional enterprise software functionality and AI-driven behaviours. The project requires advanced testing capabilities spanning deterministic business workflows, complex system integrations, AI agents, LLM-powered features, and autonomous decision-making processes

Technology Stack

AI & Agentic Technologies: LLMs, AI Agents and Agent Workflows, RAG, MCP integrations;
Agent orchestration frameworks: LangGraph, CrewAI, AutoGen;
Prompt engineering and evaluation;
Vector databases;
AI Evaluation & Quality Frameworks: DeepEval, LangSmith, Braintrust, etc.
Test Automation & Quality Engineering: Playwright, Selenium, Cypress, PyTest, JUnit, Postman, Rest Assured;
Integrations: REST APIs, Event-driven integrations, Databases (SQL/NoSQL), Enterprise messaging platforms, Warehouse system integrations (ERP/TMS/WCS integrations);
CI/CD & DevOps: GitHub Actions / Azure DevOps, Jenkins, Docker
AI Observability & Monitoring: MLflow, Prometheus/Grafana
Cloud Platforms: Azure (preferred), AWS (nice to have)

Position requirements

* 5+ years of experience in Quality Assurance and Test Automation.
* 2+ years of experience testing AI-enabled applications, LLM solutions, or agentic systems.
* Strong Python/ Java/ Javascript skills.
* Experience with Playwright, Selenium, Cypress, or similar automation frameworks.
* Solid understanding of software testing methodologies and test design techniques.
* Experience with API, integration, and database testing.
* Experience with CI/CD pipelines and DevOps practices.
* Knowledge of AI evaluation frameworks.
* Understanding of AI-specific failure modes: hallucinations, prompt injection, jailbreak attacks, model drift, bias and fairness issues, RAG failures.
* Experience working in cloud-native environments (Azure or AWS).
* Understanding of MCP architecture and agent-to-tool integrations.
* Experience with LangSmith, Braintrust, MLflow, Arize, or Evidently AI.
* Familiarity with EU AI Act and AI governance frameworks.

**Strong plus:**
* Background in Supply Chain, Logistics, WMS, TMS, OMS, or ERP platforms.
* ISTQB CT-AI certification or equivalent AI testing credentials.
* Contributions to AI testing frameworks or open-source projects.
* Leadership experience driving AI quality strategy across multiple teams.
* Knowledge of warehouse operations and fulfillment processes.

Position responsibilities

**Focus:** application-layer LLM QA, and enterprise/AISDLC quality engineering.

**AI Quality Engineering:**
* Design and execute quality strategies for AI-powered warehouse assistants, copilots, and autonomous agents.
* Build evaluation frameworks for LLM-based warehouse workflows and agentic systems.
* Define and maintain golden datasets and benchmark scenarios for warehouse operations.
* Validate AI-generated recommendations, decisions, and workflow actions.
* Perform hallucination, factuality, grounding, and response-quality testing.
* Validate RAG implementations and retrieval accuracy.
* Design agent trajectory tests for multi-step autonomous workflows.
* Conduct adversarial testing, prompt injection testing, and jailbreak assessments.
* Monitor model drift, output quality degradation, and AI performance trends.

**Test Automation & Enterprise Quality:**
* Create and maintain scalable automation frameworks for functional, API, integration, and end-to-end testing.
* Design CI/CD quality gates for AI and non-AI releases.
* Validate enterprise integrations across WMS, ERP, TMS, OMS, WCS, and third-party platforms.
* Implement automated regression suites for business-critical warehouse processes.
* Ensure release readiness through risk-based testing and quality metrics.
* Perform root-cause analysis of complex system-level defects.
* Drive quality best practices across engineering teams.

**Governance & Compliance**
* Define AI quality standards and testing methodologies.
* Support compliance with EU AI Act, ISO 42001, and responsible AI principles.
* Maintain audit-ready quality evidence and evaluation reports.
* Establish measurable KPIs for AI reliability, safety, and business accuracy.
* Contribute to AI quality engineering standards and Centers of Excellence.

**Cross-Functional Collaboration**
* Partner with AI Product Engineer, Solution Architect, and UX Lead.
* Participate in AI solution design reviews.
* Manage warehouse business requirements and maintain validation strategies.