Role description
Role Summary
This role requires deep understanding of AI fundamentals, hands-on experience with LLM testing, guardrail design, data sanitization,
and evaluation of agent trajectories. The tester will work closely with AI engineers, data scientists, and product teams to ensure safe, reliable, and high-quality AI behavior.
5-8+ years in QA, with at least 2-3 years in AI/ML or LLM testing.
Strong understanding of AI fundamentals: embeddings, vector stores, RAG, LLMs, agentic frameworks.
Hands-on experience testing:
• LLM prompts and outputs
• Agentic flows (e.g., LangChain, AutoGen, Semantic Kernel, AIP agents)
• Model evaluation metrics
Experience writing guardrails, safety rules, and evaluation prompts.
Knowledge of data sanitization, PII detection, and data governance.
Familiarity with pipeline validation and MLOps workflows.
Ability to evaluate model performance using Low-Pick Score or similar ranking metrics.
Strong Python skills for test automation and evaluation scripting.
