Senior QA Automation Engineer / SDET - GenAI & LLM Testing
6 Month Contract
Banking Client
Remote / Hybrid (UK)
Competitive Day Rate
We're supporting a leading banking client on a large-scale AI transformation programme and are looking for an experienced Senior QA Automation Engineer / SDET with a strong background in GenAI and LLM testing.
This role will focus on defining and implementing automated testing frameworks for AI-driven products, ensuring the quality, reliability and safety of LLM-powered solutions deployed within a highly regulated banking environment.
Key Responsibilities
- Design and develop scalable automated test frameworks using Python.
- Create test strategies from architecture and solution design documentation.
- Test RESTful APIs and microservice-based applications.
- Define and execute testing approaches for event-driven architectures and Kafka-based systems.
- Validate GenAI and LLM solutions through functional, non-functional and behavioural testing.
- Develop automated prompt testing and evaluation frameworks.
- Verify AI guardrails, safety controls and compliance requirements.
- Implement validation approaches for non-deterministic LLM outputs.
- Build and maintain contract tests using Pact Flow.
- Collaborate closely with Engineering, Architecture, Data Science and Product teams.
- Support CI/CD pipelines and quality engineering best practices.
Required Skills & Experience
Core Automation
- Strong Python 3.x development skills with experience building testing frameworks from the ground up.
- Extensive experience with Pytest.
- Strong API testing experience.
- Experience creating test strategies and quality approaches from architectural diagrams and solution designs.
Cloud & Platform
Working knowledge of AWS services including:
- Lambda
- Bedrock
- S3
- EKS
Event-Driven Systems
- Experience testing Kafka-based architectures.
- Strong understanding of asynchronous messaging and event-driven systems.
GenAI / LLM Testing (Essential)
- Proven experience testing LLM and GenAI applications.
- Prompt testing and prompt evaluation.
- Non-deterministic output validation.
- AI guardrail verification and safety testing.
- Hallucination and response quality assessment.
- Experience establishing quality standards for AI solutions.
LLM Evaluation Tooling
Experience with one or more of:
- DeepEval
- Promptfoo
- RAGAS
- LangSmith
- Similar LLM evaluation frameworks
Contract Testing
- Hands-on experience with Pact Flow.
- Consumer-driven contract testing within distributed systems.
Desirable
- Banking or Financial Services experience.
- Experience working in highly regulated environments.
- Exposure to RAG architectures.
- Knowledge of LLM observability and monitoring.
- Kubernetes experience.
- CI/CD pipeline integration.
Guidant, Carbon60, Lorien & SRG - The Impellam Group Portfolio are acting as an Employment Business in relation to this vacancy.
Senior QA Automation Engineer / SDET - GenAI & LLM Testing 6 Month Contract Banking Client Remote / Hybrid (UK) Competitive Day Rate We're supporting a leading banking client on a ...