Agentic AI Testing & the AI-SDET Interview

01How the SDET role changed

For a decade, the SDET job was a fairly stable stack: a language (usually Java), an automation library (Selenium, now often Playwright), API testing, a framework you designed yourself, and a CI/CD pipeline to run it all. That stack has not disappeared — but in 2026 a second layer sits on top of it. Interviewers now expect you to know how to work alongside AI agents: to generate tests with them, to review what they produce, and to test the AI-powered features your own company is shipping.

The important shift is one of judgement, not tools. The market has moved from "can you write a test?" to "can you decide what to test, direct an agent to write it, and catch the agent when it is wrong?" AI removed some of the typing; it raised the bar on design and review. That is the lens to bring into every answer below.

02AI coding agents & the Planner-Generator-Healer pattern

The single most common AI question in an SDET screen now is some version of: "How do you use AI coding agents for test automation?" The strong answer is not a tool name — it is a pattern. The pattern that has emerged for using agents on a test suite has three roles:

Why this pattern impresses: it shows you know where AI helps (volume, boilerplate, first drafts, self-healing brittle locators) and where a human still owns the decision (what is worth testing, whether the generated assertion is meaningful, whether a "heal" is hiding a real bug). The failure mode interviewers listen for is the candidate who lets the generator run unchecked and ships duplicated or meaningless tests.

How to talk about it well

• Generate one file at a time and review it — never accept a bulk dump of tests.
• Pin selectors to data-testid so generated and healed locators stay stable.
• Run a self-critique pass — ask the agent to critique its own output against your rubric before you merge.
• Reject anything that duplicates existing coverage — more tests is not better tests.
• Keep the healer on a leash — a heal that changes an assertion should trigger human review, because it may be masking a regression.

03MCP for testers

MCP (Model Context Protocol) is an open protocol that lets any LLM call any tool through one standard interface, instead of every model needing custom glue for every tool. It has become common enough that "explain MCP" is now its own interview question for AI-adjacent SDET roles.

For a tester, MCP matters in two directions. First, it is how an AI agent reaches your world — your CI, a browser, a database, your test-management tool — through MCP servers that expose those as callable tools. Second, and more interview-relevant, when you are asked to test an MCP-connected agent, you are really testing three things:

• Tool discovery — does the agent correctly find the tools available for the current context and user?
• Tool selection — given a goal, does it pick the right tool, or make unnecessary calls?
• Permission & constraint checking — does it respect what the current user is allowed to do, and refuse tools it should not use?

You do not need to have built an MCP server to answer well. You need to show you understand that an agent with tools is a system with a new attack surface, and that testing it means verifying the boundaries as much as the happy path.

04Evals: the unit tests of AI

This is the concept that separates candidates who have used AI from candidates who have only read about it. Traditional tests use a fixed assertion: input X gives exactly output Y. But an LLM is non-deterministic — the same prompt can give different, still-valid answers — so a hard assertEquals does not work. The answer is evals: the unit tests of AI systems.

An eval scores an AI's output rather than checking it for exact equality. The main approaches:

Ground-truth (golden set) evals

Build a curated dataset of 50–200 known-good input/output pairs — a "golden set". Run the AI over it and score how close each answer is to the reference. This is your regression suite for a model or prompt: run it on every change and watch the score.

LLM-as-a-judge

Use a second model to grade the first model's output for quality, correctness or reasoning, against a rubric you write. It scales far past what humans can grade by hand — but you validate the judge itself with periodic human spot-checks so you are not trusting a grader you never checked.

Component evals for agents

For agentic systems you go beyond "was the final answer right?" and score the process:

• Tool-use evaluation — did the agent call the correct tool for each step, and efficiently?
• Process adherence — did it follow the intended plan, or skip a required step such as verifying a fact before using it?
• Adversarial evaluation — feed it contradictory information or instructions to ignore its safety rules, and check it stays robust.

Metrics worth naming

Grounding (does the answer cite the source it was given?), correctness, relevance, toxicity, latency per step, and cost per call. In production you also track escalation rate, resolution rate and user-satisfaction score. Being able to reel off two or three of these signals real hands-on experience.

05Prompt engineering for test cases

Generating test cases from a user story is now a practical exercise you may be asked to do live. The reliable structure is a four-part prompt — Role, Context, Task, Format:

What turns this from a party trick into an engineering skill is the review discipline around it: run a self-critique pass, check the output against a rubric (are negatives and boundaries actually covered? any duplicates? any missing security or accessibility case?), and only then fold the survivors into your test-management tool. The prompt drafts; you decide.

06Testing LLM-powered features

More products now ship an LLM inside them — a chatbot, a summariser, a "generate with AI" button. Testing those features is a distinct skill, because the usual assumptions break:

• Non-determinism — the same input can produce different valid outputs, so you assert on properties (does it cite a source? is it under N tokens? is it on-topic?) not exact strings.
• Hallucination & grounding — verify the answer is supported by the provided context, not invented. Grounding checks compare the output against the source documents.
• Prompt injection — a user can embed "ignore your instructions" in their input; test that the feature resists it. This is the adversarial eval from section 4, applied to a product.
• Toxicity & safety — check the feature refuses harmful requests and does not emit unsafe content, across a battery of red-team prompts.
• Cost & latency — these are now test concerns; a feature that is correct but too slow or too expensive still fails.

The mental model to bring: you are no longer testing a function with one right answer, you are testing a probabilistic system against a set of properties and risks. Frame it that way and you will stand out.

07AI governance & risk in QA

The senior signal in an AI-SDET interview is judgement about risk. Expect at least one question on governance: how do you keep an AI-assisted testing process trustworthy? Points worth making:

• Human-in-the-loop on merge — AI can draft and heal, but a human approves what enters the suite and what a "heal" changes.
• Traceability — keep prompts, model versions and golden sets in version control, so a test change is auditable and reproducible.
• Data handling — do not paste secrets, PII or proprietary code into a model that is not approved for it; know your company's boundary.
• Regression on the AI itself — evals run in CI so a prompt or model change that degrades quality is caught before release, exactly like a normal regression.

You do not need to have written a governance policy. You need to show you would not let an unreviewed agent silently rewrite the test suite — that instinct is what separates a senior answer from a junior one.

08How to prepare — a focused 30-day plan

You cannot fake hands-on AI experience in an interview, so build a little of it:

• Week 1 — fundamentals stay sharp. Drill core Selenium, Java and API questions. AI augments these; it does not replace them. Use the interview simulators below.
• Week 2 — one real agent project. Use an AI coding agent to generate a small Playwright or REST-Assured suite end to end. Note where it broke — that story is your best interview material.
• Week 3 — learn evals. Build a tiny golden set (even 20 pairs) for a prompt, and try an LLM-as-judge. Read up on MCP at a high level.
• Week 4 — practise out loud. Run full mock rounds, mixing traditional and AI questions, and rehearse the Planner-Generator-Healer and evals answers until they are automatic.