Senior Product Manager, AI Agents Testing, Dublin

Role Zendesk AI Agents are fully autonomous agents that resolve customer issues end-to-end— reasoning over knowledge bases, executing multi-step procedures, taking actions via APIs, and handing off to humans when needed. They operate across messaging, email, and voice channels, handling millions of conversations for brands like Liberty London, Unity, and Motel Rocks. As these agents grow more capable and more autonomous, the stakes of every deployment decision increase: a misconfigured procedure, a hallucinated response, or a broken escalation path can erode customer trust at scale.

Key Responsibilities

Own product strategy and roadmap for AI agent testing— simulation, quality scoring, experimentation, regression detection, and conversation tracing

Ship testing as an integrated experience embedded in the builder and deployment flow

Define how simulation works end-to-end: scenario generation from real conversation patterns, automated pass/fail evaluation, and results that point admins to exactly what broke and where

Build the experimentation layer— A/B testing of agent behavior, staged rollouts with statistical rigor, safe iteration on tone and resolution strategies

Design a pre-publish readiness gate that gives admins a quantified view of risk before every deployment— specific issues, coverage gaps, comparison to current production behavior

Partner with ML, QA, and platform teams on scoring methodology, simulation infrastructure, and tracing architecture

Make all of this usable by non-technical admins— CX managers, bot builders, operations leads who need answers without writing code or filing engineering tickets

Required Qualifications

Several years of product management experience, with 2+ years building for non-technical users in complex technical domains (QA tooling, no-code platforms, admin consoles, workflow builders) in B2B SaaS

Experience shipping AI/ML products where evaluation and reliability were real concerns, not afterthoughts

Understand why traditional testing doesn’t work for LLM-based systems and have opinions about what does

Ability to ship platform capabilities through user-facing product surfaces— you don’t just build infrastructure, you make it usable

Experience integrating acquired or adjacent products into a unified experience— combining capabilities from different teams, codebases, or organizations into something that feels like one product

Track record coordinating across 3+ engineering teams and multiple departments to deliver one coherent product experience

Bonus Qualifications

Experience building simulation, synthetic data, or automated testing products

Background in conversational AI, chatbot platforms, or customer service technology

Familiarity with LLM evaluation approaches— human-in-the-loop scoring, automated rubrics, AI-as-judge

Experience with experimentation infrastructure— A/B testing, staged rollouts, feature flagging at scale

Experience turning internal prototypes into customer-facing products

Success in the Role

Testing becomes part of how customers build and deploy agents— not something they do separately, but part of the flow

Customers can quantify whether their agent is ready to go live, and catch regressions before end users hit them

Automated resolution rates improve because customers can actually diagnose and fix quality issues instead of guessing

The testing platform becomes a shared capability used beyond AI Agents— consumed by other product teams that need to validate AI-powered experiences

Equal Opportunity Employer&EEO Statement Zendesk is an equal opportunity employer, and we’re proud of our ongoing efforts to foster global diversity, equity,&inclusion in the workplace. Individuals seeking employment and employees at Zendesk are considered without regard to race, color, religion, national origin, age, sex, gender, gender identity, gender expression, sexual orientation, marital status, medical condition, ancestry, disability, military or veteran status, or any other characteristic protected by applicable law. We are an AA/EEO/Veterans/Disabled employer. If you are based in the United States and would like more information about your EEO rights under the law, our policies comply with applicable law. Zendesk endeavors to make reasonable accommodations for applicants with disabilities and disabled veterans pursuant to applicable federal and state law. If you are an individual with a disability and require a reasonable accommodation to submit this application, complete any pre-employment testing, or otherwise participate in the employee selection process, please send an e-mail to with your specific accommodation request.

#J-18808-Ljbffr