AI QA Test Lead

About the role

Truliant's mission is to improve lives by putting our members first, providing great service and straightforward financial solutions. Our core values of Member Focus, Service, Guidance, Relationships, and Community define how we interact with members to fulfill our mission of improving lives and providing a foundation on which we conduct ourselves. Our Promise to Our Teammates: Truliant strives to provide all teammates with an enjoyable place to work where they feel valued, empowered and rewarded for all that they do. Purpose of the Job The AI QA Test Lead is a critical quality and risk-management leader responsible for ensuring the reliability, accuracy, and production readiness of the credit union's AI-driven member and staff experiences. This role owns the overall AI quality assurance strategy, directly safeguarding millions of AI-powered interactions each year by ensuring solutions meet defined standards for accuracy, compliance, brand alignment, and ethical use. Serving as the primary authority on AI testing and validation, the AI QA Test Lead designs and manages comprehensive testing plans across conversational AI, generative AI, and agentic AI workflows. The role partners closely with the AI Program team and Truliant's business stakeholders, and key vendors to validate new use cases, manage AI-related risk, and ensure all AI releases are thoroughly tested before deployment. The role manages the end-to-end correction loop with third-party vendors integral to the delivery of Truliant's AI use cases. Key responsibilities include defining QA plans, leading end-to-end UAT, validating ethical guardrails, and establishing scalable QA practices that enable safe and confident expansion of Truliant's AI capabilities. Essential Functions and Responsibilities Owns and leads the end-to-end quality assurance strategy for the credit union's AI self-service and internal support solutions, with a primary focus on conversational AI platforms such as Kore.ai. Defines, documents, and maintains AI QA standards, test frameworks, methodologies, and acceptance criteria across all AI initiatives. Designs and executes comprehensive AI test plans covering functional accuracy, conversational flows, edge cases, integrations, compliance, and regression scenarios. Leads and governs all AI-related user acceptance testing (UAT), serving as the final quality gate for approving AI solutions prior to production release. Validates AI-generated responses for accuracy, consistency, brand alignment, and adherence to member-first service standards. Tests and monitors ethical AI guardrails, including bias prevention, hallucination detection, inappropriate response handling, and escalation logic. Partners closely with the AI Solutions Architect and AI Optimization Analyst to review solution designs and ensure testability throughout the AI development lifecycle. Establishes testing processes for prompt changes, workflow updates, model enhancements, and new AI capabilities to ensure safe iteration and deployment. Analyzes defects, testing outcomes, and production issues to identify root causes and drive continuous improvement in AI quality. Collaborates with business units and frontline teams to validate AI behavior against real-world use cases and operational expectations. Tracks, documents, and reports on AI quality metrics such as defect trends, release readiness, risk severity, and post-release issues. Stays current on AI QA best practices, regulatory considerations, and emerging testing techniques, proactively recommending improvements to strengthen AI reliability and governance. Should possess a strong commitment to providing excellent service to Truliant's members. Other Duties and Responsibilities Assists with other tasks and projects as assigned. Knowledge, Skills, and Abilities Must have a strong understanding of conversational AI, generative AI, NLP, and intent-based bot frameworks. Must have the ability to evaluate AI responses for tone, brand voice, compliance, and member experience impact. Must have the ability to understand credit union business processes and assess AI solutions against operational and member needs. Must have knowledge of ethical AI principles, including bias detection, guardrail validation, and hallucination mitigation. Must have excellent collaboration and communication skills to work across technical teams, business stakeholders, and frontline staff. Must have the ability to translate business and regulatory requirements into clear AI testing criteria and validation scenarios. Must have familiarity with analytics, dashboards, automated QA tools, or conversation intelligence platforms used to support testing and monitoring. Must have strong problem-solving skills with the ability to identify systemic quality issues and propose scalable solutions. Must have attention to detail and high standards when reviewing AI outputs, workflows, and system changes. Must have an understanding of UX a