AI Quality Engineer

External

Communitybrands · Atlanta, GA

Full-timeRemote5d ago

AWSAzureCI/CDData AnalysisGCPGenerative AI

Cover Letter Connect

Prepare for this interview

Elite

AI-generated questions, company research, and talking points tailored to this role

About the role

Momentive Software amplifies the impact of over 20,000 purpose-driven organizations in over 30 countries, with over $11 billion raised and 55 million members served to date. Mission-driven nonprofits and associations rely on Momentive's cloud-based software and services to address their most pressing challenges - from engaging their communities to simplifying operations and growing revenue. Designed to help organizations connect more, manage more, and ultimately expect more, Momentive's solutions are built with reliability at the core and strategically focus on fundraising, learning, events, careers, volunteering, accounting, and association management. Momentive partners with organizations that believe "good enough" is never enough - so they can bring on better outcomes for everyone they serve. Learn more at momentivesoftware.com . Why Work Here? At Momentive Software, we're a team of passionate problem-solvers, innovators, and volunteers who believe in using technology to make a real difference. We dream big, support each other, and take pride in creating solutions that help our customers drive meaningful change. If you're looking for a place where your work matters and your ideas are valued, you'll find it here. Medical, Dental & Vision Benefits 401(k

Requirements

Required

Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.

3-5 years of professional software engineering or quality engineering experience.

Hands-on experience working with LLMs or agentic AI systems (e.g., GPT-4, Claude, Gemini, or open-source models).

Proficiency in Python for scripting, test automation, and data analysis.

Experience designing and running evaluations (evals) for generative AI or LLM-powered features.

Solid understanding of software testing principles: unit, integration, regression, and end-to-end testing.

Familiarity with agentic frameworks and concepts (e.g., tool use, multi-step reasoning, retrieval-augmented generation, memory).

Experience with CI/CD pipelines and integrating automated tests into development workflows.

Strong analytical skills - able to interpret probabilistic outputs and distinguish meaningful regressions from expected variance.

Strong written and verbal communication skills; ability to clearly document findings and present quality data to non-technical stakeholders.

Detail-oriented, with a structured approach to exploring edge cases and failure scenarios.

Ability to work in a fast-paced environment and manage multiple priorities effectively.

Experience with prompt engineering and systematic prompt evaluation methodologies.

Familiarity with AI safety, alignment, or responsible AI concepts (e.g., hallucination mitigation, bias detection, guardrails).

Exposure to agentic orchestration frameworks (e.g., LangChain, LangGraph, AutoGen, CrewAI, or similar).

Experience with vector databases or RAG pipelines (e.g., Pinecone, Weaviate, pgvector).

Knowledge of observability and monitoring tools for AI systems (e.g., LangSmith, Weights & Biases, Arize).

Background in data science or ML experimentation practices.

Experience with version control systems (Git) and defect-tracking tools (e.g., Jira).

Exposure to cloud platforms (e.g., AWS, Azure, GCP) in the context of deploying or testing AI services.

What Success Looks Like

Builds robust eval frameworks that catch meaningful regressions in AI behavior before they reach production.

Reduces time-to-detection for quality issues in agentic workflows through effective automation and monitoring.

Contributes clear, actionable quality signals that help the team make confident release decisions.

Grows into a trusted voice on AI quality standards, influencing engineering practices across the team.

#LI-MH1 #momentivesoftware

AI Quality Engineer

About the role

Responsibilities

Requirements

Benefits

Additional Information

Your Match

Company Intel

What employees say

Interested in this role?