Staff Applied Scientist, AI Quality & Meta Evaluation

About the role

As a Principal Applied Scientist on the Human Centered AI team, you will be the technical engine behind our Data Quality Validation framework. This is a high-impact individual contributor role for a scientist who wants to architect and build - not just advise. You will own the data science methodology underpinning our data quality validation models, design the statistical frameworks that govern judge reliability, and work hands-on to close the loop between automated evaluation and human ground truth. You will be the person who answers the hardest question in our stack: "Can we trust the evaluators that are evaluating our models?" ","responsibilities":"Design, develop, and iterate on the reasoning agent that serves as our adjudicator, auditing Production LLM Judge outputs for hallucination, drift, and systematic bias Develop the statistical and ML approaches that detect when Production LLM Judges diverge from ground truth, including confidence calibration, entropy-based uncertainty quantification, and out-of-distribution detection Define the algorithms that determine what gets routed for deeper review, moving the team from random sampling to principled, risk-stratified smart sampling Design the hierarchical weighting model and the confidence interval framework that replaces misleading point estimates with statistically rigorous ranges Establish the standards for how immutable ground truth sets are built, versioned, and validated, including inter-annotator agreement protocols Partner with Autograder Developers to validate new LLM Judge through our standard validation processes, ensuring LLM Judges are rigorously validated before reaching production Serve as the scientific authority on data quality evaluation methodology for partner teams across ASE, translating complex statistical findings into clear decision-readiness signals for engineering and leadership stakeholders