ML Engineer - Evaluation Analysis, Metric and Data Strategy

External

Apple · Culver City, CA

Full-timeOn-siteToday

AuditingLangChainLeadershipMachine LearningPandasPython

Cover Letter Connect

Prepare for this interview

Elite

AI-generated questions, company research, and talking points tailored to this role

About the role

Day-to-day work involves analyzing evaluation results, identifying trends, regressions, and segment-level patterns across multiple AI features. This includes collaborating with partner teams on data collection strategies, ensuring evaluation data is representative of real-world usage, and designing the metrics framework that leadership uses to make decisions on AI features. Typical deliverables include: feature-level quality metrics and dashboards, evaluation analysis reports, data collection requirements, dataset representativeness audits, multi-turn evaluation frameworks and session-level scoring rubrics, and concise metric summaries for decision-makers.

Requirements

Experience designing evaluation or quality metrics for AI-powered or ML-driven features in consumer-facing products

Familiarity with productivity software or creative applications, with an ability to distinguish between technically correct and genuinely useful AI outputs

Experience partnering with engineering or data teams to define data collection requirements and schemas

Track record of translating complex analytical findings into concise recommendations for non-technical decision-makers

Experience evaluating tool-use accuracy, retrieval quality, or function-calling reliability within AI systems

Experience with evaluation methodology including inter-annotator agreement, evaluation bias detection, and dataset representativeness auditing

Familiarity with agentic orchestration frameworks (LangChain, LangGraph, CrewAI, AutoGen) and emerging agent interoperability protocols (A2A, MCP), with an understanding of how architectural choices in agent design affect evaluability

Understanding of ML model development processes, with the ability to specify what evaluation signals are useful for model improvement

Experience managing evaluation across multiple features or product areas simultaneously, with systematic rather than ad-hoc approaches

Graduate degree in a relevant quantitative field

Bachelor's degree in Statistics, Data Science, Applied Mathematics, Computer Science, or a related quantitative field

5+ years of experience in applied science, data science, or evaluation research, with a focus on defining and operationalizing quality metrics

Experience with statistical analysis methods including significance testing, sampling design, effect size estimation, and experimental design

Experience working with production user data, understanding its biases and limitations compared to controlled evaluation data, including familiarity with sequential interaction data where context and turn order affect quality assessment

Ability to design evaluation approaches where the unit of analysis is a session or conversation rather than a single model output

Track record of independently designing metrics frameworks and driving data-informed decisions across cross-functional teams

Proficiency in Python (pandas, scipy, scikit-learn) or R for da

ML Engineer - Evaluation Analysis, Metric and Data Strategy

About the role

Responsibilities

Requirements

Additional Information

Your Match

Company Intel

What employees say

Interested in this role?