LLM Quality Engineer

Responsibilities

• As part of the Assurance & Insight team, you will focus on ensuring the quality and reliability of our AI-driven systems. This role sits at the intersection of testing, AI evaluation, and operational quality.
• You will help build the foundations for how we test, benchmark, and monitor both software systems and LLM-driven outputs ensuring that what we deploy performs as expected in real-world scenarios.
• You will take ownership of quality across our AI systems, from automated testing to LLM evaluation and production monitoring. Your work will help reduce regressions, improve reliability, and free up engineering time to focus on building.
• Must-Have Qualifications
• Degree in computer science, software engineering, or a related technical field or equivalent demonstrable experience
• Exposure to LLMs or AI systems (even personal projects or coursework counts)
• Familiarity with test frameworks such as pytest or similar
• Understanding of CI/CD concepts and release pipelines
• Comfortable working with Python and APIs
• Experience writing or evaluating prompts for language models
• Strong attention to detail and a methodical approach to problem-solving
• Preferred Skills
• Exposure to AI/LLM tools or projects
• Experience with testing, QA, or automation (academic or personal projects)
• Interest in data quality, monitoring, or analytics
• Experience working in team-based or project-based environments
• Our Hiring Process
• 25-minute screening call
• Take-home challenge: A hands-on task to assess your problem-solving and technical skills
• Combined technical and cultural interview (in-person) Whiteboard Interview: 1-hour with 2 of our engineers to discuss your solution to the take-home challenge
• Culture fit: 30-minute meeting with our leadership team
• Why Join Us?
• Work on real-world problems where AI creates measurable impact.
• Be part of a team where your work matters, and your ideas become real.
• Collaborate with sharp, driven colleagues in a culture of trust, ownership, and high standards.
• Contribute to making the built environment smarter and more sustainable.

Additional Information

In AECOM's AI Engineering team your code will directly shape the physical world around us. We build AI-driven technology that revolutionises how infrastructure and buildings are designed and engineered; reducing waste, cutting CO₂, and making the built environment more efficient and sustainable. This is where software has measurable, real-world impact. With our AI Engineering team we've created a unique setup: a lean, highly technical team with the speed and ownership of a start up, backed by the scale, resources, and domain expertise of one of the world's leading engineering firms. There has never been a better time to be at AECOM. We are leading the industry's AI transformation, and with our people and technology we deliver excellence and innovate with impact. We invite you to bring your bold ideas and big dreams to solve the world's most complex challenges. We're one global team driven by our common purpose to deliver a better world. Join us.

Your Match

How well this role fits your profile.

Company Intel

What employees say

Worked at Aecom2? Share your experience