Lead AI Quality Engineer

About Phase2

Our remote first employees join and stay at Phase2 to be a force. In pursuit of digital experience that advances the human experience, you will do your best work, live fully, and succeed as a team. 

Phase2 a digital agency passionate about customer experience, grounded in data & insights, and rooted in more than two decades of successful technology delivery. We guide industry-leading companies to what’s next in their digital landscape.

And it all begins with finding the right people who align with our values and passion for impact.

About the Role

The Lead AI Quality Engineer owns the evolution of quality assurance at Phase2. Working closely with leaders within the Engineering Department, including other Quality Specialists, you will lead setting the vision for how Phase2 maintains a high level of quality in the software we produce, in an era when our engineers’ use of agentic AI has led to unprecedented rates of code production. You will also own setting and executing a roadmap that implements this vision: establishing consistent guidelines, methodologies, and tools that serve as the standard for quality assurance across our projects, and building a quality system capable of keeping pace with AI-accelerated delivery.

Your ownership of quality assurance spans two fronts: testing our AI-powered products and features, and applying AI-driven tools and techniques to test our legacy software systems. This includes establishing standards and guidelines for creating test plans for AI-powered features (LLM-based applications, agentic workflows, ML-driven components), and for using AI-powered testing tools responsibly with appropriate human oversight. Your ownership of quality strategy runs end-to-end, from requirements and design through production monitoring, across complex digital and AI-powered delivery engagements.

This is a hybrid role: allocated half-time to work on client projects, and half-time to overseeing and growing the Quality practice. Your direct work with clients involves leading quality strategy design and execution across one or more client delivery teams, presenting quality outcomes directly to client stakeholders, and coaching Phase2 project team members and client teams toward a quality ownership mindset. Your Quality practice oversight responsibilities include setting the vision and roadmap, executing on that plan, and managing, mentoring, and coaching other members of the quality team.

In collaboration with the VPs of Engineering, you will help socialize our approach to quality across other departments, including project management, strategy, sales, and marketing, ensuring that everyone has a shared understanding of the vision and how it is being executed.

This position reports to: VP, Engineering

Who You Are

• Quality strategist and technical leader: You've built your career hands-on in QA, and you're ready to own quality strategy rather than just execute test plans. You can translate coverage decisions into a business narrative and advise clients directly.
• Systems thinker for quality: You design coverage models anchored to business-critical outcomes, and you're hands-on, building automation frameworks and evaluation pipelines yourself.
• AI-native testing mindset: You bring a modern, AI-first testing approach to every project, including directing AI-powered testing tools, using prompt engineering to generate adversarial test scenarios, and evaluating LLM and ML outputs for accuracy, hallucination, and bias.
• Technical range and adaptability: You are resourceful across unfamiliar client stacks and delivery models, ramping up quickly on new tools and AI systems through hands-on exploration and collaboration with subject-matter experts.
• Client-facing consultant: You act as a quality consultant to client stakeholders, coaching engineering teams to adopt a quality ownership mindset and reporting outcomes in business terms.
• Mentor and practice builder: You manage and mentor the QA team, growing their skills, and bringing emerging AI testing methodologies back to the team proactively.

What You'll Do

• Set the vision and roadmap for software quality. Define the standards for how Phase2 ensures quality in the AI era, from how we write tickets to how we test our work. Evaluate our current suite of AI-enabled and traditional testing tools, and our current QA practices and methods, and maintain a roadmap that outlines planned improvements to our tools and methods.
• Define testing standards and defaults. Set the standards and defaults for what gets tested manually by humans, what is handled through automation, and what human oversight of that automation looks like. Lead this work in partnership with Engineering leaders, supported by AI Quality Specialists.
• Innovate approaches to accelerate quality assurance testing. Identify and implement tools and methodologies to enable testing to keep pace with AI-accelerated software development.
• Lead the use of agentic AI to write tests. Teach people across a variety of roles how to use agentic AI to write automated tests.
• Recommend improvements to execution of projects and the SDLC. Identify how quality should be integrated through the project lifecycle, including when and how QA specialists should be involved at each stage of a project.
• Socialize the quality assurance approach across the company. Educate and enroll other departments in the company about our evolving approach to QA, ensuring everyone has a shared vision and understanding.
• Oversee agentic QA tooling. Build, run, maintain, and enhance agentic AI tools that do QA work and contribute those to Phase2’s AI delivery framework.
• Test Phase2-built AI tools. Evaluate Phase2's internally built agentic AI tools (both for internal use and client use) and identify areas for improvement with clear, specific detail.
• Set approaches for testing AI features. Set guidelines for designing and executing test plans for AI-powered features, including LLM-based applications, agentic workflows, and ML-driven components.
• Develop AI evaluation frameworks. Develop evaluation frameworks for AI outputs, covering accuracy, relevance, consistency, hallucination detection, and bias across protected attributes.
• Own integration of testing with automated development workflows. Define and help implement improvements to our CI/CD testing tools, including traditional unit, functional, integration, and VRT tests, as well as agentic AI-enabled tests.
• Advise clients on quality best practices. Coach client engineering teams to adopt a quality ownership mindset, acting as a quality consultant to client stakeholders.
• Lead the quality team. Manage, mentor, and coach other Quality Specialists and Quality Engineers.
• Follow the cutting edge of quality engineering and share knowledge back to Phase2. Stay current on AI testing methodologies, responsible AI principles, and emerging tooling, bringing insights back to the team proactively.

What You'll Need

• Experience: 8+ years of software QA experience, preferably with demonstrated experience testing AI/ML-powered features.
• Leadership: Experience managing, mentoring, and coaching direct report employees.
• QA Strategy: Experience recommending and then executing improvements to QA methods and tools.
• Automation proficiency: Proficiency in writing automated tests in at least one framework.
• AI testing knowledge: Working knowledge of AI testing concepts, including hallucination detection, output evaluation, prompt engineering for test generation, and bias testing.
• Communication: Strong verbal and written communication skills, with the ability to present quality outcomes both to technical and to non-technical stakeholders.
• Accessibility: Experience with WCAG 2.1/2.2 accessibility standards.
• Agency or consulting background (preferred): Experience working in a digital agency, technology consultancy, or professional services environment.

What Success Looks Like

• Faster testing: The amount of time spent per project performing QA activities decreases, without sacrificing thoroughness.
• Clearly documented QA standards: The default methods and tools for performing QA, including guidelines describing which ones to use on specific types of projects, are spelled out in documented guidelines for all project teams to use.
• Improvement of QA methods and tools: Phase2’s internally-built QA tools are periodically evaluated and enhanced, new tools are built as needed, and improvements to the methods of conducting QA are rolled out.
• Upskilling the team: The rest of the QA team, and some engineers, improve their testing methods, learn new skills, and contribute to the creation and improvement of our testing tools.
• Cross-departmental understanding of QA tools and methods: Other departments understand the evolution of our quality practice. For example, sales and marketing can articulate our quality practice to prospective clients, and project managers can factor our QA approach into project plans.
• Reliable AI systems: Phase2 has established methods and tools for testing AI-powered features for accuracy, hallucination, and bias.
• Business-facing outcomes: Quality results are reported in terms leadership and client stakeholders act on.