Quality Lead, YouTube Global Vendor Operations
Skills
- Quality Assurance
- AI Model Evaluation
- Precision and Recall
- Prompt Engineering
- Stratified Sampling
- Contact Center Operations
- Cross-functional Collaboration
- LLM Evaluation
About the role
Responsibilities:
Design Quality Lead (QA) Frameworks to ensure alignment with key business outcomes. Monitor AI QA model performance and accuracy using metrics such as Precision, Recall, and Human Inter-rater Agreement.
Manage the ingestion of human-validated Golden Labeling from manual overrides to drive continuous, high-fidelity model retraining.
Implement blind review protocols and complex multi-tiered human + AI arbitration to prevent Automation Bias.
Design stratified sampling methodologies that prioritize high-risk or ambiguous interactions (e.g., cases where AI confidence is low) for human review.
Partner with the training team to remediate quality gaps.
Minimum qualifications:
Bachelor's degree or equivalent practical experience.
5 years of experience identifying process issues or working with cross-functional teams.
5 years of experience with contact center metrics and QA methodologies.
Preferred qualifications:
Experience with LLM-based evaluation frameworks and prompt engineering.
Experience in stratified sampling to identify specific interactions for human validation, ensuring that human efforts are focused on high-signal or ambiguous cases.
Familiarity with QA automation and sampling tools.
Ready to apply?
Sign up first — takes a minute — and you get Hiro’s take on this role, a resume tailored to it, and (if available) a referral from a real employee at Google. All free with your Pro gift.
Sign up to apply