Senior Automation Test Engineer

This role serves as the principal steward of quality for our software and AI product portfolio. The incumbent will ensure that all products meet the highest standards of reliability, accuracy, and performance throughout the development lifecycle, while acting as a trusted advisor to internal and external stakeholders on matters of testing, release assurance, and continuous quality advancement.

Scope

The role encompasses the architecture, execution, and ongoing refinement of comprehensive test frameworks for both traditional software and AI-driven systems. A core focus is placed on the unique demands of media and advertising platforms, including audience intelligence, creative optimization, bid automation, and campaign analytics. The position drives cross-functional collaboration to embed quality from inception, reduce defect exposure, and translate quality metrics into strategic business insight.

Key Accountabilities

• Strategic Test Leadership: Architect and implement comprehensive test strategies spanning functional, integration, regression, performance, security, and user acceptance dimensions for software and AI/LLM products.
• Automation & Delivery Excellence: Establish and scale robust automated testing frameworks fully integrated with CI/CD pipelines to enable high-velocity, high-confidence releases.
• AI & LLM Evaluation Frameworks: Define rigorous evaluation standards for machine learning and large language model systems, encompassing accuracy, precision and recall, hallucination incidence, bias and fairness, latency, and cost efficiency. Develop both offline and online evaluation pipelines, inclusive of human-in-the-loop methodologies.
• Test Data Governance: Design sophisticated test data strategies, including synthetic data generation and curated golden datasets, with specialized prompt and response libraries tailored to advertising and media applications.
• Quality Incident Stewardship: Lead comprehensive root cause analysis and remediation for production defects, model regressions, and client-impacting issues, in close partnership with Engineering and Data Science.
• Quality Intelligence: Define and govern key Quality Control metrics—including defect density, escape rate, model drift, test coverage, and MTTR—and synthesize findings into actionable recommendations for technical and commercial leadership.
• Embedded Quality Practice: Partner proactively with Product, Engineering, Data Science, and Ad Operations to integrate quality considerations at the earliest stages of design and development.
• Innovation in Quality Engineering: Continuously evaluate and implement emerging best practices, tools, and methodologies in software quality assurance, ML testing, and LLM evaluation.

Professional Attributes

• Exceptional analytical acumen with meticulous attention to detail
• Demonstrated ability to manage multiple priorities while consistently meeting commitments
• Superior interpersonal and communication skills, with the ability to influence across technical and business domains
• Strong sense of ownership and accountability for outcomes
• Proven capability to deliver excellence both independently and as a collaborative team member
• Advanced critical thinking and structured problem-solving abilities

Technical Expertise

• Comprehensive knowledge of software testing disciplines and their application to complex, AI-enabled systems
• Demonstrated proficiency with leading test automation frameworks such as Selenium, Playwright, Cypress, PyTest, JUnit, TestNG, and seamless CI/CD integration
• Substantive experience evaluating AI/ML and LLM systems, including model validation, prompt and response assessment, A/B experimentation, and monitoring for drift, bias, and hallucination
• Proficiency in Python or JavaScript/TypeScript, with strong SQL capabilities and fluency in REST and GraphQL API testing
• Familiarity with enterprise observability platforms including Grafana, Datadog, and the ELK stack for both pre-production and production quality monitoring
• Sound engineering judgment to identify material quality risks, correlate insights across diverse data sources, and articulate commercial and product implications with clarity