Medior AI Engineer

Mission Overview:

Keystone Solutions is seeking a Medior AI Engineer to join our consultancy mission at a client site. You will be part of the AI Center of Excellence, focusing on the Run phase of AI services in production. This role involves ensuring the reliability, security, and cost-efficiency of AI services built on a hybrid operating model, utilizing both managed Azure services and self-hosted open-source components.

Key Responsibilities:

• Operate the AI services in production: monitor availability, latency, and errors, handle incidents and service requests, analyze root causes, then fix or escalate.
• Keep answer quality under control: run and maintain evaluation sets with business content owners, track answer quality, non-answer rate, and user feedback, and detect and investigate quality regressions.
• Operate the ingestion pipelines: manage incremental refresh, failed documents, index health, and onboarding of new content sources or business domains.
• Operate agent integrations: monitor tool and integration calls to business systems, handle failures and timeouts, and review action logs and exceptions.
• Manage change safely: model version upgrades and retirements, prompt and configuration changes, and dependency and security updates, validated against evaluation sets before release.
• Follow run cost: monitor token consumption and cost per service, alert on drift, and propose optimizations.
• Maintain Terraform, Helm, and Azure DevOps artifacts, ensuring consistency across DEV, TST, ACC, and PROD environments.
• Apply data protection rules: manage trace retention, erasure, and access restrictions as defined in each service’s data protection impact assessment.
• Maintain runbooks and documentation, and report service health to service owners and the AI CoE regularly.
• Deliver small enhancements and fixes; structural changes and new features will be handled by the Build team.

Why Join Keystone Solutions?

As a consultant with Keystone Solutions, you will experience:

• Consultancy Nature: Engage in on-site work, embodying the consultant identity.
• Dynamic Projects: Tackle diverse challenges across various client environments.
• Turbo-Charged Learning: Benefit from professional development and broad learning experiences.
• Skyrocketing Ambition: Enjoy a commitment to career growth within our framework.
• Values: Being a "K-Stone" means bringing our core values to every engagement.

If you are ready to tackle technical and strategic challenges in a dynamic consultancy environment, apply today at Keystone Solutions Career Portal.

Duration: 04/01/2027 - 31/12/2027 12 months • (full time)

Skills required:

• Azure DevOps - Level: Intermediate - Most recent: This year
• Azure Kubernetes Service - AKS - Level: Intermediate - Most recent: This year
• Azure OpenAI Service - Level: Intermediate - Most recent: This year
• LangChain - Level: Intermediate - Most recent: Any time
• LLM / Large Language Models - Level: Intermediate - Most recent: This year
• Python - Level: Intermediate - Most recent: This year
• RAG (Retrieval-Augmented Generation) - Level: Intermediate - Most recent: This year
• Terraform - Level: Intermediate - Most recent: This year
• Vector databases - Level: Intermediate - Most recent: This year

Language requirements:

Dutch or French
Level Native or bilingual proficiency
English
Level Full professional proficiency