Production AI stack signals for DACH/EU B2B teams

Model registry + eval gates

Versioned models with promotion criteria tied to offline and online evals.

MLflow

Open-source model registry and experiment tracking - default starting point for mid-market MLOps.

Feature store with lineage

Centralized features with training/serving parity and lineage to source tables.

Feast

Open feature store for point-in-time correct training and low-latency serving.

Continuous model retraining

Automated retrain pipelines triggered by data drift or schedule.

LLM gateway + cost tags

Single ingress for models with routing, caching, rate limits, and per-team cost tags.

LiteLLM

Self-hosted LLM proxy - OpenAI-compatible API, budgets, routing, and 100+ providers.

Portkey

Managed LLM gateway with guardrails, observability, and enterprise support.

RAG evaluation harness

Golden-set evals for retrieval quality, faithfulness, and latency before release.

Langfuse

LLM traces, datasets, evals, and prompt management - self-host or cloud.

Promptfoo

CLI/CI evals for prompts, RAG, red-team, and provider comparison.

Prompt & tool registry

Versioned prompts and tool definitions with approval workflow.

AI-specific observability

Traces, metrics, and logs for inference latency, error rates, and quality - not just infra uptime.

OpenTelemetry (GenAI)

Vendor-neutral tracing with GenAI semantic conventions for tokens, models, and retrieval.

GPU FinOps attribution

Tag GPU and inference spend by team, model, and environment.

OpenCost / Kubecost

Kubernetes cost allocation by namespace, label, and GPU workload.

Inference routing by cost/latency

Route requests to smaller/cheaper models when quality thresholds allow.

Golden paths for AI teams

Opinionated templates for deploy, observe, and rollback AI workloads.

Backstage

Internal developer portal for service catalog and software templates.

GitOps for ML manifests

Declarative deploys for models, configs, and infra with PR-based promotion.

Argo CD

GitOps continuous delivery for Kubernetes - including ML serving manifests.

Kubernetes for AI workloads

Running training and inference on shared K8s clusters with GPU scheduling.

KServe

K8s-native model serving with canary, scale-to-zero, and multi-framework support.

EU AI Act evidence packs

Engineering artifacts - logs, eval reports, change history - mapped to Annex IV-style documentation.

PII redaction at inference boundary

Scan and redact sensitive fields before prompts leave your trust boundary.

Microsoft Presidio

Open-source PII detection and anonymization for text pipelines.

Coding agents without harness

Letting agents commit without spec, lint, test, and review gates in the loop.

Shadow MCP integrations

Unapproved MCP servers connecting agents to production data and external APIs.

Terraform AI landing zones

Reusable IaC modules for network, identity, logging, and tagging baselines before the first production model.

Terraform / OpenTofu

Declarative IaC for cloud baselines - default for mid-market platform teams in DACH.

Production vector store

Versioned embeddings store with backup, tenancy, and filtered retrieval SLOs - not a laptop Chroma dump.

Qdrant

Open-source vector DB optimized for filtered search and self-host or cloud deployment.

pgvector

Postgres embeddings extension - lowest-friction vector store for mid-market teams already on Postgres.

Pinecone

Managed vector database for teams that want RAG index ops off their plate.

LLM input/output guardrails

Programmable rails that block jailbreaks, off-topic prompts, and unsafe outputs before users see them.

NeMo Guardrails

NVIDIA open-source conversational guardrails with Colang policy definitions.

Guardrails AI

Output validators and correctors for schemas, toxicity, and PII in LLM responses.

Prompt injection defenses

Layered controls against indirect and direct injection - especially for agents with tools.

Lakera Guard

Managed detection for prompt injection and content risk on LLM traffic.

LLM API key secret scanning

CI and repo scanning that blocks committed provider keys, plus rotation playbooks.

Ragas / DeepEval metric suites

Automated faithfulness and retrieval metrics wired into CI for every RAG release.

Ragas

Open-source RAG evaluation metrics for faithfulness, context quality, and answer relevancy.

DeepEval

Pytest-native LLM evaluation framework for correctness, safety, and RAG scorers.

Document ingestion pipelines

Production parsing, chunking, and re-index jobs for PDFs, tables, and messy enterprise docs.

Unstructured

Document ETL for LLM apps - PDFs, tables, images, and layout-aware partitioning.

Canary prompt & model releases

Shadow or percentage rollouts for prompt/model changes with eval and latency abort criteria.

Self-hosted inference (vLLM)

Serve open-weight models on your GPUs with OpenAI-compatible APIs and PagedAttention throughput.

vLLM

High-throughput OSS inference engine with OpenAI-compatible endpoints.

Dataset versioning (DVC)

Git-linked dataset and artifact versions so every model can name its training data.

DVC

Data Version Control - Git workflows for datasets, models, and pipelines.

Model packaging with BentoML

Package classical ML and LLM runners as versioned deployable services.

BentoML

Open framework to build, ship, and scale model inference APIs.

ML/LLM pipeline orchestration

Durable scheduled workflows for ingest, train, eval, and reindex - not cron on a laptop.

Prefect

Python-native workflow orchestration for data and ML jobs.

Data quality gates before train

Expectation suites that block training or retrain when schemas and distributions break.

Great Expectations

Data quality expectations as code for warehouses and training inputs.

Open LLM observability (Phoenix)

OSS traces, embeddings visualization, and eval datasets as a Langfuse peer or complement.

Arize Phoenix

Open-source AI observability for traces, evaluations, and embedding analysis.

Redis semantic / response cache

Low-latency cache layer for near-duplicate LLM queries - often Redis behind the gateway.

FOCUS-aligned AI cost export

Normalize cloud + LLM spend into a FOCUS-style cost dataset for showback and anomaly alerts.

Flux CD for AI workloads

CNCF GitOps alternative to Argo CD for reconciling ML and inference manifests.

Flux CD

GitOps toolkit for continuous delivery on Kubernetes.