Technical / stack

Stack. Inference, vectors, eval, agents.

Four tables. The serving frameworks that turn weights into tokens, the vector databases that hold your embeddings, the eval and observability tools that catch regressions, and the agent frameworks you may or may not need.

SOURCE STATUS SEE EACH TABLE // TABLES 4 // SORTABLE CLICK ANY COLUMN HEADER // VERIFY BEFORE CITING

ON THIS PAGE

Inference Frameworks
Vector Databases
Eval and Observability Tools
Agent Frameworks

TABLE // INFERENCE CHANGES QUARTERLY

Inference Frameworks

Servers that turn weights + GPUs into tokens. Pick by workload shape, not popularity.

Earlier reference snapshot · Source verification date not recorded. Confirm current information before use.

Name	License	PagedAttention	Continuous batching	Structured output	Quantization	Best fit
vLLM	Apache 2.0	yes	yes	yes	AWQ, GPTQ, FP8, INT4, INT8	High-throughput multi-tenant serving. The default choice if you have no other constraint.
TGI (HuggingFace Text Generation Inference)	Apache 2.0	yes	yes	limited	AWQ, GPTQ, EETQ, FP8	Teams already deep in the HuggingFace ecosystem. Solid OpenAI-compatible API.
SGLang	Apache 2.0	yes	yes	yes	AWQ, GPTQ, FP8, INT4	Workloads with heavy prefix reuse (agents, RAG, long system prompts). RadixAttention helps.
Ollama	MIT	no	no	yes	GGUF (Q2-Q8)	Local dev, laptops, single-user. Not a production server. Built on llama.cpp.
LMDeploy	Apache 2.0	yes	yes	limited	AWQ, GPTQ, W4A16, W8A8	TurboMind backend can edge vLLM on throughput for some models. Less community than vLLM.
llama.cpp	MIT	no	yes	yes	GGUF (Q2-Q8, mixed)	CPU and Apple Silicon inference. Heavy quantization. Edge devices. Not for multi-GPU serving.
NVIDIA TensorRT-LLM	Apache 2.0	yes	yes	limited	FP8, INT4 AWQ, INT8 SmoothQuant	Last-mile latency on NVIDIA hardware. Compile step is real. Worth it only if you've tuned vLLM and still need more.

TABLE // VECTOR DBs CHANGES QUARTERLY

Vector Databases

Storage for embeddings + fast nearest-neighbor search. Pick by deployment shape and existing stack.

Earlier reference snapshot · Source verification date not recorded. Confirm current information before use.

Name	Deployment	License	Metadata filter	Hybrid search	Authorization evidence	Best fit
pgvector	Postgres extension	PostgreSQL License	strong (full SQL)	partial	Not established here; verify the exact managed offering or agency system boundary.	You already run Postgres and your vector workload fits in one server. One database to back up, one to monitor.
Chroma	Self-host or managed	Apache 2.0	medium	no	Not established here; verify the exact managed offering or agency system boundary.	Prototypes and small RAG apps. Easy to start. Not where you want to be at scale.
Pinecone	Managed only	Proprietary	strong	yes	Not established here; verify the exact managed offering or agency system boundary.	You want zero ops and will pay for it. Serverless tier scales to zero. No self-host option.
Weaviate	Self-host or managed	BSD-3-Clause	strong	yes	Not established here; verify the exact managed offering or agency system boundary.	Hybrid search out of the box. Modules for embedding, generation, reranking. Heavier to operate than Qdrant.
Milvus	Self-host or managed (Zilliz)	Apache 2.0	strong	yes	Not established here; verify the exact managed offering or agency system boundary.	Billion-vector scale. Distributed by design. Operationally heavy. Don't pick it unless you need the scale.
Qdrant	Self-host or managed	Apache 2.0	strong	yes	Not established here; verify the exact managed offering or agency system boundary.	Rust-based, fast, good filter ergonomics. Strong default for self-host when pgvector isn't enough.
LanceDB	Embedded or managed	Apache 2.0	medium	partial	Not established here; verify the exact managed offering or agency system boundary.	Embedded (no server) vector DB. Good for desktop apps, notebooks, and multimodal data. Newer, with fewer war stories.

TABLE // EVAL + OBS CHANGES QUARTERLY

Eval and Observability Tools

Two adjacent categories. Eval gauges quality before you ship. Observability watches what happens after.

Earlier reference snapshot · Source verification date not recorded. Confirm current information before use.

Name	Category	License	Self-host	OpenTelemetry	Best fit
promptfoo	eval	MIT	yes	no	YAML-driven eval suites in CI. Local-first. Cheap and fast to set up. Good first eval tool.
Langfuse	both	MIT (core) + commercial	yes	yes	Self-hosted observability with eval features attached. Open core. Strong choice when you can't ship traces to a vendor.
DeepEval	eval	Apache 2.0	yes	no	Pytest-style eval framework with built-in metrics (faithfulness, hallucination, relevance). For Python teams.
Phoenix (Arize)	both	Elastic License 2.0	yes	yes	OpenTelemetry-native tracing and evals. Self-host or use Arize's commercial AX product. Good if you want OTel everywhere.
Helicone	observability	Apache 2.0 (core) + commercial SaaS	yes	partial	Proxy-based logging, drop-in for OpenAI/Anthropic calls. Fastest path to 'I can see my LLM calls'.
LangSmith	both	Commercial SaaS	partial (enterprise tier)	yes	If you're already on LangChain. Closed source. Self-host requires enterprise contract.
OpenAI Evals	eval	MIT	yes	no	Reference eval framework. Maintenance pace has slowed (_status as of 2026-06, verify before adopting_). Most teams pick promptfoo or DeepEval instead.

TABLE // AGENTS CHANGES QUARTERLY

Agent Frameworks

Loops, planners, tool-use orchestrators. "Pure Python" is still the right answer surprisingly often.

Earlier reference snapshot · Source verification date not recorded. Confirm current information before use.

Name	Paradigm	License	Language	Observability	Best fit
LangGraph	Graph	MIT	Python/TS	built-in (via LangSmith)	Stateful multi-step agents with explicit control flow. Heavy if all you need is a tool loop.
Haystack	Pipeline	Apache 2.0	Python	via library	RAG-shaped pipelines with reusable components. Strong document processing. Less agent-y than LangGraph.
CrewAI	Multi-agent	MIT	Python	via library	Role-based agent teams. Reads well in demos. Production teams often find the abstraction in the way.
AutoGen	Multi-agent	MIT (Microsoft)	Python/.NET	via library	Conversational multi-agent patterns. v0.4 rewrite changed the API substantially (_status as of 2026-06, check current version_).
OpenAI Agents SDK	Multi-agent	MIT	Python/TS	built-in	Successor to OpenAI Swarm. Tied to OpenAI's API. Use if you're committed to that provider.
Anthropic Claude Agent SDK	Multi-agent	MIT	Python/TS	built-in	First-party SDK for Claude agents. MCP integration. Use if you're committed to Anthropic.
Pure Python (no framework)	DIY	n/a	Python	DIY	A while loop, tool definitions in JSON, direct API calls. Wins more often than people admit.

A source-check date applies only to the facts in that table. Tables without a date retain older orientation data and have no recorded source verification date. Verify current prices, versions and capabilities with primary sources before spending or citing them. Licenses and brand names do not establish authorization.