Tagged llm

Build automated and human evals with benchmarks to measure LLM app quality

Compile docs, notes and code into a linked LLM wiki with an index and concept pages

Set up Arize AI integrations that store LLM provider credentials for evaluators

Create LLM-as-judge evaluators on Arize and run them on spans or experiments

Add Arize AX tracing to an LLM app in two phases after analyzing the codebase

Optimize LLM prompts using production trace data, evaluations and annotations

Export and inspect Arize traces and spans to debug what an LLM app did

Create an llms.txt file that describes a repository for language models

Build and run Phoenix evaluators that score AI and LLM application outputs

Instrument LLM apps with OpenInference conventions and custom spans for Phoenix

Update an existing llms.txt file so it matches recent documentation changes

Find the most expensive users in AI observability and explain their LLM cost drivers

Find and categorize failure patterns in a production LLM app from real traces

Explore LLM traffic clusters and compare their cost, latency and trace behavior

Break down LLM spend by model, provider, user and feature to find cost spikes

Inspect code, LLM judge and sentiment evaluations and rerun them on generations

Debug LLM and agent traces to find what went wrong and why tools failed

Stream canonical use cases of an AI feature to Slack with an LLM judge

Monitor LLM traces for cost, latency, error and eval regressions

Red-team LLM apps in CI with Promptfoo and DeepTeam against OWASP LLM Top 10 presets

Add Llama Guard, NeMo Guardrails and LLM Guard filters to screen prompts and responses

Flag prompt injection in LLM inputs using regex, heuristics and a DeBERTa classifier

Filter LLM inputs and outputs with NeMo Guardrails, PII checks and Guardrails AI

Run multi-turn adversarial tests on LLM apps with Microsoft PyRIT orchestrators

Probe LLM endpoints for jailbreaks, prompt injection and data leakage with garak

Test LLM apps for system prompt leakage with direct and encoded extraction tricks

Test and benchmark LLM agents for behavior, reliability and production

Build AI product features with LLM patterns, RAG, prompt design and cost control

Manage LLM context windows with summarizing, trimming and routing

Add short-term, long-term and entity memory to chat-based LLM apps

Design and run evals for LLM, agent and retrieval systems with clear grading

Trace, evaluate and manage prompts for LLM apps with Langfuse

Apply production patterns for RAG pipelines, agents and LLM monitoring

Evaluate LLM apps with automated metrics, human review and A/B tests

Run LLM systems with RAG, embeddings, fine-tuning and quality evals

Cut LLM cost and latency with prompt caching and response caching

Design LLM prompts with structure, few-shot examples and output formats

Optimize LLM prompts with few-shot, chain-of-thought and structured outputs

Publish knowledge pages as password-protected HTML without any LLM calls

Assess AI and LLM systems for prompt injection, jailbreaks and tool abuse

Score and rank parallel agent results by metric or LLM judge

Cut LLM API spend with model choice, prompt caching and token budgets

Build an Obsidian second brain that an LLM keeps linked and current

Optimize prompts and score LLM outputs and RAG retrieval against eval sets

Browse API cookbook examples and tutorials for building LLM-powered apps

Audit and fix site readiness for AI tools, including llms.txt, WebMCP and robots rules

Track brand citation rates across ChatGPT, Perplexity and other LLMs over time

Meter LLM token use and screen prompts with an Apigee proxy in front of agents

Deploy LLM inference servers on GKE with GPU or TPU resources