Building Agents

Run local Ollama models, chat and create embeddings from the command line

Design agent evals with pass@k metrics, verifiers and LLM-as-a-Judge rubrics

Build event-driven agents that respond to webhooks, timers and new messages

Build coding agent harnesses with guardrails, tests and clear execution limits

Harden agents against loops, dropped streams, overflow and rate limit errors

Design agent eval tasks, verifiers and datasets that resist leakage

Build RAG pipelines with chunking, hybrid retrieval and reranking

Run and debug RAG evaluations on long file attachments using test fixtures

Choose embedding models and chunking strategies for semantic search and RAG

Combine vector and keyword search to improve retrieval in RAG systems

Design LLM apps and agents with LangChain 1.x and LangGraph

Build automated and human evals with benchmarks to measure LLM app quality

Build RAG pipelines with vector databases and semantic search for document Q&A

Design an AI agent architecture from requirements or audit an existing one

Add policy checks, access controls and trust gates to AI agent tool calls

Improve agent outputs with self-critique loops, rubrics and evaluator passes

Set up Arize AI integrations that store LLM provider credentials for evaluators

Set up Arize annotation configs, review queues and human labels on spans

Create, append to and export Arize datasets and eval examples with the ax CLI

Create LLM-as-judge evaluators on Arize and run them on spans or experiments

Run and compare Arize experiments to measure model and prompt performance

Add Arize AX tracing to an LLM app in two phases after analyzing the codebase

Generate shareable Arize UI links for traces, datasets, evaluators and queues

Export and inspect Arize traces and spans to debug what an LLM app did

Build agentic apps with the GitHub Copilot SDK, custom tools and MCP servers

Build Microsoft 365 Copilot declarative agents with TypeSpec and the Agents Toolkit

Score an AI workflow on prompts, context, tools, architecture and safety

Improve an AI app through evals: define criteria, build golden datasets and act on results

Register prompt-based agents in Azure AI Foundry from a local JSON manifest

Build a CopilotKit React app wired to a Foundry hosted agent over AG-UI

Create declarative agent definitions that connect to MCP servers

Deploy and manage agents built on MCP servers across their lifecycle

Build, review and explain Microsoft Agent Framework solutions in .NET and Python

Fetch Phoenix traces, review errors and inspect datasets and experiments from the CLI

Build and run Phoenix evaluators that score AI and LLM application outputs

Instrument LLM apps with OpenInference conventions and custom spans for Phoenix

Build, review and explain Semantic Kernel solutions in .NET and Python

Add GET, POST, PATCH and DELETE operations to a TypeSpec API plugin with adaptive cards

Generate a TypeSpec declarative agent with instructions and conversation starters

Generate a TypeSpec API plugin with REST operations, auth and adaptive cards

Check a proposed agent action against its approval before it runs

Compare offline AI evaluation runs, diagnose failing cases and track score changes

Set up continuous online evaluations that score live AI generations and traces

Find and categorize failure patterns in a production LLM app from real traces

Explore LLM traffic clusters and compare their cost, latency and trace behavior

Break down LLM spend by model, provider, user and feature to find cost spikes

Inspect code, LLM judge and sentiment evaluations and rerun them on generations

Debug LLM and agent traces to find what went wrong and why tools failed

Set up a CopilotKit agent that answers in Slack or Microsoft Teams end to end