--- name: mcp-builder description: Guide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK). license: Complete terms in LICENSE.txt --- # MCP Server Development Guide ## Overview Create MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. The quality of an MCP server is measured by how well it enables LLMs to accomplish real-world tasks. --- # Process ## 🚀 High-Level Workflow Creating a high-quality MCP server involves four main phases: ### Phase 1: Deep Research and Planning #### 1.1 Understand Modern MCP Design **API Coverage vs. Workflow Tools:** Balance comprehensive API endpoint coverage with specialized workflow tools. Workflow tools can be more convenient for specific tasks, while comprehensive coverage gives agents flexibility to compose operations. Performance varies by client—some clients benefit from code execution that combines basic tools, while others work better with higher-level workflows. When uncertain, prioritize comprehensive API coverage. **Tool Naming and Discoverability:** Clear, descriptive tool names help agents find the right tools quickly. Use consistent prefixes (e.g., `github_create_issue`, `github_list_repos`) and action-oriented naming. **Context Management:** Agents benefit from concise tool descriptions and the ability to filter/paginate results. Design tools that return focused, relevant data. Some clients support code execution which can help agents filter and process data efficiently. **Actionable Error Messages:** Error messages should guide agents toward solutions with specific suggestions and next steps. #### 1.2 Study MCP Protocol Documentation **Navigate the MCP specification:** Start with the sitemap to find relevant pages: `https://modelcontextprotocol.io/sitemap.xml` Then fetch specific pages with `.md` suffix for markdown format (e.g., `https://modelcontextprotocol.io/specification/draft.md`). Key pages to review: - Specification overview and architecture - Transport mechanisms (streamable HTTP, stdio) - Tool, resource, and prompt definitions #### 1.3 Study Framework Documentation **Recommended stack:** - **Language**: TypeScript (high-quality SDK support and good compatibility in many execution environments e.g. MCPB. Plus AI models are good at generating TypeScript code, benefiting from its broad usage, static typing and good linting tools) - **Transport**: Streamable HTTP for remote servers, using stateless JSON (simpler to scale and maintain, as opposed to stateful sessions and streaming responses). stdio for local servers. **Load framework documentation:** - **MCP Best Practices**: [📋 View Best Practices](./reference/mcp_best_practices.md) - Core guidelines **For TypeScript (recommended):** - **TypeScript SDK**: Use WebFetch to load `https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md` - [⚡ TypeScript Guide](./reference/node_mcp_server.md) - TypeScript patterns and examples **For Python:** - **Python SDK**: Use WebFetch to load `https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md` - [🐍 Python Guide](./reference/python_mcp_server.md) - Python patterns and examples #### 1.4 Plan Your Implementation **Understand the API:** Review the service's API documentation to identify key endpoints, authentication requirements, and data models. Use web search and WebFetch as needed. **Tool Selection:** Prioritize comprehensive API coverage. List endpoints to implement, starting with the most common operations. --- ### Phase 2: Implementation #### 2.1 Set Up Project Structure See language-specific guides for project setup: - [⚡ TypeScript Guide](./reference/node_mcp_server.md) - Project structure, package.json, tsconfig.json - [🐍 Python Guide](./reference/python_mcp_server.md) - Module organization, dependencies #### 2.2 Implement Core Infrastructure Create shared utilities: - API client with authentication - Error handling helpers - Response formatting (JSON/Markdown) - Pagination support #### 2.3 Implement Tools For each tool: **Input Schema:** - Use Zod (TypeScript) or Pydantic (Python) - Include constraints and clear descriptions - Add examples in field descriptions **Output Schema:** - Define `outputSchema` where possible for structured data - Use `structuredContent` in tool responses (TypeScript SDK feature) - Helps clients understand and process tool outputs **Tool Description:** - Concise summary of functionality - Parameter descriptions - Return type schema **Implementation:** - Async/await for I/O operations - Proper error handling with actionable messages - Support pagination where applicable - Return both text content and structured data when using modern SDKs **Annotations:** - `readOnlyHint`: true/false - `destructiveHint`: true/false - `idempotentHint`: true/false - `openWorldHint`: true/false --- ### Phase 3: Review and Test #### 3.1 Code Quality Review for: - No duplicated code (DRY principle) - Consistent error handling - Full type coverage - Clear tool descriptions #### 3.2 Build and Test **TypeScript:** - Run `npm run build` to verify compilation - Test with MCP Inspector: `npx @modelcontextprotocol/inspector` **Python:** - Verify syntax: `python -m py_compile your_server.py` - Test with MCP Inspector See language-specific guides for detailed testing approaches and quality checklists. --- ### Phase 4: Create Evaluations After implementing your MCP server, create comprehensive evaluations to test its effectiveness. **Load [✅ Evaluation Guide](./reference/evaluation.md) for complete evaluation guidelines.** #### 4.1 Understand Evaluation Purpose Use evaluations to test whether LLMs can effectively use your MCP server to answer realistic, complex questions. #### 4.2 Create 10 Evaluation Questions To create effective evaluations, follow the process outlined in the evaluation guide: 1. **Tool Inspection**: List available tools and understand their capabilities 2. **Content Exploration**: Use READ-ONLY operations to explore available data 3. **Question Generation**: Create 10 complex, realistic questions 4. **Answer Verification**: Solve each question yourself to verify answers #### 4.3 Evaluation Requirements Ensure each question is: - **Independent**: Not dependent on other questions - **Read-only**: Only non-destructive operations required - **Complex**: Requiring multiple tool calls and deep exploration - **Realistic**: Based on real use cases humans would care about - **Verifiable**: Single, clear answer that can be verified by string comparison - **Stable**: Answer won't change over time #### 4.4 Output Format Create an XML file with this structure: ```xml <evaluation> <qa_pair> <question>Find discussions about AI model launches with animal codenames. One model needed a specific safety designation that uses the format ASL-X. What number X was being determined for the model named after a spotted wild cat?</question> <answer>3</answer> </qa_pair> <!-- More qa_pairs... --> </evaluation> ``` --- # Reference Files ## 📚 Documentation Library Load these resources as needed during development: ### Core MCP Documentation (Load First) - **MCP Protocol**: Start with sitemap at `https://modelcontextprotocol.io/sitemap.xml`, then fetch specific pages with `.md` suffix - [📋 MCP Best Practices](./reference/mcp_best_practices.md) - Universal MCP guidelines including: - Server and tool naming conventions - Response format guidelines (JSON vs Markdown) - Pagination best practices - Transport selection (streamable HTTP vs stdio) - Security and error handling standards ### SDK Documentation (Load During Phase 1/2) - **Python SDK**: Fetch from `https://raw.githubusercontent.com/modelcontextprotocol/python-sdk/main/README.md` - **TypeScript SDK**: Fetch from `https://raw.githubusercontent.com/modelcontextprotocol/typescript-sdk/main/README.md` ### Language-Specific Implementation Guides (Load During Phase 2) - [🐍 Python Implementation Guide](./reference/python_mcp_server.md) - Complete Python/FastMCP guide with: - Server initialization patterns - Pydantic model examples - Tool registration with `@mcp.tool` - Complete working examples - Quality checklist - [⚡ TypeScript Implementation Guide](./reference/node_mcp_server.md) - Complete TypeScript guide with: - Project structure - Zod schema patterns - Tool registration with `server.registerTool` - Complete working examples - Quality checklist ### Evaluation Guide (Load During Phase 4) - [✅ Evaluation Guide](./reference/evaluation.md) - Complete evaluation creation guide with: - Question creation guidelines - Answer verification strategies - XML format specifications - Example questions and answers - Running an evaluation with the provided scripts FILE:reference/mcp_best_practices.md # MCP Server Best Practices ## Quick Reference ### Server Naming - **Python**: `{service}_mcp` (e.g., `slack_mcp`) - **Node/TypeScript**: `{service}-mcp-server` (e.g., `slack-mcp-server`) ### Tool Naming - Use snake_case with service prefix - Format: `{service}_{action}_{resource}` - Example: `slack_send_message`, `github_create_issue` ### Response Formats - Support both JSON and Markdown formats - JSON for programmatic processing - Markdown for human readability ### Pagination - Always respect `limit` parameter - Return `has_more`, `next_offset`, `total_count` - Default to 20-50 items ### Transport - **Streamable HTTP**: For remote servers, multi-client scenarios - **stdio**: For local integrations, command-line tools - Avoid SSE (deprecated in favor of streamable HTTP) --- ## Server Naming Conventions Follow these standardized naming patterns: **Python**: Use format `{service}_mcp` (lowercase with underscores) - Examples: `slack_mcp`, `github_mcp`, `jira_mcp` **Node/TypeScript**: Use format `{service}-mcp-server` (lowercase with hyphens) - Examples: `slack-mcp-server`, `github-mcp-server`, `jira-mcp-server` The name should be general, descriptive of the service being integrated, easy to infer from the task description, and without version numbers. --- ## Tool Naming and Design ### Tool Naming 1. **Use snake_case**: `search_users`, `create_project`, `get_channel_info` 2. **Include service prefix**: Anticipate that your MCP server may be used alongside other MCP servers - Use `slack_send_message` instead of just `send_message` - Use `github_create_issue` instead of just `create_issue` 3. **Be action-oriented**: Start with verbs (get, list, search, create, etc.) 4. **Be specific**: Avoid generic names that could conflict with other servers ### Tool Design - Tool descriptions must narrowly and unambiguously describe functionality - Descriptions must precisely match actual functionality - Provide tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) - Keep tool operations focused and atomic --- ## Response Formats All tools that return data should support multiple formats: ### JSON Format (`response_format="json"`) - Machine-readable structured data - Include all available fields and metadata - Consistent field names and types - Use for programmatic processing ### Markdown Format (`response_format="markdown"`, typically default) - Human-readable formatted text - Use headers, lists, and formatting for clarity - Convert timestamps to human-readable format - Show display names with IDs in parentheses - Omit verbose metadata --- ## Pagination For tools that list resources: - **Always respect the `limit` parameter** - **Implement pagination**: Use `offset` or cursor-based pagination - **Return pagination metadata**: Include `has_more`, `next_offset`/`next_cursor`, `total_count` - **Never load all results into memory**: Especially important for large datasets - **Default to reasonable limits**: 20-50 items is typical Example pagination response: ```json { "total": 150, "count": 20, "offset": 0, "items": [...], "has_more": true, "next_offset": 20 } ``` --- ## Transport Options ### Streamable HTTP **Best for**: Remote servers, web services, multi-client scenarios **Characteristics**: - Bidirectional communication over HTTP - Supports multiple simultaneous clients - Can be deployed as a web service - Enables server-to-client notifications **Use when**: - Serving multiple clients simultaneously - Deploying as a cloud service - Integration with web applications ### stdio **Best for**: Local integrations, command-line tools **Characteristics**: - Standard input/output stream communication - Simple setup, no network configuration needed - Runs as a subprocess of the client **Use when**: - Building tools for local development environments - Integrating with desktop applications - Single-user, single-session scenarios **Note**: stdio servers should NOT log to stdout (use stderr for logging) ### Transport Selection | Criterion | stdio | Streamable HTTP | |-----------|-------|-----------------| | **Deployment** | Local | Remote | | **Clients** | Single | Multiple | | **Complexity** | Low | Medium | | **Real-time** | No | Yes | --- ## Security Best Practices ### Authentication and Authorization **OAuth 2.1**: - Use secure OAuth 2.1 with certificates from recognized authorities - Validate access tokens before processing requests - Only accept tokens specifically intended for your server **API Keys**: - Store API keys in environment variables, never in code - Validate keys on server startup - Provide clear error messages when authentication fails ### Input Validation - Sanitize file paths to prevent directory traversal - Validate URLs and external identifiers - Check parameter sizes and ranges - Prevent command injection in system calls - Use schema validation (Pydantic/Zod) for all inputs ### Error Handling - Don't expose internal errors to clients - Log security-relevant errors server-side - Provide helpful but not revealing error messages - Clean up resources after errors ### DNS Rebinding Protection For streamable HTTP servers running locally: - Enable DNS rebinding protection - Validate the `Origin` header on all incoming connections - Bind to `127.0.0.1` rather than `0.0.0.0` --- ## Tool Annotations Provide annotations to help clients understand tool behavior: | Annotation | Type | Default | Description | |-----------|------|---------|-------------| | `readOnlyHint` | boolean | false | Tool does not modify its environment | | `destructiveHint` | boolean | true | Tool may perform destructive updates | | `idempotentHint` | boolean | false | Repeated calls with same args have no additional effect | | `openWorldHint` | boolean | true | Tool interacts with external entities | **Important**: Annotations are hints, not security guarantees. Clients should not make security-critical decisions based solely on annotations. --- ## Error Handling - Use standard JSON-RPC error codes - Report tool errors within result objects (not protocol-level errors) - Provide helpful, specific error messages with suggested next steps - Don't expose internal implementation details - Clean up resources properly on errors Example error handling: ```typescript try { const result = performOperation(); return { content: [{ type: "text", text: result }] }; } catch (error) { return { isError: true, content: [{ type: "text", text: `Error: ${error.message}. Try using filter='active_only' to reduce results.` }] }; } ``` --- ## Testing Requirements Comprehensive testing should cover: - **Functional testing**: Verify correct execution with valid/invalid inputs - **Integration testing**: Test interaction with external systems - **Security testing**: Validate auth, input sanitization, rate limiting - **Performance testing**: Check behavior under load, timeouts - **Error handling**: Ensure proper error reporting and cleanup --- ## Documentation Requirements - Provide clear documentation of all tools and capabilities - Include working examples (at least 3 per major feature) - Document security considerations - Specify required permissions and access levels - Document rate limits and performance characteristics FILE:reference/evaluation.md # MCP Server Evaluation Guide ## Overview This document provides guidance on creating comprehensive evaluations for MCP servers. Evaluations test whether LLMs can effectively use your MCP server to answer realistic, complex questions using only the tools provided. --- ## Quick Reference ### Evaluation Requirements - Create 10 human-readable questions - Questions must be READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE - Each question requires multiple tool calls (potentially dozens) - Answers must be single, verifiable values - Answers must be STABLE (won't change over time) ### Output Format ```xml <evaluation> <qa_pair> <question>Your question here</question> <answer>Single verifiable answer</answer> </qa_pair> </evaluation> ``` --- ## Purpose of Evaluations The measure of quality of an MCP server is NOT how well or comprehensively the server implements tools, but how well these implementations (input/output schemas, docstrings/descriptions, functionality) enable LLMs with no other context and access ONLY to the MCP servers to answer realistic and difficult questions. ## Evaluation Overview Create 10 human-readable questions requiring ONLY READ-ONLY, INDEPENDENT, NON-DESTRUCTIVE, and IDEMPOTENT operations to answer. Each question should be: - Realistic - Clear and concise - Unambiguous - Complex, requiring potentially dozens of tool calls or steps - Answerable with a single, verifiable value that you identify in advance ## Question Guidelines ### Core Requirements 1. **Questions MUST be independent** - Each question should NOT depend on the answer to any other question - Should not assume prior write operations from processing another question 2. **Questions MUST require ONLY NON-DESTRUCTIVE AND IDEMPOTENT tool use** - Should not instruct or require modifying state to arrive at the correct answer 3. **Questions must be REALISTIC, CLEAR, CONCISE, and COMPLEX** - Must require another LLM to use multiple (potentially dozens of) tools or steps to answer ### Complexity and Depth 4. **Questions must require deep exploration** - Consider multi-hop questions requiring multiple sub-questions and sequential tool calls - Each step should benefit from information found in previous questions 5. **Questions may require extensive paging** - May need paging through multiple pages of results - May require querying old data (1-2 years out-of-date) to find niche information - The questions must be DIFFICULT 6. **Questions must require deep understanding** - Rather than surface-level knowledge - May pose complex ideas as True/False questions requiring evidence - May use multiple-choice format where LLM must search different hypotheses 7. **Questions must not be solvable with straightforward keyword search** - Do not include specific keywords from the target content - Use synonyms, related concepts, or paraphrases - Require multiple searches, analyzing multiple related items, extracting context, then deriving the answer ### Tool Testing 8. **Questions should stress-test tool return values** - May elicit tools returning large JSON objects or lists, overwhelming the LLM - Should require understanding multiple modalities of data: - IDs and names - Timestamps and datetimes (months, days, years, seconds) - File IDs, names, extensions, and mimetypes - URLs, GIDs, etc. - Should probe the tool's ability to return all useful forms of data 9. **Questions should MOSTLY reflect real human use cases** - The kinds of information retrieval tasks that HUMANS assisted by an LLM would care about 10. **Questions may require dozens of tool calls** - This challenges LLMs with limited context - Encourages MCP server tools to reduce information returned 11. **Include ambiguous questions** - May be ambiguous OR require difficult decisions on which tools to call - Force the LLM to potentially make mistakes or misinterpret - Ensure that despite AMBIGUITY, there is STILL A SINGLE VERIFIABLE ANSWER ### Stability 12. **Questions must be designed so the answer DOES NOT CHANGE** - Do not ask questions that rely on "current state" which is dynamic - For example, do not count: - Number of reactions to a post - Number of replies to a thread - Number of members in a channel 13. **DO NOT let the MCP server RESTRICT the kinds of questions you create** - Create challenging and complex questions - Some may not be solvable with the available MCP server tools - Questions may require specific output formats (datetime vs. epoch time, JSON vs. MARKDOWN) - Questions may require dozens of tool calls to complete ## Answer Guidelines ### Verification 1. **Answers must be VERIFIABLE via direct string comparison** - If the answer can be re-written in many formats, clearly specify the output format in the QUESTION - Examples: "Use YYYY/MM/DD.", "Respond True or False.", "Answer A, B, C, or D and nothing else." - Answer should be a single VERIFIABLE value such as: - User ID, user name, display name, first name, last name - Channel ID, channel name - Message ID, string - URL, title - Numerical quantity - Timestamp, datetime - Boolean (for True/False questions) - Email address, phone number - File ID, file name, file extension - Multiple choice answer - Answers must not require special formatting or complex, structured output - Answer will be verified using DIRECT STRING COMPARISON ### Readability 2. **Answers should generally prefer HUMAN-READABLE formats** - Examples: names, first name, last name, datetime, file name, message string, URL, yes/no, true/false, a/b/c/d - Rather than opaque IDs (though IDs are acceptable) - The VAST MAJORITY of answers should be human-readable ### Stability 3. **Answers must be STABLE/STATIONARY** - Look at old content (e.g., conversations that have ended, projects that have launched, questions answered) - Create QUESTIONS based on "closed" concepts that will always return the same answer - Questions may ask to consider a fixed time window to insulate from non-stationary answers - Rely on context UNLIKELY to change - Example: if finding a paper name, be SPECIFIC enough so answer is not confused with papers published later 4. **Answers must be CLEAR and UNAMBIGUOUS** - Questions must be designed so there is a single, clear answer - Answer can be derived from using the MCP server tools ### Diversity 5. **Answers must be DIVERSE** - Answer should be a single VERIFIABLE value in diverse modalities and formats - User concept: user ID, user name, display name, first name, last name, email address, phone number - Channel concept: channel ID, channel name, channel topic - Message concept: message ID, message string, timestamp, month, day, year 6. **Answers must NOT be complex structures** - Not a list of values - Not a complex object - Not a list of IDs or strings - Not natural language text - UNLESS the answer can be straightforwardly verified using DIRECT STRING COMPARISON - And can be realistically reproduced - It should be unlikely that an LLM would return the same list in any other order or format ## Evaluation Process ### Step 1: Documentation Inspection Read the documentation of the target API to understand: - Available endpoints and functionality - If ambiguity exists, fetch additional information from the web - Parallelize this step AS MUCH AS POSSIBLE - Ensure each subagent is ONLY examining documentation from the file system or on the web ### Step 2: Tool Inspection List the tools available in the MCP server: - Inspect the MCP server directly - Understand input/output schemas, docstrings, and descriptions - WITHOUT calling the tools themselves at this stage ### Step 3: Developing Understanding Repeat steps 1 & 2 until you have a good understanding: - Iterate multiple times - Think about the kinds of tasks you want to create - Refine your understanding - At NO stage should you READ the code of the MCP server implementation itself - Use your intuition and understanding to create reasonable, realistic, but VERY challenging tasks ### Step 4: Read-Only Content Inspection After understanding the API and tools, USE the MCP server tools: - Inspect content using READ-ONLY and NON-DESTRUCTIVE operations ONLY - Goal: identify specific content (e.g., users, channels, messages, projects, tasks) for creating realistic questions - Should NOT call any tools that modify state - Will NOT read the code of the MCP server implementation itself - Parallelize this step with individual sub-agents pursuing independent explorations - Ensure each subagent is only performing READ-ONLY, NON-DESTRUCTIVE, and IDEMPOTENT operations - BE CAREFUL: SOME TOOLS may return LOTS OF DATA which would cause you to run out of CONTEXT - Make INCREMENTAL, SMALL, AND TARGETED tool calls for exploration - In all tool call requests, use the `limit` parameter to limit results (<10) - Use pagination ### Step 5: Task Generation After inspecting the content, create 10 human-readable questions: - An LLM should be able to answer these with the MCP server - Follow all question and answer guidelines above ## Output Format Each QA pair consists of a question and an answer. The output should be an XML file with this structure: ```xml <evaluation> <qa_pair> <question>Find the project created in Q2 2024 with the highest number of completed tasks. What is the project name?</question> <answer>Website Redesign</answer> </qa_pair> <qa_pair> <question>Search for issues labeled as "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question> <answer>sarah_dev</answer> </qa_pair> <qa_pair> <question>Look for pull requests that modified files in the /api directory and were merged between January 1 and January 31, 2024. How many different contributors worked on these PRs?</question> <answer>7</answer> </qa_pair> <qa_pair> <question>Find the repository with the most stars that was created before 2023. What is the repository name?</question> <answer>data-pipeline</answer> </qa_pair> </evaluation> ``` ## Evaluation Examples ### Good Questions **Example 1: Multi-hop question requiring deep exploration (GitHub MCP)** ```xml <qa_pair> <question>Find the repository that was archived in Q3 2023 and had previously been the most forked project in the organization. What was the primary programming language used in that repository?</question> <answer>Python</answer> </qa_pair> ``` This question is good because: - Requires multiple searches to find archived repositories - Needs to identify which had the most forks before archival - Requires examining repository details for the language - Answer is a simple, verifiable value - Based on historical (closed) data that won't change **Example 2: Requires understanding context without keyword matching (Project Management MCP)** ```xml <qa_pair> <question>Locate the initiative focused on improving customer onboarding that was completed in late 2023. The project lead created a retrospective document after completion. What was the lead's role title at that time?</question> <answer>Product Manager</answer> </qa_pair> ``` This question is good because: - Doesn't use specific project name ("initiative focused on improving customer onboarding") - Requires finding completed projects from specific timeframe - Needs to identify the project lead and their role - Requires understanding context from retrospective documents - Answer is human-readable and stable - Based on completed work (won't change) **Example 3: Complex aggregation requiring multiple steps (Issue Tracker MCP)** ```xml <qa_pair> <question>Among all bugs reported in January 2024 that were marked as critical priority, which assignee resolved the highest percentage of their assigned bugs within 48 hours? Provide the assignee's username.</question> <answer>alex_eng</answer> </qa_pair> ``` This question is good because: - Requires filtering bugs by date, priority, and status - Needs to group by assignee and calculate resolution rates - Requires understanding timestamps to determine 48-hour windows - Tests pagination (potentially many bugs to process) - Answer is a single username - Based on historical data from specific time period **Example 4: Requires synthesis across multiple data types (CRM MCP)** ```xml <qa_pair> <question>Find the account that upgraded from the Starter to Enterprise plan in Q4 2023 and had the highest annual contract value. What industry does this account operate in?</question> <answer>Healthcare</answer> </qa_pair> ``` This question is good because: - Requires understanding subscription tier changes - Needs to identify upgrade events in specific timeframe - Requires comparing contract values - Must access account industry information - Answer is simple and verifiable - Based on completed historical transactions ### Poor Questions **Example 1: Answer changes over time** ```xml <qa_pair> <question>How many open issues are currently assigned to the engineering team?</question> <answer>47</answer> </qa_pair> ``` This question is poor because: - The answer will change as issues are created, closed, or reassigned - Not based on stable/stationary data - Relies on "current state" which is dynamic **Example 2: Too easy with keyword search** ```xml <qa_pair> <question>Find the pull request with title "Add authentication feature" and tell me who created it.</question> <answer>developer123</answer> </qa_pair> ``` This question is poor because: - Can be solved with a straightforward keyword search for exact title - Doesn't require deep exploration or understanding - No synthesis or analysis needed **Example 3: Ambiguous answer format** ```xml <qa_pair> <question>List all the repositories that have Python as their primary language.</question> <answer>repo1, repo2, repo3, data-pipeline, ml-tools</answer> </qa_pair> ``` This question is poor because: - Answer is a list that could be returned in any order - Difficult to verify with direct string comparison - LLM might format differently (JSON array, comma-separated, newline-separated) - Better to ask for a specific aggregate (count) or superlative (most stars) ## Verification Process After creating evaluations: 1. **Examine the XML file** to understand the schema 2. **Load each task instruction** and in parallel using the MCP server and tools, identify the correct answer by attempting to solve the task YOURSELF 3. **Flag any operations** that require WRITE or DESTRUCTIVE operations 4. **Accumulate all CORRECT answers** and replace any incorrect answers in the document 5. **Remove any `<qa_pair>`** that require WRITE or DESTRUCTIVE operations Remember to parallelize solving tasks to avoid running out of context, then accumulate all answers and make changes to the file at the end. ## Tips for Creating Quality Evaluations 1. **Think Hard and Plan Ahead** before generating tasks 2. **Parallelize Where Opportunity Arises** to speed up the process and manage context 3. **Focus on Realistic Use Cases** that humans would actually want to accomplish 4. **Create Challenging Questions** that test the limits of the MCP server's capabilities 5. **Ensure Stability** by using historical data and closed concepts 6. **Verify Answers** by solving the questions yourself using the MCP server tools 7. **Iterate and Refine** based on what you learn during the process --- # Running Evaluations After creating your evaluation file, you can use the provided evaluation harness