Test tool use

Tool-use evaluation separates four questions that are often collapsed into a single score: did the model choose the right tool, produce valid arguments, handle the result, and complete the task?

Connect a repository

Open Tools, choose Deploy tool, and provide a public repository URL. Trunchbull validates the contract, builds the runtime, and deploys it to an isolated Worker.

Define the contract

Use JSON Schema to make tool arguments explicit. Tight schemas improve the quality of both model execution and deterministic grading.

export const searchIssues = {
  name: "search_issues",
  description: "Search repository issues by a concise query",
  inputSchema: {
    type: "object",
    properties: {
      query: { type: "string", minLength: 2 },
      limit: { type: "integer", minimum: 1, maximum: 20 },
    },
    required: ["query"],
    additionalProperties: false,
  },
}

Read the trace

Each invocation records:

• the tool selected by the model;
• the raw and parsed arguments;
• validation results;
• runtime output and errors;
• elapsed time and cost attribution;
• the model response that followed.

Failure classes

Classify failures as selection, arguments, runtime, interpretation, or final answer. This makes model and tool regressions visible as separate engineering problems.