TerminalBench 2.0
Trunchbull portEvaluates whether AI agents can autonomously complete 89 hard, realistic, end-to-end tasks in containerized terminal environments, using task-specific verifiers.
89 task previewsagent · sandboxedApache-2.0version 2.0 / 2fd12b8
Run on your models
Requires an isolated Linux sandbox. Some cases also require a task-specific container image.
What it measures
TerminalBench evaluates agents on real terminal work: installing software, repairing repositories, transforming files, and operating system tools. Each case includes an instruction, an environment, and a verifier.
Trunchbull imports the Harbor task format and marks compatibility per case. Portable cases run in the generic Trunchbull sandbox; image-specific cases require their published environment to be deployed.
trunchbull port
Trunchbull packages all 89 upstream Harbor tasks as one immutable benchmark release while preserving the original project, commit, license, case metadata, and verifier assets.
Tasks
89 in this release| id | task | runtime |
|---|---|---|
| terminal-bench/adaptive-rejection-sampler | adaptive-rejection-samplerEvaluates the ability to implement an adaptive rejection sampler in R with proper statistical algorithms, modular design, input validation, log-concavity checking, and formal testing.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/bn-fit-modify | bn-fit-modifyEvaluates the ability to recover a Bayesian Network DAG structure from data, perform causal interventions, and sample from the modified network.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/break-filter-js-from-html | break-filter-js-from-htmlEvaluates the agent's ability to bypass an HTML sanitization filter by crafting malicious HTML that triggers JavaScript execution after filtering.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/build-cython-ext | build-cython-extEvaluates the ability to compile and install a Python package with Cython extensions from source while fixing NumPy 2.x compatibility issues.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/build-pmars | build-pmarsEvaluates the ability to build pMARS from Debian source packages with X11 support disabled, requiring Makefile modification and headless compilation.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/build-pov-ray | build-pov-rayEvaluates the ability to locate, download, patch, and compile legacy POV-Ray 2.2 raytracer from 1990s source archives on a modern system.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/caffe-cifar-10 | caffe-cifar-10Evaluates the ability to install and configure BVLC Caffe 1.0.0, train a CNN on CIFAR-10 for exactly 500 iterations in CPU-only mode, and achieve specified accuracy thresholds.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/cancel-async-tasks | cancel-async-tasksEvaluates the ability to implement async task concurrency control with proper cleanup on cancellation, including the edge case of queued tasks.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/chess-best-move | chess-best-moveEvaluates the agent's ability to analyze a chess position from an image, use a chess engine to find the best move(s), and handle multiple valid solutions.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/circuit-fibsqrt | circuit-fibsqrtEvaluates the agent's ability to implement complex mathematical functions (Fibonacci of integer square root) using only combinational and sequential logic gates in a hardware description format.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/cobol-modernization | cobol-modernizationEvaluates the ability to reverse-engineer and reimplement a COBOL program's business logic in Python with exact output reproduction.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/code-from-image | code-from-imageEvaluates an agent's ability to extract code from an image using OCR or vision models, implement the pseudocode logic with cryptographic hashing, and produce the correct output.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/compile-compcert | compile-compcertEvaluates the ability to build the CompCert verified C compiler from source with proper configuration for the host architecture and dependencies.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/configure-git-webserver | configure-git-webserverEvaluates the ability to configure a Git server with automatic deployment to an nginx web server using post-receive hooks.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/constraints-scheduling | constraints-schedulingFind an optimal 1-hour meeting slot for three people with complex availability constraints by parsing ICS calendars and applying constraint satisfaction with tie-breaking preferences.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/count-dataset-tokens | count-dataset-tokensEvaluates the ability to count tokens in a filtered HuggingFace dataset using a specific tokenizer.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/crack-7z-hash | crack-7z-hashEvaluates the ability to crack a password-protected 7z archive using John the Ripper and extract secret contents.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/custom-memory-heap-crash | custom-memory-heap-crashEvaluates the ability to debug and fix a C++ program that crashes in release mode due to a static initialization order issue with custom memory allocators and STL locale facets.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/db-wal-recovery | db-wal-recoveryTests the ability to decrypt an XOR-encrypted SQLite WAL file and recover complete database contents including write-ahead log changes.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/distribution-search | distribution-searchTests the ability to find a probability distribution satisfying precise dual KL divergence constraints through numerical optimization.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/dna-assembly | dna-assemblyEvaluates the ability to design PCR primers for Golden Gate assembly by applying molecular biology knowledge and bioinformatics tools to meet complex cloning constraints.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/dna-insert | dna-insertEvaluates the ability to design PCR primers for site-directed mutagenesis by analyzing plasmid sequences and applying molecular biology constraints on primer length and melting temperature.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/extract-elf | extract-elfEvaluates ability to parse ELF binary format and extract memory values from executable sections using Node.js.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/extract-moves-from-video | extract-moves-from-videoEvaluates the agent's ability to download a YouTube video, extract text commands through OCR or transcription, and produce a formatted text file with 90% accuracy.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/feal-differential-cryptanalysis | feal-differential-cryptanalysisEvaluates the ability to implement differential cryptanalysis on a FEAL-like cipher to recover a round key through chosen plaintext attacks.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/feal-linear-cryptanalysis | feal-linear-cryptanalysisEvaluates the ability to perform linear cryptanalysis on a FEAL-like cipher to recover encryption keys from known plaintext-ciphertext pairs.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/filter-js-from-html | filter-js-from-htmlEvaluates the agent's ability to create a robust XSS filter that removes JavaScript from HTML files while preserving legitimate HTML structure and content.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/financial-document-processor | financial-document-processorEvaluates OCR, document classification, financial data extraction from mixed JPG/PDF documents, and CSV generation with totals.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/fix-code-vulnerability | fix-code-vulnerabilityEvaluates the ability to identify and fix a CRLF injection vulnerability (CWE-93) in HTTP header handling code by adding input validation to reject control characters.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/fix-git | fix-gitEvaluates the ability to recover lost Git commits from a detached HEAD state and merge them back into the master branch.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/fix-ocaml-gc | fix-ocaml-gcEvaluates ability to debug and fix a runtime crash in the OCaml garbage collector's C implementation, requiring low-level debugging skills and understanding of compiler internals.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/gcode-to-text | gcode-to-textTests the agent's ability to extract and decode text from a 3D printer G-code file by parsing movement commands, rendering them visually, and performing OCR.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/git-leak-recovery | git-leak-recoveryEvaluates the ability to recover secrets from unreachable git objects and completely remove them from repository history while preserving legitimate commits.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/git-multibranch | git-multibranchEvaluates the ability to set up a Git server with SSH authentication, implement post-receive hooks for automated multi-branch deployment, and configure Nginx to serve branch-specific content over HTTPS.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/gpt2-codegolf | gpt2-codegolfEvaluates the ability to implement a minimal, dependency-free C program that performs GPT-2 inference from TensorFlow checkpoints in under 5000 bytes.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/headless-terminal | headless-terminalImplement a Python class that provides a headless terminal interface supporting interactive bash shells, modifier keys, startup file sourcing, and state persistence between commands.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/hf-model-inference | hf-model-inferenceEvaluates the ability to download a Hugging Face transformer model, create a Flask API for sentiment analysis, and run the service in the background with proper error handling.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/install-windows-3.11 | install-windows-3.11Evaluates the ability to configure and run Windows 3.11 in QEMU with VNC display, web interface, and programmatic keyboard control for automated testing.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/kv-store-grpc | kv-store-grpcEvaluates the ability to build and deploy a gRPC-based key-value store server with Protocol Buffers, including service definition, code generation, implementation, and background process management.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/large-scale-text-editing | large-scale-text-editingEvaluates the ability to efficiently transform a 1-million-row CSV file using keystroke-efficient Vim macros with strict command restrictions.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/largest-eigenval | largest-eigenvalEvaluates the agent's ability to optimize eigenvalue computation by implementing a faster alternative to numpy's default algorithm while maintaining mathematical correctness.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/llm-inference-batching-scheduler | llm-inference-batching-schedulerImplement a shape-aware batching scheduler for static-graph LLM inference that optimally packs requests into batches while meeting strict performance thresholds on cost, latency, and padding.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/log-summary-date-ranges | log-summary-date-rangesEvaluates the ability to analyze date-stamped log files, calculate counts across multiple date ranges, and generate structured CSV output.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/mailman | mailmanEvaluates the ability to configure a functional mailing list server by integrating postfix and mailman3 with proper join/leave/announce workflows.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/make-doom-for-mips | make-doom-for-mipsEvaluates ability to cross-compile the DOOM game engine for MIPS architecture using LLVM toolchain and verify execution in a JavaScript emulator.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/make-mips-interpreter | make-mips-interpreterImplement a complete MIPS interpreter in JavaScript that can execute a DOOM ELF binary, handle system calls, and render the first game frame correctly.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/mcmc-sampling-stan | mcmc-sampling-stanEvaluates the ability to implement and run a hierarchical Bayesian model using R and Stan, including package installation, model specification with custom priors, MCMC sampling configuration, and posterior inference.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/merge-diff-arc-agi-task | merge-diff-arc-agi-taskEvaluates git bundle merging, conflict resolution, and ARC-AGI style pattern recognition by requiring agents to fetch two git bundles, merge branches, and implement a generalizable transformation function from input/output examples.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/model-extraction-relu-logits | model-extraction-relu-logitsExtracts hidden layer weights from a black-box ReLU neural network by querying outputs and identifying critical points where neurons activate.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/modernize-scientific-stack | modernize-scientific-stackEvaluates the ability to migrate legacy Python 2.7 scientific computing code to modern Python 3 with proper dependencies and data processing.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/mteb-leaderboard | mteb-leaderboardEvaluates the ability to research and identify the best-performing embedding model on the Scandinavian MTEB leaderboard using data science tools and techniques.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/mteb-retrieve | mteb-retrieveEvaluates the agent's ability to perform semantic text retrieval using MTEB embeddings, computing cosine similarities and correctly ranking documents to find the 5th most similar match to a query.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/multi-source-data-merger | multi-source-data-mergerEvaluates an agent's ability to merge multi-format data sources (JSON, CSV, Parquet) with inconsistent schemas, applying field mappings and priority-based conflict resolution to produce standardized outputs.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/nginx-request-logging | nginx-request-loggingEvaluates the ability to install and configure Nginx with advanced request logging, rate limiting, and custom error pages.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/openssl-selfsigned-cert | openssl-selfsigned-certEvaluates an agent's ability to generate self-signed TLS certificates using OpenSSL, manage cryptographic keys with proper permissions, and create verification scripts.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/overfull-hbox | overfull-hboxEvaluates the ability to fix LaTeX overfull hbox warnings by replacing words with valid synonyms while satisfying compilation and constraint requirements.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/password-recovery | password-recoveryEvaluates an agent's ability to perform digital forensics by recovering a deleted password from fragmented data within a disk image using command-line tools.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/path-tracing | path-tracingEvaluates the ability to reverse-engineer and implement a path tracing renderer in C by analyzing a reference image and recreating it algorithmically.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/path-tracing-reverse | path-tracing-reverseEvaluates the ability to reverse-engineer a compiled path tracing renderer and recreate functionally identical C source code under size constraints.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/polyglot-c-py | polyglot-c-pyCreate a single polyglot source file that computes Fibonacci numbers when executed as both Python 3 and C code.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/polyglot-rust-c | polyglot-rust-cEvaluates the ability to write a polyglot program that compiles and runs correctly as both Rust and C++ code, computing Fibonacci numbers.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/portfolio-optimization | portfolio-optimizationEvaluates the ability to implement a high-performance C extension for Python that performs portfolio risk and return calculations at least 1.2x faster than a pure Python baseline while maintaining numerical accuracy.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/protein-assembly | protein-assemblyEvaluates the ability to design a fusion protein gBlock by querying bioinformatics APIs, selecting proteins based on spectral properties, and applying codon optimization with GC content constraints.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/prove-plus-comm | prove-plus-commEvaluates the ability to complete an incomplete Coq proof of addition commutativity using inductive reasoning and formal verification tactics.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/pypi-server | pypi-serverEvaluates the ability to create a Python package, build it, set up a local PyPI server, and make the package installable from the server.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/pytorch-model-cli | pytorch-model-cliEvaluates the ability to convert PyTorch model weights to JSON, implement neural network inference in C, and create a command-line tool for MNIST digit prediction.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/pytorch-model-recovery | pytorch-model-recoveryEvaluates the ability to reverse-engineer a PyTorch Transformer model architecture from a state dictionary, load pre-trained weights, and selectively fine-tune specific layers to improve performance on a dataset.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/qemu-alpine-ssh | qemu-alpine-sshEvaluates the ability to start an Alpine Linux VM in QEMU and configure SSH server access with proper networking and authentication.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/qemu-startup | qemu-startupEvaluates the agent's ability to configure and start a QEMU virtual machine with telnet-accessible serial console, requiring knowledge of QEMU command-line options, network configuration, and system readiness verification.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/query-optimize | query-optimizeEvaluates the ability to optimize a slow SQL query with correlated subqueries by rewriting it using CTEs and window functions while preserving exact output.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/raman-fitting | raman-fittingEvaluates the ability to fit Lorentzian curves to Raman spectroscopy data, extract peak parameters, and perform scientific data analysis using Python.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/regex-chess | regex-chessEvaluates the ability to implement a complete chess move generator using only regular expression transformations on FEN notation.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/regex-log | regex-logTests the ability to construct a complex regular expression that matches dates in log lines containing valid IPv4 addresses while handling edge cases and boundary conditions.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/reshard-c4-data | reshard-c4-dataEvaluates the ability to create Python scripts for bidirectional data resharding with file size and directory constraints, using proper dependency management.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/rstan-to-pystan | rstan-to-pystanEvaluates the ability to convert an RStan Gaussian Process script to functionally equivalent PyStan 3.10.0 code, including complex installation, hyperparameter mapping, and numerical verification of posterior estimates.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/sam-cell-seg | sam-cell-segEvaluates the ability to implement a histopathology image segmentation pipeline using MobileSAM to convert rectangular cell masks to precise polyline contours.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/sanitize-git-repo | sanitize-git-repoEvaluates the ability to identify and sanitize sensitive API keys and tokens from a Git repository by replacing them with placeholders without modifying unrelated files.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/schemelike-metacircular-eval | schemelike-metacircular-evalEvaluates the ability to implement a metacircular evaluator in Scheme that can interpret itself and a comprehensive suite of Scheme programs.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/sparql-university | sparql-universityEvaluates the ability to write complex SPARQL queries with multiple constraints, aggregations, and date filtering against an RDF knowledge graph.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/sqlite-db-truncate | sqlite-db-truncateEvaluates the ability to recover data from a corrupted SQLite database using binary file analysis and data recovery techniques.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/sqlite-with-gcov | sqlite-with-gcovEvaluates the ability to compile SQLite from source with gcov instrumentation and make it available in the system PATH.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/torch-pipeline-parallelism | torch-pipeline-parallelismEvaluates the ability to implement pipeline parallel training for LLaMA using PyTorch distributed primitives with all-forward-all-backward scheduling.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/torch-tensor-parallelism | torch-tensor-parallelismEvaluates the ability to implement tensor parallelism for PyTorch linear layers with correct weight sharding, distributed forward/backward passes, and gradient computation across multiple processes.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/train-fasttext | train-fasttextTrain a FastText text classification model on Yelp review data that achieves >0.62 accuracy while staying under 150MB in size.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/tune-mjcf | tune-mjcfEvaluates the ability to optimize MuJoCo physics simulation parameters to achieve a 40% speedup while maintaining physical accuracy within specified tolerances.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/video-processing | video-processingEvaluates the ability to build a computer vision script that analyzes hurdle jump videos and extracts takeoff/landing frame numbers using OpenCV.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/vulnerable-secret | vulnerable-secretEvaluates the agent's ability to analyze a binary executable, identify and exploit a buffer overflow vulnerability to bypass authentication, and extract a hidden secret flag.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/winning-avg-corewars | winning-avg-corewarsEvaluates the ability to write a competitive CoreWars Redcode warrior that achieves specific win rates against five diverse opponent strategies.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
| terminal-bench/write-compressor | write-compressorEvaluates the agent's ability to reverse-engineer a custom compression format and write a compatible compressor program.tools: sandbox.files, sandbox.commands, sandbox.processes | sandbox |
Tools
3 requested| tool | description | kind | required |
|---|---|---|---|
| sandbox.files | Read, write, move, list, and inspect files inside the lease. | sandbox | yes |
| sandbox.commands | Execute bounded commands inside the isolated environment. | sandbox | yes |
| sandbox.processes | Start and inspect managed background processes. | sandbox | no |