AIRAS

AIRAS - an open-source project for research automation

AIRAS is open-source software for automated research. It gives a coding agent (Claude Code, Cursor, or any MCP client) everything it needs to take a research topic through literature survey, hypothesis, experiments, and a finished paper, and it makes the paper's claims verifiable: the paper is preregistered in git before any experiment runs, every reported number is realized from run outputs, and CI re-checks all of it before the PDF of record is produced.

AIRAS ships as one PyPI package (airas) that provides:

• an MCP server with the research tools the flow needs (paper search, hypothesis and experimental design, experiment results, figure rendering, LaTeX, Overleaf, and the record/verification tools),
• a Claude Code plugin that bundles the server with the auto-research workflow skills,
• a small CLI (airas verify-record, airas verify-paper) that the experiment repository's CI uses as the verification gate.

Currently, it focuses on the automation of machine learning research.

Quick Start

No clone, no Docker. Only uv is required; uvx fetches the package on first run.

1. Install

AIRAS is meant to be driven from Claude Code through its plugin. The plugin installs the MCP server together with the auto-research workflow skills and the hooks that record the agent's state:

/plugin marketplace add airas-org/airas
/plugin install airas@airas

The MCP server can also be used on its own, without the skills and hooks. In Claude Code:

claude mcp add airas -- uvx airas

In any other MCP client, add it to the client's MCP configuration (e.g. .mcp.json):

{
  "mcpServers": {
    "airas": {
      "command": "uvx",
      "args": ["airas"]
    }
  }
}

Upgrading. uvx keeps the version it fetched the first time, so an existing install does not move to a new release on its own. Run uv cache clean airas (or uvx airas@latest) once to pick up the latest version.

2. Configure credentials

Credentials live in ~/.airas/credentials.json and are re-read on every tool call, so you can create or edit the file at any time:

mkdir -p ~/.airas
cat > ~/.airas/credentials.json <<'EOF'
{
  "GH_PERSONAL_ACCESS_TOKEN": "ghp_...",
  "SEYVAL_API_KEY": "..."
}
EOF
chmod 600 ~/.airas/credentials.json

Key	Purpose
GH_PERSONAL_ACCESS_TOKEN	Required. Creates and drives the experiment repository (repo + workflow scopes, admin on the repository).
SEYVAL_API_KEY (+ optional SEYVAL_COMPUTE_ID, SEYVAL_WORKSPACE_ID)	Needed for backend="seyval": running experiments on the Seyval compute platform and cross-checking their provenance. See Execution platforms.
OPENAI_API_KEY / ANTHROPIC_API_KEY / GEMINI_API_KEY / OPENROUTER_API_KEY / AWS_BEARER_TOKEN_BEDROCK / VERCEL_AI_GATEWAY_API_KEY	Not needed for the flow. The agent driving AIRAS authors every generated artifact itself via get_generation_prompt. A key is only used when you call the backend-LLM generation tools directly.

3. Start a research project

In Claude Code, invoke the orchestrator skill and give it a topic:

/airas:auto-research

It walks through the flow below, asking you to settle the operational choices (repository visibility, execution platform, compute target) once up front. Other MCP clients can start from the server's start_research prompt or call the tools directly.

The auto-research flow

auto-research owns only the ordering and the rules that span steps. Each step is its own skill with a stated contract, so you can also invoke a single step on an existing repository.

Step	Skill	What it leaves in the repository
1	setup-repository	An experiment repository created from airas-template, cloned, with Actions secrets provisioned and main protected. All research state lives here from now on.
2	discover-papers	A study list distilled from multi-source paper search, including airas-papers-db, and full-text reading.
3	hypothesize-and-design	A falsifiable hypothesis and an experimental design that fixes run ids, metrics, models, datasets, and the compute environment.
4	preregister-paper	The full paper, written before any experiment, as numbered claims with criteria and predicted intervals. Its commit is the freeze point; .research/record.json is created here.
5	write-experiment-code	Experiment code against a fixed execution contract (Hydra entrypoint, sanity / pilot / full modes), environment fixed by lockfile and Dockerfile. Metrics are produced by airas-eval, not by the code itself.
6	run-experiments	Runs executed on the chosen backend (GitHub Actions or Seyval), outputs brought back under .research/results/ with provenance. See Execution platforms.
7	analyze-results	The analysis and verifiable figures (Vega-Lite charts, text-defined diagrams).
8	publish-paper	Every stated number realized from the record, compile and verification green locally, then pushed. CI re-runs the verification and commits paper.pdf to the protected branch: the paper of record.

What the flow guarantees

• Declare before you run. Nothing is dispatched before the preregistration commit exists, and every run descends from it. Fixes are committed on top, never rebased away.
• The record only grows. .research/record.json is an append-only tree of hypotheses, claims, designs, runs, and results. A reworded claim, a changed run condition, or a dropped result all fail the same check. A claim that misses its criterion is reported as a negative result, not rewritten.
• No experimental number is ever typed. Numbers reach the paper only through \airasval{...} references to a run's measured metric or declared parameter, rendered from the record. Anything else is marked \unverified{...}.
• The gate is enforced, not advisory. Branch protection makes the verification workflow a required check, so a red run cannot be pushed past.
• The repository is the state. Everything a later step needs is committed, so a fresh session can resume from the clone alone.

Execution platforms and LLMs

Experiments run through the same three MCP tools on either backend: dispatch_experiment starts the run, get_experiment_run_status follows it, and import_run_outputs copies its outputs from where the backend keeps them (Seyval's storage, or the workflow's artifact on GitHub Actions) into .research/results/ with a provenance manifest. The record gate cross-checks the committed bytes against that same store, and once the store has dropped the run, against the sha256 hashes the import recorded. Seyval (bring-your-own Slurm compute) and GitHub Actions are supported; a backend for machines you run yourself (RunPod, a lab cluster) is not yet, since it needs a store the agent cannot rewrite.

Generation steps need no LLM key: get_generation_prompt hands the agent the curated prompt and output schema, and the agent authors the artifact itself. The same steps also exist as backend-LLM tools (generate_hypothesis, generate_paper, ...) for use outside the flow; those need a provider key (get_available_llms lists the models your keys allow). Supported providers: OpenAI, Anthropic, Google Gemini, OpenRouter, Amazon Bedrock, and Vercel AI Gateway.

Companion repositories

AIRAS relies on three sibling repositories under the airas-org organization. Each keeps one piece of the workflow outside the agent's reach.

Repository	Role
airas-template	The template every experiment repository is created from. It ships the CI workflows, the execution contract, and the verification gate.
airas-papers-db	A curated database of papers from top conferences that the agent can search through search_papers, alongside OpenAlex, Semantic Scholar, and arXiv.
airas-eval	The evaluation logic, in one place. Metrics are computed by airas-eval from the run's evaluation inputs, not by the experiment code, so the agent cannot tamper with its own scores.

MCP tools

The auto-research flow uses the following tools; the skills above are thin contracts over them. The server exposes more, but these are the ones a research project goes through.

Step	Tools
setup-repository	prepare_repository, set_github_actions_secrets, protect_branch, upload_research_history
discover-papers	search_papers, fetch_paper_fulltext, get_input_schema
hypothesize-and-design	retrieve_models, retrieve_datasets, get_generation_prompt
preregister-paper	preregister_record, append_to_record, update_record, verify_latex
write-experiment-code	get_library_docs
run-experiments	dispatch_experiment, get_experiment_run_status, import_run_outputs, fetch_experiment_results
analyze-results	fetch_experiment_results, render_chart, render_diagram, append_to_record, update_record
publish-paper	generate_bibfile, verify_latex, open_in_overleaf, get_workflow_runs, download_research_history

See the MCP documentation for descriptions, credentials per tool, and configuration options.

CLI

uvx airas                 # MCP server on stdio (default)
uvx airas verify-record   # check .research/record.json against run outputs, git history, and the platform
uvx airas verify-paper    # verify the paper's values and provenance against the record — no PDF build (the CI gate)
uvx airas publish-paper   # build the paper's PDF for publishing (values already verified by the gate)

verify-record and verify-paper are the two required checks the experiment repository's CI runs on the protected branch; publish-paper builds the PDF in the separate, non-required publish workflow.

Development

cd backend
uv sync
uv run airas

Lint and type checks are wired through pre-commit (pre-commit install once per clone). See CONTRIBUTING.md.

Roadmap

AIRAS is developed in stages. Reliability comes first: an automated research pipeline is only worth scaling once its outputs can be trusted and reproduced.

1. Reliability (in progress). The record now guarantees that every number in the paper traces back to a declared run and its outputs, that each claim's verdict is derived from the criterion it declared before running, and CI enforces it. That covers the paper but not everything upstream of it: experiment code that games a benchmark, leaks test data, or deviates from the design still passes the gate. Closing that gap, from the experiment code and evaluation inputs back to the design, is the current focus.

• Preregistration: claims, criteria, and predicted results frozen in git before any experiment runs
• Append-only research record with numbers realized from run outputs and verified in CI
• Integrity of the experiment code and evaluation itself (benchmark hacking, data leakage, design deviation)

2. Reproducibility. When an agent drives the research, the agent's trajectory is part of the method. Keeping it, and being able to replay the workflow from it, is a necessary condition for a reproducible result.

• Persist the agent trajectory alongside the repository state
• Replay a research workflow from its recorded trajectory

3. Research quality. Once reliability and reproducibility are in place, raise the quality of the research itself: better hypotheses, stronger experimental designs, and more rigorous analysis.

In parallel: other fields. Machine learning is the first target because experiments are code. Alongside the stages above we are collaborating with other domains, such as life sciences, robotics, and materials science, to extend the same integrity model to their workflows.

Contact

We aim to build an operating system for automated research that enables humanity to discover scientific breakthroughs it has not yet reached.

If you are interested in this topic, please feel free to contact us at ulti4929@gmail.com.

About AutoRes

This OSS is developed as part of the AutoRes project.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contributions

By contributing to this project, you agree that your contributions are subject to the Contributor License Agreement (CLA) and may be used, modified, redistributed, and relicensed by the project owner, including for commercial, enterprise, and SaaS offerings.

See CLA.md for details.

Related projects

• AI-Research-SKILLs (Orchestra Research, MIT) — a library of library-specific ML engineering skills. AIRAS's experiment template installs it on code-generation runners so agents get framework-level guidance (fine-tuning, distributed training, inference); AIRAS's get_library_docs MCP tool complements it by pointing agents at each library's living documentation (llms.txt endpoints).

Citation

If you use AIRAS in your research, please cite as follows:

@software{airas2025,
  author = {Toma Tanaka, Takumi Matsuzawa, Yuki Yoshino, Ilya Horiguchi, Shiro Takagi, Ryutaro Yamauchi, Wataru Kumagai},
  title = {AIRAS},
  year = {2025},
  publisher = {GitHub},
  url = {https://github.com/airas-org/airas}
}