Category Brief: Computer Science / Artificial Intelligence / Autonomous Research Harnesses
Generated: 2026-06-01
Scope
This brief covers cs-ai/research-harnesses using arXiv categories cs.SE, cs.AI, cs.LG and keywords autonomous research, research harness, AI scientist, cross-model, adversarial collaboration, claim audit, research wiki, paper writing pipeline, agent laboratory, automated scientific discovery.
Current queue snapshot
- Papers in top queue: 8
- Primary-category distribution: cs.AI=7, cs.SE=1
Candidate clusters
cs.AI(7): 2504.08066v1, 2603.28589v1, 2507.23276v2, 2511.04583v4, 2509.08713v2, 2506.01372v2, 2405.13352v1cs.SE(1): 2605.03042v1
Strong candidate papers
2605.03042v1— ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration
- Why queued: primary:cs.SE; title:autonomous research; abstract:research harness; abstract:cross-model; abstract:adversarial collaboration; abstract:claim audit; abstract:research wiki; fresh:30d
- URL: https://arxiv.org/abs/2605.03042v1
2504.08066v1— The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
- Why queued: primary:cs.AI; title:AI scientist; title:automated scientific discovery; fresh:2y
- URL: https://arxiv.org/abs/2504.08066v1
2603.28589v1— Towards a Medical AI Scientist
- Why queued: primary:cs.AI; abstract:autonomous research; title:AI scientist; fresh:180d
- URL: https://arxiv.org/abs/2603.28589v1
2507.23276v2— How Far Are AI Scientists from Changing the World?
- Why queued: primary:cs.AI; title:AI scientist; abstract:automated scientific discovery; fresh:1y
- URL: https://arxiv.org/abs/2507.23276v2
2511.04583v4— Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper
- Why queued: primary:cs.AI; title:AI scientist; fresh:1y
- URL: https://arxiv.org/abs/2511.04583v4
2509.08713v2— The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems
- Why queued: primary:cs.AI; title:AI scientist; fresh:1y
- URL: https://arxiv.org/abs/2509.08713v2
2506.01372v2— AI Scientists Fail Without Strong Implementation Capability
- Why queued: primary:cs.AI; title:AI scientist; fresh:1y
- URL: https://arxiv.org/abs/2506.01372v2
2405.13352v1— "Turing Tests" For An AI Scientist
- Why queued: primary:cs.AI; title:AI scientist
- URL: https://arxiv.org/abs/2405.13352v1
Evidence-backed claims
Auto-produced claim table. supported is reserved for full-text verified rows. Abstract-derived rows are preliminary-linked: traceable to a paper_id, but not yet verified enough for final claims.
| Claim | Supporting paper_id(s) | Evidence row notes | Status |
|---|---|---|---|
| ARIS is an open-source research harness for autonomous ML research, including its architecture, assurance mechanisms, and early deployment experience. | 2605.03042v1 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. Machine-verified 2026-06-01: source_quote is a verbatim substring of cached arXiv PDF (deterministic, no LLM/human trust). | supported |
| We introduce The AI Scientist-v2, an automated scientific discovery framework enhanced by agentic tree search, VLM feedback, and parallel experiment execution. | 2504.08066v1 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. | preliminary-linked |
| The Medical AI Scientist is the first autonomous research framework tailored to clinical autonomous research. | 2603.28589v1 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. Machine-verified 2026-06-01: source_quote is a verbatim substring of cached arXiv PDF (deterministic, no LLM/human trust). | supported |
| AI-generated research papers having been accepted at the ICLR 2025 workshop, suggesting that a human-level AI Scientist capable of uncovering phenomena previously unknown to humans may soon become a reality. | 2507.23276v2 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. Machine-verified 2026-06-01: source_quote is a verbatim substring of cached arXiv PDF (deterministic, no LLM/human trust). | supported |
| We developed Jr. AI Scientist, a new system that starts from a baseline paper and its associated codebase, and is capable of handling complex, multi-file implementations, overcoming a major limitation of previous AI Scientist systems. | 2511.04583v4 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. Machine-verified 2026-06-01: source_quote is a verbatim substring of cached arXiv PDF (deterministic, no LLM/human trust). | supported |
| We identify four potential failure modes in contemporary AI scientist systems: inappropriate benchmark selection, data leakage, metric misuse, and post-hoc selection bias. | 2509.08713v2 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. Machine-verified 2026-06-01: source_quote is a verbatim substring of cached arXiv PDF (deterministic, no LLM/human trust). | supported |
| AI Scientists have yet to produce a groundbreaking achievement in the domain of computer science on par with automated scientific tools. | 2506.01372v2 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. | preliminary-linked |
| This paper aims to establish a benchmark for the capabilities of AI in scientific research and to stimulate further research in this exciting field. | 2405.13352v1 | LLM finding (openai/gpt-4o-mini, extracted 2026-06-01); structured fields cached to workspace/pdf_cache/.finding..json. Machine-verified 2026-06-01: source_quote is a verbatim substring of cached arXiv PDF (deterministic, no LLM/human trust). | supported |
Automated production audit notes
- Claims are generated from local artifacts, not from unstored conversation context.
- Abstract-derived evidence is acceptable only as
preliminary-linkeddraft context, not as final scientific support. - The next audit pass should replace abstract-derived rows with full-text evidence where claims are important.
System comparison scaffold
| System / paper | Workflow scope | Evidence / assurance mechanism | Reported evaluation | Local-ledger implication |
|---|---|---|---|---|
ARIS (2605.03042v1) | Research harness with execution, orchestration, and assurance layers | Cross-family review, research wiki, experiment-audit, result-to-claim, paper-claim-audit, citation audit | Observational overnight run; 65+ skills; 3 tested executor platforms; no controlled causal evaluation | Anchor architecture for evidence-ledger-first paper research |
AI Scientist-v2 (2504.08066v1) | End-to-end autonomous ML paper generation | Agentic tree search, experiment manager, VLM feedback loop; abstract does not foreground claim ledgers | Three autonomous ICLR workshop submissions; one exceeded average human acceptance threshold | Capability frontier; needs external claim audit before trusting generated papers |
Medical AI Scientist (2603.28589v1) | Clinical autonomous research with domain conventions | Clinician-engineer co-reasoning, medical evidence grounding, ethical policies | 171 cases, 19 tasks, 6 modalities; human/LLM/Agentic Reviewer evaluations | Shows need for domain-specific evidence rows and policy fields |
How Far Are AI Scientists (2507.23276v2) | Survey of AI Scientist achievements and bottlenecks | Survey synthesis, not a harness audit | Prospect-driven review; no primary metric in abstract | Provides taxonomy/open-problem framing |
Jr. AI Scientist (2511.04583v4) | Baseline-paper-driven autonomous exploration | Iterative experiments from real papers; risk report | DeepReviewer, author-led, and Agents4Science evaluations | Useful near-term model: constrained automation plus explicit risk ledger |
NORA (2605.02092v1) | Spatial data-science autonomous research agent | Needs backfill from paper queue/full text | Needs backfill | Candidate for next evidence row |
| Agent Laboratory | Human-in-the-loop AI research assistant workflow | Needs manual source backfill | Needs backfill | Compare human checkpoints against ARIS reviewer-independence |
| data-to-paper | Data-to-manuscript workflow with traceability emphasis | Needs manual source backfill | Needs backfill | Compare traceability mechanisms with evidence matrices |
Deep analysis: 2605.03042v1 ARIS
Positioning. ARIS is best read as a research-harness architecture paper, not as a model-performance paper. Its unit of contribution is the workflow substrate around LLM agents: Markdown skills, artifact contracts, reviewer routing, persistent memory, and claim audits.
Core design thesis. The paper adopts a deliberately conservative assumption: any long-term task performed by a single agent is unreliable. From that assumption it derives three required capabilities: persistent research state, modular execution, and independent assurance. This maps cleanly to ARIS's research wiki, single-file skill/workflow design, and cross-model assurance stack.
Most useful mechanism for our system. The strongest transferable piece is the evidence-to-claim cascade: first audit experiment/evaluation integrity, then map results to explicit claim verdicts, then have a fresh reviewer check manuscript claims against the claim ledger and raw evidence. Our evidence_matrix.csv and discipline_claim_index.csv should be treated as this category's lightweight version of the same idea.
What not to overclaim. The paper's empirical support is observational. It reports one overnight run where reviewer score improved from 5.0 to 7.5/10 over about eight hours with four review-revise rounds and more than 20 GPU experiments, but it explicitly says this is not causal evidence that cross-family review is superior.
Integration consequence. This category should become the anchor for papers on AI Scientist-style systems, agent laboratories, research-harness engineering, and autonomous paper-writing systems. ARIS should be cited for the assurance/architecture pattern; AI Scientist-v2 and domain-specific systems should be used to compare search strategy, benchmark protocol, and domain outcomes.
Open questions
- What controlled benchmark can isolate the value of cross-family review from model quality, researcher taste, and task difficulty?
- How should claim ledgers represent qualitative literature-review claims, not only numerical experiment claims?
- Can local reviewer models provide enough independence for confidential codebases without sending repository context to external APIs?
- Which older autonomous-research systems are missed by arXiv-first discovery and need manual backfill?
Next actions
- Fill evidence rows for AI Scientist-v2, NORA, Agent Laboratory, data-to-paper, and AutoResearchClaw-style papers.
- Add a category-level comparison table: workflow scope, persistent memory, cross-family review, audit stack, artifact contracts, and controlled evaluation.
- Run
compose-discipline --discipline cs-aiafter at least three category briefs have evidence-backed claims.