KOLAB: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows

Abstract

The KOLAB Dataset

Existing datasets of human-agent work are often not public, rely on predefined tasks constructed by researchers, or focus on a single domain such as software development. KOLAB instead captures professionals solving their own real-world problems with an agentic AI (Claude), while using whatever other software and AI tools they normally rely on.

Problem criteria

• 🌍 Real-world: The problem originates from the professional's actual work, not from an existing workflow dataset.
• 🎓 Domain-specific: It requires professional knowledge, skills, or judgment. Routine tasks and generic information seeking were excluded.
• ⏳ Complex & long-horizon: At least one hour of work and at least five interdependent steps, each with multiple computer actions.
• 📦 Deliverable-oriented: A concrete final deliverable, plus an intermediate output at every step that can be evaluated.

Workflows as graphs

Each workflow is a temporal graph of semantically meaningful human and agent actions and their dependencies. In the example below, an architect works through five steps. A task delegated to the agent in Step 2 yields an output that is later MODIFY-ed in Step 3, and the final plan is ADOPT-ed in Revit, an action that never appears in the chat interface.

What we release

How Humans Use Agent Outputs

Two human annotators labeled how the human response to each agent output incorporated that output, including actions that happen outside the chat interface:

In total, all 100 sessions (50 professionals) have human-curated temporal graphs (avg. 5.2 steps/session, 8.7 nodes/step) and adoption labels, yielding 692 labeled instances for benchmarking.

Collaboration Dynamics

We characterize collaboration with three metrics: burstiness B of the human–agent interaction rhythm (Goh & Barabási), workflow topology W (the fraction of branch nodes, i.e. how much the human works concurrently while the agent generates), and AI retention R_A, R_M (the share of agent outputs adopted or modified).

(a) Collaboration burstiness

(b) Workflow topology

(c) AI retention rate

Benchmark: Can MLLMs Understand Human-Agent Collaboration?

We evaluate Gemini 3.1 Pro, Gemini 3.8 Flash, GPT-6 Astra, Claude Opus 5.5, and two open-weight video MLLMs (Qwen3-VL-30B-A3B and Qwen3-VL-32B), against majority-class and mode-rating baselines. Video is the primary input; we also test all seven combinations of video, workflow graph, and chat log.

Results

RQ1 · Recognizing and forecasting AI adoption

Video-only input, mean ± std over 3 runs. Shaded = best excluding Claude (the agent used in data collection, which may be biased toward its own interface). Claude was run once; GPT-6 Astra was evaluated on fewer instances (279 for 1a and 349 for 1b, vs. 692 and 691 for the other models).

Does longer context help?

On the 114 instances whose source and target AI nodes both fall in the last step of a session, we add the preceding 1–3 steps (Exp. 1c) or the entire workflow up to the target AI node (Exp. 1d). Longer context gives limited gains: macro-F1 changes stay below 0.07 for most video and graph settings, and none of the video improvements are statistically significant. Chat logs benefit most (Gemini 3.1 Pro +0.12 F1 with three preceding steps, p < 0.01), but the gain does not persist with the full workflow, and longer context can hurt (Qwen-32B drops from 0.43 to 0.32 F1 with the full graph, p < 0.01).

RQ2 · Predicting collaboration satisfaction and output quality

Averaged over 3 runs (GPT-6 Astra on 276 steps). Because human ratings skew heavily toward 5, the trivial mode baseline beats every MLLM on MAE.

Which modality helps?

No single modality wins everywhere. Video alone yields the highest recognition performance (GPT-6 Astra, F1 = 0.71, Exp. 1a), while graphs, alone or combined with video or chat, tend to give the best forecasting (GPT-6 Astra, F1 = 0.52, Exp. 1b). Chat alone particularly hurts recognition of REJECT cases: relative to video, F1 changes by +0.07 for ADOPT, +0.01 for MODIFY, and −0.07 for REJECT, likely because chat does not capture how outputs are used outside the AI interface. Human-curated workflow graphs help the non-agentic Qwen models most: for Qwen-32B, adding the graph lifts forecasting F1 from 0.23 to 0.37, and for output quality it raises Gemini Flash and Qwen-30B from ρ ≈ 0.05 to 0.23–0.30.

Failure Cases

Models favor particular labels. Gemini 3.1 Pro and Flash over-predict ADOPT: with video, Gemini Flash predicts it for 83.8% of examples against a 65.9% ground-truth rate. Qwen3-VL-32B instead predicts MODIFY for 87.3% of cases (19.1% in the ground truth). These biases largely persist with graph and chat inputs (e.g., Flash predicts ADOPT for 82.3% and 74.0%, and Qwen3-VL-32B predicts MODIFY for 67.7% and 62.0%).

Models give near-constant ratings. With video, Gemini 3.1 Pro predicts 4.83 for a true rating of 1 and 4.95 for a true 5 in Task 2a. Adding the graph widens the gap between true ratings of 1 and 5 to 1.95 and 2.45 for Tasks 2a and 2b; GPT-6 and Claude already separate low and high ratings with video (gaps of 0.54–1.34).

Explore the biases

The charts below use the latest benchmark runs. Switch the input to see how each model's behaviour changes with what it can see.

Conclusion

KOLAB benchmarks frontier multimodal agentic models on their ability to support human-agent collaboration in long-horizon workflows. Even frontier MLLMs, including GPT-6, struggle to predict and understand human–AI collaboration. In future work, we will explore constructing training datasets and sandbox environments where models can practice collaborating with simulated users before real-world deployment.

BibTeX

Copy@misc{hayati2026kolab,
  title   = {Kolab: Benchmarking Human-Agent Collaboration in Long-Horizon Workflows},
  author  = {Hayati, Shirley Anugrah and Setiani, Patricia Anugrah and Wang, Ruizi and Li, Zhouyi and Le, Khanh Chi and Lee, Young-Jun and Kang, Dongyeop},
  year    = {2026}
}