CogBench

Do LLMs think like humans?

Papers

On cognitive functions:

Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs (Preprint)

On narrative understanding:

How LLMs Comprehend Temporal Meaning in Narratives (ACL 2025)
How “human” is AI? A comparison of temporal meanings in 8 LLMs and 3 populations (to appear in AAAL 2026)
Mary, the Cheeseburger-Eating Vegetarian: Do LLMs Recognize Incoherence in Narratives? (Preprint)

Contributors

Student Members (Computer Science): Karin de Langis*, Khanh Chi Le, Jong Inn Park, Bin Hu

Contributors: Laura K. Allen (Educational Psychology),
Puren Oncel (Educational Psychology),
Andreas Schramm (Lingusitics),
Andrew Elfenbein (English, Cognitive Science),
Mike Mensink (Psychology)

Principal Investigator: Dongyeop Kang (Computer Science)

If you are interested in joining our team, please contact Karin (dento019 [AT] umn.edu) or Dongyeop (dongyeop [AT] umn.edu).

Objective

The project’s goals include:

Identify and formalize cognitive and linguistic distinctions between language models and humans

Test theories/models/frameworks developed in human studies within an artificial intelligence setting

Build foundation for understanding 'internals' of LLM cognition that drive LLM behavior

Explore feasibility of LLM assistance in materials development and validation for human studies

Our Experimental Settings

We adapt the Wisconsin Card Sorting Test to assess strategic planning and cognitive flexibility, finding LLMs are below human standards across multiple settings and instruction variations. (Paper)

We adapt a series of working memory span measures for LLMs and find LLMs have larger working memory capacity than humans. (Paper)

We assess how manipulation of grammatical aspect influences narrative comprehension; our results indicate LLMs lack human-like pragmatic interpretations. (Paper)

We investigate how LLMs respond to coherent versus incoherent narratives and find that by most metrics, LLMs fail to satisfactorily separate coherent and incoherent stories. (Paper)

Artificial Cognition Project

Do LLMs think like humans?

Our project bridges computer science and human-centered disciplines.

We aim to broadly study cognition across both language models and humans.

Papers

Contributors

Objective

Our Experimental Settings

We adapt the Wisconsin Card Sorting Test to assess strategic planning and cognitive flexibility, finding LLMs are below human standards across multiple settings and instruction variations. (Paper)

We adapt a series of working memory span measures for LLMs and find LLMs have larger working memory capacity than humans. (Paper)

We assess how manipulation of grammatical aspect influences narrative comprehension; our results indicate LLMs lack human-like pragmatic interpretations. (Paper)

We investigate how LLMs respond to coherent versus incoherent narratives and find that by most metrics, LLMs fail to satisfactorily separate coherent and incoherent stories. (Paper)

Demo Video

Sponsors

Artificial Cognition Project

Do LLMs think like humans?

Our project bridges computer science and human-centered disciplines. We aim to broadly study cognition across both language models and humans.

Papers

Contributors

Objective

Our Experimental Settings

We adapt the Wisconsin Card Sorting Test to assess strategic planning and cognitive flexibility, finding LLMs are below human standards across multiple settings and instruction variations. (Paper)

We adapt a series of working memory span measures for LLMs and find LLMs have larger working memory capacity than humans. (Paper)

We assess how manipulation of grammatical aspect influences narrative comprehension; our results indicate LLMs lack human-like pragmatic interpretations. (Paper)

We investigate how LLMs respond to coherent versus incoherent narratives and find that by most metrics, LLMs fail to satisfactorily separate coherent and incoherent stories. (Paper)

Demo Video

Sponsors

Our project bridges computer science and human-centered disciplines.

We aim to broadly study cognition across both language models and humans.