Under the Surface: Tracking the Artifactuality of LLM-Generated Data

Abstract

This work delves into the expanding role of large language models (LLMs) in generating artificial data. LLMs are increasingly employed to create a variety of outputs, including annotations, preferences, instruction prompts, simulated dialogues, and free text. As these forms of LLM-generated data often intersect in their application, they exert mutual influence on each other and raise significant concerns about the quality and diversity of the artificial data incorporated into training cycles, leading to an artificial data ecosystem. To the best of our knowledge, this is the first study to aggregate various types of LLM-generated text data, from more tightly constrained ones like “task labels” to more lightly constrained “free-form text”. We then stress test the quality and implications of LLM-generated artificial data, comparing it with human data across various existing benchmarks. Despite its capability to match human performance, this paper reveals significant hidden disparities, especially in complex tasks where LLMs often miss the nuanced understanding of intrinsic human-generated content. This study critically examines diverse LLM-generated data and emphasizes the need for ethical practices in data creation and use using LLMs. It highlights the LLMs' shortcomings in replicating human traits and behaviors, underscoring the importance of addressing biases and artifacts produced in LLM-generated content for future research and development.

Research Questions

RQ1 What is the nature of SOTA artificial data? How does it compare to human data, and what artifacts does it contain?

RQ2 Does training on artificial data compromise performance outcomes compared to similar human-generated data?

RQ3 How specific are the artifacts of artificial data to certain types of data produced by large language models (LLMs), and how much do they apply to all types of data generated by LLMs?

Types of Artificial Data

Task Labels for classification tasks over textual input, bypassing the need for human annotators. (Dataset: Sentiment, SBIC, GHC, Social Chem.)
Preference specifically evaluating which text is better, is useful for human alignment tasks such as training reward models used in RLHF. (Dataset: CoBBLEr, P2C)
Instruction prompts written by LLMs for instruction fine-tuning, eliminating the need for humans to write comprehensive sets of diverse instructions.

Simulation where two large language model (LLM) agents converse, has potential for enhancing model performance and simulating intricate social interactions, according to initial studies. (Dataset: Grid-World, CAMEL)
Free-Form Text written by an LLM, often used to compensate for data scarcity issues and used for pretraining and/or finetuning. (Dataset: HC3, Scarecrow, Deepfake, Workers)

Research Contributions

We present a pioneering effort in gathering a diverse range of text data produced by LLMs, covering everything from more structured "task labels" to open-ended "free-form text." This comprehensive collection is significant as it allows for a unique and holistic examination of LLM outputs and provides insights into how LLMs perform under varying degrees of structure and freedom, which is essential for both understanding their current state and guiding future improvements and applications.
We aggregate and conduct comprehensive stress tests on various data generated by LLMs using the existing benchmarks, offering a thorough evaluation of the quality, consistency, and reliability of LLM outputs across diverse models and scenarios, thereby providing a groundbreaking insight into their strengths and weaknesses for future research and development.
Our research emphasizes the critical need for responsible and ethical practices in creating and using LLM-generated data, advocating for collaborative efforts among stakeholders to address biases, increase diversity, and deepen the understanding of complex human opinions in LLM outputs, thereby ensuring their development benefits society ethically and sustainably.

Key Takeaways

LLMs demonstrate a subpar understanding of complex human opinions and interactions.

Task Labels - section 5.1

While generating task labels, LLMs over-represent majority opinions and do not match minority opinions well. They also show a tendency to be misleadingly confident concerning sentences with age, gender, religion, and race bias.

Free-Form Text - section 9.1

In creating free-form text, LLMs exhibit styles that differ significantly from human approaches in diverse social contexts. Moreover, unlike human discourse, which varies widely across domains, LLM discourse patterns remain relatively consistent.

Preferences - section 6.1

For elicitation of preferences, we find that LLM preferences are tightly coupled with standalone lexical cues, whereas human preferences appear to take a more holistic approach.

LLMs struggle to respond effectively when faced with unknown or unfamiliar situations.

Simulation - section 8.1

In LLM simulations (i.e. conversations between two LLM agents), one of the most common errors is role flipping, or the agents swapping their assigned roles. This happens most frequently when the agent becomes confused, indicating that the simulation breaks down when an agent does not understand how to respond.

Instructions - section 7.1

In instruction writing, LLMs may provide incorrect outputs for a written instruction rather than an output that indicates uncertainty or lack of knowledge of the answer. In downstream training, these incorrect outputs result in more hallucinations produced by instruction-tuned models.

LLMs are deficient in accurately mirroring human behavior for particular tasks.

Simulation - section 8.1

In simulations, where LLM agents engage in conversations focused on problem-solving, these agents often stray from the main topic, negatively impacting task performance. This contrasts with human digressions, facilitating team building and contributing to more effective problem resolution.

Models trained on LLM data containing the above issues have degraded performance.

Free-Form Text - section 9.2

We find that models trained on several types of LLM-generated data -- specifically instructions, free-form text, and preferences -- are at risk for degraded performance when compared to models trained on corresponding human data. In addition, these performance deficits appear to be tied to artifacts in LLM-generated data.

Task Labels - section 5.2

Instructions - section 7.2

Preferences - section 6.2

Limitations

Our research acknowledges various limitations in studying LLMs. We aim to offer initial insights into LLM data quality and impact, rather than conclusive findings, due to the unpredictability and complexity of LLM outputs. Our study predominantly uses existing public datasets, focusing on text data relevant to NLP, and highlights the differences between LLM and human outputs, with an emphasis on ethical considerations.

However, our approach may introduce biases and limits the study's breadth. We employ human validation and qualitative analysis for assessing creativity and bias, while facing challenges in artifact analysis. Our experiments don't fully leverage the latest LLM methodologies due to resource constraints. This research, transparent in its limitations, seeks to balance practicality with relevance, providing a comprehensive understanding of the scope and implications of our findings.

BibTeX

@misc{das2024surface, title={Under the Surface: Tracking the Artifactuality of LLM-Generated Data}, author={Debarati Das and Karin De Langis and Anna Martin and Jaehyung Kim and Minhwa Lee and Zae Myung Kim and Shirley Hayati and Risako Owan and Bin Hu and Ritik Parkar and Ryan Koo and Jonginn Park and Aahan Tyagi and Libby Ferland and Sanjali Roy and Vincent Liu and Dongyeop Kang}, year={2024}, eprint={2401.14698}, archivePrefix={arXiv}, primaryClass={cs.CL} }