When Models Talk: The New Frontier of Social Sciences

If there is one thing we can all agree on, it is that the development of generative AI, particularly large language models (LLMs), is advancing at breakneck speed. This rapid progress is already being felt in the social sciences. Beyond their obvious applications in conducting literature reviews and assisting with document editing, LLMs are poised to transform data collection, analysis and interpretation. They offer social scientists new avenues to generate precise insights into human behavior, societal trends and policy impacts.

For example, LLMs have the potential to simulate human behavior for research purposes. This includes generating realistic survey responses, creating experimental primes and modeling dynamic interactions in online experiments. In 2024, we published a study in the Proceedings of the National Academy of Sciences (PNAS) where we asked respondents from eight low-fertility countries about their perceptions of the ideal family. Participants were presented with descriptions of hypothetical families and asked to rate them on a scale from 1 (not at all ideal) to 10 (perfectly ideal). Using regression analysis, we found that the number of children was not particularly important in individuals’ perception of what constitutes the ideal family (a finding that may explain global fertility declines) – while communication and community respect emerged as highly significant factors.

Building on this work, we explored whether LLMs could provide similar rankings. Using several well-established models, we collected synthetic data generated by these LLMs. Remarkably, the regression results closely mirrored those from the real respondents. This suggests that LLMs can approximate human responses to a considerable extent, also when the survey instrument is rather sophisticated.

Agent-based models (ABMs) have long been used to create artificial worlds populated by interacting agents. These models can be made more realistic by integrating metadata to reflect actual trends and characteristics. However, a common critique of ABMs is that researchers must specify the agents’ decision rules, which can introduce bias or reflect the researchers' own agendas. The advent of LLMs presents an opportunity to replace these ad-hoc rules with AI generative agents capable of autonomous, dynamic interactions.

Yet, a critical question arises: Can we trust LLMs to provide realistic behaviors for AI generative agents and derive reliable trends and patterns? These models are trained on vast datasets of human-generated content, meaning they inherently inherit – and sometimes amplify – existing societal biases. A further concern is the increasing prevalence of AI-generated content online, which could lead future models to perpetuate and even exacerbate these biases.

Traditional surveys are therefore unlikely to become obsolete. Validating AI-derived conclusions with real-world measures of preferences and behaviors will remain essential. Moreover, there are ethical considerations to address. Issues such as privacy and transparency demand urgent attention. Many LLMs operate as "black boxes," with proprietary algorithms that are inaccessible to researchers. But even when models are transparent, their complexity make them difficult for social scientists to fully comprehend.

To mitigate these challenges, social scientists must adopt certain best practices. One critical step is to analyze research hypotheses using a range of LLMs to ensure robustness. Another promising direction is the development of “personalized” LLMs through fine-tuning: while most LLMs demonstrate strong general performance, they tend to be less reliable when applied to specific, task-oriented problems – an area of particular relevance for researchers working with focused hypotheses. Fine-tuning involves training an existing LLM on task-specific datasets, enabling the model to deliver more targeted and reliable results.

The incorporation of AI into social science will necessarily require significant upgrades to our current teaching practices. By equipping students with the skills to use, critique and fine-tune LLMs, we can unlock the full potential of AI while maintaining rigorous scientific standards. Still, while LLMs hold tremendous promise for the social sciences, they must be approached with the same rigor. Ethical considerations, transparency and validation against real-world data are paramount.