Reinventing Social Science Research With AI

Monday - June 19 2023, 09:50 UTC - 1 year ago

A team of researchers is highlighting how artificial intelligence (AI) can redefine social science research. LLMs have already demonstrated their ability to generate realistic survey responses concerning consumer behavior, and can be used to generate novel hypotheses that could then be confirmed in the future. However, AI models cannot be used to study socio-cultural biases as they are often trained to exclude such biases, and clear guidelines for LLM governance is needed for AI-assisted social science research to be successful.

In a new study, a team of researchers is highlighting how artificial intelligence (AI), particularly large language models (LLMs), could redefine social science research. This is according to a report by Phys.org published on Friday.

"What we wanted to explore in this article is how social science research practices can be adapted, even reinvented, to harness the power of AI," said Igor Grossmann, professor of psychology at Waterloo. Leading researchers from the University of Waterloo, University of Toronto, Yale University and the University of Pennsylvania note in their novel study that large language models trained on vast amounts of text data are increasingly capable of simulating human-like responses and behaviors. This opens the doors to testing theories and hypotheses about human behavior at great scale and speed.

So far, social sciences have used a variety of methods for their research, including questionnaires, behavioral tests, observational studies, and experiments in order to obtain a generalized representation of characteristics of individuals, groups, cultures, and their dynamics. Now, emerging AI tech may just shift the landscape of data collection in the field.

"AI models can represent a vast array of human experiences and perspectives, possibly giving them a higher degree of freedom to generate diverse responses than conventional human participant methods, which can help to reduce generalizability concerns in research," said Grossmann.

"LLMs might supplant human participants for data collection," added UPenn psychology professor Philip Tetlock. "In fact, LLMs have already demonstrated their ability to generate realistic survey responses concerning consumer behavior. Large language models will revolutionize human-based forecasting in the next three years. It won't make sense for humans unassisted by AIs to venture probabilistic judgments in serious policy debates. I put an 90 percent chance on that. Of course, how humans react to all of that is another matter." .

The researchers further argue that studies using simulated participants could be used to generate novel hypotheses that could then be confirmed in human populations.

The approach however is not without possible pitfalls. LLMs are often trained to exclude socio-cultural biases that exist for real-life humans meaning that sociologists using AI in this way couldn't study those biases. Professor Dawn Parker, a co-author on the article from the University of Waterloo, highlights the importance of introducing clear guidelines for the governance of LLMs in social science research.

"Pragmatic concerns with data quality, fairness, and equity of access to the powerful AI systems will be substantial," Parker told Phys.org.

"So, we must ensure that social science LLMs, like all scientific models, are open-source, meaning that their algorithms and ideally data are available to all to scrutinize, test, and modify. Only by maintaining transparency and replicability can we ensure that AI-assisted social science research truly contributes to our understanding of human experience." .