Large language models fall short as replacements for human respondents in policy and opinion research, though they can help draft survey questions and run pilot tests, according to research from Northeastern University’s AI-Media Strategies Lab.

Study of ‘Silicon sampling’ across 30 academic papers

The working paper, published in October 2025 by John Wihbey and Samantha D’Alonzo, reviewed approximately 30 academic studies on “silicon sampling” — the use of LLMs to simulate public opinion. The authors analyzed where in the survey workflow LLMs are useful and where they fail.

The research found that models provide results that only loosely resemble real human responses when trying to simulate human preferences, with risks of bias, stereotyping and oversimplification. The authors propose a decision framework for communications practitioners that emphasizes hybrid approaches, transparency and validation against human benchmarks.

Broader AI research ecosystem

“Hopefully it is doing what it says it’s going to try to do, which is to provide good ethical research, evidence-based insights and advice to people who are in media industries,” said Wihbey, associate professor of media innovation and technology who directs the lab.

The research shows Northeastern’s approach to AI across research, teaching and institutional policy. The university is building infrastructure to support AI work while examining where the technology succeeds and where it fails.

On Jan. 30, Northeastern hosted a symposium on AI tools and research methods for social and behavioral science at its Boston campus. The event brought together scholars working on AI applications across disciplines.

“Northeastern has a very robust ecosystem of AI-focused research everywhere from fundamental computer science and algorithms-focused research to the use of AI tools to innovate in social science, to the use of LLMs in journalism to explore humanities and humanics spaces,” Wihbey said.

That ecosystem includes researchers like Christoph Riedl, a professor in the Khoury College of Computer Sciences and the D’Amore-McKim School of Business whose work focuses on human-AI teaming and collective intelligence, and Saiph Savage, assistant professor and director of the Civic AI Lab in Khoury College, whose research focuses intelligent civic technology and battling misinformation.

Faculty in the College of Arts, Media and Design work through the Institute for Information, the Internet and Democracy, which examines AI’s implications for media platforms and democratic systems.

Governance and campus-wide adoption

The university established an AI Review Committee to evaluate proposed uses of AI in classrooms and research. Javed Aslam, chief of artificial intelligence and a professor in the Khoury College of Computer Sciences, co-leads the committee. A Policy on the Use of Artificial Intelligence Systems governs AI deployment across the institution.

In April 2025, Northeastern partnered with Anthropic to provide enterprise access to Claude AI across its 13 campuses.  University data shows approximately 15,000 weekly Claude users and 43 percent student adoption.

The Center for the Advancement of Teaching and Learning and Research under the leadership of Dr. Gail Matthews-DeNatale and Dr.Michael Sweet, has recruited more than 30 faculty members as AI fellows over the past several years to pioneer classroom applications. All colleges at Northeastern are undergoing an AI readiness process, with faculty and students developing implementation plans.

Implication for media practitioners

The silicon sampling research from the AI-Media Strategies Lab offers a case study in the university’s broader approach: using AI tools while critically examining their limitations. For communications practitioners and researchers, the paper’s framework provides guidance on when to use LLMs in survey workflows and when human respondents remain essential.