When measuring the influence of online information on political opinions, do mouse clicks speak louder than words?

In a polarized country, to what extent does the media influence people’s political views? A new study co-authored by MIT researchers finds that the answer depends on people’s media preferences and, importantly, how those preferences are measured.

The researchers combined a large online survey investigation experiment with web tracking data that recorded all news sites visited by participants in the month preceding the study. They found that the media preferences individuals reported in the survey generally reflected their actual news consumption, but that important differences emerged.

First, there was substantial variation in the actual news consumption habits of participants who reported identical media preferences, suggesting that survey-based measures may not fully reflect the variance in individuals’ experiences. Additionally, people with divergent media preferences in the survey often visited similar online media. These findings challenge common assumptions about the polarized nature of Americans’ media habits and raise questions about the use of survey data when studying the effects of political media.

“There is good reason to think that the information people report in surveys may not be a perfect representation of their actual media habits,” says Chloe Wittenberg, Ph.D. ’23, a postdoctoral fellow in the Department of political science from MIT and co-author of a new paper detailing the findings.

The open access document, “Media Measurement Matters: Estimating the Persuasive Effects of Partisan Media Using Surveys and Behavioral Data,” appears in the Politics Journal. The authors are Wittenberg; Matthew A. Baum, professor at the Harvard Kennedy School; Adam Berinsky, Mitsui Professor of Political Science at MIT and director of the MIT Political Experiments Research Lab; Justin de Benedictis-Kessner, assistant professor of public policy at the Harvard Kennedy School; and Teppei Yamamoto, professor of political science and director of the Political Methodology Lab at MIT.

Stated and revealed preferences

The study was prompted by a split in some academic research. Some researchers believe that current polarization results in highly partisan media consumption; others believe that partisan media sources influence citizens to adopt more polarized views. But few have measured media self-selection and its persuasive effects simultaneously, using both survey and behavioral data.

To conduct the experiment, researchers contracted with media analytics company comScore to recruit a diverse sample of U.S. adults in 2018. ComScore then combined the survey responses from more than 3,300 of these participants with detailed information about their web browsing history over the previous month. the study.

“In this study, we adopted a new experimental model called the Preference-Incorporating Choice and Assignment model – or PICA model – for which we invented and derived a formal statistical framework in previous work,” says Yamamoto. “The PICA design was a perfect fit for the study, given its objectives.”

In the first part of the experiment, participants were asked to indicate their media preferences, including the amount and type of information they like to read. In the second part, participants were divided into one of two groups. The first group could select the type of media (Fox News, MSNBC, or an entertainment option) they wanted to read, while the second group had to view articles from one of these three sources.

This approach allowed researchers to assess both the stated preferences of individuals in the survey in relation to their online news consumption, and how persuasive partisan media can be to different consumer groups.

Overall, the study found differences in the persuasive power of partisan media across news audiences. Looking at the amount of news participants consumed, the authors found that people who generally visited fewer news sites than entertainment sites tended to be more easily persuaded by partisan media.

However, when examining the political orientation of participants’ news consumption, the authors observed a small but striking discrepancy between their survey and behavioral measures of media preferences. On the one hand, findings based on survey data suggest that members of the public may be receptive to information from ideologically opposing sources. In contrast, results based on web browsing data showed that people with a more extreme media diet are primarily persuaded by media they already agree with.

“Together, these results suggest that inferences about media polarization may depend heavily on how individuals’ media preferences are measured,” the authors state in the paper.

“Our results confirm the value of using real-world data to study political media,” adds de Benedictis-Kessner. “Accurate measurement of people’s behavior online news environments is difficult, but it is important to confront these measurement challenges because of the different conclusions that can arise from the dangers of political polarization.

Expand search

As the researchers acknowledge, their work necessarily leaves some questions open. On the one hand, the current study focused on providing media content related to Education policy, including issues such as school choice and charter schools. Although education is an important issue to many citizens, it is an area that generally does not exhibit as much polarization as other topics in American life. It is possible that studies of other policy issues will reveal different dynamics.

“An interesting extension of this work would be to look at different fields, some of which might be more polarized than education,” says Wittenberg.

She adds: “I hope the field can move toward testing a wider range of metrics to see how they stack up, and I think there will be a lot of interesting and actionable information.” Our goal is not to say, “Here’s a perfect solution.” measure that you should go out and use. It’s about getting people to think about how they measure these preferences. »