First Monday

An integrated framework for online news quality assurance by Besiki Stvilia

This paper introduced a synthesized theoretical framework of online news quality assurance. The framework includes conceptual models of quality evaluation, value assessment, and intervention. The framework also provides typologies of user activities, information agents, and the relationships among them. The framework is grounded in prior frameworks of information quality and the analysis of two cases of large-scale online news aggregators: Google News and Facebook News. The framework can be used as a knowledge source to guide the design and evaluation of quality assurance processes of online news providers and aggregator ecosystems.


Research design
Conceptual framework of news quality assurance
Cases of online news aggregators
Case comparison and discussion




Americans’ patterns of news consumption have been changing. Increasingly more people, including seniors, are getting their news online, whereas the share receiving news from television is shrinking (Shearer and Gottfried, 2017). The evaluation of online news quality and the development of interventions to counter the creation, spread, and consequences of misinformation and disinformation have increasingly come into the spotlight since Russian interference in the 2016 presidential election in the United States. Advances in deep learning have introduced more challenges to the assurance of news quality. In one study, 72 percent of readers found machine-generated fake news convincing, compared with 83 percent who found real news articles convincing [1]. There are calls for research to identify effective approaches to fighting the creation and dissemination of fake news and to design and promote a twenty-first-century news infrastructure and culture that incentivizes and rewards the spreading of true information (Lazer, et al., 2018).

In an earlier study, we developed a general framework of information quality assessment (Stvilia, et al., 2007) and a model of information quality change (Stvilia and Gasser, 2008). Multiple models have been proposed in the literature to evaluate the credibility of online news, and actual interventions have been proposed and deployed in the field to limit the spread and effects of false information (e.g., Ferrara, 2017; Horne, et al., 2018; Lazer, et al., 2018; Zaller, 2003; Vosoughi, et al., 2018; Wineburg, et al., 2016). To design effective interventions, however, the interventions need to be conceptualized within the overall context of online news quality assurance (NQA), including the models of quality evaluation and dynamics (Stvilia, 2006). Except for a few examples (e.g., Allcott and Gentzkow, 2017; Wardle and Derakhshan, 2017), attempts at conceptualizing an online NQA ecosystem have been limited. This study contributes to that need. It builds on the previous work and develops an integrated, general theoretical framework for online NQA.



Research design

In this study, we used a literature analysis and exploratory case studies (Yin, 1984) to develop a theoretical framework for online NQA. In particular, guided by a prior general framework of information quality evaluation (IQE), we analyzed the literature on NQA and synthesized a conceptual framework of online NQA. This initial conceptualization of the framework was then used to guide case studies of the NQA practices of Facebook News and Google News. Thus, as recommended in the literature (e.g., Bailey, 1994; Harlow, 2010), we combined deductive and inductive reasoning to build this theoretical framework.

We used convenience-based sampling to select the targets of the case studies. When selecting cases for the analysis, we used two criteria. The cases had to represent systems that were large enough to embody the multitude of relationships of news ecosystems. In addition, the systems had to provide sufficient information on their NQA practices. Facebook and Google News are large-scale information systems and aggregators of online news information that serve the global audience. Hence, they can function as good empirical examples of the complexity and diversity of online news ecosystems and quality assurance challenges. Furthermore, these systems supply a significant amount of documentation on their news selection and quality assurance practices on the open Web. We harvested that documentary data (i.e., policies, procedures, quality evaluation guidelines, blogs, Q&As) from the Facebook News and Google News sites in January 2020. Thus, the cases provided the temporal snapshots of those systems’ NQA practices of that particular time period. We analyzed those documentary data using the method of content analysis. The central unit of the analysis was an individual case. The subunits of the analysis were the activity systems of the four phases of the IQE: quality conceptualization, evaluation, dynamics, and intervention. We used the components and relationships of the initial conceptualization of the framework as a priori codes in the content analysis of the documentary data. The findings of the case analyses were then compared and contrasted to each other and the initial conceptualization of the framework to triangulate, modify, and expand the framework with new theoretical constructs and empirical examples.



Conceptual framework of news quality assurance

This section of the paper builds an initial conceptualization of the online NQA framework. It analyses the literature for the four phases of quality assurance activity systems of the IQE framework: conceptualization, evaluation, dynamics, intervention. Next, it maps and categorizes findings of the analyses using typology building (Bailey, 1994) to identify the types of information, activities, actions, agents, and relationships of the online NQA ecosystem.

The ultimate goal of information quality assurance (IQA) is to gain some degree of control over information quality. In the case of the news ecosystem, there is a need to connect the changes in news quality to changes in the outcome of an activity in a systematic and meaningful way (see Figure 1). If the objective of the activity is to have a well-informed electorate, then the quality of the outcome of that activity is the quality of informedness of the electorate. To degrade the informedness of the electorate, one can spread inaccurate or incomplete information about candidates or the issues that are important to the electorate. Alternatively, disseminating accurate and complete information and debunking disinformation or misinformation can enhance the quality of the outcomes of the activity. The value of the outcome of the activity is the value an individual, a group, or the society assigns to that objective. Hence, some information may be more valuable or critical than others to voters’ deliberation and decision-making. Information about someone who serves on a gas company’s board whitewashing the image of that company and its corrupted founder is probably less critical than information about the government of a foreign country interfering in the U.S. elections.


Relationship between news quality and the outcome of an activity
Figure 1: Relationship between news quality and the outcome of an activity.


Information quality assurance can be conceptualized as follows: conceptualizing or defining what quality means in a particular context, determining how to measure it, and, determining when to intervene and how to enhance or degrade the quality (Stvilia, 2006). Thus, the NQA framework should include a conceptualization of quality — a set of virtues or criteria used to define or reason about the quality of news. Each of those criteria must be measurable, and the model needs to provide a metric(s) that can be measured objectively. Information quality is dynamic. The NQA framework must characterize or predict changes in news quality over time and in space (i.e., when moving information from one context to another), or as they relate to different activities. Finally, to be effective, the NQA framework should include intervention models that conceptualize how to modify online news quality and when to enhance or degrade the quality of the outcome of a particular activity.

News quality conceptualization and evaluation

The conceptualization of online news quality in the NQA framework is grounded in a general IQE framework developed by the author (Stvilia, et al., 2007). The IQE framework was synthesized from a comprehensive review of the information quality literature and has been used successfully to guide conceptual model development in different contexts, such as Wiki-based open encyclopedia articles, an aggregated collection of OAI (Open Archives Initiative)-harvested Dublin Core metadata, an online biodiversity information repository, consumer health information, and data management in condensed-matter physics. The IQE framework is a knowledge base that can be used for defining context-specific conceptual models in a systematic and inexpensive way. It defines general types of information-dependent activities; information quality problems, dimensions, or criteria; and the relationships among them. Once the activity types of a particular context are identified, the framework can help generate a conceptual information quality measurement model for that context.

In general, quality is defined as fitness for use (Juran, 1992), pointing to the contextual nature of quality. The online news ecosystem is diverse and comprises different actors and activities. It includes original creators or generators of news articles or stories, secondary publishers who aggregate and recompile news stories, and tertiary publishers who subscribe to and reuse news stories from aggregated news banks and collections. Other agents outside the profession also produce news-related content and commentary. These include organizations, individuals, and algorithms that generate and share claims and comments on news stories, or the underlying entities and events themselves. Examples of the latter are user postings and advertisements on various online publishing platforms that contain various newsworthy claims. This category also includes commentaries or annotations on the quality of news stories and claims. Finally, there are end users, who only consume news and do not contribute to the news ecosystem in any other way.

The IQE framework defines four types of information-dependent activities: representation dependent, decontextualizing, stability dependent, and provenance dependent. Representation-dependent activities depend on the quality of mapping between an information entity and another entity or condition the information entity represents. Decontextualizing activities use information outside the context of their creation. The success of stability-dependent activities depends on the stability of the information or its underlying entity(s). Finally, provenance-dependent activities are affected by the quality of the provenance record or metadata of the information. The news creation and use ecosystem may include activities from all four types, and the same activity can be of multiple types. A reporter reporting on an event on location is a mapping or representation-dependent activity. The story is expected to reflect the event accurately and completely. In a different scenario, a reporter who does not have firsthand experience with a particular event or is not familiar with a particular region, its history, and its culture may write a news article by using “secondary” data obtained from a news bank or wire service or by using information from a local collaborator. One would expect the original story to be cast through the context of the reuser and for some information about the original context to be excluded or underemphasized; the local reporter may not include in her or his story information about the local context that is common knowledge locally but unknown to the reuser or aggregator, and thus may unintentionally degrade the quality of the news (Stvilia, et al., 2004). News writing and use activities are also stability dependent, especially when the news reports are on highly dynamic contexts or entities, and information quickly becomes outdated (e.g., reporting on war zones or stock markets). Finally, news activities such as investigative reporting and fact checking are provenance dependent. Often those activities involve reviewing and evaluating the provenance and audit trails of the information and data used in a news story to establish and assess their reliability.

The IQE framework also includes a taxonomy of quality criteria that is connected to the types of activities and types of quality problems. In particular, for each activity type, the framework specifies a relevant subset of quality criteria. Because all four activity types can be found in the news ecosystem, all the criteria included in the taxonomy are relevant to the news ecosystem and should be considered for inclusion in the news quality evaluation model of the NQA framework. These include intrinsic qualities, activity-specific (i.e., relational) qualities or virtues of news, as well as community or cultural virtues of news, such as the authority dimension.

Information creation, evaluation, and consumption, as well as other activities, can be motivated by multiple different needs and related motivations (Kaptelinin, 2005). A typology of news information can be defined by juxtaposing the levels of news information quality with differences in the creator’s motivations or intent. The typology can then serve as an effective tool for analyzing, theorizing, and communicating about news information behaviors. Wardle and Derakhshan (2017) defined three types of information: misinformation, disinformation, and malinformation. The typology identified two characteristics of information: intent and accuracy. Activity motivations can be categorized into general binary categories: malicious and nonmalicious. Misinformation is defined as inaccurate information created or spread with nonmalicious intent. Disinformation is defined as information that is false and created with a malicious or harmful intent. Malinformation refers to information that is accurate but created or shared with a harmful intent (e.g., security leaks; inciting xenophobia and racism by referring to a deadly virus by its geographic origin). This two-by-two typology scheme points to a fourth type: information that is accurate and created or shared with nonmalicious intent. That can be called good information (see Table 1).


Table 1: Types of information based on quality and intent.
Low qualityDisinformationMisinformation
High qualityMalinformationGood information


Attempts have been made in the literature to conceptualize news content quality. Gladney, et al. (2007) identified 38 criteria of online news quality, ranging from credibility, to utility, to editorial vigor. The criteria were then grouped into six categories: content, navigation, look and feel, functionality, community relevance, and interactivity. Information quality is contextual (Stvilia, 2006). Even if the same conceptual model of news quality is used, the metrics and cues that can be used to assess the quality of news along a specific dimension are context specific and grounded in the components of that context. Zaller (2003) argued that a high-quality news standard or benchmark that attends to the accuracy, completeness, and neutrality of reporting may be too expensive and may not be required for most of the mass news. Instead, a parallel standard for less complete but sensational, engaging reporting that focuses on acute problems or events and allows for the inclusion of partisan views could be practiced.

Furthermore, information quality can be evaluated directly by checking its accuracy and completeness against community-approved reference sources, or it can be assessed indirectly. Consumers may not have enough expertise and knowledge to assess the quality of information directly. Hence, they may have to resort to indirect evaluation and use various reputation or credibility cues and heuristics. Also, an indirect quality evaluation is less expensive. The literature identifies the following dimensions of source or message credibility on social media: reputation, endorsement, consistency, expectancy violation, self-confirmation, and persuasive intent (Lewandowsky, et al., 2012; Metzger and Flanagin, 2013; Wardle and Derakhshan, 2017).

To evaluate the quality of information in a systematic and objective way, evaluators need other infrastructure components in addition to an evaluation model. These components may include IQE metrics, best practices, workbenches, and a set of reference sources and cues in which evaluators can ground their evaluations (Stvilia, et al., 2007). Choi and Haigh (2019) examined the types of claims checked by media fact checkers and the types of sources they used in their evaluations. They found that fact checkers most frequently used government data and experts as their reference sources. Information is often encoded in Web pages in the form of structured data (e.g., micro data, JSON-LD [JavaScript Object Notation for Linked Data]). This allows secondary and tertiary publishers, such as search engines, to more accurately interpret and aggregate information. Structured data, however, can contain inaccurate or conflicting information. Automated tools that can help evaluate the accuracy of structured information embedded in a Web site can help with the quality evaluation (Kalchgruber, et al., 2018).


The NQA framework uses the intent of the activity and the mode of quality change to define the types of agents in the online news information ecosystem. In an earlier work, Gasser and Stvilia (2003) began investigating a dynamic information quality model by using an agent-based computational simulation. This simulation modeled the process of a collection of agents differentially interacting with a large information base to accomplish tasks that were driven by the agents’ strategic goals. The four types of agents — user, environmental, malicious, and IQA agents — use and change individual information units and the relationships among them to execute tasks that achieve strategic goals. In the context of the online news ecosystem, one can define four types of roles: providers, users, IQA agents, and environmental agents. Based on Wardle and Derakhshan’s (2017) information typology, providers can be further divided into misinformers, disinformers, malinformers, and good providers. Good providers can be defined as those who provide high-quality information without malicious or harmful intent. Each entity in the ecosystem may wear multiple hats and play different roles in different contexts: user, provider, and IQA agents. Providers and IQA agents change the quality of information units actively or intentionally, whereas environmental or contextual agents may affect information quality passively, without a direct intent (see Table 2, Figure 2).

Changes in information quality can be active (e.g., caused by changes made to the article itself) or passive (e.g., caused by changes in the underlying entity and context — culture, sociotechnical structures, and domain knowledge; Stvilia and Gasser, 2008). What is admissible and aligned with the value structure of the previous culture may not be admissible or may be interpreted differently in the current culture. Likewise, the composition of the community may change. It can become more polarized, hence less tolerant to information that does not match the individual’s attitudes and beliefs. The set of activities and their composition may change as well. New activities can be introduced into the ecosystem that may have new quality requirements for information or emphasize information of a specific quality (e.g., news satire television programs focusing on the entertainment criterion of information quality instead of accuracy). The agent pool of the activity may change as well in favor of quality assurance agents or, alternatively, in favor of malicious agents, and that may affect the quality of information produced by the activity (e.g., the quality of news coverage of an event on social media). Another source of change in information quality may be changes to the knowledge bases that are used to evaluate the quality of information. New knowledge can be introduced in an information ecosystem, and what was considered accurate in the past may not be perceived as accurate now. In the case of news quality, an example of the latter may be sharing confidential or secret information publicly that confirms or, alternatively, debunks the current interpretation of an event (e.g., a whistleblower sharing confidential diplomatic communication). The development and adoption of new technologies can affect the cost structure of information creation, sharing, and evaluation and, consequently, the quality of information available in a particular information ecosystem. For example, new machine learning-based technologies can detect with greater accuracy machine-generated fake news, fake social media accounts, or bot farms spreading inaccurate information. Those technologies can reduce the cost of identifying low-quality news and sources and, consequently, can reduce the cost of quality assurance of news collections and improve the quality of information on social media platforms or news aggregators such as Google News. Alternatively, the same or similar technologies can be used by malicious agents to generate fake news stories automatically and spread disinformation.


Table 2: Types of information agents based on intent or goal of the action and intervention type.
Note: IQA, information quality assurance.
Spread disinformation and misinformation
Good provider
IQA agent
Spread high quality information in a responsible way
Fact check
Correct or retract any inaccurate or incomplete information
PassiveEnvironmental agent
Changes in organization, community, or cultural context in time or space that make the information less accurate or complete (e.g., decontextualization, in which the original contextual information or reference sources are lost or become unavailable)
Environmental agent
Changes in organization, community, or cultural context in time or space that make the information more accurate and complete (e.g., new evidence emerges that increases the validity of the claim reported in the news)



In general, the quality of products can be improved by improving the production process or imposing stricter quality control on the ready products (i.e., moving toward the desired quality levels through stricter scrap and rework; Cook, 1997). In distributed peer production communities such as Wikipedia, the quality of the information production process can be enhanced by improving the quality of the information provider and IQA agent pools. These interventions may comprise the introduction of stricter selection criteria and performance evaluation benchmarks and more effective training mechanisms for those agents. These may also include the development of new tools and other infrastructure components (e.g., bots, templates, policies) that reduce the cost of contribution for high-quality contributors and IQA agents and penalize low-quality or malicious contributors (Stvilia, et al., 2008). Distributed ecosystems of online news need to manage the quality of their news information “supply chain” (i.e., providers) and the quality of their IQA agents. Hence, the NQA framework should include the related models of agent quality control (see Figure 2).

The news ecosystem includes different kinds of IQA agents, such as search engines, social and news media companies, who apply automatic and semiautomatic mechanisms to block or reduce the spread of low-quality information on their platforms and counter the financial motivations of fake news creators; news organizations that provide fact-checking services; and libraries, schools, and university departments that teach information literacy courses and workshops (Weedon, et al., 2017). To be successful in evaluating the quality of news and for quality enhancement interventions to be successful, users need to have the appropriate information quality literacy levels, attitudes, and motivations. These include users’ ability to notice and recognize information quality cues in news documents and use them in their evaluations, as well as their willingness to expend effort on a sufficiently deep analysis of news content and sources. It is important to note, however, that an indirect quality evaluation based solely on surface features may not always be sufficient because providers may fake or misuse those same quality cues to deceive users (McGrew, et al., 2017). Information literacy courses and related guides can also teach consumers how to fact check and triangulate the accuracy of claims found in news (Neely-Sardon and Tignor, 2018). Furthermore, malicious agents can use farms of fake accounts to create and amplify false news and disinformation. Twitter and other social media platforms make regular efforts to identify and then block or delete those fake accounts and malicious bots (Crowell, 2017).

The NQA framework should include models of agent motivation. Creators and consumers of news can be driven by different, often conflicting motivations. For instance, the consumer’s need and related motivation to search for and consume accurate news can conflict with the consumer’s psychological need for self-efficacy and the related utility the consumer receives from reading news confirming her or his prior beliefs and evaluations (Allcott and Gentzkow, 2017; Flynn, et al., 2017). Whereas the producers of fake news are often motivated by the short-term financial gain from selling advertisements, their influence may persist if the consumer’s ability to check the validity of fake news is limited or her or his prior beliefs and the related psychological utility from receiving confirmatory news are very strong (Allcott and Gentzkow, 2017). It is important to note that the producers of fake news have a much lower cost than the producers of accurate news because they expend no resources on ensuring or evaluating the accuracy of their information.

Ideally, one would apply a variety of IQA techniques to maintain the highest possible quality of the news collection as a whole relative to the requirements of the current strategies and tasks. However, in a world of limited resources that is not always possible (Stvilia, 2006). News ecosystems need to apply IQA resources differentially and condition their application for maximum effect based on the probability of increased value, reduced cost, or both. In particular, NQA algorithms should provide a value-based assessment of quality and intervention for different kinds of news genres. In addition, the added or lost value of an IQA action can be tangible (e.g., generating greater use of the news) or intangible and experience related (e.g., conflicting with someone’s cultural, religious, or ethical beliefs, norms, or values or their emotions).

Most of the earlier models of information value can be grouped into two categories. The first is the information theory-based approach, which uses the statistical structure of an information system. In this approach, information value equals information quantity or the gain in an information system, that is, how much information or how much unexpected information is contained in a given information object or item (e.g., see Machlup [1983]). Because fake news reports on fake events, activities, properties, or relationships, not surprisingly, these news items may contain more novel information than true news. This could be one of the reasons fake news is often shared at a higher rate than true news on social media. Fake news also causes different emotive responses than true news. Vosoughi, et al. (2018) found that whereas fake news caused surprise, fear, and disgust in readers, people responded to true news with sadness, anticipation, joy, and trust. However, like information quality, information value is a multidimensional concept, and novelty is only one of its facets. If the success of an activity requires the use of accurate information (e.g., teaching, COVID-19-related planning and policy design), sensational but inaccurate information will be of little value to that activity.

In the second, the decision theory model, the value of information equals the size of the welfare or net payoff increase the agent achieved from the use of the information (e.g., see Marschak, 1971; Radner, 1986). That is, the value of information equals the difference between the action payoffs obtained with and without the information. Likewise, the value of an information quality intervention is defined by the change in value of the outcome of the activity that uses the information. Hence, as with any design, the design of a news quality intervention needs to be evaluated against the alternatives in terms of its impact on news consumers and their activities. For instance, Ecker, et al. (2017) found that a retraction of misinformation that repeats the misinformation before debunking it may be a more effective intervention than a retraction that does not explicitly repeat the misinformation. Furthermore, information quality value can be functional (e.g., adding a key word to a metadata record that makes the information item discoverable to users who search by that keyword) or nonfunctional (e.g., news information conveyed in a satirical or entertaining way that increases the consumer’s engagement with the news). For instance, the effects of fact checks can be diminished by their limited reach to the intended users (Lazer, et al., 2018). Individuals’ exposure to ideologically cross-cutting news content on social media platforms (i.e., Facebook) can be limited by their friendship networks, algorithms, and individual choices (Bakshy, et al., 2015).

Orr (1998) proposed a basic control theory model of information quality dynamics and suggested a connection between information use and information quality. In particular, he argued that information use affects quality and quality affects use in a feedback cycle over time, and that more frequently used information units are more likely to grow in quality. The argument is that the frequency of use increases the criticality or value of news information and attracts more quality improvement or maintenance interventions. Similarly, one would expect that malicious agents may target the same news information and attempt to degrade its quality by pushing low-quality versions of it to the news ecosystem, or they may spread related low-quality news and commentary to further influence readers (Stvilia, 2006; see Table 3). Furthermore, criticality is context dependent. Information that is critical or valuable to one user group may not be valuable to another user group. For instance, researchers at the Pew Research Center found that older adults, black adults, and adults with less education in the United States exhibited a greater interest in local news (Barthel, et al., 2019).


Table 3: Where to intervene.
 Low criticalityHigh criticality
Low qualityNoYes
High qualityNoNo



Conceptualization of the news quality assurance (NQA) framework
Figure 2: Conceptualization of the news quality assurance (NQA) framework. Note: IQA, information quality assurance.




Cases of online news aggregators

The previous section of the paper provided the conceptualization of the NQA framework summarized in Figure 2. The framework includes synthesized typologies of information activities, agents, quality criteria, and relationships derived from the literature. This section presents descriptive case studies of the NQA practices of two online news aggregators: Facebook News and Google News. The NQA framework was used to guide the analysis of these cases.

Case of Facebook

Quality conceptualization and evaluation

Facebook is a leading social media platform in the United States. Approximately 70 percent of U.S. adults use Facebook, and almost 75 percent of users visit the site daily. More importantly, 43 percent of U.S. adults get their news from Facebook [2]. Facebook defines a typology of news content quality that comprises nine types or categories: false, mixture or partially true, true, false headline, not eligible, satire, opinion, prank generator, and not rated. The typology is used as a rating schema by third-party fact checkers to rate the quality of news content. The definitions of the categories are mostly based on one dimension of quality, accuracy, and the intended uses of the information (e.g., satire, prank generator). The exceptions are the categories “mixture or partially true” and “not eligible.” The mixture or partially true category is defined as content that contains a mixture of accurate and inaccurate claims or whose primary claim is incomplete. Thus, the definition of this category references one more quality dimension, completeness, although the emphasis is still on accuracy. The category “not eligible” comprises types of information that Facebook does not evaluate. These include nonverifiable claims, claims that were true in the past but are no longer true, and claims made by political entities [3]. Hence, this category references the third information quality dimension, currency.

In October 2019, Facebook announced the Facebook News project, which bears considerable similarity to Google News [4]. It is intended to provide centralized access to a curated collection of news stories. To be considered for inclusion in this service, news publishers must sign up through the Facebook News Page Index [5]. Facebook News has a curation team that comprises professional journalists. The curation team analyzes publisher activity across different subject areas and decides what topics to cover on a particular day. The team makes that determination by using four criteria: the prevalence of the topics in the news ecosystem; the impact of the topics on the lives of people, organizations, and society in general; the broad interest or appeal of the topics; and the diversity in the range of topics covered. Regarding selecting stories for specific news topics, Facebook News references the use of the following quality criteria: factual accuracy, diversity of publishers, original reporting, on-the-record sourcing, timeliness, depth and context, fairness (i.e., diversity of perspectives on the issue), and local reporting. To provide a personalized experience, Facebook News users’ interests are captured through prior likes, shares, comments, previously read news stories, and their publishers.

Content providers and information quality evaluation agents

Facebook News expects the news publishers it includes in its index to have a functioning Web site with a verified domain. They should not share misinformation, parody, or satire. Their news stories should be original, current, non-user-generated, and with properly documented sources. Facebook expects the participating publishers to serve marketing and advertising content in such a way that users can easily distinguish it from the actual news content [6]. Finally, in addition to meeting these selection criteria, publishers need to be prominent enough to be included in the Facebook News index. A publisher’s prominence is determined by the size of the audience it serves. Facebook News defines four types of news publishers: general news, topical news, diverse news, and local news. The diverse news category comprises news publishers that focus on the five major racial and ethnic groups identified by the U.S. Census. For each of the four types of news, Facebook News uses a separate threshold for audience size to determine a publisher’s eligibility.

Facebook uses its own algorithms, reports from its members, and its fact-checking partners around the world to identify news that needs to be evaluated for quality. The fact-checking partners then review the news stories for accuracy and assign ratings based on Facebook’s quality ratings schema [7]. To serve as Facebook’s fact-checking partners, organizations must be certified by the International Fact-Checking Network (IFCN). The IFCN is a nonpartisan organization that uses a code of five principles with related criteria to evaluate applicant organizations. The five principles are nonpartisanship and fairness, transparency of sources, transparency of funding and organization, transparency of methodology, and open and honest corrections policy. Evidence shared by applicants for each principle can be rated as “compliant,” “partially compliant,” or “noncompliant.” Applicants must provide evidence to the IFCN showing that they regularly verify and publish nonpartisan evaluations of the accuracy of newsworthy statements and claims and that they comply with the rest of the IFCN code [8]. Although the code does not explicitly reference specific skills, the criteria require applicants to demonstrate that they are following an unbiased, nonpartisan, transparent, replicable, and fair fact-checking process.


The not eligible category of Facebook’s quality evaluation model includes news that was accurate at the time of writing but may not be accurate any longer. Thus, Facebook’s quality evaluation model recognizes that news quality is dynamic and that it changes in time. Furthermore, the model acknowledges that the evaluation of news quality is contingent on the objective of the activity and its overall context. The definition of the satire category points to the use of news in humor and comedy and the importance of that context in its interpretation and evaluation [9]. In particular, it indicates that in some contexts, the entertainment quality of news can be emphasized more than the accuracy. Although Facebook’s model does not explicitly reference the role of a spatial context change as a source of news quality problems, one can predict that removing news from its original context of creation can lead to a quality problem (Stvilia, et al., 2007). The reference sources used to create or interpret news in its original context may be different or unavailable in the destination context, leading to problems of inaccuracy or incompleteness. Indeed, Facebook News uses local reporting as one of the criteria when selecting news stories for specific topics [10], and it employs fact checkers around the world who have knowledge of the local contexts and cultures.

Facebook prioritizes its quality intervention efforts based on the criticality of news. News criticality itself is determined by the degree of extant influence or the reach of the news among Facebook users, or the cultural and societal importance of the entity associated with the news as its creator or topic (CSPAN, 2019).

Facebook uses several types of interventions to confront the spread of false news [11]. It algorithmically reduces the distribution of posts evaluated by fact checkers as false or partially false. In addition, Facebook removes billions of fake accounts that could be used to amplify false information [12]. Furthermore, Facebook explicitly targets the financial incentives of false news creators. Pages and sites that repeatedly create or share false content are punished by having their ability to monetize and advertise removed. The inability to advertise and monetize on Facebook can be a significant penalty to news publishers. One study estimated the value of a click on a news link in Western European countries at 0.04–0.06 Euros [13]. Google projected that it sent more than 14 billion U.S. dollars’ worth of advertising revenue to news publishers around the world in 2018. In addition to these approaches to managing the quality of the news creation and sharing process, Facebook has mechanisms for controlling the quality of already produced or shared news (i.e., rework). An offending content provider is allowed to make a correction to its flagged content and notify the fact checker about the action taken. The correction is then reviewed by the fact checker and, if approved, Facebook lifts the penalty imposed on the provider [14].

Finally, Facebook provides additional context and reference sources to help users evaluate the news stories they encounter in their news feeds. For instance, Facebook may link a news story to a Wikipedia article about its publisher or accompany the story with related evaluations from fact checkers [15].

At the time of this writing, January 2020, Facebook does not subject posts and ads that originate directly from active politicians to fact checking. It considers those as types of direct speech, and hence ineligible for fact checking. This policy, however, does not apply to secondary content (i.e., content that does not originate from politicians but is shared by them). In addition, as of January 2020, Facebook has refused to follow the practice of Twitter and Google and block or limit the targeting of political ads. However, it has decided to provide its users with more transparency about the reach and audience of political ads and with control over what ads to see [16].

Case of Google

Quality conceptualization and evaluation

The Google search engine (Google Search) indexes and actively promotes news content from its interface. In addition, Google aggregates and provides topical access to news from its news portal, Google News. Google News is one of the most frequently visited aggregator sites of online news content. Google Search dominates the global search engine market, with an almost 93 percent share of that market as of January 2020 [17]. Google’s digital advertising revenue from news content was estimated at $US4.7 billion in 2018, which is very close to the $US5.1 billion digital advertising revenue made by the U.S. news industry as a whole [18]. Google News does not use an editorial team to determine what topics to present users with on a particular day. Instead, Google News selects topics and ranks stories within those topics algorithmically by using the following signals: relevance, as determined by the user’s prior search behavior; interest, as specified in the user’s settings for Google News; location; the prominence of a topic or article; the authoritativeness of the source; freshness; and the usability of the site [19].

To improve the effectiveness of its algorithms in promoting high-quality content and demoting low-quality or malicious content in general, Google uses human raters. These raters evaluate the quality of the search results, and their ratings are then used to train Google’s algorithms. Google uses an extensive set of rater guidelines to guide the raters’ assessment of Web site quality. The guidelines include examples of high- and low-quality content of various genres, lists of reputation and quality indicators, and procedures that the raters should follow to determine the quality rating of Web content. The guidelines cover the genres of news content as well (i.e., news Web sites, articles, blogs), including examples of misleading, deceptive, and inaccurate content [20]. The guidelines reference the following quality criteria: the reputation of the Web site or the creator of the main content, as indicated by signals such as awards received (e.g., a Pulitzer Prize); the purpose of the page (e.g., helpful vs. harmful); and its expertise, authoritativeness, and trustworthiness, as signaled by factually accurate content, journalistic professionalism, well-defined editorial and review policies, original in-depth reporting, and referenced sources.

Content providers and information quality evaluation agents

Google News specifies 13 quality criteria its publishing partners are expected to meet. Publishing partners should provide original content, these partners should be transparent about its authorship, ads should not surpass content, and sponsored content should be clearly disclosed. In addition, publishing partners should not disclose personal and confidential information; violate intellectual property rights; produce and share content of an explicit sexual nature or content that incites violence or hate; provide medical advice and treatment; support dangerous and illegal activities; engage in harassment, cyberbullying, deceptive practices, and impersonation; and spread spam and malware [21]. To become a publishing partner with Google News, any owner of a Web site can submit that site to crawling and inclusion in the Google News index.

Google does not disclose how it selects raters and whether and how it evaluates the quality of ratings submitted by those raters. Google also does not specify whether it uses a specific selection model or a third-party organization to select fact checkers for the fact-checking module of Google News.


By including the location and freshness of the news in its content ranking model, Google acknowledges that the quality of news stories is dynamic and may change in time and space [22]. The inclusion of the freshness criterion indicates that Google News expects users to find current news more useful. Fresh content may reflect the state of the underlying entity or thing more accurately than older content. Users’ needs evolve, accompanied by changes in their activities and the need for related information services and content. Google estimates that around 15 percent of searches submitted to its search engine are new [23]. Because existing information content or an existing service can be selected by human intermediaries or algorithms to meet those new needs, the quality of that content, including news content, can be evaluated differently (Stvilia, et al., 2007). Similarly, the location criterion points to the importance of the local context and local preferences in assessing the relevance and quality of news.

Like Facebook, Google uses fact checkers’ evaluations as one of the interventions to combat misinformation and disinformation on the Web. Fact checkers select claims to check on the basis of their criticality, such as the notability or influence of the authors of claims, the relevance of the subject to the local audience, and the potential value of proving or disapproving the claims [24].

In addition to the automated ranking of news sites and stories, selecting publishers for inclusion in Google News’ index, and sharing fact-checker evaluations, Google invests in developing and maintaining an infrastructure to enable fact-checking data creation and sharing. In particular, Google, in collaboration with and academic communities, developed a schema or vocabulary for fact-check metadata called ClaimReview [25]. Google also developed the Fact Check Markup Tool [26] to reduce the cost of fact-check metadata creation and sharing. Fact-check metadata generated by Google’s markup tool are automatically deposited to; are openly accessible and searchable through a Web application, a readwrite API (application programming interface); and are downloadable as a JSON data file [27]. Originally, the IFCN certified journalists and organizations who could have access to Google’s fact-checking tools and who could submit fact-check data to the collection [28]. Currently, however, access to the Fact Check Markup Tool and the API seems more open. Any full or restricted user of a Web site can submit fact-check markup [29].

In addition, Google platforms such as YouTube provide links to established reference sources such as Wikipedia and Encyclopædia Britannica when users search for topics that often have been exposed to or have generated false information [30]. The hope is that users will use these reference sources to verify the accuracy of claims made in a YouTube video or obtain more complete coverage of the video topics, or both.

Furthermore, both Google and Facebook support the creation and operation of third-party quality evaluation bodies, such as the Trust Project consortium. The Trust Project consortium assigns its trust badge to news organizations that implement and comply with its eight quality indicators: best practices, author/reporter expertise, type of work, citations and references, methods, locally sourced, diverse voices, and actionable feedback [31].



Case comparison and discussion

This section compares, contrasts, and discusses the two case analyses’ findings in the context of the literature. The analysis findings are then used to triangulate the initial conceptualization of the NQA framework and expand it with empirical types and relationships.

Quality conceptualization and evaluation

Facebook uses nine content types in its rating schema, some of which are based on well-known news genres, such as opinions, pranks, and satire. Document genres have culturally justified typified communication purposes and structures, including typified information quality conceptualizations and cues (Orlikowski and Yates, 1994; Stvilia, 2006). The use of genres reduces the complexity and cost of content classification for fact checkers, as well as users’ cost of understanding or making sense of fact-checkers’ evaluations. The definitions of the news content types seem to be grounded in four facets. Three of these facets are quality criteria: accuracy, completeness, and currency. The fourth dimension is intent. Google too references the purpose of the page in its general quality evaluation guidelines (see Table 4).

Facebook News and Google News use similar sets of criteria to evaluate the quality of news articles (see Table 4). Their shared criteria include factual accuracy, original reporting, documenting sources/on-the-record sourcing, timeliness/currency, in-depth reporting, and fairness.

They differ in some respects, however. When selecting articles for inclusion, Facebook News uses the diversity of publishers and local reporting as criteria. Whereas local reporting can be viewed as placing greater importance on information collected locally, a diversity of publishers signals the ability to gain more complete and diverse perspectives on a news topic. Google does not reference these signals in its quality evaluation and ranking models.

In its general quality evaluation model, Google references reputation as well as expertise, authoritativeness, and trustworthiness (EAT), which are not explicitly included in the Facebook News model. However, Google defines EAT by using the criteria found in Facebook’s model. Furthermore, it is important to note that reputation and EAT are grounded in the same quality signals: cultural, community, or organizational records of expertise and trustworthiness (Choi and Stvilia, 2015; Stvilia, et al., 2007). Hence, the use of both reputation and authoritativeness can be considered redundant. Prominence, however, may not necessarily be associated with the person’s expertise or trustworthiness, but rather other characteristics, such as personal wealth, position in an organizational or administrative hierarchy, or clever marketing strategies.

Google News and Facebook News use interest, prevalence, impact, or prominence to select noteworthy topics. Although these four criteria can be considered importance or value-related characteristics, there are subtle differences among them. The news topic of a famous football player switching teams can be interesting, prevalent, and prominent but not as impactful as the discovery of an effective treatment for a deadly viral disease such as COVID-19. The ranking model of Google News also references location and usability. The latter criteria are relevance signals and can be grouped under relevance.


Table 4: Comparison of the quality evaluation models of Facebook News and Google News.
Note: Shared criteria are highlighted in bold. Italics indicate unique criteria.
Facebook NewsGoogle News
Criteria used to define genres
Facebook News’ article selection criteria
    Factual accuracy
    Diversity of publishers
    Original reporting
    On-the-record sourcing
    Depth and context
    Fairness (i.e., diversity of perspectives on the issue)
    Local reporting
Facebook News’ topic selection criteria
    Diversity of topics
General quality evaluation criteria
    Reputation of the Web site or the creator of the main content (e.g., receiving awards, such as a Pulitzer Prize)
    Purpose of the page
    Expertise, authoritativeness, and trustworthiness signaled by the following cues:
      Factual accuracy
      Original reporting
      In-depth reporting
      Document sources
      Meet journalistic standards
Google News’ topic and article selection criteria


For this study, we next mapped the quality evaluation models of Facebook News and Google News to the set of quality criteria of the IQE framework. This mapping of models to the IQE framework, which is grounded in the information quality literature, helped identify and illustrate the differences in the models from the information quality theory perspective. The mapping also helped determine aggregate, synthesized models of news quality and criticality evaluation that were grounded in the prior theoretical framework and in data supplied by the two empirical cases.

Eight article selection criteria, three criteria used to define news genres, and one criterion used for topic selection by Facebook News were mapped to six quality dimensions of the IQE framework: accuracy, currency, relational accuracy, relational completeness, relational informativeness, and verifiability. Likewise, most of the Google News quality model was successfully mapped to the following eight criteria of the IQE framework: accuracy, currency, relational completeness, accessibility, relational informativeness, relevance, verifiability, and authority. Thus, the Google News model had a greater number of relational and reputational dimensions of quality, including accessibility, relevance, and authority, that were not found in the Facebook News model. The comparison of the two mappings showed that the Google News model had a focus on properties related to information relevance and accessibility (i.e., relevance, interest, location, purpose, language, usability), whereas the Facebook News model placed additional emphasis on the accuracy and completeness of representation by including criteria such as local reporting, fairness, diversity of publishers, and diversity of topics. Facebook News also defined news genres that were grounded in the dimensions of accuracy and completeness.


Mapping of the quality evaluation models of Facebook News and Google News to the information quality evaluation (IQE) framework
Figure 3: Mapping of the quality evaluation models of Facebook News and Google News to the information quality evaluation (IQE) framework. Note: Shared criteria are highlighted in bold. Italics indicate unique criteria.


Thus, the mapping of the Facebook News and Google News quality models to the IQE framework identified a news quality evaluation model that included nine criteria: accuracy, currency, relational accuracy, relational completeness, relational informativeness, relevance, accessibility, verifiability, and authority (see Figures 3, 4). In addition, both Google’s general quality evaluation criteria and Facebook’s definitions of news genres referenced the purpose or intent of news content use. Because intent refers to the communication objective of a provider, it was not included in the quality evaluation model but was represented by the types of provider agents in the NQA framework (see Figure 4).

Finally, both models were composed of criteria for assessing the value or criticality of news topics, such as prevalence, interest, impact, and prominence. Because prevalence, prominence, and interest refer to the same frequency-based characteristic of topic value, the criticality model of the NQA framework retained only two of the criteria found in the cases: impact and prevalence (see Figure 4). The Facebook News model included a diversity of topics, which can be considered a measure of collection level completeness (i.e., quality).

Content providers and information quality evaluation agents

Less overlap was found between Facebook News and Google News as it concerned the criteria used to select content providers (see Table 5). Facebook News uses a model that is composed of 10 criteria, and Google News has a model with 12 criteria. The models shared four criteria: not share misinformation, nonredundant, verifiability, and clear differentiation of sponsored content. It is noteworthy that Facebook News disqualified providers that provided satire or parody, whereas Google News disqualified providers that provided medical advice.

The Google News model focused more on the intent or behavior of providers. Six criteria out of 12 stipulated that news providers should not engage in illegal, malicious, or high-risk behaviors, such as disclosing personal information, violating intellectual property rights, providing medical advice, engaging in harassment, cyberbullying, or spreading spam and malware. When specifying what kind of content news providers should provide, the model referenced three content quality criteria: original/nonredundant, verifiability, and accurate.

The Facebook News content provider selection model, on the other hand, used the misinformation genre and two of its subgenres (i.e., parody and satire), which were grounded in the accuracy dimension, to specify undesired content. In addition, Facebook referenced four content quality criteria when stipulating what kind of content the providers had to provide: original/nonredundant, current, not user generated, and verifiable.

More than a quarter of a century ago, Mason (1986) proposed a general framework for ethical information services. The framework is composed of four dimensions: privacy, accuracy, property, and access (PAPA). The framework posits that ethical information services should preserve users’ privacy, provide accurate information, preserve intellectual property owners’ rights, and provide access to information with proper safeguards. Applying the PAPA framework as another theoretical lens in the analysis of the provider selection models of Facebook News and Google News helped shed further light on where these organizations’ priorities rested with regard to the four ethical dimensions of information provision. The mapping of the two models to the PAPA framework showed that Facebook News placed emphasis solely on its providers’ practices of supplying high-quality content. Google News, on the other hand, required its providers to fare well along all four dimensions of ethical information services. Google News expected its providers to produce high-quality information, preserve users’ privacy and the intellectual property rights of content creators, and safeguard the public from accessing content that incites violence, hate, bullying, harassment, and other dangerous or illegal activities (see Table 5).


Table 5: Comparison of the publisher selection models of Facebook News and Google News.
Note: Labels representing the PAPA framework dimensions (privacy, accuracy, property, and access) are highlighted in bold.
Facebook NewsGoogle News
    • Functioning Web site with a verified domain
    • Prominent (Quality)
Should not share
    • Misinformation (Quality)
      ◦ Parody (Quality)
      ◦ Satire (Quality)
News stories should be
    • Original (Quality)
    • Current (Quality)
    • Not user-generated (Quality)
    • With documented sources (Quality)
    • With marketing and advertising clearly differentiated from the actual news content (Quality)
    • Provide original content (Quality)
    • Be transparent about content authorship (Quality, Ownership)
    • Ads should not surpass content (Quality)
    • Sponsored content should be clearly disclosed (Quality)
Should not
    • Disclose personal and confidential information (Privacy)
    • Violate intellectual property rights (Ownership)
    • Produce and share content of an explicitly sexual nature (Access)
    • Post content that incites violence or hate (Access)
    • Provide medical advice and treatment (Access)
    • Support dangerous and illegal activities (Access)
    • Engage in harassment, cyberbullying, deceptive practices, and impersonation (Quality, Access)
    • Spread spam and malware (Quality, Access)


Both Facebook News and Google News used fact checkers as one of their main sources of news quality evaluation. As with any other content, they use fact-checkers’ evaluations for evaluating the quality of news or the quality of news providers, or for educating the public. Facebook and Google need to have mechanisms to evaluate the quality and reliability of fact checkers. Both organizations seemed to rely on third-party organizations, such as the IFCN, to conduct the screening and certification of fact checkers. The IFCN has an elaborate evaluation model that emphasizes non-partisanship, the transparency of sources, funding, the organization and fact-checking methodology, and the provision of an open and honest correction policy as requirements for reliable fact checkers. Google, in general, employed thousands of human quality evaluators to evaluate the quality of different types of content, including news genres. It provided extensive guidelines to guide the fact-checkers’ evaluations. However, Google did not specify how those evaluators are selected and evaluated.


The quality evaluation models of both Facebook News and Google News attempted to capture the dynamic nature of quality. The Google News model included time- and space-related criteria in its ranking model. Likewise, the quality evaluation criteria of Facebook News included timeliness and local reporting to capture and reflect the temporal and spatial nature of information quality. The models also tried to represent the dynamic nature of information criticality or value. Both models included criteria such as prevalence, impact, and prominence to capture the criticality of news topics and stories in their news selection and ranking algorithms.

Quality assurance resources are limited. Facebook and Google manage billions of information objects. They cannot evaluate the quality of all those resources by using human raters. Hence, as with any other large-scale system, they need to determine where to apply those limited resources and when. The typology from Table 3 posits applying IQA resources to news content that has low quality but high criticality. To determine where to intervene, the NQA framework needs to include models for both the quality dynamics and the criticality of news information. CSPAN (2019) reported that Facebook uses the degree of influence of a news story in its network or the importance of the entity(s) associated with the news, or both, to determine its criticality. In addition, in the topic selection part of its quality evaluation model, Facebook News included three criticality evaluation criteria: prevalence, impact, and interest. Google News did not disclose whether or how it prioritized its quality assurance resources, but it too included prominence as a criterion in its quality evaluation model. Another Google product, YouTube, used the low-quality and high-criticality facets of news to determine where to intervene. It showed links to authoritative reference sources when users searched for topics that could have been exposed to considerable false information.

Both Facebook and Google fought misinformation through reducing the spread and influence of false content by demoting the creators in their indices and removing their ability to advertise and make money in their systems. They also fought the effects of already published low-quality information by sharing the fact-checkers’ evaluations for influential news stories, providing alternative viewpoints, and providing the broader context (i.e., completeness) by connecting or juxtaposing those stories with references sources such Wikipedia and Encyclopædia Britannica. That helped users in making sense of the news and evaluating its quality (see Figure 4).


News quality assurance (NQA) framework
Figure 4: News quality assurance (NQA) framework. Note: IQA, information quality assurance.





This paper introduced a synthesized theoretical framework of online NQA. The framework includes conceptual models of quality evaluation, value assessment, and intervention. The framework also provides typologies of user activities, information agents, and the relationships among them. The framework is grounded in the information quality literature, and two cases of large-scale online news aggregators were examined. The framework can be used as a knowledge source to guide the design and evaluation of quality assurance processes of online news providers and aggregator ecosystems.

The study has limitations. The case studies used only one type of data — the systems’ self-descriptions of the aggregators’ news curation practices published on the Web. Although documentary/archival data is one of the main types of empirical evidence used in case study research (Eisenhardt and Graebner, 2007; Yin, 1984), these descriptions may not provide complete accounts of their NQA activities. In addition, the study used only two cases to develop this theoretical framework. Eisenhardt and Graebner (2007) have suggested that to build a theory, a higher number of cases is desired. A future related study that uses additional cases of online NQA practices would help further test and expand the NQA framework.

Furthermore, the paper’s scope was limited to synthesizing a general theoretical framework that would provide conceptual models for NQA ecosystem, quality evaluation, and intervention. The paper does not define sets of news quality and value evaluation metrics that can be used to operationalize the concepts and relationships of the NQA framework. Those operationalizations are context specific, embedded in the system’s algorithms, and can be hidden from outside investigators’ view. Future research could examine how concepts such as prevalence and impact are operationalized and measured in different news ecosystems to facilitate fair and equitable news coverage.

Another future fruitful research direction would be a more in-depth investigation of the IQA practices of individual components (e.g., fact checkers, news providers) of the online news ecosystem using different types of data, such as interviews. In particular, it would be helpful to examine what motivates fact checkers and how to enhance their motivation, as well as how to extend the NQA framework with related models. End of article


About the author

Besiki Stvilia is a professor in the College of Communication & Information at Florida State University in Tallahassee.
E-mail: bstvilia [at] fsu [dot] edu




































H. Allcott and M. Gentzkow, 2017. “Social media and fake news in the 2016 election,” Journal of Economic Perspectives, volume 31, number 2, pp. 211–236.
doi:, accessed 10 June 2021.

K. Bailey, 1994. Typologies and taxonomies: An introduction to classification techniques. Thousand Oaks, Calif.: Sage.

E. Bakshy, S. Messing, and L. Adamic, 2015. “Exposure to ideologically diverse news and opinion on Facebook,” Science, volume 348, number 6239 (5 June), pp. 1,130–1,132.
doi:, accessed 10 June 2021.

M. Barthel, E. Grieco, and E. Shearer, 2019. “Older Americans, Black adults and Americans with less education more interested in local news,” Pew Research Center (14 August), at, accessed 10 June 2021.

W. Choi and M. Haigh, 2019. “Analyzing divergent methodologies for political fact checking: United States and South Korea,” Proceedings of the Association for Information Science and Technology, volume 56, number 1, pp. 627–628.
doi:, accessed 10 June 2021.

W. Choi and B. Stvilia, 2015. “Web credibility assessment: Conceptualization, operationalization, variability, and models,” Journal of the Association for Information Science and Technology, volume 66, number 12, pp. 2,399–2,414.
doi:, accessed 10 June 2021.

H. Cook, 1997. Product management: Value, quality, cost, price, profits, and organization. New York: Chapman & Hall.

C. Crowell, 2017. “Our approach to bots and misinformation,” Twitter blog (14 June), at, accessed 10 June 2021.

CSPAN, 2019. “Facebook CEO testimony before House Financial Services Committee” (23 October), at, accessed 10 June 2021.

U. Ecker, J. Hogan, and S. Lewandowsky, 2017. “Reminders and repetition of misinformation: Helping or hindering its retraction?” Journal of Applied Research in Memory and Cognition, volume 6, number 2, pp. 185–192.
doi:, accessed 10 June 2021.

K. Eisenhardt and M. Graebner, 2007. “Theory building from cases: Opportunities and challenges,” Academy of Management Journal, volume 50, number 1, pp. 25–32.
doi:, accessed 10 June 2021.

E. Ferrara, 2017. “Disinformation and social bot operations in the run up to the 2017 French presidential election,” First Monday, volume 22, number 8, at, accessed 10 June 2021.
doi:, accessed 10 June 2021.

D. Flynn, B. Nyhan, and J. Reifler, 2017. “The nature and origins of misperceptions: Understanding false and unsupported beliefs about politics,” Political Psychology, volume 38, number S1, pp. 127–150.
doi:, accessed 10 June 2021.

L. Gasser and B. Stvilia, 2003. “Using multiagent models to understand the quality of organizational information bases over time,” Proceedings of the NAACSOS Conference.

G. Gladney, I. Shapiro, and J. Castaldo, 2007. “Online editors rate Web news quality criteria,” Newspaper Research Journal, volume 28, number 1, pp. 55–69.
doi:, accessed 10 June 2021.

E. Harlow, 2010. “Contribution, theoretical,” In: A. Mills, G. Durepos, and E. Wiebe (editors). Encyclopedia of case study research. Los Angeles, Calif.: Sage, pp. 236–238.
doi:, accessed 10 June 2021.

B. Horne, W. Dron, S. Khedr, and S. Adali, 2018. “Assessing the news landscape: A multi-module toolkit for evaluating the credibility of news,” WWW ’18: Companion Proceedings of the The Web Conference 2018, pp. 235–238.
doi:, accessed 10 June 2021.

J. Juran, 1992. Juran on quality by design: The new steps for planning quality into goods and services. New York: Free Press.

P. Kalchgruber, W. Klas, N. Jnoub, and E. Momeni, 2018. “FactCheck — Identify and fix conflicting data on the Web,” In: T. Mikkonen, R. Klamma, and J. Hernández (editors). Web Engineering: 18th International Conference, ICWE 2018, Cáceres, Spain, June 5–8, 2018, Proceedings. Cham, Switzerland: Springer International, pp. 312–320.
doi:, accessed 10 June 2021.

V. Kaptelinin, 2005. “The object of activity: Making sense of the sense-maker,” Mind, Culture, and Activity, volume 12, number 1, pp. 4–18.
doi:, accessed 10 June 2021.

D. Lazer, M. Baum, Y. Benkler, A. Berinsky, K. Greenhill, F. Menczer, M. Metzger, B. Nyhan, G. Pennycook, D. Rothschild, M. Schudson, S. Sloman, C. Sunstein, E. Thorson, D. Watts, and J. Zittrain, 2018. “The science of fake news,” Science, volume 359, number 6380 (9 March), pp. 1,094–1,096.
doi:, accessed 10 June 2021.

S. Lewandowsky, U. Ecker, C. Seifert, N. Schwarz, and J. Cook, 2012. “Misinformation and its correction: Continued influence and successful debiasing,” Psychological Science in the Public Interest, volume 13, number 3, pp. 106–131.
doi:, accessed 10 June 2021.

F. Machlup, 1983. “Semantic quirks in studies of information,” In: F. Machlup and U. Mansfield (editors). The study of information: Interdisciplinary messages. New York: Wiley, pp. 641–671.

J. Marschak, 1971. “Economics of information systems,” Journal of the American Statistical Association, volume 66, number 333, pp. 192–219.

R. Mason, 1986. “Four ethical issues of the information age,” MIS Quarterly, volume 10, number 1, pp. 5–12.
doi:, accessed 10 June 2021.

S. McGrew, T. Ortega, J. Breakstone, and S. Wineburg, 2017. “The challenge that’s bigger than fake news: Civic reasoning in a social media environment,” American Educator, volume 41, number 3, p. 4–9, 30.

M. Metzger and A. Flanagin, 2013. “Credibility and trust of information in online environments: The use of cognitive heuristics,” Journal of Pragmatics, volume 59, part B, pp. 210–220.
doi:, accessed 10 June 2021.

A. Neely-Sardon and M. Tignor, 2018. “Focus on the facts: A news and information literacy instructional program,” Reference Librarian, volume 59 number 3, pp. 108–121.
doi:, accessed 10 June 2021.

W. Orlikowski and J. Yates, 1994. “Genre repertoire: The structuring of communicative practices in organizations,” Administrative Science Quarterly, volume 39, pp. 541–574.
doi:, accessed 10 June 2021.

K. Orr, 1998. “Data quality and systems theory,” Communications of the ACM, volume 41, number 2, pp. 66–71.
doi:, accessed 10 June 2021.

R. Radner, 1986. “Normative theory of individual decision: An introduction,” In: C. McGuire and R. Radner (editors). Decision and organization: A volume in honor of Jacob Marschak. Second edition. Minneapolis: University of Minnesota Press, pp. 1–18.

E. Shearer and J. Gottfried, 2017. “News use across social media platforms 2017,” Pew Research Center (7 September), at, accessed 10 June 2021.

B. Stvilia, 2006. “Measuring information quality,” unpublished doctoral thesis, University of Illinois at UrbanaChampaign, at, accessed 10 June 2021.

B. Stvilia and L. Gasser, 2008. “An activity theoretic model for information quality change,” First Monday, volume 13, number 4, at, accessed 10 June 2021.
doi:, accessed 10 June 2021.

B. Stvilia, M. Twidale, L. Smith, and L. Gasser, 2008. “Information quality work organization in Wikipedia,” Journal of the American Society for Information Science and Technology, volume 59, number 6, pp. 983–1,001.
doi:, accessed 10 June 2021.

B. Stvilia, L. Gasser, M. Twidale, and L. Smith, 2007. “A framework for information quality assessment,” Journal of the American Society for Information Science and Technology, volume 58, number 12, pp. 1,720–1,733.
doi:, accessed 10 June 2021.

B. Stvilia, L. Gasser, M. Twidale, S. Shreeves, and T. Cole, 2004. “Metadata quality for federated collections,” Proceedings of the Ninth International Conference on Information Quality (ICIQ-04), pp. 111–125.

S. Vosoughi, D. Roy, and S. Aral, 2018. “The spread of true and false news online,” Science, volume 359, number 6380 (9 March), pp. 1,146–1,151.
doi:, accessed 10 June 2021.

C. Wardle and H. Derakhshan, 2017. “Information disorder: Toward an interdisciplinary framework for research and policy making,” Council of Europe Report (27 September), at, accessed 10 June 2021.

J. Weedon, W. Nuland, and A. Stamos, 2017. “Information operations and Facebook” (27 April).

S. Wineburg, S. McGrew, J. Breakstone, and T. Ortega, 2016. “Evaluating information: The cornerstone of civic online reasoning,” Stanford Digital Repository, at, accessed 10 June 2021.

R. Yin, 1984. Case study research: Design and methods. London: Sage.

J. Zaller, 2003. “A new standard of news quality: Burglar alarms for the monitorial citizen,” Political Communication, volume 20, number 2, pp. 109–130.
doi:, accessed 10 June 2021.


Editorial history

Received 28 August 2020; revised 16 September 2020; accepted 20 October 2020.

Copyright © 2021, Besiki Stvilia. All Rights Reserved.

An integrated framework for online news quality assurance
by Besiki Stvilia.
First Monday, Volume 26, Number 7 - 5 July 2021