Category: Databases

Have you wondered what an article’s citation count means? Who cited the article? How?

Take the now-retracted Wakefield et al. article “Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children” (1998). Though most citations are negative, this is not reflected in the overall citation count (Suelzer, 2019). A unidimensional metric like citation count does not capture the diversity of how and where citations appear. 

Scite aims to contextualize the citation count

Scite, founded by Josh Nicholson and Yuri Lazebnik and previously funded by NSF and NIDA, is a database of over 800 million citation statements (Herther, 2021) categorized as 

• Supporting: providing supporting evidence for the cited work
• Mentioning: mentioning the cited work
• Disputing: providing disputing evidence for the cited work

The statements are also tagged by where they appear in citing articles (intro, results, methods, discussion, or other).  

Users can search the website and install plug-ins for browsers and reference management tools.  

Scite uses text mining and artificial intelligence

Scite uses machine learning to enhance the database of citation statements. “The corpus on which the model was trained included 43,665 citation statements classified by trained annotators with experience in a variety of scientific fields” (Nicholson et al., 2020). Scite continues to build partnerships with publishers to gain access to articles for text mining. While the tool is imperfect and evolving, it begins to demonstrate how and where works are being used.

Publishers and researchers can use scite

In addition to citation counts and altmetrics, scite smart citations are beginning to appear on databases and journal websites. They are already available in EuropePMC records (Herther 2021, Araujo & Europe PMC, 2020). 

Researchers also use scite to see how others use their own publications and how their results fit into the larger landscape. With a free account, researchers can create a limited number of reports and visualizations and set up author alerts. Researchers can use the digital badge when presenting their works. A paid account provides access to the reference check feature, which alerts authors to potentially disputed or retracted references in an uploaded manuscript.

Finally, the founders of scite hope that smart citations will encourage researchers to “report unsuccessful attempts to [test] reported claims, as so called negative results often go unpublished because they are considered inconsequential” (Grabitz et al., 2017).

Use scite with a grain of salt

The technology is evolving and the corpus to which scite has access is incomplete. As with all citation indexes, the methods and data sources vary, which is why different numbers are found across platforms like Scopus, Web of Science, PubMed, and Google Scholar. Scite’s founders caution users “given the limitations of the model precision [and] the current limited coverage of articles analyzed by scite” (Nicholson et al., 2020).

Take the Wakefield et al. (1998) article. As of March 8, 2021, the article has been cited by 349 articles in PubMed, 1,443 in Web of Science, 1,590 in Scopus, and over 3,000 in Google Scholar. The scite browser plug-in shows over 1,000 citation statements (note one citing article may include multiple citation statements). Most are classified as mentions, with 2 confirming and 7 disputing (See: A retracted article has no disputing cites, does that mean scite is not working? | scite help desk). In contrast, a recent analysis found 838 negative citations in a collection of 1,153 citing works (Suelzer et al. 2019). 

Methods and tools to evaluate research and improve reproducibility continue to evolve, and researchers can contribute to improving the model by flagging mis-classified citations. While many AI-based tools are still in development, they offer hope for a multidimensional approach to publication metrics. 

Still curious?

Read more about how scite classifies citations. Search the website. Install the plug-ins. Visit the Himmelfarb guide on How To Measure Impact?

References

Araujo, D., Europe PMC. (2020, January 20). Europe PMC integrates smart citations from scite.ai. Retrieved February 26, 2021, from http://blog.europepmc.org/2020/01/europe-pmc-integrates-smart-citations.html

Grabitz, P., Lazebnik, Y., Nicholson, J., & Rife, S. (2017). Science with no fiction: Measuring the veracity of scientific reports by citation analysis [Preprint]. Scientific Communication and Education. https://doi.org/10.1101/172940

Herther, N. K. (2021, February 15). Scite. Ai update- part 1: Creating new opportunities- an atg original. Charleston Hub. https://www.charleston-hub.com/2021/02/scite-ai-update-part-1-creating-new-opportunities-an-atg-original/

Khamsi, R. (2020). Coronavirus in context: Scite.ai tracks positive and negative citations for COVID-19 literature. Nature. https://doi.org/10.1038/d41586-020-01324-6 

Nicholson, J. M., Uppala, A., Sieber, M., Grabitz, P., Mordaunt, M., & Rife, S. C. (2020). Measuring the quality of scientific references in Wikipedia: An analysis of more than 115M citations to over 800 000 scientific articles. The FEBS Journal, febs.15608. https://doi.org/10.1111/febs.15608

Suelzer, E. M., Deal, J., Hanus, K. L., Ruggeri, B., Sieracki, R., & Witkowski, E. (2019). Assessment of citations of the retracted article by wakefield et al with fraudulent claims of an association between vaccination and autism. JAMA Network Open, 2(11), e1915552. https://doi.org/10.1001/jamanetworkopen.2019.15552

Wakefield, A., Murch, S., Anthony, A., Linnell, J., Casson, D., Malik, M., Berelowitz, M., Dhillon, A., Thomson, M., Harvey, P., Valentine, A., Davies, S., & Walker-Smith, J. (1998). RETRACTED: Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. The Lancet, 351(9103), 637–641. https://doi.org/10.1016/S0140-6736(97)11096-0