On Wikipedia, many policies and conventions are implicit; they are not integrated into editing experiences. An example is the policy of verifiability which prompts people to add new content to attribute that information to a published, reliable source. Without a source for each major fact, people might come to doubt how trustworthy Wikipedia is. Despite the importance of this policy, many newcomers don’t know it exists and as a result, they forget to cite their sources.

The Editing team has set up the Edit check project to present Wikipedia’s policies when people need them most: when they are editing an article. The project began with a simple and fundamental check: cite your sources. 

We were able to quantify the addition of sources. Only 19% of logged-out users (IP users) or beginners spontaneously add a source when they add a new paragraph to an article. Our goal was to increase the probability that people who are new would publish a new content edit that includes a reference and is not reverted within 48 hours.  

A simple encouragement

When selected users attempt to publish an edit that includes a new paragraph without a source, the visual editor highlights the added paragraph and displays a message inviting people to add a citation:

If someone selects “yes” to add a source, then the visual editor opens the reference insertion interface. If someone thinks the paragraph they are adding does not need a source and select “no”, the interface asks  them to indicate why they think a reference is not necessary. 

We tested this principle of Reference Check on several Wikipedias at the beginning of 2024, under the name ‘References check’. And the results were encouraging!

2.2 times more references added

We set up an A/B test on 11 Wikipedias. There, we targeted users who were making an edit that involved adding at least one new paragraph without a reference. Half of the IPs and new contributors were invited to add a reference using References Check, the other half had the usual experience, without invitation. Megan Neisler, Wikimedia Foundation Staff Data Scientist working with the editing team, analyzed the results.

As mentioned above, overall, 19% of new content edits include a reference by people who added a reference spontaneously. When they are advised to add a reference, this number rises to 42.4% of new content edits with references added and are not reverted within 48 hours. We observed increases on both desktop and mobile. On mobile, users are 4.2 times more likely to include a reference with their new content when the reference check is shown to eligible edits.

The revert rate fell by 8.6% for people who saw Reference Check.  We looked at cancellations originating from people editing within Sub-Saharan Africa, as these people have been historically underrepresented on the wikis. Contributors from Sub-Saharan Africa are 53% less likely to be reverted when Reference Check is shown to eligible edits.

A few other effects were observed. Firstly, contributors that are shown Reference Check and successfully save a non-reverted edit are 16 percent more likely to return to make a non-reverted edit in their second month. Overall, there was a 10% decrease in edit completion rate for edits where Reference Check was shown. We are not sure, but it could be because people realize they need a reference to add a fact, and so abandon their edit.

Configuring Checks 

For people who patrol recent edits, every edit that has shown Edit Check is tagged. Similarly, when someone has indicated that a reference is not required, a tag indicating the reason marks the recorded modification. As these tags are visible for each modification concerned, this gives more context to the modification and allows more precise intervention. For example, when someone chooses that ‘the fact is known to everyone’ when this is not the case, it is easier to explain how to improve a modification thanks to the context given by the tag. 

When a reference is added, we have also introduced a reference reliability check. If someone decides to use one of the links on the lists of links rejected by the community, this is indicated when the reference is inserted. This is an improvement on the default experience, which indicates that the link has not been accepted when you click on ‘publish’. 

Edit Check is community configurable. For now, it is possible to define parameters for References check, like where the length of a paragraph, or where the citation should be added in the sentence.

The first check in a long series

These initial results encourage us to continue in this direction. We’re going to roll out References Check to all Wikipedias. You can test it at your wiki right now.

This work on references is not over yet: in the feedback we have received from our discussions with communities, the need to help users identify what makes a source a good reference for Wikipedia is often highlighted. It is also possible to configure the feature so that it fits your community’s particularities.

The Editing team is currently working on defining a new type of ‘check’, to confirm whether the concept works. We’re also thinking about how to show several checks during the editing experience, through the ‘Multi Check’ project. And as the funding principle of Edit check is to be used in many projects, we will provide a set of ‘bricks’ that will enable tool builders to build their own checks.

We can’t invent future checks without you. We welcome you to any of our community conversations, to talk about the future of Edit check!