Page Previews (formerly known as Hovercards) is a new software feature on the desktop version of Wikimedia sites, designed to reduce the cost of exploring of a link, as well as to promote learning by allowing readers to gain context on the article they are reading or to define an unfamiliar event, idea, object, or term without navigating away from their original topic.
Page Previews were first introduced as a beta feature in 2014. Over several months in 2017, following various earlier tests (qualitative and quantitative), page previews were rolled out to anonymous users on all Wikipedias except German and English. To confirm that the feature is working as intended (after numerous improvements and instrumentation fixes following the earlier tests), we evaluated two A/B tests that were run in October/November 2017 and December 2017-February 2018 on the German and English Wikipedias. There, the feature was activated for a small percentage of anonymous users, and data was collected measuring their interactions with the feature and with links in general, compared with a control sample of equal size.
Page Previews are designed to reduce the cost of exploring a link, as well as to promote learning, by allowing readers to gain context on the article they are reading or to quickly check the definition of an unfamiliar event, idea, object, or term without navigating away from their original topic.
How often do people use the feature to preview other pages? We set a threshold of 1000ms (one second) for a preview card to stay open before we counted it as viewed.
How often do people disable the feature? It can be disabled by clicking on a settings icon at the bottom of each preview. A high rate in disabling Page Previews would indicate that this is not a feature users want. A low rate indicates users like the feature and would continue using it.
A reader who would have opened the linked article in the absence of previews - for context about the main article they are reading, or to get a definition - might also then have gone back to the article containing the link (which registers as another pageview). Therefore we expected part of the decrease in pageviews to come from reduced usage of the back button.
Impact on fundraising: since we are hypothesizing that implementing Page Previews will result in a decrease in desktop pageviews overall, we measured the impact of the feature on donations coming from desktop banners.
Impact on fundraising: No negative impact was observed. In fact, although banner impressions were reduced, our test results suggest that the implementation of Page Previews may even have a positive impact on donation rates.
To ensure that these rates were not artificially low due to usability issues, we also confirmed in a separate qualitative test that users were indeed able to find and operate the disable functionality if they desired to disable Page previews.
Since Page Previews are an important new way of reading Wikipedia content (based on the above rate of 0.26 previews seen per pageview, there will be more than 50 million previews seen per day after full deployment), we are currently working on a separate instrumentation that will count Page Previews in the same way we currently count regular pageviews. This will eventually enable editors to know how often their articles are being viewed in this way, and the Foundation and the movement to better assess overall quantitative trends in Wikipedia readership.
Details about the EventLogging instrumentation can be found on the schema page; see also this visual overview of the different kinds of events that were logged.
The first experiment was launched on Oct 18, 2017, taking a day or more to reach the full event rate because of caching, and was stopped on November 15. To account for the strong weekly seasonality of reader behavior on desktop, the analysis above is limited to the three weeks of Oct 23 to Nov 12.
A second experiment was launched in December 2017, mainly for the purpose of gathering data for a newly added instrumentation field (time to first interaction, informing work on performance improvements), but we also used the opportunity to evaluate the same metrics as in the first test. The data from this experiment used above covers the eight weeks from December 21, 2017 to February 14, 2018.
Sampling rates and method: in both experiments, sampling was done per browser session, with probability 1.5% for test and control group each on enwiki, and 4% each on dewiki. See here for more detail.
Truncation effects for session-based data, due to some browser sessions starting before the begining of the experiment, or ending after the experiment. In particular, this means that per-session numbers are slightly lower for the first, shorter experiment.
The pageview definition used here may differ slightly from the standard one, in particular in that it counts view to special pages.
The data is limited to sendBeacon-capable browsers.
The result for the total number of pages interacted with is based on a query that makes the slightly simplifying assumption that a browser session won’t contain pages with the same name in different namespaces.
The average number of previews seen per pageview should normally be calculated based on data that is sampled per pageview, but in order to answer the other research questions, this instrumentation used sampling by browser session instead. This can introduce inaccuracies because the numbers of previews in each pageview may not be statistically independent within one session; however we assume that this error is very small, considering the small average session lengths.
This page was last edited on 11 May 2018, at 18:06.