Analytics/Data/Pagecounts-raw

See also the pageviews API, available since the end of 2015.

pagecounts-raw holds the desktop sites' pageview data, in the same format that webstatscollector used to emit. pagecounts-raw data is still getting generated for legacy consumers, but if possible use pagecounts-all-sites instead.

This stream is owned by the Analytics Team.

Contained data

The dataset consists of files with names

${YEAR}/${YEAR}-${MONTH}/pagecounts-${YEAR}${MONTH}${DAY}-${HOUR}0000.gz
${YEAR}/${YEAR}-${MONTH}/projectcounts-${YEAR}${MONTH}${DAY}-${HOUR}0000

.

The pagecounts are gzipped text files holding hourly per page aggregates of pageviews and total response bytes, and projectcounts are plain text files holding hourly per domain-name^[1] aggregates of pageviews and total response bytes, and projectcounts.

The time used in the filename is in UTC timezone refers to the end of the aggregation period, not the beginning.

Both pagecounts and projectcounts are made up of lines having 4 space-separated fields:

domain_code page_title count_views total_response_size

Field name Description

domain_code

Domain name of the request, abbreviated.

The domain coding scheme in pagecounts-all-sites is on purpose downward compatible with pagecounts-raw, thus retaining quirks and inconsistencies in the coding scheme (and perhaps adding to the confusion with new added complexity). Our apologies if the scheme looks a bit complex (it is), but codes are unambiguous, and are primarily for machine-reading.

Common trailing parts in the domain name have been abbreviated in pagecounts-all-sites just as they are for pagecounts-raw. Main inconsistency was and is: project 'wikipedia.org' doesn't add a suffix for project name, where 'wikibooks.org' adds .b., 'wiktionary.org' adds .k', etc. (the original scheme predates Wikimedia's mobile site).

Domain_code now can also be a abbreviation for mobile and zero domain names, in which case .m or .zero is inserted as second part of the domain name (just like with full domain name). E.g. 'en.m.v' stands for "en.m.wikiversity.org". (Again, as project Wikipedia is not coded in the abbreviation: 'en' stands for "en.wikipedia.org", and 'en.m' stands for "en.m.wikipedia.org".

Domain trailing part	Coded as	Database name
`.wikipedia.org`		`*wiki` (be careful about the other non- wikipedia sites using this however)
`.wikibooks.org`	`.b`	`*wikibooks`
`.wiktionary.org`	`.d`	`*wiktionary`
`.wikimediafoundation.org`	`.f`	`foundationwiki`
`.wikimedia.org`	`.m` Only the following domains are considered `commons.wikimedia.org` `meta.wikimedia.org` `incubator.wikimedia.org` `species.wikimedia.org` `strategy.wikimedia.org` `outreach.wikimedia.org` `usability.wikimedia.org` `quality.wikimedia.org`	commonswiki metawiki incubatorwiki specieswiki strategywiki outreachwiki usabilitywiki qualitywiki
`.m.${WHITELISTED_PROJECT}.org`	`.mw` (See below's explanation)
`.wikinews.org`	`.n`	`*wikinews`
`.wikiquote.org`	`.q`	`*wikiquote`
`.wikisource.org`	`.s`	`*wikisource`
`.wikiversity.org`	`.v`	`*wikiversity`
`.wikivoyage.org`	`.voy`	`*wikivoyage`
`.mediawiki.org`	`.w`	`mediawikiwiki`
`.wikidata.org`	`.wd`	`wikidatawiki`

page_title

For pagecounts files, it holds the title of the unnormalized part after /wiki/ in the request Url (E.g.: Main_Page, Berlin).

For projectcounts files, it is -.

count_views The number of times this page has been viewed in the respective hour.

total_response_size The total response size caused by the requests for this page in the respective hour. This is a sum over field #7 of Cache log format fields.

So for example a line

en Main_Page 42 50043

means 42 requests to "en.wikipedia.org/wiki/Main_Page", which accounted in total for 50043 response bytes. And

de.m.voy Berlin 176 314159

would stand for 176 requests to "de.m.wikivoyage.org/wiki/Berlin", which accounted in total for 314159 response bytes.

Each domain_code and page_title pair occurs at most once.

The file is sorted by domain_code and page_title.

Data not included

This dataset does not contain per language, or per title counts for a project's mobile site. See pagecounts-all-sites, if you need them.

(note: this line should be be moved from template to parent page) So pagecounts-raw does not contain counts for mobile or zero sites. Use file version pagecounts-all-sites if you need them.

Aggregation for `.mw`

Note: anomaly retained for backward compatibility! These lines better belong in projectcounts file. Best to ignore .mw lines.

The .mw abbreviation aggregates the mobile sites across all projects per language. The page_name gets set to the used language.

So consider a given hour only sees the following requests:

https://en.m.wikipedia.org/wiki/Main_Page
https://en.m.wikipedia.org/wiki/Berlin
https://en.m.wiktionary.org/wiki/House

(and assuming each request accounted for 100 bytes), the hour's pagecounts file would consist only of the line

 en.mw en 3 300

. The corresponding projectcounts file would be

 en.mw - 3 300

. So while the .mw abbreviation counts the mobile site, it throws wikipedia, wiktionary into the same bucket. And also, it does not distinguish between page_titles.

Availability

dumps.wikimedia.org

The stream is available unsampled as gzipped hourly files from http://dumps.wikimedia.org/other/pagecounts-raw/.

The date in the file name refers to the end of the capturing period, not the beginning.

stat1002.eqiad.wmnet

The most recent ~11 days of data are available as hourly files at /mnt/data/pagecounts/incoming on stat1002.