stream.wikimedia.org
stream.wikimedia.org is a service to provide a live data stream of edits on Wikimedia wikis that anyone can tap and use to power editor tools and web apps, create beautiful visualisations, inform research, and extend MediaWiki.
It uses RCStream to subscribe to the RCFeed of Wikimedia's production cluster, and publishes this on the endpoint stream.wikimedia.org/rc
. As a web developer, one can tap the stream using JavaScript. As an app developer, one can use a suitable client library for your platform.
Consuming RCStream is a cleaner approach than parsing the change messages from irc.wikimedia.org. Wikimedia Foundation continues to operate both (configured via $wgRCFeeds
in wmf-config).
The RCStream server also responds at http://stream.wikimedia.org/rcstream_status with a simple text message; check this if you do not receive any events.
Usage
- Several researchers are pulling the whole recent changes stream with all associated content (see wiki-research-l, December 2014)
- CVN (Not yet, as of February 2015)
- Cocytus project for tracking citations on Wikipedia
Clients and alternative access points
- Demo client (CodePen), Example listener for stream.wikimedia.org using JavaScript
- Datasift: Stream of edits to the English Wikipedia (includes page content)
- http://wikimedia.meteor.com/
- pywikibot
Server setup
RCStream runs on a set of backend servers (currently, rcs100x
; puppet node; puppet role). Backend nodes: rcs1001, rcs1002.
The backend servers run instances of RCStream, with an nginx reverse proxy on each server.
An LVS load balancer (stream-lb
) is situated in front of the backend servers.
The Beta cluster has a simplified setup on a single VM instance running the rcstream role, exposed as http://stream.wmflabs.org.
See also
- RCStream: The application running on this domain.
- irc.wikimedia.org: The service that will be superseded by stream.wikimedia.org.
- puppet-rcstream module: Wikimedia's puppet module for RCStream.
- puppet-rcstream role: Wikimedia's puppet role for stream.wikimedia.org.