WMF Projects/Data Dumps
 | This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. |
|
Wikimedia data dumps
|
Status as of 31 January 2012: A problem with the rsync to our mirror site was located and fixed. Another organization agreed to mirror the dumps as well, and we are waiting for their server to come online. Back issues of dumps from 2002 through 2006 were made available, for folks interested in historical data. New hardware has arrived in our Virginia datacenter, and we'll be copying all dumps over there as soon as it's ready. We're thinking about how to provide image dumps in some fashion, even if we don't keep local copies of the dumps or they are not run on a regular basis. We also cleaned up the dumps documentation and drafted 2012 development plans. Finally, we have a contractor, Christian Aistleitner, who will be working on a test suite for dump generation.
Feature justification
[edit]We produce XML-formatted copies of the content of each of the wiki projects in each active language. These files are used by researchers doing analysis, by contributors who run bots to manipulate content, and by organizations that mirror the projects, among others. In addition the existence of these files permits the existence of a fork of the content at any time, much as the existence of downloadable source code for various open source projects permits the community to fork it when it is deemed appropriate.
Since 2007 the availability of the dumps has been pretty variable. The dumps of the English-language Wikipedia in particular, since they are so huge, take a long time to run, and if something goes wrong during that run: a network issue, a power outage, a problem with the server hosting the files or the server generating the dumps, a change to some part of the MediaWiki code that causes corruption, the entire job is lost and must be restarted.
Speed and availability of current files are not the only issues; we have had no mirrors of these files, nor backups.
We have had many requests for additional fields to be added to the dumps or for subsets of dumps to be provided as well.
This project will address these issues.
Open tasks
[edit]- Development tasks
See Obsolete:Dumps/Development 2011 on wikitech.wikimedia.org.
- Product tasks
Current Status
[edit]See Obsolete:Dumps/Development status 2011 on wikitech.wikimedia.org.
Related documents
[edit]See:
- Dumps on wikitech
- Dumps Parallelization on wikitech
- Mirroring Wikimedia project XML dumps
- Obsolete Research Data Proposals from 2010
Schedule
[edit]Ongoing (fill this in)