Development policy/Until 2018: Difference between revisions
cc Tags: Replaced Visual edit blanking |
m Reverted edits by 194.199.117.197 (talk) to last version by Wargo Tag: Rollback |
||
Line 1: | Line 1: | ||
{{development policy}} |
|||
c'est nul |
|||
The [[MediaWiki]] software is developed collaboratively by hackers from all around the world. To make sure that none of WMF's hundreds of [[Special:Sitematrix|wikis]] breaks by an upgrade, '''but to''' also make sure that all our wikis are reasonably up to date, we have a general '''development policy''', outlined below. |
|||
== Gerrit access == |
|||
Development is done using Gerrit to manage our git repositories. To gain access to [[Gerrit]], our code review system, anyone can register at [[wikitech:Special:UserLogin/signup]]. Git with Gerrit is a lot like wiki with FlaggedRevs: Everything must be reviewed and approved, and can be reverted. |
|||
== Master branch as unstable and Wikimedia deployments == |
|||
Main development takes place in the master branch of Git; it is unstable, but must always be kept in a working state. See also [[+2]]. Major modifications should take place in a development branch and only be merged when ready. |
|||
Code is automatically synced from the git master branch to a staging cluster ("Beta Cluster"), generally within a few minutes of being merged. This allows further testing before deployment as an additional layer of defence against bugs; it is not a substitute for testing in advance of merge. |
|||
Code is generally deployed to production in stages. The WMF git branch is made from master, and this is released to the [https://test.wikipedia.org Test Wikipedia], the [https://test2.wikipedia.org Test2 Wikipedia], the [https://test.wikidata.org Test Wikidata], and [[mw:MediaWiki|MediaWiki.org]] before it goes live on other production sites. This is to ensure that everything isn't taken out by the update. It is important to remember that this is not a substitute for testing and all code should thoroughly be tested before being merged into Git master. |
|||
== Releases == |
|||
Every few months, the master branch is cut into a release branch of the form <code>REL*</code>. If you care about release management or want to try out one of the stable versions, these are the only branches that matter. Typically there will be a testing phase when a new release is begun. There might be a few release candidates, a 1.x.0 version, and finally a few minor bugfix releases. Releases are also packaged into <code>.tar.gz</code> files and made accessible from the <code>releases.wikimedia.org</code> webserver. |
|||
This ensures that users of our software can easily get a reasonably stable version without messing with Git. Anyone submitting patches should preferably use the latest version of master to avoid conflicts, and of course people with commit access need to use Gerrit to be able to push stuff for review. It's inadvisable to run production wikis off of development versions unless you know what you're doing (as do, for instance, the [[meta:System administrators|Wikimedia tech staff]], a number of whom are also MediaWiki developers). |
|||
The [[Release checklist]] provides a more comprehensive list of tasks to be done for a release. |
|||
== Database policy == |
|||
Per the RFC "Increase the strictness of MediaWiki SQL code" ([[phab:T112637]]): |
|||
* WMF will be enabling MariaDB/MySQL's strict mode ([[phab:T108255]]), which will be the default anyway in MySQL 5.7. Prior to this, all new code must not generate any warning. |
|||
* Code that touches the database must be compatible with the following MySQL SQL modes: |
|||
** <code>TRADITIONAL</code> (equivalent to <code>STRICT_TRANS_TABLES, STRICT_ALL_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER</code>) |
|||
** <code>ONLY_FULL_GROUP_BY</code> |
|||
* Database code must be compatible with older versions of databases as listed in the [[Manual:Installation_requirements#Database_server|MediaWiki installation requirements for database server]] (e.g. the [[Version_lifecycle|stable MediaWiki release]] requires MySQL {{MW stable mysql requirement}} or later). However, performance improvements that would only apply to the newest, supported versions (or its default or widely recommended defaults) should be favoured over those that apply only to unsupported releases (5.1 and lower). |
|||
* All tables must have a primary key. When a candidate for primary key could not be created (for example, if all columns can be repeated), an auto_increment or another arbitrary value, depending on the case, has to be added. |
|||
* Non-deterministic queries and unsafe statements for binlog should be avoided as they would return/write different results in a replication environment. The latter can be detected as warnings with the text "[Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT". Those include <code>INSERT ... SELECT</code> when using an auto_increment key, <code>UPDATE ... LIMIT </code> without an <code>ORDER BY</code>, and using non-deterministic functions like <code>SYSDATE()</code>. https://dev.mysql.com/doc/refman/5.6/en/replication-rbr-safe-unsafe.html has more info |
|||
* Primary keys, and fields that reference them, should be [[w:Signedness|unsigned]], in order to increase the [https://dev.mysql.com/doc/refman/5.6/en/integer-types.html maximum values]. For more details, see http://www.percona.com/files/presentations/WEBINAR-MySQL-Indexing-Best-Practices.pdf . [[Database optimization]] practices should also be followed. |
|||
== Database patches == |
|||
If you change the database schema, follow these rules: |
|||
* '''Update the installer''' – Please update <code>includes/installer</code> and add an appropriate SQL update file to <code>maintenance/archives/</code>. The naming convention, if you're adding a field, is <code>patch-''table''-''field''.sql</code>. If you're removing a field, it's <code>patch-drop-''table''-''field''.sql</code>. If you're adding a table, it's <code>patch-''table''.sql</code>. Look at the commit history of <code>includes/installer/({{git file|action=history|project=mediawiki/core|branch=HEAD|file=includes/installer/MysqlUpdater.php|text=MySQL}}|{{git file|action=history|project=mediawiki/core|branch=HEAD|file=includes/installer/PostgresUpdater.php|text=Postgres}}|{{git file|action=history|project=mediawiki/core|branch=HEAD|file=includes/installer/SqliteUpdater.php|text=SQLite}}|{{git file|action=history|project=mediawiki/core|branch=HEAD|file=includes/installer/MssqlUpdater.php|text=Mssql}}|{{git file|action=history|project=mediawiki/core|branch=HEAD|file=includes/installer/OracleUpdater.php|text=Oracle}})Updater.php</code> to find examples of how it's done. If you're adding a bunch of fields to the same table, make all those changes in one query in one patch file. |
|||
* '''Make your schema change optional''' – All schema changes must go through a period of being optional. Some examples: |
|||
** Instead of changing the format of a column, create a new column, make all writes happen to the old and new column (if it exists) and deprecate use of the old column. Check if the new column exists before blindly assuming that it does. Only eliminate support for the old column after it's clear the schema migration has completed and there's no chance that we'll need to roll back to the old version of the software. If this doesn't seem feasible, send mail to [[mail:wikitech-l|wikitech-l]] asking for advice. |
|||
** You could set your new feature to only work if a config option is set to true, and set the option to false by default. Then the commit can be safely deployed before the schema change is made. To deploy your feature to the Wikimedia cluster, [https://phabricator.wikimedia.org/maniphest/task/create/?projects=schema-change file a ticket in Phabricator] in the relevant project with the <code>#schema-change</code> tag. Once you've confirmed the change has been made, you can remove the config option to enable your feature. |
|||
** Note that this means your schema change should be optional in code - for wikimedia deployments, it is expected that every wiki with the relevant database table(s) will have the schema change applied to them. If you need different schema for different wikis, then apply the change using an extension and creating new tables dependent on that extension. |
|||
There might be cases where the "make your schema change optional" rule will be prohibitive from a performance or logistics perspective. However, schema changes like that should be rare to begin with, and should have prominent discussion on the wikitech-l mailing list. In the case where it's impossible to make your schema change optional, it is still critical to write scripts to roll back to the pre-change state. |
|||
* '''Search for input from a WMF Database Administrator''' – MediaWiki is deployed to Wikimedia websites every week, and it takes considerable planning to apply schema changes to MySQL-based sites the size of Wikipedia. {{As of|2017|September}}, Jaime Crespo (jcrespo on LDAP, jynus on irc and Manuel Arostegui, marostegui) are the best people to add to database reviews. In most cases, input is just needed on the logistics of the change. |
|||
* '''Test your changes on Beta''' - in particular, it is a common mistake to change indexes and column definitions that would result in different query plans. Try to test the generated queries' plan with tools such as [http://dev.mysql.com/doc/refman/5.7/en/explain-output.html EXPLAIN]; not doing so could mean that, when scaled to production, queries that only take 1 second locally, they pileup on production when they receive much more traffic and have larger tables. |
|||
=== Deploying to production === |
|||
Once a migration has been reviewed and agreed, if it requires WMF deployment, please follow the please [[wikitech:Schema changes|Deploying schema changes policy]] on wikitech; which basically means creating a new Phabricator task with the #Blocked-on-schema-change tag with all necessary details, then get in touch for deployment scheduling. A one week of leading time is asked, more in case it affects one of the core tables (page, revision, image on Commons) or one of the most used/largest ones (*links tables). Ask for an estimation if you are in doubt. |
|||
== Test wiki policy == |
|||
There is an official test wiki at https://test.wikipedia.org. Some developers may also have test servers deployed. |
|||
Generally, the purpose of such "fresh from Git" servers is to test the latest unstable branch of the code, ''not'' to have a testbed for demonstrating unfinished features. Features should be demonstrated with GUI mock-ups before they are implemented (these can be uploaded to [[commons:Category:MediaWiki|Wikimedia Commons]]), and implementations should only be pushed to Gerrit if they are in a reasonably complete state. This ensures that we avoid features that are never finished. To test your own code, please make sure you have a fully working local MediaWiki installation; if you want ''others'' to test your unstable code, you can always set up your own test server. |
|||
== Documentation policy == |
|||
If a feature is going to be visible to end users or administrators, it's important that the [[Developers|developer]] documents the feature. Broadly speaking, you can document |
|||
* in code comments as part of your code change that turn into generated PHP and JavaScript documentation at https://doc.wikimedia.org |
|||
* in updates to the {{git file |file=RELEASE-NOTES-1.30|text=RELEASE NOTES}} file as part of your code change to MediaWiki core |
|||
* in [[Manual:Coding conventions#Text files|text files]] in git as part of your code change |
|||
* in the vast trove of general MediaWiki documentation on this wiki |
|||
* in wiki pages on [[wikitech:|Wikitech]] that describe how MediaWiki is set up on the Wikimedia sites. |
|||
Just a few sentences are fine: what it does, how to use it. Others will fix your prose up – the important part is to have features documented. [[Documentation/Style guide]] has some suggestions. |
|||
Wiki pages on {{SERVERNAME}} use templates so that you just have to edit a few lines and the template will handle everything else. As the developer, you are the subject matter expert and therefore possibly in a better position than anyone else to document the feature accurately and thoroughly, especially if some aspects of it are not particularly self-explanatory. If you're too lazy, on the other hand, there is fortunately at least the theoretical possibility that someone will be willing and able to do it for you, since this is an open-edit wiki. |
|||
==See also== |
|||
*[[How to become a MediaWiki hacker]] |
|||
*[[Code of Conduct]] |
|||
{{TNT|Conventions navigation}} |
|||
voila |
|||
[[Category:MediaWiki development|Development policy]] |
[[Category:MediaWiki development|Development policy]] |
Revision as of 14:27, 21 June 2018
This page documents an official Wikimedia development policy. There is no current mechanism to make changes, as the TechCom RFC process is defunct. |
Development policies |
---|
See also |
Development guidelines |
The MediaWiki software is developed collaboratively by hackers from all around the world. To make sure that none of WMF's hundreds of wikis breaks by an upgrade, but to also make sure that all our wikis are reasonably up to date, we have a general development policy, outlined below.
Gerrit access
Development is done using Gerrit to manage our git repositories. To gain access to Gerrit, our code review system, anyone can register at wikitech:Special:UserLogin/signup. Git with Gerrit is a lot like wiki with FlaggedRevs: Everything must be reviewed and approved, and can be reverted.
Master branch as unstable and Wikimedia deployments
Main development takes place in the master branch of Git; it is unstable, but must always be kept in a working state. See also +2. Major modifications should take place in a development branch and only be merged when ready.
Code is automatically synced from the git master branch to a staging cluster ("Beta Cluster"), generally within a few minutes of being merged. This allows further testing before deployment as an additional layer of defence against bugs; it is not a substitute for testing in advance of merge.
Code is generally deployed to production in stages. The WMF git branch is made from master, and this is released to the Test Wikipedia, the Test2 Wikipedia, the Test Wikidata, and MediaWiki.org before it goes live on other production sites. This is to ensure that everything isn't taken out by the update. It is important to remember that this is not a substitute for testing and all code should thoroughly be tested before being merged into Git master.
Releases
Every few months, the master branch is cut into a release branch of the form REL*
. If you care about release management or want to try out one of the stable versions, these are the only branches that matter. Typically there will be a testing phase when a new release is begun. There might be a few release candidates, a 1.x.0 version, and finally a few minor bugfix releases. Releases are also packaged into .tar.gz
files and made accessible from the releases.wikimedia.org
webserver.
This ensures that users of our software can easily get a reasonably stable version without messing with Git. Anyone submitting patches should preferably use the latest version of master to avoid conflicts, and of course people with commit access need to use Gerrit to be able to push stuff for review. It's inadvisable to run production wikis off of development versions unless you know what you're doing (as do, for instance, the Wikimedia tech staff, a number of whom are also MediaWiki developers).
The Release checklist provides a more comprehensive list of tasks to be done for a release.
Database policy
Per the RFC "Increase the strictness of MediaWiki SQL code" (phab:T112637):
- WMF will be enabling MariaDB/MySQL's strict mode (phab:T108255), which will be the default anyway in MySQL 5.7. Prior to this, all new code must not generate any warning.
- Code that touches the database must be compatible with the following MySQL SQL modes:
TRADITIONAL
(equivalent toSTRICT_TRANS_TABLES, STRICT_ALL_TABLES, NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER
)ONLY_FULL_GROUP_BY
- Database code must be compatible with older versions of databases as listed in the MediaWiki installation requirements for database server (e.g. the stable MediaWiki release requires MySQL 5.7.0 or later). However, performance improvements that would only apply to the newest, supported versions (or its default or widely recommended defaults) should be favoured over those that apply only to unsupported releases (5.1 and lower).
- All tables must have a primary key. When a candidate for primary key could not be created (for example, if all columns can be repeated), an auto_increment or another arbitrary value, depending on the case, has to be added.
- Non-deterministic queries and unsafe statements for binlog should be avoided as they would return/write different results in a replication environment. The latter can be detected as warnings with the text "[Warning] Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT". Those include
INSERT ... SELECT
when using an auto_increment key,UPDATE ... LIMIT
without anORDER BY
, and using non-deterministic functions likeSYSDATE()
. https://dev.mysql.com/doc/refman/5.6/en/replication-rbr-safe-unsafe.html has more info - Primary keys, and fields that reference them, should be unsigned, in order to increase the maximum values. For more details, see http://www.percona.com/files/presentations/WEBINAR-MySQL-Indexing-Best-Practices.pdf . Database optimization practices should also be followed.
Database patches
If you change the database schema, follow these rules:
- Update the installer – Please update
includes/installer
and add an appropriate SQL update file tomaintenance/archives/
. The naming convention, if you're adding a field, ispatch-table-field.sql
. If you're removing a field, it'spatch-drop-table-field.sql
. If you're adding a table, it'spatch-table.sql
. Look at the commit history ofincludes/installer/(MySQL|Postgres|SQLite|Mssql|Oracle)Updater.php
to find examples of how it's done. If you're adding a bunch of fields to the same table, make all those changes in one query in one patch file. - Make your schema change optional – All schema changes must go through a period of being optional. Some examples:
- Instead of changing the format of a column, create a new column, make all writes happen to the old and new column (if it exists) and deprecate use of the old column. Check if the new column exists before blindly assuming that it does. Only eliminate support for the old column after it's clear the schema migration has completed and there's no chance that we'll need to roll back to the old version of the software. If this doesn't seem feasible, send mail to wikitech-l asking for advice.
- You could set your new feature to only work if a config option is set to true, and set the option to false by default. Then the commit can be safely deployed before the schema change is made. To deploy your feature to the Wikimedia cluster, file a ticket in Phabricator in the relevant project with the
#schema-change
tag. Once you've confirmed the change has been made, you can remove the config option to enable your feature. - Note that this means your schema change should be optional in code - for wikimedia deployments, it is expected that every wiki with the relevant database table(s) will have the schema change applied to them. If you need different schema for different wikis, then apply the change using an extension and creating new tables dependent on that extension.
There might be cases where the "make your schema change optional" rule will be prohibitive from a performance or logistics perspective. However, schema changes like that should be rare to begin with, and should have prominent discussion on the wikitech-l mailing list. In the case where it's impossible to make your schema change optional, it is still critical to write scripts to roll back to the pre-change state.
- Search for input from a WMF Database Administrator – MediaWiki is deployed to Wikimedia websites every week, and it takes considerable planning to apply schema changes to MySQL-based sites the size of Wikipedia. As of September 2017, Jaime Crespo (jcrespo on LDAP, jynus on irc and Manuel Arostegui, marostegui) are the best people to add to database reviews. In most cases, input is just needed on the logistics of the change.
- Test your changes on Beta - in particular, it is a common mistake to change indexes and column definitions that would result in different query plans. Try to test the generated queries' plan with tools such as EXPLAIN; not doing so could mean that, when scaled to production, queries that only take 1 second locally, they pileup on production when they receive much more traffic and have larger tables.
Deploying to production
Once a migration has been reviewed and agreed, if it requires WMF deployment, please follow the please Deploying schema changes policy on wikitech; which basically means creating a new Phabricator task with the #Blocked-on-schema-change tag with all necessary details, then get in touch for deployment scheduling. A one week of leading time is asked, more in case it affects one of the core tables (page, revision, image on Commons) or one of the most used/largest ones (*links tables). Ask for an estimation if you are in doubt.
Test wiki policy
There is an official test wiki at https://test.wikipedia.org. Some developers may also have test servers deployed.
Generally, the purpose of such "fresh from Git" servers is to test the latest unstable branch of the code, not to have a testbed for demonstrating unfinished features. Features should be demonstrated with GUI mock-ups before they are implemented (these can be uploaded to Wikimedia Commons), and implementations should only be pushed to Gerrit if they are in a reasonably complete state. This ensures that we avoid features that are never finished. To test your own code, please make sure you have a fully working local MediaWiki installation; if you want others to test your unstable code, you can always set up your own test server.
Documentation policy
If a feature is going to be visible to end users or administrators, it's important that the developer documents the feature. Broadly speaking, you can document
- in code comments as part of your code change that turn into generated PHP and JavaScript documentation at https://doc.wikimedia.org
- in updates to the RELEASE NOTES file as part of your code change to MediaWiki core
- in text files in git as part of your code change
- in the vast trove of general MediaWiki documentation on this wiki
- in wiki pages on Wikitech that describe how MediaWiki is set up on the Wikimedia sites.
Just a few sentences are fine: what it does, how to use it. Others will fix your prose up – the important part is to have features documented. Documentation/Style guide has some suggestions.
Wiki pages on www.mediawiki.org use templates so that you just have to edit a few lines and the template will handle everything else. As the developer, you are the subject matter expert and therefore possibly in a better position than anyone else to document the feature accurately and thoroughly, especially if some aspects of it are not particularly self-explanatory. If you're too lazy, on the other hand, there is fortunately at least the theoretical possibility that someone will be willing and able to do it for you, since this is an open-edit wiki.