User:Kmenger/ToolLabsSprint
Plan: July 23-25, a bunch of us are going to work on documentation, here on Etherpad, on the wikitech.wikimedia.org wiki, and in #wikimedia-labs on Freenode. Join us :)
Participants include: Kirsten Silke (part of the time) Sumana Ryan Coren Johannes (don't have lots of time, but would like to add some documentation about catgraph) Addshore (don't have lots of time, but will try to keep track of what happening and participate when and where i can) [= Ladsgroup (Amir) (add yourself)
This document contains: 1. An initial stab at an outline for a Tool Labs guide, along with source links--to draw from or point to--for each section 2. Additions to the "Migrating of Toolserver guide", with source links 3. An inventory of current documentation relating to Tool Labs, along with a proposal for a) updating b) redirection c) Appears current
All help much appreciated!
I also have some questions/notes here: http://www.mediawiki.org/wiki/User:Kmenger/Tool_Labs
Tool Labs guide
1 What is Tool Labs
Tool Labs is a reliable, scalable hosting environment for community developers working on tools and bots that help users maintain and use wikis. The cloud-based infrastructure was developed by the Wikimedia Foundation and is supported by a dedicated group of Wikimedia Foundation staff and volunteers.
Tool Labs is a part of the Labs project, which is designed to make it easier for developers and system administrators to try out improvements to Wikimedia infrastructure, including MediaWiki, and to do analytics and bot [1] work.
1.1 Rationale
Tool Labs was developed in response to the need to support external tools and their developers and maintainers. The system is designed to make it easy for maintainers to share responsibility for their tools and bots, which helps ensure that no useful tool gets âorphanedâ when one person needs a break. The system is designed to be reliable, scalable and simple to use, so that developers can hit the ground and start coding.
1.2 Features
In addition to providing a well supported hosting environment, Tool Labs provides:
¡ support for Web services, continuous bots, and scheduled tasks ¡ access to replicated production databases ¡ easily shared management of tool accounts, where tools and bots are stored ¡ a grid engine for dispatching jobs ¡ support for mosh, SSH, SFTP without complicated proxy setup ¡ time-travel backups for short-term data recovery ¡ Version control via Gerrit and Git
support for Redis
¡ 1.3 Architecture and terminology
Tool Labs has essentially four components: the bastion hosts, the grid, the web cluster, and the databases. Users access the system via one of two Tool Lab projects: âtoolsâ or âtoolsbetaâ. To request an account on the âtoolsâ project, where most tool and bot development is hosted and maintained, please see Special:FormEdit/Tools Access Request
1.3.1 Bastion hosts, grid, web cluster, databases
Bastion hosts The bastion host is where users log in to Tool Labs. Currently, Tool Labs has two bastion hosts:
tools-login.wmflabs.org
tools-dev.wmflabs.org
The two hosts are functionally identical, but we request that heavy processing (compiles, etc) be done only on tools-dev.wmflabs.org to keep interactive performance on tools-login.wmflabs.org snappy.
The grid
The Tool Labs grid, implemented with Open Grid Engine (the open-source fork of Sun Grid Engine) permits users to submit jobs from either a log-in account on the bastion host or from a Web service. Submitted jobs are added to a work queue, and the system finds a host to execute them. Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once. If a continuous job fails, the grid will automatically restart the job so that it keeps going.
For more information about the grid, please see [#Submitting, managing and scheduling Tool jobs on the grid]
The Web cluster
The Tool Labs Web cluster is fronted by a Web proxy, which supports SSL and is open to the Internet. Any of the servers in the cluster can serve any of the hosted Web tools as Tool Labs uses a shared storage system; the proxy distributes between the Web servers. The cluster uses suPHP (www.suphp.org/) to run scripts and CGI, and will soon support WSGI (www.wsgi.org/â).
Note that individual tool accounts have both a ~/public_html/ and a ~/cgi-bin/ directory in the home directory for storing Web files. For more information, please see [#Web services]
The databases
Tool Labs supports two sets of databases: the production replicas and user-created databases, which are used by individual tools.
The production replicas follow the same setup as production, and the information that can be accessed from them is the same as that which normal registered users (i.e.: not +sysop or other types of advanced permissions) can access on-wiki or via the API. Note that some data has been removed from the replicas for privacy reasons.
User-created databases can be created by either a user or a tool on the replica servers or on a local âtoolsâ project database.
1.3.2 Projects: Tools and Toolsbeta
Like the rest of Labs, Tool Labs is organized into âprojectsâ. Currently, Tool Labs consists of two projects: âtoolsâ and âtoolsbetaâ, which are described in more detail here:
Tools project: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools
Toolsbeta project: https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta
The âtoolsâ project is where tools and bots are developed and maintained. âToolsbetaâ is used for experiments in the Tool Labs environment itself--things like new systems or experimental versions of system libraries that could affect other users.
In general, every tool maintainer should work primarily on the "tools" project, only doing work on toolsbeta when changes to Tool Labs itself need to be tested to support their tool.
1.3.3 Instances
Developers working in Tool Labs do not have to create or set up virtual machines (i.e., Lab âinstances') as the Tool Lab project admins create and manage them. The term will come up in the Labs documentation; otherwise, donât worry about it.
1.4 Rules of use
1.4.1 Tool Labs policies
All tools and bots developed and maintained on Tool Labs must adhere to the terms of use that will be available here when they are finalized: Tool Labs > Rules (https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Rules)
Specifically, tools must be ¡ Open source ¡ Open data Private information must be handled carefully, if at all. Note that private user information has been redacted from the replicated databases provided by the system.
As the Tool Labs environment is shared, we ask that you strive not to break things for others, and to be considerate when using system resources.
For more information about general Labs policies, please see Wikimedia Labs > Terms of use ( https://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use) [MAP: Those terms of use are for "generic" Labs users, the Tool Labs will have its own different TOU since the audience isn't the same]
1.4.2 Individual wiki policies (these differ!)
When developing on Tool Labs, please adhere to the bot policies of the wikis your bot interacts with. Each wiki has its own guidelines and procedures for obtaining approval. The English Wikipedia, for example, requires that a bot be approved by the Bot Approvals Group before it is deployed, and that the bot account be marked with a âbotâ flag.http://en.wikipedia.org/wiki/Wikipedia:Bot_policy
For general information and guidelines, please see âBot policyâ https://meta.wikimedia.org/wiki/Bot_policy
1.5 Contact
Weâd love to hear from you! You can find us here:
On IRC: #wikimedia-labs on Freenode A great place to ask questions, get help, and meet other Tool Lab developers. See Help:IRC for more information. https://wikitech.wikimedia.org/wiki/Help:IRC
Via mailing list: Labs-l@lists.wikimedia.org A list for announcements and discussion related to the Wikimedia Labs project. You can find the archives here: http://lists.wikimedia.org/pipermail/labs-l/
Found a bug?: Bugs can be posted to Bugzilla: https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs
> Sources: :http://www.mediawiki.org/wiki/File:Tool_Labs_presentation_(Hackaton_2013).pdf https://wikitech.wikimedia.org/wiki/Help:Terminology http://meta.wikimedia.org/wiki/Bot http://en.wikipedia.org/wiki/Wikipedia:Bot_Approvals_Group https://meta.wikimedia.org/wiki/Bots https://meta.wikimedia.org/wiki/Bot_policy https://www.mediawiki.org/wiki/Wikimedia_Labs https://www.mediawiki.org/wiki/Wikimedia_Labs/Terms_of_use https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005332.html http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Languages http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Projects (tools/toolsbeta info) https://wikitech.wikimedia.org/wiki/Help:IRC https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia%20Labs (bugzilla)
2 Getting access to Tool Labs Anyone can view the source code and the output of most tools and bots, and anyone can get an account of their own as well.
To access Tool Labs you need:
to create a Labs account, which provides shell access (you must upload an SSH key) to request access to the 'tools' project
Steps for creating a Labs account, creating and uploading an SSH key, and for requesting access to the 'tools project' are described in the next sections.
2.1 Creating a Labs account on Wikitech
Before you can access Tool Labs, you must create a Labs account on Wikitech, which is the general interface for everything Labs.
Sign up for a Labs account here: https://wikitech.wikimedia.org/wiki/Special:UserLogin/signup
The "Instance shell account name" you specify in the Create Account form will be your Unix username on all Labs projects. If you forget your username, you can always find it under Preferences > Instance shell account name.
Once you have created a Labs account you will be added to a list of users to be approved for shell access, which you can see here: Shell Access Requests. (https://wikitech.wikimedia.org/wiki/Category:Shell_Access_Requests)
2.2 Generating and uploading an SSH key
In order to access Labs servers using SSH, you must provide a public SSH key. Once you have created a Labs account, you can specify a public key on the 'OpenStack' tab of your Wikitech preferences.
Specify the SSH key here: https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack
2.2.1 Generating a key in Windows
To generate an SSH key in Windows:
1. Open PuttyGen 2. Select an SSH-2 RSA key 3. Click the Generate button 4. Move your mouse around until the progress bar is full 5. Type in a passphrase (you will need to remember this) and confirm it 6. Save the private key and public key onto your local machine 7. From the text field 'Public key for pasting into OpenSSH authorized_keys file' right click and copy 8. Insert this into your 'OpenStack' tab of your Wikitech preferences
2.2.2 Generating a key in Linux
Modern Unix systems include the OpenSSH client (if not then install it). To generate a key, use:
ssh-keygen -t rsa
This will store your private key in $HOME/.ssh/id_rsa, and your public key in $HOME/.ssh/id_rsa.pub. You can use different filenames (with -f parameter), but these are the default filenames, so it's easiest to not change them.
2.3 Requesting access to the 'tools' project
Once you have created a Labs account, you must request access to the âtoolsâ project by submitting a Tools Access Request.
Submit a Tools Access Request here: https://wikitech.wikimedia.org/wiki/Special:FormEdit/Tools_Access_Request
Requests for access are generally dealt with within the day (often faster), though response-time may be longer depending on admin availability. If you need immediate assistance, please contact us on IRC.
2.4 Receiving access to the 'tools' project Once your 'tools' project access request has been processed, you will become a member of the 'tools' project, and will be able to access it using the "Instance shell account name" provided when creating your Labs account and the private key matching the public key you supplied for authentication. For more information about accessing the project, please see #Accessing Tool Labs.
2.4.1 Notification
You will be notified on Wikitech that your user rights were changed, that your request was linked from 'Nova Resource:Tools', and that you have been added to the project Nova Resource:Tools. You will also receive email explaining that your user rights have been changed, and that you are now a member of the group 'shell'. In other words, your Tool Labs account is ready for you to use!
2.4.2 Storage and use
Although you access Tool Labs via your Labs account, we strongly recommend against saving data or tools in any space that is accessible to individuals only. Tools and bots should be maintained in Tool accounts, which have flexible memberships (i.e., multiple people can help maintain the code!). For more information about Tool accounts, please see [#Joining and creating a Tool account]
>Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Getting_access
https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_instances (advice about SSH keys) https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents (advice about multiple SSH keys) https://www.mediawiki.org/wiki/Wikimedia_Labs/Account_creation_text https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Create_a_User_Account https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Home_directory_storage http://lists.wikimedia.org/pipermail/toolserver-l/2013-April/005952.html (re 2.4.1: note to clarify notification/initial experience) https://wikitech.wikimedia.org/wiki/Help:Shared_storage (not using indiv. account for shared storage)
3 Accessing Tool Labs Tool Labs can be accessed in a variety of ways--from its public IP to a GUI client. Please see Help:Access (https://wikitech.wikimedia.org/wiki/Help:Access ) for general information about accessing Labs. Pointers to more information on specific means of access below.
3.1 Tools home page The Tools home page:http://tools.wmflabs.org/
The Tools home page is publicly available and contains a list of all currently hosted Tool accounts along with the name(s) of the maintainers for each. Individual tool accounts that have an associated web page will appear as links. Users with access to the 'tools' project can create new tool accounts here, and add or remove maintainers to and from existing tool accounts.
3.2 SSH/SFTP/SCP Users can SSH to the 'tools' project via its bastion host: tools-login.wmflabs.org, provided that a public SSH key has been uploaded to the Labs account.
ssh your-instance-shell account name@tools-login.wmflabs.org
Note that if you plan to do heavy processing (compiling, etc), you should SSH to tools-dev.wmflabs.org.
3.2.1 Using 'take' to transfer ownership of uploaded files
Once you have logged in via SSH, you can transfer files via sftp and scp. Note that the transferred files will be owned by you. You will likely wish to transfer ownershihp to your tool account. To do this:
1. Become your tool account using 'become':
maintainer@tools-login:~$ become toolaccount local-toolaccount@tools-login:~$
2. As your tool account, 'take' ownership of the files:
local-toolaccount@tools-login:~$ take FILE
The 'take' command will change the ownership of the file(s) and directories recursively to the calling user (in this case, the tool account).
3.2.2 Using multiple ssh agents
If you use multiple ssh-agents (to connect to your personal or company system, for example), see Managing Multiple SSH Agents (https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents ) for more information about setting up a primary and a Labs agent.
3.4 Putty and WinSCP Note that instructions for accessing Tool Labs with Putty and WinSCP differ from the instructions for using them with other Labs projects. Please see Help:Access to ToolLabs instances with PuTTY and WinSCP (https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP) for information specific to Tool Labs.
3.5 Other graphical file managers (e.g., Gnome/KDE) For information about using a graphical file manager (e.g., Gnome/KDE), please see Accessing Tool Labs > Accessing instances with a graphical file manager (https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_a_graphical_file_manager)
>Sources: https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_public_and_private_instances
https://wikitech.wikimedia.org/wiki/Managing_Multiple_SSH_Agents
https://wikitech.wikimedia.org/wiki/Help:Access_to_ToolLabs_instances_with_PuTTY_and_WinSCP https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_a_graphical_file_manager http://www.mail-archive.com/labs-l@lists.wikimedia.org/msg00375.html for Gnome/KDE users and https://wikitech.wikimedia.org/wiki/User:Wikinaut/Help:Access_to_instances_with_PuTTY_and_WinSCP for Windows users https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Access_Project_Instances https://wikitech.wikimedia.org/wiki/Help:Access https://wikitech.wikimedia.org/wiki/Help:Putty https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#access_to_instances:_http.2Fsftp https://wiki.toolserver.org/view/Transferring_files (toolserver docs on transferring files)
4 Joining and creating a Tool account
4.1 What is a Tool account?
Tool accounts, which can be created by any âtoolsâ project member, are fundamental to the structure and organization of Tool Labs. Although each tool account has a user ID, they are not personal accounts (like a Labs account), rather services that consist of a user and group ID (i.e., a unix uid-gid pair) that are intended to run the actual tool or bot.
> Unix user: local-toolname > Unix group: local-toolname
Members of the Unix group include:
> the tool account creator > the tool account itself > (optionally, but encouraged!) additional tool maintainers
Maintainers may have more than one tool account, and tool accounts may have more than one maintainer. Every member of the group has the authorization to sudo to the tool account. By default, only members of the group have access to tool account's code and data.
A simple way for maintainers to switch to the tool account is with âbecomeâ:
maintainer@tools-login:~$ become toolname local-toolname@tools-login:~$
In addition to the user/group pair, each tool account includes:
¡ A home directory on shared storage: /data/project/toolname ¡ A ~/public_html/ and ~/cgi-bin/ directory, which are visible at http://tools.wmflabs.org/toolname/ and http://tools.wmflabs.org/toolname/cgi-bin, respectively ¡ Database access credentials: replica.my.cnf, which provide access to the production database replicas as well as to project-local databases. ¡ Access to the continuous and task queues of the compute grid
4.2 Joining an existing Tool account
All tool accounts hosted in Tool Labs are listed on the Tools home page (http://tools.wmflabs.org/). If you would like to be added to an existing account, you must contact the maintainer(s) directly.
If you would like to add (or remove) maintainers to a tool account that you manage, you may do so with the 'add' link found beneath the tool name on the Tools home page.
4.3 Creating a new Tool account
Members of the âtoolsâ project can create tool accounts from the Tools home page:
1. Navigate to the Tools home page: http://tools.wmflabs.org/ 2. Select the âcreate new toolâ link (found beside âHosted toolsâ near the top of the page 3. Enter a âService group nameâ. The service group name will be used as the name of your tool account. Do not prefix your service group name with local-; the management interface will do so automatically where appropriate, and there is a known issue that will cause the account to be created improperly if you do. Note: If you have only recently been added to the âtoolsâ project, you may get an error about not having appropriate credentials. Simply log out and back in to Wikitech to fix this
The tool account will be created and you will be granted access to it within a minute or two. If you were already logged in to your Labs account through SSH, you will have to log off then back in before you can access the tool account.
4.4 Deleting a Tool account
You can't delete a tool account yourself, though you can delete the content of your directories. If you really want a tool account to be deleted, please contact an admin.
4.5 Using Toolsbeta
Nearly all tool development is done on the 'tools' project, and 99.9% of the time, creating a tool account on this project will serve your needs. However, if your tool or bot requires an experimental library or a significant change to the 'tools' infrastructure--anything that could potentially negatively impact existing tools--you should experiment with the new infrastructure on toolsbeta.
To request access to toolsbeta, please visit #wikimedia-labs on IRC. You can also request acces via the labs-l mailing list or via bugzilla.
>Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Your_tool_account
https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Create_a_new_tool https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Project_storage https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Join_Projects https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools [note time lag btw tool account creation and user becomming tool]
https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Published_directories https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Can_I_have_a_subdomain_for_my_web_service.3F https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#Other_implementation_details [note about requesting tool resources] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools#Can_I_delete_a_tool.3F
https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuff (on bot credentials/multiple maintainer tool accounts) http://lists.wikimedia.org/pipermail/labs-l/2013-June/001311.html (info on toolsbeta 'bots') http://lists.wikimedia.org/pipermail/labs-l/2013-June/001313.html (more on toolsbeta) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001194.html (more on toolsbeta) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001198.html (more on toolsbeta) https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta http://lists.wikimedia.org/pipermail/labs-l/2013-March/001018.html (re: new package install)
5 Customizing a Tool account Once you have created a tool account, there are a few things that you can customize to make the tool more easily understood and used by other users. These include:
> Adding a tool account description (the description will appear on the Tools home page beside the tool name) > Creating a home page for your tool (if you create a home page for the tool, it will be linked from the Tools home page automatically)
Tool Labs will soon support mail to both Labs users and tool accounts (mail to a tool account will go to all maintainers by default). You can customize mail settings as well.
5.1 Creating a tool web page
To create a web page for your tool account, simply place an index.html file in the tool account's ~/public_html/ directory. The page can be a simple description of the tool or bot with basic information on how to set it up or shut it down, or it contain an interface for the web service. To see examples of existing tool web pages, click any of the linked Tool names on the Tools home page http://tools.wmflabs.org/
Note that some files, such as PHP files, will give a 500 error unless the owner of the file is tool account.
5.2 Creating a tool description
To create a tool description:
1. Log into your Labs account and become your tool account:
maintainer@tools-login:~$ become toolname
2. Create a â.descriptionâ file in the tool accountâs home directory. Note that this file must be HTML:
local-toolname@tools-login:~$ vim .description
3. Add a brief description (no more than 25 words or so) and save the file.
4. Navigate to the Tools home page (http://tools.wmflabs.org/). Your tool account description should now appear beside your tool account name.
5.3 Configuring mail -- mail forwarding
Mail from system daemons (grid, cron, etc.) is delivered to tool and user accounts. By default, tool accounts forward their mail to their maintainers' accounts, while user accounts store mail locally and users can read it (e.g., with mail).
To forward mail to your personal mail address from a Labs account:
1. Log in to your Labs account 2. In your home directory, create a file â.forwardâ
maintainer@tools-login:~$ vim .forward
3. Add the forwarding email address on a single line, e.g.
me@example.invalid
4. Ensure that .forward is only writable only by you, the account user. If â.forwardâ is writable by anyone other than you, mail is not delivered at all!
maintainer@tools-login:~$ chmod 600 ~/.forward
You can also use a .forward file in a tools account to redirect mail to a specified address (e.g., a mailing list) instead of sending all messages to the individual maintainers (which is the default).
... any other things people would like to use/configure?...
5.4 Configuring bots -- running them
In order to run a bot first of all you need to configure it. At first you should become your tool and after that you can command:
git clone --depth 3 https://gerrit.wikimedia.org/r/pywikibot/core.git cd core git submodule update --init cd externals git clone https://gerrit.wikimedia.org/r/pywikibot/spelling.git
you can use "compat" (formerly "trunk") instead of "core" (formerly "rewrite") if you're more familliar with trunk branch: git clone https://gerrit.wikimedia.org/r/pywikibot/compat.git pywikipedia cd pywikipedia git submodule update --init cd externals git clone https://gerrit.wikimedia.org/r/pywikibot/spelling.git
installing your bot in core or compat has two diffrenet methods, for core command python setup.py and answer the questions, for compat, command: python login.py and answer the questions. After installing you can run your codes directly via shell access but this is highly discouraged you must run your codes via jsub, cron, or SGE. jsub will use GRID engine (see above) For submitting a job you must command: jsub -once -N YOURJOBNAME python /data/project/YOUR-TOOL/PATHOFYOURCODE for example for running welcome.py in jsub you should command: jsub -once -N welcome python /data/project/YOUR-TOOL/core/scripts/welcome.py if you're using compat: jsub -once -N welcome python /data/project/YOUR-TOOL/pywikipedia/welcome.py
If you want to see status of your job command: qstat and if you want to see output of your job you can see them via: vim YOURJOBNAME.err vim YOURJOBNAME.out the former shows errors and the latter shows outputs for deleting a job, command: qdel NUMBEROFJOB you can find numberofjob in qstat. For further information see below
For further information about running bots you can see this help > Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Tool_Labs.27s_landing_page
https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Mail http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Mail:_to_do https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Tool_accounts https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools#What_about_file_permissions.3F_Who_can_see_my_code.3F http://lists.wikimedia.org/pipermail/labs-l/2013-May/001177.html & http://lists.wikimedia.org/pipermail/labs-l/2013-May/001179.html (note about permission) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001181.html (note about permission for public_html) http://lists.wikimedia.org/pipermail/labs-l/2013-February/000886.html (idea for what to put in html page) http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Gerrit http://www.mediawiki.org/wiki/Manual:Pywikipediabot/Installation https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Submitting_Jobs
6. Setting up code review and version control
Although it's possible to just stick your code in the directory and mess with it manually every time you want to change something, your future self and your future collaborators will thank you if you instead use source control, a.k.a. version control and a code review tool. Wikimedia Labs makes it pretty easy to use Git for source control and Gerrit for code review, but you also have other options.
6.1 Gerrit/Git Access to Git is managed via Wikimedia Labs and integrated with Gerrit. In order to use them for code review and version control with your tool accounts, you must request access. For more information, please see Gerrit/New repositories https://www.mediawiki.org/wiki/Gerrit/New_repositories
For more information about using Git and Gerrit in general, please see https://www.mediawiki.org/wiki/Gerrit
> Sources: https://www.mediawiki.org/wiki/Gerrit/New_repositories (gerrit)
https://www.mediawiki.org/wiki/Gerrit https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver (other versioning options) https://wikitech.wikimedia.org/wiki/Help:Getting_Started#to_Wikimedia_Code_Review https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Git_committers
7 Database access
Tool and Labs accounts are granted access to replicas of the production databases. Private user data has been redacted from these replicas (some rows are elided and/or some columns are made NULL depending on the table), but otherwise the schema is, for all practical purposes, identical to the production databases, and are sharded into clusters in much the same way.
<<<<<<< is the ABOVE accurate re replica databases/how they differ from production??? -------------<<<<<<<<<<<<<<<<<<
<<<<<<< are Labs accounts granted access ???????
The schema is, for all practical purposes, identical except that some rows are elided and/or some columns are made NULL depending on the dable (to redact private information). The database /name/ is altered systematically, however, to be suffixed with '_p' (so enwiki becomes enwiki_p).
All labs account are granted credentials to access th
Database credentials (user name/password) are stored in the replica.my.cnf file found in the tool accountâs home directory. To use these credentials with command-line tools by default , copy 'replica.my.cnf' to '.my.cnf'.
7.1 Naming conventions
As a convenience, each mediawiki project database (enwiki, bgwiki, etc) has an alias to the server it is hosted on. The alias has the form:
project.labsdb
where project is the name of a hosted mediawiki project (enwiki bgwiki bgwiktionary cswiki enwikiquote enwiktionary eowiki fiwiki idwiki itwiki nlwiki nowiki plwiki ptwiki svwiki thwiki trwiki zhwiki commonswiki dewiki wikidatawiki arwiki eswiki...
for a complete list, look at the /etc/hosts file on tools-login).
The database names themselves consist of the mediawiki project name, suffixed with _p (an underscore, and a p), for example:
enwiki_p (English Wikipedia database replica)
7.2 Connecting to the database replicas
You can connect to the database replicas by specifying access credentials and the host of the replicated database. For example:
To connect to the English Wikipedia replica:
mysql --defaults-file="${HOME}"/replica.my.cnf -h enwiki.labsdb enwiki_p [MAP: the host name is an alias that points at "where the enwiki_p database is" so that mysql knows where to connect to]
To connect to Wikidata: mysql --defaults-file=~/replica.my.cnf -h wikidatawiki.labsdb
To connect to Commons: mysql --defaults-file=~/replica.my.cnf -h commonswiki.labsdb
There is also a shortcut for connecting to the replicas: sql <dbname>[_p] [MAP: _p is optional, but implicit (i.e. the sql tool will add it if absent)]
To connect to the English Wikipedia database replica using the shortcut, simply type:
sql enwiki
7.3 Creating new databases
User-created databases can be created on the database hosting the replica servers or on a database local to the 'tools' project: tools-db. The latter tends to be a bit faster since that server has less heavy activity, and tools-db is the recommended location for user-created databases when no interaction with the production replicas is needed. Users have all privileges on the created database and grant options.
Database names must start with the name of the credential user, which can be found in your ~/replica.my.cnf (the name looks something like 'p50252g21636'), followed by two underscores:
username__
Note that users are granted complete control over there username__, but nothing else.
7.3.1 Steps to create a user database on the replica servers
If you would like your database to interact with the replica databases (i.e., if you need to do actual SQL joins with the replicas, which can only be done on the same cluster) you can create a database on the replica servers. <<<<<<<<< ------------??? example or two of when its appropriate to do this <<<<<<<<<<<
- this is only in the cases where you need to do actual SQL joins with the replicas; which can only be done on the same cluster.
To create a database on the replica servers:
1. Connect to the replica servers with the replica.my.cnf credentials. You must specify the host of the replica (e.g., enwiki.labsdb):
> mysql --defaults-file="${HOME}"/replica.my.cnf -h xxwiki.labsdb <---- is this right? <<<<<<<<<<<<<<<<< (no, you need to specify it)
2. In the mysql console, create a new database (where USERNAME is your credentials user and DBNAME the name you want to give to your database): <<<<---------- is this correct? the same naming convention as on tools-db ??????? It should be the same naming convention; users are granted complete control over theiruser__ and nothing else.
MariaDB [(none)]> CREATE DATABASE USERNAME__DBNAME ^^^ that works.
You can then connect to your database using: > mysql --defaults-file="${HOME}"/replica.my.cnf -h xxwiki.labsdb USERNAME__DBBAME
7.3.2 Steps to create a user database on tools-db
To create a database on tools-db:
1. Connect to tools-db with the replica.my.cnf credentials:
> mysql --defaults-file="${HOME}"/replica.my.cnf -h tools-db
2. In the mysql console, create a new database (where USERNAME is your credentials user and DBNAME the name you want to give to your database):
MariaDB [(none)]> CREATE DATABASE USERNAME__DBNAME
You can then connect to your database using: > mysql --defaults-file="${HOME}"/replica.my.cnf -h tools-db USERNAME__DBBAME
7.3 Joins between commons and wikidata and other project databases
??????? -- can you help with this section---???? <<<<<<<<<<<<<<<<<<
That needs to be written from scratch. I'll sit down and look at it tonight.
> Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Database_access
http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features (status of join) http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005382.html http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Database_replication.2Faccess https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Create_new_databases https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Replicated_databases https://wikitech.wikimedia.org/wiki/ToolLabsDatabasePlan https://wiki.toolserver.org/view/Database_access#Program_access [toolserver docs have these program access examples. would be nice to do something similar for Tool Labs]
8 Submitting, managing and scheduling Tool jobs on the grid Every non-trivial task performed in Tool Labs should be dispatched by the grid engine, which ensures that the job is run in a suitable place with sufficient resources. The basic principle of running jobs is fairly straightforward:
¡ You submit a job to a work queue from a submission server (e.g., -login) or web server ¡ The grid engine master finds a suitable execution host to run the job on, and starts it there once resources are available ¡ As it runs, your job will send output and errors to files until the job completes or is aborted.
Jobs can be scheduled synchronously or asynchronously, continuously, or simply executed once. If a continuous job fails, the grid will automatically restart the job so that it keeps going.
¡ ¡ 8.1 What is the grid engine? The grid engine is highly flexible system for assigning resources to jobs, including parallel processing. The Tool Labs grid engine is implemented with Open Grid Engine (the open-source fork of Sun Grid Engine). You can find more documentation on the Open Grid Engine [2] website.
Commonly used Grid Engine commands include:
qsub: submit jobs to the grid qalter: modify job settings (while the job is waiting or running) qstat: get information about a queued or running job qdel: abort or cancel a job
You can find detailed information about these commands in the Grid Engine Manual (http://gridscheduler.sourceforge.net/htmlman/manuals.html).
The Open Grid Engine commands are very flexible, but a little complex at first â you might prefer to use the helper scripts instead (jsub, jstart, jstop) described in more detail in the next sections.
8.2 Submitting simple one-off jobs using jsub
Jobs with a finite duration can be submitted to the work queue with either Open Gridâs qsub command or the jsub helper script, which is simpler to use and described in this section. (For information about qsub, please see the Open Grid Engine Manual http://gridscheduler.sourceforge.net/htmlman/manuals.html).
To run a finite job on demand (at interval from cron, for instance, or from a web tool or the command line), simply use the jsub command:
$ jsub [optionsâŚ] program-or-script [argsâŚ]
By default, the jsub command will schedule the job to be run as soon as possible, and print the eventual output to files (âjobname.outâ and âjobname.errâ) in your home directory. Unless a job name is explicitly specified with jsub parameters, the job will have the same name as the program, minus extensions (e.g., if you had a program named foobot.pl which you started with jsub, the job's name would be foobot.)
Example: (from current docs/not pasting correctly, but whatever. Will fix in final version)
local-shtest@tools-login:~$ jsub mybot.sh
prog?:
Your job 105033 ("mybot") has been submitted
local-shtest@tools-login:~$ qstat
job-ID prior name user state submit/start at queue slots ja-task-ID -----------------------------------------------------------------------------------------------------------------
105033 0.25000 mybot
local-shtest r 05/24/2013 08:52:00 task@tools-exec-02.pmtpa.wmfla 1 local-shtest@tools-login:~$ qstat local-shtest@tools-login:~$ ls access.log mybot.err mybot.sh replica.my.cnf cgi-bin mybot.out public_html local-shtest@tools-login:~$ cat mybot.out user_editcount 1016
jsub options In addition to a number of customized options, jsub supports many, but not all qsub options:
-stderr Send errors that occur during job submission to stderr rather than the error output file (errors that occur while runnning the script are always sent to the error file).
-mem value Request <value> amount of memory for the job, where value is number suffixed by 'k', 'm' or 'g'. The default is 256m
-once Run the named job only once; fail the job if a job by the same name is already running or queued
-continuous Start a self-restarting job on the continuous queue. Please see the section on continuous jobs and jstart for more information.
-quiet Suppress output if job has been submitted successfully (e.g. if set, cron jobs will not send mail on successful submit)
-i, -o, and -e Selects the file used for standard input, output and error of the job, respectively. By default, jsub will append stdout and stderr to the files jobname.out and jobname.err in the tool account's home directory, and will not have standard input. If a directory is given for -o or -e, new files jobname.ojobid and jobname.ejobid are created there for each job (for more info, see qsub docs)
-j y send standard output and error together to the output file (for more info, see qsub docs)
-sync y Normally, jsub queues up the job and returns immediately. This allows you to wait for the job to be complete instead.(for more info, see qsub docs)
-cwd Start the script in the same directory you invoked jsub from (for more info, see qsub docs)
-N jobname
Pick a different job name (for more info, see qsub docs).
8.2.1 Naming jobs
The job name identifies the job and can also be used to control it (e.g., XXX). By default, jobs are assigned the name of the program or script, minus its extension. For instance, if you started a program named foobot.pl with jsub, the job's name would be foobot.
It's important to note that you can have more than one job, running or queued, bearing the same name. Some of the utilities that accept a job name may not behave as expected in those cases.
Specify a different name for the job using the jsubâs âN option:
$ jsub âN NewName program-or-script [argsâŚ]
8.2.2 Allocating additional memory
By default, jobs are allowed 256MB of memory; you can request more (or less) with jsubâs -mem option (or qsub's -l h_vmem=memory). Keep in mind that a job that requests more resources may be penalized in its priority and may have to wait longer before being run until sufficient resources are available.
$ jsub âmem 500m program-or-script [argsâŚ]
8.2.3 Synchronizing jobs By default, jobs are processed asynchronously in the background. If you need to wait until the job has completed (for instance, to do further processing on its output), you can add the -sync y option to the jsub command:
$ jsub âmem 500m program-or-script [argsâŚ]
8.2.4 Running a job only once
If you need to make certain that the job isn't running multiple times (such as when you invoke it from a crontab), you can add the -once option. If the job was already running or queued, it will simply mark the failed attempt in the error file and return immediately.
¡ $ jsub -once -N jobname php /data/project/tool/task/execute.php
8.3 Submitting continuous jobs (such as bots) with jstart
8.3 Managing jobs
8.3.1 Job status
8.3.2 Stopping jobs
8.3.3 Suspending and unsuspending jobs
8.4 Scheduling jobs at regular intervals with cron
8.4.1 Creating a crontab
8.4.2 Specifying time zones
>Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#The_grid_engine
http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Job-system http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#2.29_Computing_grid http://lists.wikimedia.org/pipermail/labs-l/2013-July/001416.html https://wikitech.wikimedia.org/wiki/Help:Cron https://en.wikipedia.org/wiki/Cron
9. Developing on Tool Labs 9.1 Licensing 9.2 Heavy processing 9.3 Installing additional packages 9.4 Installing and running your bot 9.5 Where to put shared Tool code 9.5.1: Shared code: Use git submodules 9.5.1: Shared config files / other files - put it in /shared (readable by all) or add the other tool to current tool's group (In progress, doesn't work yet - https://bugzilla.wikimedia.org/show_bug.cgi?id=51990) 9.5.1: Files shared by multiple tools? 9.6 Web services 9.6.1 Running scripts 9.6.2 Using cookies 9.6.3 Web logs 9.6.4 Tips and tricks (add them here!) 9.7 PyWikipediaBot 9.7.1 Setting up PyWikipediaBot for your account 9.7.2 Using PyWikipediaBot with virtualenv 9.7.3 Tips and tricks (add them here!) ... what else?... 9.8 Tips for working collaboratively
.... do you have any? ...
(1) Use source control
9.9 how to use <programming-language-X> to write tools on labs' > Sources: http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Licenses:_Info https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Do_I_explicitly_have_to_specify_the_license_of_my_tools.3F (licencing)
http://www.mediawiki.org/w/index.php?title=File:Tool_Labs_presentation_(Hackaton_2013).pdf&page=11 (heavy processing info) http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design#1.29_Interactive_.2F_Development_environment (more on heavy processing) https://wikitech.wikimedia.org/wiki/Obsolete:Bots_project_documentation http://lists.wikimedia.org/pipermail/labs-l/2013-February/000903.html (shared tool code) https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Cookies https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Published_directories (running scripts)
https://wikitech.wikimedia.org/wiki/Help:Getting_Started#Log_your_actions
http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Logs.2FStats
https://wikitech.wikimedia.org/wiki/User:Russell_Blau/Using_pywikibot_on_Labs (pywikibot set up info) http://lists.wikimedia.org/pipermail/labs-l/2013-May/001253.html (collaboratively working tip) https://wikitech.wikimedia.org/wiki/User:Legoktm/pywikibot_on_tools_lab (pywikipediabot in virutalenv) Also, if you ever want to submit a job via python, I wrote another simple library to help do that: https://wikitech.wikimedia.org/wiki/User:Legoktm/wmflib (from legoktm)
10 ¡ Redis
10.1 Security
10.2 A note about memcache
> Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Redis
http://lists.wikimedia.org/pipermail/labs-l/2013-June/001354.html http://lists.wikimedia.org/pipermail/labs-l/2013-July/001413.html (memcache thread)
11. Public dataset dumps
>Sources:https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Dumps
https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Public_datasets https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#More
12. CatGraph (aka Graphserv/Graphcore)
In-memory graph database for fast access to Wikipedia category structure.
> Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource:Catgraph
13. Troubleshooting
>Any tips? Please add them here!
If you run into problems, please feel free to come into the #wikimedia-labs IRC (chat) channel using http://webchat.freenode.net/?channels=#wikimedia-labs and look for Coren (Marc-Andre Pelletier) or petan (Petr Bena). The labs-l mailing list at https://lists.wikimedia.org/mailman/listinfo/labs-l is another good place to ask for help, especially if the people in chat are not responding. You can also search wikitech.wikimedia.org for help pages, or look more widely with the custom search at https://www.google.com/cse/home?cx=010768530259486146519:twowe4zclqy .
14 ¡ Backups 14.1 What gets backed up? The basic rule is: there is a lot of redundancy, but no backups of labs projects beyond the filesystem's time travel feature for short term disaster recovery. Labs users should make certain that they use source control to preserve their code, and make regular backups of irreplacable data. 14.2 Time travel
>Sources: https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Backup https://blog.wikimedia.org/2013/05/30/preparing-for-the-migration-from-the-wikimedia-toolserver-to-tool-labs/ (notes on time travel) http://lists.wikimedia.org/pipermail/labs-l/2013-July/001426.html (instructions)
15 Moving a tool from Toolserver to Tool Labs (brief overview here, with a link to "Migration of Toolserver tools" https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools
16 ¡ FAQ (additions to current list)
Is there a way on Labs to easily share data across my Tool accounts? http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#Filesystem_.2F_shared_storage Is there a GUI tool for database work?
Not in Tool Labs, but you can run one locally on your computer (for example the MySQL Workbench http://dev.mysql.com/downloads/tools/workbench/). Here is how you connect to the database: For the login: username@tools-login.wmflabs.org For the database, it depends on the exact one you want to use, of course - for example: enwiki.labs
https://wikitech.wikimedia.org/wiki/Nova_Resource_Talk:Tools/Help#GUI_tool_for_databasework
Why does public_html not work in my home directory? Users do not and cannot have a public_html folder to themselves. The only web accessible directories are in /data/project/<toolname>/public_html/*. To have a url such as http://tools.wmflabs.org/<username>/ there must be a tool called <username> and therefor a folder called /data/project/<username>/public_html/
https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ
I get a Permission denied error when running my script. Why's that? Make sure that you are running your script from your tool account rather than your user account.
https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ
How can I detect if I'm running in Labs? How can I detect in which project (tools or toolsbeta) it's running? There is a file that contains the project name on every labs instance: /etc/wmflabs-project Testing for its presence tells you "you are in the WMF Labs", and checking its contents will tell you which project; that would be "tools" for the Tool Labs, or "toolsbeta" for the experimental tool labs. > http://lists.wikimedia.org/pipermail/labs-l/2013-July/001394.html
My connection seems slow. Any advice? When connecting to Tool Labs from Europe you might have higher ping times. Try using mosh (mosh.mit.edu). To connect, use mosh -a tools-login.wmflabs.org (-a to force predictive echo) (instead of ssh tools-login.wmflabs.org) > https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Slow_connections
I want to ssh to bot instances outside of Labs. Any advice?
If you want to ssh to specific bot instances other than tools-login.wmflabs.org, it is helpful to create a new SSH key: $ ssh-keygen $ cat ~/.ssh/id_rsa.pub ssh-rsa .... user@host Copy the ssh-rsa ... user@host line to your authorized keys in the labs console (https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack ). > https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#Connecting_to_different_servers
How can I check my filesystem usage? >https://wikitech.wikimedia.org/wiki/Help:Shared_storage <- Not answered there, says come ask Ryan on IRC. :)
Where can I see my page stats? > Webserver page view/visit statistics: https://graphite.wikimedia.org/ (requires Labs account), https://gdash.wikimedia.org/ (open to the world)
What languages are supported? > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Languages <- I think here the question has not been answered - i18n support for what exactly? This probably refers to the i18n support provided for by https://wiki.toolserver.org/view/Toolserver_Intuition which is currently being ported. (Krenair might have a status update)
Can I use any free license? You can use any OSI-approved license. Read more on the licenses on the Open Source Initiative's website: http://opensource.org/licenses > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Licenses:_Info
My Tool requires a package that is not currently installed in Tool Labs. How can I add it? You might not be the only one missing that package. Please submit a ticket in bugzilla and ask the admins to install it project-wide. If they have reasons not to do so you can always install software locally / just for yourself. > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bots_and_webservices_project
How can I change the permissions of a file in my Tool account? You cannot use chown. http://lists.wikimedia.org/pipermail/labs-l/2013-July/001377.html General documentation about your tool account is here: https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Your_tool_account
Can I override the default VIM settings? > http://lists.wikimedia.org/pipermail/labs-l/2013-June/001294.html >http://lists.wikimedia.org/pipermail/labs-l/2013-June/001297.html
My tool needs a more specialized infrastructure than Tool Labs provides. What should I do? Tool Labs is a simplified environment intended to be a direct toolserver replacement for most small tools. When you need something more complicated or when you need to manage specialized infrastructure, what Tool Labs can't do you can probably do with your own Labs project! (Instead of inside one of the projects "tools" or "toolsbeta".) > http://lists.wikimedia.org/pipermail/labs-l/2013-June/001350.html
I keep reading about puppet. What is it? As a tool maintainer you don't have to worry about puppet.
The nutshell: puppet is a system by which you describe the configuration of a machine. When used, it will apply the necessary changes to make the machine you apply it to match that configuration.
In practice, the sysadmin would make any change of configuration intended for the machine in puppet (including what to install, files to edit, etc) so that it can be reapplied to a blank machine to configure it "just like it was", or to make a clone, and so on. In the case of a project where the tool maintainer do the system administration, it might be desirable to acually configure and install the tool /itself/ through puppet so that it is easy to return to a known state.
In the case of the Tool Labs, however, the actual tools would not normally be configured through puppet (it's possible, but not worthwile): they live on a shared filesystem rather than on the individual machines. What puppet /is/ used for is to maintain the components of the grid, making adding "one more compute node" or "an extra webserver" as simple as to create a new instance and set puppet accordingly. When we find a tool has a dependency, the tools labs sysadmins will add it to puppet so that every host part of the grid (current and future) will have that configured accordingly without manual intervention. (source: http://lists.wikimedia.org/pipermail/labs-l/2013-March/001047.html)
What is the labsconsole? [is it important to keep this concept? I think it's worth the sentence below because in some places the docs still refer to "labs console".] https://wikitech.wikimedia.org/wiki/Help:Access It's the old name of what is now Wikitech (wikitech.wikimedia.org)
I'm being prompted for a password when I try to 'become my-tool-account'. What's wrong? If you are seeing "a password is required" message when you try to become your tool (i.e., sudo yourtoolaccount), it is likely because you were logged in to your Labs account when you created the tool account. Unix group membership is checked on login only, so an existing session will not have access to the new tool group. Log out and then log in to your Labs account again to fix this problem.
Are there any plans for adding monitoring or profiling tools? Yes, in the very long term and not guaranteed. Want to help? Find out more here: https://wikitech.wikimedia.org/wiki/User:Yuvipanda/Icinga_for_tools
Can I use BitBucket or a local git for versioning and backups? If you wish to use BitBucket or a local git for versioning and backups, you can find some notes from a user who has done this here: https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#GIT
Moving from Toolserver to Tool Labs: https://www.mediawiki.org/wiki/Wikimedia_Labs/Migration_Of_Toolserver_Tools This is a very helpful doc.
...I suggest we collect more toolserver-transition specific help in this document and link to it from Tool Labs Guide...
Modifications: Replace "bot project" with "toolsbeta project" (the current name of the project)
(modify) 6.9 What's a "service group"? --> Note that Tool Labs terminology is "Tools account" Perhaps clarify: "service group" is a general Labs concept; "tool account" is what we use service groups for in the Tool Labs project.
Additional help topics: 6.18 What should I do about Toolserver links?
> http://lists.wikimedia.org/pipermail/toolserver-l/2013-May/006143.html (.htacess redirect info)
6.19 Anything I should know about Ubuntu Linux? Not familiar with Ubuntu Linux? Below, you can find some good sources of information. You have to do with a Debian-based Linux distribution. Basically if there are different documents you'll probably want to look for the server (not the desktop) edition. The current stable release that has long term support is "Ubuntu Precise", a.k.a "Precise Pangolin", a.k.a. "12.04". This is what's running in Tool Labs. Once you know what a software/command line tool is called you can read the manual directly in Tool Labs by entering "man <name_of_software>".
Official Ubuntu Documentation: https://help.ubuntu.com/
Find groups or IRC channels in your language: http://loco.ubuntu.com/teams/
Stuff in German: http://wiki.ubuntuusers.de - very good explanations of software in German, command line as well as graphical tools. (In contrast, the forum.ubuntuusers.de contains relatively much bullshit.)
Please add more links to docs in different languages if you can really recommend them!
> Context for question: http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features#End-user-support > Source for answer: You!
6.20 Can I move my SVN repository? > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Version_control
> http://lists.wikimedia.org/pipermail/toolserver-l/2013-June/006175.html > http://lists.wikimedia.org/pipermail/toolserver-l/2013-June/006177.html
6.22 Will Tool Labs have meta database tables like the ones in Toolserver? > See thread: http://lists.wikimedia.org/pipermail/labs-l/2013-July/001390.html https://bugzilla.wikimedia.org/show_bug.cgi?id=48626 https://bugzilla.wikimedia.org/show_bug.cgi?id=48625
6.23 Can I join user databases with wiki ones? https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features That's actually working today; databases created by users can be joined with the replica databases so long as they are created on the same DB (i.e.: connect to the same place)
6.24 Will the commons database be replicated to all clusters, like it is on the Toolserver? https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features It is not. However, there is a federated database link to commons and wikidata on every shard. There are some caveats in using them for good performance, but they are available. (Needs documenting)
6.25 Can I import my JIRA bug tracker information to Bugzilla? If so, how? No, sorry. At the moment, we don't have a solution for you to keep the bugreport from jira. Some people have done such a migration but this would have to be thoroughly tested and would probably need a lot of manual intervention. The good news is: If you wish you can request to have your own "component" in bugzilla for supporting your tool. > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bug_tracker > any other suggestions?
6.26 Does the Tool Labs grid support qcronsub? >https://wikitech.wikimedia.org/wiki/User:Magnus_Manske/Migrating_from_toolserver#qsub_et_al Not directly. Similar functionality is available with the -once option to jsub but the common use case of "continually try to restart my tool if it failed" is better served with the support for continuous applications (-continuous, jstart)
Can I move a multimaintainer project to Tool Labs? >https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuff The short answer; all Tool Labs tools are "multimaintainer" as meant on toolserver (i.e.: shared project space, group membership)
My bot has many maintainers. How should we handle credentials? > https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Other_project-related_stuff Credentials the bot itself uses to edit/perform actions is subject to the target wiki's policies on sharing credentials and may need to be cleared with them.
Does Tool Labs support all the packages Toolserver does?
>https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_toolserver_features#Bots_and_webservices_project (Is this yes? or were requested packages dif. from total ones?) > https://wiki.toolserver.org/view/User:Dab/Debian-Packages > https://jira.toolserver.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+TS+AND+%28summary+~+install+OR+description+~+install+OR+comment+~+install%29+AND+issuetype+%3D+Task+AND+resolution+%3D+Fixed
It's a "yes" insofar as "If you need it and it is open source, we can get it for you." We have not, on the other hand, preemptively every version of every dependency that has been enumerated; but do so on demand (partly to avoid installing obsolete packages that are no longer relevant).
INVENTORY OF CURRENT TOOL-LAB RELATED DOCUMENTATION, AND PROPOSED UPDATE/REDIRECT PLAN:
https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [UPDATE/REPLACE WITH NEW, EXPANDED GUIDE OUTLINED ABOVE]
https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools [UPDATE WITH HIGH-LEVEL OVERVIEW/INFO ABOUT CREATING ACCOUNT]
Sources for expansion: :http://www.mediawiki.org/wiki/File:Tool_Labs_presentation_(Hackaton_2013).pdf
https://wikitech.wikimedia.org/wiki/Main_Page [UPDATE: Under Labs/add note about Tool Labs and add link to Tools project and Tool Labs guide]
https://wikitech.wikimedia.org/wiki/Help:Terminology [UPDATE with additional terms--Tool Labs, Toolsbeta project, Tools project, grid, cluster... other terms?]
http://tools.wmflabs.org/ [UPDATE to clarify what these links are (e.g. "Request access to Tool Labs"; add additional context at top of page to explain what is on page]
https://www.mediawiki.org/wiki/Wikimedia_Labs [Appears current/recently updated]
https://www.mediawiki.org/wiki/Developer_access [UPDATE with context-- what does 'developer access' mean exactly? ]
https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features [update]
https://wikitech.wikimedia.org/wiki/Help:Getting_Started [Appears current]
https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta [UPDATE with more info about Toolsbeta and how to get access]
https://wikitech.wikimedia.org/wiki/Help:Shared_storage#Project_storage [UPDATE: there's a TODO in the page]
https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Migration_of_Toolserver_tools [Update: replace 'bot' with 'toolsbeta'/also add content generated/organized in sprint]
https://www.mediawiki.org/wiki/Wikimedia_Labs/TODO [Appears current]
http://www.mediawiki.org/wiki/Toolserver/List_of_Tools [Appears current]
http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design [Appears current]
https://wikitech.wikimedia.org/wiki/Help:Shared_storage [appears current]
https://www.mediawiki.org/wiki/Special:PrefixIndex/Wikimedia_Labs/:
https://www.mediawiki.org/wiki/Wikimedia_Labs/Authentication_improvement_project [Appears current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Instance_creation_improvement_project [Appears relatively current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/FAQ [recently updated, but would love to redirect this to guide/combine info there] https://www.mediawiki.org/wiki/Wikimedia_Labs/Create_shared_sql_service_for_all_projects (last updated jan 2012) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Per-project_saltstack_remote_execution (last update june 2012) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Shared_home_directories_per_project (last update dec 2011) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/TODO (Appears current) https://www.mediawiki.org/wiki/Wikimedia_Labs/Things_to_fix_in_beta (Last updated June 2012) Redirect to roadmap/ https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/List_of_Toolserver_Tools/tool [Appears current] https://www.mediawiki.org/wiki/Wikimedia_Labs/Log_bot_for_wikimedia-labs_channel (last update Dec 2011) Redirect https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Nagios_management_without_exported_puppet_resources (last update Dec 2011) Redirect to roadmap/https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Roadmap_en [Update/to replace 'bots' with 'toolsbeta' to avoid confusion] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Rules [Update with rules] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs [Update overview at top of page] https://www.mediawiki.org/wiki/Wikimedia_Labs/Reverse_proxy_for_web_services [last update march 2012/redirect to roadmap] https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design (appears current)
https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/List_of_Toolserver_Tools [appears current]