Arturo and Chase Onboarding Sessions

Dec 19, 2017

https://phabricator.wikimedia.org/T181647

commands agnostic to distro? (upgrades for distro and security and wmf? -- trusty, jessie...stretch)

https://gerrit.wikimedia.org/r/#/c/398458/ https://gerrit.wikimedia.org/r/#/c/398079/

https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Attended_package_upgrades

http://tools-services-01.tools.eqiad.wmflabs/repo/ (APTLY) (to collab with external contributors, instead of using prod reprepro)
- probably merge into prod reprepro, using another component or something

1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/thirdparty amd64 Packages

    release v=14.04,o=Wikimedia,a=trusty-wikimedia,n=trusty-wikimedia,l=Wikimedia,c=thirdparty
    origin apt.wikimedia.org

1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/universe amd64 Packages

    release v=14.04,o=Wikimedia,a=trusty-wikimedia,n=trusty-wikimedia,l=Wikimedia,c=universe
    origin apt.wikimedia.org

1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/main amd64 Packages

    release v=14.04,o=Wikimedia,a=trusty-wikimedia,n=trusty-wikimedia,l=Wikimedia,c=main
    origin apt.wikimedia.org

1500 http://tools-services-01.tools.eqiad.wmflabs/repo/ trusty-tools/main amd64 Packages

    release o=. trusty-tools,n=trusty-tools,l=. trusty-tools,c=main
    origin tools-services-01.tools.eqiad.wmflabs
500 http://security.ubuntu.com/ubuntu/ trusty-security/universe Translation-en
500 http://security.ubuntu.com/ubuntu/ trusty-security/main Translation-en
500 http://security.ubuntu.com/ubuntu/ trusty-security/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-security,n=trusty,l=Ubuntu,c=universe
    origin security.ubuntu.com
500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-security,n=trusty,l=Ubuntu,c=main
    origin security.ubuntu.com
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/universe Translation-en
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/restricted Translation-en
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/multiverse Translation-en
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/main Translation-en
100 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/multiverse amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=multiverse
    origin nova.clouds.archive.ubuntu.com
100 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=universe
    origin nova.clouds.archive.ubuntu.com
100 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/restricted amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=restricted
    origin nova.clouds.archive.ubuntu.com
100 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-backports,n=trusty,l=Ubuntu,c=main
    origin nova.clouds.archive.ubuntu.com
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/universe Translation-en
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty-updates/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty-updates,n=trusty,l=Ubuntu,c=universe
    origin nova.clouds.archive.ubuntu.com
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe Translation-en
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/main Translation-en
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty,n=trusty,l=Ubuntu,c=universe
    origin nova.clouds.archive.ubuntu.com
500 http://nova.clouds.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
    release v=14.04,o=Ubuntu,a=trusty,n=trusty,l=Ubuntu,c=main
    origin nova.clouds.archive.ubuntu.com

Nov 28, 2017

Travel!
going through pending tickets and patches assigned

Package upgrade workflow (https://phabricator.wikimedia.org/T181647):

Unattended things:

   * All cloud instances get all unattended upgrades from WMF and distro by default
    - Security updates <-- add a patch (Arturo)
    - distro package upgrades  https://gerrit.wikimedia.org/r/#/c/390431/2/modules/apt/manifests/unattendedupgrades.pp 
    - wmf package upgrades https://gerrit.wikimedia.org/r/#/c/389480/
    :* Add a patch to put this behind a hiera setting (Chase)
    :* kernel updates still sleeping in toolforge (task?) https://phabricator.wikimedia.org/T180809
    :* packages handling configuration files correctly (which means preserving settings) https://gerrit.wikimedia.org/r/#/c/392421/
   - backports is an open question

Choosing to handle updates manually:

   * A project can choose to set a hiera key that will stop these upgrades from happening ( one key per type of upgrade candidate)
   * A script exists to run on an instance to generate a report for available package upgrades. (https://phabricator.wikimedia.org/P6365)
   :* Broken down by wmf vs distro?
   * The script that is used to generate the report or another script can be used to do the upgrades. This is a replacment for unattended and is ...attended upgrade solution.

Nov 21, 2017

Unattended-upgrades done w/ Chase: https://phabricator.wikimedia.org/T177920 notes: https://etherpad.wikimedia.org/p/389480
- pending https://phabricator.wikimedia.org/T180811
Unattended upgrades pending: https://phabricator.wikimedia.org/T180254
wiki replicas automation. Step 1: documentation https://phabricator.wikimedia.org/T180513
role::puppetmaster::standalone has no firewall rule for port 8140 https://phabricator.wikimedia.org/T154150

root@tools-bastion-03:~# host enwiki.web.db.svc.eqiad.wmflabs enwiki.web.db.svc.eqiad.wmflabs is an alias for s1.web.db.svc.eqiad.wmflabs. s1.web.db.svc.eqiad.wmflabs has address 10.64.37.15

 user_properties:
   source: user_properties
   view: select up_user, up_property, up_value
   where: >
     up_property in ( 'disablemail', 'fancysig', 'gender', 'nickname' )

 user_properties_anon:
   limit: 2
   source: ["user_properties", "user", "meta_p.properties_anon_whitelist" ]
   view: select cast(extract(year_month from user_touched)*100+1 as date) upa_touched, up_property, up_value
   where: user_id=up_user and up_property like pw_property

Nov 2, 2017

tools-bastion-03

Recurrent problem.

arturo's onboarding page

Make a network diagram

Openstack: everything is liberty execpt horizon which is mitaka.

Wiki replicas <-- look at them.

Next week: shadow clinic duty person. Madhu?

Oct 31, 2017

2017-11-01 is a public holiday for Arturo
- We should get some/all of these for the next few months on the team calendar
Arturo trying to understand which servers are physical, which are virtual, and how they link together
- Wants a map of how things fit together
Nick poked Arturo about setting up his User page on metawiki

Chase to find the newly formed ongoing topographical docs
Everything is physical *except* Cloud VPS tenents and a few things on Ganeti in "production"

https://wikitech.wikimedia.org/wiki/Ganeti <--- KVM + DRBD (NOTE: 2017-10-31: already read the docs)

https://tools.wmflabs.org/openstack-browser/project/
https://tools.wmflabs.org/openstack-browser/project/tools <-- all of the VMs in Toolforge
- names of the vms give a hint to what they do:
  - tools-k8s-* -- kubernetes core services
  - tools-docker-* -- kubernetes related Docker hosts (Docker registry, Docker image builder host)
  - tools-worker-* -- kubernetes exec nodes
  - tools-paws-* -- a second kubernetes cluster that powers PAWS <https://paws.wmcloud.org/>, run by Yuvi
  - tools-exec-* -- Grid Engine execution nodes for "normal" tasks
  - tools-webgrid-* -- Grid Engine execution nodes for "web" tasks

Oct 26, 2017

topics?
I've been working on this task today: nfsiostat diamond collector

https://phabricator.wikimedia.org/T179024

To test a patch, depool a node and test in a node:

    https://phabricator.wikimedia.org/P6194

Oct 24, 2017

Puppet (how does it work)

- LDAP is the "same sign-on" solution for all things that are not MediaWiki
- Unix user accounts outside of Cloud VPS are not connected directly to LDAP
- Data is managed by Puppet based on modules/admin/data/data.yaml

puppetmaster1001.eqiad.wmnet
- puppet-merge

<change> y/n?

new installs

https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Installation

New installs

Getting the MAC address for a new server
https://wikitech.wikimedia.org/wiki/Platform-specific_documentation
https://wikitech.wikimedia.org/wiki/Server_Lifecycle

New server: foo.eqiad.wmnet management network: foo.mgmt.eqiad.wmnet management network: <asset tag>.eqiad.wnet == mgmt

https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/HP_DL3N0 show system1/network1/Integrated_NICs

files/dhcpd/linux-host-entries.ttyS1-115200:host labcontrol1001 {

   # onboard management

Host *.mgmt.*.wmnet

       StrictHostKeyChecking ask
       UserKnownHostsFile /Users/cpettet/.ssh/wmf_mgmt_hosts
       
https://gerrit.wikimedia.org/r/#/admin/projects/operations/dns

https://phabricator.wikimedia.org/diffusion/

baham.eqiad.wment authdns-update

From pupetmaster1001: new-install <server>