Docker/SIG/Meetings/20190228
2019-02-28
[edit]Lead: Greg Grossmeier Participants: 18
- What is this group? What is a SIG? What are we trying to do? - Greg
- Everyone here is working on something to do with containers, this meeting gives people space to talk about projects they are working on
- Current plan is to have a meeting per month
- Notes posted on wiki -- you are being recorded ;)
- Description/status of the in-flight projects
- Mark H/hexmode
- not using a lot right now, using Jenkins docker images that WMF uses in CI, wanting to containerize the services so I can deploy for his clients
- David Barratt
- https://hub.docker.com/_/mediawiki / https://github.com/wikimedia/mediawiki-docker
- Not intended for WM Production
- Official for Docker not for MW
- Alexandros K
- SRE, to manage by end of next fiscal, to move to k8s for most services and MW
- I dislike docker, doing too many things, but like as the env with k8s ontop
- end-goal of work currently is to have alignment of production and development envs
- Addshore
- mediawiki-docker-dev - https://github.com/addshore/mediawiki-docker-dev
- also have images for wikibase and wdqs - https://github.com/wmde/wikibase-docker
- Working on a new node server side render to be deployed in production using blubber etc - https://phabricator.wikimedia.org/T212189
- Tyler
- Deployment Pipeline - https://wikitech.wikimedia.org/wiki/Deployment_pipeline
- Blubber - https://wikitech.wikimedia.org/wiki/Blubber
- Jeena Huneidi
- Working on local development environment using Minikube
- Minikube: https://github.com/kubernetes/minikube
- https://gerrit.wikimedia.org/g/releng/local-charts
- has mariadb, MW, and parsoid, currently working on Restbase
- Haven't created any Docker images others should use, still working on that part
- The goal here is just to make local development easy, and to get parity with deployment pipeline
- Working on local development environment using Minikube
- Mateus
- working on a maps stack for development
- Cassandra, Redis, Tilerator
- https://github.com/mateusbs17/gisdock
- Mark H/hexmode
- Discussion / Questions
- Greg: We've got a lot of stuff - it seems like there's a lot of specific-use-case Docker work going on. Is there some possibility of us using some of the same basic foundations to build on - does anyone see potential overlap that we could start working on?
- Jeena: Sharing base images
- Kosta Harlan: mediawiki-docker-dev and the local-charts could probably be using the same base image for php / apache?
- Adam: one of the biggest things people have been wanting is VE, parsoid/etc to have a real VE running
- Adam: also EventLogging from Analytics - https://github.com/wikimedia/eventlogging/tree/master/docker (for example)
- TODO: Greg to make the list of all in-flight things on the docker SIG wiki page
- Tyler: random list of images in the wikimedia-docker registry, FWIW: https://people.wikimedia.org/~thcipriani/docker/
- Jeena: list ones that have gone through some sort of approval
- David: agreed ^, and also re the "official" images from Dockerhub are *required* to extend from the official dockerhub base images. Our images don't have that policy nor extend from those.
- Alex: https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/
- Cole: Base image lifecycle?
- Alex: SRE is working on that a bit now, very early. Will be sharing later if you're interested
- Tyler: wanted to share link of an occasionally (hourly-ish) updated list production docker registery: https://people.wikimedia.org/~thcipriani/docker/
- Tyler: those are images used in production or available for use in production
- Alex: you can build images on top of that, keep them up to date to address the security concerns linked above
- Alex: you should start relying on these for things you want to see in production, feel free to use whatever else for random other non-production things you're working on
- Tyler: docker-pkg project, from Giuseppe of SRE, manages base images for production and currently for CI
- Antoine: docker-pkg recently received lot of updates. Give it a try!:)
- Dan G: What criteria would something be assessed by for officilanes
- What is meant by "official"
- dockerhub-official vs wmf-official vs wmfproduction-official
- plan to use same image for development as well as production
- Eric: documentation/social project question: one of the best use cases is new developers. QUickly hit limtes of MW-V with wikibase/ES. Docker seems the way to go. The work that david and adam have been doing is super helpful, but on top of that some documentation for new people: here are some base images, here are things you can plug in. Have some good documetation for what parts can be assembled with guidance to get you what you need.
- Brennen: there's not one clear entry point for a new developer. MW-V, docker-dev, and a bunch of people are rolling their own solution. Documentation is a solid idea. "this is the funnel"
- TODO: Brennen and Eric Gardner volunteer to help documentation on this
- Alex: documentation portal with at least Pipeline focused
- Make sure those are aligned ^^^
- Kosta: can we suppor the local-charts initiative. I'd want to put time into that if it's going to become the new dev env
- Jeena: please do help out with the local-charts work
- Brennen: we want feedback, can people try spinning up these charts and if they work or not etc?
- Greg: We've got a lot of stuff - it seems like there's a lot of specific-use-case Docker work going on. Is there some possibility of us using some of the same basic foundations to build on - does anyone see potential overlap that we could start working on?
Existing docs / entry points
[edit]- https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker
- https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker#Docker_development_environment
- https://www.mediawiki.org/wiki/Selenium/Node.js
- https://wikitech.wikimedia.org/wiki/Blubber
- https://www.mediawiki.org/wiki/MediaWiki-Vagrant (not for Docker per se, but relevant to new devs)
Followups
[edit]- TODO - Greg to add a list of all in-progress projects on the Docker SIG page
- TODO -- some folks to look at https://gerrit.wikimedia.org/g/releng/local-charts
- TODO - Brennen: Make an issue for followup on documentation stuff / coordinate a chat
Video call chat transcription
[edit]You 9:00 AM o/ Greg Grossmeier 9:02 AM https://etherpad.wikimedia.org/p/docker-sig David Barratt 9:06 AM https://hub.docker.com/_/mediawiki https://github.com/wikimedia/mediawiki-docker Eric Gardner 9:07 AM David, as a new developer, that image was very useful for me in getting a proper environment set up locally – was able to adapt it very easily. Thanks! David Barratt 9:09 AM Eric, Great! :) Mark A. Hershberger 9:10 AM link to node renderer addshore mentioned? blubber? Lars Wirzenius 9:10 AM https://wikitech.wikimedia.org/wiki/Blubber Antoine Musso 9:11 AM sorry long commute :( Jeena Huneidi 9:11 AM https://gerrit.wikimedia.org/g/releng/local-charts Tyler Cipriani 9:13 AM Etherpad for people just joining: https://etherpad.wikimedia.org/p/docker-sig (not sure how hangouts handles scrollback) Mateus Santos 9:13 AM http://github.com/mateusbs17/gisdock Antoine Musso 9:15 AM GO GREG GO ! :D David Barratt 9:16 AM Jeena, will the local-charts work with https://microk8s.io/ ? or just minikube? Kosta Harlan 9:16 AM mediawiki-docker-dev and the local-charts could probably be using the same base image for php / apache? Jeena Huneidi 9:17 AM David - not sure, I'll check it out! Kosta - that's possible yeah Tyler Cipriani 9:18 AM random list of images in the wikimedia-docker registry, FWIW: https://people.wikimedia.org/~thcipriani/docker/ Alexandros Kosiaris 9:19 AM https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-least-30-vulnerabilities/ Greg Grossmeier 9:20 AM haha Adam Shorland 9:20 AM i guess we already kind of have our own bases images to extend form? Tyler Cipriani 9:20 AM was just looking for that link :) Adam Shorland 9:21 AM :D Tyler Cipriani 9:21 AM https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/master/images/ Adam Shorland 9:22 AM https://docker-registry.wikimedia.org/v2/_catalog isnt loading :( Tyler Cipriani 9:22 AM https://github.com/wikimedia/operations-docker-images-docker-pkg Alexandros Kosiaris 9:22 AM takes 30s it's very slow currently, working on upgrading to make it faster Adam Shorland 9:22 AM :( Antoine Musso 9:23 AM docker-pkg recently received lot of updates. Give it a try!:) Tyler Cipriani 9:23 AM what's the definition of "official" :) Dan Garry 9:23 AM Exactly! :-D Adam Shorland 9:24 AM dockerhub-official, wmf-official, wmfproduction-official not sure either Adam Shorland 9:27 AM I really want to make the mediawiki-docker-dev thing more plugable Kosta Harlan 9:28 AM Seems like everyone finds their own way to suffer through this problem Eric Gardner 9:28 AM haha, so true You 9:28 AM perhaps the equivalent of https://www.mediawiki.org/wiki/MediaWiki-Vagrant Eric Gardner 9:28 AM yes, exactly Adam Shorland 9:28 AM i guess the page right now is https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker Tyler Cipriani 9:28 AM https://www.mediawiki.org/wiki/How_to_become_a_MediaWiki_hacker#Docker_development_environment Adam Shorland 9:29 AM it lists vagrant and mediawiki-docker-dev or manual install [= David Barratt 9:29 AM someone created this page as well (but it needs some love) https://www.mediawiki.org/wiki/Docker Adam Shorland 9:29 AM i had no idea the mediawiki-docker-dev thing was there Kosta Harlan 9:29 AM Would be great to add more recipes there https://www.mediawiki.org/wiki/MediaWiki-Docker-Dev Mateus Santos 9:29 AM Still, the endeavors are split between several projects. Vagrant has its problems but everyone suffers together. Adam Shorland 9:30 AM dev env goals, use wmf built images in the environemt :) Jeena Huneidi 9:31 AM yeah Greg Grossmeier 9:31 AM https://www.mediawiki.org/wiki/Selenium/Node.js Tyler Cipriani 9:31 AM we ripped that off for blubber docs https://wikitech.wikimedia.org/wiki/Blubber Adam Shorland 9:33 AM I'm in another call now so cant talk but would love to try and include more services into my dev env, in a nice way where you can turn them on and off etc if you do or dont want them, if anyone has thoughts or services it might make sense for https://github.com/addshore/mediawiki-docker-dev/issues/71 is probably the place Antoine Musso 9:34 AM make sure to capture the above chat on the etherpad https://etherpad.wikimedia.org/p/docker-sig :) Eric Gardner 9:34 AM I think expanding on mediawiki-docker-dev to make it more pluggable is a good way forward even if it's just a matter of documenting how to get common setups working Tyler Cipriani 9:35 AM https://gerrit.wikimedia.org/g/releng/local-charts#readme-for-local_charts looks like there's a FIXME in MacOS if folks are interested :) Kosta Harlan 9:35 AM eek :) Adam Shorland 9:35 AM i look forward to trying local-charts over the coming week :) Kosta Harlan 9:35 AM Yeah I'll try it out Brennen Bearnes 9:35 AM cool. :)