User:JaimeLyn/Weekly Reports

These are weekly reports on the User:JaimeLyn/Historical OpenStreetMap FOSS OPW project for User:JaimeLyn
My OPW-related blog posts can be viewed on Blogger.

May 2014

Community Bonding Period

1. Most of this has been done through the mailing list.

Lessons learned since 21 April

1. Started from scratch in using IRC. Picked up the basics here and here
2. There is an overwhelming amount of information about both the OHM project and MediaWiki in general on the web, which makes Googling for links problematic at best (and hair-tearing at second best). I've learned to use this page as a key to some of the more frequently needed resources.

Communication Plan

1. Our team uses the Historical Mailing list, Google Hangouts and IRC #ohm
2. I will complete weekly progress reports here with frequent check-ins to my mentor via Hangouts and/or IRC.

Deliverables

The MVP: Integrate the time slider created by ConorMc (need user page to properly cite) into the Rails_Port webpage codebase.

The second half of the internship will be spent integrating the slider into the map renderer.

Week 1: May 19 to May 25

1. Researched history of OHM project in archives and GitHub issues here and here.
2. First OHM planning meeting on Google Hangouts, synopsis by Robert H. Warren posted on User/JaimeLyn/Weekly_Reports/May 22 OHM Hangout Synopsis
3. Added to Github OHM organziation and (scarily) given push access.
4. Assigned and began work on microtask of investigating methods for client-side rendering

Week 2: May 26 to June 1

1. Went spelunking through the OHM Rails Port to find where to place the slider (and was later introduced to (and made a small link fix within) the official OHM UI wiki page
2. Attended #OPW IRC chat for OPW Interns on May 27th, 3PM UTC.
   1. We had a round of intros and some general questions
   2. Sumana Harihareswara is now the official "so where do we go after this is over" post-internship planning mentor
   3. Sumana also posted a link to a very helpful article on when to ask for help First You Must Try, Then You Must Ask
3. During this week's Google Hangout meeting with mentor, reviewed task from last week and discussed next steps
   1. find a place for the TimeSlider on the OHM site
   2. start finding a way to make input from the TimeSlider feed into the website and request the tiles needed (this one's a higher level task and will break out into other pieces)
4. I posted mockups of two different ideas for TimeSlider placement:
   
   1. within the header of the site
   2. attached to the search box on the site

June 2014

Week 3: June 2 to June 9

1. Snagged TimeSlider code from themusictree.net website
2. Continuing work on getting the slider integrated into the website and I'm learning:
   1. Building consensus can be hard - there are a lot of differing opinions over what the TimeSlider should look like and how it should function.
   2. While I know Rails, finding one's way around a new code base can be tricky, to say the least. I'm using the Rails UI Wiki page and this little guide to Erubis to try to lead the way.
3. I've been very surprised at how much of the logic of the site is in JavaScript. The other Rails projects I've dug into have been mostly Rails with a little JavaScript for buttons and hover effects. This project is a JavaScript project wrapped in a Rails app. I'm still getting a handle of its complexity.
4. I'm still working on getting the slider working but I've got the text fields showing, at least.
   1. They needed a lot of tweaking to fix and flex to the space of the site. My branch is posted on GitHub.
   2. I have hit a major roadblock with the bottom half of the slider. The divs generated in the JavaScript portions are not rendering on the site. I've used console.logs throughout, testing to be sure that the script is running but I have not yet found the problem. I'm reaching out to my mentor for some assistance for what else to try to find the answer. I've exhausted my ideas. (None thus far have worked.)

Week 4: June 9 - June 15

1. After consultation with my mentor, we were both stumped so we reached out to the author of the slider for tips and toward the end of the week, we finally synced and got the slider working! The slider required a change from rendering a single object to iterating through a group of divs to create the sliders.
2. While I was waiting to work with the slider-creator, I started investigating methods for having a form in Rails have two separate actions, depending on the button clicked. It's apparently a little non-REST-full, but there are some REST-like options:
   1. using the params and embedding if statements into the controller or
   2. capturing the name of the button that is clicked.
3. I also cast about for other options that might be more understandable for me (this was a plan Z option, in case I never was able to connect with the original author.) I found a few nice options including nouislider
4. Received a last-minute invite to the OHM/OSM Hangout at the @SotMEU Conference on June 13th.
   1. The audio was very difficult to understand, but Susanna Ånäs pointed me to SK53's Maps Matter Blog. I spent a lot of time reading through to get more familiar with the history of the project. A shared notes page is posted here.

Week 5: June 16 - June 22

1. I spent far too long this week chasing Heroku errors in an attempt to get a live version of my time_slider2 branch online. After an IRC chat with my mentor, I cut my losses and moved on to the next task:
2. Capturing the start_year value entered from the TimeSlider
3. I found a new resource: a StackOverflow-style QA page for OSM.
   1. Regrettably, this resource seems focused on map-editing users. It did help me discover when I was headed (meandering?) down the wrong track with the router file in Rails and needed to pivot and work on the router.js file.
4. This was a real Edison week. I found a dozen different ways that don't work. Still at it.

Week 6: June 23 - June 29

1. Completed and submitted my mid-session evaluation.
   1. I'm not where I thought I would be at this point in the session. There is a tremendous difference between the code used in tutorials and student-level projects and the real, live code that lives out in the wild.
   2. It's more than just the complexity (although the huge leap in complexity would be more than enough to stymie me!) There are unfamiliar conventions and blending of platforms that still surprise me. (I'm still getting caught flat-footed and discovering that the code I'm writing in Rails needs to be ripped out and put into the JavaScript portions of the code base.) I will learn.
2. I'm having a meeting with my mentor on Tuesday, and hopefully that will give me some grounding.
3. I'm making an effort to commit more often and to push up my commits to my branch more regularly. I've been fearful about doing so but last week, I caught myself trying the same failed function twice. Having all of my commits with failures noted well help stop me from doing that in the future.
4. I made some baby steps toward success:
   1. screenshot
   2. GitHub issue tracker

July 2014

Week 7: June 29 - July 5

1. On Monday, the OHM team had our monthly Google Hangout (which I have learned is every first Tuesday of the month). The very sketchy notes are posted online.
2. At the Hangout, I was introduced to a new resource: the design prototypes for Wikimaps search. I'm pouring over them today. In fact, I spent an inordinate amount of time reading new material and going through archives to catch up. At every Hangout or meeting, I come across more terms and talking points that leave me confused. I'm filling in the gaps. My reading list has included:
   1. reports on Zurich Hackathon
   2. the Wikimaps Search Design page above
   3. more on vector rendering of tiles in preparation for my next steps
   4. RfC for How to Deal with Open Datasets
   5. Overview of Wikimaps
3. I am getting most anxious about continuing to work on my own branch of the OHM for so long. I would like to push to Master soon, but I need a better-working demo before I do so.
   1. I've been spending a lot of time there attempting to craft an appropriate request to get something other than a doctored error message to appear on the site when searching by year.
   2. The most excellent Frances Hocutt introduced me to the Wikidata API Sandbox It is proving quite helpful. APIs are a whole new world to me.
4. To appease my anxiety, I submitted a pull request with my most up-to-date code: https://github.com/OpenHistoricalMap/ohm-website/pull/18

Week 8: July 6 - July 13

1. And the code I've written was deemed not yet ready for prime time - it only works halfway at this point. I will resubmit after I get the code up to production standard.
2. After some useful feedback on the importance of submitting a pull request with only one commit (and some pointers to how-to's on rebasing to make that happen) and the helpfulness of of including *lots* of screenshots showing the software in action, I closed the pull request.
3. One (!!) of the aspects of this project that I have found especially challenging is finding my way through the blending of JavaScript and Rails code. I spent an inordinate amount of time this week chasing down what turned out to be a Javascript syntax error that was manifesting as a Rails error.
   1. I've installed the "better_errors" gem to try to assist with errors like this - it will presumably be triggered if it is a true Rails error
   2. I've also learned that if I make changes to the JavaScript portions of the code, I have to restart the server in order to see any changes. Most changes to the Ruby code automagically propagates without restarting the server.
4. Screenshots of my progress so far are on GitHub. Searching by year (2008 and 2009) will bring up the Burning Man map layers for those years and recenter the map to that area. Searching by any other year brings up with message that this is a work in progress. (It no longer crashes the map and brings up the dreaded Rails error screen, so there's some progress there!)

Week 9: July 14 - July 20

1. After some discussion and direction from my mentor, I'm tackling the tile rendering server bit. I've added a test layer for "Year" in the leaflet.osm.js file.
2. I've got a first little skeleton in the code but it doesn't lead anywhere yet. I updated the issue tracker in GitHub with my latest and (not so) greatest.
   1. The screenshot is also posted here
   2. As can be seen by the mostly empty layer on the right, I've got most of the plumbing in there but there's no data coming through the pipes.
3. Digging further into Leaflet and how to use it and I also put up a white flag for assistance in the issue thread above

Week 10: July 21 - July 26

1. At the beginning of the week, I felt a little discouraged after seeing this post in the mailing list about how nomoregrapes built a working slider over an isolated OHM map during the Hackathon at SoTM-EU. I went through the code he posted on GitHub and linked to it in the GitHub OHM issues tracker.
2. After a lot of hacking and many false starts (and some help from my mentor), by the end of the week, I found the proper places in the code to add references to the {t} value for searching. My commits and code

August 2014

Week 11: July 28 - August 3

1. We're jumping into the rendering now. Most of the work will be in ohm_mod_tile, especially renderd and the master daemon program.
2. I posted the IRC log here (tried to strip out the filler /leave, /join, etc)
3. RENDERING
   1. This file is proving vital but elusive: home/tim/ohm-carto/ohm-carto/mapnik 2008.xml (UPDATE: I have the file now, but...)
4. The bulk of the renderer is written in C and C++ (with a lot of Bash and some inline SQL for good measure.)
   1. I am v e r y s l o w l y and carefully modifying the renderer to accept the {t} parameter. It's been many years since I've even looked at C code (and that was in an introductory C class.)
   2. I'm relying heavily upon the 'git grep' command (tracking here).
   3. I'm logging my progress so I can keep track of what I've changed and where: git diff log.
5. I don't want to lose this reference (is this Momento?):
   1. on line 245 in mod_tile.c, ~~bzero~~ recv() (bzero is used to zero out the bits at &resp to prepare it for the map tiles) and sends &resp to get the tiles from the database over a websocket
6. I am really deep in the weeds here in C. I am going to take a little time to read up more in the man pages and go through some more of the tutorial over on Learn Code the Hard Way

Week 12: Aug 4 - Aug 10

1. Made some changes to my changes (meta!) to the renderd.py byte packing/unpacking directive.
   1. IRC log on discussion here
   2. updated diff log here
   3. link on languages and urls from Chippy
   4. link on parameterization of mapnik tiles from Chippy
2. The OHM Team had our monthly Google Hangout notes
3. Re-attempting to install dependencies to run test server, instructions here for Mapnik and here for mod_tile
4. After much searching (and an upgrade to gcc, I finally got it all running well enough to attempt to compile.
   1. Sadly, MediaWiki's spam filter is preventing me from posting a link to the very helpful blog that walked me through the task of upgrading my C/C++/Java compiler. If you visit charette.no-ip.com:81/programming/2011-12-24_GCCv47, you'll get to the instructions.
5. I am now working on fixing the massive list of warnings and errors that running make generated for me!

Week 13: Aug 11 - Aug 17

1. The number one thing that I have learned so far through this internship is that all of those adages and warnings about people being terrible at estimating how long a software task will take to complete are vast understatements!
   1. I initially thought that I would be spending this week polishing up the documentation for my project, I am still chin deep, trying to make the renderer accept a year as a query.
   2. I have a pow-wow planned with my mentor tomorrow (Tuesday), so we will come up with a plan of attack then. In the meantime, I'm still slogging through the errors I've generated on my now second attempt at compiling.
2. Still on the compile, fix those errors, compile again for more errors cycle.
   1. I appear to have a bigger problem, though: a Makefile.in is missing. Searching for the right way to solve this. (--Aug 14)
   2. And found one! Running autoreconf solved the problem!
3. I am now approaching the gen_tile.cpp file. A lot of magic happens here, and it all needs to reference t for time. Diving in.
4. Current changes: compare of master branch and my branch
5. Hitting a roadblock with the protocol versions. We need to be able to switch between protocols (with time or without) but I have just finished going through the entire mod_tile hard coding the {t} variable throughout. Refactor?
6. I am also working on something to present to the IRC group on Tuesday.

Week 14: Aug 18 - Aug 23

1. 🎶 A three month tour, a three month tour... 🎶
2. My notes/presentation for OPW wrap up meeting
3. This week has been spent playing Wack-A-Mole with compiler errors. Various iterations of the error logs can be found at: my GitHub notes repo

Week 14 + 1: Aug 24 - Aug 30

1. I am still working on getting the server running properly on my machine. At this moment, I'm stitching together the ohm-website files with the ohm_mod_tile and mapnik.
2. A few websites and wikis have been giving me some breadcrumbs
   1. The FLOSS Manual for OSM is helpful, but also directs users to other sources
   2. Build Your Own OpenStreetMap Server tutorial is a little old, but still seems helpful
   3. postgresl/postgis tutorial is helpful, but not perfect. Check your machine to see what it calls postgres! (It will save you a lot of headaches!) Also, see postgis from osgeo
3. I'm chin deep here, but fighting my way through

Week 14 + 2: Aug 30 - Ongoing

1. My internship has officially ended before I could complete the project.
2. The overall OHM project page: Open Historical Map
   1. Open_Historical_Map The goal of the project was, essentially, to put a time slider on a fork of the Open Street Map to enable users to search for historical maps that have been loaded into the system.
   2. My original OPW proposal page
3. The codebase is quite complex.
   1. It is actually three separate codebases, written in several languages: Javascript, Ruby, C, C++ and Python (with a healthy serving of Bash scripts for interpreting map paths!)
   2. I was able to integrate a minimum viable slider into the website, have it feed the year portion of the query through and up to the server.
   3. I've also modified the renderer to accept time as a parameter.
   4. A big stumbling block (besides not knowing C/C++ and spending some internship time studying the language !!) has been the difficulty in keeping the renderer backwards-compatible with non-time queries, but I'm still working on it!
4. There's still so much more to do. Three months was simply not enough time for me to complete the entire project. I plan to keep contributing.