Team Practices Group/Improving burndown charts

Introduction

This page is a work in progress, reflecting current thinking. It is not authoritative in any way.

Raw kickoff meeting notes: https://etherpad.wikimedia.org/p/PhabGraphKickoff

Terminology

Burndown Chart
- https://en.wikipedia.org/wiki/Burn_down_chart
Burnup Chart
- http://brodzinski.com/2012/10/burn-up-better-burn-down.html
Phabricator
- The issue-tracking system used by the WMF
Phabricator Sprint Extension
- Adds "Sprint" type projects, which add a "Story Points" field to any task in that project
- Written by the WMF (?)
Phragile
- A tool that generates graphs of data pulled from phabricator's API
Release
- A "marketing" feature-driven release, OR a timeboxed release
- For feature-driven, the question we want to answer is "When?"
- For timeboxed, the question is "How much can we get done?"
Release Burnup Chart (or Product Burnup Chart)
- A burnup chart whose scope spans multiple sprints
Release Cumulative Flow Chart
- http://www.agilesherpa.org/agile_coach/metrics/cumulative_flow/
Sprint Burnup Chart
- A burnup chart whose scope is limited to a single sprint

Assumptions

Burnup charts are fine (in lieu of Burndown charts)
Product/Release charts are the focus of this initiative (as opposed to Sprint charts)
Forward-looking prediction is the focus of this initiative (as opposed to retrospective)
An external chart generation tool is acceptable (it doesn't have to be built into phab)
At least for now, it would be acceptable for phab to export raw data, to allow fancy charts to be generated by a spreadsheet
We have some (unknown $) budget to spend on upstream Phabricator coding, as needed
We have the ability to configure our phab instance however we wish
We have limited WMF human hours available for phab coding
Any phab coding we do should be in the form of patches pushed upstream
Is backward-looking important, given that projects wouldn't have used the necessary conventions???
Long-term reliance on the Sprint Extension is problematic
- We need Task.Estimate (aka "Story Points"), and Project.StartDate/EndDate
- (Not sure what else Sprint Extension provides that we need

Current State

Sprint extension
- Developed by Christopher at WMDE
- Requires ongoing maintenance to remain compatible with upstream changes
- Therefore requires testing with each new upstream release
- Christopher doesn't have time to maintain it
- Mukunda can maintain as needed, but not ideal
- We really need to get sprint-y features in upstream (Story points, start/end dates)
Phabricator's built-in burnup charts (part of Sprint Extension):
- Reports/Burnup, filter by a single project
- Does not track when tasks were added/removed to/from the project
- Odd pink color choice
- In FF on Linux, scrolling down leaves the graph grid behind
Phabricator's built-in burndown charts (phact?):
- Work OK within a single Sprint, but require a ton of work to span multiple sprints
- Trigger on a task being in a "Done" column, which not all teams use
  - But note that "resolved" is also problematic
- Generates Burndown charts, not burnup charts
- Does not show the scope line moving up or down as scope changes--it always just shows as a flat line, at the *current* scope level
- Handles weekends in ways that some people dislike, but that's really a Sprint chart issue more than a Release chart issue
Phact:
- Upstream reporting tool, with a "burnup chart" feature
- Disabled in our instance.
- Chase: Phact is in "prototype" mode, effectively disowned by upstream
- Generally phab doesn't have good reporting, so phact was a response to that
- But they didn't have clear requirements, so it has gone stale
- We could push for phact to take on a reporting role, but it doesn't seem to have much of use to build on
Phragile:
- Is being developed by wmde
- Provides ??? charts (I haven't seen examples yet)
- Is an external tool, which would be acceptable
- Would need to interface with phab authentication/authorization system?
  - To be able to handle security issues, yes, but for normal projects, is this true?
- Currently provides BOTH graph generation AND sprint creation
  - Should these be separated?
Robla's "wbstatus" script:
- Is written in python
- Scrapes phab html to generate a state model of issues on a workboard
- Could be used as a template to create a similar tool for burnup purposes
- Example output: https://mw-core-wbstatus.wmflabs.org/?r=2015-03-16_to_2015-03-20
- Source code: https://github.com/robla/phab-wbstatus

Project Roles

Kevin: Product Owner.
- Gather and prioritize requirements
  - https://www.mediawiki.org/wiki/Team_Practices_Group/Improving_burndown_charts
- Project management of externals (e.g. phab, phragile)
- Reporting to stakeholders
Mukunda:
- could code in phab
- maintain phab in general (features, integration, deployment)
- could maintain/fix the sprint extension as needed
  - Complexity of integrating Sprint extension with upstream changes (we have no Continuous Integration of Phab+Sprint extension)
Greg:
- OK'ing Mukunda's time
- Delegating interface w/Evan
- Neither an implementor nor a (knowledgeable) consumer
Arthur:
- Shepherding, supporting
- Conduit/buffer as necessary for exec requirements
Chase:
- Ops eng
- mostly 'historically significant'
- dealt with deployments previously (handing off to Mukunda)
- much experience dealing with upstream
- is pregnant ;)

Issues to Remember

Backlog stories may not have estimates
- Assign arbitrary value (e.g. average story size)?
"Release" might mean fixed-date/train, or feature-driven
Not all teams will have "sprints"--some are Kanban
Need to avoid double-counting when task and subtasks were both estimated
- Could be an issue both for "work completed" and for "target scope" calculations

Data Model

It is "relatively easy" to expose other db elements via new custom Conduit API calls
Helpful phab queries, if they are/were possible:
- List all tasks that were in Column C of Project P on Date D (NO?)
- List all projects matching wildcard name search (YES?) [Maybe not necessary]
- List all tasks which were EVER in Project P (NO?)
- List all transactions for Task T, with timestamps (YES except not sure about timestamps)
- List all transactions for Project P, with timestamps (NO?)
If we did a nightly snapshot of all tasks in Project P,Q,R..., then where would we store that data
Can a Release project also be a Sprint project? (YES)
- (would be nice to have estimated future epics only be in the Release project, and not to force them to also exist in a Sprint project that isn't a real Sprint)

Desired Datasets/Charts

Points "done" per sprint over time (velocity)
- Note: Should scale by sprint length, to accommodate Kanban teams
- Note: Is this based on a "done" column, or marked "resolved"?
Current points of all tasks in all columns of all sprints PLUS non-sprint backlog
Historical points of all tasks in all columns of all sprints PLUS non-sprint backlog
Average task estimate across multiple sprints (or maybe just within each sprint?)
- To provide a placeholder size for unestimated tasks
Average Lead Time per story (from entering a sprint backlog until being "done")
- Especially helpful for Kanban projects

Possible Solution(s)

Completed points are calculated from a set of wildcard-filtered Sprint projects
- CONVENTION: Tasks are considered completed if they are in a workboard column named "Done"
  - Graph-requesting user could specify the column name when they give the wildcard spec
- Brute-force algorithm:
  - For each matching Sprint project,
  - Sum story points of all tasks in the "Done" column
  - Note the timestamp at which that Sprint ended (assuming it is in the past)
Scope of release is calculated from a Release project
- FEATURE REQUEST: New Phab API call: List all transactions that refer to a specific project
- CONVENTION: The Release project must contain *every* task that is part of that release
- Any task with subtasks must exclude any subtask estimates from its own current estimate
- Brute-force algorithm:
  - Given a Release project, find all transactions affecting it
  - Create event history of tasks being added/removed on that project
  - Replay the history to know which tasks were in that project at any timestamp
  - Record the story point sum at desired timestamps
Phragile, with the following changes:
- Ability to view graphs for sprints not created by phragile (e.g. existing sprints)
- Make sure it will work with a months-long sprint
- Burnup charts instead of burndown
  - NOTE: Should use historical estimates, as well as historical statuses
  - Must include tasks which used to be in the sprint, but no longer are
- Calculate scope line based on points in the sprint at that time
  - NOTE: Must use the historical estimates, as well as historical statuses
  - Must include tasks which used to be in the sprint, but no longer are
  - Need a rule to handle unestimated tasks (TBD)
- Ability to export underlying data that built the chart
  - Allows power users to generate alternative graphs
- Make the "ideal points" line optional
- Would prefer subset of the app which only does anonymous queries
  - For increased security
- Would prefer subset of the app that doesn't require its own database
  - For simpler deployment and less ongoing maintenance
  - Only feature to drop would be snapshots?