Jump to content

Data³/metric-dictionary

From mediawiki.org
Metrics
Name Submitted By Description Status Purpose Category(s) Source(s) Used In Impacts/Contributes to Calculation UOM Granularity Aggregations Notes
Code Coverage jbranaa Measures the amount of code that is covered by unit tests. Current Version

Tasks:

To help drive the adoption of unit testing practices. Code Health PHPUnit, QUnit Code Health dash, Code Stewardship dash Code Health score Lines Code unit - class, function, etc File, Component, Extension, Service
Code Review Responsiveness jbranaa Measure how quickly code reviews are performed. Current Version

Tasks:

To help drive improved responsiveness to overall developer community. Code Stewardship Gerrit Code Stewardship dash Cycle Time Days commit or patchset Repo, Component/Extension/Service, Patch Submitter - Team, Staff, non-Staff
Bug Triage responsiveness jbranaa Measures the amount of time it takes to triage a new bug. Tasks: Drive improved responsiveness when new prospective bugs are reported. Will also potentially help manage backlog growth. Code Stewardship Phabricator Code Stewardship dash Days task Phab project, team, organizationally. Patch Submitter - Team, Staff, non-Staff
Bug resolution responsiveness jbranaa Measures the amount of time it takes to address/resolve a bug. Current Version

Tasks:

To help improve overall software maintenance and manage bug backlog. Code Stewardship Phabricator Code Stewardship dash Days task Phab project, team, organizationally. Patch Submitter - Team, Staff, non-Staff
Test Execution Duration - Test metrics (eg: quibble) Greg Measure how long our tests take and identifying problematic (long running) tests. Current Version

Tasks:

To reduce the time to feedback for our developers using CI. Code Health? Zuul, quibble CI health dashboard? Cylce Time seconds test run Repo, Component/Extension/Service
Page load time Gilles Real world measurement of how long it takes to fully load a page Current Version

Tasks:

To help improve reader experience, equity of access, search engine ranking and catch regressions Performance Coal/Grafana Site performance Median navigationTiming loadEventEnd - fetchStart for 1 in every 1000 requests ms Labeled as "loadEventEnd" on our performance dashboards
Edit save time Gilles Real world measurement of how long it takes to save an edit Current Version

Tasks:

To help improve key editor experience and catch regressions Performance Coal/Grafana Site performance ms
Cycle time (mediawiki/core + extensions) thcipriani Time it takes for a patch to make it from gerrit to prod Tasks: Reduce time to production by streamlining delivery/review/feedback/etc code review scap, gerrit developer satisfaction hours Repo, team
Bugs per train thcipriani Tasks: deployment phab Site stability count Component/extension/service
Test wait time thcipriani time spent waiting on hardware availibility Tasks: Tighten feedback loop testing zuul developer satisfaction ms test/extension
Patches per train thcipriani How many patches are we deploying Current Version

Tasks:

gerrit site stability count
Team/Technology bugs reported/fixed thcipriani count bugs fixed and created overtime Current Version

Tasks:

phab
Team/Tech patches created over time thcipriani Count of patches created/merged over time by team Current Version

Tasks:

gerrit
Lead time Naïké Time needed to move a ticket from the first stage to the last stage of a process. This metric calculates the number of work days that passed since the ticket was created. Tasks: 1) See how the team reduces time to finish

2) Identify what is stalling the process

Performance Phabricator Code stewardship dashboard Code stewardship Count of days Days Task Phabricator board Only reportable by looking at each task manually
Cycle time Naïké Cycle time is the total elapsed time you need to move a task from the moment it enters the “in progress” (working) stage to the moment it is considered as finished. Cycle Time = End Date – Start Date Tasks: 1) Optimize the process by identifying bottlenecks

2) Create value for our users more often

Performance Phabricator Code stewardship dashboard Code stewardship Cycle Time = End Date – Start Date Days Task Phabricator board Only reportable by looking at each task manually
Throughput Naïké The number of tickets completed per period (i.e moved to “Done” and not moved backward). Current Version

Tasks:

1) Forecast how much work the team is able to complete at a given period

2) Measure team performance over time

Performance Phabricator Code stewardship dashboard Code stewardship Count Days, weeks, months, and quarters Task Phabricator board Only reportable by looking at each task manually
Total work in progress (WIP) Naïké The number of tasks that can be in progress at any one time. Tasks: 1) Prevent tasks from getting stuck

2) Reduce loss of productivity from multitasking 3) Get more work done

Performance Phabricator Code stewardship dashboard Code stewardship Count Count Task Phabricator board Only reportable by looking at each task manually
Blockers Naïké Anything that stops or slows down progress on a ticket Tasks: Remove impediments as soon as possible Performance Phabricator code stewardship dashboard Code stewardship Count Count Task Phabricator board Can be reported by manually counting
Deployment frequency Naïké Measure of software deployment to production Tasks: Performance Gerrit? Code stewardship dashboard Code stewardship Count Hours, days, weeks, months, and quarters Task
Mean time to restore Naïké Measures the availability of systems. Time it takes to restore a system. Tasks: Know whether or not we are meeting our SLAs/SLOs Performance Code stewardship dashboard Code stewardship Mean hours, days Task calculated by dividing the total unplanned maintenance time spent on an asset by the total number of failures that asset experienced over a specific period.
Change fail percentage Naïké Change fail percentage is a measure of the percentage of changes that result in a failure. A failure is a change that

“result[s] in degraded service or subsequently require remediation (e.g., leads to service impairment or outage, require a hotfix, a rollback, a fix-forward, or a patch.)

Tasks: Measure delivery performance to reduce rework Performance Code stewardship dashboard Code stewardship Percentage Task
Time to first response Naïké First Response Time is the number of minutes, hours, or days between when a customer submits a support ticket and when a customer support representative provides an initial response. Tasks: Reduce the time stakeholders have to wait before we respond to an issue Performance Code stewardship dashboard Code stewardship Minutes, hours, days Hours Task
Average age of issues Naïké Average age of unresolved issues Tasks: Making sur the team meets its SLAs/SLOs by preventing too many issues from aging Performance Code stewardship dashboard Code stewardship average Days Task
Workload distribution Naïké Number of tickets assigned per individual contributor Tasks: Prevent too many tickets from being assigned to the same person Performance Code stewardship dashboard Code stewardship count Count Task
Defect Escapes jbranaa Number of unknown defects that escape to production. Sometimes we find defects prior to deployment, but decide to deploy without fixing them. This is NOT them. Tasks: To help measure testing effectiveness. Testing Phabricator Code stewardship dashboard Site Reliability, Test Effectiveness count Quantity Repo, Patch (?)
Test Case Failure Rate jbranaa Percentage of time that a testcase catches a problem. Tasks: To help identify tests are no longer relavant. Testing Gerrit + TBD Test Effectiveness # of failures/total # of executions. Percent Testcase This one requires the historical tracking of testcase exectuion results. Automated tests can be pulled from Gerrit, but we don't currently have a common way to track manual testcase execution (this may not be a big deal as much of our manual testing is exploritory/non-repetitive).
nDAR jbranaa The rate of defects being found normalized by testing effort. Tasks: To help assess release readiness by the rate of defects being found over time normalized by testing effort. The testing effort can be measured in hours and/or test cases being executed. Testing Phabricator + TBD Release readiness dashboard # of pre-deployment defects reported by WMF divided by effort measured in testing hours or test cases executed. Count Repo, Patch (?)
Pre-production defect backlog jbranaa The number of defects that where found pre-deployment that are still open Tasks: To help assess release readiness by identifying the size of the defect backlog that will be deployed to production. Code Health Phabricator Release readiness dashboard, Code Health Dashboard Code Health, Site Reliability # of defect found during development cycle that are open. Count Repo, Patch (?)
Pre-production UBNs reported per train Rummana The Number of UBNs found during pre-deployment testing/ code-review period Tasks: To measue test effectiveness Code Health, Testing Gerrit, Phabricator Site Reliability # of pre-deployment UBNs reported on Phabricator per deployment cycle Count Task
Pre-production UBNs fixed per train Rummana Pre-deployment count of UBNs fixed Tasks: To measure reposiveness and release readiness Code Health Gerrit,

Phabricator

Test Effectiveness # of UBNs marked as resolved on Phabricator before deployment per release cycle Count Task and/or Patch
Rate of re-surfaced bugs Rummana Number of defects that were fixed but re-surfaced Tasks: To identify unstable or less stable area that can be improved and/or identify test completeness in those area. Code Health, Testing Gerrit, Phabricator Site Reliability + Test effectiveness # of issues that got re-opened on Phabricator per deployment cycle per repo. Count Task and/or Patch