📖 Developer Satisfaction 2024 Report

🔁 Continuous Integration

tl;dr

Queue time satisfaction – Only half of respondents are satisfied with CI queue time, similar to last year.
Overall satisfaction – 64% are satisfied with CI overall, slightly higher than the 57% last year who said were satisfied with CI features aside from queue times.

# CI use in the past year

We asked survey takers, “In the past year, have you used Wikimedia’s continuous integration?”

The majority (52.4%) of respondents said they used Wikimedia's CI in the past year.
42.2% said they did not use it in the past year
5.4% were unsure

# CI queue time satisfaction

We asked survey takers, “How satisfied are you with the Continuous Integration queue time (how long it takes to report test results in Gerrit)?”

Of the survey takers who indicated that they did use Wikimedia's CI in the past year,

Half (50%) said they were satisfied with CI queue time
14% said they were neither satisfied nor dissatisfied
30% said they were dissatisfied
6% were unsure

top

# CI overall satisfaction

We asked survey takers, “How satisfied are you overall with continuous integration?”

Of the survey takers who indicated that they did use Wikimedia's CI in the past year,

a majority (64%) said they were satisfied with CI overall
14% said they were neither satisfied nor dissatisfied
16% said they were dissatisfied
6% were unsure

top

# CI onboarding satisfaction

We asked survey takers, “How satisfied are you with onboarding new people into continuous integration?”

Of the survey takers who indicated that they did use Wikimedia's CI in the past year, the most common response was uncertainty: 39% were unsure if they were satisfied with CI onboarding.

only (19%) said they were satisfied with CI onboarding
24% said they were neither satisfied nor dissatisfied
28% said they were dissatisfied

top

# Feedback

We asked survey takers, “Please share any other feedback you may have about Wikimedia’s continuous integration” Some themes emerged from the answers.

Finding test errors

5 out of 53 responses talked about difficulty finding the cause of test failures in the test output.

it can be hard to find the real error that caused the CI run to fail in hundreds of lines of logs

It would be nice if the actual failure was noted on Gerrit

Confusion

8 out of 53 open answers found some aspect of our continuous integration confusing and/or under-documented.

We shouldn't have to tell people to type magic words like 'recheck' or 'check experimental'!

It's very unclear which tests run on which repo, and why.

[new developers] have difficulty understanding why they received errors, or what to do about it.