Here are the high level priorities of the DPE SRE team. The detailed backlog can be found on our main Phabricator board. Our current work can be followed on our "milestone" Phabricator board (there is no stable link to the current milestone, but it can be found as a link in the menu of our main board).

Current main projects

Migrate Airflow to DSE k8s cluster

To simplify operations and increase availability, we are migrating Airflow to k8s.

Links

Main phab task: T362788
Design doc

Migration from Graphite to Prometheus

To support the deprecation and removal of Graphite

Links

Main phab task: T359033

WDQS Graph Split

To support work by the Search Platform team. In particular, DPE SRE is focused on migration of the internal WDQS clients and the operational support of the underlying servers / platform.

Links

Main phab task: T374453
Migration of internal traffic: T379023

Replace Archiva with Gitlab artifact repositories

Archiva is our current solution for artifact hosting for Java / Scala projects and mirroring of external Maven repositories. It is unsupported and as a critical piece of our development and deployment infrastructure needs to be replaced. Gitlab is a component that provides the functionality that we need and is already deployed in our infrastructure, it is the obvious solution.

This project is driven by DPE SRE, but most of the implementation work is done by Search Platform, Data Engineering and Data Products. It is prioritized on top of the usual work for those teams and thus is slow moving.

Links

Main phab task: T367315
Decision brief

Usual operational work

Incidents
Various minor software upgrades
Access requests
SPARQL Federation requests

High level backlog of projects

Migration of the Search cluster from Elasticsearch to OpenSearch: T370147
Kafka upgrade: design doc
Mutualized OpenSearch cluster: T362105 & design doc
Hadoop upgrade: T379385
Kubernetes upgrade
Spark upgrade
Migration of additional services to k8s
- Presto
- JupyterHub