Data Platform Engineering/Data Platform SRE/Decisions and Design Docs

The line between decision briefs and design docs is blurry. Design docs obviously contain a number of decisions as well. In general, decision briefs are focused on a single decision, which requires buy in from multiple people and teams.

We make 100's of decisions per day, not all of them need to be documented. Not all of them need to be documented, not all of them are decision briefs. Decisions happen in phab tickets, while writing code, and while discussing with people. If a decision is controversial, impact multiple people or teams, might be revisited or challenged in the future, then it makes sense to link it from here.

Main design folder for DPE SRE

Artifact repositories

• Archiva to Gitlab migration

Kafka

• Kafka clusters upgrade plan

Superset

• Superset on Kubernetes

Stats machines / Jupyter

• Jupyter and Analytics Client Enhancements

Airflow

• Airflow: High-Availability Strategy
• Airflow on Kubernetes - Test and migration plan
• Design a suitable DAG deployment method (T368033)

Query Service (W[CD]QS)

• WDQS Categories Migration (in progress)