[Epic] Splitting the graph in WDQS
Open, HighPublic
Actions

Assigned To

None

Authored By

	Lydia_Pintscher
	May 19 2023, 9:35 AM

Description

In order to stabilize the Wikidata Query Service we are looking into splitting the graph inside Blazegraph into 2 (or potentially more) subgraphs. This ticket is for tracking the investigation into what a sensible split would be, what the consequences are and then making it happen.

Related Objects
Search...

Status	Assigned	Task
Open	None	T335067 Epic: Wikidata Query Service stabilization
Open	None	T337013 [Epic] Splitting the graph in WDQS
Resolved	Gehel	T337018 Refine the definitions of the potential splits
Resolved	• Manuel	T337020 [Analytics] Understand the size and connectedness of Wikimedia-internal concepts in the Wikidata graph
Resolved	AndrewTavis_WMDE	T337021 [Analytics] Find out size of term subgraph
Declined	None	T337407 1st experiment in splitting the WDQS graph
Declined	None	T337406 Setting up a test server exposing an experimental split of the WDQS graph
Resolved	dr0ptp4kt	T347989 Adapt rdf-spark-tools to split the wikidata graph based on a set of rules
Resolved	dcausse	T349519 Determine if IGUANA and TFT would fit our query analysis needs
Resolved	Lydia_Pintscher	T349829 List and document all known public SPARQL endpoints serving wikidata
Resolved	Gehel	T350464 Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph
Resolved	dcausse	T349069 Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers
Resolved	bking	T347504 WDQS graph split: load data from dumps into new hosts
Resolved	bking	T349011 Improve data-reload cookbook based on graph split needs
Resolved	bking	T347505 Prepare new WDQS hosts for graph splitting
Resolved	RKemper	T351650 Expose 3 new dedicated WDQS endpoints
Resolved	RKemper	T350465 Load Wikidata split graphs into test servers
Resolved	dr0ptp4kt	T350106 Implement a spark job that converts a RDF triples table into a RDF file format
Resolved	RKemper	T354658 Create 3 microsites for wdqs full graph, main graph, & scholarly articles
Resolved	Dzahn	T355273 ProbeDown - *_experimental_wikidata_org
Resolved	RKemper	T354661 Generate TLS certs for new WDQS endpoints
Resolved	RKemper	T354662 Create DNS records for 3 new WDQS endpoints
Open	None	T354043 Decide the name, domain and logo of WDQS for scholarly articles
Resolved	RKemper	T355593 Re-generate webserver-misc-apps.discovery.wmnet cergen certificate
Resolved	Gehel	T355888 Enable cross federation between experimental WDQS endpoints
Resolved	bking	T352878 Troubleshoot recurring systemd unit failures and availability issues for wdqs1022-24
Resolved	Gehel	T352538 [EPIC] Evaluate the impact of the graph split
Resolved	AndrewTavis_WMDE	T349512 [Analytics] Collect multiple sets of SPARQL queries
Duplicate	AndrewTavis_WMDE	T350157 [Analytics] Extract a representative sample of SPARQL queries from the query logs
Open	None	T337015 review sample of queries to get a better sense of impact of various graph split options
Resolved	Gehel	T349911 Explore the feasibility of using SPARQL federation for scholia queries
Resolved	dcausse	T351819 Create a tool that records and compares a set of sparql query results
Resolved	dr0ptp4kt	T355037 Compare the performance of sparql queries between the full graph and the subgraphs
Resolved	dcausse	T355040 Compare the results of sparql queries between the fullgraph and the subgraphs
Invalid	Sannita	T356773 [tracking] Community feedback for the WDQS Split the Graph project
Resolved	Lucas_Werkmeister_WMDE	T355298 Investigate the impact of the WDQS graph split on constraints checks
Resolved	dcausse	T357966 Document limitations of blazegraph federation
Resolved	dcausse	T357980 Compile a set of queries rewritten with federation across the two graph splits
Resolved	pfischer	T361935 Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs
Resolved	EBernhardson	T361950 Ensure that WDQS query throttling does not interfere with federation
Resolved	dcausse	T362060 Generalize ScholarlyArticleSplitter
Open	None	T364363 [Epic] Productionize federated wdqs graph-split endpoints
Resolved	RKemper	T364364 Provision DNS and certificates for wdqs graph split domains
Resolved	Stevemunene	T364366 Implement puppet logic for wdqs graph split
Resolved	Stevemunene	T364367 Create dedicated UIs for wdqs graph split endpoints
Resolved	Stevemunene	T364368 Create separate pybal pools for wdqs graph split (main vs scholarly)
Resolved	Stevemunene	T372919 Bring wqds2024 back into service
Resolved	RKemper	T373145 Create new service catalog entries for wdqs-main and wdqs-scholarly
Resolved	RKemper	T364077 Adapt the wdqs data-transfer cookbook to operate with federated subgraphs
Invalid	Gehel	T368972 Use Envoy instead of LVS to route internal federation traffic for WDQS
Resolved	RKemper	T370754 Import WDQS subgraphs to production nodes
Resolved	ItamarWMDE	T371346 Add a banner to query.wikidata.org to inform users about the scholarly articles split
Resolved	Stevemunene	T371833 Tear down wdqs graph split experimental endpoints
Duplicate	None	T372816 Add federated queries in the test queries used to validate deployment
Resolved	RKemper	T372818 Verify and correct WDQS Grafana dashboard with graph split
Resolved	dcausse	T373391 Create wdqs-main and wdqs-scholarly specific test queries
Resolved	bking	T373791 Transfer a sane journal (subgraph:main) to wdqs2021 from wdqs2022
Resolved	dcausse	T373812 Internal federation sometimes fail with HttpConnectionOverHTTP
Open	RKemper	T367409 Measure the time to recovery of WDQS after the graph split
Resolved	Gehel	T367510 Request permission to create 4 kafka topics in kafka-main (WDQS graph split)
Resolved	AudreyPenven_WMDE	T369079 Update `UniqueValueChecker` to query a list of endpoints
Resolved	pfischer	T369729 RDF Updater Consumer: Allow "empty" patches
Open	None	T372094 WDQS: Document review/refresh for https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service
Open	None	T374453 [EPIC] Migration of traffic to new Split Endpoints for WDQS
Open	AndrewTavis_WMDE	T377211 Estimate the number of queries needing to migrate away from the full WDQS graph
Open	None	T379023 Create WDQS split endpoints for internal traffic and reconfigure clients to use the new endpoints
Open	None	T377956 Make WikimediaCampaignEvents use split-graph query service
Open	None	T376151 Cutover wdqs-internal to new split endpoints
Open	None	T376150 Prepare 5 codfw hosts to serve wdqs-internal from main graph
Open	RKemper	T379329 Create puppet config for wdqs-internal-main and wdqs-internal-scholarly roles
Open	None	T379334 Create DNS records for wdqs-internal-main and wdqs-internal-scholarly
Open	RKemper	T380556 Provision VIPs for wdqs-internal-[main,scholarly]
Open	None	T379333 Create envoy config for wdqs-internal-main and wdqs-internal-scholarly
Open	RKemper	T379330 Create pybal pools for wdqs-internal-main and wdqs-internal-scholarly
Open	None	T374021 Make WikibaseQualityConstraints use split-graph query service
Open	RKemper	T380555 Enable LVS for wdqs-internal-[main,scholarly]
Open	None	T380594 Inform current wdqs-internal consumers about new internal graph split endpoints
Open	None	T374967 wdqs-categories migration: decide where to migrate
Resolved	bking	T376079 eqiad: request 1 VM for wdqs-categories

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Lydia_Pintscher mentioned this in T335067: Epic: Wikidata Query Service stabilization.May 19 2023, 10:04 AM

Gehel triaged this task as High priority.May 22 2023, 12:58 PM

Gehel moved this task from Incoming to Scaling on the Wikidata-Query-Service board.

Gehel moved this task from Scaling to Epics on the Wikidata-Query-Service board.

karapayneWMDE subscribed.May 24 2023, 2:43 PM

Gehel removed a subtask: T337015: review sample of queries to get a better sense of impact of various graph split options.May 24 2023, 3:25 PM

Gehel removed a subtask: T337406: Setting up a test server exposing an experimental split of the WDQS graph.

elal subscribed.May 24 2023, 3:28 PM

• Manuel added a project: Wikidata Analytics.May 30 2023, 6:38 PM

• Manuel moved this task from Incoming to Waiting Tasks on the Wikidata Analytics board.

• Manuel mentioned this in T337799: [EPIC] Analytics support around splitting the WDQS graph [up to milestone 3].May 30 2023, 8:54 PM

• Manuel removed a project: Wikidata Analytics.Jul 7 2023, 9:12 PM

tfmorris subscribed.Jul 7 2023, 9:29 PM

Gehel closed subtask T337407: 1st experiment in splitting the WDQS graph as Declined.Jul 17 2023, 3:37 PM

• Manuel closed subtask T337020: [Analytics] Understand the size and connectedness of Wikimedia-internal concepts in the Wikidata graph as Resolved.Jul 18 2023, 1:16 PM

• Manuel closed subtask T337021: [Analytics] Find out size of term subgraph as Resolved.Jul 20 2023, 9:32 AM

bking subscribed.Sep 15 2023, 1:37 PM

~~When T345475 is done, we should have 3 new WDQS hosts in CODFW that could be used for the graph splitting experiment. @RKemper let us know if you have any objections to this plan.~~

We'll use eqiad hosts instead, see T347505

bking added a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet.Sep 15 2023, 2:02 PM

dr0ptp4kt subscribed.Sep 22 2023, 4:07 PM

bking mentioned this in T347504: WDQS graph split: load data from dumps into new hosts.Sep 27 2023, 3:27 PM

bking added a subtask: T347504: WDQS graph split: load data from dumps into new hosts.

bking removed a subtask: T345475: Service implementation for wdqs202[3-5].codfw.wmnet.Sep 27 2023, 3:36 PM

bking added a subtask: T347505: Prepare new WDQS hosts for graph splitting.Sep 27 2023, 3:39 PM

Gehel added a project: Discovery-Search (Current work).Oct 17 2023, 9:18 AM

Gehel moved this task from Incoming to Epics on the Discovery-Search (Current work) board.

bking closed subtask T347505: Prepare new WDQS hosts for graph splitting as Resolved.Oct 20 2023, 1:58 PM

dcausse added a subtask: T349512: [Analytics] Collect multiple sets of SPARQL queries.Oct 23 2023, 1:41 PM

dcausse added a subtask: T349519: Determine if IGUANA and TFT would fit our query analysis needs.Oct 23 2023, 2:51 PM

dcausse added a subtask: T349829: List and document all known public SPARQL endpoints serving wikidata.Oct 26 2023, 2:59 PM

dcausse added a subtask: T349911: Explore the feasibility of using SPARQL federation for scholia queries.Oct 27 2023, 2:19 PM

Gehel closed subtask T337018: Refine the definitions of the potential splits as Resolved.Nov 3 2023, 9:55 AM

Gehel removed a subtask: T349069: Design and implement a WDQS data-reload mechanism that sources its data from HDFS instead of the snapshot servers.Nov 3 2023, 10:21 AM

Gehel removed a subtask: T347504: WDQS graph split: load data from dumps into new hosts.

Gehel removed a subtask: T347505: Prepare new WDQS hosts for graph splitting.

dr0ptp4kt closed subtask T347989: Adapt rdf-spark-tools to split the wikidata graph based on a set of rules as Resolved.Nov 20 2023, 5:45 PM

Gehel removed a subtask: T351650: Expose 3 new dedicated WDQS endpoints.Nov 20 2023, 7:41 PM

dcausse added a subtask: T351819: Create a tool that records and compares a set of sparql query results.Nov 22 2023, 2:01 PM

Gehel closed subtask T349519: Determine if IGUANA and TFT would fit our query analysis needs as Resolved.Dec 1 2023, 9:20 AM

Gehel removed a subtask: T349512: [Analytics] Collect multiple sets of SPARQL queries.Dec 1 2023, 2:49 PM

Gehel removed a subtask: T349911: Explore the feasibility of using SPARQL federation for scholia queries.

Gehel removed a subtask: T351819: Create a tool that records and compares a set of sparql query results.

SEgt-WMF subscribed.Jan 8 2024, 6:58 PM

Gehel closed subtask T350464: Expose SPARQL endpoints with full wikidata data set and with split graph to enable experimentation on federation with a split graph as Resolved.Jan 26 2024, 8:41 AM

dcausse added a subtask: T357966: Document limitations of blazegraph federation.Feb 20 2024, 11:03 AM

dcausse added a subtask: T357980: Compile a set of queries rewritten with federation across the two graph splits.Feb 20 2024, 2:00 PM

Lydia_Pintscher closed subtask T349829: List and document all known public SPARQL endpoints serving wikidata as Resolved.Mar 4 2024, 7:36 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-05T22:37:10Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 60 days, 0:00:00 on wdqs[1022-1024].eqiad.wmnet with reason: T337013

Mentioned in SAL (#wikimedia-operations) [2024-03-05T22:37:14Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 60 days, 0:00:00 on wdqs[1022-1024].eqiad.wmnet with reason: T337013

Mentioned in SAL (#wikimedia-operations) [2024-03-07T18:22:18Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 60 days, 0:00:00 on wdqs[1022-1025].eqiad.wmnet with reason: T337013

Mentioned in SAL (#wikimedia-operations) [2024-03-07T18:22:38Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 60 days, 0:00:00 on wdqs[1022-1025].eqiad.wmnet with reason: T337013

Kanashimi subscribed.Mar 7 2024, 10:37 PM

Gehel closed subtask T357980: Compile a set of queries rewritten with federation across the two graph splits as Resolved.Mar 8 2024, 10:19 AM

dcausse added a subtask: T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs.Apr 5 2024, 12:53 PM

Gehel closed subtask T357966: Document limitations of blazegraph federation as Resolved.Apr 5 2024, 1:01 PM

dcausse added a subtask: T361950: Ensure that WDQS query throttling does not interfere with federation.Apr 5 2024, 3:29 PM

Daniel_Mietchen subscribed.Apr 6 2024, 1:04 AM

dcausse added a subtask: T362060: Generalize ScholarlyArticleSplitter.Apr 8 2024, 11:48 AM

Gehel added a subtask: T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs.May 10 2024, 8:27 AM

Gehel closed subtask T362060: Generalize ScholarlyArticleSplitter as Resolved.May 24 2024, 7:28 AM

Gehel closed subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Resolved.Jun 21 2024, 1:17 PM

Gehel reopened subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Open.Jun 21 2024, 1:26 PM

Will this also solve T261764? or is the graph used by the Query Service different than the one used by the API?

Lydia_Pintscher added a subtask: T369079: Update `UniqueValueChecker` to query a list of endpoints.Jul 3 2024, 6:52 AM

In T337013#9923203, @Nosferattus wrote:

Will this also solve T261764? or is the graph used by the Query Service different than the one used by the API?

I don't believe this is using the Query Service. This means it would not be affected.

diegodlh subscribed.Jul 8 2024, 12:34 AM

pfischer changed the status of subtask T369729: RDF Updater Consumer: Allow "empty" patches from Open to In Progress.Jul 11 2024, 8:28 AM

Lucas_Werkmeister_WMDE closed subtask T355298: Investigate the impact of the WDQS graph split on constraints checks as Resolved.Jul 16 2024, 8:28 AM

Gehel closed subtask T367510: Request permission to create 4 kafka topics in kafka-main (WDQS graph split) as Resolved.Jul 19 2024, 12:42 PM

Gehel closed subtask T361935: Adapt the WDQS Streaming Updater to update multiple WDQS subgraphs as Resolved.

Gehel closed subtask T369729: RDF Updater Consumer: Allow "empty" patches as Resolved.

Gehel closed subtask T361950: Ensure that WDQS query throttling does not interfere with federation as Resolved.Jul 19 2024, 12:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-08-23T15:52:57Z] <bking@cumin2002> START - Cookbook sre.hosts.downtime for 17:00:00 on wdqs[1023-1024].eqiad.wmnet with reason: noisy alerts related to graph split T337013

Mentioned in SAL (#wikimedia-operations) [2024-08-23T15:53:13Z] <bking@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 17:00:00 on wdqs[1023-1024].eqiad.wmnet with reason: noisy alerts related to graph split T337013

Nosferattus unsubscribed.Aug 23 2024, 4:00 PM

bking mentioned this in T372094: WDQS: Document review/refresh for https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service.Sep 4 2024, 2:06 PM

bking added a subtask: T372094: WDQS: Document review/refresh for https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service.

bking mentioned this in T374016: Consider separating wdqs-categories from the rest of the wdqs stack.Sep 4 2024, 2:45 PM

Lucas_Werkmeister_WMDE closed subtask T369079: Update `UniqueValueChecker` to query a list of endpoints as Resolved.Sep 4 2024, 3:26 PM

Gehel closed subtask T352538: [EPIC] Evaluate the impact of the graph split as Resolved.Sep 6 2024, 12:52 PM

Gehel removed a subtask: T374021: Make WikibaseQualityConstraints use split-graph query service.Sep 10 2024, 2:06 PM

bking mentioned this in T374967: wdqs-categories migration: decide where to migrate.Sep 23 2024, 2:36 PM

bking mentioned this in T375520: EPIC: WDQS categories migration.Sep 30 2024, 9:35 PM

Stevemunene closed subtask T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs as Resolved.Oct 4 2024, 7:31 AM

bking added a subtask: T374967: wdqs-categories migration: decide where to migrate.Oct 4 2024, 1:21 PM