I resumed working on this now that we have instrumentation for EventBus.

This task has been superseded by {T363587: [Event Platform] Instrument EventBus with prometheus MW Statslib}.

In T366836#10157337, @Snwachukwu wrote:

@gmodena , We have the mediawiki group and wmde group (recently added by developer-experience) available in gitlab. So I have added them to the members of the Secondary event schema repo in Gitlab and given them the developer role.

So everyone in that group should be able to clone the repo and create an MR directly and merge once approved. No need to fork + MR.

Let me know if this is okay.

In T363587#10117871, @gerritbot wrote:

Change #1070252 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] Fix buggy events_accepted_total metric

https://gerrit.wikimedia.org/r/1070252

We mirror repos (read-only) from Gerrit to GitHub, is there a way we can do the same to Gitlab? That could help soft landing this roll out.

@Snwachukwu might you need inspiration on drafting a contribution policy for the repos your are migrating, maybe these could be of help: https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/blob/main/CONTRIBUTING.md and https://www.mediawiki.org/wiki/Platform_Engineering_Team/Event_Platform_Value_Stream/Contribution

@Fabfur thanks for the heads up. We'll need to coordinate a bit on roll out, because the previous ingestion and processing infra has been decommissioned as part of T372456: Rollback haproxy feed automated ingestion.

Are you planning to start shipping to Kafka anytime soon? It would be great to touch base and iron out the timelines in case, just to stay on the same page.

@dcausse @pfischer and I had a chat about this phab today. Here are some notes from our conversation,

In T374055#10122014, @Ottomata wrote:

nit: this said, I would prefer not to use page_change to describe the Dumps 2.0 streams.

Generally +1. But, do you think we can use approximately the same data model, perhaps even using some of the MW state entity schema fragments? Where it is easy to do so anyway?

And page_change_kind? and maybe changelog_kind?

In T373086#10116254, @dcausse wrote:

In T373086#10116237, @gmodena wrote:

Looks like the spike is only related to CirrusSearch?

yes this is my understanding as well, it is quite noisy so it might hide others, I'm shipping a fix so hopefully we'll get some more clarity soon.

Ack. Investigating.
My original patch only addressed EventBus. I can't seem to find any new warn entry for that service in logstash.

I suppose we should target running Flink reconciled enrichment job in dse-k8s at first anyway, even if we decide we want to run the job multi-DC in wikikube eventually. So let's go for it!

I don't know why matching on the job label alone did not work, by a more extensive match (all labels except instace, that is empty) did the trick:

@colewhite ack. Thanks for the heads up.

f/up from a convo with @fgiunchedi in IRC.

The alert compares the ratio of flink_taskmanager_job_task_operator_event_process_function_events_out_total over flink_taskmanager_job_task_operator_event_process_function_events_in_total.

I did some investigation both on the affected Kafka topics and related hive event tables. Quick update with my findings so far.

Remove the Gobblin MapReduce job that loads Kafka topics.

I'd also like to do 2 more things before we resolve this ticket:

In T372456#10064121, @Ottomata wrote:

@gmodena I just pushed https://gerrit.wikimedia.org/r/c/operations/puppet/+/1062707, it prob has to go before the .pull file is removed.

Ack. Thanks for the heads up and CRs @fgiunchedi

In T346046#9992635, @Ottomata wrote:

Couple of WIP patches up for discussion.

• Use StreamConfigs to determine if an event should be sent in EventBus send function. (1053785)
• WIP - Refactor EventBus to be a Service (1054934)

The second depends on the first.

@gmodena I like the way this is headed, but I'm not sure if following through (and fully deprecating EventBusFactory) is worth it. Let's get together and discuss.

When gobblin moves to airflow, we can use Artifact sync to deploy the jar, instead of relying on analytics/refinery.

A dashboard for EventBus is available in Grafana

After re-scoping both Config Store and MPIC, we decided to not move forward with refactoring ESC at this stage. A summary of this task is available on wikitech. A design doc is available at Stream Registry.

DQ job and airflow dag have been updated. Deployment requires a new release of refinery-source and a dag deployment on analytics.

DQ job and airflow dag have been implemented. Deployment requires a new release of refinery-source and a dag deployment on analytics.

Instrumentation has been enabled in beta. You can test it by modifying a page on https://simple.wikipedia.beta.wmflabs.org.

In T368787#9949570, @Ottomata wrote:

But can we assume the stream will be produced directly into jumbo, and won't have multi dc / replication requirements?

I think we decided we want to do reconcilliation of page_change and page_content_change in general, so that it can be used for Search and others.

Consume this new event stream