In T369547#10323793, @dchan wrote:

Thanks @EAkinloose, that's really helpful. It looks like the Citoid service might be returning 404 on beta cluster. I'm seeing a 404 error from the REST API:

https://en.wikipedia.beta.wmflabs.org/api/rest_v1/data/citation/mediawiki/https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F9%2F98%2FColoring_page_for_Wikipedia_Day_2019_in_NYC.pdf?action=query&format=json

That means we never see a 415 error, and therefore never see the more specific error text. @Mvolz is this an expected issue on beta cluster?

It's working now (though outdated.)

Example get response:

WSJ.com is one of the publishers we can't cite because of CloudFront. I'm not sure if there are others.

Resolving as deployed (Went out to group 1 and 2 Tuesday.)

Any news?

In T378461#10309264, @fnegri wrote:

• for refill I have just opened a phab ticket: T378686: refill: review citoid usage
• for citations we have the on-wiki bug report opened by @Mvolz

I think both have been resolved/addressed. @Mvolz do you think this task can be resolved?

Which graph would these data go into? The main graph, or the scholarly metadata one?

@ppelberg we only keep the last 3 months of data. For the 6 months, should I download now and wait until we collect the next 3 months worth of data and compile it, or just do 3 months.

Now we have better tooling we can examine this more in depth; NPR requests are actually timing out and responding with 504. This would be slightly weird for a deliberate block as you're more likely to get a 403 and for the response to be relatively fast where indeed these are actually timing out.

I see in the other thread you don't have Turnilo access, sorry. The user-agent is 'reFill/2 (http://en.wikipedia.org/wiki/User:Zhaofeng_Li/reFill)'

In T378686#10282814, @Curb_Safe_Charmer wrote:

There is certainly nothing in the service log that indicates that reFill was called 30,000 times in any 12 hours.

In T369547#10079818, @ppelberg wrote:

In T369547#10063836, @Mvolz wrote:

I find this message kind of confusing because the statements aren't explicitly connected.

"We couldn't make a citation for you because the resource is in a format we don't support" sounds clearer to me.

Great spot, @Mvolz . What do you think about the proposed version below?

Separately: what page do you think should be linked from the format we don't support portion of the message?

We couldn't create a citation for you.<br>
The resource is in a [[TBD | format we don't support]]. Create a citation manually below.

Should we trial this for a month or something? I don't like it for the long term but maybe worth seeing if there's an impact?

In T377828#10250601, @colewhite wrote:

"Count Percentage" isn't a calculation that OpenSearch returns in its response. It's not even a percent of all hits returned by the query. Count percentage is calculated client-side by dividing each leaf aggregation doc_count by the sum of all branch root aggregations doc_count. This makes it heavily influenced by the size parameter.

I don't think it's possible in the current implementation. OpenSearch Dashboards would need a new feature to do it.

So something like: https://logstash.wikimedia.org/app/visualize#/edit/a2d1a480-473d-11ef-8912-517dff44673b?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1M,to:now))&_a=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'logstash-*',key:outgoingReqResult.hostname.keyword,negate:!t,params:(query:en.wikipedia.org),type:phrase),query:(match_phrase:(outgoingReqResult.hostname.keyword:en.wikipedia.org)))),linked:!f,query:(language:kuery,query:''),uiState:(),vis:(aggs:!((enabled:!t,id:'1',params:(),schema:metric,type:count),(enabled:!t,id:'3',params:(exclude:'200',field:outgoingReqResult.status.keyword,include:'',missingBucket:!f,missingBucketLabel:Missing,order:desc,orderBy:'1',otherBucket:!f,otherBucketLabel:Errors,size:5),schema:bucket,type:terms),(enabled:!t,id:'2',params:(field:outgoingReqResult.hostname.keyword,missingBucket:!f,missingBucketLabel:Missing,order:desc,orderBy:'1',otherBucket:!f,otherBucketLabel:Other,size:100),schema:bucket,type:terms)),params:(perPage:10,percentageCol:Count,showMetricsAtAllLevels:!f,showPartialRows:!f,showTotal:!f,totalFunc:sum),title:'Top%20errors%20by%20requested%20domain',type:table))

In T349118#10214863, @Ottomata wrote:

What service are you using this for?

https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service
T361769: Migrate and re-deploy eventstreams using service-utils

Unfortunately we need [...] Open API spec generation.

OpenAPI spec generation sounds great. File a task...and help implement it!?

I don't believe service-runner or service-template-node ever supported this though, did it? FWIW, EventStreams also generates some spec, but it is manual, and not handled by a lib.

It was part of the list of things "not included" by service-utils so I assumed it was service-runner, but yeah now that I look that's in service-template-node, not service-runner: https://github.com/wikimedia/service-template-node/blob/main/lib/swagger-ui.js

Unfortunately we need [...] workers

Curious, why do you need workers? Because WMF services are deployed on k8s now, we decided to rely on k8s pod replicas and k8s routing, instead of node.js worker / clustering.

Another user report: https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#Adding_Hindustan_Times_sources

In T349118#10081252, @Ottomata wrote:

FYI, service-utils (a replacement for service-runner) is nearing its first production deployment. In case you want to wait for it instead of solving service-runner problems :) cc @tchin

I think your best bet to accomplish something like this is a userscript/gadget.

Quoting from T372438 thread:

In T372438#10196433, @DLynch wrote:

Looking at how it was done...

automatic-generate-fail-searchResults	The request made it all the way through the reliability check, tried to build the templates from the citoid results, and failed at that point (probably because there were no results, but could also include any other errors in building the templates)
automatic-generate-fail-network	The promise from the mw.Api request was rejected - without any testing for further details, so I think this is a consolidation of any possible API errors + any network errors that interrupted the request

I think these events are misleadingly labelled, then.

automatic-generate-fail-network is more like no search results from the api. When we don't have results, it returns a 404 (or a 415 if we don't have results because it's a pdf). This likely is the majority of cases and the numbers roughly match the api numbers. Only rarely would it be some sort of connection issue.

You don't get to network success unless there are search results. So the automatic-generate-fail-searchResults thing means there were definitely search results from the API. However, in some rare cases even if there are results from the API, we fail to build a template. One reason might be if the template listed for it doesn't have any template data. Then we can't build the template, so it fails.

In T372438#10196542, @DLynch wrote:

Which is to say: if citoid is returning an API error when we encounter a non-200 status code, all of that should get swept into the network category.

1. How do the following metrics compare in the two weeks before and after 3 August 2024?

403 responses:

In T372438#10188550, @ppelberg wrote:

In T374624#10158451, @MNeisler wrote:

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

Overall proportion of failed citoid editing sessions by reason:

Reason	July 2024	August 2024
Automatic citation generation failed	19.1%	19.3%
* Network error while attempting to retrieve the necessary information	18.4%	18.7%
* No search results were found	0.7%	0.6%
Session aborted	80.9%	80.7%

As found in T368988#10123059, about 19% of all citoid editing sessions that failed to insert a reference were due to an automatic citation generation failure. Of these sessions, the majority of citation generation failures were due to a network error while retrieving citation info. Network errors account for 18.7% of all citoid sessions that failed and 97% of all citoid sessions that failed due an automatic citation generation failure.

There were no observed changes in observed failure rates before or after the decrease in Citoid traffic on August 3rd.

Per platform proportion of failed citoid editing sessions by reason:
Per platform results are very similar to the overall results. The majority of citation generation failures on both desktop and mobile were due to a network error. 97% of all citation generation failures on desktop were due to a network error and 95% of all citation generation failures on mobile were due to a network error.

Time Series Chart
I further investigated the daily number of Citoid editing events and confirmed there were no any sudden changes in the number of automatic citation generation failure events right around August 3rd. The number of daily citoid sessions that have failed to insert a reference due to a network failure to retrieve results has remained stable before and after the change in Citoid traffic.

From this, it still appears that the 3 August 2024 change had no impact on overall user success or failure rates using Citoid on desktop or mobile.

cc @ppelberg

This was mostly specifically related to our probe, i.e. only requests to wikipedia.org were timing out; @akosiaris determined this was due to it only being available in iPv4 and node 18 preferring iPv6.

In T374831#10167374, @Trizek-WMF wrote:

For Tech News:

[technical] "When Citoid generates a reference based on an archive.org URL, we currently have no way of populating the archive-url and archive-date parameters in the citation template. The archive.org URL is considered the "original" URL, not as an archive link. The Editing Team is working on fixing this. We are asking communities to preemptively add mappings for these parameters to the citoid map within the TemplateData for each citation template."

Though might get complaints i.e. if we add a link when we're blocked, and we're de facto marking nytimes links as dead because that's the default if we don't include the parameter... but I don't see an easy way around this anyway because if we're blocked we don't know for sure by code... could assume 415 is blocked and 404 is not available but so many different codes are used here)

In T374831#10148459, @Esanders wrote:

@Mvolz I'm not sure how urlStatus would work given the value type is different per wiki:

• on en.wiki it is a one of various strings (dead/live/deviated/unfit...)
• on fr.wiki it is a boolean/datetime (brisé le broken since = 4 juin 2018 or oui).
• on es.wiki it is a boolean (urlmuerta = [sí|no])

@dchan @zoe before we actually start using the zotero repo in gitlab we need to set up ci to build the docker image. @dchan would this be something you could do?

In T370809#10095847, @zoe wrote:

I was thinking of having an allow or denylist to only log certain headers, and possibly simply logging if cf-mitigated is present and what its value is. That way, when we're looking at metrics we can start to segment by whether we hit a web application firewall or are having a site-specific issue.

No, we still need to actually log these. I wasn't sure who was working on this but I just started a PR... right now I'm just JSON.stringifying the headers, but were there particular headers we wanted? Or do we juts stringify the whole thing?

FYI Citoid upgrade is blocked by dependency on service-runner: https://github.com/wikimedia/service-runner/pull/251

In T370702#10053072, @ppelberg wrote:

In T370702#10044977, @Mvolz wrote:

I've added some stats here.

Wonderful.

I'm looking into getting some longer term data from metrics.

Excellent.

To get the above stats, I've added some new panels to the dashboard:

"Outgoing requests by status" in the lower right hand corner reports the percentage which are 200, so this is our "success" rate, and the failure rate is just 100% minus that.

Unfortunately this includes some pdfs are "succeeding" because some pdfs return us a 200, so for the above data I've filtered out the pdfs in a hacky way by excluding *.pdf on the request url.

@mvols: 2 questions in response:

1. What field did you filter out *.pdf from? [i]
2. With .pdf approximately excluded, are there any other media types that we ought to try to exclude such that the metrics Outgoing requests by status return are specific to URLs? [ii]

And hey! Thank you for sharing how you arrived to these metrics. You doing so helps equip me with the know-how I need to use this dashboard to ask and answer questions of Citoid independently.

---

i. This is the interface I'm assuming you used to create the filter you described above:

ii. One thought: maybe the effort required to do this is not worthwhile considering how minimal we assume the traffic to be for formats like .mov?

I find this message kind of confusing because the statements aren't explicitly connected.

I've added some stats here.

We can start logging response headers after https://gerrit.wikimedia.org/r/c/mediawiki/services/citoid/+/1056571 is merged, as this change makes the get response (and consequently the headers) available. (currently still wip... hopefully almost done though!)

I've implemented this.

In T115224#6415888, @Cyberpower678 wrote:

I would like to emphasize that the availability API has seen improvements since 2017. Is it possible you can redo your investigation a bit?

Something to note: preq library will automatically retry requests in certain situations so potentially every preq request is two.

In T369928#9979219, @DLynch wrote:

Is this for purely client-side cache (in-memory, just affecting people who repeatedly click "generate" in a single session)? If so it seems harmless, but I can't imagine it'd have much impact.

In T365583#9962638, @ppelberg wrote:

In T365583#9946857, @Mvolz wrote...

@Mvolz, three questions in response to this update...

1) What work would be involved to, as you described, "...detect content-type in citoid first."?

A patch in the backend, but I would do this after 2) is resolved.

2) How – if at all – does allowing, "...restbase/hyperswitch to correctly pass through the error itself." impact our ability to detect the content-type that caused the error to be activated and subsequently, offer people feedback specific to the content-type they're trying to cite?

We can do this now but not with the specified content-type, as we are returning 415 but not with a specific rejected content type.

In T364594#9790608, @Trizek-WMF wrote:

Can we create a link to the "Manual tab"? If so the message should invite users to fill a template.

I think @Esanders mentioned in that task earlier that we cannot check user's browser for caches of other sites they've visited as it violates our privacy policy.

I had a look at our error percentage from when we deployed this and it doesn't seem to have done much for citation success rate in any particular direction. (New panel alert!)

This caused T368971.

I've looked into this more and this seems to be a direct consequence of switching from head requests from get requests in order to follow redirects here: T367452

In T368971#9942816, @Vahurzpu wrote:

After a bit of poking around, it appears the underlying issue is with the website, which (depending on user-agent) will send a redirect to google.com.hk. It has the same behavior in curl:

$ curl https://www.clydeships.co.uk/view.php?ref=10727
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="http://www.google.com.hk//?ref=10727">here</a>.</p>
</body></html>

I'm not sure exactly what user-agents cause it to have issues, though. Zotero can fetch the website just fine, and the text "Custom user agent" causes it to send back the actual page content. Maybe Citoid is explicitly deny-listed?

I'm not sure how long it will take to fix the hyper switch issue. We might need to switch from restbase to the api gateway first.

No QA needed.

In T367870#9936182, @colewhite wrote:

In T367870#9936065, @Mvolz wrote:

Okay, so if I redeploy right before midnight, we should all be good?

That can work, but note that if you do change the field type, you will lose the ability to query or filter that field while the field exists as multiple types. This conflict could take 90 days to resolve.

In T367870#9935760, @colewhite wrote:

In T367870#9935427, @Mvolz wrote:

This week I tried to deploy a change that would have changed outgoingReqResult.error from a JSON obj to a string.

A change like that would cause OpenSearch to throw:

mapper_parsing_exception: object mapping for [error] tried to parse field [error] as object, but found a concrete value

All logs that did not match the previous schema would be dropped until index rollover (midnight each day).

In T367870#9903841, @colewhite wrote:

Indexed fields list is refreshed and outgoingReqResult.* is now searchable and aggregatable.

Please reach out regarding ECS. We're happy to help folks navigate the migration. :)

In T361728#9933441, @JMeybohm wrote:

For zotero I'm afraid we can't add additional logging. AIUI the software does not have any proper logging options. But I think it's might also be more valuable to check on the citoid side as there is clearly where things change with the update.
From what I can see in the logs from citoid and the tls terminator of zotero is these requests failing: https://logstash.wikimedia.org/goto/208ad22ca41ab54070c889a59297b369
I would suggest to check which requests are actually send to zotero and then see how they differ between nodejs versions.

Tried redeploy on just codfw with the upgraded chart, it did Not Go Well.