I think their arguments are that dowloading a rendered PNG file are both more accessible and have a higher quality then trying to take a screen capture. This would likely present more if they are trying to copy a section of text intertwined with formulas, as a resulting paste would then be lacking only the formulas.

In T311620#8442275, @Xaosflux wrote:

I think their arguments are that dowloading a rendered PNG file are both more accessible and have a higher quality then trying to take a screen capture. This would likely present more if they are trying to copy a section of text intertwined with formulas, as a resulting paste would then be lacking only the formulas.

I can understand that. I filed a task for this specific issue T324451.

Can we migrate directly to node 16 (skipping 12 and 14)?

In T311620#8450878, @Physikerwelt wrote:

Can we migrate directly to node 16 (skipping 12 and 14)?

Yes, if you can do all the changes in one go https://docker-registry.wikimedia.org/nodejs16-slim/tags/ is available right now.

Change 865787 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/mathoid@master] Remove PNG support

https://gerrit.wikimedia.org/r/865787

Change 866662 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/mathoid@master] Update svgo

https://gerrit.wikimedia.org/r/866662

Change 865787 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Remove PNG support

https://gerrit.wikimedia.org/r/865787

In T311620#8480958, @gerritbot wrote:

Change 865787 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Remove PNG support

https://gerrit.wikimedia.org/r/865787

This should now be deployed to beta (e.g. https://en.wikipedia.beta.wmflabs.org). @Physikerwelt

In T311620#8481013, @dom_walden wrote:

In T311620#8480958, @gerritbot wrote:

Change 865787 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Remove PNG support

https://gerrit.wikimedia.org/r/865787

This should now be deployed to beta (e.g. https://en.wikipedia.beta.wmflabs.org). @Physikerwelt

It seems to have broken beta as you predicted :). Error message:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "/mathoid/local/v1/":): {\displaystyle f(z) = \sum_{n=0}^{\infty} a_n (z - z_0)^n}

You see the error if you contact RestBase directly,

curl -vd 'q=E=mc^2 \text{221220}' https://en.wikipedia.beta.wmflabs.org/api/rest_v1/media/math/check/tex

{"type":"https://mediawiki.org/wiki/HyperSwitch/errors/internal_http_error","method":"post","detail":"getaddrinfo ENOTFOUND deployment-docker-mathoid02.deployment-prep.eqiad.wmflabs deployment-docker-mathoid02.deployment-prep.eqiad.wmflabs:10044","uri":"/en.wikipedia.beta.wmflabs.org/v1/media/math/check/tex"}%

It seems something is misconfigured?

(Note that you need to use a new query string whenever you make a successful attempt to bypass caching.)

It's using the old FQN (deployment-docker-mathoid02.deployment-prep.eqiad.wmflabs) but it seems correct in puppet config: https://gerrit.wikimedia.org/g/operations/puppet/+/d77677bed318c726f8f4bf519d4242576cab7d7d/hieradata/cloud/eqiad1/deployment-prep/common.yaml
(see https://openstack-browser.toolforge.org/server/deployment-docker-mathoid02.deployment-prep.eqiad1.wikimedia.cloud)

I'll dig

Change 869742 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/mediawiki-config@master] Labs: Use the correct address for mathoid

https://gerrit.wikimedia.org/r/869742

Change 869742 merged by jenkins-bot:

[operations/mediawiki-config@master] Labs: Use the correct address for mathoid

https://gerrit.wikimedia.org/r/869742

That needed fixing regardless but I still need to figure out where that's coming from because it's trying to connect to mathoid02 (correctly, just FQN is wrong) while the mw-config had on mathoid01. I should dig in HyperSwitch.

I assume that there is a restbase config similar to this one https://github.com/wikimedia/restbase/blob/c1d391c74d39d38f7b1a8175e510fa714b4d91f1/config.fullstack.test.yaml That should specify the mathoid host.

The value in restbase04 in /etc/restbase/config-vars.yaml is correct. I restarted it and that I think fixed it.

Thank you. The deployment is fixed. However, future changes in restbase seem to be necessary. See the current output https://en.wikipedia.beta.wmflabs.org/api/rest_v1/media/math/render/svg/de352569c2c921965ffe883b3eb11f48d4af300f

{
  "type": "https://mediawiki.org/wiki/HyperSwitch/errors/server_error#empty_response",
  "method": "get",
  "detail": "Math: missing or malformed response for format png",
  "uri": "/en.wikipedia.beta.wmflabs.org/v1/media/math/render/svg/de352569c2c921965ffe883b3eb11f48d4af300f"
}

hmm, it looks correct to me

I see a similar error when I ask for a png, e.g. https://en.wikipedia.beta.wmflabs.org/api/rest_v1/media/math/render/png/61b3082f39193dd9aa038a8f7d8cafbdb5103db4

I am also seeing errors in logstash, but they don't always get shown to the user.

I occasionally see errors when creating/editing maths formulae in VisualEditor. May be worth testing further.

Change 869763 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/restbase@master] Remove pngs from math endpoint

https://gerrit.wikimedia.org/r/869763

I still think we need to adjust restbase along the lines of https://gerrit.wikimedia.org/r/c/mediawiki/services/restbase/+/869763/. However, it was 2016 when I made the last change to restbase (https://github.com/wikimedia/restbase/commit/4fccb85ffaf154be3ef12aa684c2a8172a2d9d62). At that time, restbase was stored on GitHub.

Change 866666 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/mathoid@master] Update dependencies and require node16

https://gerrit.wikimedia.org/r/866666

Change 873490 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/extensions/MathSearch@master] Remove png support

https://gerrit.wikimedia.org/r/873490

Change 873490 merged by jenkins-bot:

[mediawiki/extensions/MathSearch@master] Remove png support

https://gerrit.wikimedia.org/r/873490

@Physikerwelt I have managed to get both Mathoid and Restbase running locally.

I am testing out https://gerrit.wikimedia.org/r/869763.

After checking it out, I no longer get the error (printed to stdout by the restbase service):

{"name":"restbase","hostname":"x200","pid":2757,"level":50,"message":"500: server_error#empty_response","res":{"name":"HTTPError","message":"500: server_error#empty_response","status":500,"body":{"type":"server_error#empty_response","description":"Math: missing or malformed response for format png"}, ...

which is good.

However, I am getting a new error when loading a path with <math> tags:

{"name":"restbase","hostname":"x200","pid":2822,"level":40,"err":{"message":"404: not_found#route","name":"restbase","stack":"HTTPError: 404: not_found#route\n    at HyperSwitch._request (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:270:19)\n    at _createFilteredHandler (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:172:26)\n    at tryCatcher (/home/drw/wikimedia/srv/restbase/node_modules/bluebird/js/release/util.js:16:23)\n    at /home/drw/wikimedia/srv/restbase/node_modules/bluebird/js/release/method.js:15:34\n    at handlerWrapper (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:424:37)\n    at next (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:410:42)\n    at Object.module.exports [as filter] (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/filters/http.js:24:16)\n    at handlerWrapper (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:422:27)\n    at /home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:428:28\n    at HyperSwitch._filteredRequest (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:173:19)\n    at HyperSwitch.(anonymous function) [as post] (/home/drw/wikimedia/srv/restbase/node_modules/hyperswitch/lib/hyperswitch.js:462:21)\n    at MathoidService._invalidateCache (/home/drw/wikimedia/srv/restbase/sys/mathoid.js:187:22)\n    at MathoidService._storeRenders (/home/drw/wikimedia/srv/restbase/sys/mathoid.js:149:24)\n    at hyper.post.then (/home/drw/wikimedia/srv/restbase/sys/mathoid.js:166:18)\n    at tryCatcher (/home/drw/wikimedia/srv/restbase/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/home/drw/wikimedia/srv/restbase/node_modules/bluebird/js/release/promise.js:547:31)","status":404,"body":{"type":"not_found#route","title":"Not found.","internalURI":"/wikimedia.org/sys/events/","method":"post","depth":2},"root_req":{"method":"get","uri":"/localhost/v1/media/math/render/svg/6e865e5b244a7a5d4fdf7a95fe4cea6d25e581ac","headers":{"cache-control":"no-cache","user-agent":"Mozilla/5.0 (X11; Linux ppc64le; rv:91.0) Gecko/20100101 Firefox/91.0","x-client-ip":"::ffff:192.168.10.117","x-forwarded-for":"::ffff:192.168.10.117","x-request-id":"2971a6e0-9270-11ed-9678-6fffb2fa919e","x-request-class":"external"}},"request_id":"2971a6e0-9270-11ed-9678-6fffb2fa919e","api_path":"/{domain}/v1/media/math/render/{format}/{hash}","levelPath":"warn/bg-updates"},"msg":"404: not_found#route","time":"2023-01-12T11:56:34.906Z","v":0}

But, in spite of this, on my local mediawiki using my local restbase the formulae display fine and no signs of anything going wrong. I just get the above error printed by restbase.

My setup is:

• mathoid commit a35cb15a2e85f52d09ba2068a0ec1eae719816fe running in a docker container built from .pipeline/blubber.yaml on host Debian Bullseye.
• restbase checkout out from 869763 running on "bare metal" Debian Buster (npm install && npm start).
• mediawiki Math commit 1c4e3fd6ca56f8eb288ee428e7f0ecb38eb9e72c docker running on the same Debian Bullseye host as mathoid.

In T311620#8519411, @dom_walden wrote:

However, I am getting a new error when loading a path with <math> tags:

This error might be T216433. I tried to modify my config.yaml in the way suggested by T216433#4966821 but I couldn't get it to work.

Hi @dom_walden,

Thank you for taking the time and testing the change. I propose, instead of testing from the math extension, testing from restbase. Otherwise, there are too many intermediate steps.

Are you aware of

https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/875405

from @daniel ?

I guess if you test that, you should be able to get the SVG and MathML from Restbase but no longer PNG. Does that work?

If so, the next step would be to check if you use your local restbase server from your math extension. The error message, i.e.,

{"type":"not_found#route","title":"Not found.","internalURI":"/wikimedia.org/sys/events/","method":"post","depth":2}

suggests to me that this is unrelated to https://gerrit.wikimedia.org/r/869763. So you should get the same error message wether or not you try with master or https://gerrit.wikimedia.org/r/869763 or restbase. I think the config provided by @daniel solves the localhost vs. wikimedia.org issue.

In T311620#8527717, @Physikerwelt wrote:

Hi @dom_walden,

Thank you for taking the time and testing the change. I propose, instead of testing from the math extension, testing from restbase. Otherwise, there are too many intermediate steps.

I have given the restbase API a brief test. I could use the three endpoints (/media/math/check/tex, media/math/formula/$hash, /media/math/render/$format/$hash) to generate an SVG and MML representation of a formula.

/media/math/render/png/$hash just returns: "data.params.format should be equal to one of the allowed values: [svg, mml]"

Are you aware of

https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/875405

Yes, that was what I was using to get my local setup working.

I guess if you test that, you should be able to get the SVG and MathML from Restbase but no longer PNG. Does that work?

Yes. Locally, maths formulae display fine.

I tried with different values of $wgDefaultUserOptions['math'] and they all worked (png just returned an MML+SVG).

If so, the next step would be to check if you use your local restbase server from your math extension. The error message, i.e.,

{"type":"not_found#route","title":"Not found.","internalURI":"/wikimedia.org/sys/events/","method":"post","depth":2}

suggests to me that this is unrelated to https://gerrit.wikimedia.org/r/869763. So you should get the same error message wether or not you try with master or https://gerrit.wikimedia.org/r/869763 or restbase. I think the config provided by @daniel solves the localhost vs. wikimedia.org issue.

I was using their config. But I think you are right that it is unrelated. I don't think it needs to be investigated further.

In T311620#8488924, @gerritbot wrote:

Change 866666 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/services/mathoid@master] Update dependencies and require node16

https://gerrit.wikimedia.org/r/866666

I have also tested this in my local mediawiki + mathoid + restbase setup. I have not seen any problems. Maths formulae display fine.

Thank you. What is the next step? Can I do anything to foster progress?

Change 869763 abandoned by Physikerwelt:

[mediawiki/services/restbase@master] Remove pngs from math endpoint

Reason:

review on github

https://gerrit.wikimedia.org/r/869763

In T311620#8528551, @Physikerwelt wrote:

Thank you. What is the next step? Can I do anything to foster progress?

Sorry for the late reply.

Hopefully @Jgiannelos can review https://github.com/wikimedia/restbase/pull/1311.

I will look into getting it deployed to beta as well as https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/866666.

Thank you. The review for https://gerrit.wikimedia.org/r/c/mediawiki/services/mathoid/+/866662/7 was already done; however, CI-merge failed. I think a consultation with OPS is required here.

Change 890466 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] mathoid: Switch to using node instead of nodejs

https://gerrit.wikimedia.org/r/890466

Change 890466 merged by jenkins-bot:

[operations/deployment-charts@master] mathoid: Switch to using node instead of nodejs

https://gerrit.wikimedia.org/r/890466

Change 866662 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Update svgo and switch to node v16

https://gerrit.wikimedia.org/r/866662

Change 866666 merged by jenkins-bot:

[mediawiki/services/mathoid@master] Update dependencies and require node16

https://gerrit.wikimedia.org/r/866666

Change 898882 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/extensions/Math@REL1_39] Remove PNG rendering mode

https://gerrit.wikimedia.org/r/898882

Change 898882 merged by jenkins-bot:

[mediawiki/extensions/Math@REL1_39] Remove PNG rendering mode

https://gerrit.wikimedia.org/r/898882

Change 899589 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/extensions/Math@REL1_35] Remove PNG rendering mode

https://gerrit.wikimedia.org/r/899589

Change 899589 abandoned by Physikerwelt:

[mediawiki/extensions/Math@REL1_35] Remove PNG rendering mode

Reason:

quite a few conflicts

https://gerrit.wikimedia.org/r/899589

Per discussion elsewhere: we need to retain some level of support for the png endpoints, since wmf is offering them as apis for use by the math extension on 3rd party wikis.

Current thinking is to return SVG from the PNG endpoints, with the appropriate Content-Type header. This breaks the contract of course, but is likely to just work in browsers.

In T311620#8804762, @daniel wrote:

Per discussion elsewhere: we need to retain some level of support for the png endpoints, since wmf is offering them as apis for use by the math extension on 3rd party wikis.

I don't agree. Can you please link to this task and mark it as a blocker?

I agree with James. Mathoid API in wikimedia production is to serve Wikimedia wikis, not random 3rd party users. The default was set to point to wikimedia for ease of installation for development (which I don't even agree it was a good idea) not "here is some free service for you".

Some usecases already moved away from it like Fandom because they mistakenly relied on our API (=not knowing they are relaying on Wikimedia services because defaults).

Honestly we should change the default

In T311620#8804767, @Jdforrester-WMF wrote:

I don't agree. Can you please link to this task and mark it as a blocker?

It's on github: https://github.com/wikimedia/restbase/pull/1311

In T311620#8804804, @Ladsgroup wrote:

I agree with James. Mathoid API in wikimedia production is to serve Wikimedia wikis, not random 3rd party users. The default was set to point to wikimedia for ease of installation for development (which I don't even agree it was a good idea) not "here is some free service for you".

That's exactly what the documentation sais though: https://www.mediawiki.org/wiki/Extension:Math

Some usecases already moved away from it like Fandom because they mistakenly relied on our API (=not knowing they are relaying on Wikimedia services because defaults).

As far as I know, Fandom is still using our rendering. They just moved from PNG to SVG.

In T311620#8805001, @daniel wrote:

In T311620#8804767, @Jdforrester-WMF wrote:

I don't agree. Can you please link to this task and mark it as a blocker?

It's on github: https://github.com/wikimedia/restbase/pull/1311

In T311620#8804804, @Ladsgroup wrote:

I agree with James. Mathoid API in wikimedia production is to serve Wikimedia wikis, not random 3rd party users. The default was set to point to wikimedia for ease of installation for development (which I don't even agree it was a good idea) not "here is some free service for you".

That's exactly what the documentation sais though: https://www.mediawiki.org/wiki/Extension:Math

It's a wiki anyone can edit :P In the older reversions it's not recommended and I don't think this is an official WMF-backed suggestion. Having mathoid service as an official service for any 3rd party mediawiki installation opens the Pandora box of maintenance overhead, deprecation policy, communication, so on and so forth. And unless we have a dedicated team to maintain the services and approval from legal and more. I don't think it's a reasonable option.

Also note that the mathoid that's default is the mathoid in beta cluster. It's not even the actual mathoid.

Some usecases already moved away from it like Fandom because they mistakenly relied on our API (=not knowing they are relaying on Wikimedia services because defaults).

As far as I know, Fandom is still using our rendering. They just moved from PNG to SVG.

Yeah but they are okay with setting up their own mathoid service and move away from our setup

Pinging @VirginiaPoundstone. As the PM of API, I'd say it's up to her to decide what level of compatibility we need to retain.

In the future, we will hopefully have a way to distingish stable from unstable APIs just by looking at the URL...

I do get user-facing APIs needing to be stable but this is internal API to mediawiki. If someone wants png support for their own mw installation, they can setup up either their own mathoid service or cli mathoid. They already have a mw setup. To me, it's like having a service that provides memcached to 3rd party mw installations.

I do get user-facing APIs needing to be stable but this is internal API to mediawiki.

As someone who set up Math four years ago, how would you have known that?

In T311620#8805581, @daniel wrote:

I do get user-facing APIs needing to be stable but this is internal API to mediawiki.

As someone who set up Math four years ago, how would you have known that?

Given that the Math extension was entirely unsupported until a few months ago when we added it to the tarball, how would said human have expected this to magically keep working?

And the default is the mathoid in beta cluster that is completely unmaintained and unstable and goes down for weeks (services, mw is "just" days usually). People are already relying on an unstable API (infra-wise).

In T311620#8805725, @Ladsgroup wrote:

And the default is the mathoid in beta cluster that is completely unmaintained and unstable and goes down for weeks (services, mw is "just" days usually). People are already relying on an unstable API (infra-wise).

The default config that Math comes with is this:

		"MathFullRestbaseURL": {
			"value": "https://wikimedia.org/api/rest_"
		},

That's not an unstable endpoint on beta, is it? if it is, how would people know?

The restbase is production but the actual mathoid is beta cluster: from my understanding https://github.com/wikimedia/mediawiki-extensions-Math/blob/master/extension.json#L122

		"MathMathMLUrl": {
			"value": "https://mathoid-beta.wmflabs.org"
		},

In T311620#8810174, @Ladsgroup wrote:

The restbase is production but the actual mathoid is beta cluster: from my understanding https://github.com/wikimedia/mediawiki-extensions-Math/blob/master/extension.json#L122

		"MathMathMLUrl": {
			"value": "https://mathoid-beta.wmflabs.org"
		},

Uh - where's the real mathoid then, and how and when do we route to it? I mean, doesn't https://wikimedia.org/api/rest_v1/media/math/render/svg/c70ed61797014507069e84897d9a80acca352bcf use the same config?

Oh, but this is just for MathML, right? So it's not used for other kinds of rendering?

I guess it's partially goes to production mathoid (through restbase?) and then partially to beta cluster which is problematic but oh well.

Anyway, the point James said still stands.

In T311620#8813089, @Ladsgroup wrote:

I guess it partially goes to production mathoid (through restbase?) and then partially to beta cluster which is problematic but oh well.

MathMathMLUrl should be only used if MathML is used as input, or LaTeXML is used for rendering. Both should be disabled in production.

To me, it is much more important, to deliver mathematics to people with limited vision than to support browsers that are incapable of rendering SVG:-)

https://github.com/wikimedia/restbase/pull/1311 has been open for a while. Should I go bother Virginia about this?

@Ladsgroup yes, that would be great.

Change 965923 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/extensions/Math@master] Stop storing PNG images

https://gerrit.wikimedia.org/r/965923

From https://gerrit.wikimedia.org/r/q/Ib2c216f54e6817ee2c3be0355ba72bd4769ba6ea

Doesn’t MediaWiki keep track of patches it has already applied? Ouch. Then I guess we have to remove the old migration indeed, but I’d like to get a confirmation from James or someone else more familiar with the MW database handling.

In that context, I was also wondering if beta has a mathoid table? I guess production does not have one, or at least it is not used.

Change 965923 abandoned by Physikerwelt:

[mediawiki/extensions/Math@master] Stop storing PNG images

Reason:

probably too big, split into several changes

https://gerrit.wikimedia.org/r/965923

Change 969508 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/extensions/Math@master] Explain the significance of the img.tex css element

https://gerrit.wikimedia.org/r/969508

Change 969510 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[mediawiki/extensions/Math@master] Remove functional dead code about PNG images

https://gerrit.wikimedia.org/r/969510

Change 969508 merged by jenkins-bot:

[mediawiki/extensions/Math@master] Explain the significance of the img.tex css element

https://gerrit.wikimedia.org/r/969508

Change 969510 merged by jenkins-bot:

[mediawiki/extensions/Math@master] Remove functional dead code about PNG images

https://gerrit.wikimedia.org/r/969510

Change 971244 had a related patch set uploaded (by Physikerwelt; author: Physikerwelt):

[operations/mediawiki-config@master] Enable native math rendering mode on testwiki

https://gerrit.wikimedia.org/r/971244

Change 971244 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable native math rendering mode on testwiki

https://gerrit.wikimedia.org/r/971244

Mentioned in SAL (#wikimedia-operations) [2023-11-02T20:18:49Z] <mabualruz@deploy2002> Started scap: Backport for [[gerrit:971244|Enable native math rendering mode on testwiki (T311620)]]

Mentioned in SAL (#wikimedia-operations) [2023-11-02T20:20:11Z] <mabualruz@deploy2002> mabualruz and physikerwelt: Backport for [[gerrit:971244|Enable native math rendering mode on testwiki (T311620)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-11-02T20:32:55Z] <mabualruz@deploy2002> Finished scap: Backport for [[gerrit:971244|Enable native math rendering mode on testwiki (T311620)]] (duration: 14m 06s)

Anything left to do here?