Moreover, I'd like to ask if there is any point of having all of the donate.$project.org combinations. Do people actually ever go to e.g. donate.wikiversity.org? We never advertise that link anywhere, as far as I know.

@MeganHernandez_WMF can you let us know about the rest of the domains? Is this how the sidebar linking works or anything?

Once these two domain names, www.donate.wikimediafoundation.org and www.donate.mediawiki.org are removed, wikimediafoundation.org and mediawiki.org can be preloaded. Fortunately, searching them on Google returns no results: https://www.google.com/search?q=%22www.donate.wikimediafoundation.org%22 and https://www.google.com/search?q=%22www.donate.mediawiki.org%22. So I think it is safe to remove at least these two.

[1] https://raw.githubusercontent.com/wikimedia/operations-dns/master/templates/mediawiki.org
[2] https://raw.githubusercontent.com/wikimedia/operations-dns/master/templates/wikimediafoundation.org

Oh, actually http://www.donate.wikimediafoundation.org/ redirects to https://wikimediafoundation.org/wiki/Home, and http://www.donate.mediawiki.org/ shows an "unconfigured domain" error page. So they are broken already.

Change 222876 had a related patch set uploaded (by Chmarkine):
Remove www.donate.wikimediafoundation.org from DNS

https://gerrit.wikimedia.org/r/222876

Change 222877 had a related patch set uploaded (by Chmarkine):
Remove www.donate.mediawiki.org from DNS

https://gerrit.wikimedia.org/r/222877

Change 222880 had a related patch set uploaded (by Chmarkine):
Remove www.donate.wiktionary.org from DNS

https://gerrit.wikimedia.org/r/222880

Change 222883 had a related patch set uploaded (by Chmarkine):
Remove www.donate.wikipedia.org from DNS

https://gerrit.wikimedia.org/r/222883

Regarding the mkt41.net cnames, see also T74514 and T60373.

In T102827#1374685, @faidon wrote:

Moreover, I'd like to ask if there is any point of having all of the donate.$project.org combinations. Do people actually ever go to e.g. donate.wikiversity.org? We never advertise that link anywhere, as far as I know.

Seems unlikely indeed, though it's not unlikely people may mistype donate.wikimedia.org for donate.mediawiki.org or donate.wikipedia.org. And donate.$project.org shouldn't a problem as those are covered by wildcard certs. The problem is the extra www. prefixed variants. I certainly hope we'll keep the donate.$project.org, even as mere redirects.

In T102827#1428781, @Krinkle wrote:

Seems unlikely indeed, though it's not unlikely people may mistype donate.wikimedia.org for donate.mediawiki.org or donate.wikipedia.org. And donate.$project.org shouldn't a problem as those are covered by wildcard certs. The problem is the extra www. prefixed variants. I certainly hope we'll keep the donate.$project.org, even as mere redirects.

I respectfully disagree :) I think this is the kind of logic that has resulted in us hoarding domains and redirects for every conceivable user error, increasing our complexity for very little benefit. I sincerely doubt many people type URLs at the address bar nowadays and for those who do (like you and me, I guess!), I'm sure they'll be able to handle their browser's error page and take corrective action, rather than say "oh snap, I guess I'm not going to donate after all".

That's a matter of my guess and our opinion, though, so I thought it would help to have some data. I ran a grep over oxygen's sampled 1:1000 logs for the past month — the data has some flaws (sampling was probably broken in some cases, and we have at least a large gap of a few days, so numbers are only meaningful as relative to each other, if at all):

faidon@oxygen:~$ awk '{ sum += $1 } END { print sum }' per-domain-count
588179523
faidon@oxygen:~$ grep donate per-domain-count
   7939 "donate.wikimedia.org"
    754 "donate.wikimedia.org:80"
      8 "donate.wikipedia.org"
      3 "donate.wikiquote.com"
      3 "donate.wikimediafoundation.org"
      3 "donate.mediawiki.org"
      2 "donate.wikiquote.org"
      1 "donate.wiktionary.org"
      1 "donate.wikiversity.com"
      1 "donate.wikimediafoundation.com"
      1 "donate.wikimedia.com"
      1 "donate.wikibooks.org"
      1 "donate.wikibooks.com"
faidon@oxygen:~$ head -5 per-domain-count
270110534 "upload.wikimedia.org"
98696288 "en.wikipedia.org"
18542227 "ru.wikipedia.org"
18277072 "meta.wikimedia.org"
17523536 "es.wikipedia.org"
faidon@oxygen:~$ tail -5 per-domain-count
      1 "033153132050"
      1 "031563056612"
      1 "0301.0244.0205.0110:443"
      1 "012565462272"
      1 "000298-1.l.windowsupdate.com"

Change 223044 had a related patch set uploaded (by BBlack):
Remove (www\.)?donate from most domains

https://gerrit.wikimedia.org/r/223044

^ Patch above is a cleanup on this based on stats/arguments above. Seems like a near-zero-impact change to me, and clears up a lot of TLS problems for us in the process.

Change 223044 merged by Faidon Liambotis:
Remove (www\.)?donate from most domains

https://gerrit.wikimedia.org/r/223044

Change 222876 abandoned by Chmarkine:
Remove www.donate.wikimediafoundation.org from DNS

Reason:
Done in If010437c

https://gerrit.wikimedia.org/r/222876

Change 222877 abandoned by Chmarkine:
Remove www.donate.mediawiki.org from DNS

https://gerrit.wikimedia.org/r/222877

Change 222880 abandoned by Chmarkine:
Remove www.donate.wiktionary.org from DNS

https://gerrit.wikimedia.org/r/222880

Change 222883 abandoned by Chmarkine:
Remove www.donate.wikipedia.org from DNS

https://gerrit.wikimedia.org/r/222883

Change 223048 had a related patch set uploaded (by BBlack):
Remove dead donate hostnames from redirects

https://gerrit.wikimedia.org/r/223048

Change 223048 merged by Faidon Liambotis:
Remove dead donate hostnames from redirects

https://gerrit.wikimedia.org/r/223048

Actually http://www.email.donate.wikimedia.org/ can be removed too.

Change 223245 had a related patch set uploaded (by Chmarkine):
Remove www.email.donate.wikimedia.org from DNS

https://gerrit.wikimedia.org/r/223245

I'm fine with dropping the www.* subdomains if they're going to cause extra work/cost. Couldn't find anywhere that they are linked to.

Please do keep the project domains as redirects though unless there's a compelling reason not to. Especially donate.wikipedia.org, that's a common (and understandable) mistake. I've even seen WMF staff directing people to that domain.

@CCogdill_WMF knows more about our various email domains, but maybe we should split that out to a separate task if it's more complicated.

@Chmarkine, @BBlack, @faidon - can one of you give me a summary of the domains you're proposing we delete? I'm not totally clear if it's just www.* domains at this moment or others as well.

If @MeganHernandez_WMF and @Pcoombe see no reason to keep the www.* subdomains, I don't see a need for them in order to send email either.

The only one(s) I'd specifically ask we keep around if possible are any containing "email.donate", including the "www.email.donate" subdomain. The reason behind all these DNS entries is different email providers will check different domains based on domains within the email headers. We suspect it's possible that certain email providers check that domain to see if it’s valid, and that factors into Inbox placement.

In T102827#1452705, @CCogdill_WMF wrote:

The only one(s) I'd specifically ask we keep around if possible are any containing "email.donate", including the "www.email.donate" subdomain. The reason behind all these DNS entries is different email providers will check different domains based on domains within the email headers. We suspect it's possible that certain email providers check that domain to see if it’s valid, and that factors into Inbox placement.

Do you have more information on this? What exactly do they check? The existence of the name in DNS? What headers? What do we put into those headers? How do they end up with these domains?

These are the recommend settings from our email service provider (IBM).
When it comes to deliverability, we have to take every possible approach as
there are millions (tens of millions?) of email services in the world, so
we can’t know exactly what each and every one is doing (and they are all
doing thing differently).

Do you have a link to those recommendations?

@Pcoombe @CCogdill_WMF - We've already dealt with the generic/per-project issues in https://gerrit.wikimedia.org/r/#/c/223044 back on July 6. The only concern left here is the "email.donate.wikimedia.org" case, which is all of these interrelated records:

wikimedia.org:spop1024._domainkey.email.donate 1H IN TXT  "k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCnPE/TvQbvCk0FbqekfcpB7R6R9yXFulninKCCagEijzf/55WqL3o9mNMddmmEAIQvhlscYqtrEerd2RYQTo0pNjJQAXIddvhUaJo+r66TYoyTSoxtL6zF7oJ9ZOL9i7PIntN7zq7m448S/3llWKK8MDvkeG+f5MN/qZ+P2wyD+QIDAQAB"
wikimedia.org:bounce.email.donate     1H  IN A    74.121.50.42
wikimedia.org:bounce.email.donate     1H  IN MX   5 bounce.email.donate
wikimedia.org:bounce.email.donate     1H  IN TXT  "v=spf1 ip4:74.121.51.111 ip4:208.80.155.11 -all"
wikimedia.org:email.donate            600 IN DYNA geoip!text-addrs-v4/eqiad
wikimedia.org:email.donate            1H  IN MX   10 reply.email.donate
wikimedia.org:email.donate            1H  IN MX   20 mail3880.email.donate
wikimedia.org:email.donate            1H  IN TXT  "v=spf1 ip4:74.121.51.111 ip4:208.80.155.11 -all"
wikimedia.org:mail3880.email.donate   1H  IN A    74.121.51.111
wikimedia.org:mail3880.email.donate   1H  IN MX   5 mail3880.email.donate
wikimedia.org:mail3880.email.donate   1H  IN TXT  "v=spf1 ip4:74.121.51.111 ip4:208.80.155.11 -all"
wikimedia.org:reply.email.donate      1H  IN A    74.121.50.42
wikimedia.org:reply.email.donate      1H  IN MX   5 reply.email.donate
wikimedia.org:reply.email.donate      1H  IN TXT  "v=spf1 ip4:74.121.51.111 ip4:208.80.155.11 -all"
wikimedia.org:links.email.donate      1H  IN CNAME recp.mkt41.net.
wikimedia.org:open.email.donate       1H  IN CNAME open.mkt41.net.
wikimedia.org:www.email.donate        1H  IN CNAME wikimedia.org.

The primary issue here is that if any of these are used for HTTP traffic, as in emails end up containing http://www.email.donate.wikimedia.org (or similar for the rest of the names above), then that blocks us from preloading HSTS on wikimedia.org as a whole in the long run. If we turned on HSTS for wikimedia.org without solving this problem first, then users who had ever visited a primary wikimedia.org site (and thus picked up HSTS for wikimedia.org) would automatically re-interpret those links as https://..., which would break them all.

Not a single one of those hostnames responds to the HTTPS protocol legitimately, and that's not an easily fixable thing. Only www.email.donate and email.donate resolve to our own servers where we could theoretically fix the situation, but we'd still prefer to rewrite those hostnames as e.g. "www-email-donate" and "email-donate" if possible, or some other such scheme. Alternatively, we could purchase a certificate for "*.email.donate.wikimedia.org", but again that would only fix the ones that map to our own servers. All of the others map to server IPs or hostnames owned by Silverpop/IBM, so they wouldn't have legitimate certificates on our behalf in the first place regardless of the multi-subdomain thing, and again they don't even have HTTPS listeners so I doubt they're capable.

Stepping back a bit from how this blocks our HSTS preloading for the rest of wikimedia.org: donations and fundraising are probably one of the most sensitive areas of any of our traffic. In general, I'd suspect we'd want solutions for this that protect users' privacy to the degree possible, at least by not sending cleartext HTTP traffic in response to them opening, clicking around in, or replying to emails from us on this subject. It's also a questionable experience from the user's point of view when they click on a link supposedly from our donations team and get sent to a suspicious-sounding third party like "mkt41.net". This is referenced also in the two tickets @JanZerebecki linked earlier: T74514 and T60373 ...

Thanks for this explanation, @BBlack. I got a little further clarification from @faidon so I think I understand how I need to follow up with IBM/Silverpop. I will update this ticket once I've been able to get more information on why these domains are http right now. We've been told in the past it isn't possible to track our (very valuable) open and click data with https in the domain, but I'm not sure why that is, so I will work on this with them.

Update from IBM/Silverpop: it turns out there is a way for us to use https without losing all of our click data in emails. The tradeoff is we lose the ability to customize our domain in those links so the domain will be "links.mkt4988.com" instead of "links.wikimedia.org".

We are going to put in a support case with Silverpop to change our link domain so that we can use https. This is not a switch we can flip ourselves, so I will update this task once Silverpop confirms its done.

@BBlack I wanted to make sure you saw this as it relates to T107059. I don't have an ETA yet but expect this should be done within a few weeks.

Confirming that IBM updated our links so that all Silveprop links will be
https from now on. Can we close this task?

I think the problems with email.donate run deeper than anything IBM could fix without changes on our side as well. We should start with an evaluation of what all of the related hostnames are even used for in what ways, especially which ones are used for HTTP(S) links in emails.

Currently there seem to be multiple potential problems, but without that information it's hard to even categorize the solutions for them:

1. email.donate - A-record to our production text clusters, but our wildcard certificate does not cover this domainname because it's a multi-level subdomain (TLS wildcards only cover a single level of hostname). Works over HTTP (just goes to default portal for an unconfigured HTTP domain), but not HTTPS (cert mismatch).
2. bounce.email.donate - A-record to 3rd party, serves HTTP (unknown what content is supposed to be there, the root path gives 403 Forbidden) but not HTTPS at all (connection refused).
3. mail3880.email.donate - A-record to 3rd party, no HTTP or HTTPS service on it at all AFAICS.
4. reply.email.donate - same A-record as bounce.email.donate, so same basic situation.
5. links.email.donate - CNAME to 3rd party. Serves both HTTP and HTTPS already, but doesn't have a correct certificate for links.email.donate.wm.o.
6. open.email.donate - CNAME to 3rd party. Serves HTTP with a page containing a (rather ironic, in light of all of this) statement about how privacy is important. No HTTPS service (connection refused).
7. www.email.donate - Basically identical to email.donate situation at the top.

If the solution is to convert all of the links to foo.mkt4988.com, I think that's still a pretty poor solution (it makes the whole email look like a scam or spam attempt since the linked domain has no branding or legitimacy and has nothing to do with us), and regardless we still need to get rid of the problem service hostnames in our zone data before we close this and move forward.

IBM tells us it's not possible to have a customized domain with an active ssl cert in place; they aren't able provide that for each client, apparently. That's why we pushed back on this task initially, but it seemed like we would have to change one way or another. I agree this doesn't look good from a branding perspective, and we were optimistically hoping there would be a tradeoff in offering an https-only experience to those of our donors that care.

We only do link tracking on the email.donate subdomain, so that's the one we changed. We should be able to change all the others to the mkt4988.com domain, but it will definitely impact deliverability. I'm not sure we have any alternative.

After talking about this more with our email consultants, I don't think we have any other option but to change all the Silverpop subdomains for bounce, open, and click tracking to the mtk4988 domain. The good news is they'll give us a double-nested sub-domain like the one now being used for click tracking (links.wikimedia.mtk4988.com), so there is some customization.

We're going to put in the request with Silverpop to change the other domains on their side today. There is an expected impact to deliverability here, but we can't estimate how much until the change happens. If the effect is drastic, I will update here.

links.wikimedia.mtk4988.com is a really bad solution. As already pointed by BBlack, it looks like a phishing attempt (and for a donation campaign, none the less!). Why can't directly point to WMF servers? (it can keep those silverpop ids and provide them to IBM if that's really needed -which is probably arguable-)

See also https://pt.wikipedia.org/wiki/Wikip%C3%A9dia:Pedidos/Outros#Den.C3.BAncia_de_scam.2Fphishing and today's #wikimedia log (starting at 2015-08-19 22:40 UTC), for a real case of user reporting it to us as phishing against our brand.

I agree this is not a great solution, and have made that argument in the past. However, changing the domain was the only way we could continue our email program and support https. IBM needs to be able to track the various levels of email behavior data, from bounces to clickthroughs, which is not a capability we have with in-house tools.

If Fundraising doesn't have to make this change right now, I would very much like to change the links back to http through our wikimedia domain, but I don't think that's an option. If there's a solution I don't know of which allows us to send and track email and do it under the wikimedia.org domain without breaking HSTS, let me know!

Thanks for pointing me to the comment on Wikipedia. I've followed up there, and Fundraising will continue to collect our own data on user response to this change.

In T102827#1555784, @CCogdill_WMF wrote:

I agree this is not a great solution, and have made that argument in the past. However, changing the domain was the only way we could continue our email program and support https. IBM needs to be able to track the various levels of email behavior data, from bounces to clickthroughs, which is not a capability we have with in-house tools.

Is there software we could deploy in-house to do this, without sending that private data off to a third party and/or running into all kinds of legitimacy / privacy / TLS issues? Looking shallowly around in google search results, I stumbled on https://github.com/mautic/mautic and https://github.com/pimcore/pimcore . There are bound to be others, as these aren't fundamentally difficult problems to solve on a code level.

If Fundraising doesn't have to make this change right now, I would very much like to change the links back to http through our wikimedia domain, but I don't think that's an option. If there's a solution I don't know of which allows us to send and track email and do it under the wikimedia.org domain without breaking HSTS, let me know!

It is still an option. Nothing technical about the email.donate situation has been changed recently on ops' end of things. HSTS includeSub/preload for wikimedia.org is still blocked/broken regardless of this issue, and will be until the whole benefactorevents situation is sorted out at some later date this year (hopefully), so it's not a big change in the status of the situation to roll back to where we were on the issues in this ticket and re-evaluate options.

As for solutions: basically we can either do it similarly to the good parts of the benefactorevents situation (3rd party, but with us sending them certs to use TLS for everything, and no multi-level subdomains, and all of those services HTTPS-only), or we can bring it in-house. Even a well-configured 3rd party solution could run into HPKP-related issues with us down the road. Exactly what our HPKP policies will become is still an unknown at this time, though, so it's not as urgent.

Personally, I think we're better off bringing all of these things in-house. All of this data related to fundraising emails (the emails themselves, the responses, open/click/link/bounce data, etc) is privacy-sensitive and branding/legitimacy-sensitive. While I understand FR signs contracts related to privacy with vendors, contracts do not actually prevent breaches, leaks, or misappropriations - they merely allow the assigning of blame after an incident has occurred, assuming we're ever even aware of it. As a general rule, the entire industry of email marketing (which these 3rd parties are necessarily a part of) is in the business of gathering, selling, and generally dealing loosely with private customer data compared to how we treat it, so it's hard to trust that they're going to take the utmost concern with ours as a special case just because we're a different kind of organization.

On a separate level from these privacy issues, I tend to be opposed to us unnecessarily outsourcing any functionality to commercial 3rd party services because it breaches our commitment to Open Source software as detailed here: https://wikimediafoundation.org/wiki/Resolution:Wikimedia_Foundation_Guiding_Principles#Freedom_and_open_source . Outsourcing to a commercial entity using closed-source software on our behalf is at least equivalent to, if not worse than, using closed-source software for said functionality internally on our own servers. It shouldn't become normal for us to to evade the spirit of that Principle through outsourcing.

I also tend to think that, in general, we're such a sufficiently prominent and unique organization on the Internet and in the world that we'll virtually always want more customization and self-determination in our services than a commodity 3rd party can provide.

I looked into Pimcore, Mautic, and OpenEMM. All documentation I can find on these tools suggests they are meant for very small databases. For example, from OpenEMM's FAQ[1]:

If you want to work with more than 120,000 addresses in your database, you have to change the property import.maxrows in file emm.properties in directory /home/openemm/webapps/core/WEB-INF/classes accordingly. However, the bigger your database, the more the performance of your OpenEMM installation will suffer.

OpenEMM’s quote does a good job of illustrating why we don’t have an intention to switch email clients: we’re currently using the most robust system there is, partly because very few email systems are designed to handle databases with over 1 million contacts — outside of the many other sophisticated things Silverpop allows us to do. More importantly, in the event the system breaks, who will fix it? The Fundraising tech team has more than enough on their plate, and we can’t afford to lose our email program for 3 months because it’s broken and no one has time to fix it. Silverpop has never gone down for me in 3 cumulative years of use, and they continue to improve it. Our team could not provide me with that same level of consistency and support.

Setting up our own email system would require ~5 new FTEs - a full-time DBA to focus specifically on email database production, a deliverability expert to handle bounce authentication, and more engineering/analytics staff to support the platform. It does not seem justifiable to spend hundreds of thousands of donor dollars to build a tool that already exists and likely functions better than what we could afford to build.

Our two departments have a time scheduled to discuss third party usage, so I will leave that discussion for the meeting.

[1]https://www.agnitas.de/en/e-marketing_manager/email-marketing-software-variants/openemm/openemm-support/open-emm-faqs/

Note this should get resolved via T130414 's https://gerrit.wikimedia.org/r/#/c/278353

Change 284106 had a related patch set uploaded (by BBlack):
Common VCL: remove wikimedia.org subdomain HTTPS redirect exception

https://gerrit.wikimedia.org/r/284106

Change 284106 merged by BBlack:
Common VCL: remove wikimedia.org subdomain HTTPS redirect exception

https://gerrit.wikimedia.org/r/284106

Change 223245 abandoned by Faidon Liambotis:
Remove www.email.donate.wikimedia.org from DNS

Reason:
A variant of this has been already applied for a while, see Ie758bd4af13604f6966a6abab3c2e3eff86db167

https://gerrit.wikimedia.org/r/223245