I honestly love the idea of BBB being available for Wikimedians to use, but I have a number of gentle concerns about a volunteer effort to support such a platform.

The biggest concern I have initially is actually what access control can and will be used on the BBB install? We have other resources like etherpad and OSM tiles that are actively used by bad faith actors who are not contributing to the Wikimedia movement. These uses consume resources, but as far as I know do not cause any harm to actual humans. An open to the internet at large video chat platform will almost certainly become a resource drain. It may also become a platform of both random and targeted harassment if the maintainers are not prepared for those types of threats.

Looking at https://docs.bigbluebutton.org/greenlight/gl-overview.html I see mention of local auth, several OAuth2 providers, and LDAP for authentication. Local auth would be subject to the restrictions documented at https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#If_my_tools_have_account_creation.... LDAP authentication from inside Cloud VPS using Wikimedia Developer accounts (Wikitech/Gerrit/Cloud VPS) is specifically excluded by the same TOU. That leaves some OAuth2 integration, but it looks like that will require custom development by my reading of https://docs.bigbluebutton.org/greenlight/gl-config.html.

The test node in digital ocean required quite amount of memory.

Do you have data from that experiment? Two 36G instances is quite a large ask from the memory side of things. The standard Cloud VPS project quota is 16G. We do not oversubscribe RAM for our OpenStack instances, so this resource usage is "real" even if the instances themselves are mostly idle.

If it's too successful, we can migrate this to production.

That's an interesting claim. How will we measure "too successful"? How will this volunteer project get priority for Wikimedia production deployment if that threshold is hit?

How soon you are hoping this can be fulfilled: "As soon as possible" because of the pandemic and the lockdown.

This task missed the weekly review meeting for new project requests by ~4 hours so at the soonest you should expect next week. That is of course assuming that the other questions can be answered.

I am fairly sure this project also needs to ask for a public IP address. The info at https://docs.bigbluebutton.org/2.2/configure-firewall.html shows that in addition to HTTPS (which can be handled by the shared Cloud VPS proxy) BBB needs to accept UDP traffic on ports 16384 - 32768 for its RTP streams layer.

As to restricting access, what if we say that every Wikimedia chapter or usergroup has the right to use the tool, and can give access to host calls to 5 of its designated members? I could help administer a process for this on meta.

In T249159#6023545, @Pharos wrote:

As to restricting access, what if we say that every Wikimedia chapter or usergroup has the right to use the tool, and can give access to host calls to 5 of its designated members? I could help administer a process for this on meta.

I think for pilot purposes this is a good idea - give it to a circle of trusted folks first for the testing phase so it doesn't become a publicly exploitable service.

Thanks for notes @bd808. Some comments:

Looking at https://docs.bigbluebutton.org/greenlight/gl-overview.html I see mention of local auth, several OAuth2 providers, and LDAP for authentication. Local auth would be subject to the restrictions documented at https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#If_my_tools_have_account_creation.... LDAP authentication from inside Cloud VPS using Wikimedia Developer accounts (Wikitech/Gerrit/Cloud VPS) is specifically excluded by the same TOU. That leaves some OAuth2 integration, but it looks like that will require custom development by my reading of https://docs.bigbluebutton.org/greenlight/gl-config.html.

I couldn't find the part to configure OAuth2 for custom clients (like Wikimedia) OTOH, we might be able to use Google OAuth2. Does it comply with ToU? Also my idea for now is to have a dedicated LDAP server for this for now and then move to proper authentication once we hit the large enough scale. Something like this (having dedicated LDAP authentication) is already done in logstash-beta.wmflabs.org. I can even built a quick webserver to manage LDAP authentication.

Do you have data from that experiment?

Kevin knows better.

Two 36G instances is quite a large ask from the memory side of things.

My main idea for now is to have one bigram for the main thing + free space for another one or LDAP server and stuff like that. We might get away with one bigram + medium but don't be angry if people like it and I have to ask you for more :D

That's an interesting claim. How will we measure "too successful"? How will this volunteer project get priority for Wikimedia production deployment if that threshold is hit?

I genuinely don't know but I have seen several projects done by volunteers have been picked up by WMF and resourced properly once we end up being under-resourced.

I am fairly sure this project also needs to ask for a public IP address. The info at https://docs.bigbluebutton.org/2.2/configure-firewall.html shows that in addition to HTTPS (which can be handled by the shared Cloud VPS proxy) BBB needs to accept UDP traffic on ports 16384 - 32768 for its RTP streams layer.

Yeah I think so too. Thanks for this.

In T249159#6025021, @Ladsgroup wrote:

That's an interesting claim. How will we measure "too successful"? How will this volunteer project get priority for Wikimedia production deployment if that threshold is hit?

I genuinely don't know but I have seen several projects done by volunteers have been picked up by WMF and resourced properly once we end up being under-resourced.

The only tool I can think of that happening for was Magnus' WDQ which was not really taken into production so much as replaced by a different system. There might be something I missed in the last 7 years, but that is literally the only thing that I know of that was once a Cloud VPS project and now a production service. I suppose a similar claim could be made about map tiles, but that is actually also a parallel implementation rather than a promotion from Cloud VPS to prod.

I guess what I am trying to say here is that there is no clear path from an experiment to production for anything. And honestly this sounds like the sort of project that the end user community of Wikimedians will flock to. That's a good problem to have if you are ready for it, but as a side project it can be very disruptive for both the project maintainers and the service users.

I'm asking these questions out of a sense of self-preservation honestly. The WMCS folks are the ones who will be asked for help when and if this service fails. Taking the history of WDQ as an example, Yuvi ended up spending more than 3 months of nearly full time work just keeping it running once it became popular and Magnus hit his limit of being able to support it. I would like to know what the plan is so I can fall back on that rather than needing to make real-time decisions about the importance of BBB vs whatever else my team is handling.

In T249159#6025021, @Ladsgroup wrote:

I couldn't find the part to configure OAuth2 for custom clients (like Wikimedia) OTOH, we might be able to use Google OAuth2. Does it comply with ToU?

My reading of the docs at https://docs.bigbluebutton.org/greenlight/gl-config.html#user-authentication is that there is not a generic OAuth connector, so integration with SUL accounts would likely be custom code that you could potentially try to upstream to them.

Google OAuth2 would be fine by Cloud VPS TOU, but it also means that you are forcing folks into Google interactions to use to the service. That may or may not work with your target audience. I do know that Google OAuth2 has some restrictions on domain validation that will probably require you to ask WMCS folks for help if you go that direction (T226052).

Also my idea for now is to have a dedicated LDAP server for this for now and then move to proper authentication once we hit the large enough scale. Something like this (having dedicated LDAP authentication) is already done in logstash-beta.wmflabs.org. I can even built a quick webserver to manage LDAP authentication.

Adding an LDAP server to maintain doesn't sound like it would really add any value to your project. Just use the native local database auth system.

In T249159#6025021, @Ladsgroup wrote:

Do you have data from that experiment?

Kevin knows better.

Two 36G instances is quite a large ask from the memory side of things.

My main idea for now is to have one bigram for the main thing + free space for another one or LDAP server and stuff like that. We might get away with one bigram + medium but don't be angry if people like it and I have to ask you for more :D

Growing quota is easier than shrinking it. :) The upstream docs about AWS hosting suggest starting with a c5.xlarge instance. That's a 4 vCPU + 8GiB instance size and the equivalent of our m1.large instance flavor.

In T249159#6020745, @bd808 wrote:

Do you have data from that experiment?

I temporarily spun up a $160/mo instance with 16 GB memory, 8 dedicated CPUs, and 100 GB SSD. It was way overkill for our small experiment - double the minimum BBB recommends (4 CPUs, 8 GB memory).

We tested with 10-12 connections, 6-8 video streams.

• Average CPU usage: 12-13%
• Bandwidth Minimum:
  • Inbound 2 Mbps
  • Outbound 12 Mbps
• Bandwidth Maximum:
  • Inbound 3 Mbps
  • Outbound 40 Mbps

In general, I find this kind of projects related to what I understand as "Technological Sovereignty" to be very interesting. I want to believe this is a trend in other communities as well (see https://wiki.debian.org/Teams/DebianSocial for just an example).
From that point of view, I really want to see this project started :-)

That being said, I fully share the concerns detailed by @bd808. We already have several cases of successful experiments and community projects that have little to no maintenance. People expect us to maintain them, and we would like to, but we don't have the human power, but they are important to the community, but we don't have time, but yes, but no... but anxiety!

Anyway, my initial impulse is to let experiment, and don't shutdown legit project ideas out of fear even before they start. I would give my +1 to this project if you clearly state everywhere that this can be shutdown anytime if this goes south in any way (unmaintained, etc).

I guess I could ramble some more about this topic, but stopping now just to say:
<rainbows>I would really love to see some similar initiative happening with jitsi meet, matrix, mastodon and many other services</rainbows>

In T249159#6025818, @aborrero wrote:

In general, I find this kind of projects related to what I understand as "Technological Sovereignty" to be very interesting. I want to believe this is a trend in other communities as well (see https://wiki.debian.org/Teams/DebianSocial for just an example).

Exactly, and Wikimedia is a pioneer in "Technological Sovereignty", specially when it comes to privacy. We maintain our own CDN (which is quite a hassle I'd imagine), we don't use AWS, we don't outsource our media storage, I can go on for hours.

Also security engineers call Zoom a security disaster: https://www.theguardian.com/technology/2020/apr/02/zoom-technology-security-coronavirus-video-conferencing and https://blogs.harvard.edu/doc/2020/03/27/zoom/

I guess I could ramble some more about this topic, but stopping now just to say:
<rainbows>I would really love to see some similar initiative happening with jitsi meet, matrix, mastodon and many other services</rainbows>

I would love to see a mattermost installation inside WMF infra, that would be awesome. Or discord maybe?

For the record I'd be willing to lend a hand if needed.

I'm happy to try to help with traffic problems. Don't think we run many UDP services within Cloud VPS but it's certainly possible, a floating IP should do the trick if we need to expose it to the internet. Even if you have a floating IP it may still be possible to use the shared HTTPS proxy for HTTPS traffic, assuming you can configure different hostnames for HTTPS vs. UDP traffic as Bryan wrote.

Thank you for accepting this. I won't let you down. I created Wikimedia Meet to track the work. Once the horizon project is ready, I start working on it.

In T249159#6041151, @Ladsgroup wrote:

Thank you for accepting this. I won't let you down. I created Wikimedia Meet to track the work. Once the horizon project is ready, I start working on it.

Would "meet" be a better project name than "virtual-rendezvous"?

In T249159#6041156, @bd808 wrote:

In T249159#6041151, @Ladsgroup wrote:

Thank you for accepting this. I won't let you down. I created Wikimedia Meet to track the work. Once the horizon project is ready, I start working on it.

Would "meet" be a better project name than "virtual-rendezvous"?

For the URL that's for sure. Mostly because rendezvous is pretty hard to write (like rendezvous.wmflabs.org) so people suggested meet.wmflabs.org and I went with "Meet" for phabricator so outside users making tickets wouldn't be confused but if you want to create the project in the cloud and "meet" is okay for you. Please do that. Otherwise virtual-rendezvous is fine. If also "Meet" is unacceptable in phabaricator, I can change it.

In T249159#6041208, @Ladsgroup wrote:

For the URL that's for sure. Mostly because rendezvous is pretty hard to write (like rendezvous.wmflabs.org) so people suggested meet.wmflabs.org and I went with "Meet" for phabricator so outside users making tickets wouldn't be confused but if you want to create the project in the cloud and "meet" is okay for you. Please do that. Otherwise virtual-rendezvous is fine. If also "Meet" is unacceptable in phabaricator, I can change it.

"Meet" seems good to me for the project, the BBB public hostname, and the phab project. I agree on the spelling challenges with rendezvous. I also think it works out pretty well when a project has a name that is reflective of its purpose rather than the tech stack it is using at the moment. That kept me from suggesting "bbb" or something similar as the project name. I BOLD'ly updated the task description.

meet.wmflabs.org

We don't have support for non-wmflabs hostnames in the shared proxy yet, but if this project ends up needing to do its own TLS termination using meet.wmcloud.org as the hostname would be a bit more future proof. We have the ability to create DNS entries in the wmcloud.org TLD now and will probably be working on rolling out the use of that to replace wmflabs.org in the next couple of months. With project local LE cert management you could even do something like bbb.meet.wmcloud.org if you wanted, although https://meet.wmcloud.org is probably a nicer thing for folks to remember and type for the initial use-case.

Mentioned in SAL (#wikimedia-cloud) [2020-04-09T09:34:34Z] <arturo> create project with ladsgroup and kevinpayravi as projectadmins (T249159) with 1 floating IP

@Ladsgroup: The Phabricator tag Wikimedia Meet should be under Cloud-Services > Tools.

In T249159#6051135, @Aklapper wrote:

@Ladsgroup: The Phabricator tag Wikimedia Meet should be under Cloud-Services > Tools.

VPS-Projects is the umbrella for these.

@JJMC89: Whoops, you are right, thanks. Still no idea how to move to two sublevels deep (Cloud-Services > VPS-Projects) when looking at https://wikitech.wikimedia.org/wiki/Phabricator#Converting_a_parent_project_into_a_subproject , meh

In T249159#6052312, @Aklapper wrote:

@JJMC89: Whoops, you are right, thanks. Still no idea how to move to two sublevels deep (Cloud-Services > VPS-Projects) when looking at https://wikitech.wikimedia.org/wiki/Phabricator#Converting_a_parent_project_into_a_subproject , meh

@Aklapper worth double checking with @mmodell to confirm, but I think you would use the PHID of VPS-Projects as the --parent. I know that Mukunda made these kinds of project moves for me in the past (which I think is where that script may have started its life).

meet.wmcloud.org is now accessible. You can't join meeting until you enter a valid user and password. If you want it, let me know. (I'm working on improving authentication now)

In T249159#6067951, @Ladsgroup wrote:

meet.wmcloud.org is now accessible. You can't join meeting until you enter a valid user and password. If you want it, let me know. (I'm working on improving authentication now)

Cool! It looks like it is a hosted instance of Jitsi? The original task description talked about Big Blue Button so I'm curious about what swayed you towards Jitsi. (I have no in depth knowledge of or preference for one over the other.)

In T249159#6071413, @kostajh wrote:

In T249159#6067951, @Ladsgroup wrote:

meet.wmcloud.org is now accessible. You can't join meeting until you enter a valid user and password. If you want it, let me know. (I'm working on improving authentication now)

Cool! It looks like it is a hosted instance of Jitsi? The original task description talked about Big Blue Button so I'm curious about what swayed you towards Jitsi. (I have no in depth knowledge of or preference for one over the other.)

Yes, The reasoning was that BBB doesn't have out-of-the-box solution for debain (It's being tracked) and making BBB work for debian would take really long time and wouldn't worth it for something that we have to throw away and re-do every couple of months (as we should throw away VMs in the cloud once a couple of months to pick up new images). Jitsi OTOH, is waaay easier to set up and maintain (like out-of-the-box Let'sEncrypt support and so many other things, like Docker support). so I went with Jitsi for now, if it turns out that we really need some BBB features missing in jitsi (like breakout rooms), then I need to spend time to make it work (or we can wait until the issue I mentioned is resolved)

In T249159#6067951, @Ladsgroup wrote:

meet.wmcloud.org is now accessible. You can't join meeting until you enter a valid user and password. If you want it, let me know. (I'm working on improving authentication now)

I would love to use this. Documentation is not very clear yet, so my understanding is that we need to request a token directly to you? If that is the procedure, please give me a ticket, I can also help as ticket master if needed.

Thanks for this work, it is really amazing!

In T249159#6082365, @diego wrote:

In T249159#6067951, @Ladsgroup wrote:

meet.wmcloud.org is now accessible. You can't join meeting until you enter a valid user and password. If you want it, let me know. (I'm working on improving authentication now)

I would love to use this. Documentation is not very clear yet, so my understanding is that we need to request a token directly to you? If that is the procedure, please give me a ticket, I can also help as ticket master if needed.

Thanks for this work, it is really amazing!

I'm sorry about the documentation, feel free to improve any of those (or tell where it's unclear). I sent you an email so you would be a ticketmaster now but if there's anything else I can do to help. Let me know. Thanks for using it.

It says incorrect username or password when I would like to authenticate as host. Where do users register?

In T249159#6130849, @Gryllida wrote:

It says incorrect username or password when I would like to authenticate as host. Where do users register?

Hey, please see https://meta.wikimedia.org/wiki/Wikimedia_Meet