I'v started to prepare for this and a realistic time schedule is that I can demo on a device setup at my home but running the tests from a gui deployed somewhere else. I mostly did the work for this around Christmas but is some tuning that needs to be done, and realistic is to demo it in the end of March.

I've be tuning this for a while and I think this plan will work:

17:th October: deploy the setup on Wikicloud
24th October: let one of our synthetic test suite go through our new setup. That means for example all emulate mobiled tests use the new setup, so that the result of the tests are stored in a database. The rest will work exactly as before. By collecting data early from one type of tests, we will have data to verify that everything works ok.

Next step is to do the work comparing metrics. There's part in Fresnel and there some logic that the Mozilla team has created for their setup that we probably can re-use or use as a start. The idea is that with the compare functionality we can first compare two test (run two tests, compare them) and know if the metrics has changed. Also the next step is to turn that functionality on our synthetic testing so it will be an alternative to the alerts we have in Grafana today.

Lets aim for having that last functionality deployed the 27 of November. Then we we have December where everything can just run and I can verify it work and I can tune what's needed.

This could be some inspiration for comparing metrics: https://grafana.com/blog/2023/09/25/better-anomaly-detection-in-system-observability-and-performance-testing-with-grafana-k6/

Today I configured our direct tests to us the new ondemand test service. This gives me time to verify that it works and fix bugs and watch how it works. The plan is to keep that running during June and July and then have its prepared for mobile phone on demand testing.