[go: up one dir, main page]

Page MenuHomePhabricator

Run a synthetic test for client side preferences
Closed, ResolvedPublic3 Estimated Story Points

Description

In https://gerrit.wikimedia.org/r/c/mediawiki/core/+/932819 we plan to make a change to how we handle client side preferences. The significant changes are switching from storing these in cookies to storing them in localStorage and storing multiple values in a JSON string that must be parsed,

Previously, in T327798#8554853, we profiled localStorage to get a rough idea and found that it was 1.7ms longer but still under the 50ms cut-off to be considered a long task however we didn't profile the use of JSON.parse. We'd like to get more confidence going into the change.

To do this, we will compare the current version of the inline script VS the new script.

Two static HTML files are provided in https://gist.github.com/jdlrobson/407cde7f0b894c23244fda83581bee3e that simulate https://en.wikipedia.org/wiki/Brazil, with the following modifications:

TODO

Hypothesis to prove/disprove: Parsing JSON stored in localStorage at this point does not introduce any long tasks and has more or less identical performance.

Method

  • Prep the test for version 1 vs version 2 - make sure the following code is run in the environment that will run the test and disable the:
mw.cookie.set('mwclientprefs', 'vector-feature-limited-width');
localStorage.setItem('mw-client-preferences', JSON.stringify( { 'vector-feature-limited-width' : 1 } ) );
  • Locally compare HTML version 1 VS HTML version 2
  • Analyse the differences in performance that you are seeing and document in the ticket

Stretch goal:

sign off steps

  • Evaluate the analysis work.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Jdlrobson raised the priority of this task from Medium to High.May 16 2023, 5:09 PM
LGoto set the point value for this task to 3.May 18 2023, 5:11 PM
LGoto changed the task status from Open to Stalled.May 22 2023, 5:28 PM
Jdlrobson changed the task status from Stalled to Open.May 25 2023, 4:58 PM
Jdlrobson lowered the priority of this task from High to Medium.

Moving to next sprint and unblocking.

To the person who is assigned this ticket - Please note that I wrote step-by-step instructions for how I've been adding synthetic tests in my offboarding document at https://docs.google.com/document/d/13NypRIkc3-jOFNR6wTOnMfUXMjL11ny5OiyX4XulDK4/edit#heading=h.a26i7l3crf5f under the heading "Comparative synthetic tests" that I hope you will find helpful

Jdlrobson renamed this task from Run synthetic tests for Vector 2022 with Codex to Run a synthetic test for client side preferences.Jul 5 2023, 4:27 PM
Jdlrobson updated the task description. (Show Details)
Jdlrobson removed a subscriber: nray.

Change 935873 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[mediawiki/core@master] [playground] Run a synthetic test for client side preferences

https://gerrit.wikimedia.org/r/935873

The patch here is just for quick local test, and for feedback on the test code itself before starting to move into synthetic performance tests

local results:

Json parse and loop: 0.10000002384185791ms
Local storage loop: 0.19999998807907104ms

it varies from run to run, sometimes the Local storage loop is faster with the same code.

regardless for now it shows both are a fraction of a millisecond, and both will not effect general performance. but we will continue with the more controlled synthetic tests to have a better idea which is more stable and performant, and also to gain the experience of running these kind of tests.

Capturing what I said in the standup this morning I am a little confused by the existence of the patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/935873/ so it would be helpful to have some context with your intentions around that patch.

My expectations with the work needed for this task are as follows:

  1. Finalize the solution in T339268.
  2. Upload current HTML vs new proposed HTML to mediawiki-config (example: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/905769)
  3. Add some tests in performance/synthetic-monitoring-tests to compare the 2 (example: https://gerrit.wikimedia.org/r/c/performance/synthetic-monitoring-tests/+/905777)

For the above you'll likely need to add a class to the HTML element e.g. synthetic-test-0 and set client preferences on the device `localStorage.setItem( 'mw-clientprefs', '{\"synthetic-test\":1}' ) - the end result should switch the body class from synthetic-test-0 to synthetic-test-1.

(We can talk tomorrow in our 1 on 1 if anything is misaligned between us around what we want to do here)

Change 935873 abandoned by Mabualruz:

[mediawiki/core@master] [playground] Run a synthetic test for client side preferences

Reason:

https://gerrit.wikimedia.org/r/935873

Change 937092 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[operations/mediawiki-config@master] Run a synthetic test for client side preferences

https://gerrit.wikimedia.org/r/937092

Capturing what I said in the standup this morning I am a little confused by the existence of the patch https://gerrit.wikimedia.org/r/c/mediawiki/core/+/935873/ so it would be helpful to have some context with your intentions around that patch.

My expectations with the work needed for this task are as follows:

  1. Finalize the solution in T339268.
  2. Upload current HTML vs new proposed HTML to mediawiki-config (example: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/905769)
  3. Add some tests in performance/synthetic-monitoring-tests to compare the 2 (example: https://gerrit.wikimedia.org/r/c/performance/synthetic-monitoring-tests/+/905777)

For the above you'll likely need to add a class to the HTML element e.g. synthetic-test-0 and set client preferences on the device `localStorage.setItem( 'mw-clientprefs', '{\"synthetic-test\":1}' ) - the end result should switch the body class from synthetic-test-0 to synthetic-test-1.

(We can talk tomorrow in our 1 on 1 if anything is misaligned between us around what we want to do here)

No I know the steps, I just wanted to make sure the code we testing is showcased with the 1st patch, nvm I will submit normal patches asap

Change 936818 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[performance/synthetic-monitoring-tests@master] Run a synthetic test for client side preferences

https://gerrit.wikimedia.org/r/936818

For the above you'll likely need to add a class to the HTML element e.g. synthetic-test-0 and set client preferences on the device `localStorage.setItem( 'mw-clientprefs', '{\"synthetic-test\":1}' ) - the end result should switch the body class from synthetic-test-0 to synthetic-test-1.

Changing any class names in the test, will not help us compare between using 1 object or multi key value inside local storage

That's fine. The scope of this task is just to confirm using JSON.parse does not cause a considerable performance degradation

Change 936818 abandoned by Mabualruz:

[performance/synthetic-monitoring-tests@master] Run a synthetic test for client side preferences

Reason:

https://gerrit.wikimedia.org/r/936818

Change 937092 abandoned by Mabualruz:

[operations/mediawiki-config@master] Run a synthetic test for client side preferences

Reason:

https://gerrit.wikimedia.org/r/937092

Change 939312 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[operations/mediawiki-config@master] Run a synthetic test for client side preferences

https://gerrit.wikimedia.org/r/939312

Change 939323 had a related patch set uploaded (by Mabualruz; author: Mabualruz):

[performance/synthetic-monitoring-tests@master] Run a synthetic test for client side preferences

https://gerrit.wikimedia.org/r/939323

  • Seems performance numbers are around the same for both.
  • The local storage approach seems to be more stable on benchmarks on my machine.
  • The Cookie can get faster and slower than the local storage.

Max diffs between the two approaches around 0.25s in anyway:

  • We can proceed with after confirmation on the result from another dev with the inline script. ( Blocking )
  • Try also try to confirm the results on our performance platform with the attached patches. ( Non-Blocking )

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/939312/

How to verify/test locally:

The local folder in the provided patch have 2 sub folders with index.html files

  • Open both files in a browser
  • Open development tools
  • Performance insights tab
  • Click Measure Page Load
  • Make sure you wait for each page to finish the benchmark before running the other one
  • you can look at the 1st print or any of the metrics in the results
  • Run them 2 - 3 each to make sure your machine’s limitations is understood ( like what are the ranges of times you get for each page )
  • Verify that the findings that is attached on the ticket are correct or wrong.

PS: you can download each run for sharing as you can upload the reports on any chrome based browser and look at them

We chatted earlier and it sounds like Jan was able to replicate these results. @Jdrewniak could you please summarize your findings here as well? Thanks in advance!

I've gotten the same results as @Mabualruz when profiling the test page locally.

With a 6x cpu slowdown and slow 3g, both approaches showed similar results, sometimes the localStorage was a bit faster:
cookies: Total blocking time - 1250.16ms
localStorage: Total blocking time - 1275.57ms
so a 25ms difference.

Thanks @Jdrewniak

We can unblock the workflow and move dependant tasks along for now, we can treat the server runs of synthetic tests as complimentary for now, and if we get contradicting results we can do the changes accordingly in retrospect.

Change 939312 merged by jenkins-bot:

[operations/mediawiki-config@master] Run a synthetic test for client side preferences

https://gerrit.wikimedia.org/r/939312

Mentioned in SAL (#wikimedia-operations) [2023-07-24T13:33:09Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:939312|Run a synthetic test for client side preferences (T336527 T339268)]]

Mentioned in SAL (#wikimedia-operations) [2023-07-24T13:34:36Z] <samtar@deploy1002> samtar and mabualruz: Backport for [[gerrit:939312|Run a synthetic test for client side preferences (T336527 T339268)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-07-24T13:49:39Z] <TheresNoTime> [[gerrit:939312]] not synced. T336527 T339268

Change 940912 had a related patch set uploaded (by Samtar; author: Samtar):

[operations/mediawiki-config@master] Revert "Revert "Run a synthetic test for client side preferences""

https://gerrit.wikimedia.org/r/940912

Change 940912 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert "Revert "Run a synthetic test for client side preferences""

https://gerrit.wikimedia.org/r/940912

Mentioned in SAL (#wikimedia-operations) [2023-07-24T14:09:24Z] <samtar@deploy1002> Started scap: Backport for [[gerrit:940912|Revert "Revert "Run a synthetic test for client side preferences"" (T336527 T339268)]]

Mentioned in SAL (#wikimedia-operations) [2023-07-24T14:11:03Z] <samtar@deploy1002> samtar: Backport for [[gerrit:940912|Revert "Revert "Run a synthetic test for client side preferences"" (T336527 T339268)]] synced to the testservers mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)

Mentioned in SAL (#wikimedia-operations) [2023-07-24T14:23:30Z] <samtar@deploy1002> Finished scap: Backport for [[gerrit:940912|Revert "Revert "Run a synthetic test for client side preferences"" (T336527 T339268)]] (duration: 14m 05s)

Change 939323 merged by jenkins-bot:

[performance/synthetic-monitoring-tests@master] Run a synthetic test for client side preferences

https://gerrit.wikimedia.org/r/939323

it's not clear to me what the findings were of the synthetic test. Did we get results? If so what were they?

Based on data from: https://grafana-rw.wikimedia.org/d/IvAfnmLMk/page-drilldown?orgId=1

Before localStorage and JSON.parse:

MetricMinMaxLastRange
Time to first byte (TTFB)68 ms72 ms71 ms4 ms
When something first was painted on the screen900 ms1 s1 s100 ms
When the largest heading was painted900 ms1 s1 s100 ms
When the largest image was painted900 ms1 s1 s100 ms
Last visual change on the page2.40 s2.70 s2.60 s300 ms
SpeedIndex920 ms1.02 s1.02 s104 ms
When all requests/responses are loaded (fullyLoaded)2.42 s2.70 s2.54 s276 ms
When the largest area on screen was painted (LCP)902 ms1.05 s1.05 s150 ms
The last CPU long task happened at1.32 s1.53 s1.53 s211 ms

After:

MetricMinMaxLastRange
Time to first byte (TTFB)70 ms72 ms70 ms2 ms
When something first was painted on the screen900 ms1 s1 s100 ms
When the largest heading was painted900 ms1 s1 s100 ms
When the largest image was painted900 ms1 s1 s100 ms
Last visual change on the page2.40 s2.60 s2.60 s200 ms
SpeedIndex919 ms1.02 s1.02 s103 ms
When all requests/responses are loaded (fullyLoaded)2.40 s2.65 s2.56 s251 ms
When the largest area on screen was painted (LCP)886 ms1.05 s1.05 s161 ms
The last CPU long task happened at1.29 s1.51 s1.51 s214 ms

From the Data mentioned I recon that the difference between using cookies and local storage with json parse is within margin of error, and either solution is as performant as the other.

The synthetic tests don't show much of a difference on our local machines, however after talking to @Krinkle about the loading characteristics of cookies vs. localStorage in the browser, we determined that this synthetic test might not be adequate in determining the performance impact of this potential change, so in regards to the hypothesis layed out in the description:

Hypothesis to prove/disprove: Parsing JSON stored in localStorage at this point does not introduce any long tasks and has more or less identical performance.

we agreed that we would have to test this approach in a real-world scenario, taking into account low-end devices, before moving forward with using localStorage during pageload for this inline script.