I always assumed that it was a slow network causing the issues, but I opened up the browser tools to check and it turns out the slowdown was happening while rendering the page, not when fetching external data. Goes to show it's a good idea to open up the browser tools once in a while to check out what's happening under the hood!