Page is hidden at first causing a visibilitychange event during testing

[Generosus]

Good Day,

PSI (i.e., Lighthouse) yields inconsistent results every time we run it. Why? DebugBear and WebPageTest are consistent all the time.

Specifically:

(1) Sometimes, PSI detects Google third-party files such as googletagmanager.com, analytics.google.com, or ads.google.com although we have properly delayed those files. Sometimes, they are not detected - which is the expected result. Details: https://prnt.sc/lrtXbIhHQFLq

(2) Sometimes, PSI detects CSS files that we have properly omitted from loading via RUCSS or Async methods. Sometimes, they are not detected - which is the expected result. Details: https://prnt.sc/N3sekB7-epen

Noteworthy: The waterfall results displayed by WebPageTest are correct and consistent every time and serve as our indicator to determine our site's performance and payload. By comparison, the results displayed by PSI are not so we cannot rely on it anymore. We have cleared all cache and allowed our cache to build over a period 3-5 days to ensure all pages load the same way, yet we still get inconsistent results with PSI.

Bottom line, why is PSI so inconsistent in its results? Can you kindly look into this and fix this condition? What did we miss? 100% sure that if you run any site with PSI these days, you will notice the same behavior with those sites.

Help appreciated. Thank you!

[adamraine]

Just making sure I understand the problem correctly, when testing https://www.omniaaerospace.com/ this CSS file showing up in the results is unexpected:
https://www.omniaaerospace.com/wp-content/uploads/fusion-styles/faace9bba7c2d374a6155b18282a6305.min.css?ver=3.11.6

This is a weird one because I got a trace fro PSI and it looks like the CSS file is never fetched:
https://trace.cafe/t/uJpTrYJfHA

BUT it does appear in CSSUsage which is where the audit result is coming from. Seems like a bug that this stylesheet is being emitted by Chrome in the first place if there was not request for it.

[adamraine]

Looking at the source code of the page, the CSS file in question is always fetched 6 seconds after page load. My current hypothesis:

1. Lighthouse stops listening for network requests before the 6 seconds is up
2. Enough time passes for the 6 second timer to go off
3. Then CSSUsage.getArtifact is called which notices the new stylesheet

This appears to be backed up by the artifacts.Timing which shows CSSUsage.getArtifact called a full 3 seconds after DevtoolsLog.getArtifact:

{
  "name": "lh:driver:navigate",
  "entryType": "measure",
  "startTime": 3230.6000003814697,
  "duration": 3344.3999996185303
},
{
  "name": "lh:gather:getArtifact:DevtoolsLog",
  "entryType": "measure",
  "startTime": 7123.300000190735,
  "duration": 0.09999990463256836
},
{
  "name": "lh:gather:getArtifact:Trace",
  "entryType": "measure",
  "startTime": 7123.400000095367,
  "duration": 0
},
...
{
  "name": "lh:gather:getArtifact:CSSUsage",
  "entryType": "measure",
  "startTime": 10516.800000190735,
  "duration": 2011.4000000953674
},

Possible solution: Change these conditions to also exclude navigation mode.

[Generosus]

Hey Adam,

You are 100% correct. That seems to be the behavior. Sometimes the CSS file loads, sometimes is doesn't. The same occurs for Google Analytics' JS files, which we have delayed from loading. If you run a WebPageTest on our site, you'll notice the waterfall is clean (all the time).

What do you recommend as the next step? We are using the Avada theme, so many other Avada users (or the general public) could be affected by your finding.

Note: The above affects both Mobile and Desktop modes.

Cheers.

[adamraine]

What do you recommend as the next step? We are using the Avada theme, so many other Avada users (or the general public) could be affected by your finding.

I think this is just a bug in Lighthouse. Stylesheets fetched after Lighthouse determines the page is done loading should not show up in the results.

[Generosus]

Thank you. Agree.

What about the Google Analytics (third-party) JS files that are loading in the same fashion as the CSS file(s)?

Perhaps that needs to be looked at as well. At our end, those JS files have been delayed from loading but are still being picked up at random by Lighthouse.

If you re-run our site several times, you’ll clearly see that behavior.

Cheers!

[Generosus]

Update:

So we gave our code a second look (thanks to input received from the developers of Perfmatters), and noticed that we had set a long time ago a JS delay trigger of 6 seconds (in lieu of the plugin's default of 10 seconds). This -- compounded with the fact that Avada recently issued an update which increased the size of their JS and CSS files -- contributed to the issue reported above.

After re-setting our JS delay trigger to 10 seconds (default value) and running 10 PSI tests, we noticed the results became more consistent. HOWEVER, this did not fix 100% the condition we reported since we're still seeing those delayed files detected by PSI once in a while. Details: https://prnt.sc/DaH60xBmak7b

So, in conclusion, it appears that the inconsistencies are being created by both our JS delay trigger (< 10 seconds) and PSI (which may need to be fixed as you suggested above).

Hope this helps. Cheers.

[adamraine]

What about the Google Analytics (third-party) JS files that are loading in the same fashion as the CSS file(s)?

Perhaps that needs to be looked at as well. At our end, those JS files have been delayed from loading but are still being picked up at random by Lighthouse.

I was able to reproduce this, but something on the page might be causing the deferred resources to fetch early. Inspecting the trace, it looks like those resources are fetched well before 6s:

https://trace.cafe/t/cz6E35CIQH

It doesn't happen all the time like you said, for reference here is a trace where the deferred resources are not fetched:

https://trace.cafe/t/oxGGfnLkn9

[Generosus]

Thanks for confirming and the additional feedback. At this point, not sure what else we can do to help.

In addition to many other checks, we have extended our Delayed JS loading (timeout) to 12 seconds (from 6 originally) and are getting better results (i.e., more consistent), but the issue still occurs at random - as you confirmed. Frustrating of course since we won't know at what point we'll obtain reliable performance results using PSI or Lighthouse.

Again, thank you.

PS: For our learning, what does the tag "pending-close" mean in this case?

[adamraine]

PS: For our learning, what does the tag "pending-close" mean in this case?

For future reference, it means we will discuss closing the issue on the next bug triage.

---

I did find something that looks like a bug in either Lighthouse, Chrome, or PSI. On the provided page, a visibilitychange event will trigger the deferred scripts to be fetched. I think this is what we are seeing right now.

Taking a look at the trace I linked to above (https://trace.cafe/t/cz6E35CIQH), the page appears to be invisible until the first content is painted:

Compare this to a typical load where the page is visible but is just blank:

Seems like the page being initially hidden and then becomes visible later which triggers a visibilitychange event which happens to affect the page state in this case. This visibility change seems unexpected in PageSpeed insights.

FWIW I can reproduce this in a real world scenario by opening a new tab (which is hidden at first) and then clicking on the new tab to make it visible. The deferred gtag scripts and stuff are fetched immediately once I make the tab visible.

[Generosus]

Hey Adam,

Thank you so much for your quick, practical feedback and assistance. Yes, makes perfect sense.

If necessary, we'll be glad to volunteer our site for testing.

Looking forward to the fix.

Cheers!

[Generosus]

Good Day,

Any updates?

Thank you!

[Generosus]

Update:

We researched this topic further (i.e., conducted more testing) and can confirm the JS issue reported above is caused by a plugin we're using for loading Google Tag Manager. (Details) JS issue fixed.

The CSS issue, however, is still open. We have opened a ticket with the developers of Perfmatters to see if their plugin is causing the CSS issue reported above.

Will update this ticket upon receiving feedback from the developers of Perfmatters.

Cheers :)

[Generosus]

Update:

I can confirm the issue lies with PSI, not Perfmatters. We keep getting inconsistent CSS results. This issue is not a biggie, but it's frustrating because we cannot rely 100% on the scores provided by PSI -- as compared with DebugBear and WebPageTest.

Hopefully, you can address and fix this soon.

Thank you!