[test] Improve regression test suite debugging #20194
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
if they are in the blacklist they are not needed. The comment should explain the why not the what
eps1lon
commented
Mar 20, 2020
test/regressions/index.js
Outdated
| function excludeTest(suite, name) { | ||
| if (/^docs-premium-themes(.*)/.test(suite)) { | ||
| // eslint-disable-next-line no-console | ||
| console.log('ingoring premium themes pages'); |
There was a problem hiding this comment.
I know, I know this should return the reason and logging is a different concern but it's friday :)
Details of bundle changes.Comparing: 0b25383...6584a70 Details of page changes
|
|
Well it works if you include the bundle locally but vrtest does not display any console calls. |
|
Ok I honestly don't want to deal with |
| 'docs-getting-started-templates-album/Album.png', // Flaky image loading | ||
| 'docs-getting-started-templates-blog', // Not needed |
There was a problem hiding this comment.
// Not needed Maybe we should rename it // Nothing to test? Basically, these pages are supposed to yield close to no value or a low ROI.
There was a problem hiding this comment.
Why is there nothing to test? They're not empty as far as I know.
There was a problem hiding this comment.
In this case, I imagine the random images from unsplash make it flaky. So the "label" isn't correct. Happy to answer this question for any other test case if you have a doubt.
There was a problem hiding this comment.
Feel free to add any descriptive label you have. The current ones I removed where classic "comments for comments sake"-noise.
There was a problem hiding this comment.
Ok, cool, let's add more specific comments. They should add information. "No needed" is equivalent to having the test inside the blacklist array: noise.
There was a problem hiding this comment.
Perfect. That helps a lot. Is this good to merge in your opinion? Would be nice to get this in before #20187
vrtest ignores errors in index This reverts commit 3c86385.
eps1lon
force-pushed
the
test/fix-regression-tests-blacklist
branch
from
ec2b298 to
766b0ae
Compare
oliviertassinari
approved these changes
Mar 20, 2020
|
Merge after it is rebase with #20195 |
|
Don't need to rebase since we don't run |
eps1lon
commented
Mar 20, 2020
| @@ -42,18 +42,20 @@ const blacklist = [ | |||
| 'docs-components-badges/BadgeAlignment.png', // Redux isolation | |||
| 'docs-components-badges/BadgeVisibility.png', // Needs interaction | |||
| 'docs-components-breadcrumbs/ActiveLastBreadcrumb.png', // Redundant | |||
| 'docs-components-buttons/ButtonBases.png', // Flaky image loading | |||
EsoterikStare
pushed a commit
to EsoterikStare/material-ui
that referenced
this pull request
Mar 30, 2020
This was referenced Mar 31, 2020
This was referenced Apr 7, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The meat of this PR: b68aedd
The rest is refactoring and debugging features.
Makes the log more verbose but helps debugging which tests are run and which are excluded. It also warns if patterns are unused. This revealed: