Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable parallel for small test coverage runs #19067

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

samford
Copy link
Member

@samford samford commented Jan 9, 2025

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same change?
  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your changes? Here's an example.
  • Have you successfully run brew style with your changes locally?
  • Have you successfully run brew typecheck with your changes locally?
  • Have you successfully run brew tests with your changes locally?

brew tests --coverage can fail to produce coverage information when run on a small number of tests (e.g., --only utils/curl). We use ParallelTests::last_process? in tests/spec_helper.rb to handle the SimpleCov output but due to the way the method is implemented, it doesn't work as expected if the number of processes is greater than one but lower than the number of cores.

For example, this is the current output for a small run (no coverage generated):

$ brew tests --only utils/curl --coverage
Randomized with seed 26057
1 process for 1 spec, ~ 1 spec per process
.........................................



Took 0 seconds

This is the output with this change (coverage generated as expected):

$ brew tests --only utils/curl --coverage
Randomized with seed 9280
Run options: exclude {:needs_linux=>true, :needs_network=>true, :needs_ci=>true, :needs_svn=>true}

Randomized with seed 9280
.........................................

Finished in 0.10351 seconds (files took 0.56166 seconds to load)
41 examples, 0 failures

Randomized with seed 9280

Coverage report generated for brew:80041 to /opt/homebrew/Library/Homebrew/test/coverage.
Line Coverage: 17.05% (8454 / 49580)
Branch Coverage: 2.32% (204 / 8797)
Coverage report generated for brew:80041 to /opt/homebrew/Library/Homebrew/test/coverage/coverage.xml. 8454 / 49580 LOC (17.05%) covered; 204 / 8797 BC (2.32%) covere

I tried to address this through changes to spec_helper.rb and/or changes to ParallelTests but didn't meet with any success.

This works around the issue by disabling parallel test execution when the --coverage option is used and the number of files to be tested is lower than the number of CPU cores. I've been using this workaround for months and it works as expected on my machine (M1 Pro and now M4 Pro).

`brew tests --coverage` can fail to produce coverage information when
run on a small number of tests (e.g., `--only utils/curl`). We use `ParallelTests::last_process?` in `tests/spec_helper.rb` to handle
the SimpleCov output but due to the way the method is implemented, it
doesn't work as expected if the number of processes is greater than
one but lower than the number of cores. I tried to address this
through changes to  `spec_helper.rb` and/or changes to `ParallelTests`
but didn't meet with any success.

This works around the issue by disabling parallel test execution when
the `--coverage` option is used and the number of files to be tested
is lower than the number of CPU cores. I've been using this workaround
for months and it works as expected on my machine.
@samford samford force-pushed the single-process-test-coverage branch from 4168558 to 42a2433 Compare January 9, 2025 20:59
Copy link
Member

@MikeMcQuaid MikeMcQuaid left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for explanation and PR! Have a proposed, simpler change.

@@ -75,7 +76,7 @@ def run
end
end

parallel = false if args.profile
parallel = false if args.profile || (args.coverage? && files.length < Hardware::CPU.cores)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
parallel = false if args.profile || (args.coverage? && files.length < Hardware::CPU.cores)
parallel = false if args.profile || args.only

would be simpler and I think would actually be pretty reasonable

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants