Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempt to fix VM llvm-project submodule contamination #1605

Merged
merged 2 commits into from
Feb 3, 2021

Conversation

StephanTLavavej
Copy link
Member

@StephanTLavavej StephanTLavavej commented Feb 2, 2021

We're seeing test runs fail due to apparent submodule contamination. For example, #1604 didn't change llvm-project at all, but failed with:

1: Test command: "C:\Program Files\Python39\python.exe" "D:/build/tests/utils/stl-lit/stl-lit.py" "--timeout=240" "--shuffle" "-j30" "--xunit-xml-output=D:\build/test-results.xml" "--num-shards=8" "--run-shard=1" "C:/a/1/s/llvm-project/libcxx/test/std" "C:/a/1/s/tests/std/tests" "C:/a/1/s/tests/tr1/tests"
1: Test timeout computed to be: 10000000
1: stl-lit.py: C:\a\1\s\llvm-project\llvm\utils\lit\lit\TestingConfig.py:99: fatal: unable to parse config file 'D:\\build\\tests\\libcxx\\lit.site.cfg', traceback: Traceback (most recent call last):
1:   File "C:/a/1/s/llvm-project/llvm\utils\lit\lit\TestingConfig.py", line 88, in load_from_path
1:     exec(compile(data, path, 'exec'), cfg_globals, None)
1:   File "D:\build\tests\libcxx\lit.site.cfg", line 37, in <module>
1:     stl.test.params.getDefaultParameters(config, lit_config),
1:   File "C:\a\1\s\tests\utils\stl\test\params.py", line 13, in getDefaultParameters
1:     Parameter(name='long_tests', choices=[True, False], type=bool, default=True,
1: TypeError: __init__() got an unexpected keyword argument 'feature'
1: 
1/1 Test #1: stl ..............................***Failed    0.14 sec

0% tests passed, 1 tests failed out of 1

We believe this is because #1594, which did change llvm-project, ran on the same VM. (Timeline: #1594 ran build 20210201.6 starting at 3:53 PM Pacific Time, followed by #1604 running build 20210202.1 starting at 4:27 PM. I can't find enough timestamps to see exactly when the first build finished, given our 10 minute idle VM cleanup setting, but this indeed looks like a contamination scenario.)

After investigating, we determined that the cause was:

git fetch --filter=tree:0 --depth=1 llvm $(${{ parameters.llvmSHAVar }})
git reset --quiet FETCH_HEAD
git sparse-checkout init --cone
git sparse-checkout set libcxx/test libcxx/utils/libcxx llvm/utils/lit

These commands will perform a sparse checkout of a given SHA - but if they're run again, on the same VM but asked to check out a different SHA, they won't update the llvm-project working tree. The result is running new STL bits against old llvm-project bits, or vice versa. (In this case, the "unexpected keyword argument" was caused by a change in llvm-project with a corresponding change in the STL.)

(We saw this after merging #1505, but the issue was introduced in #1513 which was merged earlier.)

After reproing this locally, I believe that a reasonable fix is:

  • We don't need to perform a mixed reset between fetch and setting up the sparse-checkout.
  • After setting up the sparse-checkout, git reset --hard will set the working tree to the sparse set of files that we want, regardless of whether this is the first run, a later run with the same SHA, or a later run with a different SHA.
  • For good measure, let's git clean. This will remove any files left over from previous builds, although they weren't responsible for today's problem.

I've locally verified that this is capable of switching SHAs, and that it doesn't result in a full checkout.

@StephanTLavavej StephanTLavavej added the infrastructure Related to repository automation label Feb 2, 2021
@StephanTLavavej StephanTLavavej requested a review from a team as a code owner February 2, 2021 01:50
@StephanTLavavej

This comment has been minimized.

@azure-pipelines

This comment has been minimized.

azure-devops/checkout-sources.yml Outdated Show resolved Hide resolved
Co-authored-by: Curtis J Bezault <[email protected]>
@StephanTLavavej

This comment has been minimized.

@azure-pipelines

This comment has been minimized.

@StephanTLavavej StephanTLavavej dismissed cbezault’s stale review February 3, 2021 01:46

Added --quiet as requested.

@StephanTLavavej StephanTLavavej self-assigned this Feb 3, 2021
@StephanTLavavej StephanTLavavej merged commit d936bd8 into microsoft:main Feb 3, 2021
@StephanTLavavej StephanTLavavej deleted the clean_submodule branch February 3, 2021 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
infrastructure Related to repository automation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants