Remove node if VM doesn't exist even if computer is not idle #552
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When working with spot instance, when the agent is evicted/removed the node in jenkins is not removed because of this check which causes the build to hang.
If we move the idle check below the vm-exists check this solves the issue.
Testing done
Tested this change by adding more logging to see what was causing the issue of why the node wasn't being removed when the agent no longer existed.
The logging show it was being stop by the not idle check.
[FINE][com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs] Node cnp-jenkins-builderse8d030 is not idle, skipping
I next moved the idle check further down the code and tested removing the agent when the build was running. This time the clean-up task would remove the node.
[INFO][com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists] Checking VM exists for cnp-jenkins-builderse8d030 [INFO][com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists] cnp-jenkins-builderse8d030 doesnt exist [FINE][com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs] Node cnp-jenkins-builderse8d030 doesnt exist, removing
Submitter checklist