Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove node if VM doesn't exist even if computer is not idle #552

Merged
merged 11 commits into from
Jul 23, 2024

Conversation

Tyler-35
Copy link
Contributor

@Tyler-35 Tyler-35 commented Jul 23, 2024

When working with spot instance, when the agent is evicted/removed the node in jenkins is not removed because of this check which causes the build to hang.
If we move the idle check below the vm-exists check this solves the issue.

Testing done

Tested this change by adding more logging to see what was causing the issue of why the node wasn't being removed when the agent no longer existed.
The logging show it was being stop by the not idle check.
[FINE][com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs] Node cnp-jenkins-builderse8d030 is not idle, skipping
I next moved the idle check further down the code and tested removing the agent when the build was running. This time the clean-up task would remove the node.
[INFO][com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists] Checking VM exists for cnp-jenkins-builderse8d030 [INFO][com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists] cnp-jenkins-builderse8d030 doesnt exist [FINE][com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask cleanVMs] Node cnp-jenkins-builderse8d030 doesnt exist, removing

Submitter checklist

  • Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
  • Ensure that the pull request title represents the desired changelog entry
  • Please describe what you did
  • Link to relevant issues in GitHub or Jira
  • Link to relevant pull requests, esp. upstream and downstream changes
  • Ensure you have provided tests - that demonstrates feature works or fixes the issue

Copy link
Member

@timja timja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Tyler,

I'm taking a look at this, in future please check out the submitter checklist, specifically this wasn't done:

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!

(you've opened from your master branch)
Its easy to accidentally push unrelated changes to your master branch and affecting open PRs

@timja timja added the bug label Jul 23, 2024
@timja timja changed the title move vm-exist-check for spot instance Remove node if VM doesn't exist even if computer is not idle Jul 23, 2024
@Tyler-35 Tyler-35 changed the title Remove node if VM doesn't exist even if computer is not idle move idle-check for spot instance Jul 23, 2024
@Tyler-35 Tyler-35 changed the title move idle-check for spot instance Remove node if VM doesn't exist even if computer is not idle Jul 23, 2024
@Tyler-35 Tyler-35 marked this pull request as ready for review July 23, 2024 14:03
@timja timja merged commit 0b47dc6 into jenkinsci:master Jul 23, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants