Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add spotVM and maxRunDuration feature to VM provisioning #492

Open
wants to merge 32 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
742724e
Add maxRunDuration feature to VM provisioning
gbhat618 Dec 10, 2024
0f38202
spot vm and max run duration settings put
gbhat618 Dec 11, 2024
d8e0bca
update comments
gbhat618 Dec 11, 2024
615e605
change from radio to dropdown descriptor selector
gbhat618 Dec 12, 2024
d57ea0c
update comments
gbhat618 Dec 12, 2024
579410e
add an integration test for spot vms with maximum run durations
gbhat618 Dec 13, 2024
17a0113
Fix trailing slash
gbhat618 Dec 16, 2024
411e510
Merge branch 'develop' of github.com:jenkinsci/google-compute-engine-…
gbhat618 Dec 16, 2024
3ddd4d3
update tests and new tests for scheduling code
gbhat618 Dec 16, 2024
363925a
remove the @deprecated annotation that would otherwise break casc com…
gbhat618 Dec 16, 2024
f4c7154
remove the todo added during code analysis
gbhat618 Dec 16, 2024
74fe8c3
fix the access specifier in descriptor
gbhat618 Dec 16, 2024
cfae906
comment add license
gbhat618 Dec 16, 2024
937abdf
remove unused field leftout during refactor
gbhat618 Dec 16, 2024
8fa2222
fix spelling mistake in default
gbhat618 Dec 17, 2024
1708135
re-order setters to reduce diff
gbhat618 Dec 17, 2024
e806d25
fix the default provisioning type considering preemptible could be al…
gbhat618 Dec 17, 2024
50a74b8
spotless
gbhat618 Dec 17, 2024
3e7e787
fix duplication, handle field deprecation and migration
gbhat618 Dec 18, 2024
244da59
fix the preemptive task rescheduling integration tests
gbhat618 Dec 18, 2024
c076896
Update src/main/java/com/google/jenkins/plugins/computeengine/Instanc…
gbhat618 Dec 19, 2024
f656ffc
Update src/main/java/com/google/jenkins/plugins/computeengine/Instanc…
gbhat618 Dec 19, 2024
361bdd0
removing the ProvisioningTypeValue enum
gbhat618 Dec 19, 2024
11c7068
fix the test instead of casc compatibility
gbhat618 Dec 19, 2024
8d7f256
add DataBoundConstructors to classes
gbhat618 Dec 19, 2024
e602674
keep casc compatibility for preemptible field
gbhat618 Dec 19, 2024
ba2d713
fix the test throwing error in RemainingActivityListener, by waitUnti…
gbhat618 Dec 20, 2024
a2dc12c
rename the package from ui.helps to better name configs
gbhat618 Dec 20, 2024
3a2cea4
config (not confi)
gbhat618 Dec 20, 2024
7442934
revert the integration test infra chagnes as these are handled by oth…
gbhat618 Dec 20, 2024
ea30baf
Merge branch 'develop' of github.com:jenkinsci/google-compute-engine-…
gbhat618 Jan 8, 2025
2b59050
update tests
gbhat618 Jan 8, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@

import static com.google.cloud.graphite.platforms.plugin.client.util.ClientUtil.nameFromSelfLink;
import static com.google.jenkins.plugins.computeengine.ComputeEngineCloud.checkPermissions;
import static com.google.jenkins.plugins.computeengine.ui.helpers.ProvisioningTypeValue.PREEMPTIBLE;
import static com.google.jenkins.plugins.computeengine.ui.helpers.ProvisioningTypeValue.SPOT;

import com.google.api.client.json.GenericJson;
import com.google.api.services.compute.model.AcceleratorConfig;
import com.google.api.services.compute.model.AttachedDisk;
import com.google.api.services.compute.model.AttachedDiskInitializeParams;
Expand All @@ -42,6 +45,8 @@
import com.google.jenkins.plugins.computeengine.ssh.GoogleKeyCredential;
import com.google.jenkins.plugins.computeengine.ssh.GoogleKeyPair;
import com.google.jenkins.plugins.computeengine.ssh.GooglePrivateKey;
import com.google.jenkins.plugins.computeengine.ui.helpers.ProvisioningType;
import com.google.jenkins.plugins.computeengine.ui.helpers.ProvisioningTypeValue;
import edu.umd.cs.findbugs.annotations.Nullable;
import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
import hudson.Extension;
Expand All @@ -63,6 +68,7 @@
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.logging.Level;
Expand Down Expand Up @@ -120,7 +126,19 @@
private String machineType;
private String numExecutorsStr;
private String startupScript;

/**
* Use the {@link #provisioningType} field instead.
*/
@Deprecated
private boolean preemptible;

/**
* Succeeds {@link #preemptible}.
*/
private ProvisioningType provisioningType;
gbhat618 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for reviewers: this class is using Lombok (not a good idea!):


private long maxRunDurationSeconds;
private String minCpuPlatform;
private String labels;
private String runAsUser;
Expand Down Expand Up @@ -494,7 +512,26 @@

private Scheduling scheduling() {
Scheduling scheduling = new Scheduling();
scheduling.setPreemptible(preemptible);

if (provisioningType != null) {
if (provisioningType.getValue() == PREEMPTIBLE) {
scheduling.setPreemptible(true);
} else if (provisioningType.getValue() == SPOT) {
scheduling.setProvisioningModel("SPOT");
// only the instance is deleted, the disk deletion is based on bootDiskAutoDelete config value
scheduling.setInstanceTerminationAction("DELETE");
}
} else if (preemptible) { // keeping the check for `preemptible` for backward compatibility
scheduling.setPreemptible(true);
} // else: standard provisioning

if (maxRunDurationSeconds > 0 && provisioningType.getValue() != PREEMPTIBLE) {
GenericJson j = new GenericJson();
j.set("seconds", maxRunDurationSeconds);
scheduling.set("maxRunDuration", j);
// only the instance is deleted, the disk deletion is based on bootDiskAutoDelete config value
scheduling.setInstanceTerminationAction("DELETE");
}
return scheduling;
}

Expand Down Expand Up @@ -554,6 +591,7 @@
@Extension
public static final class DescriptorImpl extends Descriptor<InstanceConfiguration> {
private static ComputeClient computeClient;
private static List<ProvisioningType.ProvisioningTypeDescriptor> provisioningTypeDescriptors;

public static void setComputeClient(ComputeClient client) {
computeClient = client;
Expand Down Expand Up @@ -783,6 +821,29 @@
return FormValidation.ok();
}

public FormValidation doCheckMaxRunDurationSeconds(
Fixed Show fixed Hide fixed
Fixed Show fixed Hide fixed
@QueryParameter String value, @QueryParameter(value="provisioningType") String provisioningTypeIdx) {
try {
long maxRunDurationSeconds = Long.parseLong(value);
if (maxRunDurationSeconds < 0) {
return FormValidation.error("Max run duration must be greater than or equal to 0");
}

if (StringUtils.isNotBlank(provisioningTypeIdx)) {
ProvisioningTypeValue provisioningTypeValue = provisioningTypeDescriptors
.get(Integer.parseInt(provisioningTypeIdx))
.getProvisioningTypeValue();
if (provisioningTypeValue == PREEMPTIBLE) {
return FormValidation.warning("Max run duration is not supported for preemptible VMs and will be ignored.");
}
}

return FormValidation.ok();
} catch (NumberFormatException e) {
return FormValidation.error("Max run duration must be non-negative number");
}
}

public ListBoxModel doFillMinCpuPlatformItems(
@AncestorInPath Jenkins context,
@QueryParameter("projectId") @RelativePath("..") final String projectId,
Expand Down Expand Up @@ -943,6 +1004,15 @@
return FormValidation.ok();
}

@SuppressWarnings("unused")
gbhat618 marked this conversation as resolved.
Show resolved Hide resolved
public List<ProvisioningType.ProvisioningTypeDescriptor> getProvisioningTypes() {
if (provisioningTypeDescriptors == null) {
provisioningTypeDescriptors =
Objects.requireNonNull(Jenkins.getInstanceOrNull()).getDescriptorList(ProvisioningType.class);
}
return provisioningTypeDescriptors;
}

public List<NetworkInterfaceIpStackMode.Descriptor> getNetworkInterfaceIpStackModeDescriptors() {
return ExtensionList.lookup(NetworkInterfaceIpStackMode.Descriptor.class);
}
Expand All @@ -959,6 +1029,8 @@
instanceConfiguration.setNumExecutorsStr(this.numExecutorsStr);
instanceConfiguration.setStartupScript(this.startupScript);
instanceConfiguration.setPreemptible(this.preemptible);
instanceConfiguration.setProvisioningType(this.provisioningType);
instanceConfiguration.setMaxRunDurationSeconds(this.maxRunDurationSeconds);
instanceConfiguration.setMinCpuPlatform(this.minCpuPlatform);
instanceConfiguration.setLabelString(this.labels);
instanceConfiguration.setRunAsUser(this.runAsUser);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package com.google.jenkins.plugins.computeengine.ui.helpers;
gbhat618 marked this conversation as resolved.
Show resolved Hide resolved

import hudson.Extension;
import org.kohsuke.stapler.DataBoundConstructor;
import org.kohsuke.stapler.DataBoundSetter;

@SuppressWarnings("unused")
public class PreemptibleVm extends ProvisioningType {

@DataBoundConstructor
public PreemptibleVm() {
super(ProvisioningTypeValue.PREEMPTIBLE);
}

@Extension
public static class DescriptorImpl extends ProvisioningTypeDescriptor {
@Override
public String getDisplayName() {
return "Preemptible VM";
}

@Override
public ProvisioningTypeValue getProvisioningTypeValue() {
return ProvisioningTypeValue.PREEMPTIBLE;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
package com.google.jenkins.plugins.computeengine.ui.helpers;

import org.kohsuke.stapler.DataBoundSetter;

import hudson.model.AbstractDescribableImpl;
import hudson.model.Descriptor;

/**
* ProvisioningType represents the type of VM to be provisioned.
*/
public abstract class ProvisioningType extends AbstractDescribableImpl<ProvisioningType> {
public ProvisioningTypeValue value;

public ProvisioningType(ProvisioningTypeValue value) {
this.value = value;
}

public ProvisioningTypeValue getValue() {
return value;
}

@DataBoundSetter
public void setProvisioningTypeValue(ProvisioningTypeValue value) {
this.value = value;
}

public abstract static class ProvisioningTypeDescriptor extends Descriptor<ProvisioningType> {
public abstract ProvisioningTypeValue getProvisioningTypeValue();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
package com.google.jenkins.plugins.computeengine.ui.helpers;

public enum ProvisioningTypeValue {
gbhat618 marked this conversation as resolved.
Show resolved Hide resolved
STANDARD,
SPOT,
PREEMPTIBLE;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package com.google.jenkins.plugins.computeengine.ui.helpers;

import hudson.Extension;
import org.kohsuke.stapler.DataBoundConstructor;
import org.kohsuke.stapler.DataBoundSetter;

@SuppressWarnings("unused")
public class SpotVm extends ProvisioningType {
gbhat618 marked this conversation as resolved.
Show resolved Hide resolved

@DataBoundConstructor
public SpotVm() {
super(ProvisioningTypeValue.SPOT);
}

@Extension
public static class DescriptorImpl extends ProvisioningTypeDescriptor {
@Override
public String getDisplayName() {
return "Spot VM";
}

@Override
public ProvisioningTypeValue getProvisioningTypeValue() {
return ProvisioningTypeValue.SPOT;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
package com.google.jenkins.plugins.computeengine.ui.helpers;

import hudson.Extension;
import org.kohsuke.stapler.DataBoundConstructor;
import org.kohsuke.stapler.DataBoundSetter;

@SuppressWarnings("unused")
public class Standard extends ProvisioningType {

@DataBoundConstructor
public Standard() {
super(ProvisioningTypeValue.STANDARD);
}

@Extension
public static class DescriptorImpl extends ProvisioningTypeDescriptor {
@Override
public String getDisplayName() {
return "Standard";
}

@Override
public ProvisioningTypeValue getProvisioningTypeValue() {
return ProvisioningTypeValue.STANDARD;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,12 @@
<f:entry field="machineType" title="${%Machine Type}">
<f:select/>
</f:entry>
<f:entry field="preemptible" title="${%Preemptible?}">
<f:checkbox/>
<f:descriptorRadioList descriptors="${descriptor.provisioningTypes}"
gbhat618 marked this conversation as resolved.
Show resolved Hide resolved
title="Provisioning Type"
instance="${instance}"
varName="provisioningType"/>
<f:entry field="maxRunDurationSeconds" title="${%Max Run Duration Seconds}">
<f:textbox/>
</f:entry>
<f:entry field="minCpuPlatform" title="${%Minimum Cpu Platform}">
<f:select/>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
<!--
Copyright 2024 CloudBees, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<div>
The maximum duration (in seconds) after which the VM will be automatically deleted by GCP.
See <a href="https://cloud.google.com/compute/docs/instances/limit-vm-runtime">Limit the run time of a VM</a> for more details.
</div>

This file was deleted.

gbhat618 marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
<!--
Copyright 2024 CloudBees, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

<div>
GCP offers three VM provisioning options, each with different availability, pricing, and termination policies.
The "<code>Max Run Duration Seconds</code>" field below applies to all three types.
<ul>
<li>
<strong>Standard:</strong> Stable provisioning model with guaranteed availability by GCP SLAs.
Higher pricing compared to Spot and Preemptible VMs.
</li>
<li>
<strong>Spot:</strong> Low-cost instances that may be terminated if resources are needed elsewhere. Availability varies.
Unlike Preemptible VMs, they are not always terminated after 24 hours. Spot and Preemptible VMs are priced the same.
Set the "<code>Max Run Duration Seconds</code>" field below to ensure termination after a specific duration.
<p>
See <a href="https://cloud.google.com/compute/docs/instances/spot">Spot VMs</a> for more details.
</p>
</li>
<li>
<strong>Preemptible:</strong> Low-cost instances that may be terminated if resources are needed for other tasks.
Availability varies with usage. Spot and Preemptible VMs are priced the same.
<p>
See the <a href="https://cloud.google.com/compute/docs/instances/preemptible">Preemptible VM Instances</a> documentation for more information.
</p>
</li>
</ul>
</div>
Loading