(aws-ecs): add a warning when the default value for minimumHealthyPercent (50%) is used for a service

[MarrickLip]

Describe the feature

Add a warning message when minimumHealthyPercent isn't set for an ECS service, resulting in the implicit use of CDK's default value of 50%. This warning wouldn't appear if minimumHealthyPercent is explicitly set to 50%.

Use Case

CDK overrides the default value of 100% used by CloudFormation's AWS::ECS::Service and the CreateService API. This allows the number of running tasks to drop by up to 50% during deployments and Fargate maintenance etc.

This is an unsafe default for services which must support a consistent load e.g. handle web traffic via an ALB.

Proposed Solution

Implementation would be similar to the warning when creating an ASG with an explicit desired capacity.

Wording would be along the lines of:

minimumHealthyPercent has not been configured so the default value of 50% is used. The number of running tasks will decrease below the desired count during deployments etc. See [link to this issue]

Other Information

#14277 proposed changing the default value to 50%. This is a breaking change (e.g. it can cause stuck deployments) so I believe a warning is more practical. I also considered implementing this as a cdk-nag rule, but I believe a warning in "mainstream CDK" is warranted as this is a quirk of CDK itself.

I wasn't able to find any context for why CDK chose to override CloudFormation's default value. The ECS Developer Guide says 50% should be used to speed up deployments for services which are "idle for some time and don't have a high utilization rate." (source), but 100% is recommended for stateless workloads:

For example, when deploying a stateless application as an Amazon ECS service, such as a web or API server, customers should deploy multiple task replicas and set the minimumHealthyPercent to 100% (source)

---

I'm happy to open a PR but just wanted to get thoughts on this before putting pen to paper :)

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

Latest

Environment details (OS name and version, etc.)

N/A

[khushail]

Hi @MarrickLip , thanks for reaching out.

I see that in the cloduformation docs, this minimumHealthyPercent is having different values for different scenarios -

The default value for a replica service for minimumHealthyPercent is 100%. The default minimumHealthyPercent value for a service using the DAEMON service schedule is 0% for the AWS CLI, the AWS SDKs, and the APIs and 50% for the AWS Management Console.

Yes, the CDK chose to keep the minimumHealthyPercent to 50 %

https://github.com/aws/aws-cdk/blob/17c81fe8362a0b5dcb59f059ade298a904b2aba3/packages/%40aws-cdk/aws-ecs/lib/base/base-service.ts#L129C1-L136C39

when the AWS suggested it as 100%.

It would be great to display a warning message,so appreciate you volunteering for that.

I would be marking this issue as P3 which means its a good to have feature and would be open for contribution from community as well as team.
Thanks.

[davidjmemmett]

Contrary to this, when initially setting up a new ECS stack, minHealthyPercent should be zero for some containers, and then increased later when dependencies (DBs which may be controlled outside of CDK etc) are healthy.

I've just watched a video from this year's re:Invent whereby even the internal EC2 teams have rolling dependencies when bootstrapping a new AWS region, which means that services should be allowed to fail during a deployment until other things have reached a steady state.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.