DnsValidatedCertificate does not fail if certificate is both "pending validation" and "validation: success"

[nsvarich]

As of this morning (Feb 3, 2020) a CDK stack deploy that previously worked, started failing with a Service: AmazonElasticLoadBalancingV2; Status Code: 400; Error Code: CertificateNotFound;

The AWS Certificate Manager console shows the certificate stuck in a Pending Validation state (with Validation Status: Success). I'm guessing this is the root cause of the issue?

I've tried to deploy several times over the last 3 hours with the same error.

Reproduction Steps

Deploy a CDK stack with the following:

const certificate = new DnsValidatedCertificate(stack, 'Cert', {
    domainName,
    hostedZone
});

const lb = new elbv2.ApplicationLoadBalancer(stack, 'ALB', {
    vpc,
    internetFacing: true,
    securityGroup
});

const listener = lb.addListener(getId(resourceMoniker, 'Listener'), {
    port: 443,
    protocol: elbv2.ApplicationProtocol.HTTPS,
    open: false,
    certificateArns: [certificate.certificateArn],
});

Error Log

 13/16 | 10:24:19 PM | CREATE_FAILED        | AWS::ElasticLoadBalancingV2::Listener     | Fargate-
Test-ALB/Fargate-Test-Listener (FargateTestALBFargateTestListenerDDAC2B18) Certificate 
'arn:aws:acm:us-west-1:<account>:certificate/<id>' not found (Service: 
AmazonElasticLoadBalancingV2; Status Code: 400; Error Code: CertificateNotFound; Request ID: 
2747ec92-de97-4e37-890c-06485abcb879)

Environment

• CLI Version : 1.20.0
• Framework Version: 1.20.0
• OS : Ubuntu 18.04
• Language : English

Other

---

This is 🐛 Bug Report

[rix0rrr]

I don't know why this is happening, but I think it might be a bit of a stretch to call it a CDK bug without further evidence.

You mentioned the stack deployed before. Did the CDK version change in the mean time? Did the template change?

The thing to do would be to figure out why your certificate didn't validate. You must be destroying+deploying it, instead of updating, if the certificate is getting recreated. Are you sure you're not hitting account limits?

If anything, the stack deployment should have failed on creating the certificate, if there's a bug it's that we pretended that the certificate creation succeeded. That's not going to help you get your cert created, but that's what I'm repurposing this ticket into.

[rix0rrr]

The combination of states "pending" and "success" is very odd to me, so it might also have been an ACM hiccup. Have you checked the health dashboard?

[nsvarich]

I just confirmed the problem has gone away as of this morning by simply re-running the same stack deploy, so I think you're right it must've been an ACM hiccup (though the health dashboard didn't show anything). Appreciate the quick response!

[sousmangoosta]

Same bug for me @rix0rrr .

If you take a look on the waiter, we are waiting for Certificate.DomainValidationOptions[].ValidationStatus be "SUCCESS" to return success
Here it is : https://github.com/aws/aws-sdk-js/blob/master/apis/acm-2015-12-08.waiters2.json
but when the dns validation is success, the certificate can be not ISSUED, have a look here : aws/aws-sdk-js#2920

I think you can ovveride in the WaiterConfiguration interface to wait on Certificate.Status to be on ISSUED state, or ask for modification (or creation of new waiter) on the sdk-js project.

What do you think about this guys and girls ?

[jogold]

I think that a rewrite of this custom resource using an async provider from the custom resource framework (@aws-cdk/custom-resources) would solve all issues here (+ would make code cleaner).