Add support for AWS ACM resources

[marcosnils]

Pending on aws/aws-sdk-go#523

[paultyng]

👍

[phinze]

We're pretty excited about this release! Unfortunately it looks like the shape of the API is not very automation-friendly. 😞

There's a required email-based validation step after calling RequestCertificate.

We'll have to figure out if there's a reasonable way for Terraform to work with the validation step.

Stay tuned!

[apparentlymart]

Perhaps an alternative way to think of this support is that setting up a certificate is something you do out of band of Terraform, but then Terraform has a read-only resource (or a data source!) to retrieve certificates using DescribeCertificate/GetCertificate so that they can then be used to configure other dependent resources automatically in Terraform.

It's a shame this API doesn't separate validation from certificate requesting like the ACME API does; for my LetsEncrypt design in #3599 I modeled the certificate itself as a resource but expected that the validation step would be handled out-of-band. Doesn't seem like there's any way to do that for ACM. 😢

[HenryTheHamster]

Even being able to provide the ARN of existing certificates in aws_elb resource definitions would be a huge help.

[marcosnils]

agree with @HenryTheHamster and @apparentlymart comments. Certificate creation might be handled outside terraform but It'd be nice if assignment can be automated.

[phinze]

Even being able to provide the ARN of existing certificates in aws_elb resource definitions would be a huge help.

I believe this requires no change in Terraform, right? Just a different value for ssl_certificate_id.

Perhaps [...] setting up a certificate is something you do out of band of Terraform, but then Terraform has a read-only resource (or a data source!) to retrieve certificates

I was indeed thinking of this as a good scenario for a data source!

The other model that might be worth considering is a long (several minute) poll cycle during which Terraform waits for the validation step to be completed. We'd need to somehow make it eminently clear in the plan that the apply will block on a human-triggered action. It would definitely be weird. Perhaps too weird! But it would allow Terraform to manage the full cert cycle. So, food for thought. 😀

[marcosnils]

I believe this requires no change in Terraform, right? Just a different value for ssl_certificate_id.

correct.

The other model that might be worth considering is a long (several minute) poll cycle during which Terraform waits for the validation step to be completed. We'd need to somehow make it eminently clear in the plan that the apply will block on a human-triggered action. It would definitely be weird. Perhaps too weird! But it would allow Terraform to manage the full cert cycle. So, food for thought

IMHO it's not worth to bring this type of behavior just to handle these cases in terraform. I'd still prefer to handle certification creation manually and then just parametrize terraform with the corresponding certificate ids.

[apparentlymart]

@phinze the other thing I thought of in the mean time was that maybe once we have infrastructure in place for #4149 we could build on that by giving resource implementations a way to force deferral of their own dependencies. Not sure exactly how this would play out but I'm thinking something like this:

• Add a flag to ResourceDiff to say "I'm not going to be able to complete in one step"
• If that flag is set, all dependencies of the given resource are automatically deferred, as described in Partial/Progressive Configuration Changes #4149. A flag is placed in the resource state so that Terraform remembers that this resource is currently blocking its downstream deps.
• During Refresh, a provider can decide whether or not to clear the "I'm blocking my downstream deps" flag based on information it discovers. In this case, the flag would be cleared once the validation status reflects that validation has been completed.
• During subsequent plans, Terraform will use the "I'm blocking my downstream deps" flag to decide whether to continue to defer the dependencies or to now include them in the plan, allowing them to complete.

This doesn't feel exactly right yet, since it's not clear to me how Terraform would then explain to the user what needs to be done to proceed, but maybe there's something in this general area if we continue to work on the details. Perhaps there are some adjustments to the partial plan/apply model we discussed that would make this sort of flow more reasonable.

[catsby]

aws/aws-sdk-go#523 has been merged and released in v1.0.11

[HenryTheHamster]

Changing the ssl_certificate_id results in an error of
Error creating ELB: CertificateNotFound: Server Certificate not found for the key: arn:aws:acm:us-east-1...
I had assumed this is because its looking for a different resource type (acm), but happy to be told I'm doing it wrong :)

[jfexyz]

ACM = AWS Certificate Manager (for searchability)

[marcosnils]

@HenryTheHamster I had no problems using the ssl_certificate_id with the ACM arn. Could you make it work?

[HenryTheHamster]

@marcosnils The mistake was mine. I'm in ap-southeast-2, which is not available for ACM support for ELBs as yet. The CloudFront ACM certs were working fine, as they are unconstrained by region, which was what was confusing me.

[munhitsu]

+1

[pimvernooij]

With regards to automating the manual verification step; probably requiring a Route53 hosted zone is available and willing to have subdomains created automatically (terraform-acm-verifications.mydomain.com), one could automatically create a "Receipt Rule" using SES and receive the message in S3 and read the confirmation from there (or invoke an SNS notification or Lambda function).

https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-receipt-rule-set.html

[omarabid]

If the domain is registered with Route 53, do you still need to do the email confirmation?

I've been thinking of the read-only mode. I think it's useless unless you are getting the id of the certificate by the "domain name" and not reading through the resource.

[marcosnils]

@omarabid yes, you still need to do the verification anyways.

[errordeveloper]

Would love to see this!

[omarabid]

@marcosnils What's the verification purpose anyway?

[nathanielks]

@omarabid I imagine so that people can't do man-in-the-middle attacks? Just a guess though.

[BerndWessels]

Is anybody still working on this issue?

[kamoljan]

+1

[erutherford]

Providing a datasource to grab the available certs in ACM would be a huge improvement, it'd be great if we could at least do that. It's better than manually maintaining a map with these values.

[willejs]

@phinze Is there any update on this, is it on the roadmap? Sorry to bother you!

[dwmkerr]

Would love to see this as well! Currently I set up all my infrastructure with terraform, except Route53 and the certificates, would be great to be able to just specify my certificate ARN.

[croess]

+1

[robertstettner]

+1

[paultyng]

I second the usefulness of a data source, especially if you could query by domain name (or wildcard, etc).

[paultyng]

#8359 A data source for querying ARN's is available.

[jen20]

@paultyng Yup, the data source will be in the next release. I don't know if we can usefully do anything other than that, though I will read through the thread and verify. Perhaps we can create the requests and verify that they are in one of the valid states.

[phinze]

From my review, the only thing that makes sense as a resource would be something that drives ImportCertificate and has fields for shoving raw cert / key / chain contents into ACM.

This is a valid use case for users who want to be able to reference their certificates via ACM arns, but as far as I can tell its utility is pretty limited - as there's no benefit from auto-renewal.

I think @apparentlymart's description is potentially a useful path, but reviewing the backlog here I think that the data source we just merged covers enough to be able to close this issue. Anybody who disagrees feel free to petition to reopen! 👍

[joelthompson]

I think that the data source we just merged covers enough to be able to close this issue. Anybody who disagrees feel free to petition to reopen!

Not to necessarily petition to re-open, but I've implemented automated, out-of-band approvals of ACM requests (and I've also asked our AWS reps for the ability to pre-verify an entire domain with ACM so these approvals aren't necessary), with a turnaround time of about 10 seconds. Given this, I'd love to to have an ACM resource in Terraform. If I implemented this, would it be something you'd consider merging, or would it still be too fringe of a use case that I should just keep it as a separate plugin?

[Jonty]

@joelthompson Could you detail how you implemented automated out-of-band verification, and how you propose to implement it in terraform?

[joelthompson]

Well, so, I wouldn't implement the out-of-band verification in TF. TF would just need to assume something exists (either some sort of automated verification or somebody waiting by an email inbox), or it'll just fail. But the implementation isn't that difficult -- I just have a script which parses the verification email, ensures that it comes from an authorized account, and if so, opens the link in the email, scrapes the approval webpage, and submits the form (implemented under the hood with SES email receipt + S3 + Lambda).

[tomelliff]

@apparentlymart I just took a quick look at the SDK and it looks like you could use a resource to create the certificate request easily enough (http://docs.aws.amazon.com/sdk-for-go/api/service/acm/#ACM.RequestCertificate).

Am I misunderstanding what you'd want to do with an ACM resource here? Simply using Terraform to create the request (and then the pre-existing data source to look up the certificate once it has been validated) would be good enough for me I think. Obviously I then have to handle the validation out of band but that's not too bad. I have the code for a Lambda function lying around somewhere that did exactly what @joelthompson is talking about that I can use for things at scale if necessary.

Maybe the resource would have to be called aws_acm_request to be explicit that it's just doing the request and not actually returning a valid certificate at that point.

[apparentlymart]

Hi @tomelliff!

It's been a while since I looked at this one so I'm not sure I remember all of the details here, but from skimming the API docs you linked to I see that the RequestCertificate function does return a certificate ARN, so in principle Terraform could call that API and return it.

What I think we were concerned about above is the likely possibility that the certificate would not be immediately ready for use, and thus downstream resources depending on it would fail. I don't remember exactly how this behaves, but a hypothetical example of the config we wanted to work would be:

resource "aws_acm_certificate" "example" {
  # ... arguments ...
}

resource "aws_elb" "example" {
  # ...
  listener {
    # ...
    ssl_certificate_id = "${aws_acm_certificate.example.arn}"
  }
}

The above would presumably fail to apply because the certificate would not be approved in time for the ELB to be created.

However, it would in principle be possible to make one configuration that only deals with ACM certificate requests, and another separate configuration that uses them via the data source, and then coordinate applying them so that the second is not applied until the manual approval is complete.

I think if we took care to mention in the docs the limitations and caveats here then we could have a resource that creates the request. We could potentially include on it an optional setting for an automatic validation timeout, where the resource would poll the GetCertificate API politely until either the timeout is reached or the certificate becomes approved, which would then help Terraform coordinate with an out-of-band auto-approval mechanism to make something like my above example work.

If you'd like to talk about this some more, I'd encourage opening a new issue in the new separated aws provider repo, since that is where future development of the AWS provider will take place.

[tilgovi]

Would a wait_for_approval = (true|false) work for the certificate resource? Then the resource wouldn't finish being created—its dependents would be blocked—until it's approved.

I don't know enough about the API off the top of my head to know how resumable this process would be if the resource timed out waiting for that approval.

[apparentlymart]

That's essentially what I was suggesting, though I would instead make it something like approval_timeout_secs = 60 so that the user can specify how long to wait for the approval before failing, so Terraform doesn't end up running indefinitely. (Running forever would be problematic because other resources created at the same time would not necessarily be persisted to state if e.g. Terraform were forcefully interrupted by a CI system, etc.)

[tilgovi]

Ah, yes, I re-read. This is your suggestion precisely.

[telepath]

I do petition to re-open.
I'm really missing the ability to create certificate requests via Terraform. I think it's absurd that right now I need to create them by hand or shell-script or whatever, when I create all other AWS resources with Terraform.
Of course the validation needs to be done, so I cannot use a new certificate as a dependency. But why should that be a reason to do it by hand? I can either create a standalone script just for all needed certificates, or run the script only for the certificate first, take care of the validation, and then run the full script. There are enough useful things to do with an acm resource to merit implementing it.

[w32-blaster]

I'd like to see this reopened. If nothing but making certificate import possible would be helpful to keep our certificates in ACM and manage it with TF. Import doesn't require approval so it should be easier to implement

[ozbillwang]

@apparentlymart && @phinze && @vancluever

I went through the comments, understand that we can't implement ACM resource because:

There's a required email-based validation step after calling RequestCertificate.

But we can do something for free Letsencrypt certificates (ACME)

Provider paybyphone/terraform-provider-acme can be used to both manage registrations and certificates. There are no email-based validation step required.

After implemented terraform-provider-acme resources, I successfully get below terraform tfstate:

                "acme_certificate.certificate": {
                    "type": "acme_certificate",
                    "depends_on": [
                        "acme_registration.reg",
                        "tls_cert_request.req",
                        "tls_private_key.reg_private_key"
                    ],
                    "primary": {
                        "id": "https://acme-v01.api.letsencrypt.org/acme/cert/xxxx",
                        "attributes": {
                            "account_key_pem": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAtjLxxxxV2dlT0EpeNWbvuydrc6Xf6PL\n-----END RSA PRIVATE KEY-----\n",
                            "account_ref": "https://acme-v01.api.letsencrypt.org/acme/reg/xxx",
                            "certificate_domain": "stag.newsnow.io",
                            "**certificate_pem**": "-----BEGIN CERTIFICATE-----\nMIIFGDCCBACgAwIBxxxxkaLBVsGY6vA=\n-----END CERTIFICATE REQUEST-----\n",
                            "certificate_url": "https://acme-v01.api.letsencrypt.org/acme/cert/031408ad563ff33291d02f21bdd3e2a3c595",
                            "dns_challenge.#": "1",
                            "dns_challenge.1056487162.config.%": "0",
                            "dns_challenge.1056487162.provider": "route53",
                            "http_challenge_port": "80",
                            "id": "https://acme-v01.api.letsencrypt.org/acme/cert/031408ad563ff33291d02f21bdd3e2a3c595",
                            "**issuer_pem**": "-----BEGIN CERTIFICATE-----\nMIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2xxxV2dlT0EpeNWbvuydrc6Xf6PL\n-----END RSA PRIVATE KEY-----\n",
                            "key_type": "2048",
                            "min_days_remaining": "7",
                            "must_staple": "false",
                            "**private_key_pem**": "",
                            "registration_url": "https://acme-v01.api.letsencrypt.org/acme/reg/21546469",
                            "server_url": "https://acme-v01.api.letsencrypt.org/directory",
                            "tls_challenge_port": "443"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": ""
                },

With help from @vancluever, I now understand how to map the PEMs to the information which AWS Certificate manager (ACM) required.

• Certificate body: acme_certificate.certificate.certificate_pem
• Certificate private key: acme_certificate.certificate.private_key_pem
• Certificate chain: acme_certificate.certificate.issuer_pem

So I just need one more resource to import above three PEMs (and re-import as well???) to AWS Certificate manager. Currently I have to do that manually.

Will this be easier implemented in terraform? Can you help?

aws cli command for reference:

aws acm import-certificate --certificate file://Certificate.pem
                           --certificate-chain file://CertificateChain.pem
                           --private-key file://PrivateKey.pem

data source aws_acm_certificate exists already.

With the last miss part, It will release the pain that I need to take care for the certificate issuer in non-prod environments anymore

[pimvernooij]

With CAA record support in route53, this should now be easier to implement (when using r53 for your domains): http://docs.aws.amazon.com/acm/latest/userguide/troubleshooting-caa.html

…

On Fri, 22 Sep 2017 at 14:24, Bill Wang ***@***.***> wrote: @apparentlymart <https://github.com/apparentlymart> && @phinze <https://github.com/phinze> && @vancluever <https://github.com/vancluever> I went through the comments, understand that we can't implement ACM resource because: There's a required email-based validation step after calling RequestCertificate. But we can do something for free Letsencrypt certificates (ACME) With provider paybyphone/terraform-provider-acme <https://github.com/paybyphone/terraform-provider-acme>, this provider can be used to both manage registrations and certificates. I successfully get the terraform tfstate as below: "acme_certificate.certificate": { "type": "acme_certificate", "depends_on": [ "acme_registration.reg", "tls_cert_request.req", "tls_private_key.reg_private_key" ], "primary": { "id": "https://acme-v01.api.letsencrypt.org/acme/cert/xxxx", "attributes": { "account_key_pem": "-----BEGIN RSA PRIVATE KEY-----\nMIIEowIBAAKCAQEAtjLxxxxV2dlT0EpeNWbvuydrc6Xf6PL\n-----END RSA PRIVATE KEY-----\n", "account_ref": "https://acme-v01.api.letsencrypt.org/acme/reg/xxx", "certificate_domain": "stag.newsnow.io", "certificate_pem": "-----BEGIN CERTIFICATE-----\nMIIFGDCCBACgAwIBxxxxkaLBVsGY6vA=\n-----END CERTIFICATE REQUEST-----\n", "certificate_url": "https://acme-v01.api.letsencrypt.org/acme/cert/031408ad563ff33291d02f21bdd3e2a3c595", "dns_challenge.#": "1", "dns_challenge.1056487162.config.%": "0", "dns_challenge.1056487162.provider": "route53", "http_challenge_port": "80", "id": "https://acme-v01.api.letsencrypt.org/acme/cert/031408ad563ff33291d02f21bdd3e2a3c595", "issuer_pem": "-----BEGIN CERTIFICATE-----\nMIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2xxxV2dlT0EpeNWbvuydrc6Xf6PL\n-----END RSA PRIVATE KEY-----\n", "key_type": "2048", "min_days_remaining": "7", "must_staple": "false", "private_key_pem": "", "registration_url": "https://acme-v01.api.letsencrypt.org/acme/reg/21546469", "server_url": "https://acme-v01.api.letsencrypt.org/directory", "tls_challenge_port": "443" }, "meta": {}, "tainted": false }, "deposed": [], "provider": "" }, With help from @vancluever <https://github.com/vancluever>, I now understand to map the PEMs to the informations which AWS Certificate manager (ACM) required. - Certificate body: acme_certificate.certificate.certificate_pem - Certificate private key: acme_certificate.certificate.private_key_pem - Certificate chain: acme_certificate.certificate.issuer_pem So I just need one more resource to import the above three PEMs to AWS Certificate manager. Will this be easier implemented in terraform? reference aws cli command is: aws acm import-certificate --certificate file://Certificate.pem --certificate-chain file://CertificateChain.pem --private-key file://PrivateKey.pem — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4782 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABxOWBAFKzgaWEZ7xNKgxti1Jgi_yTAsks5sk6bwgaJpZM4HJxqp> .

[tomelliff]

I'm not sure why this hasn't been migrated to https://github.com/terraform-providers/terraform-provider-aws/ any chance someone could do that?

On topic, having CAA records makes zero difference to this as CAA records simply prohibit certificates being issued for a domain by a provider not included in the CAA record. On the other hand AWS have just released DNS validation of ACM certs using CNAMEs so as long as Terraform can manage your DNS provider (eg Route53 or DNSimple or similar) then this should now be doable in a single workflow.

I'd need to check how this actually works because if the ACM request through the API returns the CNAME record key and value that needs creating then the ACM resource would need to return after creating the request but before validation. That might then have some impact if you are creating the ACM certificate, verifying the request with the CNAME record and attaching it to a load balancer/Cloudfront/API Gateway all in the same terraform apply even if you use a depends_on on the CNAME record being created before the load balancer etc is created.

[pdecat]

@tomelliff I've just opened hashicorp/terraform-provider-aws#2418

[martinssipenko]

Maybe someone finds this helpful, the I worked around this was by creating a CloudFormation stack using Terraform. In the CFN stack I only crated the ACM resource and added its ARN as output, which can then be used as variable in Terraform script.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.