TF keeps recreating a '@' TXT record

[FransUrbo]

I've been tasked with moving all domains that the company owns from register.com to AWS Route53.

In a few domains, there's a TXT record named @. Moving those into TF and then have TF apply it leads to TF wanting to create the record each time.

Terraform Version

0.11.3

Affected Resource(s)

• aws_route53_record

Terraform Configuration Files

resource "aws_route53_zone" "bnf_org_uk" {
  name                 = "bnf.org.uk"
  comment              = "bnf.org.uk"
  force_destroy        = "true"
}

resource "aws_route53_record" "bnf_org_uk-at_txt" {
  zone_id                     = "${aws_route53_zone.bnf_org_uk.zone_id}"
  name                        = "@.bnf.org.uk"
  type                        = "TXT"
  ttl                         = 3600
  records                     = [
    "Domain for:tisl"
  ]
}

Expected Behavior

Created once and never any changes again.

Actual Behavior

TF want's to add the resource every time.

Steps to Reproduce

[Turbo-Fredrikssons-MacBook-Pro]$ terraform plan -target=aws_route53_record.bnf_org_uk-at_txt
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_route53_zone.bnf_org_uk: Refreshing state... (ID: SOMETHING)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_route53_record.bnf_org_uk-at_txt
      id:                 <computed>
      allow_overwrite:    "true"
      fqdn:               <computed>
      name:               "@.bnf.org.uk"
      records.#:          "1"
      records.4097833795: "Domain for:tisl"
      ttl:                "3600"
      type:               "TXT"
      zone_id:            "SOMETHING"


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

[Turbo-Fredrikssons-MacBook-Pro]$ terraform apply -target=aws_route53_record.bnf_org_uk-at_txt
aws_route53_zone.bnf_org_uk: Refreshing state... (ID: SOMETHING)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_route53_record.bnf_org_uk-at_txt
      id:                 <computed>
      allow_overwrite:    "true"
      fqdn:               <computed>
      name:               "@.bnf.org.uk"
      records.#:          "1"
      records.4097833795: "Domain for:tisl"
      ttl:                "3600"
      type:               "TXT"
      zone_id:            "SOMETHING"


Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_route53_record.bnf_org_uk-at_txt: Creating...
  allow_overwrite:    "" => "true"
  fqdn:               "" => "<computed>"
  name:               "" => "@.bnf.org.uk"
  records.#:          "" => "1"
  records.4097833795: "" => "Domain for:tisl"
  ttl:                "" => "3600"
  type:               "" => "TXT"
  zone_id:            "" => "SOMETHING"
aws_route53_record.bnf_org_uk-at_txt: Still creating... (10s elapsed)
aws_route53_record.bnf_org_uk-at_txt: Still creating... (20s elapsed)
aws_route53_record.bnf_org_uk-at_txt: Still creating... (30s elapsed)
aws_route53_record.bnf_org_uk-at_txt: Still creating... (40s elapsed)
aws_route53_record.bnf_org_uk-at_txt: Still creating... (50s elapsed)
aws_route53_record.bnf_org_uk-at_txt: Creation complete after 59s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
[Turbo-Fredrikssons-MacBook-Pro]$ terraform plan -target=aws_route53_record.bnf_org_uk-at_txt
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_route53_zone.bnf_org_uk: Refreshing state... (ID: SOMETHING)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + aws_route53_record.bnf_org_uk-at_txt
      id:                 <computed>
      allow_overwrite:    "true"
      fqdn:               <computed>
      name:               "@.bnf.org.uk"
      records.#:          "1"
      records.4097833795: "Domain for:tisl"
      ttl:                "3600"
      type:               "TXT"
      zone_id:            "SOMETHING"


Plan: 1 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

References

This is a screenshot from the register.com TXT record list:

This is a screenshot of the resource created in R53:

[bflad]

Hi @FransUrbo 👋 , if you're not familiar, @ is generally reserved for declaring apex records in BIND configurations, AWS Route 53 does not require using that record name (and I believe discourages its usage if you try to do it in the web console). You can instead declare just the apex name.

e.g. @.bnf.org.uk -> bnf.org.uk

[FransUrbo]

I have quite a lot of experience with Bind, everything from v4 to v9. But I've never seen that kind of record before. Not that I can remember anyway. It's been a couple of years since I last work with Bind.

Will that change be "the same"? I'm guessing it's for some form of domain validation and I don't want to risk breaking that with the move.

[bflad]

You can verify it does the correct behavior via dig/nslookup before you flip the DNS name servers with your registrar:

# Verify the existing domain

$ dig -t TXT +short bnf.org.uk
"Domain for:tisl"

# Verify the new DNS on Route 53 (use one of the name servers listed in the Route 53 zone NS record)

$ dig @ns-XXX.awsdns-XX.XXX -t TXT +short bnf.org.uk

That said, if you're really paranoid about the migration and want to be sure the records are created exactly the same, you may want to consider using Route 53's built in import functionality if your old registrar allows you to export in the correct format. Then you can import the records into Terraform (modifying the configuration until it shows no difference).

[FransUrbo]

Unfortunately, register.com don't offer an export function :(. I'll try it that way and just hope for the best. Worst case, we'll have to do it some other way (or revert back to register.com for those few domains).

Thanx for the help. I consider this issue closed.

Although, some documentation update somewhere that mention this? As in, tagging this "Documentation" (and possibly putting it at a low prio) will suffice for me. I'll leave it open for now, but feel free to close it.

[Crazybus]

@bflad I just hit this as well so thanks for the advice and explanation!

I dove into it a bit more to try and understand why the record (which is valid but not recommended) was being recreated every time. The weird thing is that terraform creates the resource but doesn't put it into the state.

My test.

resource "aws_route53_zone" "example_com" {
  name = "elastictest.co"
}

resource "aws_route53_record" "100_example_com" {
  zone_id = "${aws_route53_zone.example_com.zone_id}"
  name    = "@"
  type    = "TXT"
  records = ["text1"]
  ttl     = "3600"
}

A terraform apply works as expected and the record is correctly created. However the record doesn't show up in state for some reason.

❯ terraform state list
aws_route53_zone.example_com

When looking at the output from aws route53 list-resource-record-sets I see that record is stored differently in AWS.

 "Name": "\\100.elastictest.co.",

I tried changing the name to match what I was getting back from AWS.

resource "aws_route53_record" "100_example_com" {
  zone_id = "${aws_route53_zone.example_com.zone_id}"
  name    = "\100.elastictest.co"
  type    = "TXT"
  records = ["text2"]
  ttl     = "3600"
}

However exactly the same thing happened. The correct record (@.elastictest.co) was updated but nothing was added to the terraform state. So for my use case it is fine to switch to the apex domain however this still feels like a genuine terraform bug to me. I can't imagine it is normal for terraform to successfully create a resource and not store it in the state.

[bflad]

I can't imagine it is normal for terraform to successfully create a resource and not store it in the state.

You're correct. We should be returning a log message in this case:

https://github.com/terraform-providers/terraform-provider-aws/blob/0095b64b0015be13a10a0a97c6e7375d05b60763/aws/resource_aws_route53_record.go#L523-L526

We may need to allow the resource to convert to and from octal codes if its not doing so already. AWS Documentation

If the domain name includes any characters other than a to z, 0 to 9, - (hyphen), or _ (underscore), Route 53 API actions return the characters as escape codes. This is true whether you specify the characters as characters or as escape codes when you create the entity. The Route 53 console displays the characters as characters, not as escape codes.

Although understandably in the @ record situation, we will likely want to ensure the Terraform documentation clearly states that @ is not a reference to the apex record in Route53, like how it is in some other DNS systems, but rather the actual character.

[Crazybus]

@bflad Just to be 100% clear. The record was actually created successfully and appeared in AWS when using both @ and \100 but then didn't appear in the state file. So if a user wrote this record, ran terraform apply once, then checked the console everything looked like it worked perfectly.

So it would appear that terraform (or the AWS api) is correctly handling the octal codes. Also if I set allow_overwrite to false terraform actually prevents me from running terraform apply a second time since it detects that the record does actually exist.

[Crazybus]

Yesterday I made a switched the TXT records we had at @ to the apex domain. This worked as expected. However there was a very odd side affect which I think related to my findings about these changes not being stored in state.

So the side affect was that an unrelated record was deleted in the same zone. I still need to try and reproduce this all again however I wanted to document it here first since I find it quite scary.

Before I made the change an unrelated change in a different zone did something weird:

08:06:33 aws_route53_record.100-elastictest-co-TXT: Modifying... (ID: [email protected]_TXT)
08:06:33   records.2049461970: "container-registry.elastictest.co" => ""
08:06:33   records.3087383794: "" => "globalsign-domain-verification=removed"
08:06:33   ttl:                "300" => "3600"

The value it changed to comes from push.docker.elastictest.co which was not modified in the change (and was randomly deleted after the next run)

When the change removing the @ records was deployed this resource was deleted even though up until now it had never existed in state properly.

This record was removed.

resource "aws_route53_record" "100-elastictest-co-TXT" {	
  zone_id = "${aws_route53_zone.elastictest_co.zone_id}"	
  name    = "@.elastictest.co"	
  type    = "TXT"	
  records = ["globalsign-domain-verification=removed"]	
  ttl     = "3600"	
}

In the same plan that line was added to the apex record which already included many other TXT records:

resource "aws_route53_record" "elastictest-co-TXT" {
  zone_id = "${aws_route53_zone.elastictest_co.zone_id}"
  name    = "elastictest.co"
  type    = "TXT"
  ttl     = "300"
  records = [
    "_globalsign-domain-verification=removed",
    "status-page-domain-verification=removed",
  ]
}

When this change was ran the @ record for elastictest.co was destroyed. Looking in the route53 cloudtrail logs I can see that this actually removed push.docker.elastictest.co

13:27:20 aws_route53_record.100-elastictest-co-TXT: Destroying... (ID: [email protected]_TXT)

A bit of extra info: All of our changes are controlled via jenkins and I have confirmed via route53 cloudtrail that no records were modified by anything other than terraform running from jenkins.

[Crazybus]

Created #4183 to fix the issues I found above.

[bflad]

The non-alphanumeric character fix for aws_route53_record has been released in version 1.15.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!