custom-resources: How to iterate over list returned by custom resource

[SvenMarquardt5772]

Describe the bug

When using an AWSCustomResource to retrieve the managed prefix list entries of one managedprefixlist, all entries are encoded like this.

"Data": {
        "apiVersion": null,
        "region": "eu-central-1",
        "Entries.0.Cidr": "3.69.126.10/32",
        "Entries.1.Cidr": "3.72.182.106/32",
        "Entries.2.Cidr": "3.65.30.76/32"
    }

Because of this, their is no way to iterate over the entries.

Expected Behavior

I would like to have a way to get all entries. So i can use

prefixEntries.getResponseField('Entries');

And not `prefixEntries.getResponseField('Entries.0.Cidr'); n times.
There is no way to know how many entries a list contains.

Current Behavior

Calling prefixEntries.getResponseField('Entries'); throws an error because there is no key with this name.

Reproduction Steps

export class Cidr {

    public readonly cidrs: string;

    constructor(scope: Construct, ownerId: string, listName: string) {
        const currentPrefixLists = new AwsCustomResource(scope, 'GetList', {
            onUpdate: {
                service: 'EC2',
                action: 'describeManagedPrefixLists',
                parameters: {
                    Filters: [
                        {
                            Name: 'prefix-list-name',
                            Values: [listName],
                        },
                        {
                            Name: 'owner-id',
                            Values: [ownerId]
                        }
                    ],
                },
                physicalResourceId: {},
            },
            policy: AwsCustomResourcePolicy.fromSdkCalls({ resources: AwsCustomResourcePolicy.ANY_RESOURCE }),
        });
        const prefixListId = currentPrefixLists.getResponseField('PrefixLists.0.PrefixListId');
        const prefixEntries = new AwsCustomResource(scope, 'GetCidrs', {
            onUpdate: {
                service: 'EC2',
                action: 'getManagedPrefixListEntries',
                parameters: {
                    PrefixListId: prefixListId
                },
                physicalResourceId: {},
            },
            policy: AwsCustomResourcePolicy.fromSdkCalls({ resources: AwsCustomResourcePolicy.ANY_RESOURCE }),

        });

        this.cidrs = prefixEntries.getResponseField('Entries');

    }

}

Possible Solution

Have a key for all entries, or a key with the body encoded as string.

Additional Information/Context

No response

CDK CLI Version

2.50.0 (build 4c11af6)

Framework Version

No response

Node.js Version

v18.10.0

OS

Fedora 36

Language

Typescript

Language Version

No response

Other information

I think this is similiar to the issue #9969

[peterwoodworth]

I don't think there is a way to iterate through a list returned by a custom resource - at least in your CDK code. This would require synth time logic to operate on values that aren't known until deploy time.

There might be some ways to accomplish this with a custom resource using your own provider, or with multiple stacks. However I can't think of a clean way to do this off the top of my head.

I'm going to convert this to a discussion since this could have some user solution 🙂

[mrpackethead]

This might be ugly, pass the number of items in the list as a seperate attribute.

return {
		'Data': {
			'Targets': targets,
			'TargetCount': len(targets)
		}
	}

let proxyTargets: elbv2_targets.IpTarget[] = []
    for(var i = 0; i < parseInt(proxyEndpointIP.getAttString('TargetCount')); i++ ) {
      proxyTargets.push(new elbv2_targets.IpTarget(proxyEndpointIP.getAttString(`Targets.${i}`)))
    }

[tiagolisalves]

In my case, I ended up on this solution, but the number is not parsed correctly with parseInt. Sad. CDK should have an intrinsic function with a foreach functionality. I think it's the case, the parseInt doesn't process it correctly because at deployment time it is a Token.

[peterwoodworth]

There isn't a good solution here, I think the easiest, best solution we have is as follows for a list of arbitrary length:

Forego the custom resource entirely. Instead you will first write a shell script that retrieves the list and writes it to a JSON file. Then, read that JSON file in your cdk app and do the loop there

There are a number of problems with this however. Namely, the list cannot be deployed as part of your app - it has to be deployed before you synth. Additionally, if the list ever changes you will need to manually invalidate it. And lastly, if you need to do this in multiple accounts then you need to do have credentials to all of them during synth

[hrsh7th]

I may be facing a problem related to this issue.

My use case is to build ElastiCache in CDK and also create Alarm for it.

In this case, the developer gives numCacheClusters explicitly.
However, the process with custom resources will still fail.

In the example below, numCacheClusters is used to reference custom resources response.
It works when decreasing numCacheClusters, but does not seem to work when increasing numCacheClusters.

I think that if this behavior could be supported, it would really expand the possibilities of AWS CDK.
If anyone knows of a solution, I would really appreciate it if you could provide it. Thanks in advance.

class ExampleStack extends Stack {
  constructor(scope: Construct, id: string) {
    super(scope, id);

    // create CfnReplicationGroup.
    const replicationGroup = new CfnReplicationGroup(this, 'ReplicationGroup', {
      ...
      numCacheClusters: 3,
      ...
    });

    // get CfnReplicationGroup details.
    const action = {
      physicalResourceId: PhysicalResourceId.of(`${this.group.ref}-resource`),
      service: 'ElastiCache',
      action: 'describeReplicationGroups',
      parameters: {
        ReplicationGroupId: replicationGroup.ref,
      },
      outputPaths: ['ReplicationGroups']
    }
    const replicationGroupResource = new AwsCustomResource(this, 'ReplicationGroupResource', {
      policy: AwsCustomResourcePolicy.fromSdkCalls({
        resources: AwsCustomResourcePolicy.ANY_RESOURCE
      }),
      onCreate: action,
      onUpdate: action,
      onDelete: action,
    });
    replicationGroupResource.addDependency(replicationGroup)

    // create Alarm for each CacheCluster.
    for (const cacheClusterIndex = 0; i < replicationGroup.numCacheClusters; i++) {
      const cacheClusterId = replicationGroupResource.getResponseField(`ReplicationGroups.0.MemberClusters.${cacheClusterIndex}`);
      const alarm = new Alarm(this, `Alarm${cacheClusterIndex}`, {
        ...
        // use `cacheClusterId` in dimensionsMap
        ...
      })
      alarm.addDependency(replicationGroupResource)
    }
  }
}

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.