(cli): sso tokens not refreshed by cdk cli, refreshed by aws cli

[diranged]

Describe the bug

When using AWS SSO in the "Refreshable Token" mode the cdk ... commands do not know how to refresh the token properly, while the aws commands do.

App Versions

yarn run v1.22.19
2.69.0 (build 60a5b2a)

aws-cli/2.11.2 Python/3.11.2 Darwin/22.3.0 source/arm64 prompt/off

Expected Behavior

I would hope that the cdk commands would trigger the token refresh in the same way that the aws CLI does.

Current Behavior

Example Failure:

Here's an example of how this works... The first call to yarn cdk deploy fails:

% AWS_PROFILE=eng yarn cdk deploy MYSTACK-Pipeline/MYSTACK/MyApp
yarn run v1.22.19
$ /Users/myuser/git/myorg/myapp/node_modules/.bin/cdk deploy MYSTACK-Pipeline/MYSTACK/MyApp
There are expired AWS credentials in your environment. The CDK app will synth without current account information.
Bundling asset MYSTACK-Pipeline/Pipeline/CodePipelinePostToGitHub/Function/Code/Stage...
$ esbuild --bundle /Users/myuser/git/myorg/myapp/src/constructs/github/funcs/github_status_lambda.ts --target=node18 --platform=node --outfile=/Users/myuser/git/myorg/myapp/cdk.out/bundling-temp-f3dd5251d5b3272245fed42e3671961fe3b95239a8df323bd7858983eb7c416e/index.js --external:aws-sdk --external:@octokit/rest --external:@types/aws-lambda

  ../../../../cdk.out/bundling-temp-f3dd5251d5b3272245fed42e3671961fe3b95239a8df323bd7858983eb7c416e/index.js  1.0mb ⚠️

warning package.json: No license field
warning No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...

success Saved lockfile.

✨  Synthesis time: 6.29s

MYSTACK-Pipeline/MYSTACK/MyApp (MYSTACK-MyApp): building assets...


 ❌ Building assets failed: Error: Building Assets Failed: Error: Need to perform AWS calls for account XXXXXXXXXXXX, but no credentials have been configured
    at buildAllStackAssets (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:374:115279)
    at async CdkToolkit.deploy (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:374:143496)
    at async exec4 (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:429:51795)

Building Assets Failed: Error: Need to perform AWS calls for account XXXXXXXXXXXX, but no credentials have been configured
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Then if we just call aws sts get-caller-identity the CLI refreshes the creds..

% AWS_PROFILE=eng aws sts get-caller-identity                                       
{   
    "UserId": "AROAWB3X2QYCDHSUU7JG6:[email protected]",
    "Account": "XXXXXXXXXXXX",
    "Arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/AWSReservedSSO-Role_64dad7cc9f3c9292/[email protected]"
}

Finally a followup call to yarn cdk deploy will work fine:

% AWS_PROFILE=eng yarn cdk deploy MYSTACK-Pipeline/MYSTACK/MyApp
yarn run v1.22.19
$ /Users/myuser/git/myorg/myapp/node_modules/.bin/cdk deploy MYSTACK-Pipeline/MYSTACK/MyApp

✨  Synthesis time: 3.87s

MYSTACK-Pipeline/MYSTACK/MyApp (MYSTACK-MyApp): building assets...

[0%] start: Building 7670bc5e7ac35fd59238f64edf0326b9e70f52c545419110a9ff8f7c324b7d8a:XXXXXXXXXXXX-us-east-1
[0%] start: Building c260b700560a0f19f0f7be6abae8ab4526d4671aa5f6ff875a8ef0a9db260257:XXXXXXXXXXXX-us-east-1
[50%] success: Built 7670bc5e7ac35fd59238f64edf0326b9e70f52c545419110a9ff8f7c324b7d8a:XXXXXXXXXXXX-us-east-1
[100%] success: Built c260b700560a0f19f0f7be6abae8ab4526d4671aa5f6ff875a8ef0a9db260257:XXXXXXXXXXXX-us-east-1

MYSTACK-Pipeline/MYSTACK/MyApp (MYSTACK-MyApp): assets built

MYSTACK-Pipeline/MYSTACK/MyApp (MYSTACK-MyApp): deploying... [1/1]
[0%] start: Publishing 7670bc5e7ac35fd59238f64edf0326b9e70f52c545419110a9ff8f7c324b7d8a:XXXXXXXXXXXX-us-east-1
[0%] start: Publishing c260b700560a0f19f0f7be6abae8ab4526d4671aa5f6ff875a8ef0a9db260257:XXXXXXXXXXXX-us-east-1
[50%] success: Published c260b700560a0f19f0f7be6abae8ab4526d4671aa5f6ff875a8ef0a9db260257:XXXXXXXXXXXX-us-east-1
[100%] success: Published 7670bc5e7ac35fd59238f64edf0326b9e70f52c545419110a9ff8f7c324b7d8a:XXXXXXXXXXXX-us-east-1
MYSTACK-MyApp: creating CloudFormation changeset...

yarn cdk deploy -v ...

% yarn cdk deploy MYSTACK-Pipeline/MYSTACK/MyApp -v 
yarn run v1.22.19
$ /Users/myuser/git/myorg/myapp/node_modules/.bin/cdk deploy MYSTACK-Pipeline/MYSTACK/MyApp -v
[09:45:57] CDK toolkit version: 2.69.0 (build 60a5b2a)
[09:45:57] Command line arguments: {
  _: [ 'deploy' ],
  v: 1,
  verbose: 1,
  lookups: true,
  'ignore-errors': false,
  ignoreErrors: false,
  json: false,
  j: false,
  debug: false,
  ec2creds: undefined,
  i: undefined,
  'version-reporting': undefined,
  versionReporting: undefined,
  'path-metadata': true,
  pathMetadata: true,
  'asset-metadata': true,
  assetMetadata: true,
  'role-arn': undefined,
  r: undefined,
  roleArn: undefined,
  staging: true,
  'no-color': false,
  noColor: false,
  ci: false,
  all: false,
  'build-exclude': [],
  E: [],
  buildExclude: [],
  force: false,
  f: false,
  parameters: [ {} ],
  'previous-parameters': true,
  previousParameters: true,
  logs: true,
  concurrency: 1,
  'asset-prebuild': true,
  assetPrebuild: true,
  '$0': '/Users/myuser/git/myorg/myapp/node_modules/.bin/cdk',
  STACKS: [
    'MYSTACK-Pipeline/MYSTACK/MyApp'
  ],
  'S-t-a-c-k-s': [
    'MYSTACK-Pipeline/MYSTACK/MyApp'
  ]
}
[09:45:57] cdk.json: {
  "app": "npx ts-node -P tsconfig.json --prefer-ts-exts src/main.ts",
  "context": {
    "@aws-cdk/core:newStyleStackSynthesis": true
  },
  "output": "cdk.out",
  "build": "npx projen bundle",
  "watch": {
    "include": [
      "src/**/*.ts",
      "test/**/*.ts"
    ],
    "exclude": [
      "README.md",
      "cdk*.json",
      "**/*.d.ts",
      "**/*.js",
      "tsconfig.json",
      "package*.json",
      "yarn.lock",
      "node_modules"
    ]
  },
  "//": "~~ Generated by projen. To modify, edit .projenrc.ts and run \"npx projen\"."
}
[09:45:57] merged settings: {
  versionReporting: true,
  pathMetadata: true,
  output: 'cdk.out',
  app: 'npx ts-node -P tsconfig.json --prefer-ts-exts src/main.ts',
  context: { '@aws-cdk/core:newStyleStackSynthesis': true },
  build: 'npx projen bundle',
  watch: {
    include: [ 'src/**/*.ts', 'test/**/*.ts' ],
    exclude: [
      'README.md',
      'cdk*.json',
      '**/*.d.ts',
      '**/*.js',
      'tsconfig.json',
      'package*.json',
      'yarn.lock',
      'node_modules'
    ]
  },
  '//': '~~ Generated by projen. To modify, edit .projenrc.ts and run "npx projen".',
  debug: false,
  assetMetadata: true,
  toolkitBucket: {},
  staging: true,
  bundlingStacks: [ '**' ],
  lookups: true,
  assetPrebuild: true
}
[09:45:57] Determining if we're on an EC2 instance.
[09:45:57] Does not look like an EC2 instance.
[09:45:57] Toolkit stack: CDKToolkit
[09:45:57] Setting "CDK_DEFAULT_REGION" environment variable to us-east-1
[09:45:57] Resolving default credentials
[09:45:57] Reading cached notices from /Users/myuser/.cdk/cache/notices.json
[09:45:57] Looking up default account ID from STS
There are expired AWS credentials in your environment. The CDK app will synth without current account information.
[09:45:58] context: {
  '@aws-cdk/core:newStyleStackSynthesis': true,
  'aws:cdk:enable-path-metadata': true,
  'aws:cdk:enable-asset-metadata': true,
  'aws:cdk:version-reporting': true,
  'aws:cdk:bundling-stacks': [ '**' ]
}
[09:45:59] outdir: cdk.out
[09:45:59] env: {
  CDK_DEFAULT_REGION: 'us-east-1',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '31.0.0',
  CDK_CLI_VERSION: '2.69.0'
}
Bundling asset MYSTACK-Pipeline/Pipeline/CodePipelinePostToGitHub/Function/Code/Stage...
$ esbuild --bundle /Users/myuser/git/myorg/myapp/src/constructs/github/funcs/github_status_lambda.ts --target=node18 --platform=node --outfile=/Users/myuser/git/myorg/myapp/cdk.out/bundling-temp-f3dd5251d5b3272245fed42e3671961fe3b95239a8df323bd7858983eb7c416e/index.js --external:aws-sdk --external:@octokit/rest --external:@types/aws-lambda

  ../../../../cdk.out/bundling-temp-f3dd5251d5b3272245fed42e3671961fe3b95239a8df323bd7858983eb7c416e/index.js  1.0mb ⚠️

warning package.json: No license field
warning No license field
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 🔨  Building fresh packages...
success Saved lockfile.

✨  Synthesis time: 5.95s

MYSTACK-Pipeline/MYSTACK/MyApp (MYSTACK-Thingy): building assets...


 ❌ Building assets failed: Error: Building Assets Failed: Error: Need to perform AWS calls for account XXXXXXXXXXX, but no credentials have been configured
    at buildAllStackAssets (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:374:115279)
    at async CdkToolkit.deploy (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:374:143496)
    at async exec4 (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:429:51795)
[09:46:02] Reading cached notices from /Users/myuser/.cdk/cache/notices.json

Building Assets Failed: Error: Need to perform AWS calls for account XXXXXXXXXXX, but no credentials have been configured
[09:46:02] Error: Building Assets Failed: Error: Need to perform AWS calls for account XXXXXXXXXXX, but no credentials have been configured
    at buildAllStackAssets (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:374:115279)
    at async CdkToolkit.deploy (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:374:143496)
    at async exec4 (/Users/myuser/git/myorg/myapp/node_modules/aws-cdk/lib/index.js:429:51795)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
myuser@Matts-Air myapp % 

Reproduction Steps

See above

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.69.0 (build 60a5b2a)

Framework Version

No response

Node.js Version

n/a

OS

Mac OS

Language

Typescript

Language Version

No response

Other information

Our $HOME/.aws/config file looks like this:

[sso-session myorg]
sso_region = us-east-1
sso_start_url = https://myorg.awsapps.com/start
sso_registration_scopes = sso:account:access

[profile eng]
sso_session = myorg
sso_region = us-east-1
sso_account_id = XXXXXXXXXX
sso_role_name = Role

[pahud]

Thank you for the report. I am not sure if this is related to #24744 but we will need more investigation before we can find the root cause. Any further feedback or upvotes are appreciated.

[blytheaw]

We are experiencing this issue as well using the new refresh token approach for SSO credentials in the AWS CLI.

[diranged]

@pahud I don't know what further help you need - are there logs or other bits of information that would be useful?

[pahud]

@diranged This seems to be related to SDK and we are tracking in aws/aws-sdk#531

[spoco2]

we are having this exact issue. Using CDK all my developers are having their sessions time out after 1 hour, and have to re-run aws sso login.

Being that it's been directed over to issue 531, I'll comment on there, as it seems somewhat related, but maybe not entirely.

[benkehoe]

I'm a little concerned that the error message "Need to perform AWS calls for account XXXXXXXXXXXX, but no credentials have been configured" sounds a bit different from the error messages involved in aws/aws-sdk-js#4441

If it is in fact, giving this error message because the SDK is incorrectly saying the credentials are expired, it's worth opening a feature request for the CDK recognizing expired credentials as separate from no credentials configured, and making this error message more helpful to the user.

[diranged]

Just checking in here... this is still a pain point for us. Any progress on fixing it?

[tom-dibble-powerschool]

Anyone looking at this issue? It makes using the aws sso login approach with CDK nearly impossible. Our team has gone back to just using environment variables despite the security implications.

[TheRealAmazonKendra]

I'm fairly certain this should have been fixed when we upgraded our AWS SDK to v2.1691.0 because of aws/aws-sdk-js#4443. Can you please upgrade your cdk cli to our most recent version and see if this is still impacting you?

If it is, you should see a resolution when #31702 is merged.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.