-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
‼️ NOTICE: new S3 Buckets trying to use ACLs fail to create if not explicitly opted in #25358
Comments
@corymhall Thank you for summarizing this problem! NITS: I found the sentence |
…S3 defaults (#812) Related to aws/aws-cdk#25358 Resolves #806 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
All CDK versions <2.60.0 are impacted (that's when we introduced the feature flag). Since we can't check on feature flags in the CLI notices, alert only the definitely impacted crowd.
All CDK versions <2.60.0 are impacted (that's when we introduced the feature flag). Since we can't check on feature flags in the CLI notices, alert only the definitely impacted crowd. Fixes #
Still failing with cdk 2.77. Tring to create a bucket with |
@alexhddev did you upgrade the version of |
@corymhall yes. Currenlty I solved the issue by using the blockPublicAccess option when creating the bucket:
|
I had tried this and upgraded to what was JUST released today even after trying 77 and now on 78. Same errors when applying The only way I could even work around it was to deploy the bucket with the privacy settings configured, change them through console, even though they aren't configured from default there in the global settings. Then to redeploy with my cloudfront association to the bucket. Which there was no errors there. This type of breakage with the CDK, with S3, which is considered something that is managed with stability, is honestly very frustrating and quite embarrassing. Wasted significant time reading everything that indicated there should not have been an issue with explicit settings defined, WITH an administrator access account even, to eliminate all other potential permission issues. That said, for anyone else running into this issue, you might as well save yourself the 3 hours of reading broken documentation until these bugs that are inevitably popping up because it only affects NEWLY created buckets, which will eventually resolved. Again, this was probably my most frustrating experience to date with CDK and AWS, and I have been using it since 2018, prior to version 1.0.0. We are now at version 2.0.0. These types of things really are, as I said, beyond frustrating. ^ Rant aside, with relevant info. If you are trying to save yourself time, follow the above ^. EDIT:
for this to work. I also assume publicReadAccess: true potentially as well. |
@thekevshow are you saying that this configuration will not work? new s3.Bucket(this, 'Bucket', {
blockPublicAccess: {
blockPublicAcls: false,
blockPublicPolicy: false,
ignorePublicAcls: false,
restrictPublicBuckets: false
},
accessControl: s3.BucketAccessControl.PUBLIC_READ,
objectOwnership: s3.ObjectOwnership.OBJECT_WRITER,
}) |
For the time being, I managed to solve the issue overriding the S3 resource property like the below: bucket = aws_s3.Bucket(...)
# PublicAccessBlockConfiguration.BlockPublicPolicy property override
s3_resource = bucket.node.default_child
s3_resource.add_override("Properties.PublicAccessBlockConfiguration.BlockPublicPolicy", "False") |
In https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html it's described how since april 2023, new bucket are created with ACLs disabled. Cloudfront still requires them, so it needs to be set. In aws/aws-cdk#25358, specifically under 'Using an S3 Bucket for CloudFront logs' there's a solution posted. This implements that option, to prevent the cloudformation error: ""Invalid request provided: AWS::CloudFront::Distribution: The S3 bucket that you specified for CloudFront logs does not enable ACL access"
|
Edit: Created the following issue to re-track this bug + suggested ways of improving the CDK to prevent this issue properly: I'm running into what I think is this same issue currently:
Which also seems to correspond with the issues described (and solved) on this Reddit thread: My CDK code looks like this: const siteBucket = new Bucket(this, 'SiteBucket', {
bucketName: appDomain,
websiteIndexDocument: 'index.html',
websiteErrorDocument: 'error.html',
publicReadAccess: true,
cors: [
{
allowedOrigins: ['*'],
allowedMethods: [HttpMethods.GET],
},
],
// The default removal policy is RETAIN, which means that cdk destroy will not attempt to delete
// the new bucket, and it will remain in your account until manually deleted. By setting the policy to
// DESTROY, cdk destroy will attempt to delete the bucket, but will error if the bucket is not empty.
removalPolicy: RemovalPolicy.DESTROY, // NOT recommended for production code
})
// ..snip..
const distribution = new CloudFrontWebDistribution(
this,
'SiteDistribution',
{
viewerCertificate: {
aliases: [appDomain],
props: {
acmCertificateArn: certificate.certificateArn,
sslSupportMethod: SSLMethod.SNI,
minimumProtocolVersion: SecurityPolicyProtocol.TLS_V1_1_2016,
},
},
viewerProtocolPolicy: ViewerProtocolPolicy.REDIRECT_TO_HTTPS,
originConfigs: [
{
s3OriginSource: {
s3BucketSource: siteBucket,
},
behaviors: [{ isDefaultBehavior: true }],
},
],
errorConfigurations: [
{
errorCode: 403,
responseCode: 200,
responsePagePath: '/index.html',
},
{
errorCode: 404,
responseCode: 200,
responsePagePath: '/index.html',
},
],
}
)
// ..snip..
new BucketDeployment(this, 'Deploy', {
sources: [
Source.asset('../REDACTED1'),
Source.asset('../REDACTED2'),
],
destinationBucket: siteBucket,
distribution,
distributionPaths: ['/*'],
}) Focussing in on the const siteBucket = new Bucket(this, 'SiteBucket', {
// ..snip..
publicReadAccess: true,
// ..snip..
}) Using --- Resources:
+++ Resources:
@@ -32,6 +32,27 @@
DeletionPolicy: Delete
Metadata:
aws:cdk:path: REDACTED/SiteBucket/Resource
+ SiteBucketPolicy3AC1D0F8:
+ Type: AWS::S3::BucketPolicy
+ Properties:
+ Bucket:
+ Ref: SiteBucket397A1860
+ PolicyDocument:
+ Statement:
+ - Action: s3:GetObject
+ Effect: Allow
+ Principal:
+ AWS: "*"
+ Resource:
+ Fn::Join:
+ - ""
+ - - Fn::GetAtt:
+ - SiteBucket397A1860
+ - Arn
+ - /*
+ Version: "2012-10-17"
+ Metadata:
+ aws:cdk:path: REDACTED/SiteBucket/Policy/Resource
SiteDistributionCFDistribution209CF7F5:
Type: AWS::CloudFront::Distribution
Properties: Given this seems to be incompatible with the new defaults on AWS, I would expect that this should do some measure of:
See Also |
Same problem here. Does someone has a working solution to deploy a public S3 bucket with CDK ? |
Did it work on your side ? new s3.Bucket(this, 'Bucket', {
blockPublicAccess: {
blockPublicAcls: false,
blockPublicPolicy: false,
ignorePublicAcls: false,
restrictPublicBuckets: false
},
accessControl: s3.BucketAccessControl.PUBLIC_READ,
objectOwnership: s3.ObjectOwnership.OBJECT_WRITER,
}) |
@Aarbel Did you look at the stuff in #25983 ? Eg. From the background research I did on that issue, I linked to this PR: Which seemed to just be able to add So I suspect the solution may be as simple as: new s3.Bucket(this, 'Bucket', {
publicReadAccess: true,
objectOwnership: s3.ObjectOwnership.OBJECT_WRITER,
}) But I haven't explicitly tested it. That's also based on this:
Which to me, implies that just explicitly setting We can see here that
Though if that doesn't work, then this Reddit answer suggests this: const bucket = new s3.Bucket(this, 'WebsiteBucket', {
// your other bucket configurations
publicReadAccess: true,
blockPublicAccess: new s3.BlockPublicAccess({
blockPublicAcls: false,
ignorePublicAcls: false,
blockPublicPolicy: false,
restrictPublicBuckets: false,
}), We can see what each of the Edit: To add some more 'concrete evidence' (rather than assumptions) of what each of these options does (on CDK Option 1: new Bucket(this, 'TestPublicBucket', {
publicReadAccess: true,
}) TestPublicBucket87623E53:
Type: AWS::S3::Bucket
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Resource
TestPublicBucketPolicyB64BAA28:
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: TestPublicBucket87623E53
PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Principal:
AWS: "*"
Resource:
Fn::Join:
- ""
- - Fn::GetAtt:
- TestPublicBucket87623E53
- Arn
- /*
Version: "2012-10-17"
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Policy/Resource Option 2: new Bucket(this, 'TestPublicBucket', {
publicReadAccess: true,
+ objectOwnership: ObjectOwnership.OBJECT_WRITER,
}) TestPublicBucket87623E53:
Type: AWS::S3::Bucket
+ Properties:
+ OwnershipControls:
+ Rules:
+ - ObjectOwnership: ObjectWriter
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Resource
TestPublicBucketPolicyB64BAA28:
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: TestPublicBucket87623E53
PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Principal:
AWS: "*"
Resource:
Fn::Join:
- ""
- - Fn::GetAtt:
- TestPublicBucket87623E53
- Arn
- /*
Version: "2012-10-17"
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Policy/Resource Option 3: new Bucket(this, 'TestPublicBucket', {
publicReadAccess: true,
+ blockPublicAccess: new BlockPublicAccess({
+ blockPublicAcls: false,
+ ignorePublicAcls: false,
+ blockPublicPolicy: false,
+ restrictPublicBuckets: false,
+ }),
}) TestPublicBucket87623E53:
Type: AWS::S3::Bucket
+ Properties:
+ PublicAccessBlockConfiguration:
+ BlockPublicAcls: false
+ BlockPublicPolicy: false
+ IgnorePublicAcls: false
+ RestrictPublicBuckets: false
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Resource
TestPublicBucketPolicyB64BAA28:
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: TestPublicBucket87623E53
PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Principal:
AWS: "*"
Resource:
Fn::Join:
- ""
- - Fn::GetAtt:
- TestPublicBucket87623E53
- Arn
- /*
Version: "2012-10-17"
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Policy/Resource Option 4: new Bucket(this, 'TestPublicBucket', {
- publicReadAccess: true,
+ blockPublicAccess: {
+ blockPublicAcls: false,
+ ignorePublicAcls: false,
+ blockPublicPolicy: false,
+ restrictPublicBuckets: false,
+ },
+ accessControl: BucketAccessControl.PUBLIC_READ,
}) TestPublicBucket87623E53:
Type: AWS::S3::Bucket
+ Properties:
+ AccessControl: PublicRead
+ OwnershipControls:
+ Rules:
+ - ObjectOwnership: ObjectWriter
+ PublicAccessBlockConfiguration:
+ BlockPublicAcls: false
+ BlockPublicPolicy: false
+ IgnorePublicAcls: false
+ RestrictPublicBuckets: false
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Resource
-TestPublicBucketPolicyB64BAA28:
- Type: AWS::S3::BucketPolicy
- Properties:
- Bucket:
- Ref: TestPublicBucket87623E53
- PolicyDocument:
- Statement:
- - Action: s3:GetObject
- Effect: Allow
- Principal:
- AWS: "*"
- Resource:
- Fn::Join:
- - ""
- - - Fn::GetAtt:
- - TestPublicBucket87623E53
- - Arn
- - /*
- Version: "2012-10-17" Option 5: new Bucket(this, 'TestPublicBucket', {
- publicReadAccess: true,
+ blockPublicAccess: {
+ blockPublicAcls: false,
+ ignorePublicAcls: false,
+ blockPublicPolicy: false,
+ restrictPublicBuckets: false,
+ },
+ accessControl: BucketAccessControl.PUBLIC_READ,
+ objectOwnership: ObjectOwnership.OBJECT_WRITER,
}) TestPublicBucket87623E53:
Type: AWS::S3::Bucket
+ Properties:
+ AccessControl: PublicRead
+ OwnershipControls:
+ Rules:
+ - ObjectOwnership: ObjectWriter
+ PublicAccessBlockConfiguration:
+ BlockPublicAcls: false
+ BlockPublicPolicy: false
+ IgnorePublicAcls: false
+ RestrictPublicBuckets: false
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Resource
-TestPublicBucketPolicyB64BAA28:
- Type: AWS::S3::BucketPolicy
- Properties:
- Bucket:
- Ref: TestPublicBucket87623E53
- PolicyDocument:
- Statement:
- - Action: s3:GetObject
- Effect: Allow
- Principal:
- AWS: "*"
- Resource:
- Fn::Join:
- - ""
- - - Fn::GetAtt:
- - TestPublicBucket87623E53
- - Arn
- - /*
- Version: "2012-10-17" Option 6: new Bucket(this, 'TestPublicBucket', {
publicReadAccess: true,
+ blockPublicAccess: {
+ blockPublicAcls: false,
+ ignorePublicAcls: false,
+ blockPublicPolicy: false,
+ restrictPublicBuckets: false,
+ },
+ accessControl: BucketAccessControl.PUBLIC_READ,
+ objectOwnership: ObjectOwnership.OBJECT_WRITER,
}) TestPublicBucket87623E53:
Type: AWS::S3::Bucket
+ Properties:
+ AccessControl: PublicRead
+ OwnershipControls:
+ Rules:
+ - ObjectOwnership: ObjectWriter
+ PublicAccessBlockConfiguration:
+ BlockPublicAcls: false
+ BlockPublicPolicy: false
+ IgnorePublicAcls: false
+ RestrictPublicBuckets: false
UpdateReplacePolicy: Retain
DeletionPolicy: Retain
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Resource
TestPublicBucketPolicyB64BAA28:
Type: AWS::S3::BucketPolicy
Properties:
Bucket:
Ref: TestPublicBucket87623E53
PolicyDocument:
Statement:
- Action: s3:GetObject
Effect: Allow
Principal:
AWS: "*"
Resource:
Fn::Join:
- ""
- - Fn::GetAtt:
- TestPublicBucket87623E53
- Arn
- /*
Version: "2012-10-17"
Metadata:
aws:cdk:path: REDACTED/TestPublicBucket/Policy/Resource Based on the above, I think the full command (until #25983 is fixed) might actually be: new Bucket(this, 'TestPublicBucket', {
publicReadAccess: true,
blockPublicAccess: {
blockPublicAcls: false,
ignorePublicAcls: false,
blockPublicPolicy: false,
restrictPublicBuckets: false,
},
accessControl: BucketAccessControl.PUBLIC_READ,
objectOwnership: ObjectOwnership.OBJECT_WRITER,
}) Though due to the helpers already implemented in new Bucket(this, 'TestPublicBucket', {
publicReadAccess: true,
blockPublicAccess: {
blockPublicAcls: false,
ignorePublicAcls: false,
blockPublicPolicy: false,
restrictPublicBuckets: false,
},
accessControl: BucketAccessControl.PUBLIC_READ,
}) @Aarbel Can you give that a go and see how it works for you? |
|
Nope, fails with I'm trying with both (... later) It didn't work, I think the solution above might be the correct one (and the minimum necessary), one should not set EDIT: And I thought wrong. Fails with |
…S3 defaults (awslabs#812) Related to aws/aws-cdk#25358 Resolves awslabs#806 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Now ACLs aren't enabled by default in logging buckets. So I need to specify the ACL settings myself. I don't know the right ACL settings for CloudFront distro logging buckets. But enableLogging: true creates a bucket for me. aws/aws-cdk#25358 https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudfront-readme.html#logging
What is the issue?
Amazon S3 Buckets have a property called
ObjectOwnership
. It has 3 possible settings:BucketOwnerEnforced
: access to all objects in the bucket is controlled via the bucket's Bucket Policy. Access Control Lists (ACLs) are disallowed, both on a per-object as well as bucket-wide level.ObjectWriter
: ACLs are allowed, object writers decide the ACL for each object.BucketOwnerPreferred
: ACLs are allowed, object writers can decide each object's ACL, but the default is to honor the Bucket Policy.Before April 2023, the default was to create new buckets with
ObjectOwnership: ObjectWriter
, which means it is possible to configure ACLs on the buckets and on objects by default.In April 2023, S3 has changed the default to
ObjectOwnership: BucketOwnerEnforced
, which means it is no longer possible to configure ACLs on the buckets and on objects by default.Since this change affects only the defaults for new buckets, this change has no effect on existing S3 Buckets.
Announcement: Amazon S3 will automatically enable S3 Block Public Access and disable access control lists for all new buckets starting in April 2023
What is the impact?
Depending on your CDK version and feature flags, CDK may try to configure buckets with an ACL, relying on the default
ObjectOwnership: ObjectWriter
for this to be allowed. After April 2023, buckets with ACLs will fail to deploy with the following error:What code is affected by this change?
If you meet any of the following criteria then you are not affected and can stop reading:
aws-cdk-lib >= 2.60.0
, the feature flag@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy: true
is set, and you are not using CloudFront logging; ORaws-cdk-lib >= 2.77.0
Otherwise, keep on reading to find out what patterns may lead a bucket to fail creation. You could be affected if you use
aws-cdk-lib.aws_s3.Bucket
directly, or if you use a 3rd party construct library that creates S3 Buckets.There are a couple of cases how a Bucket created by CDK may end up with an ACL configured on it. We will go over them here:
Explicit bucket ACL
If you explicitly set the ACL to one of
PUBLIC_READ
,AWS_EXEC_READ
,PUBLIC_READ_WRITE
,AUTHENTICATED_READ
, orLOG_DELIVERY_WRITE
. Example:Using a serverAccessLogsBucket
If you set one bucket as the
serverAccessLogsBucket
of another, CDK automatically configures it with permissions for writing logs. Example:If you are on a version
>= 2.60.0
and have the feature flag@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy: true
configured, CDK will use a Bucket Policy for log writing permissions, and you are not affected.Otherwise, it will use ACLs, and you are affected.
Using an S3 Bucket for CloudFront logs
When you create a CloudFront Distribution with
enableLogging: true
, the CDK will automatically create a logging bucket for you that uses ACLs:(If you are providing your own
logBucket
, you will fall into case 1, where you are explicitly configuring the bucket to use the log delivery ACL)How do I resolve this?
If you upgrade
aws-cdk-lib
to>=2.77.0
, we will automatically setObjectOwnership
for you if you configure an ACL.Only if you are not able to upgrade
aws-cdk-lib
quickly enough, see the instructions below for workarounds:Explicit ACL configuration
If you are explicitly setting the Bucket ACL then you need to now also explicitly set the ObjectOwnership to
OBJECT_WRITER
Using a serverAccessLogsBucket
If configuring server access logs and you are on
aws-cdk-lib
>=2.60.0 & <2.77.0
then set the@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy
feature flag to true. This can be done either globally in thecdk.json
file or on a per construct basis by usingsetContext
.If using CDK v1 or
aws-cdk-lib
<2.60.0
then you have to configure ObjectOwnership directly.Using an S3 Bucket for CloudFront logs
CloudFront logging still requires ACLs. If you are providing your own logging bucket to CloudFront, then you need to make sure to set the
ObjectOwnership
property.If you are using
enableLogging: true
to have a bucket automatically created, you should upgrade to2.77.0
to set theObjectOwnership
option correctly, or use escape hatches to force the option toObjectOwnership
.The text was updated successfully, but these errors were encountered: