GetRestApis failed PROTOCOL_ERROR #1022

smugcloud · 2016-12-23T00:42:29Z

Hello,

Below is the DEBUG output from a simple GET to the GetRestApis ApiGateway operation. It appears the response code is zero. Is this a bug?

GET

func main() {
	region := "us-west-2"

	awsConfig := &aws.Config{
		Region:   &region,
		LogLevel: aws.LogLevel(aws.LogDebugWithRequestErrors),
	}

	sess, err := session.NewSession(awsConfig)
	if err != nil {
		fmt.Println("failed to create session,", err)
		return
	}

	svc := apigateway.New(sess)

	params := &apigateway.GetRestApisInput{
		Limit:    aws.Int64(500),
		Position: aws.String("String"),
	}
	resp, err := svc.GetRestApis(params)

	if err != nil {
		// Print the error, cast err to awserr.Error to get the Code and
		// Message from an error.
		fmt.Println(err.Error())
		return
	}

	// Pretty-print the response data.
	fmt.Println(resp)
}

Debug

---[ REQUEST POST-SIGN ]-----------------------------
GET http://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String HTTP/1.1
Host: apigateway.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.5.7 (go1.7.1; darwin; amd64)
Accept: application/json
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20161223/us-west-2/apigateway/aws4_request, SignedHeaders=accept;host;x-amz-date, Signature=c29f4caa9fdbd559356f5ec64bd7e77dcd00011f47825c6203ec8bc71ee13bb5
X-Amz-Date: 20161223T003241Z
Accept-Encoding: gzip


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Response apigateway/GetRestApis Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 000 status code 0
Content-Length: 0


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Send Request apigateway/GetRestApis failed, will retry, error RequestError: send request failed
caused by: Get https://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String: stream error: stream ID 1; PROTOCOL_ERROR
2016/12/22 16:32:42 DEBUG: Request apigateway/GetRestApis Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String HTTP/1.1
Host: apigateway.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.5.7 (go1.7.1; darwin; amd64)
Accept: application/json
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20161223/us-west-2/apigateway/aws4_request, SignedHeaders=accept;host;x-amz-date, Signature=90db626ee75b694c5d1f35f16d7013c66536c078d3a2bd092c632e3f5138776d
X-Amz-Date: 20161223T003242Z
Accept-Encoding: gzip


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Response apigateway/GetRestApis Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 000 status code 0
Content-Length: 0


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Send Request apigateway/GetRestApis failed, will retry, error RequestError: send request failed
caused by: Get https://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String: stream error: stream ID 3; PROTOCOL_ERROR
2016/12/22 16:32:42 DEBUG: Request apigateway/GetRestApis Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String HTTP/1.1
Host: apigateway.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.5.7 (go1.7.1; darwin; amd64)
Accept: application/json
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20161223/us-west-2/apigateway/aws4_request, SignedHeaders=accept;host;x-amz-date, Signature=90db626ee75b694c5d1f35f16d7013c66536c078d3a2bd092c632e3f5138776d
X-Amz-Date: 20161223T003242Z
Accept-Encoding: gzip


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Response apigateway/GetRestApis Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 000 status code 0
Content-Length: 0


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Send Request apigateway/GetRestApis failed, will retry, error RequestError: send request failed
caused by: Get https://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String: stream error: stream ID 5; PROTOCOL_ERROR
2016/12/22 16:32:42 DEBUG: Request apigateway/GetRestApis Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET http://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String HTTP/1.1
Host: apigateway.us-west-2.amazonaws.com
User-Agent: aws-sdk-go/1.5.7 (go1.7.1; darwin; amd64)
Accept: application/json
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20161223/us-west-2/apigateway/aws4_request, SignedHeaders=accept;host;x-amz-date, Signature=90db626ee75b694c5d1f35f16d7013c66536c078d3a2bd092c632e3f5138776d
X-Amz-Date: 20161223T003242Z
Accept-Encoding: gzip


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Response apigateway/GetRestApis Details:
---[ RESPONSE ]--------------------------------------
HTTP/0.0 000 status code 0
Content-Length: 0


-----------------------------------------------------
2016/12/22 16:32:42 DEBUG: Send Request apigateway/GetRestApis failed, not retrying, error RequestError: send request failed
caused by: Get https://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String: stream error: stream ID 7; PROTOCOL_ERROR
RequestError: send request failed
caused by: Get https://apigateway.us-west-2.amazonaws.com/restapis?limit=500&position=String: stream error: stream ID 7; PROTOCOL_ERROR```

The text was updated successfully, but these errors were encountered:

xibz · 2016-12-23T00:46:01Z

Hello @smugcloud, thank you for reaching out to us. This was a bug that has been addressed in later version of the SDK. Can you try updating to latest and see if this resolves the issue?

tmaiaroto · 2016-12-23T03:25:40Z

I'm also getting the same error today. This was, very, recently working for me. Something changed in the last day or so. I just updated to the latest package here, but it didn't seem to fix the issue.

xibz · 2016-12-23T03:34:00Z

@tmaiaroto, interesting, we haven't changed anything regarding requests in the last week or so. I wonder if this is a service issue. I'll reach out to them and see if anything looks suspicious. Are you also using the service/apigateway package?

tmaiaroto · 2016-12-23T03:41:00Z

Yes, I am (for reference: https://github.com/tmaiaroto/aegis/blob/master/cmd/up.go#L510). I'm thinking service issue as well given how frequently it was just working. It has been a few hours, but I'll also check again tomorrow.

smugcloud · 2016-12-23T04:18:40Z

Thanks @xibz. I mistakenly left off the -u when trying to upgrade before I posted this 🤦‍♂️

After properly upgrading, it is working again.

tmaiaroto · 2016-12-23T04:39:04Z

Also working for me now.

xibz · 2016-12-23T05:43:26Z

Awesome @smugcloud @tmaiaroto, great to hear! If you guys have any additional questions, please let us know.

rabbbit · 2017-11-30T09:13:40Z

@xibz do you remember when was it fixed, what was the fix? Was it related to #874?

I'm curious about the root cause, but we're also seeing this issue in [an ancient] version of Terraform (when talking to aws lambda)- it would be good to know how much we have to upgrade to get it fixed.

xibz · 2017-11-30T17:44:43Z

@rabbbit - Are you still running into the issue? From what I remember this was on the service's side and was not related to #874.

rabbbit · 2017-11-30T18:01:35Z

We are running the issue since ~2 weeks ago, on our calls to lambda - we definitely cannot create/read functions or triggers. Note that we're using Terraform 0.7, and that includes aws-sdk-go 1.14.15

…

On 30 November 2017 at 18:44, xibz ***@***.***> wrote: @rabbbit <https://github.com/rabbbit> - Are you still running into the issue? From what I remember this was on the service's side and was not related to #874 <#874>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1022 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARt0qoZT1A78F21KTVpwhm1GG2mIJh-ks5s7umTgaJpZM4LUf4K> .

xibz · 2017-11-30T18:10:16Z

@rabbbit - Could please enable HTTP2 debugging? This can be enabled by using GODEBUG=http2debug=2

rabbbit · 2017-11-30T18:32:11Z

Woooahh, http2? I did see HTTP 1.1 in our logs, didn't expect 2 to make a difference.

So, without the debugging enabled, we were seeing a lot of:

2017/11/27 10:01:53 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/27 10:01:53 [DEBUG] [aws-sdk-go] Request body type has been overwritten. May cause race conditions

and

2017/11/27 10:00:12 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/27 10:00:12 [DEBUG] [aws-sdk-go] DEBUG: Response lambda/GetFunction Details:
2017/11/27 10:00:12 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/11/27 10:00:12 [DEBUG] plugin: terraform: HTTP/0.0 0 status code 0
2017/11/27 10:00:12 [DEBUG] plugin: terraform: Content-Length: 0

With debugging enabled, it's:

2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport failed to get client conn for iam.amazonaws.com:443: http2: no cached connection was available                                                                                                                                                    
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport creating client conn to 54.67.50.113:443                           
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Framer 0xc82015cdc0: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760                                                                                                                             2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Framer 0xc82015cdc0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824        
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Framer 0xc82015cdc0: read SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=1
28, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215                                                                                                                         
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Framer 0xc82015cdc0: wrote SETTINGS flags=ACK len=0 
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport encoding header ":authority" = "lambda.us-west-1.amazonaws.com"    
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport encoding header ":method" = "GET"                                  
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport encoding header ":path" = "https://lambda.us-west-1.amazonaws.com/2015-03-31/functions/test"                                                                                                           
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport encoding header ":scheme" = "https"                                
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport encoding header "user-agent" = "terraform/0.7.0 aws-sdk-go/1.2.7 (go1.6.3; linux; amd64)" 

2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Framer 0xc82015cdc0: read WINDOW_UPDATE len=4 (conn) incr=2147418112         
2017/11/30 19:22:16 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:16 http2: Transport received WINDOW_UPDATE len=4 (conn) incr=2147418112                
2017/11/30 19:22:17 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:17 http2: Framer 0xc82015cdc0: read SETTINGS flags=ACK len=0
2017/11/30 19:22:17 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:17 http2: Transport received SETTINGS flags=ACK len=0
2017/11/30 19:22:17 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:17 http2: Framer 0xc82015cdc0: read RST_STREAM stream=1 len=4 ErrCode=PROTOCOL_ERROR
2017/11/30 19:22:17 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:17 http2: Transport received RST_STREAM stream=1 len=4 ErrCode=PROTOCOL_ERROR
2017/11/30 19:22:17 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:17 RoundTrip failure: stream error: stream ID 1; PROTOCOL_ERROR
2017/11/30 19:22:17 [DEBUG] plugin: terraform: aws-provider (internal) 2017/11/30 19:22:17 [DEBUG] [aws-sdk-go] DEBUG: Response lambda/GetFunction Details:
2017/11/30 19:22:17 [DEBUG] plugin: terraform: ---[ RESPONSE ]--------------------------------------
2017/11/30 19:22:17 [DEBUG] plugin: terraform: HTTP/0.0 0 status code 0
2017/11/30 19:22:17 [DEBUG] plugin: terraform: Content-Length: 0
2017/11/30 19:22:17 [DEBUG] plugin: terraform:
2017/11/30 19:22:17 [DEBUG] plugin: terraform:

xibz · 2017-11-30T20:37:15Z

@rabbbit - Looks like the RST_STREAM is being set which is canceling that stream. I am curious if this would be resolved by updating Golang. I know the language made several changes to the HTTP2 library since then. Please let us know if this resolves the issue.

rabbbit · 2017-11-30T21:16:19Z

Sure, although it might take a while before we get to upgrade terraform, it's not trivial. What did you mean by the original comment in this task though? You were suggesting an issue that was fixed on your end, right?

…

On 30 Nov 2017 21:37, "xibz" ***@***.***> wrote: @rabbbit <https://github.com/rabbbit> - Looks like the RST_STREAM is being set which is canceling that stream. I am curious if this would be resolved by updating Golang. I know the language made several changes to the HTTP2 library since then. Please let us know if this resolves the issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1022 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARt0s_7toSLePOL7icZ08xUn_A_lzqQks5s7xIBgaJpZM4LUf4K> .

xibz · 2017-11-30T21:24:45Z

@rabbbit - There was a bug regarding HTTP2 in an earlier version of the SDK. I noticed you were using a very early version of the SDK. So, updating that would be a good idea as well.

smugcloud closed this as completed Dec 23, 2016

dhoeric mentioned this issue Jan 22, 2017

Fix PROTOCOL_ERROR on aws-go-sdk/apigateway/GetRestApi mweagle/Sparta#44

Closed

derekvanassche mentioned this issue May 18, 2018

Fail to get the lambda Sydsvenskan/lambda-resource#2

Open

skotambkar pushed a commit to skotambkar/aws-sdk-go that referenced this issue May 20, 2021

Regenerate Clients to Support Union Validation (aws#1022)

a72cd14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GetRestApis failed PROTOCOL_ERROR #1022

GetRestApis failed PROTOCOL_ERROR #1022

smugcloud commented Dec 23, 2016

xibz commented Dec 23, 2016

tmaiaroto commented Dec 23, 2016

xibz commented Dec 23, 2016

tmaiaroto commented Dec 23, 2016

smugcloud commented Dec 23, 2016

tmaiaroto commented Dec 23, 2016

xibz commented Dec 23, 2016

rabbbit commented Nov 30, 2017

xibz commented Nov 30, 2017

rabbbit commented Nov 30, 2017 via email

xibz commented Nov 30, 2017

rabbbit commented Nov 30, 2017

xibz commented Nov 30, 2017

rabbbit commented Nov 30, 2017 via email

xibz commented Nov 30, 2017

GetRestApis failed PROTOCOL_ERROR #1022

GetRestApis failed PROTOCOL_ERROR #1022

Comments

smugcloud commented Dec 23, 2016

xibz commented Dec 23, 2016

tmaiaroto commented Dec 23, 2016

xibz commented Dec 23, 2016

tmaiaroto commented Dec 23, 2016

smugcloud commented Dec 23, 2016

tmaiaroto commented Dec 23, 2016

xibz commented Dec 23, 2016

rabbbit commented Nov 30, 2017

xibz commented Nov 30, 2017

rabbbit commented Nov 30, 2017 via email

xibz commented Nov 30, 2017

rabbbit commented Nov 30, 2017

xibz commented Nov 30, 2017

rabbbit commented Nov 30, 2017 via email

xibz commented Nov 30, 2017