gRPC-Go should drop received headers containing upper ASCII (0x80+) characters

[marooou]

A have a question about a new version: v1.46.0 released on Apr 22.
The release adds a client-side validation of HTTP-invalid metadata before attempting to send (PR #4886).

One of the validation rules states that the gRPC request header value must contain one or more characters from the set [%x20-%x7E] (ref code) effectively excluding non-printable characters.

As a result, in some cases, bumping the package from v1.45 to 1.46 may become a breaking change
whenever header values pass on names and localized content (e.g. : Łukasz, Renée, Noël).

Was that an oversight or fully intentional to comply with RFC7230?

[dfawley]

The change was intended to bring us in line with the HTTP spec, but I think there is a bug in the enforcement. RFC7230 says:

     quoted-string  = DQUOTE *( qdtext / quoted-pair ) DQUOTE
     qdtext         = HTAB / SP /%x21 / %x23-5B / %x5D-7E / obs-text
     obs-text       = %x80-FF

0x80-FF seems to be there to allow for unicode, but we are rejecting it.

[marchmiel]

@dfawley Sounds reasonable to fix, I think. Is there anything I can help with to have that fixed?

[dfawley]

I did a little more digging into this. It looks like the gRPC spec itself forbids this:

https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md#requests

Custom-Metadata → Binary-Header / ASCII-Header
ASCII-Header → Header-Name ASCII-Value
ASCII-Value → 1*( %x20-%x7E ) ; space and printable ASCII

Also, gRPC-Java for sure will have trouble with these values (I'm not sure about C++/other implementations). The only reason it worked for go->go servers and clients is that both implementations assumed UTF-8, which would not be the case in other languages.

For this reason, I believe it is a bug that we were allowing these to be used before, and that the validation, as added is mostly working as intended. I do apologize for the breakage, though, which was unexpected because we did not realize that the underlying HTTP framer implementation was accepting unicode values.

Instead of transmitting the strings directly, it is recommended to percent-encode them first (and decode them on the receiving side).

However, we do still have a bug. The spec goes on to say:

The ASCII-Value character range defined is more strict than HTTP. Implementations must not error due to receiving an invalid ASCII-Value that's a valid field-value in HTTP, but the precise behavior is not strictly defined: they may throw the value away or accept the value. If accepted, care must be taken to make sure that the application is permitted to echo the value back as metadata. For example, if the metadata is provided to the application as a list in a request, the application should not trigger an error by providing that same list as the metadata in the response.

This means that on the receiving side, we also need to drop these headers, as we have no way to distinguish between metadata received over the wire and metadata created by the application.

[win5do]

Any update on this?

[jonathanrhodes]

Yeah, I just got burned by this too. Whether the original behavior was a bug or not, introducing a breaking change like this is not a small thing.

(If we're going to start silently dropping the headers on the server-side too, I'm concerned that will also lead to more confusion.)

[Patrick0308]

I'm sorry. Although I implement this validating, I getting burned by this too now. We didn't anticipate unicode is being widely used on header value.
Should do we not allow %x80-FF? Because the http RFC7230 allow it and many users pass unicode value. May be we need to modify grpc rfc?
And I agree with @jonathanrhodes. If we not allow %x80-FF, dropping invalid header on the receiving side. This covert break change is confusing.

[chowyu12]

i have the same problem, does not have a better solution

[viggin543]

any solution found besides downgrading?

[dfawley]

It should be possible to metadata key names ending with -bin, and gRPC will base64 encode/decode them on the wire.

[bananacocodrilo]

Hey @dfawley

I'm currently facing some problems due to this "If accepted, care must be taken to make sure that the application is permitted to echo the value back as metadata."

Do you know if there are any plans to get a fix for this?

On the same topic:

• I've noticed that "ASCII-Value should not have leading or trailing whitespace" is also not being validated.
  This one is not so problematic, just mentioning it in case it's not in anyone's radar.
• Also, just for reference, I've opened a similar issue in the Java implementation (Metadata: Header value validation grpc-java#11679). It might be worth tackling both issues together to reach a common place between both.

[dfawley]

@bananacocodrilo,

Do you know if there are any plans to get a fix for this?

We aren't scheduling this work right now, and I'm a bit nervous about such a behavior change, too. The only thing we can really do, given our API in Go, to follow the gRPC spec, is to start removing invalid incoming metadata. It's unclear to me whether this would be better or worse than the current state. Perhaps we need a new metadata API so that we can accept incoming invalid metadata but not permit the addition of new instances?

[ejona86]

0x80-FF seems to be there to allow for unicode

NOPE! It was there for Latin-1. Latin-1 was the only RFC-allowed use, but... Anyway, the "obs" part of the HTTP spec stands for "obsolete" and shouldn't be used.

From RFC 7230:

Historically, HTTP has allowed field content with text in the
ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
through use of [RFC2047] encoding. In practice, most HTTP header
field values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD limit their field values to
US-ASCII octets. A recipient SHOULD treat other octets in field
content (obs-text) as opaque data.

And yes, that means if you chose to support obs-text you have to define header values as bytes/bytestring in Java, Go, Python 3, and every other Unicode-enabled language. Basically nobody used RFC2047 for HTTP much; it looked like=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <[email protected]>.

[ejona86]

I'll also note quoted-string isn't what you should be looking at; that applies to only certain headers. You should be looking at field-content (obs-fold is only important to HTTP/1, so you can ignore it). VCHAR = %x21-%x7E

     header-field   = field-name ":" OWS field-value OWS

     field-name     = token
     field-value    = *( field-content / obs-fold )
     field-content  = field-vchar [ 1*( SP / HTAB ) field-vchar ]
     field-vchar    = VCHAR / obs-text