Support for nginx style X-Accel headers in reverse_proxy

[segevfiner]

Nginx has support for an upstream server to effect it by setting X-Accel-* headers in the response (NGINX | X-Accel).

This feature is very useful for various different scenarios, for example:

1. Disabling buffering for certain requests without having to write specialized configuration in the proxy server for them. e.g. A route that returns large files (X-Accel-Buffering).
2. Offloading protected/restricted static files to the proxy server, by an internal/named location only accessible ro such internal redirects (X-Accel-Redirect).
3. Offloading certain requests to a different server proxied by Nginx. Often an internal/named location, only accessible by such internal redirects (X-Accel-Redirect).

Some of this behavior is probably possible by handle_response, which is not currently available in the Caddyfile (#3707), and even if it would be available, a stock handler/config snippet for this behavior would still be useful.

Note that there are some fine details with how nginx behaves with those headers, for example: It changes the request method to a GET unless you redirect to a named location when using X-Accel-Redirect. Both changing the method (e.g. A POST that redirects to a static file), and not changing it (e.g. Offload to a different proxied server) are useful and need consideration how to make them available in caddy.

It also doesn't seem like Caddy has such named locations or internal locations/matcher to create such internal redirect only routes.

Also another interesting custom header of this sort is X-Sendfile supported by Apache.

[segevfiner]

I recently encountered a bad interaction between X-Accel-Redirect and request buffering in Nginx. If you disable request buffering, Nginx just doesn't send the body to the redirected upstream with no error on its side, with no obvious way to direct it to hold off on the body to send after the redirect.

Nginx is always such a pain in the ass...

[mholt]

Caddy's JSON config is capable of supporting very nuanced routes and handling logic, so I wouldn't worry about that. The functionality, if it does not already exist, just needs to be implemented.

This is a pretty vague description of everything though, and there seems to be a lot to unpack. Frankly, I'm not really sure what you're getting at. I think this is a feature request? Can you be more specific please?

[dmke]

I think this is a feature request?

Yes, indeed.

Can you be more specific please?

NGINX's X-Accel-Redirect, Apache/mod_xsendfile's X-Sendfile, or lighttpd's X-Lighttpd-Send-File allows reverse-proxy'd applications to hand-over a file reference (i.e. a path) to the proxying web server, which in turn answers the clients request as if the client would have requested the file itself.^{(modulo some caveats, see below)}

I'm using X-Sendfile (and Apache terms) following description, because that's what I've used the most in the past.

Rack specifics

In Ruby/Rack applications (like Rails), I can configure which kind of header the application should use. The default config for production contains these lines:

Rails.application.configure do
  # Specifies the header that your server uses for sending files.
  # config.action_dispatch.x_sendfile_header = 'X-Sendfile' # for Apache
  # config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for NGINX
end

The nitty-gritty details can be found in Rack::Sendfile middleware. There, @app.call(env) is the result of the Rails controller action, and env is a context hash containing the HTTP response headers (among others).

Use cases

There are two reasons why this is useful:

1. The application's thread pool is not blocked by slow clients (Ruby and Python use an interpreter lock, which is annoying). By handing off the actual download to the web server, the current request is finished, the thread becomes available again and the application can handle the next request.
2. The application can grant access to otherwise inaccessible resources when the incoming request bears the proper credentials (cookies, query parameter, etc.). X-Sendfile describes it as "[it] will also happily send files that are otherwise protected," meaning it can serve files outside of the DocumentRoot, or files which have a Deny from all directive applied.

There might be more, but I believe this are the main reasons.

Protocol overview

The protocol (on the application side) is quite simple: The application just finishes a response without a body:

HTTP/1.1 200 OK
X-Sendfile: /path/to/some/file.ext

That HTTP response from the proxied application is intercepted by the web server, and rewritten to something like this, if an X-Sendfile header is present:

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Disposition: attachment; filename="file.ext"
Content-Length: 1234567

1234567 bytes omitted

Some header fields are filled in when not provided by the application (IIRC, Content-Type, -Length and -Disposition), others are removed (X-Sendfile obviously), and the rest is copied without changes.

Caveats

As described above, the X-Sendfile header circumvents access protection (on purpose).

Example deployment

Given a Caddyfile like this:

example.com {
  root * /srv/app/public

  # copied from one of our production systems;
  # not sure if minimal enough - I've cut large parts
  @notStatic {
    not {
      file {
        try_files {path}
      }
    }
    not {
      path /assets/*
    }
  }

  reverse_proxy @notStatic http://127.0.0.1:3000
  file_server
}

and a file structure like

/srv/app/
+-- public/
|   +-- 404.html
|   +-- assets/
|       +-- app.js
|       +-- style.css
+-- data/files/
    +-- secret.txt

a request to app.js would succeed (GET /assets/app.js), while there's no way to get to data/files/secret.txt.

However, the application server might set X-Sendfile: /srv/app/data/files/secret.txt when requesting GET /downloads/secret.txt, and the web server should provide that file.

[mholt]

From what you've described, I believe you can already accomplish this with handle_response in the reverse_proxy handler: https://caddyserver.com/docs/json/apps/http/servers/routes/handle/reverse_proxy/handle_response/

List of handlers and their associated matchers to evaluate after successful roundtrips. The first handler that matches the response from a backend will be invoked. The response body from the backend will not be written to the client; it is up to the handler to finish handling the response.

If I were you, I'd just match on the expected status code and header, then chain together whatever handlers you need to finish handling the request.

[dmke]

Cool, I'll have a look later.

From a first glance, this isn't available from a Caddyfile, only by in the JSON config tree, correct?

[mholt]

@dmke Track #3712

[Richard87]

Following along,

I would love to use this together with my php-fpm upstream, and redirect the file download to s3proxy when applicable 🤔

[francislavoie]

@Richard87 you'll be able to do this with #4021 (or build with that now to play with it). The functionality to do this already exists in Caddy via JSON config, this just adds Caddyfile support.

[mholt]

#4021 has been merged.