Using Envoy's internal redirect for fun, glory and authentication
You have assets, they are in s3/object storage, but they shouldn't be accessible to everyone on the internet. We could serve them all through our Backend which can authenticate the request, but streaming files can be a big performance drain.
One way of handling this little dilemma would be to hand off the file streaming to our reverse proxy while keeping auth in our backend. Nginx supports this via internal locations and the the X-Accel-Redirect header, described in Nginx X-Accel Explained (Archive).
However, I'm generally using Envoy as my reverse proxy these days and was looking for a way to achieve the same. Here's the general flow we're aiming to accomplish, with the proxy in the path of serving the assets.
The moving parts are very simple, we have our Envoy Proxy, a Backend, and our object storage.
- A request for
/assets/1.jpgcomes in from a user - Envoy proxies this to the backend service
- The backend service can read headers and cookies, check session storage, or do whatever else it needs to authenticate and authorize the request, then it will do one of two things:
- Returns a 307 redirect to the actual location of the file, e.g
s3://my-asset-bucket/assets/1 - Returns 403 Forbidden, causing the request to fail
- Returns a 307 redirect to the actual location of the file, e.g
- Envoy response from the backend, and we've configured it to handle any 307 redirects with another request to the new location in our bucket
- Object storage returns the asset, and Envoy streams it to the end user
Simple, with few moving parts and agnostic to any Cloud Provider and object storage solution you might be using.
Configuring Envoy
First, we need a backend API and an Envoy configuration. Let's start with Envoy. We'll tweak a few things to make this possible:
- Setting
internal_redirect_action: 'HANDLE_INTERNAL_REDIRECT', for reasons that feel pretty obvious - Configuring
internal_redirect_policyto permit our redirects - Configuring a way to check if it was an internal redirect or not. In this case, we'll make use of a new header
X-Internal-Redirect, that shouldn't be able to be spoofed by any clients
Here's the Envoy configuration for our listener:
static_resources:
listeners:
- name: asset_listener
address:
socket_address:
address: 0.0.0.0
port_value: 10000
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: asset_ingress_http
route_config:
name: asset_route
# We create a header to denote internal redirect,
# and critically we mark it as internal so clients can't spoof it
internal_only_headers:
- "X-Internal-Redirect"
virtual_hosts:
- name: asset_service
domains: ["*"]
routes:
# This is the public route, which listents to any internal redirects
- match:
prefix: "/assets/"
route:
cluster: application_service
internal_redirect_action: 'HANDLE_INTERNAL_REDIRECT'
internal_redirect_policy:
max_internal_redirects: 1
redirect_response_codes: [307]
response_headers_to_copy:
- 'X-Internal-Redirect'
# Our internal route, which will only match if both route and header exist
- match:
prefix: "/internal-assets/"
headers:
- name: "X-Internal-Redirect"
exact_match: "true"
route:
cluster: s3_backend
# If we're using path style buckets, add the bucket to the path, otherwise '/' to remove the assets prefix depending on bucket structure.
prefix_rewrite: "/asset-bucket/"
http_filters:
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters: # [{name: application_service, ...}, {name: s3_backend, ...}]
The configuration itself is pretty verbose, but the changes we had to are quite small. Now let's build a super simple backend API just to demonstrate, what is important is that it adds the header and sets the correct redirect.
import { Elysia } from "elysia";
const app = new Elysia()
.onRequest(({ request }) => {
console.log(request.method, request.url);
})
.get("/assets/:id", ({ set, redirect, error, params: { id } }) => {
// Check for authentication/authorization
if (id === "some-test-id-unauthorized") {
return error(403, "Forbidden");
}
// redirect to our an internal route
set.headers["x-internal-redirect"] = "true";
// for Envoy to perform the internal redirect, we need a fully qualified URL that matches the same virtual host
return redirect(`http://localhost:8000/internal-assets/${id}`, 307);
})
.listen(8000);
console.log(`Server is running at on port ${app.server?.port}...`);
Finally, we omitted the cluster definition from the Envoy configuration. If you're not familiar, Envoy decouples listening and transforming incoming requests from actually serving them to an upstream service. You might have multiple listeners on different ports and domains sending data to the same service in the end.
For local testing, I've been using SeaweedFS to pose as S3, without any authentication. But in a real setup, our requests need to be authenticated or we'll be looking forward to some permissions errors from S3. In the example below we're using the AwsRequestSigning filter. This filter can read metadata from ECS, EC2, environment variables and a few other credential sources to authenticate.
static_resources:
listeners: # [...] from above
clusters: # [{name: application_service, ...}, {name: s3_backend, ...}]
# Our backend API
- name: application_service
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: application_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal
port_value: 8000
# Connect to a local s3 provider, e.g Minio or Seaweedfs
- name: s3_backend
connect_timeout: 0.25s
type: STRICT_DNS
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: s3_backend
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: host.docker.internal
port_value: 8333
# Alternatively, connect to a s3 bucket running in AWS which requires authentication
- name: s3_backend_aws
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: s3_backend_aws
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: <some-cool-bucket>.s3.amazonaws.com
port_value: 443
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
upstream_http_protocol_options:
auto_sni: true
auto_san_validation: true
auto_config:
http2_protocol_options: {}
http_filters:
# Add the transformation/filter that will sign our requests to AWS S3
- name: envoy.filters.http.aws_request_signing
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning
service_name: "s3"
region: "eu-west-1"
host_rewrite: <some-cool-bucket>.s3.amazonaws.com
use_unsigned_payload: true
match_excluded_headers:
- prefix: x-envoy
- prefix: x-forwarded
- exact: x-amzn-trace-id
- name: envoy.filters.http.upstream_codec
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.upstream_codec.v3.UpstreamCodec
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
It is possible to add the signing into the listener configuration as well, but it requires more care to not mutate any headers (like the host) that would invalidate it after having signed the request.
Conclusion
So that's it, pretty quick and easy. There are other ways of achieving the same result. In the redirect handler we could generate a pre-signed URL for the asset in S3 with some short validity. We didn't talk about uploads, but pre-signed URLs is probably also how you would handle them.
If you are on AWS, another way would be to use CloudFront signed cookies to grant access to specific assets or groups of them.
But the reason I like this solution is that it's cloud agnostic and has few moving parts, making it easy to understand and maintain (or at least to me). I also like being able to tap into the rich variety of Envoy filters and transforms. For example, it would be possible to add a simple HTTP cache after authentication, or move authentication to Envoy itself via JWTs.
Since every request is authenticated, we are also not at the mercy of the timing which our signatures expire at. Depending on your usecase this may or may not be so important.
Though you might be using another proxy (perhaps unlikely if you made it this far). Nginx has internal redirects covered with X-Accel/X-Sendfile, and Caddy should be able to use the intercept directive. As far as I can see it's not possible using Traefik, but people are asking on Github (Add middleware to allow one service to redirect to another by a response header #5154).
Another difference is that Envoy has the AWS request signing built-in while the others don't. While you might be able to add signing to your Backend and copy the headers over it is an extra bit of complexity, that I prefer not to handle.
That's it for this time, for any questions send me a hi at this domain.