Immer mehr Content im Web ist vor Scraping geschützt. Die Anbieter wollen dadurch vor allem die „hungrigen“ AI-Bots aussperren.
Einer dieser Content Delivery Network (CDN) Anbieter, der solchen Schutz offeriert, ist AKAMAI, siehe z.B.: https://www.akamai.com/solutions/ai-scraper-bot-protection
Versucht man beispielsweise folgendes PDF-Dokument mittels wget zu laden, wird man lediglich mittels eines ERROR 403: Forbidden. abgewiesen:
root@myVM5:/# wget -nv https://www.fidelityinternational.com/FILPS/Documents/de/current/mf.de.xx.LU1048684796.pdf
https://www.fidelityinternational.com/FILPS/Documents/de/current/mf.de.xx.LU1048684796.pdf:
2026-08-21 17:49:05 ERROR 403: Forbidden.
In diesem Fall ist der Grund der Abweisung aber keine Captcha-Prüfung, fehlende Cookies oder JavaScript-Ausführung. Denn wenn wir die URL mit dem PDF-Dokument in einem frischen InPrivate-Fenster abrufen (als einzigen HTTP-Request), dann erhalten wir das Dokument.
Somit kann die Abweisung nur auf Basis der HTTP-Request-Header erfolgen. Die Lösung ist somit, sich diese von einem funktionierenden Browser 1:1 zu kopieren. Hierzu öffnen wir in einem frischen InPrivate-Fenster im Browser (Edge, Chrome, Firefox – hier am Beispiel von Edge) mittels F12 die Entwickler-Konsole, öffnen das „Networking“ Tab und rufen die URL ab:
Die Header kopieren wir in hierzu passende wget Commandline-Argumente, diese sehen dann wie folgt aus:
wget --debug \
--content-on-error \
--header='Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7' \
--header='Accept-Encoding: gzip, deflate, br, zstd' \
--header='accept-language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6' \
--header='cache-control: no-cache' \
--header='dnt: 1' \
--header='pragma: no-cache' \
--header='priority: u=0, i' \
--header='sec-ch-ua: "Not=A?Brand";v="99", "Microsoft Edge";v="151", "Chromium";v="151"' \
--header='sec-ch-ua-mobile: ?0' \
--header='sec-ch-ua-platform: "Windows"' \
--header='sec-fetch-dest: document' \
--header='sec-fetch-mode: navigate' \
--header='sec-fetch-site: none' \
--header='sec-fetch-user: ?1' \
--header='upgrade-insecure-requests: 1' \
--user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36 Edg/151.0.0.0" \
https://www.fidelityinternational.com/FILPS/Documents/de/current/mf.de.xx.LU1048684796.pdf
Und voila – funktioniert:
Setting --content-on-error (contentonerror) to 1
Setting --header (header) to Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Setting --header (header) to Accept-Encoding: gzip, deflate, br, zstd
Setting --header (header) to accept-language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
Setting --header (header) to cache-control: no-cache
Setting --header (header) to dnt: 1
Setting --header (header) to pragma: no-cache
Setting --header (header) to priority: u=0, i
Setting --header (header) to sec-ch-ua: "Not=A?Brand";v="99", "Microsoft Edge";v="151", "Chromium";v="151"
Setting --header (header) to sec-ch-ua-mobile: ?0
Setting --header (header) to sec-ch-ua-platform: "Windows"
Setting --header (header) to sec-fetch-dest: document
Setting --header (header) to sec-fetch-mode: navigate
Setting --header (header) to sec-fetch-site: none
Setting --header (header) to sec-fetch-user: ?1
Setting --header (header) to upgrade-insecure-requests: 1
Setting --user-agent (useragent) to Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36 Edg/151.0.0.0
DEBUG output created by Wget 1.25.0 on linux-gnu.
Reading HSTS entries from /root/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'mf.de.xx.LU1048684796.pdf' (UTF-8) -> 'mf.de.xx.LU1048684796.pdf' (UTF-8)
--2026-08-21 17:57:30-- https://www.fidelityinternational.com/FILPS/Documents/de/current/mf.de.xx.LU1048684796.pdf
Certificates loaded: 121
Resolving www.fidelityinternational.com (www.fidelityinternational.com)... 95.101.23.88, 95.101.23.82, 2a02:26f0:11a::5f65:1752, ...
Caching www.fidelityinternational.com => 95.101.23.88 95.101.23.82 2a02:26f0:11a::5f65:1752 2a02:26f0:11a::5f65:1758
Connecting to www.fidelityinternational.com (www.fidelityinternational.com)|95.101.23.88|:443... connected.
Created socket 3.
Releasing 0x000058dacb40c3c0 (new refcount 1).
---request begin---
GET /FILPS/Documents/de/current/mf.de.xx.LU1048684796.pdf HTTP/1.1
Host: www.fidelityinternational.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36 Edg/151.0.0.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br, zstd
Connection: Keep-Alive
accept-language: de,de-DE;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6
cache-control: no-cache
dnt: 1
pragma: no-cache
priority: u=0, i
sec-ch-ua: "Not=A?Brand";v="99", "Microsoft Edge";v="151", "Chromium";v="151"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
upgrade-insecure-requests: 1
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Content-Type: application/pdf
Last-Modified: Wed, 12 Aug 2026 17:59:09 GMT
x-amz-server-side-encryption: AES256
x-amz-version-id: RUlnfeuahwIojh0e0p9l10xJbGWAGSFp
Content-Encoding: gzip
ETag: W/"eee1c17c6b4b791d2368e3dbdaa6fbd5"
Vary: Accept-Encoding
X-Amz-Cf-Pop: LHR86-P2
X-Amz-Cf-Id: BM73rR1yu-4DNgueYxqkb-FQdXZTMr4_6tyJC7DIISrt9d0ZeXJb1g==
Date: Fri, 21 Aug 2026 17:57:31 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Connection: Transfer-Encoding
Server-Timing: cdn-cache; desc=MISS
Server-Timing: edge; dur=221
Server-Timing: origin; dur=69
Alt-Svc: h3=":443"; ma=93600
Akamai-GRN: 0.4e17655f.1787335051.777420b9
Server-Timing: ak_p; desc="1787335051010_1600460622_2004099257_28939_15535_6_258_-";dur=1
---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: unspecified [application/pdf]
Saving to: ‘mf.de.xx.LU1048684796.pdf’
mf.de.xx.LU1048684796.pdf [ <=> ] 122.48K --.-KB/s in 0.02s
2026-08-21 17:57:31 (5.54 MB/s) - ‘mf.de.xx.LU1048684796.pdf’ saved [125422]
