Hi Malccolm,

ah, yes. robots.txt

Found one at http://www.spiegel.de/robots.txt

User-agent: *
Disallow: /RealMedia/
Disallow: /archiv/
Disallow: /forum/
Disallow: /cgi-bin/
Disallow: /werbung/
Disallow: /artikelversand/
Disallow: /grossbild/
Disallow: /druckbild/
Disallow: /druckrezension/
Disallow: /druckversion/
Disallow: /active/

#User-agent: Firefly/1.0
#Disallow: /

User-agent: WebReaper
Disallow: /

User-agent: Slurp
Crawl-delay: 180

Hmm, is Crawl-delay meant in seconds? With what User-agent would I be known when crawling with URLDownloadToFile? Perhaps as IE (as urlmon.dll is part of IE)? I'll check that out.

Bye, Olaf.