Hi Olaf,
>> a target="_top" href="http...
>Okay, a 'nice' example of what I didn't think of. But I could simply
>enhance the parsing of URLs. if I search for http or https instead of
>"a href".

>Although: If you meant document.write... with script generated links,
>than I surely have no real problem and would normally find them even
>with just the "a href" search, perhaps I already did.

To be sure you don't get too many pictures you'ld probably search only in already isolated

<a ...></a>

strings - at least this was my underlying assumption. But I was thinking of problems if the code morphed to

tablinkwrite('href="http://www.finanztip.de/tip/home/0bank.htm" title="Startseite: Bank - Geldanlage - Kredit">Bank   Home');
tablinkwrite('href="http://devisen.finanztip.de/" title="Wechselkurse - topaktuelle Deviseninformationen">Devisen   Home');

function tablinkwrite(tcLink) {
document.write('<div class="tab"><a target="_top" ' + tcLink + '<span></span></a></div>');
};

or

if IsHighDefinitionScreen() {
  lcLinkStart = '<b><a  target="_top" href="http';
  lcLinkStop = '</a></b> ';
  }
else {
  lcLinkStart = '<a  target="_top" href="http';
  lcLinkStop = '</a> ';
  }
end;
document.write(lcLinkStart + 's://www.finanztip.de/tip/home/0bank.htm" title="Startseite: Bank - Geldanlage - Kredit">Bank   Home' + lcLinkStop);

to create an example for misusing a good idea [lcLinkstart, lcLinkStop]
by including part of the protocol in linkstart <g>.

>But how about those links that are generated as result of a form
>submit? If I don't have them before I submit I can't control where they
>lead me. And I wouldn't also find these in the DOM.

But you can fill HTML-Forms automatically and follow the Post's.
And a large part of my decision to work through IE stems from the preliference of frames in the late nineties:
the frames would sometimes be selected via script depending on cookied values or hidden variables when cross
frame script was not always viewed as security breach (waaay back on IE3 and IE4).

>You're thinking about it too much as a crawler crawling
>the whole web. Of course the base class crawler should be able to do so.

Yes and no, I'ld like to have a tool usable/easily modified for every search on different "topic areas",
especially to get all possible pages created by a specific pattern used throughout the site.

>Still I wouldn't make use of IE and the DOM.
I am a firm believer in solitary confinement of crawling machines<g>.
They have fixed IP's in a subset blocked in all other machines. If I switch a machine over,
it boots/works from a special nearly empty small disk (already imaged) and
on the other disks power is disconnected. Harvested content is copied via USB disk or DVD.
So even if the crawler is infected, it doesn't spread and can be cured.
Works for me, but is probably a matter of personal taste.
You are experienced and tenacious enough to get your idea to work as well...

regards

thomas