Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Screen scraping from a web page
Message
From
12/12/2007 17:30:45
Dragan Nedeljkovich (Online)
Now officially retired
Zrenjanin, Serbia
 
 
General information
Forum:
Visual FoxPro
Category:
Other
Miscellaneous
Thread ID:
01273270
Message ID:
01275352
Views:
12
>If that website has over 40,000 records and I'm extracting the data that I mentioned, how long do you estimate it would take a program to read thru the 40,000 records?

Are you sure the admins of the website would allow this? You may just be suspected of trying out a new attack bot ;).

But seriously, assuming that you've worked out the details of that kind... the answer would depend on the speed of your connection and on the speed of the server, plus the time it may take your IE object to report that it's done loading (which may not be equal to the time it takes to display the page - depending on all the little images, possibly ads coming from other servers and who knows what. You may try to load a few hundred as a sample and see how long it took.

If the pages have too much content you don't need (ads etc) and the data you do need are easy to strextract() from the HTML, then you may just do a httpget() to retrieve only the page without any extras, and then take it from there, just analyzing text, not any DOM.

back to same old

the first online autobiography, unfinished by design
What, me reckless? I'm full of recks!
Balkans, eh? Count them.
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform