* NESTING ERROR MARKED BELOW - I CANNOT SEE WHY
******************************************
* http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds
* http://evolvingtrends.wordpress.com/2006/07/09/e-society-p2p-application/
* Gather information from blogs, comment sites, forums etc
* Process and determine what the group's postings suggest
* to process and determine near future events that MAY occur.
* Look at WEBBOT in Wikipedia - Linguists say this can be done.
* The CIA and China are doing it.
******************************************
* NetScrape :-) in Visual FoxPro
* Kudos to Tore in Norway and Mr. Strahl in Hawaii for help.
Close Tables All
Public lcHTML,cSearchExpression,nOccurrence,Captured,Caught
lcHTML = ""
cSearchExpression = ""
Captured = Date()
Caught = ""
* nOccurrence = 0
Set Default To C:\scraper
Use SEEDURLS In 1
* Use URLs In 1
Use results In 2
Use srchwrds In 3
Select SEEDURLS
Count To URLcount
? URLcount
If Type('lox')#'O'
lox=Createobject('internetexplorer.application')
Endif
lox.Visible=.F.
********** Find what we want and save it **********
For Num = 1 To URLcount && NESTING ERROR OCCURS - NEWS EYES NEED TO LOOK
Goto Num
lcURL = Seed &&location
lcURL = Transform(lcURL)
? lcURL
lnTime=1200 && 100 = 1 SECOND
lox.Navigate(lcURL)
lox.Visible=.F.
Wait 'Navigating!' Window Nowait

If !NavComplete(lox,lnTime)
??Chr(7)
Wait 'Timeout failed!' Window
lox.Visible=.F.
Release lox
Return
Endif

* Wait Clear
lcHTML = lox.Document.body.innertext
Select srchwrds
cSearchExpression = words && Words is the field name in srchwrds.dbf
* Processing chart material may be done here
**************************************************
For nOccurrence = 1 To 6
Caught = At_c(cSearchExpression,lcHTML[, nOccurrence]) && loop this to find others
Caught = Substrc(cSearchExpression,(nOccurrence - 30),60)
If Alltrim(Caught) > ""
* Get individual words and store them
Select results
Append Blank
Replace Captured With Date()
Replace Phrase With Caught
Else
If Alltrim(Caught) = ""
nOccurrence = 6
ENDIF
Caught = ""
Next nOccurrence
**************************************************
Select SEEDURLs
Next Num
Close Tables All
Release lox
Quit
**************************************************
Function NavComplete
Lpara toIE, tnTimeout
lnTimeout=Iif( Type("tnTimeout")="N",tnTimeout ,60 )
lnTimeElapsed=0
lnStartSeconds=Seconds()
Do While .T.
If toIE.ReadyState=4 And !toIE.Busy
Do While .T.
If toIE.Document.ReadyState="complete"
Return .T.
Endif
If (Seconds()-lnStartSeconds)>lnTimeout
Return .F.
Endif
Enddo
Endif
If (Seconds()-lnStartSeconds)>lnTimeout
Return .F.
Endif
Enddo
Endfunc