Hi All

We are at final beta testing stage of a VFP powered website which I was hoping some of you fellow VFP’ers might find a moment to give a test run and provide some feedback and even discover any last minute bugs or problems. Hopefully not too many of those left! ;)

Background:

The Scannery is a specialist, vertical content set, search site. It provides the ability to search the websites of specific content sets such as public companies or government websites from around the world. The target market for this website is professional researchers, analysts, journalists, investors, etc.

A few technical facts and figures:

The site tracks over 35,000+ public companies in 120 countries, of which 27,000+ have websites. 15,000+ government websites and 1800+ political party websites in over 190 countries. And every major stock exchange in the world. Additional content sets are being created.

The entire operation is run by VFP 8 application’s which maintain the database of websites, controls and manages the web spiders, schedules indexing jobs, handles the search requests from end-users, and records search results and statistics.

The site uses a commercial third party search engine and web spider which we are busy writing up a case study for - time permitting. (Apologies in advance but we do not wish to discuss this third party application at this time until our case study has been finished).

Search requests are passed from the end-user, via FoxWeb (sorry Claude :), to our VFP programs for processing. Our programs in turn use the third party search engine via a COM interface to perform the actual search. The results are then packaged, analyzed, and formatted in VFP before being sent back to the end-user's browser.

The website spiders process over a terabyte of data every 4 months and produce a searchable index of +/-100 gigabytes across all content sets. So far there are about 20 million pages indexed from +/-44,000 websites.

In addition the site provides direct links to some 1.6 million downloadable documents in various formats like XLS, DOC, and PDF files, that have been identified on the websites that have been indexed.

Please note that some websites do not allow themselves to be indexed or cannot be indexed if they have flash animation entry points or use certain types of scripting techniques. For instance IBM does not allow their site to be indexed.

The site is in final beta and is running on a humble development server during the testing phase. Be gentle :)

Thanks

In the End, we will remember not the words of our enemies, but the silence of our friends - Martin Luther King, Jr.