Al, see the new thread that I started: Newer functions faster than older functions? Thread #843451 Message #843451

>>>>Hi to all who are interested in this topic.
>>>>
>>>>One of my customers has a colleague who states that the application that I've written should be written in C++, which would give a better performance. I wonder whether he's right and if not, what arguments I could use.
>>>>
>>>>The core of the app hardly uses DBMS functions. Instead, it uses low-level functions like FOPEN(), FGETS() and FPUTS(), and string manipulation functions like STRTRAN(). It reads tab delimited files, not with APPEND FROM, but with FGETS(). The reason is that the fields of the import files vary and the file size can be huge. And it also writes files with FPUTS() after having done some sophisticated string manipulation. The speed is still phenominal, thousands of records/lines are processed in a second or so. But the import files can contain 100,000,000 records or even more, so it may still take a while before they're all processed.
>>>>
>>>>The big question now is: Will C++ or Delphi do this job significantly faster??
>>>
>>>Well, I don't know about your case but I do know the last time I tried to use VFP for some "relatively" heavy string manipulation I was quite disappointed.
>>>
>>>I have a client who suspected a computer of theirs was being used after hours to browse porn sites. I copied that user's INDEX.DAT file for analysis - it was about 8MB.
>>>
>>>If you open a sample in the VFP Editor or Notepad (if you use IE there's one on your HD, in W2K/XP in \Documents and Settings\USER\Local Settings\Temporary Internet Files\Content.IE5\) you can see it's binary data but contains URLs in plain text. The one I had contained a couple of hundred URLs. I wrote a QUAD routine pulling the file into a character variable using FILETOSTR(), then using standard AT(), RIGHT() functions etc. to look for strings starting with "http://". On a 1GHz Athlon with 512MB RAM it was taking about 45 seconds per URL (in VFP7 SP1).
>>>
>>>I fired up the Coverage Profiler and found that with "large" strings the AT() and RIGHT()/SUBSTR() functions are both slow. However, I couldn't figure out a way to parse out the URLs from the character variable/string without using them. If someone can post a routine based on the technique outlined above that's reasonably fast it would be enlightening because it sure escapes me.
>>
>>Here's my very first attempt:
>>

local lcAll, lnFound, ln
>>*
>>create cursor c_urls ( url c(100), thetime t )
>>*
>>t       = seconds()
>>lcAll   = filetostr( 'index.dat' )
>>lnFound = occurs( 'http://', lcAll )
>>*
>>for ln = 1 to lnFound
>>   *
>>   lcURL = substr( lcAll, at( 'http://', lcAll, ln ), 100 )
>>   lnNIL = at( chr(0), lcURL )
>>   *
>>   if lnNIL > 0
>>      lcURL = left( lcURL, lnNIL - 1 )
>>   endif
>>   *
>>   insert into c_urls values ( lcURL, datetime() )
>>   *
>>   wait wind str( ln ) nowait
>>   *
>>   if inkey() = 113    && character q from quit
>>      exit
>>   endif
>>next
>>*
>>? ln -1, seconds() - t

>>In my case the filesize is 11 Mb and the number of urls nearly 28,000. The machine is 1.8Ghz and 256 Mb internal memory. VFP7 with SP1. The number of urls as stored in the cursor is 14 to 11 per second. It was 14 in the beginning and 11 after having processed more than 8000 urls. Then I interrupted the routine.
>>
>>I'm sure that another vfp-algorithm can do it faster, but it's not your 45 seconds per url. No, it's less than 0.1 second per url.
>>
>>
>>>I eventually used a completely different technique. I created a cursor with a 1-byte character field, used low-level file functions to pull the INDEX.DAT file into the cursor one character at a time, then used SKIP etc. to move back and forth in the "string". Still wasn't instantaneous but was a big improvement and was "fast enough" for the job.
>>>
>>>I believe the job of parsing the INDEX.DAT file should be possible to do basically instantaneously with modern tools. I remember my first IBM-type PC, a true-blue IBM PC (4.77MHz 8088, 640K RAM) on which I used to do FoxBASE+ programming. I used the BRIEF editor and regularly edited procedure files that were 200 or 300K. I could search for any string I wanted, and BRIEF could find it faster than I could get my finger off the "Find" key. On the machine I have now, an 8MB file, well, it should be child's play.
>>
>>My result (see above) would still take 45 minutes to retrieve all 28,000 urls. I'm sure this is not the final world record. But the main issue here is: Is VFP underperforming when compared to C++?
>
>Your approach is similar in basic concept to the first idea I tried (but better and faster!). However, when you look at the overall performance (45 min. on your machine) well, that's pathetic.
>
>Gregory's response was very enlightening, showing that the machine is capable of much faster performance. However, Fabio's post was also very interesting showing that in some cases VFP can pretty much keep up. I discount John Ryan's post - while interesting it doesn't really deal with string performance.
>
>I now have a theory:
>
>ALINES() is fast, and it's a relatively new function in VFP.
>AT(), SUBSTR() etc. are slow, these are old functions, around since FoxBASE+ days.
>
>I wonder if the "old" functions are still mired in the 16-bit world and are losing performance with 16/32 bit conversions ("thunking") and/or hitting limitations at 64K chunks? The moral - for fast VFP performance, use new functions??