Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Measuring progress of a BCP event
Message
From
01/06/2004 12:32:00
 
General information
Forum:
Visual FoxPro
Category:
Client/server
Miscellaneous
Thread ID:
00907011
Message ID:
00908934
Views:
16
David -- I found that two factors control the speed of the process: the polling interval and the record count in each transaction. The factor that affected the process the most is the number of records in each transaction of the bcp event.
The test set of data has 29000 records: it's rather small. The SQL table getting the data has 109 fields of mixed data types. It works fine for testing the engine, but a substantially larger set is needed for serious load testing of the process. Given that, these are the results of my tests:
1. Baseline - Doing a "non-interupted" import takes 1 minute. By non-interupted, I mean one in which there is no progress polling and just one transaction.
2. Changing Transaction size only - Importing the same data with 2000 record transactions, with the keyword "BATCHSIZE = 2000" and no progress polling, the time to import dropped to 15 seconds. Time reduced by a factor of 4!
3. Changing Transaction size with polling - Importing the data with 2000 record transactions and an interval of .1 seconds resulted in an import time of 17 seconds. With no polling interval, the import time increased to 23 seconds. Changing to greater polling intervals did not appreciably affect performance. Two second penalty for polling, or 13% increase in the time to perform the BCP.
I can't tell from the size of my data if changing the polling interval will have a significant effect on the speed of the BCP. I was surprised at how much a difference the BATCHSIZE keyword had on the process.

Hope this is helpfull to you. It's been a fun experience for me.

FWIW, here are the specs on the machines I did this on: My development "server" is a 900 mhz PIII powered notebook: OS is Win2KPro SP4. (30 GB HD w/4800 RPM. ) My development "client" is a 933 mhz Duron powered notebook: OS is WinXPPro SP1. Both machines have 512 MB RAM. The SQL server is SQL2K MSDE. The app is built with VFP7 SP1.

Ken

>Kenneth,
>
>Thanks for posting the code back. I'm curious though how long your total BCP process is taking and how much it slows it down making the count(*) hit every .25 seconds. You might find the performance overall better if you set that poll interval time so that the count(*) query happens no more than say 100 times ie 1% although depending on the size of your thermometer and it's blip accuracy might allow you to use a much longer interval.
>
>>This is how you can run monitor the progress of bulk copying data into an RDBMS database such as MS SQL Server from Visual Fox Pro. The general approach is to establish two connections to the RDBMS, one connection runs the bulk-copy event, hereafter “bcp”. The other connection is used to monitor the bcp event.
>
>> = INKEY( .25 ) && This reduces the number of times per second, the monitoring request is sent to the SQL server.
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform