unless there is a way to estimate processing time rather well I'd also lean more to the table-based semaphore processing, as there is less chance of one list needing lots more processing time than the others, leaving 3 cores idling while the core with the task list taking longest still chuggs away. Generating such a table will probably take only minimal time and can be speeded along by creating dummy records, reading the first batch and then letting the workers start while the semaphore table is still in the process of being finalized. Lock contention will be minimal - and could be helped by a simple 2-pass processing strategy, giving each core on first pass a dedicated subset and only if most task is done the already finished cores will start on the other subsets - and only then lock contention will happen ;-)

I'd probably switch to a Rushmore-izable status field plus some timestamp and log info instead of deleting, but that changes nothing in basic approach IMO ;-)

>Agreed. However I'd divide the list of files to be processed into 4 or 5 or whatever separate tables so there's no messing around with row locking/contention or any possibility of skipped files.
>
>>Harsh, the way I would do it (assuming I understand the project):
>>
>>First, create a table of file names to convert - see ADIR() function.
>>
>>Now make a conversion program that accesses this file and does this:
>>
>>1) search for the first unlocked record,
>>2) attempt to lock the record - if failure then try another record (attempt once only, don't bother with retries)
>>3) convert the file as stored in the record,
>>4) delete the record,
>>5) repeat at step 1
>>
>>The program runs in a loop and the user can exit it in some way e.g. pressing Esc.
>>
>>Now I can simply load as many copies of this program as I like and the computer CPU itself will decide what core to run it in. In this way my program is simpler to manage, the computer takes care of multi-processing issues, and I can fire up as many copies of the program as I think appropriate by observing the CPU and disk usage performance.
>>
>>
>>
>>
>>
>>>In that case you mean to say That of I have 20000 Records in a Table say A. I shall divide it in A1,A2,A3,A4,A5 Each with 4000 Records first.
>>>And than I should process Each table in a separate CPU. Is it what is suggested.
>>>
>>>If yes, than kindly tell me how will I bind my A1 table and related records to CPU1 and A2 to CPU 2 ?
>>>
>>>Harsh
>>>
>>>
>>>>Harsh, make sure you read Al's post regarding not using Parallel Fox and instead modifying your application to accept a parameter to process a section or group of files. And then you load several copies of your main application, each processing a different group of files. The advantage of this are:
>>>>
>>>>1) Far simpler model - no need to mess around with Parallel Fox, threads, etc. Your app is a simple file conversion app (but needs to take a parameter to determine files to process, or it must ask the user for which files to process, or you can create a database of file names and lock the record you are processing and each copy of your app just looks for the next not-locked record. There are various options.
>>>>
>>>>2) By running multiple copies of your app it allows the processor to decide which CPU core or thread to run each copy on. More efficient.
>>>>
>>>>3) You can load as many copies of your app as you like - watching CPU and disk usage to judge performance - stopping copies and loading additional copies without code changes.
>>>>
>>>>
>>>>