Harsh, make sure you read Al's post regarding not using Parallel Fox and instead modifying your application to accept a parameter to process a section or group of files. And then you load several copies of your main application, each processing a different group of files. The advantage of this are:

1) Far simpler model - no need to mess around with Parallel Fox, threads, etc. Your app is a simple file conversion app (but needs to take a parameter to determine files to process, or it must ask the user for which files to process, or you can create a database of file names and lock the record you are processing and each copy of your app just looks for the next not-locked record. There are various options.

2) By running multiple copies of your app it allows the processor to decide which CPU core or thread to run each copy on. More efficient.

3) You can load as many copies of your app as you like - watching CPU and disk usage to judge performance - stopping copies and loading additional copies without code changes.



>Thanks very much for the Guidance Sir.
>
>>>Hello Experts,
>>>
>>>I am converting file from Type A to Type B using a I7 Machine with 8 core CPU, 8 GB RAM, 2TB HDD.
>>>
>>>Using simple Program (i.e using single core) I am able to convert around 800 files in a Minute. my CPU Shows 17%-18% Usage. Memory Shows around 15%-16% and HDD shows around 10 % usage in Task Manager.
>>>
>>>When I incorporate Parallel Fox in my code and use all the CPU. Following was the result found.
>>>
>>>No. of files converted: around 1700 Per Minute
>>>CPU Usage:-75%-95%
>>>Memory Usage 30%-35%
>>>HDD Usage :- 20%-40%
>>>
>>>Thus , using a Single CPU I am getting around 800 files but when I use Parallel fox and all cpu utilized, though my CPU works to the fullest I just get just twice the ouput than before.
>>
>>It's worth pointing out a few things about your i7-4770K processor ( http://ark.intel.com/products/75123 ):
>>
>>1. It's not a true 8-core CPU, it's a 4-core CPU with HyperThreading (HT). HT is a clever trick Intel uses to get more performance out of a computing core. Under ideal conditions Intel claims this can improve performance of a core by about 30%. In general usage it will be somewhat less. From a performance point of view you can consider your 4770K to have ~5 "real" cores, although 8 "logical" cores appear in Windows Task Manager
>>
>>2. When running only 1 intensive task, the 4770K can speed up a single core from 3.5 to 3.9GHz (about 10% faster) ("Turbo Boost/Frequency"). It won't do this if all cores are busy. So your 1-core test was probably running at 3.9GHz, and the multi-core test at 3.5GHz
>>
>>3. Windows needs processing power to run other tasks while your jobs are running. During your single core test there are other cores free for Windows to do these other things so your task can run unimpeded on its single core. If you're trying to run your job in parallel using all cores then from time to time your job threads will be interrupted so Windows can service its own threads. Performance will be reduced because of these context switches
>>
>>Comparing apples to apples, 800 files/min @3.9GHz is about 720 files/min @3.5GHz.
>>
>>Your performance gain using "all cores" @ 3.5GHz is 1700/720, or about 2.36. The theoretical best performance improvement you could expect would be about 5.0 times better, with zero overhead for Windows and no losses from context switching. In practice, it will be less than that.
>>
>>If you're consistently getting 75 - 95% CPU utilization while running your jobs, that's actually pretty impressive, that's not easy to achieve with a multi-core processor. From the stats you give it looks like the CPU is your bottleneck, despite being one of the most powerful desktop CPUs currently available.
>>
>>I don't know how ParallelFox uses multiple logical cores/threads or if you can configure it to use only a certain number of cores/threads (instead of all available). If you can limit the number of threads being used I would experiment with 3, 4, 5, and 6 threads, rather than 8 (one each for the 8 "logical" cores that appear in Windows). You may find this reduces context switching and improves your job's overall throughput.
>>
>>If your job is CPU-bound, you have some options:
>>
>>1. Make your job processing more efficient
>>
>>2. Get more desktop computers and divide the files you're processing amongst them
>>
>>3. Move to a server platform with one or more Intel Xeon processors ( http://www.intel.com/content/www/us/en/processor-comparison/compare-intel-processors.html?select=server ). E5 Xeons are available with up to 12 physical cores, and some versions with fewer cores run at up to 3.5GHz. However, server processors, mainboards and RAM are much more expensive than desktop systems; it may be more cost-effective to use multiple desktop systems.