>>Re disk: I thought I ran into an issue once with having too many files in a folder, that *might* be an issue for Harsh if he's processing many files. Found a couple of interesting threads on StackOverflow:
>>
>>http://stackoverflow.com/questions/197162/ntfs-performance-and-large-volumes-of-files-and-directories
>>http://stackoverflow.com/questions/2994544/how-many-files-in-a-directory-is-too-many-on-windows-and-linux
>>
>>Still, from the numbers he gave earlier it looks like his process is CPU-bound. Improving disk might help a bit but it seems unlikely to yield large gains in overall performance.
>
>First of all, I am not sure if the numbers reported for disc utilization can be trusted. But when switching over to a beehive of I5 working on the dir in a server based dir, adding 5 more machines will move the bottleneck to disc even if he does not change anything/the code cannot be imptoved - it is simply a question of adding more machines. Having dedicated I/O channels will open the bottleneck a bit then, or wirting out locally and piping only finised code back to the server or directly into print buffer.
>
>
>>
>>If I were in that position I'd take a hard look at the DLL. Unless it's designed for and proven to be high-performance, there may be better options.
>
>Quite possible, but by adding 10 more machines with a simple change of approach the "problem" vanishes.

Near the beginning of this thread I suggested dividing the file list up and using multiple program instances, but it segued into ParallelFox so I lost interest. I haven't been following the exact code closely, I wouldn't know what to suggest for specific optimizations.

I've always liked the idea of throwing more hardware at problems, it's often the cheapest performance fix. Of course it depends on the economics. If I were processing 1M files per day and getting paid 0.01 each, I could make a pretty good case for more hardware ;)