>>>Any reason why you could not run the 'robot' asynchronously ?
>>
>>Because those are processes totally not related to a user-hit on the Web site. Intensive processing on millions of records and so on are really not what the users do on the Web site.
>
>OK. But you say hits 'are sometimes taking a few seconds' - why is that?
>If, for example, it's because they spend time retrieving data then making them async could vastly improve the number of hits that could be handled.

Michel,


My two cents


Let me give you my impression - not directly related to this 'problem'
Seems you are working with a 'busy' system

(1) You're fully on NAS storage
Heavy transfers to and from the NAS drive on gigabit controller may take in excess of 50% of the bandwidth
The more a network (TCP/IP) is busy, the more likely are collisions. There's a random wait to retransfer - but as collisions occur the traffic increases

(2) VMWare - execution speed
Does the machine have enough memory configured and enough processors/cores

(3) Performance
Related to some other posts ( creating a StringBuilder when not ncessary, the use of object collections + boxing/unboxing)

.Net lets you create objects as needed. But when it comes to removing (destroying) an object on the heap, you have no control
( see garbage collection http://msdn.microsoft.com/en-us/library/0xy59wtx(v=vs.110).aspx )
If you create lots of objects, they will have to be garbage collected at some point in time.
After that the objects have been 'removed' the heap is compacted ( not the large object heap )
Compaction is not only moving things around in memory, but also updating the references to objects when they have been moved

Creating an object that is not needed is not a real problem if the system is not very busy

In a 'busy' system, a thread may get suspended for a while when garbage collection is busy

It is also best to use the 'using' clause for objects that implement IDisposable. If the Dispose() method is not called when you no longer need the object, they cannot be removed immediately by the garbage collector. Instead they have to be handled by a Finalizer thread