The server doesn't sound overly well specified, 48mb of Ram is a little measly, 64mb or 128mb should be preferable. However, there's a big difference in 6 seconds compared to 3 minutes. It's possible that the processor & hard disk systems are just struggling to deal with all the requests from the clients.

How about putting some more ram in?

Andy Crawshaw