>>
>>I've got mysql server running on the processing machine, but using that to store the data would mean rewriting the sorting and matching part of the program, which is a lot of code and is currently fast, getting the data from memory back onto disk as a foxpro table is the bit that is slow. once I've got that bit done it sorts and matches in seconds and then writes out to a pipe delimited file in seconds.
>
>Using MySQL (or SQL Express which would be faster) as the datastore eliminates almost all the IO. Basically a stored procedure would just spit the text back to you to save. You definitely would have to convert the processing code. It depends on how hard the code conversion would be I guess.

mysql of sqlexpress would still have to perform the io, but I see what you are saying, they would act as the buffer for those writes so that foxpro can continue reading without waiting for the write io to complete.

I think I'll try getting around it by writing something to replace the memo/fpt part with just references to the original input files, that should be fairly simple to do, converting all the code to sp would take an age and means we'd need an sql server running on any machine we wanted to process these big data files on, if I can avoid that I make everything more portable.