>>
>>the reading and splitting part takes just over a second per 100meg ie as fast as the hd array can manage
>>
>>however writing that data back to the dbf adds 14 minutes to the process.
>
>>every 5000 records took about 20 seconds
>
>sometimes it doesn't hurt to make sure of the obvious: you ARE using different physical disks for reading and writing ?

no it's just one disk (well a raid array but it's one in effect) available, if the switching between the read and write operations was the problem then it would have mostly gone away when I batched it to read/write batches at a time so each read/ write was around 5 meg, the write part is still around 70 times slower than the - read, split, trim and prepare array operation

if I read the file in and write it back out as a text file including all the trimming and pulling out needed info all to the same disk the entire 500 meg file is processed and new file produced in around 15 seconds at most. writing that out to a table rather than as text adds 14 minutes to that equation. unfortunatly if I dont get it into a table then I cant use the speedy foxpro sorting/matching/selecting to get everything right before exporting back out to a text file.