Gerald,

I think it must be the fault of your step #2, it must be doing something that is not constant time per line processed.

I just ran this code to create a 1gb file:

lcLine = replicate( "1234567890 ", 10 ) + chr(13) + chr(10)

for i = 1 to 8
   lcLine = lcLine + lcLine
endfor

fh = fcreate( "testbig.txt" )

for i = 1 to 40000
   if ( i % 100 = 0 )
      @0,0 say i
   endif
   fwrite( fh, lcLine )
endfor

fclose( fh )

it ran in about 3 minutes on a PII-300 class machine, 11gb free diskspace, WinXPPro, 192mb ram.

This code read the file back in line by line and just wrote the data right back out to the other file.

fhIn = fopen( "testbig.txt", 0 )
fhOut = fcreate( "testbig2.txt" )
i = 0
create cursor x1 ( irecs i, twhen t )
do while ! feof( fhIn )
   i = i + 1
   if ( i % 1000 = 0 )
      insert into x1 values ( i, datetime() )
      @0,0 say i
   endif
   lcStr = fgets( fhIn, 255 )
   fputs( fhOut, lcStr )
enddo

fclose( fhIn )
fclose( fhOut )

select x1
browse last nowait

It remains quite linear over the whole operation. It does take about 15 minutes to run, because it's a read + write operation instead of just a write operation.

What exactly does your step 2 do?

>(VFP 7 SP1 on Windows 2000 or Windows XP pro)
>
>I have to perform 3 operation on a text file.
>
>1 - read a line from a text file.
>2 - perform some operations on the string.
>3 - write the line in another file.
>
>The text file was really big (950 mb).
>
>The program run fast for the first ~50 mb but after that it start to slow down (crawl).
>The computer have 1 gb of memory an 100 gb of available disk space.
>From the task manager I can see that only around 200 mb was used during the operation.
>
>What is the best way to open the files? buffered/unbuffered?
>Flush when I reach a certain point in the file? every 10 mb?
>How can I optimize the process to use all available memory and speed up the process? some sys() function to use?