Step 1: Read one line

Step 2: The second step was the validation of all chars in the string.
I have the replace every chars < chr(32) by a "?" in the string.
After this validation, the string was put in an array using ALINES.
Each rows of the array was tested to determine the type of data (date,datetime,numeric and so on).
The result was put in another array. If there is 6 "fields" by row, the array have 6 elements.
For each row read from the file, 1 character was added to each of the array's elements (C for char, T for datetime, etc...).

Step 3: Write the line to a new file

Assuming that there is 6 fields by row.
After 100 000 rows parsed from the file, the array contain 6 strings of 100 000 chars.
When the entire file was read, a table can be created base on the information in array.

Anything wrong in this?
:-)

>Gerald,
>
>I think it must be the fault of your step #2, it must be doing something that is not constant time per line processed.
>
>I just ran this code to create a 1gb file:
>
>

lcLine = replicate( "1234567890 ", 10 ) + chr(13) + chr(10)
>
>for i = 1 to 8
>   lcLine = lcLine + lcLine
>endfor
>
>fh = fcreate( "testbig.txt" )
>
>for i = 1 to 40000
>   if ( i % 100 = 0 )
>      @0,0 say i
>   endif
>   fwrite( fh, lcLine )
>endfor
>
>fclose( fh )

>
>it ran in about 3 minutes on a PII-300 class machine, 11gb free diskspace, WinXPPro, 192mb ram.
>
>This code read the file back in line by line and just wrote the data right back out to the other file.
>
>

fhIn = fopen( "testbig.txt", 0 )
>fhOut = fcreate( "testbig2.txt" )
>i = 0
>create cursor x1 ( irecs i, twhen t )
>do while ! feof( fhIn )
>   i = i + 1
>   if ( i % 1000 = 0 )
>      insert into x1 values ( i, datetime() )
>      @0,0 say i
>   endif
>   lcStr = fgets( fhIn, 255 )
>   fputs( fhOut, lcStr )
>enddo
>
>fclose( fhIn )
>fclose( fhOut )
>
>select x1
>browse last nowait
>

>
>It remains quite linear over the whole operation. It does take about 15 minutes to run, because it's a read + write operation instead of just a write operation.
>
>What exactly does your step 2 do?
>
>>(VFP 7 SP1 on Windows 2000 or Windows XP pro)
>>
>>I have to perform 3 operation on a text file.
>>
>>1 - read a line from a text file.
>>2 - perform some operations on the string.
>>3 - write the line in another file.
>>
>>The text file was really big (950 mb).
>>
>>The program run fast for the first ~50 mb but after that it start to slow down (crawl).
>>The computer have 1 gb of memory an 100 gb of available disk space.
>>From the task manager I can see that only around 200 mb was used during the operation.
>>
>>What is the best way to open the files? buffered/unbuffered?
>>Flush when I reach a certain point in the file? every 10 mb?
>>How can I optimize the process to use all available memory and speed up the process? some sys() function to use?