Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Low level read-write optimization for very big file
Message
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00733923
Message ID:
00733993
Views:
36
This message has been marked as a message which has helped to the initial question of the thread.
Hi Gérard,
Try this code...

MartinJ
LOCAL liCount, liIndex, liSeek,lhIn,lhOut,lcRow,liCR,liLF
LOCAL ARRAY laRow(1)
#DEFINE CRLF CHR(13)+CHR(10)

* Open In file buffered readonly (lhIN)
* Open Out file buffered Writeonly (lhOUT)

STORE 0 TO liCount, liIndex, liSeek
=FSEEK(lhIn,0) && Seek top
DO WHILE !FEOF(lhIN)
   IF liCount=liIndex && If read buffer
      lcRow=FREAD(lhIN,245760) && read 240 KB
      IF EMPTY(lcRow) && If eof()
         EXIT
      ENDIF
      liCR=RAT(CHR(13),lcRow,2) && Find last enter (CR)
      liLF=RAT(CHR(10),lcRow,2) && Find last enter (LF)
      IF liCR=0 OR liLF=0 && If one
         liCR=RAT(CHR(13),lcRow,1) && Find last enter (CR)
         liLF=RAT(CHR(10),lcRow,1) && Find last enter (LF)
      ENDIF

      IF liCR>0 OR liLF>0
         liSeek=liSeek+IIF(liCR+1=liLF,liCR+2,IIF(liCR>0,liCR+1,liLF+1))-1
         =FSEEK(lhIN,liSeek) && Seek
         lcRow=LEFT(lcRow,IIF(liCR>0,liCR,liLF)-1)

         lcRow=IIF(liCR=0 OR liLF=0,STRTRAN(lcRow,IIF(liCR=0,CHR(13),CHR(10)),CRLF),lcRow)
         liCount=ALINES(laRow,lcRow) && Read rows
         liIndex=1
         lcRow=laRow(liIndex) && First row
      ELSE
         STORE 0 TO liCount, liIndex
      ENDIF
   ELSE
      liIndex=liIndex+1 && Increment counter
      lcRow=laRow(liIndex) && Next row
   ENDIF

   * Any operation with lcRow

   * Write row
   =FPUT(lhOut,lcRow)
ENDDO
>(VFP 7 SP1 on Windows 2000 or Windows XP pro)
>
>I have to perform 3 operation on a text file.
>
>1 - read a line from a text file.
>2 - perform some operations on the string.
>3 - write the line in another file.
>
>The text file was really big (950 mb).
>
>The program run fast for the first ~50 mb but after that it start to slow down (crawl).
>The computer have 1 gb of memory an 100 gb of available disk space.
>From the task manager I can see that only around 200 mb was used during the operation.
>
>What is the best way to open the files? buffered/unbuffered?
>Flush when I reach a certain point in the file? every 10 mb?
>How can I optimize the process to use all available memory and speed up the process? some sys() function to use?
>
>TIA :-)
"Navision is evil that needs to be erazed... to the ground"

Jabber: gorila@dione.zcu.cz
Jabber? Jabbim
Previous
Reply
Map
View

Click here to load this message in the networking platform