Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Low level read-write optimization for very big file
Message
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00733923
Message ID:
00733985
Views:
19
>Gerard,

GéraLd :-)

My client want that in this way :-)

The program need to determine the format of ANY comma delimited files that was given to it.
I know that Fox was able to guess the structure of the data in the file, but the client want a parsing and validation of the file.
After the parsing, a table was created based on the array values.
The output file was append to the table.

Following your path I have created a little program to benchmark the time it take to strech a string in an array:
DIMENSION aTest[10]
STORE "" TO aTest
lasttime=DATETIME()
FOR i =1 TO 100000
	IF i%5000=0
		WAIT WINDOW "Current="+TRANSFORM(i)+CHR(13)+"Duration for the last 5000 chars:"+TRANSFORM(DATETIME()-lasttime) NOWAIT 
		lasttime=DATETIME()
	ENDIF 
	FOR ii=1 TO 10
		atest[ii]=atest[ii]+"?"
	ENDFOR 
ENDFOR 
You are rigt, the slow down occurs in the second step.
If you run the code, you will see that after a quick start, a slowdown appear an increase at each loop...

But, if you watch the task manager's performance window, you cant notice a memory problem in the system (only a slowdonw of fox).

Fox was suppose to be able to handle a maximum # of characters per character string or memory variable of 16,777,184 bytes...
But at wich price?

:-)


>
>reply order kinda swapped around
>
>>Anything wrong in this?
>>:-)
>
>Yeah, it takes too long. *L*
>
>>Assuming that there is 6 fields by row.
>>After 100 000 rows parsed from the file, the array contain 6 strings of 100 000 chars.
>>When the entire file was read, a table can be created base on the information in array.
>
>So for every line in the file you are adding a single character to 6 different array elements. This is your non-constant speed operation
>
>Is it really necessary to know that every field you know what type of data was in that field for every line of the file? Why do you need to have 100,000 Ts in one of the array elements?
>
>Why don't you give us a better description in words what you are really trying to accomplish here.
>
>Can you also show us an example of 2-3 input lines and 2-3 outputlines that they'd generate?
>
>>Step 1: Read one line
>>
>>Step 2: The second step was the validation of all chars in the string.
>> I have the replace every chars < chr(32) by a "?" in the string.
>> After this validation, the string was put in an array using ALINES.
>> Each rows of the array was tested to determine the type of data (date,datetime,numeric and so on).
>> The result was put in another array. If there is 6 "fields" by row, the array have 6 elements.
>> For each row read from the file, 1 character was added to each of the array's elements (C for char, T for datetime, etc...).
If we exchange an apple, we both get an apple.
But if we exchange an idea, we both get 2 ideas, cool...


Gérald Santerre
Independant programmer - internet or intranet stuff - always looking for contracts big or small :)
http://www.siteintranet.qc.ca
Previous
Next
Reply
Map
View

Click here to load this message in the networking platform