>I ran it for 1, 2, 4, 8... 16384 records on a couple of my tables. What I discovered is that the eval() works almost immesurably fast until we reach hundreds of records; by that time, B, B2 and C go down from 550ms/record (resulting XML was roughly 980 bytes per record) to about one, and then as we go on, they tend to get equal footing.
>
>Also, no matter whether the string gets built in smaller (B2) or greater chunks (B), the total speed seems to be O(n**2), where n is number of records.

Well, that is relative. That the speed degradation comes from the string concatenation, and can be eliminated by appending to a file. The point where switching from concatenating strings to LL-file functions speed up is probably machine dependendant (speed of RAM, size of L2 cache, Speed of disk), but is very low: around 30 records, but the resulting XML file size is probably the better indicator of the point, where a strategy switch is called for.

Working with the low-level file functions I get consistently sub-ms times per record, even if working on a few thousand records, and using the original eval() tends to double the time needed. The compilation time is fixed, so having more records to work upon lessens the impact of compilation per record.

>Then I introduced an additional change: I've removed transform() for character fields (I've specifically excluded memos for this test). This involved using an iif() in the eval() case, and an if... \\ transform(|field|)...else \\|field| endif (I used | for textmerge delimiters, since that's what Fabio had - I started off from his collection). Now the results are a little more surprising:

I also reworked my program a bit: I made every generated XML identical, to be sure to compare apples to apples, and also removed the Trans() from the generated optimized runner. "My" runner.prg now jus returns the xml in one big statement, eliminating intermediate string concats. Since Trans() returns just the same for strings, it is not called for memo and character fields. This about triples performance compared to your first sending in the datasize I use, see below). I didn't include a iif() to the other solutions, because in numeric-only tables this would further slow down execution speed, and it is my guess that trans() will be nearly as effective as an explicit iif()-check (I know I should also check this assumption <g>).

>The results make sense as a side to side comparison only - even though the table was the same, I had less memory available the second time, having the spreadsheet open... and a solitaire, because this started taking some considerable time :).

That's why I don't run the string concats for more than a few hundred records (unless I am leaving the machine for a nap then it is crunch time again <g>).

At nearly 6000 records resulting in an XML of roughly 11MB I get

select(ms) Xml(ms) Xml(bytes)   Recs  ms/Rec    Byte/ms
     110    4897    11528250    5857   0.836    2354.145  MEESTER
     120    3986    11528250    5857   0.681    2892.185  CETIN
     120    2294    11528250    5857   0.392    5025.392  CETINNOEVAL
     150    5638    11528250    5857   0.963    2044.741  DRAGANLOWLEVEL
     120    1913    11528250    5857   0.327    6026.268  DRAGAN2_LOWLEVEL

on a probably slightly slower machine. As you see, the perf is enhanced more the ten-fold by skipping the string concats. Also, now you can see clearly the effect of the different strategies, which were masked by the memory thrashing before: another 100% increase is possible. Not too shabby I guess, but what surprised me was the speed of the CETINNOEVAL, which is still hanging in there, needing no tricks other than scatter to array.

ExecScript(), B2 and generating runner-like string to eval() should be easy enough to add.

If you want the prg, I'll be glad to mail it to you, since adding another few K to this message seems uneccessary.

regards

thomas