Hi Charlie,

Good luck with those tests. I'm betting that lots of folks are anxious to see some of your results.

Cheers,

Jim N

>Jim,
>Just yesterday one of our analysts, ran into the 2 gig limit with about 28,000,000 records. He gets hospital claims on CDs and does all sorts of statistical stuff. What other tool can do this type of work better?
>
>While 28 million is unusual, it works, so any integer binary chunk of data that points at more than 17 million things must use more that three bytes.
>
>There's no way an entire tag which is involved in a rushmore expression is read into memory. That's just not possible. It is much much faster than that. There is no speculation about that.
>
>The magic is that fox must know almost exactly the right CDX file offsets to ask for. It takes very little network activity to get the matches. If there are a lot of matches, like DelTag gives, that slows things down. The other thing is that once fox reads the tag portion that matches, it caches it, yet apparently knows when the cache is stale. I'm SPECULATING that the CDX header may store a updated stamp for each tag. This would allow fox to know when the cached index info was stale and when it was still in effect. This is a testable theory, and perhaps tomorrow I'll know.
>Charlie
>
>>Hi Charlie,
>>
>>ADDING TO SPECULATION
>>
>>3 bytes would probably be just fine for VFP as things now stand. The max. records is 1 billion but that means a 1 byte record (the other byte is the delete mark). I can't think of many uses for a 1 byte table - at least not with 1 billion entries! I would also guess that 16.7 million records would present other problems in day-to-day usage. I suspect that somewhere around 2 million records is a "practical" limit.
>>
>>The speculation about the bit-mapping doesn't seem to make allowances for B-Tree functionality. Surely that has to play some important role in all of this.
>>
>>Similarly, speculation about the bit-mapping implies (to me, as I have read this thread) that the whole index would necessarily be read into RAM. I have to wonder about this too.
>>
>>Finally, and this may be what Dragan was saying, even if the whole index *is* read in, the order should be immaterial for bit-map *building* purposes, since the record number is supplied with each index entry (thus providing the offset for the bit in question anyway).
>>
>>END ADDITIONAL SPECULATION
>>
>>I don't blame MS (or Fox before them) for keeping as tight a lid as possible on the innards of Rushmore. But I do feel that that provides sufficient warning that we must always remember that we are all only speculating when we write about this stuff.
>>
>>Finally, as we saw in a thread a few months back, "fully optimized" is **NOT** always what we want (at least not any more, though I still hope that we can figure out when this may have changed if at all).
>>
>>Cheers,
>>
>>Jim N