>>...This speed advantage is of course fully dependent on the memory your database will be allowed to access...
Right... since with 600 million rows, if there's (say) 200 bytes per row then presumably (unless my math is way off!) you'll need around 110GB Memory just to hold the thing (600,000,000*200/1,073,741,824) if it's not lazy loaded. I suppose the other issue is speed of data transfer from the SSD; can you transfer that much data in 0.6 sec?
Hence my wondering about lazy loading. FWIW, if you open a huge VFP dbf it opens almost instantaneously and you can browse or perform SQL against it without knowing or caring how much is actually in memory. I "get" that importing a csv is different, unless the data engine can process csv natively...
In any case, nice that there's renewed interest in this sort of thing after a long "everybody knows" period when it was definitely best to do all processing in stored procedures in a backend database. Ability to partition off chunks of data for local processing will be fabulous for AI, especially for those who are cautious about offloading to big tech black boxes.
"... They ne'er cared for us
yet: suffer us to famish, and their store-houses
crammed with grain; make edicts for usury, to
support usurers; repeal daily any wholesome act
established against the rich, and provide more
piercing statutes daily, to chain up and restrain
the poor. If the wars eat us not up, they will; and
there's all the love they bear us."
-- Shakespeare: Coriolanus, Act 1, scene 1
