Hi John,

"Thanks for reporting back on things like this!"

You are welcome!

"You mention security- so is DuckDB as exposed as dbf?"

Yes it is. I have no heavy need for security. But sure, it does not fully belong to this eco-system where performance still is the key issue.

What you get as a return is the ability to handle massive quantities of data. I have starter to handle the combination of Apache-Arrow via pyarraow (for the data feed), Duckdb for the SQL engines and numpy for array handling.

Numpy is an array-management eco-system of its own within the python eco-system. It has become one of the software bricks of scientific IT world. You can handle massive multicore-or-gpu aided matrix-handling operations from the ptyhon interpreter.

The Numba library can bring additional multicore power to Numpy already fast array functionality. Not your typical ascan anymore really! But you are still working in a foxlike environment from the dot prompt. Parallelizing your code from an interpreter with no headache is possible though the numba llvmlite engine. Dreamed we had this sort of stuff within VFP back in the years :-)

Thinking of Duckdb as some sort of modern day VFP rushmore and parquet as equally to-day's dbfs ain't stupid. Good news: the size limits are nowhere to be found. And, even without high-end index management, performance levels can be impressive on 'columnar workloads' (ie "accross the board"). We are, here as well, a generation forward especially in terms of ability to put parallelism at work.

All this from an interpreter à la fox that is really a joy to work with. It feels like an interpreter... The only stuff missing as a fox guy? The VFP UI layer. The native tkinter is below par. And conversely QT a bit over the top with a daunting learning experience that I have yet to start investigating...

Daniel