Hi Hank,
>Thanks for sharing enough details about duckdb to whet the appetite.
You are welcome!
>If I were to use it I would likely integrate it into a Lianja app -- where I can write the Python program and
>make the call from a web UI just as I would from the Python command line.
The Web is not my current cup of tea. Just the opposite (local data mangling of local data, say à la Excel). But for those concerned, WebAssembly is possibly a way into duckdb:
https://duckdb.org/2021/10/29/duckdb-wasm.htmlI know next to nothing about WASM... But well it looks like they consider the lean duckdb (lean when not too much loaded...) may have some sort of place in web-building area. What I appreciate here is that you can run sql calls on massive "parquet resources" from the highly flexible python interpreter.
Let us call that parquet-centric world sort of "sql on csv-s on incredible steroids". You can indeed "load enormous table resources from scratch into a db engine" in minus one second and query it back sqlwise to a very pythonesque numpy resource in the same conditions i-e massively fast.
Of course this stuff consumes memory fast, may eat up your processors... and power by the way. That's the intention... But Duckdb database belongs the kind of databases that can be set up on the fly and run (with data inside...) in a couple of secs at most. Yep, way, way faster than loading csv files into brand new VFP cursors. That says something!
Of course, you need to have some sort of needs for this crazy "data mangling" power! As far as I am concerned I can see it to cover three distinct sorts of needs:
1- arbitrarily sql-search data on-the-fly from massive data sets, say lakes of thousands or more of csv files (or much better parquet-formatted ones),
2- heavy-duty, i-e column-wise ones, searches among massive data tables without resorting to massive infrastructures à la spark, typically AI kinds of requirements,
3- complex "on-the-fly" data mangling among large arbitrarily interconnected data sources,
I look forward to explore duckdb from the third perspective. Within the new duckdb ecosystem you can do that R, should you prefer more of an interpreter-kind of environnement. I'm on the developer side, so my preference is with python with a view to deliver apps that offer massive data mangling on arbitrarily inter-connected data sources.
Of course, there is a weak point. And that's worth mentioning: "the security stuff is missing!". You are on yourself on this matter. But this project could possibly improve on the subject.
The two main developers, Hannes and Mark, are startup-ers who originate from the IT university teaching and research world, yep like David L. Fulton in his time... They are both warmly open to requests. And the development (C++ based) is indeed impressively on target!
Daniel
