>For me the lure of such a beast is to be able to query/update with SQL with data on local disc - as a side effect often minimizing the data transfered across the wire by not joining on the the server. Keeping RO data local (lookup/join data, selection tables) and synch server side changes via some data replication mechanism IMO is the best usage of smart endpoint devices with some computing power and storage ability.

Well it depends on what you need to do. Inserting into in memory data structures is pretty easy. Append to an array, list, dictionary or whatever. But again it depends on the tools available in the language.

What I was talking about in WebSurge is pure memory - no disk backup of any kind. LINQ works on in memory lists and that's what I'm getting at. You get the benefits of SQL like querying without having to have a database. And if you do need to write the data out you can serialize it which is efficient for a ~20 megabytes of data before it starts getting slow dumping so much data to disk. If you don't need a database (ie. any sort of shared data access) in memory is always preferrable to me.

This works great in .NET because of LINQ. In JavaScript you would use some sort of map/reduce solution to implement something similar. I suspect Python, and others like it have some sort of vector engines built in for similar functionality. FoxPro this would be more problematic because the in memory structure support sucks. Arrays top out and slow down under 100,000. I haven't tried collections but they are very inefficient unfortunately.

The main advantage of using something like SQLite is that you CAN make it a disk database that virtualizes the db and memory. But SQLite is also pretty limited in that multi-user access locks the entire db for writes, so it's not exactly a scalable solution either.

Lots of options obviously, but it depends on the platform you're using... But some people - especially FoxPro developers - see EVERYTHING as a database which doesn't make sense in other environments because DB access is more work.The ease of DB access in the language is certainly one of VFP's strong points, but outside of it you typically don't want to treat small data tables like a database because of overhead.

+++ Rick ---


>
>For a while it sounded like Linq2Sql was deprecated - and new effort from MS seems to target full blown ORM style, which I at least sometimes consider overarchitectured. While I certainly use biz objects, I prefer to have them working on local data [which often is just a cursor/ view of hosted data]. Disclaimer: I have not kept up on how easy it is to join/update sets of information via LINQ partly still in DB [aka SQL] and structures already read into memory.
>
>Having the option to specify RAM: as database location in most of the cases I encounter is enough to have in your back pocket if you need to tune.
>
>my 0.002€
>
>thomas
>
>>Well if you're doing an in-memory database - why not skip the database all together and just build up data structures in memory?
>>
>>For WebSurge in .NET I basically created large lists in memory and query them with LINQ which is very fast since it's all in memory - even when running against several gigabytes of data search perf is super fast. I suspect the same will be true in other environments assuming there's an efficient query interface like LINQ in .NET.
>>
>>I just think an in-memory database doesn't much of anything unless you're loading up huge amounts of data that require the in memory b-tree indexing.
>>
>>+++ Rick ---
>>
>>>Hi Rick,
>>>
>>>Thanks for dropping in,
>>>
>>>>SqLite is very dependent on proper configuration to get good performance out of it. Default config is pretty slow... but with turning off the transaction log (journal) and in >memory usage it can do OK.
>>>
>>>Yep, that's what I read up to now. You definitely need to turn the journaling off.
>>>
>>>>It works great as a single user Db, but make sure you don't need multi-user access in the future because it's terrible for that - basically locking the entire db for any >updates which is very inefficient.
>>>
>>>The intended usage, as an application session-wise-only data storage - very much in the way vfp-cursors are sometimes used within some vfp apps - requires zero multi-user access. Except when debugging where access may be done as a way to investigate and debug, the access to the db is intended from a single user and two threads, a GUI one and the (long-)process-one.
>>>
>>>I am always impressed by rushmore speed on indexed cursors. I will run a couple of tests on the relative perf of rushmore sql queries and and "in-memory" sqlite from python on this thread as soon as I'll have my feet wet on the project.
>>>
>>>Since python code does not run faster than good ol' vfp, I hope that I'll be able to replace the current set of rushmore-based data crunching with "in-memory" sqlite , properly tuned as you hinted.
>>>
>>>There are of course possibly LOTS of alternatives to speed things up in python. I would like to be able to shunt cython-based (aka python FLLs) optimizations, numpy, numba and other speed-up solutions when starting the rewrite at least for the initial version of the app.
>>>
>>>Daniel