>>[Subtitle: How VFP's cache is mapped onto the network tables]

Jim,

"Never too old to learn" could be another subtitle. For that matter : most of my earlier statements have to be revised already. That is, after my experiences from the last days.
Though I have inline comments in the below, I think it is best to tell what I've been learning first.
BTW, knowing that some MS persons are around here too, I really wonder why they don't respond to this thread. Do they not know ? is it about info not to unveil ?

Rushmore
Yeah, this made me think. That is, looking at the below. It just might be about Rushmore. If not, it's even more Rushmore :

Being around with some monitoring software (unveiling tthe commands VFP presents to the server) and a sniffer on the network cable (showing what the server comes up with), I now can withdraw all my earlier "statements" on the phenomenon "block" (or cluster if you like). That is, when talking about blocks to receive from the server opposed to retrieving the records in them, this just is not true at all. For example, a statement like "RLock a record form a block, and you'll receive not only that record in its most actual state, but you will also receive the other records from that block in their most actual state ..." not true at all.

Please note that all was derived (before) logically from the perspective that at the more physical level, the server reads a complete block indeed, just as well as that the server's cache contains the data at the block level (but note that the blocksizes in there can differ from the physical blocksize on disk). This by itself is all still true. But ... it has really nothing to do with how VFP receives that data. So how does this actually work ?

The commands from VFP as you know them (like GO 3701) are converted by already VFP to a byte-offset within the file. Thus, supposed the header-length is 263 bytes, and the record length is 100 bytes, the server is asked for a byte offset of 3701+263=3964, and from that point a stream of 100 bytes.

I know, this sounds too easy (and stupid) to be true, but that's how it works. Only too much (?) knowledgde on computers' workings could make you think more difficult.

Though from this could be derived that all software will behave like this, I wouldn't be sure of that. It just could be VFP being smart, and as we know (compared to similar products), the Fox smart.

Sidenote : the retrieved data is obviously spread over the network packets needed, opposed to their physical length. Though this is true, it doesn't interfere with all of the logic as now "unveiled".

The ones out there (UT) who already know the above, right now could be under the table form laughing. Why ? well, because there are quite some other things to add, and "they" know them already. Here we go :

At sniffing the cable, it takes quite a while before you're even after the simple fact as describe above. This is exactly because of the way it really works. Think of it :

Depending on the speed (subsequent ?) records are asked for, the number of records asked for by VFP increase.

What about that ?
The above says that when I ask for a record (e.g. GO) from the command window, I just receive that one record. Or better : VFP just aks for that one record. However, if I am able to ask for the records at say, an interval of 0.5 secs, each VFP DML command I give, is converted to a larger bytestream to receive back. Thus, asking for record #3680 as the one command only will let VFP to ask for a stream of 100 bytes (the record length). if I then ask for #3681 in some "easy" manner, VFP asks again for 100 bytes. But now within (say) 0.5 secs I ask for #3682, VFP asks for 200 bytes ...

If I'd continue with asking for #3683, it's already in the PC's (VFP's) cache, and nothing happens. That is, VFP doesn't ask the server for anything.

If -again- I ask quickly after the before actions for #3684, VFP asks the server for 400 bytes. Coming to asking for #3688 it will be 800 bytes. Etc.

Oh man, how smart this is <s>.

But you know what this means ? this means that troubleshooting whatever corruption situation will be the vast impossible job.
Now think of the next :

Before knowing this, and before I was even thinking of analyzing the above described, I just came into the situation of discovering "something like this". But, it was completely unexplainable. Sorry to be slightly (?) off-topic, but here is my corruption problem again (Thread #692378), and I managed to find the way to let Novell come up with either the null data, or the proper data. IOW, I knew the commands to let Novell behave differently;
In my situation, I discovered that a GO to the record before the really corrupted record would come up with nulls in the not-really corrupted records, and a GO to either of two records "around" the beginning of the not-really corrupted area would show the proper data from any of the not-really corrupted records (GO to either of them);
This now knowing, I wrote some small loop, trying to GO to each of the records in the table and right after that GOing to some record in the not-really corrupted area. I saw nulls only. Also when the program was in the area of GOing to the records I was GOing to from within the command window, and when that showed the proper data.

For a whole afternoon I was puzzled, also seeing that the elapse of the SET REFRESH timer influenced the thing (now there were other records to GO to in order to show the proper data). RLock influenced too, end in the end I found that it just needed an INKEY(.n) - varying the n in order to show different results.

Now back to this thread, it is my conclusion that what a VFP-PC receives, already depends on the speed it can do it. So here we go : have a faster Mhz PC, and looping through the table will imply different physical results. That is, the faster the PC, the longer the byte-stream asked for will be.

In order to let all corruption-analysis flaw, I know can fairly say that how faster the PC, the more cache it "will need" (so uses) in order to be able to deal with VFP's internal dealing with the data requests.

But there is another subject to go back to :

Supposed my earlier expressions on how to receive the actual record contents opposed to receiving of blocks was somewhat understood, this now must be explained as follows :

All of the earlier explanation(s) still stand, but before the "bytestream" all is about. Hence, this is not about some 4,096 (when a block is 4KB) stream, but this is about "some stream you will never know".
IOW, the records you obtain by an RLock of some record you want to have actualized, is unpredictable. Okay, one thing you could do : have an =INKEY(1) before the RLock, and you can be pretty sure that you will receive that one record only.

This, sadly, implies that we are not able to apply "logic" to what we will receive; it depends on the Mhz speed (and all around it) what we will receive according to the actuality of things.
And to be really clear on how this influences "us" : if we coincidentally asked for some records in a speedy manner, the next few x will be in the PC's cache, and retrieving them again some many seconds later, won't get the actual version from the server. But, if coincidentally we were not speedy at all in our previous acttions, a GO (or whatever) to the record will retrieve its actual version. What to do with that ? just nothing I guess.
One thing : an RLock of the record should refetch it from the server as always.


The below further comments obviously relate to the new info as from the above.

(Peter)



>>
>SNIP
>
>Peter,
>My apology for being so late with a reply. My comments are in-line...
>>
>>I have some perceptions of the "cacheing" operations of VFP, I'd like to be shot at. Or better, to be worked out as to how it really is.
>>It is the most easy to describe a scenario as my perception :
>>
>>1.
>>At the server we have a VFP table. It comprises of, say, 4KB blocks (clusters). At some moment in time, 100 blocks are in there. The last block just cannot hold a new record to be added.
>>
>I don't think we know if VFP is programmed to fill whole "clusters" regardless of the length of the records (thus having records spanning clusters) or if it operates allowing only complete records per cluster (thus leaving unused bytes at the end of every cluster). My guess is that it always fills all clusters (except the last, as necessary) but it would be nice to know for sure.

As we now fairly can say, VFP is not about this at all. It doesn't obtain the "physical" blocks, it doesn't contain them, hence it can't write back to them. It just passes back the record itself, and the offset of where it must go to in the file. It will be the file system dealing with that further.
Note : in order to find out how the file system is dealing with this, it will need tools I don't know. And, because of the file system dealing with these kind of things, it will be different from the OS where the file system is subject to. It would be my conclusion that this can not be the file system of the PC, since VFP itself proves that it goes beyond that (it requests at the byte level, in the end surely resulting in reading (or writing) from (to) some physical (cached) block somewhere.

>
>>2.
>>User A wants to change a record in block #100. He performs an RLock(), causing the block to be refreshed in his PC.
>>Never mind preceeding SEEK's etc.; the RLock() has to be performed in order to receive the latest version of the block. In the end it is about the latest version of the record, but the block is received in the mean time.
>>Supposed 3,700 records are in the 100-block file, and this is about record #3698.
>>
>>3.
>>User B wants to add a new record, and all he does is an Append Blank.
>>This implies a lock of the header of the table, so not two users at any same time will write to the header. Hence, the append blank from User B will result in incrementing the record count, in this case from 3,700 to 3,701.
>>After the header is updated, it is unlocked again.
>>
>OK, but I think it is fair to assume that a blank record 3,701 is also written.

You are right. Forgot to mention that.
And, note that the scenario doesn't fit anymore.


>Also, you specified above that the last block just cannot hold a new record, so we should assume that a new cluster #101 and it's EOF Marker is also written. For purposes here, let's ASSUME that the new record spans cluster 100 and cluster 101. [I see later that you assume this too]
>>Now all my assumptions start ...
>>
>>4.
>>User A performs a replace at record # 3,698. He is not performing an unlock yet. Hence, the new data from record #3,698 will stay in User A's cache only. No fresh copy of the block is obtained, because it is not necessary; the record is already locked.
>>It is here where I can be dead-wrong.
>>
>We do not know for sure that a REPLACE command remains ONLY in the workstation's cache until a unlock is executed. A SET REFRESH may cause it to be written WHILE the lock is still held. There may be other factors that could cause the record to be re-written while a lock is still held. One thing is certain - other than some performance impact it would be perfectly legitimate for a locked record to be re-written. It seems that a FLUSH would cause a re-write, don't you think?

A.
For the developer's logic reasons, it should be so that the record won't be flushed to the server until it is unlocked. And, in normal situations is surely won't get flushed.

B.
I certainly agree that there will be situations (thinkable) that records get flushed out of the developer's control. I don't think the REFRESH can imply this, and certainly should not imply this. But, the PC's (VFP's) cache certainly is limited, and I can't recall a situation that an "Out of memory" or similar will be presented because VFP is running out of (write) cache. This makes me think that just overloading the cache will already imply a flush.

C.
No, I can't agree with it being legitimate that a record will be re-written (should be re-written) when the record is still being locked. Remember, this has all to do with app-logic, and the developer has to be (able to) depend on the logic on "transactions". It is therefor offical(ly known) that records won't flush until the unlock (with the table concerned being active (Select)), and an UNLOCK ALL will imply the logical transaction. Also do not forget that Novell once (still ?) anticipated formally on XBase's working by TTS (Transaction Tracking System) incorporating with XBase (possibly this was FoxBase/FoxPro only). TTS could deal with implicit transactions by means of RLock - Unlock only, and it could deal with explicit transactions by means of commands as BEGIN TRANSACTION, ROLLBACK, COMMIT etc., and Novell came with a Fox Library especially for this;
We talk about the implicit transactions here.
Sidenote : the implicit transactions will still be supported by means of TTS still existing, but I am sure that it just doesn't work. Or because of Novell's improper working on TTS itself, or because of VFP not dealing with it properly, like flushing the record(s) out of the developer's control ...

D.
FLUSH causing a re-write of records being locked ? ...
I never tested that, but I would expect this not to be the case. To this respect, I have the feeling of expressing my thoughts on FLUSH : this command should not exist. I have a few reasons for this :
a.
FLUSH flushes all of the cache of VFP, but retains the cache anyhow.
I don't expect the FLUSH to explcitly re-fetch all the actualized mapped records from the cache. Note that I did not test this.
Also note that these findings are from several years ago, and right now it could be different.
Furthermore, FLUSH is exceptionally slow, just because it flushes all of the cache again and again. That is, without writing to the records, it does.

b.
From the tests in the past, I found that not all PC's behave the same on the FLUSH command. That is, some PC's (that is OSses) just do not flush the whole cache. What it then flushes ? don't ask me.

c.
I cannot apply any logic to the FLUSH command in the multi user environment, thinking of the locking of records being the basis of consistent transactions. I mean, either way transactions can be nested, but FLUSH cannot. Furthermore, an Unlock of a before locked record flushes for sure, so how to use the FLUSH in this environment ?

Note : the FLUSH causing a possible re-write of a locked record can obviously be tested easily. Right now I don't feel like it ...

>>5.
>>At the append blank of User B, User B will obtain a fresh copy of block #100, because its last portion contains the beginning of his new record 3,701. User B also receives block #101, because in there is the remainder of record 3,701 (the record contains blanks only at this moment).
>>Note that User B will not obtain the record #3,698 as how User A already changed it. User B will see the old contents of it (when he'd look for that).
>But we cannot be certain of this. But in any case record #3,698 is still locked, so we will see what develops.

Here too, the scenario doesn't fit anymore; We can't talk "blocks" anymore, and instead we now must talk "bytestreams". The lentgh of the stream is unpredictable ...
I can't even begin with re-writing the scenario with the knowledge as off today. One thing stays : I never see it flaw.


>
>>Also note that User B is legitimatly able to obtain a fresh copy of block #100 (as he just did), but that "fresh copy" means : as how it is on the server's (virtual) disk.
>>It is my statement that by, for example, performing an RLock() on record #3,699, all records from block #100 will be refreshed to the status as how it's on the server's (virtual) disk. A user could perform any RLock() upon a record in block #100 (say records 3,663 u/i 3,700), but for record #3,698. Hence, this record will fail to get locked at this point in time, because User A has got it locked for himself. And, because the lock won't be done, the block will not refresh either.
>>
>Well again we don't know for sure. But I would use this situation as evidence that, to be perfectly up-to-date with all data, VFP must re-write records that are (still) under lock.

This can not be true, because it will violate the transaction logic. I hope you agree on this.
I can add one thing related to the above scenario :
My tests upon retrieving the null data opposed to the real data (see explanation in the lead of this message), showed that there was only one means of getting the same results always, and this was an RLock of the record I GOed to. Though I didn't apply the sniffer on that test, I can fairly say that an RLock will read just the byte stream of the one record only. IOW, the fetch of the record as implied by an RLock, will fetch a stream of the record length only. Somehow this is logic too :
Firstly it is not allowed to fetch the record from cache because the RLock is just featured with making the record the most actual as possible.
Secondly, there won't be any reason to fetch a larger stream (i.e. get more records), because I never want to RLock the next record before doing something else. That is, when SET MULTILOCKS = OFF ... :(
So in the end it needs the sniffer again, I guess.

One thing is for sure : all the implied logic of mine from before on fetching a record, and all other records in the block, is just wrong. Instead, the VFP command to fetch a record (but RLock) will get a fresh copy from all the records as VFP thinks is "smart". Not because of logic around getting fresh copies, but because of the logics around being speedy.
Working this out furtherly, cannot be done, and won't help us whatsoever.
Instead now we can think something like : The faster the PC and its environment it, even the faster we can obtain the records from the server. "Even the faster" means : not at lineair speed (but better).

Please note that the workings as found out now, won't allow anymore to test things on the subject from the command window. Furthermore, "your" test from some program will present different results from mine. What to do with it.
I think this may be the best kept secrect from the Fox until now;
Tests will usually be done from the command window. It won't unveil the internal working. It was just the stupid coincidence that I could visually see the different results by means of receiving nulls opposed to real data.
Rushmore ...


>
>>Note that in between this all, the VFP activity on the Set Refresh is operating, and that that might cause a refresh of the block afterall. But again, the contents of record # 3,698 as how User A sees it right now, will never be obtained for that contents : it's just not on the server's (virtual) disk.
>>
>To me this is the argument supporting that VFP DOES re-write records while still under lock.

Jim, personally I read the Help on SET REFRESH a couple of hundred times (somewhere in the past). This is because it seems to be there to confuse you, which it certainly will when you don't know the working in advance.
BTW note that it helps to compare the Help on this from FPD and VFP. They are slightly different and vary in various versions as well.
Also note that SET REFRESH's 2nd parameter already operated from off FPD 2.00, though it was an undocumented feature then.
Anyway, SET REFRESH is for the reading-side only. It does nothing to the writing side.
The parameters are about refreshing the records in the PC's cache on one hand, and the Browse (and some more) able to show it on the other hand. Parameter 1 is about the last, and 2 is about the first (let's make it extra confusing ;).
When the parameter (the 2nd) 's timer elapsed, VFP asks the server for all of the records in its cache again. O yea ? no ! this works rather differently : it just throws away the cache. Remember, a far more easy way to imply the similar. But now think of your own statement on the re-write ...
What will it do ?
I did not test this explcitly, but from earlier tests (and I always test with Set Refresh to 1,1), I think (!) I can fairly say that locked records will stay in the cache.

Anyways, think of how difficult analyses become, knowing that the Refresh timer interferes with the reading ahead (that's just what it is) as described in this message; What this comes to, is that read ahead records will be removed from the cache "by accident" (i.e. you don't control that) and because of that they have to be re-read again *if* that process is current, which is by accident again. Furthermore, the accident of me asking for a record, and the Refresh timer elapsing 0.1 second after that, will *Rushmore* VFP to get more records this time.
The only thing I learn from this really, is that further tests should be at Set Refresh to be at 3600,3600, in order not to let interfere Refresh with my tests. But, for practice testing this is nothing because my app will not work with 3600,3600. Now what ?

>
>>6. User B, who just appended the new record, is performing an Rlock() to record #3,701, for his due replace.
>>In between the append blank and the replace there is a point in time we can let it all flaw. That is, when no file locking (FLock()) is used, within this period the appended #3,701 can be obtained by someone else, and can be replaced by someone else. But, this will be shooting the "system", whereas by no means logic can be found in such an operation.
>Keep in mind that if the record #3,701 is involved in a FPD/FPW "screen" GET field or a VFP "form" entry field (at least with no or pessimistic buffering) the APPEND BLANK would immediately be followed by an intrinsic RLOCK() for the record in question. I'm with you in that I feel confident that VFP covers the situation else there would be corruption all over the place. Possibly APPEND BLANK places a RLOCK() automatically and the documentation just doesn't say so.

Personally I do not believe it will work like this. I mean, thinking of some structured programming, "VFP" will just call the normal Append Blank procedure, leaving you with a time-gap for another to fetch the record. But, you could be right. And, though never tested, I already expect SQL - INSERT to even work via an internal Append Blank and Replace. However, in this case at least the record will be locked from Append until Replace (if the both are separated internally), on one hand
just because it is faster ... (opposed to the explicit Append Blank followed by Replace).

Please note that when you experienced the whole life of Fox step by step, it is rather easy to see how the development went. To my experience it's always new things on top of other existing things. And do we not work all like that ? Added to this, I am 100 % sure that even Visual FoxPro has been built on top of FoxPro for Dos, and for instance, really nothing changed in the data engine (but for some 32bit manipulation).

>>
>>7.
>>At this stage, User A unlocks record #3,698 and because of that it will be flushed to the network's disk. "It" means the whole block #100, because transfer is at the block level.

Completely not true anymore ... Only the one record will go to the network.
But ... note that somewhere in there, all comes to the same again : the whole block will be physically written.

>>How will this physically work ? It hardly can be true that some "dumn" file system is able to merge the new block #100 from User A with the current new block #100 as implied bu User B. Or can it ?
>>Two options exist :
>>a. The block is re-fetched to the PC first, and all the intelligence necessary is applied by VFP;
>>b. The block is not re-fetched, and the file system is performing a merge.
>>I can only fo for a.
>>Assumed that a. is the case, it won't be allowed to receive record #3,701 because it is locked. Furthermore, what about the #3,701 being spread over block #100 and #101 ? Will User A receive Block #101 anyway, anticipating on it now logically being connected to #100 ? Would a Go Bottom imply the presence of #3,701 anyhow ? I think so ... But, User A can't go around with replacing #3,701 because it is locked by User B.
>>
>I really believe that some kind of "merging" is done by VFP. I believe that it has to as this is the only way that I can see for FP/VFP data to always be accurate, which it always is (except for the odd corruption instances that may be bugs in the processing logic).
>As mentioned earlier, it may well be that locked records are re-written upon a replace or periodically for other reasons.

On one hand, all now can be looked at as to the file system now dealing with this. But please note that at this stage of my analysis I did not sniff the writes ! IOW, on the writes I am assuming things so far.
Note also, that it could very well be that the Fox is the only being that smart that it gives requests at the byte level (whereas other software operates at the block level, really coming to the same but asking for other offsets and a stream comprising of the block length ...). Now, if it were so that Fox is the only software operating at this byte level, it very well could be that the file system has no means of merging, and that Fox has to do this itself (!!). This implies that the sniffer should show whatever fetches of records subject to the one record written, which will come to the mapping onto the physical block afterall. But hey, it's better to sniff than to speculate. So I will do that some near day.

>>8.
>>Now User B is performing the replace, and when the assumptions so far are right, it will imply a re-fetch of block #100 and #101 first. The data from #3,701 is now applied to both block #100 and #101.
>>User B is now performing an unlock for #3,701.
>>
>>9.
>>As I assume it, now User A is according my statements at the end of #7, capable of replacing #3,701, overriding the contents as just applied by User B. But again, this is not applying proper app logic, because first User A should RLock() #3,701. When he does that, he will obtain the fresh contents of block #100 and #101, record #3,701 being part of it.
>>
>>All poper working depends on the re-fetch of the data, even when the record concerned was locked already. I personally wonder whether this is true ...
>>I believe that this has to be true.

Me too, but whether it is Fox doing it, or the file system dealing with that internally, I can't tell yet.

>SNIP
>I feel that the general integrity of data has been proven, if only by time alone.

100 % agreed.

>This doesn't, however, preclude the existence of bugs, especially as concerns the more obscure situations that may arise.

Yes. But now wonder : where these bugs are ? And remember, a bug in the file system, obviously can be worked around in VFP's kernel. Right ? So, how often does this happen ?
Furthermore, didn't we know about the x100 bugs in the client software, just causing all of the corruptions we saw ?
I now can tell, that apart from the writing-area yet unknown to me, the reading as how VFP this explicitly does is far too simple to cause anomalies. That is, once all is up and running in some environment, it becomes very hard to think that VFP could crash on it. But what about all that's behind VFP ? it's one big pile of complexity ...
Personally I would be very careful in thinking that corruption is caused by VFP itself. I always thought that already, and so far I did not find one corruption caused by VFP itself (BTW, in the FoxBase time there was). Just like my corruption from today : it is already proven that it's caused by Novell.

>An end-of-cluster situation presents at least four possible situations:
>1) There is room in the existing cluster for a whole new record and then some. Most frequent, I guess.
>2) There is exactly room in the existing cluster for the whole new record and its EOF Marker. hardly frequent I guess.
>3) There is exactly room in the existing cluster for the whole new record BUT not for its EOF Marker. Least frequent I guess.
>4) There is room for only a part of the new record so a new cluster will also be required. Second most frequent I guess.
>Logic for the above may be impacted too by the cluster situation on related .CDX or .FPT files or by the sequencing of writes to the three different files.
>
>Can you think of some way to obtain the answer to the unknowns suggested in all of the above? It seems to me that this basic missing (I assume) information could be published by MS without jeopardizing its patents.

Well Jim, it's my guess that you think differently now.
What I suggested earlier, I now make more explicit : I'd say that what I just found is that smart, and, that this causes so very much improvement on speed, that is is just this what Rushmore is about. Think of it :
It allows the server for coming up with one-command sequential reads of (eventually) very long streams of data (that is, it allows for the sequential reading of at least index blocks at, say, 100 at a time). Dropping back to the low level of disk performance, you can imagine how this would improve response dramatically. Let's say that obtaining 100 physical blocks from the server, the server knowing in advance it must do this, will improve speed at the rotation speed of the spindle, eliminating head-swaps at all, or so. Divide the rotation speed by the avarage access time (or similar), and you learn the potential improvement on performance.
In analyzing this, you must compare it with the single physical block fetch if all would be normal. Personally I always wondered how Rushmore dealt with the SKIP 10.000 coming up "at once". Though I did not test this (gee a lot to test here), I am pretty sure that it is just some offset to present to the server. That is, now I know (saw) how the fetch of one record works.
It just needs to know in advance that subsequent records are needed in order to be able to do it like this.
Personally I don't know Rushmore at all, but the ones who do (when can it optimize etc.), probably will come up with the "why's" after reading this message. I'd say, try it.

>
>cheers

Pffff.

Peter