Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Fragmentation - why it is usually good for VFP native data
Message
From
15/01/2003 13:47:35
 
 
To
All
General information
Forum:
Visual FoxPro
Category:
Databases,Tables, Views, Indexing and SQL syntax
Title:
Fragmentation - why it is usually good for VFP native data
Miscellaneous
Thread ID:
00742043
Message ID:
00742043
Views:
92
WARNING - long-winded diatribe following. . .

This tries to explain why it is that a typical VFP application benefits from file fragmentation as it occurs naturally on a NTFS system.

Important (make that CRITICAL) note:
In fact we have positively NO CONTROL whatsoever as regards file fragmentation. The best that we can do to influence its impact is to:
1) Set conditions up so that fragmentation that will naturally occur does so in the most predictable way;
2) Leave it alone once we have done so.
To set the optimal conditions we should:
a) Ensure that the volume or partition holding the tables/files is fully defragmented prior to creating any of the tables/files;
b) Ensure that no other application (things like Word or Excel or whatever) write anything to that volume/partition;
c) Ensure that table/file writing sequences are consistent between revisions of the application.
d) Ensure that VFP TEMPFILES/SORTWORK/EDITWORK CONFIG.FPW statement point to some other volume/partition.

Even then this state is nearly impossible to maintain!! A PACK or a REINDEX will muck it up. Adding (or deleting) a field in a .DBF will mess it up too. A COPY TO...RENAME (to one of the table names involved) will too! So, all in all, it is an impossible objective and will remain so until MS (or someone else) gives us the tools to manage fragmentation. That day may be a long way off UNLESS WE START CLAMOURING FOR IT.

Basics of (NTFS) space allocation
  • The file system allocates space in "clusters" (default size = 4096 bytes, changeable by the FORMAT command).
  • A default cluster is made up of 8 contiguous "sectors" and each sector is comprised of 512 bytes.
  • The file system will always try (and usually succeed) to allocate contiguous clusters for the length of data to be written.
    --- In addition to a (record) length, VFP will tell the file system the relative position (offset) within the specific file that the length specified is to be written (the current end of the file).
    ---That offset will usually mean that the last (already) allocated cluster will get some, if not all, of the data specified by the length to be written.
    --- In the case that the whole length does not fit into the last allocated cluster, the file system will then allocate sufficient clusters (usually 1, of course) in which to write the balance of the data to be written.

    Basics of a hard drive data space (for the drive I used for testing (MAXTOR 6L040J2)
    Being a more modern drive, this one uses "zones" to take maximum advantage of the recording surface area, specifically the greater surface area available for tracks moving outwards from the centre of the spindle. This one uses 15 zones per surface. The inner zone has 481 sectors per track and the outermost zone has 882 sectors per track. Any calculations following will use 481.
  • Number of platters = 1 (thus, 2 recording surfaces used (top and underside))
  • Number of R/W heads = 2
  • Tracks per surface = 58,970 (spread across 15 zones)
  • Track-to-track seek time = 0.8 ms
  • Average seek time = 8.5 ms
  • Full stroke seek time = 17.8 ms
  • Bytes per sector = 512
  • From the above the following are applicable for the standard NTFS allocation:
    --- Clusters per track (minimum) = 60.125 (don't know if the .125 is used or wasted)
    ------ This gives a total track byte capacity of 245,760 (ignoring the .125)
    --- Clusters per cylinder (minimum) = 120.250 (this is the total clusters useable before a head movement would be required, under specific conditions)
    ------ This gives a total cylinder byte capacity of 491,520 (ignoring the .250)
  • Note that at the outer tracks these numbers are 110.25 and 220.50 respectively (450,560 bytes and 901,120 bytes).

    Basics of VFP record writing in a shared environment
    Being USEd shared, VFP will want to ensure that any record insertions (INSERT INTO...., APPEND BLANK/REPLACE, with or without TABLEUPDATE()) will be written immediately to the HD.
    This means that VFP will tell the OS file system that it wants to write nnn (length of 1 record) bytes to file XXXX.DBF at offset oooooo (the current end of the table). If that record contains a filled memo field, it will also tell the OS file system the same things for the .FPT file. Finally, being a new record, the table's .CDX will need writing although in this case it could be an update for existing .CDX records or it might mean writing (some) new .CDX records or it could result in both.
    Note: The default "BlockSize" for a .FPT file is 64 bytes(including 'overhead' of 8 bytes) and the size of each .CDX record is 512 bytes. The standard "header record" sizes for each file type are ignored for this discussion.

    A scenario
    Assume a typical VFP application where a user "save" results in the creation of 3 records in 3 separate tables. These records obviously have some relationship to each other.
    Allow that these records have lengths of 239 positions, 135 positions and 293 positions, each including the DELETED byte.
    Allow that the record of 239 bytes holds one memo field and the record of 293 bytes holds 2 memo fields.
    Allow that the 239 byte records hold memo data (60 bytes) in every 3rd record and the 293 byte record holds data (60 bytes and 100 bytes) in each memo field every 4th record.
    Allow that all 3 tables have 5 TAGs (they are related, after all), 2 of which are INT, 1 is Char(16), 1 is Char(10) and one is on DELETED().
    Allow that this application writes a single record to each table for every "save".
    This will result in tables/files of the following sizes (VFP7 SP1) for 500,000 records:
    File     total Bytes Clus. Cyls Recs/Clus
    T1.DBF   119,500,553 29,175 243  17.14 
    T1.CDX    24,246,784  5,920  50   8 
    T1.FPT    21,333,760  5,209  44  64 
    
    T2.DBF    67,500,457 16,480 138  30.35
    T2.CDX    24,246,784  5,920  50   8 
    
    T3.DBF   146,500,585 35,767 299  13.98 
    T3.CDX    24,246,784  5,920  50   8 
    T3.FPT    32,000,512  7,813  66  64
    With DEfragmented tables/files (assumed to be laid on the HD as listed above). . .
    To do a SEEK to each of the 3 tables, and assuming that the records in question all happen to be in the first cylinder of each table (they are related, after all) then the best case situation would be:
    1) Read some .CDX records in T1.CDX. Good chance that these are on the first cylinder of the file.
    2) Having the record number, now read T1.DBF. This means that the heads MUST move 243 cylinders to reach the record and read it in.
    3) Allowing that the DBF record shows no content for its memo field, that SEEK is now complete.
    4) Read some .CDX records in T2.CDX. Good chance that these are on the first cylinder of the file. This means that the head MUST move at least 475 cylinders to read the T2.CDX records.
    5) Having the record number, now read T2.DBF. This means that the heads MUST move 138 cylinders to reach the record and read it in. There being no memo fields here, this SEEK isnow complete.
    6) Read some .CDX records in T3.CDX. Good chance that these are on the first cylinder of the file. This means that the head MUST move at least 487 cylinders to read the T3.CDX records.
    7) Having the record number, now read T3.DBF. This means that the heads MUST move 299 cylinders to reach the record and read it in.
    8) Allowing that the DBF record shows no content for its memo field, that SEEK is now complete.

    From the above it is clear that the total head movements were (at least) 5 and the cylinders crossed total 1642. It seems fair to allow one half of the "average seek time" for the device for each head movement, resulting inan aggregate head movement time of 21.25 ms.

    Of course the tables/files would not have been defragmented without some external action. Instead, they would have been written basicaly "interleaved" on the HD. Of course only the first cluster of each table/file would be exactly interleaved. Later interleaving would be dependent on full cluster consumption, which itself is dependent on the record sizes (DBFs) and write frequencies (CDXs and FPTs). In addition the order would have been DBF, FPT (when applicable) and CDX for each file.
    Every 18th record of T1.DBF would cause an additional cluster allocation and significantly more T1.DBF records would be required to cause additional cluster allocations for its CDX and FPT files.
    Every 31st record of T2.DBF would cause an additional cluster allocation and significantly more T2.DBF records would be required to cause additional cluster allocation for its CDX file.
    Every 14th record of T3.DBF would cause an additional cluster allocation and significantly more T3.DBF records would be required to cause additional cluster allocations for its CDX and FPT files.
    The T1.DBF and T3.DBF files will more or less keep pace with each other cluster-wise and each of these will outpace T2.DBFs cluster counts by roughly 2 to 1.
    Now because there are at least 120 clusters per cylinder (i.e. the R/W heads can access all of those clusters without movement it can still be safely assumed that "related" records remain in very close proximity to each other.

    Repeating the above but with the tables/files left as "naturally" laid on the HD. . .
    The chances are excellent that most of the data need resides on the same cylinder, but we will stipulate that each relevant table/file record is at least 1 cylinder away:
    1) Read some .CDX records in T1.CDX. Good chance that these are on the first cylinder of the file.
    2) Having the record number, now read T1.DBF. This means that the heads MUST move 1 cylinder.
    3) Allowing that the DBF record shows no content for its memo field, that SEEK is now complete.
    4) Read some .CDX records in T2.CDX. Good chance that these are on the first cylinder of the file. This means that the head MUST move at least 1 cylinder to read the T2.CDX records.
    5) Having the record number, now read T2.DBF. This means that the heads MUST move 1 cylinder to reach the record and read it in. There being no memo fields here, this SEEK isnow complete.
    6) Read some .CDX records in T3.CDX. Good chance that these are on the first cylinder of the file. This means that the head MUST move at least 1 cylinder to read the T3.CDX records.
    7) Having the record number, now read T3.DBF. This means that the heads MUST move 1 cylinder to reach the record and read it in.
    8) Allowing that the DBF record shows no content for its memo field, that SEEK is now complete.

    From the above it is clear that the total head movements were (at least) 5 and the cylinders crossed total 5! Using the track-to-track seek time for the device, the aggregate seek time is 4.0 ms.

    Now there is some "strangeness" as regards the reading of .FPT data relevant to any record. Testing has shown that "defragmenting" (using FILETOSTR()/STRTOFILE()) just the .FPT tables results in significantly faster SEEKs and Select-SQLs. However, the impact of doing this is not known as concerns subsequent writes to the tables in question. My inclination would be to leave the .FPTs alone. But if the files are not big already, then I might give consideration to "defragmenting" the .FPTs on a frequent regular basis.
  • Next
    Reply
    Map
    View

    Click here to load this message in the networking platform