>I have to create a table of 35 millions plus records. The table is to be populated from 850 fixed length text files. I get about 1/3rd of the way through with the imports and I hit the 2 Gig limit for Visual FoxPro tables.
>
>I took the 2 Gig dbf file and exported it to a delimited text file to see how much smaller it might be since there are a lot of empty fields in the table. To my surprise the 2,079,274 KB table made a 2,066,947 KB delimited text file.
>
>Other vendors have this data is some sort of file structure that only takes up about 450 Megs. They distribute this data (in .dat format whatever that is) via a CD-ROM and access it directly off of the CD via their apps on the fly without any major writing to the hard drive. They can access the data immediately upon demand without any obvious decompression. For me, just trying to zip the 2 Gig text file takes forever and slows the computer to a crawl.
>
>What type of data format could they be using that can store what would have been a 6 Gig delimited text file into only 450 Megs?
>
>I need to be able to do the same thing. I can’t distribute the data in zip format and expect the user to take 40 minutes unzipping it and also expecting him to have 6 Gigs of available disk space not counting space for the indexes that will need to be created.
>
>Any ideas or theories about the file type might be and if there is any way that I can do the same thing via Visual FoxPro (or VB or C# if I have to)?

I don't know if it works with VFP but you could look at http://www.xitech-europe.co.uk/compaxion/index.html