Level Extreme platform
Subscription
Corporate profile
Products & Services
Support
Legal
Français
Text files with different structures and delimiters
Message
From
09/12/2001 12:37:08
Dragan Nedeljkovich (Online)
Now officially retired
Zrenjanin, Serbia
 
General information
Forum:
Visual FoxPro
Category:
Coding, syntax & commands
Miscellaneous
Thread ID:
00591251
Message ID:
00591837
Views:
30
>>Not that I wish to discourage you, but while this sounds simple, there are more speed bumps down the road. You'll discover some repeated values which actually fit some of your lookups, and you'll match them to your lookups... until once you discover they've changed the set of values on you. Since they owe you nothing, they won't tell you. I was maintaining and speeding up this code for three seasons, and it's never been simple. But it's feasible, and if you don't have to do much data massaging, it may be easy.
>
>Sounds like a plan and very complicated one, but I discussed this problem already and found few things:
>1) We pay something for the data, but very cheap price
>2) We receive this data ~ once in 6 months
>3) It's unlikely what data format would be changed, but possible (as it happens this time)
>
>So, I'm going to take a simplified route, as I described in my reply to Winn: check the current table structure with what they have in the first line (if they have this structure there) and if it doesn't correspond, invoke modify structure dialog... I may use your idea of intermediate "conversion" table to make this comparision easier...

If it's twice a year, you don't really need this heavy artillery approach. In my case, it was run at least daily during the season, and had to split the data into a dozen tables (i.e. going from the flat file to the normalized tables). The guys who wrote the first version were reading it byte by byte, and fSeek()ed the previous byte if it was a quote or apostrophe and replaced it with a space... which then took about 30 minutes to import a 250K file. I had to get that part below a minute so that I could concentrate on the second part - getting the distribution of the data into child tables properly. I think the last version ran at a rate of 10K/s.

You most probably don't need this sort of automation, i.e. this seems to be better suited for the programmers of the 1st kind :)

back to same old

the first online autobiography, unfinished by design
What, me reckless? I'm full of recks!
Balkans, eh? Count them.
Previous
Reply
Map
View

Click here to load this message in the networking platform