>>I've seen all kinds of weird shit and stuff which took hours to debug, or rather to understand how it'd happen, when operators found their inventive ways to cover for their errors, thus making things even worse,
>instead of having a regular way to fix them without consequence. It seems the goal of having the correct data is forgotten, and the perceived relationship with the boss takes precedence. So they cover it up, they won't exactly say what they really did until you present them with things you found out that they thought you never would etc etc. Sherlocking out such problems is often taking more time than writing a form in such a way where they could have fixed their error right away.
>>
>>Which is exactly why these things happen - the app is not quite tested yet, there are recent additions, code is tangled here and there, not refactored into clearly separated pieces, the same piece of calculation may exist in two places but be preceded or followed by some code that's slightly different etc etc. OTOH, the boss doesn't have a clue but keeps up appearances by calling his operators stupid for any mistake... all the ills of a hierarchy. And then they don't ask you to fix the system so that they can correct any such errors in the future, they want you to help them cover their tracks without telling you what exactly did they do, and without any of it reaching their boss.
>>
>>Took me a while until I learned to avoid such problems in advance, but then I lived happily thereafter.
>
>Full ACK, usually I get along with testers and DBA - until the odd one disturbs clear water. Then IMO it is better to implement a few safety lines without spreading the news, gather data for yourself under "testing and verifying" and use when odd one starts to throw funny stuff into your back yard.... Sometimes can nip discussions before they really start if origin of "wrong" data can be pinpointed better than odd one expects.
>Even if only CRC plus encrypted edt_time+edt_usr point to fudging or some audit trails are saved elsewhere...

I wish we had such tools back then... but even what we had (few routines to just look at raw data, such as they were) was enough to instill the fear of the all-seeing admin, who knows when their excuses don't hold water. With an audit, ooo... such things would have been done ten times faster, and the fear of the all-seeing admin would be set in stone, all upercase, 72px font.