Originally Published: Friday, 3 August 2001 Author: Kristina Pfaff-Harris
Published to: opinion_articles/opinion Page: 1/1 - [Std View]

Kristina in Space: Sometimes Scripts Just Don't Help

Kristina returns with a tale of terror and taming the wild beast that is administration. You think you had a bad week? We're surprised Kristina can still even focus on the screen.

Every so often, no matter how hard you try to keep things stable, you run into one of *those* weeks. You know the kind I'm talking about: the ones that start out okay, but then sort of sneak up on you from behind, and suddenly leap out shrieking, "HA! What are you going to do about THIS?"

It's been one of those weeks. I'd been working on a project for some time now, trying to move a lot of text files and other stuff scattered all over the system into a nice unified database. (For the techies, I'm using Oracle and MySQL. Yes, both. Don't ask.) Luckily, many of the text files are generated by some sort of script or program. This means that their contents are arranged in a more or less predictable manner, so that you can write another script or program of some sort to grab all the data, moosh it around, and shove it into one of the database tables. No problem: a little tweaking here, a few manual fixes there, and everything is going oh-so-smoothly. I merged all the billing and invoicing tables: no problem. I updated all the back-end tools for the staff: again, no problem.

This was until I got to the point where I had to find out who owned each of the email addresses (roughly 20,000) on the system. I worried just a bit, as I always do when dealing with something that has the potential to affect people's email, but I went on despite my misgivings. After all, things had gone so well up until then. "Worried about email?" you ask, "It's just data. What's the big deal?"

Well, let me put it to you this way: in my experience, you can take someone's website down temporarily, and they'll be a annoyed. A few people will get extremely angry, but mostly a minority. You can break someone's dialup, cable, or DSL account for a bit, and while they might complain, everyone seems to understand that the Internet is finicky and these things happen. But make a mistake on someone's email for 5 seconds, and BAM! The world is ending! Fire, Flames, and Horror! Millions of dollars being lost every single second! Doom, I tell you, Doom!

In short, email may in fact be the single most important part of many people's very existence. That means no mistakes can be made, via computer glitch or otherwise. This ordinarily wouldn't be a problem either. After all, I've got all this data in all these different files. All I have to do is figure out for each and every email address, which actual customer account owned it. (No, none of the files had this particular piece of data readily available. Again, don't ask.)

Now we did have sendmail that comes with all those handy "virtusertable" text files, and some of the data was in a MySQL database and some in Oracle. So it was possible to write a script to just run around and check all these different places for what we were looking for until we finally found a valid account id. I wrote up a little script to look for things more or less the way I would. You know, check this file, if it's in there, match it up with this other thing from this other file, look in the billing database to make sure it's a good account ID, if not go here, etc. I let it go running around, and it finally came up with an account ID for each and every one of those thousands of email addresses. Life was good, and all was in harmony. This, of course, was Monday, and the week was just waiting to ambush me.

Tuesday arrived in beauteous splendor. Okay, it wasn't pouring rain, and it wasn't 104 degrees outside: that's beauteous in my book. Just stay with me. I was doing a random spot check of my newly database-ized email addresses, sorting them in different ways, and so on. Everything was going fine until: I noticed a mistake. Oh, it was a small mistake to be sure. For some reason, my script had assigned someone's email address to an old account ID which had been terminated in 1997. Since the whole point of matching up all this stuff was so that we can get all those old terminated accounts off the system, the idea was that every email address that was assigned to a terminated account could just be dropped. This one, however, actually belonged to a real customer who is still still using it. In addition, the real customer in question is known to be extremely dependent on this email address. The tech support database is filled with calls from this person from whenever the email just "seemed a little slow." Not good. Not good at all.

"HA!" said the week, at approximately this moment, jumping out from behind my bookshelves, "What are you going to do about THIS?"

My friends, I panicked. I couldn't bear to blithely palm that one off on the tech support folks when the phones started ringing. I've been tech support: when people can't get their email, it's not pretty. There was no help for it: I had to manually check everything.

Oh, I did manage to narrow it down a bit, so that I only had to manually check about 4,000 or so. But you know, it's things like this that make it "one of those weeks." This happened on a Tuesday. I spent 12 hours going through 4,000 records and found 5, count 'em, 5 mistakes. The worst part is, you can't win: a little known corollary of Murphy's law clearly states, "If you don't manually check all 4,000 email addresses, there will be at least 1,000 mistakes. If you do spend all day doing it, there will be 5 or fewer mistakes."

I wanted to blame it on someone, but who? Linus Torvalds for inventing Linux so I could run a mail server? Eric Allman for inventing Sendmail? Those MySQL AB guys for making it easy to run a database? Or even Larry Wall for inventing Perl so I could write my text-file-scarfing, database-filling scripts? No, no. Blame serves no purpose. In fact, in the end, I had a very nicely arranged set of data, happily tucked away in a MySQL database for quick searches and modification. The end result was accomplished. It's over now, and I should look forward and not back.

It could, after all, have been much, much worse: the problem could have affected a customer's monthly bill instead. I shudder to think of that! At least it's over...until next week.