Tuesday 19 March 2013

The Missing Data Conundrum

It's nearly quarter to two in the afternoon. It's been a busy day, but a productive one. A project I have been working on for almost four months has finally come to fruition. After months of working with a software developer, I finally have a working upgrade package for one of our vital systems. The code has been patched, the bugs eradicated and the end is in sight.

I had been given this project for the simple reason that I had heard the name of it sometime in the past. I had no idea what it was, what it did, who used it or why. I had been thrown in the deep end of the data pool. The current version we had was three years out of date, and the new version refused to work in our environment. It was my job to fix it!

To cut a very long and boring story very short. The software developer built us a new package and all was right in the universe. However, one part of it still refused to work, so on this particular Thursday, he had stripped it down once again and finally found the problem. Satisfied with a job well done, I thanked him, put the patched file in a safe place and sat back in my chair, giving myself a silent congratulations.

Then...

Like all good IT Nightmares, it all began with a dreaded phone call. Half an hour ago...

Yes, as I had been collaborating with the developer, putting the final touches to our masterpiece, a call had come in reporting that the data had mysteriously disappeared from that system. Some forty minutes after hanging up from the Developer, I receive the news.

"Why wasn't I told sooner?" I ask.

"You were on the phone" I am told.

"I was on the phone to the developer! He could have fixed the problem."

"Well, we knew you were talking with him, we thought it was something he had done, so we didn't say anything!"

"We didn't even know there was a problem!" I explain.

Isn't communication a wonderful thing!?!

In the mean time, the person who had reported the problem was non too happy that they couldn't do any work. With the data missing, the system was blank and useless, and she was more than a little unhappy.
This users workstation was playing up a little.
I call the developer back, only to find that he has gone for a well earned lunch. I am promised he would contact me within the hour, as soon as he comes back.

Now, this is not the first time that this problem has reared its ugly head. Last time a script was run, but failed part way through, leaving the database in much the same sorry state it was in now. The only way to fix it, is to run the script again, and let it complete. We pass this on to the user, only to be met with protest that they do not want to run that as it may cause further issues. Fair enough! This is followed with a stereo rendition of, the data is missing, we can't do any work. We explain the situation with the developer, and they spend the next hour getting their filing up to date.

I sit by the phone for over an hour, until I finally give in and call them back.

"Oh, um, it seems he (the developer) has finished for the day." S#*%!

"Is there anyone else there what can help?" I ask. "I have an entire department who can't do any work." (Not to mention a user that could quite well go on a murderous rampage at any moment).

"Not really no, especially as we don't know what he has been doing to your system. However, if you run that script, it usually sorts the problems out. In the mean time we will try to get hold of him and get him in touch with you."

We call the user back once more, and suggest again that they run the script to clear the problem and put the data back. They once again refuse, with more than a little hostility. Obviously they want it fixed without us or them actually doing anything. The system has now been down for over an hour and a half.

Twenty minutes later the developer calls me back and confirms that Yes! The script must be run and all would be well. He explains in depth what the script does and I pass this on up the food chain.

The user is still not convinced, they even go as far as getting a database specialist involved to check it all out. It takes him a further twenty minutes, and then he too confirms what we have been saying for the last three hours. Run the script!

"They are going to run the script!" I am told finally. "To see if that clears the problems."

"Really? That's a good idea. Why didn't I think of that?"

The script takes half an hour to complete and by the time it is done, there is less than half an hour before home time. The data is back, the system is working again, and a twenty something workforce rushes to complete as much as possible before 5 o'clock, as the whip cracks behind them.

They lost over three and a half hours of work time. If they had listened to us, they would only have lost an hour.

Which begs the question: Why did they call us in the first place, if they were not going to follow our advice?

It's what we do!


No comments:

Post a Comment

Popular Posts