In-Depth

The Project from Hell

Three continents, incompetent management and an impossible deadline is a case study in how to NOT do a major upgrade.

• By Greg Neilson
• 03/01/2004

The first I heard of this project was during a conversation with my practice leader in the consulting firm in which I worked. “Greg, we’re working on a cool proposal right now to upgrade to the latest version, and if we get it, we’ll need you to go back to Australia during part of the implementation.”

This all sounded fantastic—I was going to be utilized (the golden rule of consulting!), do my first upgrade to this new version of Domino, and better yet, get a trip back to Australia paid for by someone else.

The reality of the project hit us when our assembled staff started work on the client site. The client had business imperatives that forced a delivery date that was straight-out unattainable. More respectable consulting firms hadn’t wanted this work because they knew it would end in tears. It appears that our firm was so desperate for the business, it had agreed to deliver something it knew we couldn’t do. But the expectation was that once we got started, it would be too late to remove us; and besides, eventually things would probably turn out OK…

As we progressed through the project, it became clear that to get anywhere near the desired schedule, the team was going to have to work around the clock. However, the project was being run (“project managed” is too strong a term) by a practice leader—I’ll call him Bob—who made no attempt to engage us to committing to the project schedule. Thus, it became his schedule and not our schedule. This increased our frustration and anger, turning lunchtime breaks into mutiny sessions where we we’d voice our discontent.

With our assembled team of five, we were certainly capable of handling the task at hand, but the unrealistic schedule meant there was little commitment within the team to making the date. The other guys were contemplating resigning there and then (this was a few years ago, when jobs were plentiful), although the prospect of the free trip to Australia trip kept me from similar considerations. Bob clearly was no leader—that would imply that people would willingly follow him—but he was great at selling to clients. There was a real death march feeling within the team that we were headed for doom, and nothing we could do would change that.

After a time planning the upgrade and collecting information about the client’s environment, our implementation proceeded in earnest. We built the first servers, migrated some of the data and were ready to proceed. Then the fun really started.

The team split up to travel to the main client sites to perform the majority of the upgrade—Phoenix, Arizona; Dublin, Ireland; and Sydney, Australia (where I was). We were migrating 1,000 users to the new environment built on Windows NT 4.0 and Lotus Domino 5.01. At this time, our new manager—I’ll call him Abe—joined the project. In terms of the management hierarchy, he came somewhere between us and Bob, our practice lead. We hadn’t worked alongside him before, but we knew he had an extensive technical consulting background. But unfortunately, we’d soon learn Abe wasn’t much of a people manager, either.

The next sign of trouble was when I called after noticing some configuration changes in the environment with which I was unfamiliar. When I asked Abe about this, he told me he’d made some design changes. When I asked him if he could send me some details about what had changed, he said if there was anything I wanted to know I could call him and ask!

The 18-hour time difference between Sydney and Phoenix makes that suggestion impractical, and the arrogance of coming in at the last minute and changing everything was astounding. Abe did have a strong technical background, and it’s likely that these were sound ideas; at the same time, this lack of confidence in his own staff wasn’t conducive to an ongoing productive relationship.

As we proceeded further, we were engulfed by the internal politics of the client. The majority of the technical research and development folks were in Dublin, with the marketing staff based in Phoenix. Our project sponsor for the company that hired us was also in Phoenix and had made project decisions without knowing or caring how this would affect users. Instead, he elected to hide behind the hired consultants (in other words, our team) and have us deliver bad news. We weren’t able to act on the complaints from those whose ability to work was affected by the changes; we just had to do the dirty work and listen to the resulting profanity. Any further changes were going to have to be debated internally in the company. But given that these were usually directives from “on high,” nothing was going to be changed without a fight.

For example, the technical staff in Ireland would have much less system access in the new environment. This sounds great from a CIO viewpoint, but some of this access was needed for them to do their jobs. (On the other hand, many of them currently had more access than they really needed, and some of their complaints were more ego- than need-driven.)

Having our project team communicate these decisions onsite to the Dublin staff wasn’t much fun. The familiar refrain of “(Bleeping) Americans” was all-too-frequently uttered by the Irish staffers. They had little respect for decisions made at the Phoenix headquarters, and we were stuck in the middle, attempting to implement the already-agreed-to project.

As we went through our migration, the time zone issue really started to bite us. The different time zones of the three global locations made it impossible to have a single time during the day for a status review meeting. With team members working long days to perform the migration, having one of us wake at 2 a.m. for a teleconference was never a practical idea. The best we could hope for were updates between two of the locations at the start and end of each day, and hoping the information didn’t get mangled as it was passed around, like a game of “telephone.” There’s nothing worse than trying to be part of a team implementation when you don’t know what everyone else is doing. With no regular updates, we had no idea of what progress had been made in the other locations, what decisions had been made, what problems had been encountered or how they could be fixed. This was particularly important because this was a new version of the product, and for all of us it represented our first production deployment of it.

Ultimately, we kind of met the committed live upgrade date. But it was another two weeks until they had a completely stable environment, and for some staff, this interim period had to be frustrating, as their productivity would have been severely affected. E-mail mostly worked from the beginning; it was the applications, most of which were developed internally, that took the extra time to be completely migrated. Of course, e-mail’s important, but for many of the company’s employees, these other applications were critical to their jobs. From our consulting firm’s point of view, more hours meant more dollars for us, since this was a time-and-materials contract.

Within three months of project completion, most of us had left the company. Not long after that, the consulting firm itself had gone into bankruptcy and folded. But I did get my trip to Australia, I guess. Alas, my poor workmates that were originally excited about their trip to Ireland saw little more than the Dublin airport, their hotel room and the client site.

It was formative experiences like this that convinced me to take a management role myself. I’m not saying that I haven’t made—or won’t make—mistakes, but I’d like to think that these will be nowhere near the magnitude of those I experienced on that job. Seeing things done so poorly was source of motivation for me to demonstrate that perhaps I could do better.