In-Depth

Exchanging Pain for Pleasure

Moving from Exchange 5.5 to Exchange 2000 is a complex task. This migration took many unexpected twists, turns and hours.

• By Alan Caruth
• 08/01/2002

It can be a difficult and unpredictable maneuver. The pressure was even greater in this case, as it was my company’s first non-test, real-life customer migration.

The Background
The customer’s network: Windows 2000 mixed mode, 26 workstations, four servers, one Exchange 5.5 server running on Win2K Server, 10/100 switched Ethernet, several network printers and various other devices spread around the office.

The client who wanted the upgrade has been a customer for several years. It has set office hours and usually only a few employees who work on the weekend; this makes for easy scheduled network downtime when needed.

The plan was to do an in-place upgrade of the Exchange server during a weekend. We conducted test runs outside the live environment and gathered documentation and caveats from the Exchange documentation and the Knowledge Base. We created backups and tested Thursday night, as well as immediately before we started the upgrade process Friday night.

We were ready. We were also planning on going home before midnight Friday. That isn’t what the Exchange server had in mind.

The Upgrade Begins
We met to start the process at 5:30 p.m. Friday. With the Exchange server offline, we performed the necessary tape backups. Verification showed the backup was good. As an extra precautionary step, we made backups of the Exchange database by stopping the services and performing a manual copy of all relevant files. This can save some time if you have to do a restore, as there’s no need to wait for tape.

As a first step, we disabled all incoming mail to keep from accidentally black-holing any e-mail. A quick firewall rule-set change stopped the incoming flow of e-mail.

Following the Microsoft upgrade procedures, we prepped AD for the Exchange upgrade. Modification of the schema is always a scary thing; but, in this case, we were extra paranoid since this was a production network. The prep went well, though. AD appeared to be healthy and we received no errors. As of 7:30 p.m., we were still on track. The upgrade on the Exchange 2000 Server then began.

The No-Fix Fix
After starting the upgrade process, we figured we had 90 minutes of sitting around watching progress bars slowly creep across the monitor. About midway through the upgrade process, an error requestor appeared on the screen. It told us there was an error updating the Internet Mail Service; we could retry or cancel. A Knowledge Base search revealed that Microsoft knew about this error. We thought we’d been saved.

But no—the fix we learned is that there is no fix. You must manually uninstall Exchange 2000 and the AD integration. The fix referenced by the Microsoft article involved removing registry entries, deleting files manually, restoring Exchange 5.5 from backup and attempting the upgrade again. So we started manually removing Exchange 2000 and restoring Exchange 5.5. Approaching 10 p.m., we were becoming thankful the customer had free soda for vendors.

After the removal process was complete, we reinstalled Exchange 5.5 and the service pack and restored the database to bring up the server. Following that, the store service wouldn’t start. We checked the Knowledge Base, copied files to and fro, adjusted settings, checked registry entries, copied files back and forth (again), reinstalled a couple of other times for good measure and restored files from backup—but the store was still dead. At 2 a.m. Saturday, after working the problem for a couple of hours, we made a decision not to troubleshoot the server anymore; but we didn’t want to destroy the current server before Exchange was fully functional. We wanted to get another Exchange Server online before formatting the old one for a clean install.

Moving to Plan C
In the corner of the room was a dual Pentium 120 Compaq Proliant Server with a RAID 5 array of four 2.5GB hard drives. It was this customer’s primary server for years but had been moved to a secondary test-server role. The ProLiant was running NT 4.0, the new server Win2K; we decided to try restoring the backup to the old server under NT 4.0/Exchange 5.5 in order to save time and not destroy the poor machine’s performance. The risk here was that Win2K would treat Exchange 5.5 differently enough that we would just end up wasting hours trying to restore the data to a server with NT 4.0 installed.

After installing NT 4.0, we installed Exchange and SPed it and restored the database. Deciding the backup tapes were probably the best option, we pulled them out and started the restoration. What we didn’t know was that the online backup from the previous evening went fine, but the backup performed immediately before the upgrade didn’t successfully pull the database. Hidden in the log file was a message that the Exchange offline public and private store backup failed, even though the backup reported it a success. The file-copy backup we did wasn’t as clean as an online restore, but it allowed us to regain the most current data. After starting the file-copy, we watched the status bar creep across the screen for a half-hour, holding our breath.

After the restoration was complete we did some minor testing. There were some oddities in the way Exchange acted, which was due to an Service Pack issue. We looked over our notes and discovered we were running the wrong SP for Exchange 5.5. So, we installed the proper SP, restarted the services and verified server operation. The Exchange database was resurrected from the dead and, like us, was breathing again. It was 4 a.m. when we went home.

Six hours later, we started planning the next action. Now that the old server was up and running, and Exchange and all had tested well, we could focus on getting the Win2K box operational.

Migration Good; Upgrade Bad
Reformatting the primary Exchange server, we installed Win2K Server and Exchange 2000. Once complete, we made a decision to migrate the data rather than attempt another upgrade. Now that a second server was operational and fully functional, migration was an easy solution.

Exchange 2000 came up cleanly on the server and appeared to be functional. Testing verified that it was running well, so we started the user migration. The migration went perfectly. The second step was to move the public folders. That went less perfectly. The machines reported that the migration was successful, but the folders never appeared on the destination machine. A run through the Knowledge Base and elsewhere on the Web revealed nothing, leading to more troubleshooting. Eventually, we used an older Exchange method of migrating data when all else fails: We exported all the public folders to a .pst file and imported them from a client machine. The permissions were lost, but the data was saved. That necessitated creating a list of necessary permissions and rebuilding them on the imported data.

We finished the configuration by configuring SMTP, modifying the firewall rules to allow mail to flow again and restarting the backup system. We were feeling good. After a bit of testing, we were ready to go home for the rest of the weekend.

The Aftermath
Over the next couple of weeks, some small issues came up:

• Clients were receiving free/busy data update errors on their workstations. This turned out to be an Exchange permissions issue.
• We noticed problems with Exchange addressing e-mail and relaying mail properly when used with our relay/smart host server. This was easily resolved through small configuration changes on the Exchange server.
• We had problems with adding new users and a problem adding and modifying distribution lists. This was documented and repaired when we made the switch to Native mode.
• Some of our Distribution Lists had newly illegal “*” characters in the names. After the conversion to Win2K, we had to adjust the naming convention on the distribution lists so we could keep a standard with the new limitations in addressing.
• Personal Calendar permissions were broken because of some of our lists having the character “*” in the name. For a temporary fix, we added the users where needed, but after the switch to Native mode, we properly fixed the problem by creating all new distribution lists.
• The transaction logging function of the database started filling the hard drives quickly. We developed a plan to create proper backups of log files and maintain the size of the directory over a longer term. If you aren’t familiar with the logging methods in Exchange 2000, I’d suggest you do some reading on it before you upgrade.

Most of our minor complaints and issues were resolved with the switch to Native mode. The other problems we had were repaired by the minor adjustments above.

Learning Curve
Some of the lessons I learned from my experience:

• When planning to install Exchange 2000, migrate, don’t upgrade. This may not apply in all circumstances; but if you have the opportunity and hardware to migrate, use that method. It will help you avoid most of the adventures we experienced in our first install.
• Always verify that you have at least two good backups and can restore the data successfully. It’s much better to spend an extra hour making spare backups than spend dozens of hours explaining the lost data to the boss or client.
• Create a backout plan. Occasionally bad things happen. When they do, be prepared. Test your backout method to verify it’s fully functional and will recover all data.
• Check your permissions and do full end-user testing. We only discovered some errors after a couple days of production on the system.
• Look in the Event Logs to find out, before the upgrade or migration, if you’re receiving any errors. A failed or misconfigured installation can create errors, which might not appear at first to be related to Exchange.
• Do offline maintenance on your Exchange database before making the switch to Win2K. This can help compact the database and clean it up before you make the move, depending on how you do it. This also saves tape restoration time in case you end up having to back out of your installation.
• Read the directions for the upgrade or migration. Read every ounce of them available from Microsoft and other resources. Follow the procedures exactly as stated. Look through the Knowledge Base for articles so you have an idea of what types of installation caveats and failures might occur. Familiarize yourself with the install process inside and out. Have the backout and uninstall procedures handy.
• Plan the first migration or upgrade with a long maintenance window if available. If not, have a machine immediately ready to take the place of the machine you’re upgrading or migrating. If we’d scheduled our first upgrade for a weeknight, we would have spilled into regular business hours and cost the company money.

Remember Where the Exits Are
This was our first production Exchange 2000 migration. We’d prepared to a great extent before this upgrade and tried to take into account every possibility. Even then, an amazing number of unpredictable issues came up. The Knowledge Base, Exchange FAQs and discussion boards are great resources, but they can’t help in every situation. Be prepared to back out and do the upgrade another day, if needed. Migrate instead of upgrade and you will save yourself many headaches—and may even free up a weekend.