In-Depth

Top Guns Under Fire

They were a smooth-running, well-oiled IT machine—then a sticky, application-specific problem tested their mettle.

• By Bob Wheeler, Jr.
• 03/01/2003

YOU KNOW HOW you crave that rare and unusual feeling where everything seems to be going OK? Like when your office network was purring and your confidence level in your remote networks was in the 99 percent range? I was recently blessed with that rare feeling here in our home office. I savored it and inhaled deeply, sucking in all of the conceit I could muster. The recent migration in our engineering firm from NetWare to Windows 2000 was now complete, stable and deemed successful by the senior staff. My new firewalls and routers were working great in all offices, and I had a competent IT staff. There was no doubt—we, the IT team, were Top Guns.

An Easy Fix
The feeling lasted until a network engineer knocked on my door and informed me that one of our remote offices was having serious problems accessing time sheets on our Web server in the home office. The problem seemed to be excessive disconnects and time-outs. There’s no WAN or VPN connection for the remote office running in a separate domain but simply a 768kbps DSL Internet connection. The Win2K clients in the remote office were attempting to access a Windows NT IIS Web application via Internet Explorer. A quick check of the Web server indicated no problems with other remote offices. Sounded like an easy fix for the Top Guns.

The LAN operator in the remote office—normally a full-time engineer or designer who volunteers to monitor the local network—reported that the problem was steadily getting worse and that there had been no recent configuration or hardware changes. Outlook was starting to lock up the client computers and Web pages were taking forever to open. The HR director informed me that it was a payroll day and they needed timesheets from the remote office immediately.

I stalled and pressured my Top Guns for a quick solution. The remote office was four hours away, so there was no chance of dispatching my network engineer for a quick fix. We immediately narrowed the problem to a faulty Internet connection, router or hardware firewall. More time passed before the division director for the remote office called; he was alarmed that he couldn’t use FTP to upload computer-aided drafting drawings for a client. And meanwhile, the HR director was still waiting on an answer. The LAN operator reminded me that she didn’t have a lot of answers for users who had a lot of questions. Naturally, this caused our Top Gun status slip; we were now Lower Guns.

From Top Guns to Cap Pistols
With no WAN or VPN connection, I had no way to access the remote office except via the Internet, which was broken on the remote side. My network engineer was able to squeeze out enough bandwidth to access the remote firewall and routers. It was painfully slow but still better than our last option—a 56kbps dial-up. Strangely, the router and firewall looked good, although pinging got slow responses.

More time passed. The Lower Guns reviewed the situation. The phone was ringing off the hook. We eventually isolated the problem to the Internet connection itself. I recommended to HR and remote division directors that the users fax their time sheets to the home office. That wasn’t pretty. I retreated to the safety of my office. And we were demoted again—now we were simply Guns.

This was a time we could really have used the packet analyzer we discussed buying. Instead, we opted for more Microsoft licenses. My network engineer contacted the DSL Internet provider for the remote office. The provider claimed everything was peachy, thank you very much. My network engineer debated with him. The tech support request was escalated several levels until we were connected with an engineer who’d actually heard of “packets.” We pleaded with the engineer to analyze the Internet traffic for the remote office. “No problem,” he said. “It’ll take a few days.” Ah, the joy of DSL connectivity. It’s cheap—and so was the support that went with it.

The Big Cheese Arrives
Next, the CEO showed up at my door, wanting to know what was going on. I smiled and reassured him that everything would be fine. He winked at me, knowing I needed his support. I was grateful for his confidence. The growling in my stomach reminded me that I missed lunch—again. It was time to put this to rest. No way were the Guns going down shooting blanks.

I reminded my staff of the infamous “failure is not an option” remark from the movie Apollo 13. They suggested a solution involving duct tape, garbage bags and a modem. Not funny. My network engineer revisited his dialogue with the DSL provider and negotiated a quick-and-dirty traffic analysis. While waiting forever for them to call back, the phone continued ringing off the hook from employees. The waiting list for my network engineer was growing, and my LAN and CAD administrators were trying to fill in the gaps.

The Big Break
Finally, the provider called and informed us that there was an immense amount of broadcast traffic flowing through the DSL connection. My network engineer and I exchanged deer-in-the-headlights looks. The remote office hosted no Web, FTP or teleconferencing services, and the firewall reported no hacking of any kind. After receiving the information, we thanked the provider and scratched our heads. We also remembered that the LAN operator stated that there were no hardware or configuration changes.

The remote division director was increasingly anxious to know when the Internet connection would be functional. Again, I stalled. The network engineer and I started sifting through the firewall log reports obtained from the remote office. We were looking for broadcast traffic and noticed that some unconventional ports were open and active. We traced the port usage back to a user.

Bingo!
We then developed an unorthodox plan. We accessed a specific workstation in the remote office via Symantec pcAnywhere. From that workstation, we accessed the suspected user’s local hard drive across the local network to see what the user had loaded. Bingo! After clandestinely stomping around on the user’s hard drive, we discovered a Webcast application.

Next, we accessed another user’s drive. The Webcast software was there as well. Could it be? We contacted the LAN operator, who claimed no knowledge of the Webcast application. We instructed the LAN operator to immediately uninstall the application from all client computers. Meanwhile, in the home office, we monitored Internet bandwidth for the remote office. Victory! The bandwidth slowly started increasing. After about two hours, the bandwidth was back at 100 percent.

Oh, What a Tangled Webcast…
The Webcast application had essentially choked off all the available Internet bandwidth.

I queried the remote users regarding the Webcast application. They said the application provided them with real-time news, sports and stock statistics. They loved it and were disappointed to see it go, although it had become painfully slow. One of the users had obtained the application and passed it around discreetly, and none of the remote users realized the application could cause a problem. The LAN operator was embarrassed but not at fault. We reconfigured our firewalls to block this traffic.

Lesson learned: Know what applications your users are running, and don’t allow unauthorized programs on their machines.

This story had a happy ending, but reminded me that “monitoring” is a relative term. Supporting remote offices can be both a logistical and technical challenge, even with the best of teams. It took several days to finally correct all aspects of the problem. In a perfect world I’d have IT technicians, administrators and engineers at all my sites, along with the fastest and most secure connectivity. Then I wake up.

Small but Powerful
For now I’ll settle for my IT staff and LAN operators, who truly are Top Guns. They may be a small group, but they can hold their own and are humbled by the confidence and support of our employees. You don’t have to work for a huge company to be viewed as a professional or to find technology challenges. It’s not the amount of money that determines success, but the quality of people working on a problem.