19 Gigs, and What do You Get?

Last week started off in a not-so-fun state.

Earlier this month, I wrote about a consulting gig that our company had taken on.

The day after that post was written, we met with a couple of departments at a large medical device manufacturer with operations here in town. One department had approached us about auditing the security of web based applications and the server that hosts them.

The other department's interest was driven by a network effect. The manager of the first department was friends with the other manager and had told her about the project and decided that might be a good thing for them to do with their own applications.

In the two weeks since the initial meetings, not much had happened. There was a teleconference with the first department, where we got to meet the technical contact for their web hosting service. There was a teleconference with the second department to talk about what information we needed to do the audit.

My action item was to create a virtual machine, a software emulation environment that mirrored the setup of the staging server so that the contractors could conduct their security tests against it rather than the actual server. To accomplish this task, I had copies of the operating system installation disks and a huge three-ring binder manual that documented what was done during the install two years ago.

After a day or so of work I got the installation mostly done. Part of the setup involved installing a few gigabytes of patches on the simulated machine. Some of those patches had dependency issues that could not be easily resolved, so I held off installing them rather than forcing the install by bypassing the dependencies. Almost all of the problematic packages were peripheral utilities. The only exception was the webserver program itself.

Another issue was the collection of files that comprised the application. This was a huge collection of data, consisting of scripts and media, totalling 17 GB of disk space. On the company's weakly DSL connection or my own weakly DSL connection, it would have taken over two and a half full days. There was talk of us doing an on-site checkout of the files from a revision control server that is owned by the client site.

Finally, there were several system configuration files that would need to be edited to reflect the setup on the server. The client had not supplied these files or the contents thereof to us.

I documented my progress and listed the problematic packages in a memo to the CTO. A week passed by with no acknowledgment or further guidance. Last week early Monday morning (like right-after-midnight-early), the CTO dropped me an e-mail instructing me to send him a set of DVDs for the virtual machine so that he could set it up on his laptop. Later on in the morning, I wrote back advising him that the install was not complete as per the issues described in the memo.

During the early afternoon, while I was on a phone interview, he called my coworker to have him relay new instructions. I was to go down to the colocation facility where our external servers reside. Among those servers was the sandbox server that belongs to our client. Although this server is supposed to be close in configuration to the staging and production servers, it is not rigorously audited. I was to do a network file copy operation from that server to the virtual machine. The thinking was that a direct connection would proceed more quickly than one over the internet. Once that was done, I was to finish off the configuration of the virtual machine by copying over a select set of files and then upload the virtual machine data to one of the servers so that my boss could download it to his home, about 200 miles away.

I told my boss that I had pre-existing commitments to watch the kids after work from 5 - 9 pm, but that I would take care of this once I was done with watching them. So I headed out to the "colo" after 9 p.m. with my laptop, some Cat 5 cable, and the installation binder. I started the file copy operation. Based on the transfer rate, it looked like it would take a little under two hours.

After the two hours had passed, I realized two very disturbing things. First, the disk space consumed by the transferred data was 19 GB, 2 GB more than the original file system. Second, it seemed to be copying the same set of files over and over. A little research determined that the copy operation was following a symbolic link in an infinite loop.

I called my boss and explained the situation, and then he realized that my diagnosis was probably right. He then told me to create a TAR (UNIX Tape Archive) file of the 17 GB file collection and then download it to the virtual machine. I said that wouldn't work because the virtual disk partition that where the TAR file could reside was 24 GB. There wasn't enough room to hold a TAR file and its extracted result. He told me to go ahead and download the file to the Windows XP host operating system and then figure out how to make the file visible to the virtual machine in the morning.

I created the TAR file, which took over an hour, and then started the file transfer on Windows XP. Something wasn't right because the transfer rate was 1/4 of what it had been with the virtual machine. I tried another file transfer program but transfer came to a dead halt at 4 MB for reasons unknown. I reverted to the original program and set up my laptop to sit in the server rack. I went home and got in at 2 am. I wasn't happy because I was scheduled to have an interview with Bonded and Insured at 9:30 am. I figured I could still make the interview by getting up early and stopping by the colo on the way to work.

When I got up on Tuesday morning, I went to check my e-mail using a web interface to our mail server, which is hosted at the colo. I couldn't get to the server. I was about to have a heart attack. I checked our external website. It was down, too. I knew that I needed to get up to the colo ASAP because something was really wrong.

I made it to the colo around 8:15 am. My laptop was running, but a low power warning dialog box was visible, indicating that the laptop wasn't getting any current and hadn't received any for about an hour. The firewall through which all the external servers run was down, too.

I called the CTO and left a voice mail to let him know about the situation as I frantically tried to diagnose the problem. Both the firewall and the laptop were plugged into a uninterruptible power supply (UPS), which then plugged into the power strip of a neighboring server rack. Yes, my employers were so cheap that they stole power from other server racks. I managed to find a power strip opening for the firewall and my laptop to get them back up and running. The UPS appeared to be non-responsive, even with a change of outlet.

I looked at the clock. It was 8:45 am. I figured that I still had a ghost of a chance at making it to the interview up in the north suburbs, so I headed out. I got as far as the near north side of town, when I got a call from the CTO. He said that the external web server machine was down, so I needed to get a power strip as a stopgap measure. I clenched my teeth and said I would take care of it.

I doubled back, stopping to call my HR contact at Bonded and Insured. I said that I would need to reschedule the interview because I had to take care of a mission critical server outage. I apologized and expressed that I was still very interested in going through the interview. She said she would take care of it.

Then I stopped at an inner city drug store to buy the cheapest power strip and extension cord I could find. I then headed back to the colo and got to work. I transferred everything from the UPS to the power strip and got them back online. Then I called my coworker to verify that it was working. It was, so now I could get back to the virtual machine stuff.

Fortunately, the 17 GB archive had been transferred successfully. Now I needed to figure out how to make the host system's file system visible to the virtual machine. After fumbling my way through the product documentation, I figured out how to get that working. The next step was to do the extraction, which took well over an hour.

After the extraction was complete, I had to do some additional software installs. One package appeared to already be at the required version, so no update was needed. The second failed because of a missing user interface library, but I don't think it was lethal because the utility was used only for viewing PDFs, and that probably wasn't needed for the website.

A third tool was supposed to be built from source code, but the build failed because the header files for a library upon which it depended were not installed. So I had to go track down a matching version of the library from a download server and get that built and installed. A couple more hours were lost fooling around with that.

Once every thing was set up, I started creating a ZIP archive of the virtual machine files. I did this using Windows XP's built in Send To -> Compressed (zipped) Folder menu command. Because I started the operation by right clicking a folder a couple levels above the collection of files, the compression code did not do a sanity check to see whether the collection was too large. It proceeded to create ZIP file approximately 5 GB large over a period of an hour or so. No error or warning messages were displayed.

I transferred the 5 GB file up to the download server and sent out a note to the CTO that he could now snag the file from his home. I went out to the car and did my phone interview with ShovelSite, which used up close to an hour. Then I went home, anticipating that I would be able to relax for the evening.

I got an e-mail around 7:30 pm from the CTO saying that he had downloaded the file but that it appeared to have problems because he couldn't extract it with any utility. After some research, I turned up Microsoft Knowledge Base Article ID 301325. The gist of the article was, "Yes, there are ways to create a ZIP file larger than 4 GB. No, the archives won't be valid. Yes, there is a workaround: Don't do that."

So I wound up having to compress each virtual machine file individually. Since the largest of these files was 2 GB, they could be zipped safely. The total byte count of the compressed files was over 19 GB, and it took over two hours to create them. With the new archives in hand, I drove up once again to the colo to do another file drop. This time I turned off the virtual network adapters created by the virtual machine and got double the file transfer rate that I did the night before. I was done shortly after midnight and in bed by 1 am Wednesday morning.

It suffices to say that I was dragging on Wednesday, but I went into work anyway. The CTO called me on his way to his in-laws in the Keystone State. He said that he had gotten the files downloaded successfully and that he had gotten them to run successfully on his laptop. The CEO took my coworker and I out to lunch, and in recognition for my efforts, I got to pick the restaurant. I selected a nearby Italian eatery that has a damned fine chicken Parmesan sandwich.