Late Night Listening XXXII: Tetering on the Edge of the Known Edition

(picks self up off of pavement after being hit by what felt like the bus I ride to and from work)

Tonight we set the wayback machine for the mid 70s and find the nearest arena to rock out to Kansas' "Point of Know Return".



I always loved the cover art for the album on which this track appears, and it is of symbolic significance this week because on late Friday afternoon, I got the OK to push through a significant change way that our application works.

This project has been a long time coming. Both my ex-boss and I had long eyed the possibility of replacing a naive algorithm for content sorting with something more industrial grade. A business development opportunity which threatened to push the existing code past it limits created a the motivation to push forward with it.

Changing the algorithm wasn't without risks. The end result of this sorting algorithm is user-visible, and a major shift in the sort profiles threatened to hurt the search engine cred for some of our customers. To protect against damage, I ran extensive comparison tests for almost 20,000 different content aggregations, identifying which ones might be hurt most.

Fortunately over half of the aggregations remained unchanged. About 30 percent shrank, and the remainder increased. After some additional number crunching by other departments, and adding a bunch of subjective measures, they came back with a list of adjustments that they thought would be needed.

A couple of product support workers got the task of refining the sorting criteria, running into issues along the way. It seemed as if no matter what they did to loosen up the criteria, they still wound up with bad results. Fortunately our team was able to tune the sorting system to get better results.

There was a lot of tension along the way, with upper management at the point where they were wanting to delay the switchover to the new algorithm even later than what we had done already. In the end, I managed to persuade them that some of the shifts they were worried about would lead to better results in the long run because the sorting would mean something.

So all of that was enough to keep me busy this week, but there was more, and it put me through a wringer.

First, I had to deliver the "break up" message to the problem employee. Thankfully he took things well, and wasn't totally blindsided by the announcement. It probably helped that upper management had authorized a week's extra severance as part of the package. Still, the experience was emotionally draining, and afterward, I jokingly told the remaining workers, "Please don't make me do that again."

Still the newly ousted employee had some chutzpah. The next day, he e-mailed my boss to say that he had done some thinking about the agreement and said that he wanted one month's pay instead. Keep in mind, he had been with us only three months. My bosses did not budge, thankfully.

But there was more. On the afternoon of testing day, I found a couple of bugs in a major user interface feature for the week. They were show stoppers, and to ship with the bugs would have been insane. Since we were already doing the release on Thursday, instead of Wednesday, due to the short work week, to postpone the release would mean no release for the week at all since we don't do releases on Fridays as a rule.

So I worked late with the UI engineer to debug one of the issues. It was a tricky bug caused by a lingering subscription to a change event that fires when an asynchronous HTTP request is finished. It turned out that the developer of the user interface was using the web service code in way that it was never envisioned. We made changes that wound up making the problem just go away. By the time the UI engineer fixed the other bug, it was 2:30 am on Thursday, and both of us had to be in the office by 7:30 am for the push to production. It was a looong day.

On top of all of this, I was working with another manager to finish up the draft proposal for our outage emergency policy, attending management meetings where there was debate over the values and mission statement of the company, and preparing presentation materials for the company meeting that was supposed to be on Friday, but got moved to next week.

Which brings us back to the code switchover. This sorting change was a big effort. It was started by my ex boss, who did the foundation work and some prototyping. I had to refine it and handle a lot of edge case nastiness. None of these items, let along testing, could get done in a week's time. So we had to commit changes to a branch that lasted over a month.

At two points over the past month and a half, I was able to merge subsets of this code into trunk to handle new functionality, but the merge I did early this morning marked , for all practical purposes, a crossing of the Rubicon. Although there were some pieces in place to roll back after the switch, the reversion would be a risky operation that would only be employed in the event of an emergency. So in essence, we have now crossed that point of no return.

Indeed, when I committed the merged changes back into trunk, I included a lyrical reference to tonight's song. The sad thing is that I think not one person on my dev team was born when this song was big.

On the personal front, I didn't make it out for the testing today. My mom had to work, so the kids were all mine all the day. We went out for lunch and then went out for a couple hours later to go to the library and do some shopping.

I've been taking it easy this weekend. With my nerves as raw as they were on Friday afternoon, I need the downtime to recharge.