Sunday, September 20, 2009

Sunday Morning Sing-a-Long VII: Triphoppin'

As I mentioned in the Twittersphere last night, the local good radio station played the Nightmares on Wax song "70s 80s":



I can't get that guitar loop out of my head.

The week was busy. It was stressful, but in the end it was all for the good. Last week, I blogged about the Big Switchover in the code base at work. This week is the week that code made its way into production.

A release candidate was cut and put on the test environment on Tuesday. I felt bad for the systems engineer and user interface engineer who had to code review and test the changes because there was a lot of code going into this release. Between my ex-boss and I, there was something on the order of 1,600 new lines of code. To aid them in their efforts, I wrote up a detailed test plan of things for them to do to flex different parts of the code.

Late in the afternoon, there was a false alarm bug that was really system configuration issue. The new code relies on asynchronous execution of tasks, and the queue that was dispatching the requests was not up and running.

After I got home that evening, I found a bug in my own code for something I had not taken into account in the testing. I worked with the system engineer to get the fix merged into the release candidate.

The push to production on Wednesday morning had issues.

It turned out that code I had written for doing synchronous population of a data store had not been tested for multiple sites. I had a stray line of code that was breaking the execution of a loop, so we had to fix that.

Then once the code went live, we started getting alerts about fatal errors. A fatal error happens when a script cannot execute to completion. It was being caused by some validation code that I had written as part of this release. I was checking the value of a variable using the wrong name.

Within an hour, I had coded the fix, got the system engineer to test it, and then got it pushed into production. Lucky for us, the bug was exercised only when an "about us" type of page was requested. According to the logs, most of the hits were being generated by search engine crawlers rather than real humans.

On Thursday, another issue was detected. A database query had been written without adequate entity constraints, so there were situations where if you requested a post by its permalink URL, and there was another post in the database with an identical title, you would wind up with another person's content. Fortunately, it was a two-line fix, and the issue was resolved within a couple hours of detection.

I've also been busy with some external development stuff. We have another company who wishes to integrate with our application. Since we have ways for third parties to use our services programmatically, this is pretty easy to do.

In fact, we have one company who did just this thing this past summer, and their development team was really happy with the design. This other company isn't quite as web services savvy, so educating them on ways to get what they want done quickly has been a challenge.

To make their work much easier, I will need to make one change to our application that gives them much more flexibility. I took the lead on implementing that over the weekend. It won't be a breaking change, thankfully, but it will need to be vetted before we push it into production, so I will be using our group code review time to put it under peer scrutiny.

We also made plans this week for the development week the next couple of weeks. We will be adding a user interface for adding accounts that doesn't require a call to our support team. When I announced this at our company mini-meeting on Friday, I got cheers from the support team, which made me feel good.

To finish off the week, the President and CEO took my team and I to lunch at a brew pub downtown. It was one of my favorite restaurants, and luckily it has enough breadth to keep the diversity of my teams tastes happy. One is a vegetarian, another is a teetotaler, and another is on a high protein diet.

I think the meeting was good because we got the ear of upper management, and they got to know us better. We have a good reputation within our company because we consistently get stuff done, usually on time or pretty close. The CEO said that at the last startup he was at, software releases involved nail biting and stress. He said he doesn't worry about that stuff with us. I told him, "We worry, so you don't have to."

Despite having one less team member, I feel good about where we are headed. I gave a new job description to the President on Friday, and it's a major rewrite of what my ex-boss used. After 1 3/4 years on this job, I've got a good sense of what the good hires and not-so-great hires are like, and I've got a better sense of the soft criteria that discerns the two. The description takes that into account.

With the quarter coming to a close, I will be sitting down with my team members to set goals for their own personal development for the next quarter. While it is important for us to get the releases done on time, I know it's also important for our team to grow in our expertise and skill.

On the home front, STBX started one of her part-time jobs last week. She has to be at the workplace at 5 pm on Wednesdays, so I have to take off work early. That's not such a big deal because I go in early on Wednesdays because of releases. Since my older daughter has to be in bed earlier these days, that gives me more time to spend with the kids.

My older got her first progress report this week, and she got good marks. She's really enjoying school, and it's so neat to see her grow in her abilities. The younger daughter is in preschool three days of the week now. The other part-time job that STBX has is working as a substitute in the school cafeteria. Since that has the possibility of conflicting with getting the younger daughter to and from school, she has arranged for the preschool teacher's mom to be a sitter. The rate isn't too awful, and I'm picking up 75 % of the tab for that.

In a few hours, I'll be picking up the girls to take them up to the north burbs. My mom is house sitting for my brother, so she's invited us up there for a cookout. That's about all that's news here.

Monday, September 14, 2009

An xkcd Comic that Hits too Close to Home

Be sure to mouse over the image to get the tooltip that makes yet another analogy.

Brontosaurus

Saturday, September 12, 2009

Late Night Listening XXXII: Tetering on the Edge of the Known Edition

(picks self up off of pavement after being hit by what felt like the bus I ride to and from work)

Tonight we set the wayback machine for the mid 70s and find the nearest arena to rock out to Kansas' "Point of Know Return".



I always loved the cover art for the album on which this track appears, and it is of symbolic significance this week because on late Friday afternoon, I got the OK to push through a significant change way that our application works.

This project has been a long time coming. Both my ex-boss and I had long eyed the possibility of replacing a naive algorithm for content sorting with something more industrial grade. A business development opportunity which threatened to push the existing code past it limits created a the motivation to push forward with it.

Changing the algorithm wasn't without risks. The end result of this sorting algorithm is user-visible, and a major shift in the sort profiles threatened to hurt the search engine cred for some of our customers. To protect against damage, I ran extensive comparison tests for almost 20,000 different content aggregations, identifying which ones might be hurt most.

Fortunately over half of the aggregations remained unchanged. About 30 percent shrank, and the remainder increased. After some additional number crunching by other departments, and adding a bunch of subjective measures, they came back with a list of adjustments that they thought would be needed.

A couple of product support workers got the task of refining the sorting criteria, running into issues along the way. It seemed as if no matter what they did to loosen up the criteria, they still wound up with bad results. Fortunately our team was able to tune the sorting system to get better results.

There was a lot of tension along the way, with upper management at the point where they were wanting to delay the switchover to the new algorithm even later than what we had done already. In the end, I managed to persuade them that some of the shifts they were worried about would lead to better results in the long run because the sorting would mean something.

So all of that was enough to keep me busy this week, but there was more, and it put me through a wringer.

First, I had to deliver the "break up" message to the problem employee. Thankfully he took things well, and wasn't totally blindsided by the announcement. It probably helped that upper management had authorized a week's extra severance as part of the package. Still, the experience was emotionally draining, and afterward, I jokingly told the remaining workers, "Please don't make me do that again."

Still the newly ousted employee had some chutzpah. The next day, he e-mailed my boss to say that he had done some thinking about the agreement and said that he wanted one month's pay instead. Keep in mind, he had been with us only three months. My bosses did not budge, thankfully.

But there was more. On the afternoon of testing day, I found a couple of bugs in a major user interface feature for the week. They were show stoppers, and to ship with the bugs would have been insane. Since we were already doing the release on Thursday, instead of Wednesday, due to the short work week, to postpone the release would mean no release for the week at all since we don't do releases on Fridays as a rule.

So I worked late with the UI engineer to debug one of the issues. It was a tricky bug caused by a lingering subscription to a change event that fires when an asynchronous HTTP request is finished. It turned out that the developer of the user interface was using the web service code in way that it was never envisioned. We made changes that wound up making the problem just go away. By the time the UI engineer fixed the other bug, it was 2:30 am on Thursday, and both of us had to be in the office by 7:30 am for the push to production. It was a looong day.

On top of all of this, I was working with another manager to finish up the draft proposal for our outage emergency policy, attending management meetings where there was debate over the values and mission statement of the company, and preparing presentation materials for the company meeting that was supposed to be on Friday, but got moved to next week.

Which brings us back to the code switchover. This sorting change was a big effort. It was started by my ex boss, who did the foundation work and some prototyping. I had to refine it and handle a lot of edge case nastiness. None of these items, let along testing, could get done in a week's time. So we had to commit changes to a branch that lasted over a month.

At two points over the past month and a half, I was able to merge subsets of this code into trunk to handle new functionality, but the merge I did early this morning marked , for all practical purposes, a crossing of the Rubicon. Although there were some pieces in place to roll back after the switch, the reversion would be a risky operation that would only be employed in the event of an emergency. So in essence, we have now crossed that point of no return.

Indeed, when I committed the merged changes back into trunk, I included a lyrical reference to tonight's song. The sad thing is that I think not one person on my dev team was born when this song was big.

On the personal front, I didn't make it out for the testing today. My mom had to work, so the kids were all mine all the day. We went out for lunch and then went out for a couple hours later to go to the library and do some shopping.

I've been taking it easy this weekend. With my nerves as raw as they were on Friday afternoon, I need the downtime to recharge.

Monday, September 07, 2009

Late Night Listening XXXI: For the Love of Labor

Tonight's selection is an obscurity... Thompson Twins' "You Take Me Up".



This track is not my absolute favorite from this group, but since it touches on the theme of labor, I thought it appropos for a posting on this holiday.

Within the depth of this Aspergic mind of mine, there is a region that obsesses over the reinterpretation of songs in different styles, much like what Jack White did with the Racontuers' "Old Enough", by getting Ricky Skaggs and Ashley Monroe to sing a bluegrass version. I could see a rearrangement of "You Take Me Up" sung in a more traditional gospel/bluegrass style, and having it work pretty well.

On a superficial level, Jack White creeps me out. Yet, the Skaggs/Monroe recording and his work with Loretta Lynn (c.f. "Portland, Oregon") hints at a musical transcendence that bespeaks creative genius. I'll drink a slow gin fizz to that any time.

Getting back to Thompson Twins... my favorite song from their canon is "Lay Your Hands on Me", which has a refrain that has a religious revival feel to it. (looks away wistfully) I... miss touch... badly.

So where are things with me? Overall pretty good. House is a disaster area, mostly toy scatterage. I'm starting to catch up on laundry. Bills are paid, and the bank account balance has been slowly and steadily rising.

STBX has a couple of very part time jobs lined up that should allow her to keep our younger daughter in the preschool cooperative for her last year. First, she got on as a substitute cafeteria worker where our older daughter goes to school, and she's going to be working a couple of days a week for a frozen custard chain, doing detail on their cakes.

I didn't do too much over the weekend. Wrapped up some odds-and-ends tasks, and I'm code complete for the Tuesday end-of-business cutoff, save for some fill-in-the blank things I'm waiting to get from a coworker. That's a good thing because I have 4 1/2 hours of meetings scheduled on my calendar for Tuesday, which leaves very little for coding tasks.

I wasn't supposed to have the girls this weekend, but I got some Daddy time in. They went down to my dad and stepmom's on Friday, and I went down to pick them up yesterday afternoon. STBX had to go to training for her cake detailing job today, so I came over and got them. We took a bus trip to downtown, which is something they have been wanting to do ever since I started riding what they refer to as "the work bus". We met my mom for lunch at the big mall (those of you at the blogger gathering last October would know where this is). We also stopped at a chocolate shop for some ice cream. Here is a picture of them ogling the deserts...



... the moment that picture caught was one of those warm-and-fuzzy times that makes being a dad so much fun. Just watching them off by themselves... in a state of wonder... enjoying life.

Let's pay a visit to the office now...

Getting used to the management role has been fairly smooth. I am now at a point where I have an idea of where our department is and how the other departments are doing, and I've been able to keep up my coding commitments.

The feedback I have gotten from my boss, the President, is that I am doing really well. She also added that if I needed flexibility in my schedule for family stuff, she's give me leeway since she knew that I had put in a lot of extra hours through all of this. Their main concern is that they don't overwork me to the point that the "break 2am".

One sign of confidence from upper management... Back when I took on this role, I was promoted at a "Director" level, not the "Vice President" level that my boss held. They envisioned getting someone who was more into doing business development work for that role, and that was fine by me. I've been able to keep things going well enough that they've decided that they could postpone hiring for that VP role, with them reconsidering reopening that search on a month-to-month basis.

We've made good progress on the big project that we started at the beginning of August. My ex-boss chipped in some trailblazing work, and I was able to use that as a starting point, and refine it to handle all of the edge cases. What emerged from that is a really good system that should make our customers happier than the old system. However, our customer service team is nervous and needed more in-depth data over the impact of the change, so its release has been delayed by a week. I have been more than transparent, giving them solid and complete before/after data sets.

The one thorn in my side is the guy we hired on back in mid-June. He has been something of a slacker since day one... coming in late morning, cutting out some afternoons early, missing deadlines, writing shoddy code. Before he left, I know that my ex-boss had a talk with him about his performance.

And when I took on the director role, I had another talk with him. I had learned that he had gone to the President with his complaints... including our rapid development pace and the fact that our developers also had to do testing. He thought we needed to hire on a full-time QA person. I tried to address his complaints and explained how our system had some benefits for him as well.

When we started this new project, I gave him more leeway than usual to work on the design for a new user interface component. He spent way too much time on it, and again, deadlines crept out and out. This was starting to really annoy the other UI developer, too. We reached a point where I told him that we needed to move from the prototype to a full version. We mapped out a day-to-day schedule to ensure he understood that we needed to see this into production this past week.

Still, he dragged his feet. My boss was concerned about his progress, and she could tell he was starting be a real problem for me. We talked about what we could do, and agreed we would try to keep him on board until we wrapped up this project and then look seriously about getting rid of him.

On Tuesday afternoon last week, just before the code-complete deadline. We realized that there would be more time needed to get things done, so I agreed to have the first release candidate cut on Wednesday morning. I didn't see the slacker online when I signed on at 9 pm that night. The diligent UI guy was online, too, and was miffed that the slacker had not been available to help with some code integration work, so he started working on the slacker's code to fix the issues.

The slacker signed on after 11 pm, irritated that the other developer had worked on his code, and then he IMed me saying he wanted to speak with me the next day about his future with the company. I told him that I would be willing to speak with him, but not on testing day, when things are super hectic. I opted for a meeting after lunch on Thursday.

During the meeting, he rattled off what was bugging him about the job... his ideas weren't taken seriously, he didn't like our frequent release schedule, we put too much time into engineering things, he felt like he was being micromanaged. Everything was everybody else's fault.

His solution? Move him out of the department to be a "product manager", so that he could concentrate on less time sensitive things that required creativity, like prototyping. I asked him whom he thought he would report to. He said he thought that he should report directly to the President, my boss. He also said that if this move wasn't possible, he might have to start looking elsewhere.

The sense of entitlement was completely annoying. This was basically a shoot-from-the-hip whiz kid in his 20s who was being paid way more than what he had given us in output, and at no point had he ever owned up to mistakes he had made along the way, many of which resulted in either bug reports or emergency fixes.

I spoke with the President after that meeting, communicating what he had told me. We both agreed that he needed to go. Keeping him on while he was looking made no sense because he had never put in full effort at this job. The checkout would have made him complete deadweight. Since he had already scheduled to be out of the office on Friday, I arranged for the axe to fall on Tuesday (tomorrow), at a meeting with the Presdient, him, and me late in the afternoon.

So with two months of experience in this role, I've managed one of the largest development projects the company has had, and I've had to fire someone along the way.

On other matters...

The system engineer who overslept a few weeks back took a much deserved vacation the last week of August, half of it spent at Hilton Head, and the other half at Sin City. I went out of my way to make sure that I did not interrupt his vacation for anything, and fortunately we did not encounter any issues that would have required his help.

Also, Anais has recommended that I get involved with the local Mensa group to build some new friendships. It turns out they have admissions testing this coming Saturday. If I can get my mom to watch the girls for a few hours then, I will head out to do that. If I can't make it then, there is a larger testing event in town in mid-October.