Joe the Plumber
and
Control Theory

In-house.
Outsource.
Off-shore.

What can we say about their strengths and weaknesses?

Let's first consider physical goods and services.

• Who calls Joe the Plumber to unplug their toilet?
• Who outsources their major plumbing?
• Who off-shores their plumbing?
• Where was your cellphone or laptop manufactured?

Most people outsource their major plumbing, but nobody off-shores it.

Why not?

Let's set that question aside for a moment and look at...

---

System Control Theory

Who remembers system control theory? Bode plots? Frequency and phase? Poles and zeros?

Rather than frequency and phase, I will use bandwidth and latency - they are equivalent but more descriptive in our context.

Here is the premise: In-house, outsourcing, and off-shoring development can be modeled in systems engineering terms of bandwidth and latency.

What does control theory say about bandwidth and latency?

• If you have insufficient bandwidth for the system you are controlling, you cannot close the loop.
• If you have too much latency for the system you are controlling, you have instability (oscillations).

Either way, the result is that you build the wrong thing.

---

Bandwidth = Communications

So... what is bandwidth in our system? It is human-to-human communications.

While we do not have methods to quantify this bandwidth, it is pretty easy to rank qualitatively:

1. Face-to-face communications. Excellent bandwidth.
2. Telephone communications. You lose body language, but still have decent bandwidth.
3. IRC communications — still interactive, but you lose the body language and vocal nuances.
4. Email communications — not interactive, no nuances. It works, but when a misunderstanding occurs, it can cause major problems.
5. Crossing a language barrier is difficult. Doing it over a speakerphone is very difficult.

---

Latency = Time of Flight

The other critical system control piece is latency. Latency is the time it takes for information to travel. It is the "s" in Laplace.

1. The number of time zones increase the average latency between a question and the answer.
2. Heavyweight processes, contractual approval cycles increase latency.
3. The serial nature of a waterfall development model results in latency.

(Proponents of a global development model say it results in 24 hours of development per day. That sounds good, but the dark side is that it results in 16-24 hours of latency for every question.)

---

The Off-Shoring Advantage

1. Cost.

That.     Is.     It.

...and even that may be an illusion.

[shrug] Sometimes cost is everything.

Off-shoring has success where bandwidth and latency is less important, for instance, mass manufacturing of commodity items like cell phones and laptops (although latency can sting here too, such as building millions of defective items or the wrong toy for Christmas).

---

The Local Advantage

1. High bandwidth
2. Low latency.

Conclusion: Play to your strengths.

So, what did we learn from plumbing?

• Plugged toilet: Joe the plumber has too high of latency, unplug it in-house.
• Major plumbing: Joe the Plumber has acceptable latency and much higher bandwidth (domain knowledge, building codes).
• Plumbing parts: off-shore — latency is not critical, but cost is.

Here is one final thought. What do Agile development methods emphasize?

1. Close interaction with your customer — high bandwidth.
2. Continuous integration, short iterations, incremental improvements — low latency.

Flocks of Programmers

Flocks of birds and software developers fly on three very simple rules:

1. Separation — Don't run over your neighbor.
2. Alignment — Make your flight vector similar to your neighbors'.
3. Cohesion — Have a common destination.

The difference between the "V"s of geese and the ever shifting clouds of starlings is the level of cohesion and the variance in their alignment.

Geese have waypoints; starlings, not so much. Starling flocks have a lot more coupling - they have many neighbors - and a lot less cohesion.

So, the question for software projects is, how do we reduce the coupling and increase the cohesion?

1. As Robert Frost said many years ago, "Good fences make good neighbors." For programs, good interfaces reduce coupling and negative interactions.
2. Increase bandwidth. One of the agile techniques is to have a very structured daily 15 minute "scrum" meeting: each developer says briefly what he accomplished the previous day, what he expects to accomplish today, and any issues he has. This helps keep all the developers aware of everything that is going on, helps maintain separation, and helps improve alignment.
   
3. Decrease latency. The open source mantra is "release early and often." This is where distributed revision control systems (e.g. git, Mercurial, DARCS) shine and where the open source technique of publishing patchsets on an email list is very beneficial; long windy roads can be straightened out before they become a serious problem.
   

Sailing Adventure 2009

We have been doing an annual Sailing Adventure on our '77 Ericson 27. What we lack in boat, we make up in fun. ;-)

This year we polled the kids whether we should go north (Manistee) or south (South Haven). Molly and Melissa said "north." Jeremy said "west" (i.e. cross the lake). Sorry, Jeremy, Manistee wins this year. :-D

2009-06-20 | Mileage: 19.0nm

• Rain cleared, sunny by 10:00
  
• On the water 12:40
  
• Motorsailed W ~5nm, then headed for White Lake
  
• Wind died ~ half way there, motoring the rest of the way
  
• White Lake Municipal slip 32p
  

2009-06-21

• Layover in White Lake: sunny, high haze & warm
  
• Paddled, rowed around end of the lake, up White River (under bridges)
  
• Melissa and Jeremy rowing up dock waterway and got caught by fisherman (they floated over his line, he must have pulled it in as they went over it and snagged the dinghy)
  
• Lots of turtles, several very large
  

2009-06-22 | Mileage: 41.8nm

• Sailing for Ludington
  
• Knotmeter broken, always 11.2kts. Dream on!
  
• On the water 9:00
  
• Sunny, light wind from SE (offshore breeze), motorsailing
  
• 11:00 wind shifted S, no apparent wind
  
• It is stupid fly season. :-( There are two fly seasons: June/July is stupid fly season. They are trivial to kill because they are so stupid, but there are hundreds more for every one you kill. They don't bite, but they crawl all over, making your skin crawl. August is biting fly season. They use their serrated proboscis to painfully saw a hole in your skin and are very hard to kill because they are very jumpy.
  
• 11:10 wind from W, cool and refreshing
  
• 12:40 coming up on Little Sable Point, wind from behind us, funky again
  
• 13:25 took down main, S wind behind us
  
• 14:50 passing Pentwater, wind backed NNW
  
• 14:45 gave up on wind, motoring last 6nm
  
• Ludington Municipal, slip b22p
  
• Went to the House of Flavors for ice cream cones :-)
  

2009-06-23 | Mileage: 23.2

• Depart 9:15, light W wind, motoring w/ sail up 0.1kt boost
  
• 11:00 rounded Big Sable Point, better wind angle, 5.9 kts
  
• 11:30 Wind clocked around behind us, jibed the jenny
  
• 12:50 wind died, motoring last 3nm
  
• Manistee Municipal, slip 11s (fishing boat ahead of us jumped our original slip assignment)
  
• The White Rose (Frankfort, Il) is in the next slip - 60 ft? yacht
  
• Some changes in town: new condos by Maple St bridge, Green Apple shop gone (bummer).
  
• Molly flipped the kayak getting in off the rocks (jumped in, instantly rolled). Once we got her on the water right side up, Dad, Melissa, and Jeremy rowed the dinghy and Molly paddled the kayak up river to the lake.
  

2009-06-24

• Layover in Manistee, sunny, blue sky. Gorgeous beach day.
  

2009-06-25 | Mileage: 33.1nm

• 9:00 heading back to Ludington
  
• Patch of blue overhead, clouds all around, hazy, temp cooler, pleasant. No wind.
  
• 9:50 som wind from W, main up, little boost
  
• 10:05 Weather holding, changed destination to Pentwater
  
• 10:20 Got the jenny out, the wind died :-( see what happens around Big Sable Point 7.8nm
  
• 10:50 Picked up some wind, set jenny
  
• 11:10 Lost the wind, jenny down
  
• 11:25 Found wind, set jenny
  
• 12:00 Rounded Big Sable Point, jenny down again
  
• 12:20 Set the jenny, broad reach, maybe the breeze will hold this time
  
• 13:00 Passing Ludington, breeze nice
  
• 13:50 Lost our bet that we could beat the rain to Pentwater. Just sprinkles, no big deal.
  
• Pentwater Municipal 36p (they have 7 transient slips - this is the first time we got in the municipal)
  
• Rowed to Little Bayou with Molly & Jeremy, Melissa kayaked
  
• Supper at the Brown Bear - excellent burgers
  
• Band concert on the green at 8:00PM
  

2009-06-26 | Mileage: 39.7nm

• Pentwater municipal: just the basics - no coffee, no microwave - but clean, nice view over the channel and anchorage
  
• Breakfast at Gull Landing, on Island time: good food, but over a hour wait :-/
  
• 11:30 left
  
• Sailing W wind
  
• 12:30 wind dropped some, used motor to kick us through a dead spot
  
• 13:25 Passing Little Sable Point, motor kicker again, wind aft and following seas
  
• 13:50 Main down, jenny not doing much :-(
  
• 14:10 Wind picked up, jenny working :-)
  
• Waves building ~3ft until passed White Lake, then diminishing as we approached Muskegon
  
• Supper at Bear Lake Tavern - new owners, same excellent food
  

2009-06-27

• Slept on the boat in our slip.
  
• Queen's Cup (interesting history) Milwalkee-Muskegon, saw the boats finishing in the morning
  
• Breakfast at Mike's Landing - yum!
  

Cruising - worked:

• Dinghy and kayak - fun!
  
• Thermos e5 mugs - awesome insulation!
  

Cruising - didn't work:

• DTV was a total bust
  
• Laptop screen unreadable topside (no surprise)
  
• Towing dinghy more drag than I expected - over 5kt, it had a bigger wake than our sailboat!
  

Requirements are a finite set of rules.

[A]ny finite set of rules is going to be a very incomplete approximation of reality.

— Douglas Lenat (Stanford University)

In the above quote, substitute "requirements" for "rules." Requirements are necessary to bound the problem, but insufficient to fully define reality. This is a major reason why "Big Design Up Front" (BDUF) has an unblemished history of failure.

The quote comes from a very interesting story in the New Yorker, partially quoted below, and a whitepaper with more details.

In 1981, a computer scientist from Stanford University named Doug Lenat entered the Traveller Trillion Credit Squadron tournament, in San Mateo, California. It was a war game. The contestants had been given several volumes of rules, well beforehand, and had been asked to design their own fleet of warships with a mythical budget of a trillion dollars.
:
:
Lenat had developed an artificial-intelligence program that he called Eurisko, and he decided to feed his program the rules of the tournament. Lenat did not give Eurisko any advice or steer the program in any particular strategic direction. He was not a war-gamer. He simply let Eurisko figure things out for itself. For about a month, for ten hours every night on a hundred computers at Xerox PARC, in Palo Alto, Eurisko ground away at the problem, until it came out with an answer. Most teams fielded some version of a traditional naval fleet — an array of ships of various sizes, each well defended against enemy attack. Eurisko thought differently. "The program came up with a strategy of spending the trillion on an astronomical number of small ships like P.T. boats, with powerful weapons but absolutely no defense and no mobility," Lenat said. "They just sat there. Basically, if they were hit once they would sink. And what happened is that the enemy would take its shots, and every one of those shots would sink our ships. But it didn’t matter, because we had so many." Lenat won the tournament in a runaway.
:
:
"Eurisko was exposing the fact that any finite set of rules is going to be a very incomplete approximation of reality," Lenat explained. "What the other entrants were doing was filling in the holes in the rules with real-world, realistic answers. But Eurisko didn’t have that kind of preconception, partly because it didn’t know enough about the world."

BDUF, "IBM Master Programmer", and naive outsourcing strategies are "Eurisko programming." ... the resulting program may meet the literal requirements, but is not going to be very satisfactory.

Career Decision Tree

Do you know how to do the job?
Y: That's great, but learn something new.
N: Does it enhance my career (is it worth learning)?
   Y: Life is good, grow. —— employee sweet spot
   N: Does it pay well?
      Y: Do you need the money?
         Y: Suck it up, do the job.
         N: Are you a contractor?
            Y: To be expected. —— "hired gun" contractor
            N: Look for a new job.
      N: GET A NEW JOB. —— the coffin corner

Turing Test Take Two

One concept that Alan Turing is famous for is his test for evaluating artificial intelligence, know as the Turing Test. In the test, people attempt to identify by means of a conversation which "far end" conversationalists are people and which are computers.

I would propose an alternate version of the Turing test: the artificial intelligence side runs the source code through a tool and an engineer (fairly literally) implements the tool's recommended changes. The "control group" would be one or more experienced engineers who do a peer review of the same code and implements their improvements. If an observer cannot tell which code "improvements" were the result of the machine and which were the result of good engineering judgment, the tool has passed the Turing Test Take Two.

Conversely, if a tool cannot pass the Turing test, we must use an experienced engineer to filter the "recommendations" that the tool makes before we apply it.

---

Case study

---

In our contracts and in our work instructions, we have implicitly made our tools the "gatekeeper" and final judge of our code quality. The way that we fall into this trap is via our contracts or Plan for Software Aspects of Certification (PSAC), we specify that we will provide the artifacts that are generated by our tools to proved that our code is "good." The result is that the goal of running the tool is no longer to produce good code, but rather it is to produce clean printouts ("no faults found") for the customer.

The way back from this madness is to make engineers responsible for the code they write and the code they review. They should use tools to help them write good code and perform quality reviews, but the artifact that we take to the customer should not be "PCLint signed off on this code written by an anonymous cog" but "Gerald Van Baren wrote this code and is proud of it" and "Joe Competent Engineer reviewed this code and agrees that it is good." In other words, our engineers must taste the sausage. (In that article, map leaders => experienced engineers (aka. gatekeepers), sausage => code, broken machines that result in overtime => broken or misapplied tools that result in overtime.)

Our C Coding Standard (an internal standard consisting mostly of MISRA-C rules) is a classic example of a tool gone wrong. We sowed the seeds of a Turing Test Take Two breakdown in Rule 4 (of the internal standard): "Source code shall be run though an acceptable static source code analyzer." When we write in our PSAC that we will follow our C Coding Standard, we just jumped the shark and never saw it coming. While Rule 4 does not explicitly state that the engineer must implement the tool's "recommendations,"¹ in practice it is easier to make the tool shut up than it is to explain and defend and defend and defend good engineering judgment that is contrary to the tool's "recommendation."

The case study in creating an "acceptable static source code analyzer" is the (first try internal C Coding Standard) checking tool. We spent (lots of money) contracting (elided) to implement it and then discarded it because it was hopelessly inadequate. We followed that by spending (a tenth as much money) on the (second try internal C Coding Standard) tool which was only moderately inadequate. We are now mainly using PCLint (thousands of dollars per seat) which is almost adequate, but is still incapable of passing the Turing Test Take Two.

We actually (inadvertently) did the Turing Test Take Two on the (first try internal C Coding Standard) and (second try internal C Coding Standard) tools: we assigned engineers to implement changes to project source code in a "sea of hands" fashion on the results of running the static analysis tool on that code. That was a disaster. Management quickly realized from the howling of anguish from the affected internal engineers that it wasn't working and backed off on that approach.

---

1. When I discussed the (first try internal C Coding Standard) tool with an experienced, highly regarded engineer, he told me he ran the (first try internal C Coding Standard) tool on his code because the PSAC said he had to. He noted that the PSAC had a [X] checkbox for running the checking tool, but did not have a check box that said that the results were used for anything, so he did an incredibly practical thing: he simply discarded the verbosely bogus results. He then ran PCLint on his code, using it as a tool (not a judge), to identify problem areas in his code and applied his engineering judgment to determine which complaints were real and which were artificial.

Intellectual. Property.

Or, as our British friends would say, "intellectual (full stop) property." Management has subscribed to the term "intellectual property" to allow them to indulge in their fantasy that the company's "intellectual property" can be transferred readily to a "low cost labor region." If the "intellectual" part is embodied in employees, it is not very portable, which really balls up their plans to save money by transferring their "intellectual property" to "low cost labor regions."

I contend that intelligence is real and it does create property for companies, but the property (code, documentation, manufacturing, etc.) is an artifact of intelligence. It isn't intelligence itself and thus "intellectual property" is a jarring discord. Toner on paper does not have any ability to be intelligent. Bits on a disk do not have any ability to be intelligent.

The property half is held inside the company walls, but the intelligence half walks out the door every evening.

Management can talk all they want about the company's "intellectual property", but disks full of bits have only a residual value without the human intelligence that understands what those bits mean.

As an aside, if the intelligence half did not walk in the company door some morning, it does not lessen the value of the property the company actually owns. Management is inappropriately taking credit for the intelligence that humans bring to the business.

Transferring "intellectual property" to "low cost labor regions" is an extremely short term and an extremely destructive strategy. The fundamental reason some areas are "low cost" is because the workforce there often is inexperienced in general and always is inexperienced in the problem domain of the "intellectual property." By the time the "low cost labor region" workforce has gained the domain knowledge to effectively use the "property" part (the bits on the hard drives), they will no longer be low cost. Aggravating this, there is a substantial risk that the intrinsic value of the the bits on the hard drives will have decayed to a level that no amount of added intelligence can restore the value back to its original level.

---

As a concrete example, how much better are the "big three" domestic automakers doing after transferring their "intellectual property" (knowledge of how to build automobiles) to "low cost labor regions?" I contend it nearly killed them. Maybe it has killed them and what we now see is their death throes.

In an article on IBM layoffs, Robert E. Kennedy, a professor at the University of Michigan's Ross School of Business states "GM is stuck with high-cost, medium-skilled engineers. That's one reason it takes GM seven yearsto go from concept to design to the showroom floor [to produce a new car model],whereas it takes Toyota only three years. If they were more into tapping into the best talent wherever it is in the world, GM would be in better shape today."

This is a completely specious argument. Toyota hasn't moved their engineering to "low cost labor regions." Instead, they have worked to make their engineers higher skilled and more efficient. Dr. Kennedy's quote cuts exactly opposite of what he intended to show: it states that offshoring is killing GM. The engineers that remain are the second rate ones... by implication, the best ones have all left.

My google searches indicate that Toyota created new technology centers in...

• Ann Arbor, MI - Detroit's back yard. Not a "low cost labor region." If there truly are no good engineers in Detroit, it would follow that they wouldn't be in Ann Arbor either.
  
• Cannes, France - one of the most expensive countries in the EU.
  
• Australia - I don't know how this compares, but it definitely isn't the low cost labor region in that region.
  

First post

When it is your own blog, it is easy to score "first post!"