Initially, I was like everyone else - appalled that Mayer would have such disregard for the many hard-working remote folks at Yahoo! and surprised that she would shoot herself in the foot so badly.

On the HackerNews post, I wrote:

Why does pulling people into the office finally give Yahoo the ability to fire under-performers? You don’t need to see butts in seats in order to identify people who are not pulling their weight.

If people are coasting at home OR at the office you can tell that by their work output. Where they do their work doesn’t enter into it.

Adopting a Results Only Work Environment (http://gorowe.com) would allow them to have folks who work from anywhere and give management the tools to fire under-performers.

If this were only a temporary measure, a “reset” so to speak, then I still don’t agree with it (I think you can do that without the disruption), but it would be understandable. As they have not said that this is temporary, I have no reason to think it’ll come back in the future, which is the tragedy.

It doesn’t have to be like this.

The more I think about it though, I was wrong. Yes, the internet is right that remote workers are not inherently unproductive and that pulling everyone back punishes the many for the misdeeds of the few. More gallingly, revoking remote work shows you don’t trust your coworkers.

You know what? She probably doesn’t.

Since 2009, Yahoo! has had 5 CEOs (interim or otherwise) and gone through at least one complete reorg. Lots of people are coming out of the woodwork stating that remote work at Yahoo! was like a vacation. And, Yahoo! has long been the butt of a joke:

Q. How do you take a profitable, popular startup and kill it?

A. Get acquired by Yahoo!

So, can you really trust anyone at the company? Sure, some responsible remote workers are caught in the crosshairs but with reports of “We’ve checked and some people who work from home haven’t even logged into the VPN” how can you trust their immediate management to do the right thing? And their managers?

Whom can you trust? Who is taking advantage of Yahoo and who is merely caught up in the downward-spiral that has been Yahoo! for the last decade?

Marissa wants to look into every employee’s eyes and make that call. And I can’t really blame her. There’s probably more dead wood at Yahoo! than South Dakota in the 1870’s, and it’s going to be a long, slow journey to get things back on track.

• They’re noisy. That is to say, there’s an exhaust fan that is just the right amount of white noise.
• The urinals have an appropriately sized divider between them
• The stalls are actually little rooms. Floor to ceiling walls, a door that closes.
• The sinks are fashionable, have towels, and hand lotion.

It’s really quite nice. So, why do most employee bathrooms look like a prison?

I actually interviewed at a well-known, highly funded (tens of millions) tech company here in NYC and visited the restroom. It was dead quiet. The stalls faced the sink and had large gaps between the door and frame, so when you washed your hands and looked in the mirror, you could see right into the embarrassed eyes of the person sitting there.

I casually mentioned how quiet the bathroom was to one of the interviewing engineers and he said “Oh yeah, it’s terrible. No one uses it unless they have to. I go home to use the bathroom, other people use the coffee shop downstairs.”

Companies: it’s not hard to make humane bathrooms that don’t stress people out. Take care of your coworkers by providing reasonable accommodations. I didn’t take the job at that company for a number of reasons; the terrible bathrooms was one of them.

This chart says it all:

Yes, it’s relative share (a smaller part of a bigger pie) but PC growth has stagnated. Microsoft makes a significant bulk of its Windows revenue from new PC sales and business upgrades - very few consumers actually upgrade their Windows absent a new computer. Back in the 90’s technology moved so quickly that a computer released in 1998 had technology not even heard of in 1996. You’d be a fool not to upgrade! Nowadays? Not so much.

Why? Because Microsoft has no taste. And more importantly, neither do most Windows software developers. Open up virtually any popular Windows application and you’re presented with a really ugly interface:

Boring grids, lackluster UX and visual design, indecipherable “help wizards”.

iOS and OSX app developers have a strong design aesthetic. OSX apps look nice:

Web developers share the same design-as-a-core-competency philosophy influenced by Apple:

• Square
• MailChimp
• Wufoo (and)

And developers are on the same OSX bandwagon. At virtually every innovative conference and meetup I attend, ranging from JavaScript to Ruby to the Business of Software conference, I’m met with a sea of glowing Apple logos on MacBook Airs and Pros (or the subdued Apple logo on iPads).

Yes, if you go to a .NET conference or perhaps a big-enterprise conference, you’ll likely see a lot more Windows machines in the audience. But the “get ‘em young” strategy is likely to work well for Apple, as future developers starting with OSX and Apple are significantly more likely to continue with the platform as they enter the workforce, demanding OSX development environments or not working at predominantly Windows shops.

Microsoft realized this years ago but only recently has there been a flip-flop on the younger generation as new college students prefer Mac to PC.

This is a trend that is likely to continue as long as Windows clone manufacturers continue to underperform against Apple. And, without the halo effect of a strong phone and tablet, Microsoft has no hope of luring consumers back to the Windows ecosystem.

Because Microsoft has no taste neither do their developers, and so there’s nothing in the Windows ecosystem that is desirable:

Today’s children are tomorrow’s consumers (and business owners!) and they are overwhelmingly rejecting Microsoft’s (non-Xbox) offerings in favor of Apple (and to a lesser extent, Android).

But what about businesses? Microsoft has long held a lock on large enterprises that require strong controls and IT departments. Apple has a history of ignoring this market, and there’s no evidence of this changing. However, with the “BYOD” movement, Apple is proving to be a competent player and Microsoft is increasing costs. Enterprise Apple sales are outperforming Windows.

Microsoft also held a lock on the small and medium sized businesses that could setup Windows computers, servers, and services (Exchange, MS Office, etc.). However, SMBs are increasingly replacing expensive locally hosted solutions with cloud based that are cheaper, require significantly less management, and provide better availability and performance.

The loss of developer, youth, and app mindshare is a huge blow to the Windows platform. iOS and Android ecosystems are defined by the apps that run on their devices. And the web is increasingly becoming the way we run applications on our bigger desktop and laptop computers.

Microsoft has billions of dollars in revenue and is not likely to disappear any time soon, but the era of Microsoft as a dominant consumer platform is over. Developers are moving to the web in droves and switching to OSX (or Linux). American youth are preferring Apple and iOS products, and small and medium sized businesses are replacing Microsoft software with web-based apps to manage and run their business.

Software companies that don’t have design as a core, integral component to their software design are at an ever-increasing disadvantage against companies that do.

(Edit: Updated with a more recent image of QuickBooks.)

Facts do not cease to exist because they are ignored.

In my last post, I discussed why software effort estimation was harmful. I mentioned that you can’t add up all the individual tasks to get a completion date at some arbitrary point in the future, and you can’t (with some caveats) use estimation to “staff up” a project from the beginning.

Where does this leave us? Should we not estimate at all? These folks think so:

• Good Agile, Bad Agile
• Do not estimate software projects at all
• Impact of Schedule Estimation on Software Project Behavior

And even if you could estimate, they’re always wrong because of the difficulty in estimation coupled with the inherent variability of development:

• Saving for a Rainy Day
• The Challenge of Productivity Measurement
• Understanding Software Productivity
• Engagement Level and Impact on Productivity

How We Estimate

Some folks have asked how I/we estimate. In general, I think fine-grained estimation (hours, etc.) is a waste of time. Hours are not interchangeable (one developer may get it done in seven hours, another in six) and don’t accurately take into account risk.

At work, we use relative points-based estimation where 1 point = small, 2 points = medium, and 3 points = large (large stories are almost always broken into smaller stories before development begins). We have a guide that gives representative stories for each size point.

Initially we fill our backlog with a large number of stories, broken down into individual deliverable units (e.g. “user can login”, “system sends email reminders”). We take a rough estimate of size (for some reason a well-gel’ed team can agree on relative sizes without too much difficulty) and use that for basic prioritization.

We then pull items off of the top of the backlog and work on them. As we go, if a story feels “too big” it’s broken up into smaller tasks. In general, no one should work on a “large” story - it should be decomposed into small and medium sized stories.

We can use velocity and “gut feel” to continually keep the backlog stacked with a few weeks worth of stories, ensuring that each developer has enough work to keep them busy (Google’s “Priority Queue”).

Why Points?

Points reflect that software development is inherently a chaotic and un-knowable endeavor, full of uncertainty and icebergs. But, this doesn’t allow us to abandon accountability or plan for the future. We’re professionals, and we act like it. Points lets us spend our incredibly valuable and scarce time working on what we do best: crafting amazing software, not agonizing whether feature X will take Y or Z “ideal hours”.

Points also reflect that a given story contains the same amount of risk or size no matter who works on it. Switching from PostgreSQL to MongoDB represents the same amount of risk and relative effort no matter who works on it.

Lingering Questions

There were a few questions raised in the last article that I’d like to address.

• How do we know how to staff a project?
• When will it be done?
• How will we make internal/external commitments?

How do we know how to staff a project?

If you look at the best estimators, they have a few things in common:

• Teams are fixed (no churn in developers)
• They have a lot of historical data to depend on
• They are often working on few, long-term projects or many projects that are very similar

These things typically only exist at large, established companies. Those that probably operate under a waterfall-type model - minimize risk and cost. The software they create is rarely used to disrupt entrenched markets but to claw share from competitors - and viewed under this lens it makes sense that any given software project at these companies generates little measurable value and so requires the most control.

(This post is not for people working at those companies, although I think given enough prodding, you could get your management to be more flexible.)

But - if you’re asking “how many people?” then you’re removing the main support of effective estimation - that is, fixed team composition. Historical data estimation only works if you have the same people working on the future project. Otherwise, the estimates are significantly less valuable, especially in the short run as teams of new people fight to “gel” (and don’t forget the mythical 10x developer).

More interestingly, though, is that each additional programmer past a certain number gives diminishing returns. This is something (generally) not accounted for in historical data. You can’t take the performance of a four person team and multiply it by ten to achieve the performance of a forty person team.

If you have n-number of “programmer months” you also can’t simply throw a large number of developers at the project to get it done “faster” (also known as schedule compression).

The larger the team, the less schedule compression you achieve, and the greater diseconomies of scale. It sounds surprising at first, but intuition should help. If it took five people a year to work on a project, would it take ten to finish in six months? Maybe, but that seems doubtful. Not everything on a software project can be worked in parallel. Gut intuition tells you that it should be done later than 6 months.

What about twenty people. Could you get it done in three months? That seems a lot less likely. What about forty people? Could forty programmers get it done in a month and a half? Absolutely not.

The data backs this up:

• Haste Makes Waste When You Over-Staff to Achieve Schedule Compression
• Top Performing Projects Use Small Teams
• Small Teams Deliver Lower Cost and Higher Quality
• Mythical Man-Month Surgical Team

So what might account for these results? Over three decades of data from the
QSM database suggest that defect creation and density are highly correlated
with the number of people (and hence communication paths) present on a given
project. Larger teams create more defects, which in turn beget additional
rework. Fixed code must be retested and the team may find new defects injected
during rework. These unplanned find/fix/retest cycles take additional time,
drive up cost, and cancel out any schedule compression achieved by larger
teams earlier in the lifecycle.

Conclusion: Minimize team size. The “optimal” size of teams is relatively fixed (less than five developers). “How many developers should we have on this project?” is the wrong question to ask.

“How can we subdivide the project into 2-4 person teams” is the right question, and planning can help with that. I’ve never had estimation be a high-order term in the equation, though. Generally putting a list of all of the different features broken into small chunks, organized by rough size (Small, Medium, Large) seems to work just fine.

I think a key point of this is on larger software projects, you must subdivide the work into separate components that don’t rely on shared state, etc. between them. This means your different components will act via message passing and an API.

Steve Yegge’s infamous “API ALL THE THINGS” post is a good example. It’s not an exact fit, but if you squint a bit and cross your eyes so you can see through the picture, you get an idea of what I mean.

When will it be done?

What does “done” mean? There’s always a feature you can add. Always some refactoring or improvements to make. “Done” is just an arbitrary pile of features that are “good enough” to be shipped.

Tower of Babel

What seems to happen most often is that over time, software never gets “done” because developers don’t have a clear “done” criteria. This leads to software built very wide (lots of features) but not very deep (not fully “done”), leading to the age-old “90/90” rule.

Even if a feature has well-defined “done-ness” estimation has a strong anchoring effect that encourages “90%” done features. If you estimate a feature to be eight hours but it really takes twenty to be “done”, in most companies you’ll receive pressure to compromise on quality or take shortcuts because you’re “taking too much time”.

Absent external pressure, I’ve often seen developers put too much emphasis on the initial estimate - even if there’s no manager breathing down their neck, the selfsame developer will feel guilty that they are taking “too long”.

Conclusion: Features should be fully “done” before moving on to other features. Avoid anchoring effect and be willing to invest the appropriate time to release high-quality code.

How will we make internal/external commitments?

The creation of genuinely new software has far more in common with developing
a new theory of physics than it does with producing cars or watches on an
assembly line. ref

This is a hard problem. I argue that building software is a lot closer to writing poetry than building a bridge. Ask a poet how long until the sonnet is written and you’ll get a blank stare.

Unless you’re building something very similar to something you’ve done before, you’re fundamentally creating something that has never existed before. Even the most “scientific” estimates contain a large chunk of risk (or “unknown unknowns”) that cannot be quantified ahead of time.

If you must make a commitment to a random date in the future and scope is fixed, you’re almost always screwed. Agile development helps avoid this by encouraging (requiring?) incremental development. Start working on the most important features first; aim for always shippable/deployable software by keeping the features small.

In this manner, you’ll always have working, testable software and will be moving towards the goal. Your customer/stakeholder will see the progress (or lack thereof) instantly, and be able to re-prioritize as necessary. Since you’ve got shippable software at every point in the game, Marketing and Sales can accurately inform customers on what’s going to be in the next release in a “just-in-time” fashion.

Many software companies have been abandoning announcing arbitrary dates for their latest products and announcing at conferences and tradeshows because it’s too restrictive and places their development on someone else’s timeline.

When Apple does a product announcement, the software has been in a feature freeze for months, and the hardware inventory has already been piling up. They don’t release handwavy dates “sometime next year” because they give a date once they’re already finished (iOS betas are more or less feature complete: Apple spends a long time polishing and fixing bugs, but no major features are usually released once the betas are released to the public.)

And, the functionality that is delivered is not promised ahead of time. I suspect Apple has a high level roadmap and works on the most important features first and cuts anything that isn’t ready or good enough.

If Microsoft and Apple can’t predict the future, what chance do you have?

Conclusion: Work on the most valuable features first in an iterative fashion. Always be delivering working software.

How to be Successful With Minimal Estimating - Build Small Software

Many small things make a big thing.

Since team productivity is maximized with smaller team size, effort is fixed, and software is inherently unpredictable, this leads to an interesting conclusion: don’t build big software projects.

Large software projects are cost/productivity inefficient. Large software projects contain a large amount of risk that cannot be determined a priori. Large software projects fail more often than smaller ones.

• Use rough approximations of feature size for high-level planning and guidance
• Use small teams to build small software
• Deliver frequently, working on the highest-value features first
• API all the things to enable different teams to integrate via a standardized API

Delta Air Lines tells me it will take two hours and twenty-five minutes to fly from New York City to Atlanta, GA. American says two thirty. United? Two thirty-four. Yet, the captain just told me it was “one-hour forty-five” minutes “wheels-up to wheels-down”. What gives?

It turns out that the US Department of Transportation (US DOT) rates the airlines by “on-time performance”: whether or not the aircraft reaches the gate by the quoted time. So, like any rational actors, the airlines pad the flight time to account for ground delays (waiting by the gate for the tug driver to appear), re-routes due to bad weather, etc. - but also to goose the ratings.

They could massively pad and say the flight takes three hours, but that impacts the ability for folks to connect to other flights. Airlines constantly tweak the advertised flight time: too much padding and the wasted time on the ground costs dearly; too little and the DOT will come after you.

Software

Software developers are often called upon to estimate various tasks, features, or bugs. Oftentimes this is measured in hours, but for larger stories it could be days or weeks (month-long estimates don’t seem to be as common).

These estimates are used to estimate the effort for a given task and, in the hands of a business user/project manager, added up to get the completion date for the project as a whole (e.g. feature X takes 2 weeks, feature Y takes 3, X + Y = 5 weeks to launch). This completion date is then used in a formula to calculate total project cost (salary per day * days, or equivalent).

In really terrible places to work, someone other than the developer will actually do the estimating. This estimate will then be given to the developer as an implicit (or at especially evil places, explicit) time not to exceed. This is such a depressing situation I cannot dare think of it further.

Estimation Process

The estimation process itself is not very well defined at many software companies. Often the developer will try and compare feature X to a known feature they have completed in the past. For example, “Create login form” may have taken them four hours on a previous project, so they can reasonably estimate likewise for a login form on a new project. Or, “Add a customer” seems a little bit harder than a login form, so multiply the login form estimate by 1.5x to get 6 hours.

It gets a lot trickier for larger features, like “Customer Analytics”. What does “Customer Analytics” mean? Hopefully the developer has been given a specifications document from a business analyst (or equivalent) that spells out all of the requirements:

• Bar chart of Customers over time (group by signup date)
• Ability to change scale (view week, month, year)
• Exclude inactive Customers
• etc.

In this case, the smart developer would decompose the feature into smaller features that can be compared to past experiences. The dumb developer would try and estimate the feature as a whole. Bad, naughty developer!

Missing Data

Most estimation is drawn from past experience. What if the developer doesn’t have this experience? How can they estimate? This is where things get tricky. And by tricky, I mean “completely made up.” Sure, for things that have been done before by someone the developer can draw some parallels and make a guesstimate. But what about the stuff that has never been done before (presumably, the secret-sauce that makes you money)? Usually you just take a Wild-Ass Guess (WAG-estimation) and hope you’re not wrong. Given the rarity of being punished for under-promising and over-delivering this WAG tends to be a massive over-estimation.

Effort vs. Accuracy

It seems obvious that the more time you spend on your estimates, the more likely you are to “get it right.” Right? This thesis is basically the entire justification for the Waterfall model of software development.

Completely spec out the entire system ahead of time, spend a lot of time researching and estimating the problem, determine project dependencies, and you can determine the “finish date” before writing a line of code! It’s seductive and taps into the part of our brain that craves order and dependability.

Pretty little stories all in a row

It turns out that effort spent estimating only provides marginal returns and, eventually, becomes negative. Why? Because of overfitting.

In machine learning, overfitting generally results from having too little training data (edited: incorrect usage of test data) or focusing on irrelevant metrics. The costing algorithm you evolve into can only accurately predict your training data. When fed a new problem the algorithm provides widely variable or incorrect information.

Human estimation algorithms fall into the same trap - the more you try and dig into a problem, the more you see parallels that don’t exist and the less you account for randomness (or “unknown unknowns”). You end up being very confident in your estimation but ultimately are wildly wrong.

“Agile Estimation and Planning” Addison-Wesley

Why Estimate?

Let’s back up a bit. Earlier I wrote that folks estimate in order to determine the completion date for the project as a whole.

And? What’s the point of knowing when the entire project will be complete? Who cares? Typical answers are:

• So we know whether or not to undertake the task
• So we know how many developers to put on the problem
• So we don’t overspend
• So they can generate a ship date for marketing/PR reasons

But I think the deeper, much-less admitted reason is that managers and other “business people” don’t do a particularly good job of calculating business value, motivating, and managing software people and projects. They wield the estimate as a cudgel: perform X by Y date or else.

Business Value Knife Fight

In his 1986 book, ”Controlling Software Projects: Management, Measurement, and Estimation” Tom DeMarco (yes, of Peopleware fame) wrote “You can’t control what you can’t measure.” This is true, but is it important in software projects? Some 23 years later Tom doesn’t think so.

“You can’t control what you can’t measure.” This line contains a real truth,
but I’ve become increasingly uncomfortable with my use of it. Implicit in
the quote (and indeed in the book’s title) is that control is an important
aspect, maybe the most important, of any software project. But it isn’t.

The full article text is here.

Tom goes further to state:

My early metrics book […] played a role in the way many budding software
engineers quantified work and planned their projects. In my reflective mood,
I’m wondering, was its advice correct at the time, is it still relevant, and do
I still believe that metrics are a must for any successful software development
effort? My answers are no, no, and no.

The tl;dr of his article is that the software projects that require tight cost control are the ones delivering little or no marginal value. That is,

To understand control’s real role, you need to distinguish between two drastically
different kinds of projects:

* Project A will eventually cost about a million dollars and produce value of around $1.1 million.
* Project B will eventually cost about a million dollars and produce value of more than $50 million.

What’s immediately apparent is that control is really important for Project A but
almost not at all important for Project B. This leads us to the odd conclusion
that strict control is something that matters a lot on relatively useless projects
and much less on useful projects. It suggests that the more you focus on control,
the more likely you’re working on a project that’s striving to deliver something
of relatively minor value.

To my mind, the question that’s much more important than how to control a software project is, why on earth are we doing so many projects that deliver such marginal value?

If the value of software we’re writing is so low, is it worth being written? If the project has such tight time constraints that a schedule slip will doom the project then you’re in a world of hurt. (The solution is agile: work on the most important things first, have an always-working project, and cut scope to hit a date.)

This exposes a major weakness in most software companies: the inability to determine project value. Given a project of unknown value, the conservative business will then attempt to minimize cost. Unfortunately as we’ll see, cost minimization ends up becoming very expensive in the long run.

Hierarchy as a Service

Most companies are incredibly hierarchical - programmers report to a tech lead who reports (tangentially, to a project manager) to a mid-manager who has one or two layers in-between them and the CIO/CTO/CEO. The further up the stack the greater the nosebleed: the lack of visibility of the day-to-day machinations of the software organization and the less concrete decision making (aka vision over strategy/tactics).

This results in senior/mid tier managers that don’t know much about how software actually gets written. Given the perverse cost-minimization (because they didn’t do their job) organizations go to great lengths (of both time and money) to keep projects under tight control:

• They require big up front design before any code is worked on
• They require sign-offs and constant oversight
• Anything that isn’t directly driving completion of a task is forbidden (or at least, implicitly disincentivized)

What does this do? It encourages waste via Parkinson’s Law. It discourages refactoring when software entropy becomes unmanageable. Ultimately, estimation ensures software projects underachieve. By strictly managing cost, an organization guarantees the value is strictly limited.

(I’m not entirely sure what this means, but I’ve never seen the folks who are writting the specs, requirements, etc. ever being asked to estimate how long each spec will take to write. Or have the CEO give a date when the company will hit some arbitrary metric, although see the perversity in public company earnings estimates and reporting.)

Trust me, I’m Lying

Ultimately companies require estimation because they don’t trust anyone. They don’t trust developers to deliver results. They don’t trust mid-management to motivate their reports. The underlying reality is that the folks “in charge” are worried that everyone is going to rip off the company and only by holding people to arbitrary deadlines are they assured stuff will get done.

Committing to a date allows them to sleep better at night and shifts the blame from them to the folks on the front lines. And this is wrong, both morally and from a pure technical perspective. (See: Results Only Work Environment for how to manage based on results.)

Estimation considered harmful

Estimation is ultimately a futile effort. Software, more or less, is like writing poetry. Or solving mathematical proofs. It takes as long as it takes, and it’s done when it’s done. Perfect, or imperfect, estimation won’t change how long it takes to complete the work. If that sounds horrible to you, then go do something else.

Even if we could completely accurately estimate all of the pieces, software of any complexity has interdependencies and “unknown unknowns” that tend to not surface until implementation time. You can’t know the end date a priori by simply adding up all the components.

Estimation actually slows down development because estimation isn’t free. The developer is incentivised to take as long as possible estimating in an effort to not get beaten when they miss the estimate. Why waste that time when the metric the developer is attempting to hit ultimately generates negative value?

(In Peopleware they discovered that projects where no estimation was done had the highest average productivity; any estimation lowered productivity no matter who performed the estimation.)

Software is not like bricklaying - you can’t speed it up without major consequences. The Mythical Man-Month teaches us that adding developers to a late project will only make it later due to exponential communication and ramp-up (coined ”Brook’s Law”).

This results in folks that think adding more developers to a project at the beginning will allow them to hit an arbitrary date by spreading out the workload. Unfortunately new software doesn’t work this way, either, as the same exponential communication overhead occurs in the beginning, too. You merely end up with a giant mass of incoherent code. And you waste a lot of money employing people who don’t add much value, sitting on the bench waiting for their part of the project to begin.

The solution is “Don’t build big software”, which I’ll cover in a future post.

Two Points for Honesty

Estimation isn’t all bad, though. The act of estimating forces you to consider the problem domain in some detail. As Winston Churchill said, “Failure to plan is planning to fail.”

This is why I espouse the agile activity of relative points-based estimation. Regardless of which point system you use (linear, exponential, etc.) group-based estimation helps the entire team arrive at a shared understanding of what each feature/story/etc. is supposed to do. This is critical to keeping a high bus factor.

The key point is that these estimates are never, ever used to try and pinpoint a “ship date” (you should be using continuous deployment anyway) or used to punish a developer for “taking too long”.

The estimation process should take place relatively quickly. Break stories up into trivial tasks (which by definition don’t need estimating) and then roughly estimate the risk posed by other stories (external integrations, hard machine learning problems, etc.). In all, this process shouldn’t take very long.

This ensures minimal estimation waste, keeps developers happy, and velocity high.

(UPDATE) Evidence of What?

Some folks have asked: “What about evidence-based scheduling?”

That’s a very good question. I have a huge man-crush on Joel. Probably so much that if I saw him in person I’d babble like a child and be basically incoherent. He’s done more to advance the state of programmer work environments and software development than anyone I can think of in recent history.

Joel doesn’t strike me as the kind of guy who would use hourly estimates to punish developers. So, I think it comes from a good place, and not the deep-dark evil place that most estimation requirements come from. But, when it comes to EBS, I’m not sure it really provides that much value relative to the overhead vs. relative points-based.

It’s really close to what agile points give, namely a burndown chart, except you’ve got all the overhead of keeping track of exactly how long it takes to finish (I admit that I may be overestimating the time it takes to manage this reporting.)

Even if it’s easy to record the hours, I’m still skeptical that things like Parkinson’s Law might intrude or that the psychological effects of hours-based estimation may have unintended consequences. I’d rather stick to points.

Click here for the Hacker News discussion

Defensive Disclaimer

Building safety-critical software is necessarily out of scope, as that is a completely different animal. I’ve never built space shuttle software or something that could easily kill someone so the suitability of this technique is undefined.

It was nice, especially in a company that was full of Linux developers. My home directory was magically NFS mounted, all of our software “just worked” when I’d log in to another machine, and we could get pretty cheap computers that performed exceptionally well! All the software I had on the server I had locally (vim, psql, etc.) and it was very seamless and awesome.

When it worked, it was just magical. When it didn’t, which was rare, it wasn’t too difficult to work around whatever was going on. I learned the value of a really well-run IT department that had the budget to “do it right” instead of the typical “do it right now”.

The problem was all that infrastructure was pretty expensive to maintain. We had SAN guys who kept the home directories performant. I’d see company-wide emails go out “Please keep your home directories under xxGB!” as folks were publicly shamed for using “too much” shared storage. At the time, automatic updates wasn’t a “thing” so the IT folks would come by periodically and physically swap out machines that had the latest apt-get update run. Woe be the developer that installed anything outside of their home directory!

Anyway, after that I was hooked. Sure, Windows had some sort of “roaming profile” business but it really didn’t work as well as the Linux NFS solution. I made the decision that I was never using Windows again.

Fast forward a bit, and I found myself at an all Mac shop running OS X. I found OS X to be an amazing blend of a better GUI than Windows with the power of Linux (it’s BSD underneath). Wow! Finally, a decent GUI (GNOME was terrible at the time) but with all of my usual GNU/Linux tools! Huzzah! All was well in the land of OS X.

Except for one thing: the .dmg.

Mac Package Deployment

OS X has this peculiar method of deploying software. Instead of an “Installer” like Windows, that litters your environment with configuration, registry, and userland data files, OS X (typically) distributes software without installers as stand-alone .app directories bundled in mountable disks. This is quite an interesting and elegant solution, I think. Except it’s implemented in the worst possible way.

Download most OS X software and you end up with a .dmg file sitting in your downloads. Double-click it and you’re presented with the following:

What am I supposed to do here? There’s no discovery. What if I double click the icon? Oh, it opens the program. That’s how I run it, right? But what about the “Applications” folder there. What am I supposed to do with that? Turns out you’re supposed to DRAG the Skype icon to the Applications folder, thus “Installing” it to your machine.

It’s gotten so bad that some apps, like the most-excellent Cloak, give you instructions:

Provided I follow those instructions, I’ve now installed it to my Applications folder, I guess. Now I want to get rid of the installer, so I delete it and empty the trash:

But, cloak isn’t running? What do I need to do now? Turns out you need to “eject” the dmg. What you’re doing, under the covers, is actually unmounting the virtual disk that OS X helpfully mounted for you when you double-clicked the .dmg.

Only then can you empty your trash.

This makes no sense for the average user.

I guess eventually you can figure that out and memorize these steps except for one thing: OS X, as of Mountain Lion, hides the “Applications” folder. If you install a fresh copy of Mountain Lion you rarely see the “Applications” folder. It’s not on the dock, like “Documents” and “Downloads” are. Why not?

OS X App Store

With Mountain Lion, Apple has introduced the concept of the App Store as the way to install apps to your computer. If your app is downloaded outside of the app store, you’re presented with a scary dialog that doesn’t (easily) let you install the app.

Apple is trying to push OS X developers to distribute through the app store; the .dmg is not long for this world. I doubt we’ll see any design improvements to the .dmg process in further OS X releases. Developers should consider the App Store and, if it fits your business model, target the store first. If you must deliver software via the .dmg, please make it as easy-to-use as possible, like Cloak.

Today I happened to have iTunes open when I plugged in my iPhone for charging and I noticed that the iPhone quickly appeared in the “Devices” list before mysteriously vanishing again. To the internet!

A quick Google search showed quite a few people feeling my pain - the devices sync but don’t appear in iTunes and only reinstalling iTunes would fix it. I dug into my Mac’s system logging and saw various red herrings about being unable to sync via USB but nothing panned out. I tried turning it off and on again but nothing worked.

Defeated I gave up and went back to iTunes to resume IDD (Inception Driven Development) when my mouse cursor errantly floated over the “Devices” column. Actually, not the column itself, but the word “Devices”. And all was clear.

What they did

Clicking “Show” revealed my iOS devices.

Wow.

What they should have done

What terrible UX! The iOS devices live in an invisible panel with no visible affordances to indicate that this panel exists. Is this really necessary? Are there iOS fanatics out there with dozens of attached devices that need the extra screen space granted by hiding the list? Even if so why not put those little arrows like you see everywhere else in iOS (and Windows and websites)?

Here is what’s awesome about Vagrant:

• Uses Chef or Puppet to add packages to a base VM
• Easily maintain and rebuild VMs
• Pretty straightforward install process on OS X, Linux, and Windows
• Mounts your source project internally so you can work on your code in the VM

Here’s what I don’t like about Vagrant:

• SSH’ing into the VM all the time is a pain
• Creating your own box from scratch is a bit difficult
• Working in a group/team environment is left up to you to figure out

Enter: veewee. Veewee helps you build your own vagrant boxen instead of using predefined boxes that come with Vagrant. If you’re an untrusting person, as well you should be, you kinda want to build your own VM from scratch.

Combined with something like Dropbox or a local server, you can easily set up a group/team environment, build your own boxen, and then distribute them to your teammates.

Once you have your base VirtualBox boxen defined, you can skip straight to running the last step of loading the box into vagrant, of course, so this is really only necessary if you want to define your own boxen.

Here’s how I did it:

1. Download and install VirtualBox
2. Download and install Vagrant
3. Setup Dropbox and put my Ubuntu ISOs in a shared location
4. git clone git@github.com:FundingGates/fg_veewee_scripts.git
5. Run fg_veewee_scripts/setup.sh
6. Follow the instructions to build a box
7. Note, the validation of veewee-validation nfs mount may fail; it appears to be a transient bug in VBox. It doesn’t appear NFS is at fault, so it seems you can safely ignore it.
8. Place the VirtualBox boxen in Dropbox (box box box).
9. Load the boxen into vagrant, run vagrant up in your project (check in a Vagrantfile to all your projects). The end!

I created a little shell script to help share the virtual box definitions within our team. You can see how the shell script works for more information but basically I do two things:

1. Pull down veewee
2. Symlink /definitions and /iso into veewee from shared locations

Now that we’ve got a repeatable, sharable dev/vagrant setup, we can now work on integrating it with all of our projects!

Having just jumped on board to a newish Django project here at work, I feel their pain. Although I don’t know the Django ecosystem as well as Rails, I do know that projects should contain a readme, and I shouldn’t have to:

1. execute server runner
2. install missing lib or package that it complained about
3. goto: 1 until working

Every development project, no matter how complex, should have a repeatable, predictable (preferably scripted) new developer project bootstrap.

Why?

• New developers can get their machine(s) setup and commit/deploy on the first day
• Existing developers can easily upgrade to new machines without spending days getting it all setup again
• Light-duty hackers (support folks) can have a copy of the site running locally (super awesome for debugging and general playing around)
• Designers can tweak styling, implement new designs, etc. without needing to involve a developer
• Spinning up throwaway testing/dev servers becomes very easy (you can treat them like a dev box)

A repeatable, predictable project bootstrap should, minimally, consist of the following:

• A root-level README that explains exactly how to setup the project. This should contain step-by-step tasks (start with git clone ...)
• A script to install pre-requisites (likely one for OSX, Linux)
• A script to bootstrap their data storage (build the structure, load default users, etc.)
• A script to deploy to staging/production

Root level README

This is your project’s bible. It’s in the project because web pages, wikis, and spec docs all have a way of disappearing.

The README shouldn’t contain any marketing speak about the project. It also shouldn’t contain any architectural documents, the history of the company, or anything like that. It’s simply a list of steps that you need to perform to get the app up and running on your machine, depending on your operating system.

This is the first thing a new person on the project will read. As part of a new hire’s “orientation” you should create a task in your tracker to “Follow the README”. Sometimes, but not often, some change may have invalidated the README. Part of the new hire’s job will be to fix the process to reflect the new project state.

It should also be part of standard operating procedure to follow the README when setting up a new developer machine (as old ones are swapped out) and any bugs immediately addressed.

Like code comments, broken or out-of-date project bootstraps are worse than none at all.

The README should also include any relevant links to the project wiki, bug tracker, project management, chat rooms, etc.

It should be opinionated - if you’re on ruby pick rvm or rbenv. If you’re on Python/Django, have a virtual environment. OSX? Homebrew.

This is not a negotiation - some level of consistency and standardization is a good thing. Don’t get carried away: let developers choose their own text editor, browser, etc. but project-level standards are expected to be enforced.

My README template is (loosely):

• External Links (wiki, bug tracker, chat)
• Dev setup (assuming a fresh machine - so this may include installing git, homebrew)
• External dependencies and versioning (if applicable)
• Project conventions that deviate from the norm (if you use some weird test runner…)

Scripts

Where at all possible - script script script! Yes, you could tell the developer the series of commands to create a new user in the database (I’ve seen raw SQL insert statements in a README before! Eeek!) but it’s far better to run ./setup_db.

Obviously a script introduces overhead but a good team, over time, instinctively seems to find a good balance of scriptability vs. maintenance.

Deployability

A hallmark of a great engineering process/culture has new hires deploying on their first day. This is only possible if the deploy process is scripted. The README should say how to deploy to the various environment(s) and if there are any externalities to resolve (ssh keys).

Edit - May 4, 2012

Some folks have suggested Vagrant + Chef or Puppet as a solution to the problem. I haven’t used it at all, but it certainly looks promising. I’d love to build a Puppet-based config for my local machine (we use Puppet-based Moonshine on our servers, so I’m not entirely sure why I didn’t think of it) and I suspect that’s what I’ll try first.

My only hesitation is that I primarily use a MacBook Air and, although a perfectly capable development machine, running Ubuntu on VMWare is intolerably slow. Benchmarking is probably a task for another day.

Oh, the shock and horror. The wailing and gnashing of teeth.

Why is this news? Why do folks think that working gobs and gobs of hours, as software developers, leads to better software, faster?

Sure, for something repetitive and methodological like bricklaying, you can crank up the hours worked dial and see the wall getting built faster. And, sure, there’ll be small bugs here and there (sloppiness as the worker gets tired, etc.) but generally nothing that will materially affect the structural integrity of the wall.

Not so in software development, however. It’s been my experience that productivity falls off a cliff not far after 40 hours a week, and quickly goes negative as developers start adding more bugs than features.

Pretty soon, however, the bugs are not small, easily cleaned up after ones, but those that will cause you to go massively in debt. These kinds of wrong architectural decisions when uncovered days/weeks/months later will bring the whole project to a halt while the extent of the damage is estimated and then repaired.

It kinda looks something like this (completely made up but feels approximately correct):

That’s why one metric I have software developers track per week is whether or not they feel they are on a sustainable pace. Hours are a reasonable approximation of sustainable pace (e.g. on average, don’t work much more than 40 hrs/week) but not sufficient. Frequent check-ins are crucial to establish what “sustainable pace” means for each individual software developer.

Given that we also follow the “Results Only Work Environment” total number of hours worked doesn’t really mean anything anyway. With ROWE, you work wherever you want, whenever you want. Every meeting is optional.

All that matters is:

1. Are you getting your job done, awesomely?
2. Are you, individually, working at a sustainable pace?

How often does this happen where you work:

1. Developer begs, pleads with boss to go to technical conference
2. Boss negotiates with developer on getting a cheaper hotel. (It’s only $69 a night! Clean sheets optional. 30 minute bus ride from conference. What a steal!)
3. Boss orders developer to get cheaper, three stop flight instead of more expensive non-stop
4. Developer attends conference
5. Developer returns from conference with a pile of low value receipts ($20 for lunch, $40 for cab to airport)
6. Developer spends two hours painstakingly scanning each receipt and constructs a spreadsheet with itemized list of charges
7. Boss approves charges without even looking at them
8. goto 1

How brain-dead is this? Here you have a highly-paid software developer doing super-low value data entry. It probably cost you $200 to have them enter all this junk.

What does this tell the developer?

• I don’t trust you
• Scanning receipts is more valuable than writing awesome code
• Your time is not particularly valuable
• I don’t trust you

Give the developer a corp Amex, ask them to be a responsible adult, and be done with it. Have them forward receipts (where applicable) to a specific email address to satisfy potential IRS complications. The end.

Anything else is over-optimization of a low-probability event (being audited and needing to justify a latte at the airport).

Over the last couple of months of casual conversation over drinks and dinners, I’ve discovered the two co-founders, Ismail Colak and Jean-Marc Freuler, share my passion for building insanely great products by hiring the smartest, best developers, treating them like rock stars, and then getting out of their way.

That also means I’ll be leaving Toura, a place I’ve called home for the past two years. I leave knowing we’ve built an amazing team, awesome products, and have made our dent in the world. I expect them to continue to do great things and wish them all the best.

P.S. We’re hiring! Looking for an amazing workplace that is run by folks that understand and value software developers? It’s my job to make Funding Gates the best place that you could ever work, and I hope you’ll check us out!

P.P.S I’ll be in Phoenix, AZ April 1-4 at JSConf, followed by RailsConf April 23-25 in Austin, TX. Come talk with me about software development, good beer, or anything else that comes to mind. I’m buying!

“There are a million ways to lose a work day, but not even a single way to get one back.”

“How does a large software project get to be one year late? Answer: One day at a time!”

Apple Store Upper West Side

It’s trendy right now to produce office space that looks like an Apple store. Bright, airy, cheerful, lots of open space, big ceilings, blonde wood, attractive and friendly employees. I can’t blame the urge - I love seeing the latest architectural masterpieces the geniuses at Apple routinely cook up. Living in New York I have the opportunity of being a short hop away from no less than five of them.

Apple retail stores invoke the sprits of great cathedrals of the 14th century or some vast ancient library. If you’ve ever stepped foot into one, however, the library or cathedral-like comparison vanishes into the cacophony of hundreds of conversations, various iTunes-sponsored music clips, and the ringing of countless iPhones. Often when I walk into one of these stores I’ll put on my noise-isolating earbuds - not to listen to music, but to act as earplugs.

So why is it that companies - small and large, startup and established player - continually take high-pay, high-skilled, super-smart programmers and cram them into giant rooms full of noise and distraction?⁹

Examples are everywhere: AppNexus - Klout - Kiva - ZeroCater - JustinTV - PublicStuff

Things you’ll notice in virtually all of these include: folks talking on phones, people listening to headphones, small groups standing and talking, and the deafening sound of thousands of dollars of productivity being sucked out of the room.

Go to any of these offices and ask each programmer: how often are you distracted? Do you ever find yourself drawn into conversations that you aren’t a part of?

I interviewed at a company that had a giant office (imagine a hundred folks in one room) and one of their engineers proudly said: “We have a generous work from home policy. I love it because I can never get any work done around here!”

Flow

Programmers work in a state of “flow”¹ - it takes a tremendous amount of mental wrangling to page in to memory all the data required to comprehend the micro and the macro of software systems. Recalling the object hierarchy, database schema, or GUI architecture for a complex system is akin to building a house of cards. One slight bump and the whole thing comes tumbling down.

Some estimates have suggested it takes 15-30 minutes or more² to get in the zone³. What is less-known is how long it takes to re-start flow once you’re knocked out. Given an average NYC fully-loaded salary, a simple email “Ding!” or an answer to a “quick question” could cost a company $50.

Of course, it is too simplistic a measure and vastly underestimates the true cost of distraction⁴; many larger features or architectural decision-making requires continual effort to maintain progress. A few interruptions strategically placed through the course of a day could make a four-hour task take two days.

Sure, large companies are so full of inefficiencies and waste that the lack of agility won’t hurt them. But startups? If your programmers are less efficient than the competition that could mean you launch months late. That you can’t iterate as quickly as you need. And that next amazing idea - that idea that fundamentally transforms your company and changes the world - simply doesn’t have the time and focus to exist. It’s lost in the noise.

Solutions

The evidence is pretty clear that massive open-plan offices where the entire office is colocated are terrible interruptors of programmers⁸. At the the other extreme we can place programmers in private offices (with a door that actually closes!) to keep them isolated from the noise and movement of the office.

DeMarco and Lister, via Peopleware⁷, argue for private offices for each developer because most software development is done in isolation.

Communal Workshop

“Agile” methodology, on the other hand, states that communication is the most important factor and that private offices destroy communication¹². Oram and Wilson, via “Making Software: What Really Works and Why We Believe It” argue very strongly that the Communal Workshop (smaller units of open-plan with, say, six or seven programmers in their own room) layout is the most efficient for teams that require lots of collaboration and have loose requirements.[Oram and Wilson, pg. 346]¹²

As a counterpoint, Joel Spolsky suggests that communication isn’t harmed via private offices.^{10 11}

There are valid points to each solution but I’m not aware of any hard data that can conclusively determine the “victor”, if there is such a thing.

Regardless of the layout, here are some rules that must be enforced:

Create a quiet workplace free of unnecessary distractions

Some distractions, like going to lunch or pair programming, are necessary. Most distractions, like ringing cell phones, conversations not involving the team, eating lunch, and playing music are unnecessary and disruptive.⁵

Keep ambient noise to a minimum

At our office, we have a shared space where the developers work and everyone else is in a separate location. We treat the programmer space as we would a library. No irrelevant conversations⁶, don’t bother anyone, stick to IM and IRC whenever possible. Meetings either are team-wide (standups), so can take place in the shared space, or individual, and are taken out of the room.

Leverage Technology

We use group chat with logging, private instant messaging, wikis, blogs, etc. to keep general chit-chat down. The side effect, too, is that two programmers can have a discussion in a chat room and any passive bystander can chime in if they feel like. If they’re busy or not in the office, they should browse the chat log upon returning to work. As they catch up, they can re-engage the other programmers if they choose.

No meetings

The more meetings a programmer has the less time they have to work. We only have one meeting, our daily standup - everything else is ad hoc and optional.

Conclusion

With a little bit of time, thought, and effort, we can have office space that is much more enjoyable, productive, and focused on programmer happiness.

Footnotes

• Discuss ways to build better software
• How to hire, retain, and keep awesome software developers happy
• General this-and-that interests me