Watching “agile” move from a common-sense philosophy to an organized, detailed practice

A long time ago in a galaxy far far away — before Google, before the PC, before the widespread non-academic/military use of the Internet, perhaps even before the invention of fire and electricity — my colleagues and I were bemused and intrigued by the reaction we got from veteran software developers and project managers when we wandered all over the world (including some pretty scary places where programmers brought machine guns to work), preaching from every soap-box we could find about the benefits of structured programming, structured design, and structured analysis. During my week at Agile2010 in Orlando, I was equally bemused and intrigued to hear some of the same reactions from people when they listened to the agilistas preaching about the benefits of agile, scrum, XP, lean, and related concepts.

The first reaction that took us by surprise, back in the 70s, was the innocent and sincere statement from consulting clients, students in our classes, and people listening to our lectures, along the lines of, “Well, that’s all just common sense, isn’t it? Why are you making such a big deal out of it? Why do you have to use all these funny words to describe plain old stuff?” Our first instinct was to repeat a statement often attributed to Will Rogers and/or Voltaire: “Common sense isn’t common.” And sometimes we would say, “Well, if it’s common sense, then why are all of the systems we’ve looked at so incredibly badly specified, designed, and implemented?”

That often led to a second reaction, along the lines of: “Well, this ‘structured stuff’ you’re promoting is really just a philosophy, isn’t it? Heck, who could argue against a philosophy of breaking a system into small pieces?” A variation on this theme was the veteran programmers who said, “Aren’t you just talking about modularity and modular programming? Heck, we’ve been doing that since the 50s!” And my colleagues and I would often respond, “That’s like saying you don’t need detailed specific laws and regulations, because everyone agrees with the philosophy of the Golden Rule: do unto others as you would have them do unto you.” And we would then point them to books we had written, with hundreds of pages of specific guidelines, practices, do’s and don’ts.

I have to admit that when I first started using the “agile” word in discussions with my clients, students, and fellow IT professionals, I tended to express it more as a philosophy than a concrete set of practices. Partly, that was because even the philosophy sounded radical a couple decades ago: in a world dominated by the waterfall development cycle, the suggestion that perhaps you could develop systems in an iterative fashion, and deliver working prototypes to the business user at frequent intervals seemed … well, it seemed downright crazy. (But if you track down any of the out-of-print books like my Managing the Structured Techniques and Managing the System Life Cycle from the late 70s and early 80s, you’ll see that that was indeed the philosophy that I was advocating, along with most of my equally-crazy friends and colleagues.)

But it wasn’t until I stumbled onto Kent Beck’s work, and especially his Extreme Programming books from the late 90s, that I began to appreciate that there was a lot of codified practice supporting the old iterative philosophy. Things like pair programming, continuous refactoring, test-driven design, colocation of developers and users, and collective ownership of code were all things that made perfectly good sense … and things that, at least in a couple of cases, I could honestly say I had been aware of, and had probably been practicing in some kind of informal fashion.

But that’s what our students and clients used to say to us, too, back in the 70s. “Well, we sort of do that structured stuff you’ve been talking about … well, at least some of it, some of the time.” So, for example, we would find that they had read the first couple chapters of Tom DeMarco‘s Structured Analysis and System Specification book, and thought that data flow diagrams were really cool. But their DFD’s were riddled with technical errors, and violated many of the specific guidelines we had explained (e.g., the “infinite sink” bubble in a DFD, which has multiple inputs, but no outputs). And it turned out that they were only drawing DFD’s, without the additional support of entity-relationship diagrams, data dictionaries, or detailed mini-specs to describe the requirements of each bottom-level bubble.

I met a lot of people at Agile2010 who described themselves as agile/scrum/XP trainers and “coaches,” and I’m pretty confident that they all do a good job of explaining the principles to their students in various IT organizations, and leading through a variety of workshop exercises. But when they go back to those same IT organizations six months or a year later, the real question is whether they’re really doing agile/scrum/XP, or whether they’re simply mouthing platitudes about the universally-accepted philosophy of delivering more business value, more quickly, to their business customers.

Unfortunately, my colleagues and I often heard a lot of explanations and excuses why our former students and consulting clients really wanted to use structured methods and techniques in their project, but found that they simply couldn’t. Sometimes the reasons were technical (“we’re programming in FORTRAN, so none of your structured stuff will work,”) or (“we’re just doing maintenance work here, and your stuff is all about developing new systems”). And often the reasons were organizational or political: “we’d love to do this stuff you’ve taught us, but our boss is an idiot, and the users won’t talk to us.”

Sadly, I heard some of the same rationales and excuses in Orlando, too; it’s still not a perfect world, and there are still too many people who feel that until it is a perfect world, they can’t put any of the agile/scrum/XP principles into practical use.

One of the biggest problems that my colleagues and I had, in the early days of “structured stuff,” was that not very much had actually been written about the concepts and techniques. There was the famous “GoTo Considered Harmful” paper by Edsger Dijkstra in the March 1968 Communications of the ACM, and a paper by Larry Constantine and two IBM colleagues (Glen Myers and Wayne Stevens) in the July 1974 IBM Systems Journal, and a few others; but it was pretty sparse. Indeed, I think that one of the main reasons that my colleagues and I ultimately succeeded in making “structured stuff” a fairly widely-used systems development approach is that we wrote literally dozens of books (including an edited collection of the really old stuff, called Classics in Software Engineering ), and hundreds of papers and articles. And we did show up at every street corner with our soap box, as well as every chapter meeting of professional computer societies that are now mostly defunct, to preach and evangelize at anyone who would listen to us.

And that’s what is obviously going on in the agile/scrum/XP/lean movement today. I was delighted to see a table at the end of the conference hall, filled with a couple dozen different books on various aspects of managing, planning, and executing agile/scrum/XP/lean projects; a few of them were familiar, but I found lots of information-packed books that I had not read, and that I’ve now downloaded onto my iPad. Similarly, I was delighted to see Jurgen Appelo convert his ambitious list of Top 100 Best Software Engineering Books, Ever (which I barely made, with Death March at position #75) into a new list called (ta-da!) Top 100 Agile Books. And he’s not the only one compiling such lists: Yves Hanoulle has a new list titled simply Agile 2010 Booklist that you should take a look at.

Of course, it takes more than just a pile of books to make something a “practice” or a “discipline” or (dare I use the word?) a “methodology.” I don’t know if it really requires certification, and I don’t want to wade into that debate (which seemed to be only a tangential topic of discussion at Agile2010). But whatever more might need to be done at this point (and I’m sure there’s a long list we can come up with), I’m delighted to add my feeble concurrence to a reality that everyone in Orlando was already aware of: Agile is real. Agile is mainstream. Agile is more than just a simple statement of philosophies (which, I would argue, is what the Agile Manifesto represented to most people in the IT industry).

Having said that, I think it’s also obvious that there’s still a lot of work to do. As long as the Agile Overlords promise never to drag all of us back to a hot, humid place like Orlando and then freeze our butts off with ice-cold air-conditioning, I’ll do my best to attend the annual gathering of the clan from here on. Salt Lake City next summer? Sounds good to me!

curriculum vitae.