In university, we do a lot of waterfall in courses with project work. It isn’t the kind of thing a student would do (to) themselves, so professors feel obligated to give us that experience in class. Research shows that both business and recent graduates wish they’d been taught agile development methodologies in university and college, but course content always lags behind.
At Pythian, we used a mostly-Scrum methodology, with all the benefits and challenges that entails.
One common challenge is that of estimation and timeboxing. I have almost no experience with estimating real-world-scale projects, so I would expect to be wildly wrong in my estimates, but I was surprised that more experienced developers are also often very wrong. My favourite quip on this is about “the second 90%” of a project. The little niggling details you didn’t get to before being reassigned to a different task really add up, and can push delivery back weeks or months, depending on how priorities shift.
Timeboxing is also difficult. Sprints were sometimes full-to-overflowing, and developers wouldn’t be able to clear their task list by the end of a sprint. Hours seemed to total up correctly, but something else was keeping people from finishing everything in the sprint.
As a recovering psychology student, these are really fascinating problems. Why is it that eight hours of work don’t get done in an eight hour work day? What is it about the total time estimate for many small tasks that’s so different from the total time estimate for fewer larger tasks? Why does task switching wreak such havoc with time estimations? How can we account for these when managing developer effort?
Equally interesting are failure rates. I have a software engineering professor who points out that about 25% of software projects fail to deliver anything. But aren’t 25% of ideas crap? And crap ideas should fail! And is 25% really too high? Compare that to new companies, where something like half of them fail. Maybe software engineering is doing well to have such a low failure rate compared to producing physical products! We throw around numbers without examining our assumptions. (We sometimes don’t even examine our data!!)
I don’t have the answers, but I do have some questions. It is disappointing that there is consistently a focus on studying programs, but never studying programmers or programming. I hope this turns out to be a parochial view. As disappointing as it is that my university doesn’t take software engineering research seriously, it’d be worse if this were the case more broadly.
These research questions are hard - but we have an entire discipline of science devoted to questions of that sort. Cross-discipline co-operation is needed to get results, so I expect we’ll be waiting a while.