Software Development Still Sucks
from the it's-not-pretty dept
We’ve discussed the Mythical Man-Month concept, where adding more developers to an already late software project tends to slow it down even more. It’s an important concept that’s often quoted, but not often followed by management. It appears that Scott Rosenberg has written up a new book that picks up where the Mythical Man-Month left off, looking at how software development is still a complete mess. Specifically, he focuses on Mitch Kapor’s Chandler Project, which has taken years to not get very far. However, the point is that almost all software development seems to be an incredibly messy and unorganized process — leading to all sorts of problems both in planning and in implementation of any particular software product. Basically, as much as it’s called “computer science,” it really isn’t much of a science. It’s hardly even a process. I recently had a chat with someone who had switched jobs from one big company with a reputation for extremely complex and buggy software to one that had a reputation for very clean software. He talked about how he thought he’d learn so much at the new company, only to realize that the development process there was just as messy (in some cases, even messier). Rosenberg makes a few interesting points, about how hardware improvements have followed Moore’s Law, while it seems like software has almost been dragged along by accident — though, perhaps a more accurate way of looking at things is that the hardware improvements of Moore’s Law have actually allowed us to hide the problems of software development by simply throwing more horsepower at things. No matter what, it sounds like an interesting book that will probably ring true to many software developers.
Comments on “Software Development Still Sucks”
Processes are ignored
Basically, as much as it’s called “computer science,” it really isn’t much of a science. It’s hardly even a process.
I love that quote, and am going to use it in my class next week.
There are processes in place – the SDLC, etc… but a large part of the problem it too many people are willing to take shortcuts – th eprocess is ignored, we can fix it later.
It is s sad state of affairs.
science vs engineering
“almost all software development seems to be an incredibly messy and unorganized process”
Software engineering has a huge body of great literature behind it now, as a process that is almost 40 years old. The problem is with software engineering management since valuable texts like Sommerville are thrown aside as “adademic” and each management team thinks it knows best about how to reinvent wheels.
“Basically, as much as it’s called “computer science,” it really isn’t much of a science.”
Confusing CS with SE is a common mistake. “Computer science has as much to do with computers as astronomy has to do with telescopes” – I forget the source. Computer science is about fundamental principles, software engineering is about practical solutions to concrete problems.
“..perhaps a more accurate way of looking at things is that the hardware improvements of Moore’s Law have actually allowed us to hide the problems of software development by simply throwing more horsepower at things.”
Absolutely correct. The vast majority of software is bloated beyond comprehension. It starts with the seductive hypothesis that programmers time is expensive but cycles are cheap. With modern development environments in just a few thoughtless keystrokes a naive programmer can include gigabytes of unneeded libraries and create a circus of invisible suboptimal code. Then the complexity explodes. 75% of cost is in the last 25% of the lifecycle, test, debug and maintainance.
While experienced ones..
“… in just a few thoughtless keystrokes a naive programmer can include gigabytes of unneeded libraries and create a circus of invisible suboptimal code.”
Which is the point where an experienced developer [sic] will throw out all of those libraries and roll his own tightly coded, highly optimized, undocumented routines, creating a new circus of buggy suboptimal code.
Then the software, the server, and the company explodes…
Re: While experienced ones..
Yep that’s about the long and short of it Michael. Meanwhile the project manager, who has never written so much as a line of Basic in his whole life, fires the second team brought in to clean up the first teams mess and advertises for new blood to come and redo it all in “AJAX” or whatever the latest newfangleumjimidoo he just read about on the intermerweb.
Meanwhile a 13year old kiddy in Kazakhstan has written a much better implementation of the original idea in Brainfuck which runs on a TI-83 calculator in 2k of RAM, that nobody will ever hear about ouside the demo scene.
Lions led by donkeys.
It's not the code, it's the people
I’ve been in software dev at several well-known, large corporations as well as tiny dev houses no one ever heard of…the source of the “problem” was the same everywhere, the people.
As long as there are people with personalities empowered to make decisions and be creative software dev will always be a mess.
There are different (often better and worse) ways to implement everything, from the language itself to the talent with which it is employed.
Want mess-free software dev? Standardize everything, lobotomies all around, and love the total crap you end with 😉
preemptive deadlines
A lot of the mess is caused by the sales, marketing or Customer relationship teams that often promise something vague for a very fixed date. How often do we hear ‘we don’t know what exactly we what to do, but it needs to be done by xxx’.
'Software Development' is not 'Computer Science'
Software development is an implementation of Computer Science, it isn’t ‘Computer Science’ itself.
‘Computer science has as much to do with computers as astronomy has to do with telescopes” – Dijstrka
Ummm... wrong.
That’s some of the broadest use of hasty generalization and fallacious thinking I’ve seen in a long time.
Re: Ummm... wrong.
That’s some of the broadest use of hasty generalization and fallacious thinking I’ve seen in a long time.
It’s kinda funny how you do it yourself 🙂
Re: Re: Ummm... wrong.
doing fallacious thinking on your own, so would that be auto-fallacious?
Re: Re: Ummm... wrong.
Explain. The first could not be true, because I made no generalization, unless you are confused by my use of the words “broadest use”. The second cannot be true unless we are both incorrect, which I accept is a possibility.
The true problem....
I think the biggest problem is that we as developer haven’t developed a culture of rigorous internal and external review of algorithms and routines, and that leads to having to “reinvent” things every time a new project comes along.
Look at how medicine does things. If you develop a new procedure or technique for doing something, often you publish your techniques and your peers critique and refine your technique. This technique is then taught to new doctors along with others before it, and the body of tested, proven knowledge grows.
In Software Engineering, however, seems like everyone has their own home brew for doing every thing, and we are too quick to take the “let’s start from a blank page and make everything from scratch” approach. I guess part of this can be attributed to the fact that SE grew up in the world at a time when copyright and patents were king, making it very difficult to do external review. However, has anyone been to an internal code review lately? Seems to me that most of the ones I’ve been in degenerate into this “your member variable needs to be named this, or your comment needs to say this”. I think we would be better served to concentrate on the fundamental algorithms implemented (at code reviews) and also attempting to enforce re-use of code more often as a 1st line of development.
Basically, what I’m saying is that SE needs to develop an “institutional memory” of proven techniques to address problems that have been critically reviewed and accepted as standard. These techniques need to be taught by students and as more improved ones come along [and are peer reviewed] incorporated into the body of knowledge.
Re: The true problem....
I totally agree with you, Tyshaun. If only it could exist in universal form. There are small communities where you can share techniques and critique each other BUT nothing on full scale.
Re: The true problem....
I have dealt with this issue and have solved the problem. My solution was to never write applications, but instead write application generators. Then I would point the generator at a database of customer information and another database of generation instructions and BAM!! I had a finished application. If there was a mistake in the code or if the customer changed his mind about what he wanted, I would change the generator or the instruction database regenerate the application. Then all the mistakes would disappear and the customers changes would ripple through the entire system. The trick to making it all possible was getting the generator to remember hand coded custom changes. With only myself on the staff of programmers, I could build and maintain applications with a hundreds of tables and custom windows. It never caught on with many other programmers because programmers bill by the hour and because most of them didn’t want to believe that you could capture their knowledge and skill in an expert system.
all I see is the authors name not the name of the book….
Re: Re:
The book is called Dreaming in Code
A House of Clay
I’ve been in QA more (yikes) more than a decade and have been in a few different operations, some famous for quality and some not, some large and some small. There are two fundamental problems.
1) The software world has not really come up with the functional equivalent of a “blue print,” a straightforward diagram speaking to the customer (whoever determines what the software should do), the construction workers, and the architects. So, a customer might say they want the software equivalent of a “2000 sq.ft. house with 14 windows and 2 doors” but they won’t be able to really envision it until its built. And, if the development team is bigger than 3 engineers or has to “prefabricate the frame” before it is assembled, then the “support beams” might not actually connect properly.
2) Code is maleable like clay. It is easier and, occassionally, less time consuming to simply build the shapes of the code and then press it together than it is to create an accurate, detailed design.
Hence we build a house of clay. The word science is not really a question…its more the word engineer that I question.
The problem comes down to...
Ego.
The ego of management, coupled with the ego of the ‘star’ developers.
Software Development & Moore's Law
It is misleading to claim that hardware is improving according to Moore’s Law while software is not. Look at how most of the transistors are used on a typical present-day CPU chip: cache memory consists of a single cache line circuit replicated thousands of times; the circuit was only designed once, yet every one of those replications counts towards Moore’s Law. Similarly multiple function units for doing addition, multiplication, division etc–modern CPUs tend to have several of these, which are all identical, yet again they add to the total transistor count.
In software, we are not allowed to count such multiple repetitions of the same code. But if we did, you could easily come to the conclusion that software productivity is way ahead of hardware productivity.
My Opinion is:
Tyshaun, I utterly agree with you.
misanthropic humanist, “Lions led by donkeys.” is one of my favorite qoutes too. It really is a perfect description for this topic. 😛
Chandler
I had a pretty good hunch that this project would be vaporware when it was announced, because it seemed to have the following goals:
– establish a benchmark for how great software can be created
– create software that can be used and appreciated by ordinary folks, not just corporations
– provide a showcase and platform for open source and free software
– keep up with the latest fads, and start new ones, so the project and its principals would have buzz and be in constant demand at parties
– set Microsoft back on its heels and maybe even lower its stock price
All this is nice (especially the last), but it has little to do with making and keeping customers and growing a business. Kapor did an outstanding job launching Lotus, but after that his record wasn’t so good, in spite of all his intelligence, user interface savvy and people smarts. IIRC ON Technology was originally launched as some sort of generalized services platform on top of the desktop operating system; they hired a bunch of “rock star” types to flesh out the vision. After operating 2-3 years in stealth mode and having little to show, the company changed direction and decided to commercialize a calendar tool they’d developed called Meeting Maker. Even that took a lot longer than expected; I remember a long WSJ article written by a reporter who was given access, that probably was a short version of Rosenberg’s book.
Then there was GO Corporation, the pen computing startup that Kapor founded with Jerry Kaplan. In fairness, that idea was simply ahead of its time.
The true problem....
Have a look at Plone John, I think you would enjoy that.
Within a limited domain application factories are a good idea. If you can solve any problem just once and do it correctly the rewards are huge. The only downsides are:
1) Absolute correctness. You need to invest at least 5 times the development time in making sure your thing factory works, otherwise every single application you generate is flawed, a position far worse than cocking up only one clients code. Given that ratio you probably want to reuse your code generator on at least 10 or 20 projects.
2) Flexibility/brittleness. Can you knock up a medical spectrographic analysis tool that uses discrete wavelet transforms using your framework? I doubt it very much if you primarily designed it to create e-commerce solutions.
You definitely read those other programmers wrong. All programmers are lazy, it is a singular virtue, and they would welcome any enabling tool that reduces the number of lines of code written.
Re: The true problem....
Hi M.H.
Thank you for your thoughts. M.H. Wrote: 1) Absolute correctness. You need to invest at least 5 times the development time in making sure your thing factory works, otherwise every single application you generate is flawed, a position far worse than cocking up only one clients code. John’s Answer: Once you get used to the idea of never writing applications but rather the generators instead, development is just as fast. And if you do find a mistake, all you need to do is fix it in the generator and regenerate an app for all your customers rather than digging through the code of each customer. Also, there are rarely mistakes because the generator does all the typing. What I do is capture the technical specifications of the customer in an expert system rather than on paper. This allows me to do all kinds of “what if we do it this way” experiments with the customer. I can let the customer play and discover what he really wants rather than trying to lock him down with a static technical specification that he must sign off on. So the customer gets zero typing errors, fast development, and he can change his mind about what he wants or need all at no extra cost to me.
M.H wrote: 2) Flexibility/brittleness. Can you knock up a medical spectrographic analysis tool that uses discrete wavelet transforms using your framework? I doubt it very much if you primarily designed it to create e-commerce solutions. John’s answer: Of course the math routines would have to be coded by hand but presentation, data access, security and so on is quite common and doing all that by hand and then chasing down the bugs and then changing it all by hand because your customer really wants a different look and feel takes so much time away from the important fun and creative work that developers should be doing. I have found that software development doesn’t have to suck. The key for me was chosing to see what’s common in about what we do and passing those jobs off to a computer.
THANK YOU MIKE!!!
(disclaimer- My comments below pertain to Mike’s original post and not the chatter of comments afterward.)
*THIS* is the kind of stuff that makes me still subscribe to Techdirt. Insightful commentary — in this case, a book review — based on information and analysis, and not drama, or BPM (bitching, pissing and moaning). I’ve commented before that Techdirt’s signal-to-noise ratio has too much NOISE. THANK YOU THANK YOU THANK YOU for proving that you guys still have signal strength.
(Sorry, this doesn’t contribute much to the discussion, but I think while it’s useful to try and point out when things are going wrong, it’s equally important to cheer loudly when things go right.)
There are a number of basic problems I have seen in my 17 years as a professional software developer.
First, most people actually writing code have little or no training in how to do software development. Sure they have had a few, or even a lot of programming classes, but these classes almost never teach anything other then language syntax. This includes many CS majors I have meet who have no clue about gathering requirements, revision control, test plans, etc.
Second, speed is more important then quality. Getting things out the door as fast as possible ALWAYS trumps requirements gathering, complete testing, robust error handling, proper use of revision control, etc. In 17 years I have yet to work on a project where this was not the case. Because these quality controls are not done along the way something ALWAYS gets screwed up, thus you get more bugs and never finish in time.
Third, lost cost is more important then quality. The bottom line controls everything. No matter how much something actually costs management ALWAYS wants to spend less. This is why most projects end up overbudget. Penny pinching leads to cost overruns in the long term.
Finally, management is self serving and has no backbone. Managers main goal seems to be to make themselves look good in the eyes of the higher ups. There goal is NOT quality software. Quality does not make them look good. It actually makes them look bad. Low projected budgets, aggressive timetables, etc are what make them look good. SO even if all the expert developers under a manager say something will take X amount of time and Y amount of money the manager will always back down when those above them say it needs to be done faster and cheaper.
None of the above is unique to just software, I’ve seen it on every hardware project as well. The real core problem is not really managing projects the way they should be.
Re: Problems No Unique to Software
Hi, John
I feel that writing generators solves problems 1-3 that you have presented, which were developer training, speed of development, and cost.
I feel your comment that the problem crosses over to other disciplines is very insightful. I disagree with your assessment of what the actual problem with managers is. In my opinion, managers respond to incentive just like everyone else. They have a room full of programmers and they get a certain amount of money for each hour of development that each one of them does. Yes, they can reduce their costs if they can do the job with less programmers and in less time, but they can’t bill as much either. Since the customer pays the cost, incentive causes managers and programmers alike to believe and sell the idea that only people can program. So as long as industry spokesmen (all of us) can convince ourselves and the public that this is true, then programming will continue to suck and we will all continue to profit from that notion. No disrespect intended; it is human nature to honestly believe what it is profitable believe in.
So what?
…perhaps a more accurate way of looking at things is that the hardware improvements of Moore’s Law have actually allowed us to hide the problems of software development by simply throwing more horsepower at things.
So what? My truck is not very efficient at getting me to work every day either, and the engineering required to build it was hundreds of years in the making.
Software (and its development) has revolutionized faster than anything in recorded history. Is the process messy? yeah. so what?
Schrodingers truck
“Software (and its development) has revolutionized faster than anything in recorded history. Is the process messy? yeah. so what?”
Interesting attitude BOF, I kinda like it and I could follow that reasoning up to a point. I mean computers are only going to get faster and faster right. If it achieves its goals who cares?
Here’s where it breaks down. The problem with the truck analogy is that there is no strange hidden condition where if you pull out the throttle, with the gear stick in reverse, but the steering wheel at exactly 15 degrees while there is exactly 33% fuel in the tank…the entire fucking thing just explodes and kills everyone.
For all its inefficiencies it wont fail spectacularly and catastrophically for some arcane reason.
That’s exactly what software does once its complexity goes above a certain point.
There is only one thing that can save us now…
WE NEED A COMMITTEE!
Software Engineering
Software programs are becomming more and more complex. The reason that they are considered a “mess” is because many of them conver new ground. Science is a good name for it, because it is as much about discovery as it is about process.
With 700 page books being commonplace in the industry, that go out of date in a matter of months, it is no wonder that things in development get a little chaotic.
There is nothing like software development, to compare it to anything else doesn’t make sense. For example, above it was compared with “hardware” that keeps getting faster. Never mind that the hardware really hasn’t changed that much, just gotten faster. I don’t think many people would be lined up to buy the latest copy of Word if it was just “faster.” No they want it “simpler,” and paradoxically “with more features.”
There are literally thousands of software development methodologies. Most companies try and follow one. When you hear someone say there company doesn’t have one, chances are they do, but that particular programmer doesn’t like they one they have.
So give software engineers a break. If there were a better way, they would take it.
The Real Reason
The real reason software is so bad is that programming /
coding is fun! Planning isn’t. I agree “programmers are lazy”.
After years of fun I learned that the easiest way to get a job done is to do it right the first time. Of course that requires a
decent set of firm specs ( not a trivial requirement ).
I have been involved in several large projects ( 15 or more programmers ) that were very successful ( no reported bugs
after years in thousands of installations ). That was accomplished by months of planning before any code was written. The code was written in about 5% of the total time. The only problems found in Quality testing were due to coding typos. Wasn’t fun, but fixing bugs for irate customers isn’t fun either!
Wowee
I had no idea (until this article) that planning wasn’t taught with programming courses. Having taken Interior Design/Architecture courses, (and flow-charting in basic high school computer classes) I can’t understand how someone could sit down to write software or develop an application without [a plan] knowing where one wants to go with it. (needs, requirements, etc.)
I also didn’t know that IT Managers or Project Managers were ever hired without so much as writing one line of Basic. I’ve looked over the requirements for these positions, being a project coordinator myself, and what they want would require the years of experience that you guys have!
(I do know about managerial agendas.)
Thanks for the eye-opener.
Software Development Still Sucks
Hi This is MAG Studios i found Your Post so much Useful and Interesting and it’s helpful for Software Development and for the person who Connected to Software Thanks.
This!
“There are a number of basic problems I have seen in my 17 years as a professional software developer.
First, most people actually writing code have little or no training in how to do software development. Sure they have had a few, or even a lot of programming classes, but these classes almost never teach anything other then language syntax. This includes many CS majors I have meet who have no clue about gathering requirements, revision control, test plans, etc.
Second, speed is more important then quality. Getting things out the door as fast as possible ALWAYS trumps requirements gathering, complete testing, robust error handling, proper use of revision control, etc. In 17 years I have yet to work on a project where this was not the case. Because these quality controls are not done along the way something ALWAYS gets screwed up, thus you get more bugs and never finish in time.
Third, lost cost is more important then quality. The bottom line controls everything. No matter how much something actually costs management ALWAYS wants to spend less. This is why most projects end up overbudget. Penny pinching leads to cost overruns in the long term.
Finally, management is self serving and has no backbone. Managers main goal seems to be to make themselves look good in the eyes of the higher ups. There goal is NOT quality software. Quality does not make them look good. It actually makes them look bad. Low projected budgets, aggressive timetables, etc are what make them look good. SO even if all the expert developers under a manager say something will take X amount of time and Y amount of money the manager will always back down when those above them say it needs to be done faster and cheaper.
None of the above is unique to just software, I’ve seen it on every hardware project as well. The real core problem is not really managing projects the way they should be.”
This, this, and this! Did I say this?!
Nailed it! You Sir, just stated cleanly and succinctly why I am leaving this good-for-nothing God-forsaken so-called ‘profession’.
“Make those good-for-nothing code apes work faster so we can invoice, or else we’ll outsource them all”: your bosses’ boss. He has an MBA, don’t-you-know.
Software Development & Moore's Law
Moore’s law has nothing do do with actual transistor count, but that the transistor count per sq inch will double every 18 months.
It is very exciting to read this article now. You are right, even in today’s reality, creating software is not such an easy task. Although most developer companies such as https://www.timesolv.com/ I have long optimized the process of creating high-quality programs, there are still difficulties. The process of creating a new program is very complicated. This is a whole business process to which you need to approach very seriously and consistently