The Joel Test Anno 2017

by Rolf Smit | 25 September 2017 |

You might have heard of the Joel test (depending on your background). As a regular Stack Overflow user and development blog reader I sure have heard of it. I think it’s great, even 17 years after it was invented. The Joel test is a test of 12 questions that a software development team can answer to determinate its quality. The author of this test Joel Spolsky (CEO of Stack Overflow) calls the test in his own words: “The neat thing about The Joel Test is that it’s easy to get a quick yes or no to each question. You don’t have to figure out lines-of-code-per-day or average-bugs-per-inflection-point.” If you do not know what the Joel test is you should familiarize yourself with it directly, follow this link to read the original publication by Joel himself.

The Joel test questions:

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?


Nowadays more and more people consider the Joel test outdated and many have tried to revise the 12 key questions, some because they think the set of 12 questions does not hold-up against agile methodologies and others because anno 2017 we use different technologies and tools to manage software development, in my eyes a big misconception. Even though the last is definitely true, I think most, if not all, of the questions have withstand the ravages of time. But on the other hand I do think the test could use at least the question “do you write unit tests?”. But nevertheless I hope that more people in the software business will start to use the test (hence this blog post). Because I believe that the test actually is a great way to determinate the general development team quality. Indeed a general measurement, people should stop thinking that the questions and the result of the test is something that should be considered set in stone (that would be irresponsible) or something that measures the quality of your software. So let’s first address those misconceptions when it comes to the Joel test:

1. Some questions are too trivial, everyone will answer them with a firm “yes”

Some consider this to be true. But take for example the first question: “Do you use source control?” too trivial because anno 2017 everyone uses source control right? Well, even though it has improved over the years, this is unfortunately not true. And I’m not talking about the few left over teams that still don’t use source control I’m talking about a fundamentally different problem here. The point is that many development teams are using source control in a suboptimal way. Even though I’m young, I have seen too many software projects use a single branch in source control or even worse; projects that depend on other “single branch projects” by compiling against the latest commits. This leads to nightmare situations (I really hope you understand why). In other words both version control and branching strategy are fundamental source control principles. But what about software configurations, do you have source control for those? Next time you answer this or other Joel test questions, try to think of the Joel test as a best case scenario, answer it with “yes” only if you at least try to follow best practices.

2. We do agile we don’t need a bug database!

I once read something in the lines of “We do agile, we don’t need a bug database”. But even if you are using an agile software methodology it’s still a good idea to have a bug database. A backlog simply does not get you the benefits of a full fledged bug database. A bug database is important to keep track of known issues and bugs solved in the past, historical data and searchability are key here. There are tools out there that have merged bug tracking and backlog management into a single user interface, which is fine. But sometimes it can still be a good idea to have a separate bug database as a backlog was never really meant to be your bug database.

3. Thinking of the test as a business measurement.

This is where my opinion and personal experience comes into play. A simple but big misconception that I heard too many times is that what should happen is always based on business needs, so whether a bug needs fixing is a business decision. Obviously this is true and business needs are important, but don’t shoot yourself in the feet. In the end bugs can be complex and building on top of existing bugs often leads to even more complex situations. In some cases bugs require some architecture rethinking in order to fix them. Let developers prioritize bugs based on technical impact. Bugs with a low technical impact don’t need to be fixed directly from a technical point of view. On the other hand bugs with a high technical impact sure do, even if they don’t impact your business directly they are very likely to do so at some point in the future. But even more important and the key statement I want to make here is that the Joel test is a test to determinate the development team quality not the quality of your business as a whole.

There are probably more misconceptions, but I just had to address these. I might add some more misconceptions in the near future.

Improving the Joel test

Just like so many people I do think the Joel test could use some modern day improvements, but I don’t think we should rewrite each and every question like so many others have done or make the questions more specific. I think we should not do this because the test is obviously meant to be a quick and general tool of measurement, unlike so many other tests and tools to measure quality which are way more complicated. But if I could improve one thing, it be the following:

13. Do you write unit-tests?

Why? Because unit-tests are the fundamental principle of good quality software. But let’s be critical here, does this question really fit within the test, does it really improve it? After all this is something the programmer should do anyway, right? Yes, programmers should write unit-tests, but sometimes the business does not allow them to do so, because “time”. While this might be fine for proof-of-concept applications, for a production application programmers and actually everyone else should know better. Having unit-tests in place avoids many bugs and thus saves the business (at some point) time and the programmers a headache. So having the ability to write unit-tests as a programmer is something that is good for the development team in the context of quality and could in my opinion very well be part of the Joel test. Anno 2017 we are still developing applications and writing the unit-test after we find out it’s full of bugs and I think this will (unfortunately) continue for many years to come.

Summing it all up

The biggest Joel test misconceptions are probably; answering the questions to literally and seeing the Joel test as a measure of performance for your business, it is not! Answering the questions too literally is something you should not do, the questions should (at least in my eyes) be answered from a best practice perspective. To give an example: using source control and using it in an effective way are two completely different things. But the most important lesson is that the Joel test is a way to determinate the quality and health of your software development team in a quick and easy way, not a measure of performance for your business. Which also leads me to my last conclusion, please stop making the Joel test more complicated than it should be, keep it simple and quick, as it was meant to be. But most importantly try it, use it.