Why unit tests are not always good

Unit tests are good to detect most bugs in your code but not all bugs. When you are writing standard unit tests for a class you are doing the following

  • Create a fresh class instance (ex using Setup method in DUnit framework);
  • Run a code under test (usually a single call of a single method) on the instance;
  • Free the instance (ex using TearDown method in DUnit framework).

And this is how unit tests should be written; if your test detects a bug you immediately know the bug’s origin.

The problem with the above scenario is that it is ideal to hide some badly reproducible bugs such as access violation (AV) bugs. To detect such a bad bug with good probability you need something different, probably to do multiple calls of a method on the same instance, or to call different methods in the same test, and this approach is quite opposite to the idea of unit testing.

Advertisements

17 thoughts on “Why unit tests are not always good

  1. Topic fail imho.
    You could also say: Why hammers are not always good… if you only have screws.

    Unit tests are still good. They are just not to find these kind of bugs.

    • oh big infallible ruling master, which sentence thou would recommend to impose on this thour failing disciple?

      • The title would better read “Unit tests are not good enough”. The way it is, it can be misinterpreted as meaning that unit tests can be bad (as I did).

  2. I agree with abouchez. Nowhere is written “unit tests” should be simple. They could be fairly complex, and if you find a bug – even in “corner cases” – you should retrofit your test to check for it – recreating the conditions that triggered the bug. The only “boundary” of unit tests it’s they should performed on “units”, not the the overall application.
    Anyway of course “unit tests” are just a subset of a product “tests”. You can’t just perform them and think you did everything needed.

  3. An artificially constrained and limited view of testing your code is an artificially constrained, and limited view of testing your code. Let’s focus less on “this is unit testing, and that is not unit testing, it’s integration testing” and more on “let’s make our apps great, by whatever means are practical and effective”, and worry less about buzz-word compliance and more about effectiveness of our working time.

    • @Warren:
      I don’t agree. Getting your terms right is important in our job because if you don’t do that many misunderstandings can occur. I think if you ask 10 developers what a “great app” is like you will get 10 different answers (and I am not asking marketing…). Also “practical” and “effective” will most likely differ.

  4. @Warren: I’m with Stefan on this. Part of the reason that questions and comments like this keep popping up is that we aren’t defining our terms properly. Right now “unit test” is used to describe any automated testing, which causes confusion when someone digs a little deeper. And if we can’t clearly describe the concept and what it’s supposed to gain us, how are we supposed to convince people that it’s worth the investment?

  5. We had bugs in code talking to an SQL DB. Whenever it failed, the unit test would clean up (which meant a rollback). There was constant fighting between me, the developer, because I never discovered what bad in the DB was causing the fail, and the guy writing the unit tests who said it’s correct to rollback.

    • If you are talking to a DB technically it’s not a unit test (which should be in isolation) but an integration test (because you are testing different pieces of your software).

      The thing is that you should not only have unit tests but also integration and other tests. Each of them serve a purpose but unit tests are usually closest to the code and can cover a major part of functionality if your code is properly designed (that is why some people including me are talking about writing testable code).

    • Wow. I am completely at odds with the author’s characterisation of unit testing and TDD and with most (but not all) of his suggestions.

      • I was really wondering if that article was an attempt of trolling because it is full of bs.
        I agree that design by contracts is nice but I certainly would rather kill myself than doing a refactoring in such a system without any unit tests where I can quickly test the things I just changed. Maybe it’s his definition of “unit test” but how can you test a system where the assertions are in your code. DbC done right is something like strong typing because in such systems the compiler just prevents you from connecting pieces that have mismatching contracts to prevent any possible errors.

      • Watch the video below. It’s a less extreme take on Coplien’s views. Still lots to disagree with, though.

    • Not only my blog posts. I believe the duplication began after some update on wordpress.com and affects all blogs on this platform. I don’t know how to fix it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s