There were some interesting side conversations going on in the comments of my Java EE wins over Spring blog. In particular, a few people were arguing over the value of Mocks. I always considered Mocks a bogus pattern. Only time I ever use them is when I’m initially starting a project and I don’t have full implementations of a particular subsystem and I need to test that something works. You see to me, your GIT master branch, or your SVN trunk is a sacred place. I fully believe in the idea of continuous integration. But to have a pure Trunk, you have to test your code, as close as possible, to the target environment it is supposed to run in. Mocks go counter to this idea. And give you false sense of security.
My colleague, Stan Silvert said it best:
When you DO use mocks you are testing in an environment that is biased toward a passing test. An artificial environment will always make invalid assumptions about the behavior and stability of that environment.
What you end up with is a (hard to maintain) mock library that allows your tests to pass and gives you a warm feeling that you’ve done a great job. But you are fooling yourself.
The ONLY reason to use mocks is when test execution in the real environment is unacceptably slow. This used to be the case quite often. But with frameworks like Arquillian and super-fast startup times for app servers this is no longer a frequent concern.
I’ll end with an even more radical assertion. The difference between integration testing and unit testing is purely semantic. Even the smallest, most focused unit test pulls in a lot of the environment (JVM, OS, etc). In-container testing just pulls in more of the target environment and allows you to find more errors sooner.
Test in the real environment and get results that are truly valid. Stop kidding yourself with mocks.
Personally, I’m a very paranoid person when it comes to development. Earlier in my career I did a lot of demos and live coding in front of an audience. When you’re in this type of environment you live by the mantra that anything that can go wrong will. So, when rehearsing, you always mimic as much as possible what your setup will be. I carried this paranoia to Iona (an Irish company), when I worked on Orbix there. We had a rule, that if even one test (integration or whatever) failed, you were awarded the Guiness Penalty. If you broke the build, you had to buy every one on the team a pint of Guiness. IN this environment, Mocks are a waste of time, both in build CPU and in coding time as you still had to run integration testing before you committed anything (unless of course you just wanted an excuse to have a few beers).
So unless a particular integration test is ungodly slow, don’t use mocks. Stan again, said it best:
…the days of the 2 hour in-container smoke test are over.Since mocks no longer provide much of a speed advantage their use is outweighed by the fact that they give you a phony environment, maintenance headaches, and false confidence.But you do touch on a truth about testing that a lot of people miss. The purpose of testing is to uncover errors. And you will uncover more errors in a real environment than you would in a mock environment.
With projects like Arquillian and fast boot up times like AS7 (1.75 seconds on my laptop), there’s no need to mock anymore.
Bill the Plumber 
May 01, 2012 @ 16:06:45
Fantastic sentiments, ones we frequently champion when explaining the role of Arquillian. There are some great Getting Started Guides on arquillian.org which should help to bootstrap new users.
May 02, 2012 @ 07:10:06
Bill, it depends what do you want to test. Please do not confuse unit testing with integration testing.
The first one is the fast failing, early-warning set of tests that test the piece of code they test and nothing else. Mocking “outside systems” helps actually not depending on those systems, but mocking the expected behavior. Yes I agree in that case you are depending on a maybe-false assumption, the other piece works *as expected*, but on the other hand that prevents you from some weird dependencies and countless hours of debugging when a set of tests for ServiceA fails really because ServiceB, The Dependency is broken.
Now once you are 100% sure that the ServiceA and ServiceB in isolated environments are working *as expected* you can combine an integration test that will test them working in tandem. And here you shouldn’t use mocks.
And I don’t really understand what do you mean by big mocking library. There’s mockito etc. which allow really nice, small snippets of code to get your code working.
May 02, 2012 @ 13:24:32
I’m not confusing unit testing and integration testing. Blindly following some idea that you have to categorize your tests into unit tests and integration tests is short-sighted and many times a pedantic exercise. Test use cases and regressions (regressions really model real-world highly specific use cases), in as close to the target environment as possible or feasible. Mocking doesn’t buy you much anymore other than giving you that happy feeling that you’ve followed some idiom of some academic definition of software engineering.
May 02, 2012 @ 21:09:31
Bill, this is a very opinionated statement (I don’t mean that in any negative sense, I’m curious). But to me there is a lot of detail missing to convince me of your point of view.
What I understand is that you do quite a lot of high-level testing, accessing the public interface of your application. What is the approximate ratio of low-level tests (one object-ish) and public interface tests?
My fears are mainly about a high maintenance burden as the application grows above say 25000 lines. From my experience this burden comes from
– False errors.
– Hard to debug errors.
– Tests being hard to understand.
– Long test runtime (several minutes).
– Difficulty to get a good test coverage (from a branch perspective)
To what extent do you (or don’t) live with this? And if you don’t how do you avoid it? How often do you deliver?
May 02, 2012 @ 22:33:08
I just don’t see what the size of the project has anything to do with whether you mock or not. Or whether you run all tests of a project before committing. Just because a project is a million lines of code doesn’t mean that the code isn’t modularized into specific sub-projects. JBoss Application Server aggregates a large set of JBoss.org projects. Does this mean it mocks or runs the testsuite of each of them? No, it doesn’t. For Resteasy, does this mean I need to test the UriBuilder implementation within the context of a HTTP request? No, it doesn’t.
May 03, 2012 @ 20:54:19
I get what you mean about testing what can be tested in an isolated fashion like UriBuilder.
With the size of a project comes the maintenance issues related to “full” stack tests. They’re not visible below my totally arbitrary limit of 25000 lines 🙂 From my experience these issues are inevitable even though much can be done to make it bearable. Nevertheless I’ve come to a different conclusion – that in order to obtain virtually 0 bugs I’m better off doing loads of isolated tests, plus some full stack tests to cover most of the lacks of the isolated tests.
So from someone who has gone down another path I’m curious what results I could expect of your approach and also what learnings you’ve drawn from dealing with the potential maintenance issues I mentioned.
May 03, 2012 @ 08:18:55
I think one point that people regularly omit is. “Who tests the mocks?”. When you write mocks, they are either trivial and always return 42 (and are thus completely useless to test any real world scenario). Or they are more complicated and e.g. return pseudo-database rows. But in this case the logic in the mock grows and will be prone to the same x-errors-per-y-lines-of-code rule than the the real business logic, which means that you also need to write tests to test the mocks.
We at RHQ (http://rhq-project.org/) started using the embedded ejb3 container from Bill almost 6 years ago and while it was sometimes a bit of a pita, we were always happy to be able to execute tests against the real target databases.
And I think this embedded ejb3 container was also the point in time for me when all this dependency injection hype (you can exchange the implementation with a mock by manipulating dozens of lines of xml) made no more sense, as I was just able to test against the real environment.
Nowadays we are using an Arquillian based approach to test our as7 plugin against real running as7 servers and are thus much more agile to follow modifications in the as7 server — mocking this one out would take at least one full time employee to follow up with all the as7 progress.
tl;dr : I never was a fan of mocks and don’t see the point in using them.
May 03, 2012 @ 15:24:21
First of all, we all agree that we’ve gotta make automated tests… and for that, i’m already so thankful =)
One of the advantages to make unit tests, that i didn’t see anyone here or in the Spring’s post discussing about, is that they really help and guide your application design.
Mainly, when we are all concerned about loosely coupled between class.
For instance, with unit tests, you tend to use a lot of IoC, especially Dependency Injection, because if you don’t, you are not going to be able to test it (unless you have a container to provide it)!
Another advantage is that it is so much easier to make unit tests than integrations tests, especially to teach to a new programmer about it.
It’s already difficult to try to convince them about automated tests, you can imagine if it was something difficult to accomplish it.
well, maybe, some day, because of the facilities that tools like Arquillian bring to the tests, they are going to be useless… BUT, i don’t see how they help on the application design… because it’s has already a container started and gives you all the dependencies, even if your object are to coupled or doing so much stuff that it wasn’t supposed to do.
But ok.. let’s see how it goes..
May 04, 2012 @ 02:41:50
Mocks are a great option when it comes to interacting with a persistent data store. Integration test require you be able to to consistently restore your database to a given ‘state’, and that can be a huge task. I’ve found integration tests are great for the 90% of your app that is ‘read-only’ and mocks are great for testing the updates. deleting logic, etc. Especially we you don’t work for Red Hat and have the luxury of millions of beta testers doing your dirty work for you :p
May 04, 2012 @ 13:54:42
Sure, millions of beta testers may find a lot of bugs, but rarely do they actually fix them. We still end up writing the vast majority of tests.
May 04, 2012 @ 14:38:26
Check out the Arquillian Persistence Extension[1] to help with bootstrapping real data. 🙂
[1] https://github.com/arquillian/arquillian-extension-persistence
May 06, 2012 @ 17:09:39
The bottom line is that integration testing is not only required it is performed on 100% of released software. Unfortunately, that confidence is based on some using unwitting customers to perform integration tests, post-release!
What we are talking about here of course is automated integration testing that happens before the release. I agree, keep it as real as possible, however, but that isn’t unit testing. You can skip unit testing, but you can’t skip integration testing whether you perform unit tests or not.
People often use a unit testing framework for integration tests and call that unit testing. No harm no foul but if you decide you want to do actual unit testing you should make a distinction. In Java or C# the unit is the class. You should mock the external dependencies to perform a unit test. However, I would rather skip unit testing entirely than maintain mocks, The cure is worse than the disease.
Mock frameworks to the rescue. There are some mock frameworks, but only a couple, that actually make true unit testing practical, economical and effective. I believe Moq is in that category (C#) and have heard that Mockito (Java) is too. Not every external dependency has to be mocked, be pragmatic! As Bill points out, this isn’t an academic exercise, it’s goal-oriented not process-oriented. So adding true (practical not pure) unit testing and running those on each build is something that can improve quality and shorten overall development time. But only if combined with a separate integration testing effort.
p.s. suggesting that dependency injection is “hype” is just abject ignorance. Sorry but we should not let that slide.
Is there such a thing as the NoMock movement? « henk53
Sep 11, 2012 @ 21:30:11
Oct 30, 2012 @ 09:42:45
I do not understand how you will do integration testing with for example SMS sending service, payment system service or even just e-mail service that do not have testing API or possibility to get to know if service really did his job? For mock testing it is enough to get to know that testing code did a call to external service, because it tests application logic and not the logic of external services. And note that external services are sometimes slow or just do not work. What to do if external service is not available when integration test is running? in mock testing you do not care. you even can emulate “external service is not available” problem.
i agree that database mocking is somewhere mockery. But again if you have legacy system with oracle stored procedures it is not convenient to re-create database each time and populate it with lot of init data just to ensure for example that application saves log record to database when some problem was occurred.
Nov 27, 2012 @ 14:56:23
May 02, 2013 @ 18:55:19
“I’m not confusing unit testing and integration testing. Blindly following some idea that you have to categorize your tests into unit tests and integration tests is short-sighted and many times a pedantic exercise.”
Sorry but this is nonsense. Chaining a group of developers to an outside dependency that won’t always be available is what is stupid. There IS a distinction between integration and unit tests whether or not you chose to acknowledge it.
The point of mocking is twofold. 1. To avoid retesting a dependency that already has its own unit tests. 2. To avoid stopping all development while some outside dependency is no longer available or is broken.
Even within a project, such as an MVC structured application, it makes no sense whatsoever to retest the DB cruft functions when you are only testing the manager or controller. The point of testing is to test your own code, not somebody else’s. It is totally safe to assume that a particular service is available and will return a known result if you have decided to use it in your product, unless you are stupid and have chosen an unreliable external service or dependency. And even if so, the test should be fleshing out how your code handles cases where the outside service is unavailable.
Lastly, if you’re not mocking, you’re unable to test non-“happy path” scenarios. To test failure scenarios without mocking, you have to ask the provider to take down their service just so you can test failure scenarios. Good luck with that.
Nobody who does mocking is just “blindly following some idea”. It is useful and the myriad of consumers and producers of mock projects in multiple languages bear this out. Go ahead and argue with what has been developed over decades as good practice. Unless you have a viable alternative solution however, you are just blowing hot air.
May 02, 2013 @ 20:27:22
Obviously, if you can’t test certain situations with the real deal, you’re going to have to artificially create the test conditions. Obviously, you can’t make an automated build dependent on unreliable remote services. Geez, the whole point of the article is to make people rethink and question how they do things. Chill man, Chill…