Big Testing in Cucumber vs. JUnit

Hey all,

My team and I recently found ourselves in the interesting position of having to Big Test Rhapsody Routes written by another team.  Rhapsody is Orion Health's Integration Engine, used to create customized data workflows in the Clinical setting.  In this case it was being used as an alternative for some work we otherwise would have done in Java.  "Big Test" is a way of saying "End to End test of a live, running application".

On top of the usual Acceptance Criteria tests that you'd expect to perform at a "Big" level, we needed to have more granular, edge case type tests. In Java land, you get this kind of coverage with Unit level tests, but we don't have the ability to write these with Rhapsody routes.  As far as I know, there is no automated way of testing how every piece of Rhapsody configuration strings together or how an embedded Javascript behaves.

As a result, we had to Big Test/Black box test granular edge cases.  We decided to call these type of tests Big Little Tests.

There's a general association drawn between Big Tests and Cucumber so naturally, we considered writing these Big Little Tests in Cucumber. I insisted we do this type of test using Junit instead for a number of reasons:

1. Tests that don't directly back a high level business criteria have no place in a Cucumber test suite.
2. As much as I see the value Cucumber provides in enabling BDD, I actually dislike it a lot as a Testing Framework

To elaborate on point 2, I find Cucumber tests incredibly ugly. Readability of tests is absolutely critical for me, and because using Cucumber requires splitting up scenarios into one method per step, reading your tests becomes difficult. The step definitions for a single scenario are often dispersed across a number of classes. You can't look at one method and immediately tell the intent of a test. Figuring out what's going on usually involves starting at your feature file and systematically drilling into each step to read the code. It works, but it's not fun. On top of this, the idea of passing test context between steps has always made me nauseous.

In my mind, I have always pictured our Big Tests looking a lot prettier as JUnit tests and our decision to do pursue edge case testing in the form of Big JUnit tests became the opportunity I had been looking for to actually compare Cucumber and JUnit as Testing Frameworks.

To make this happen, we needed to seperate out "framework" type code from our Cucumber testing Maven project into its own Maven project. The framework code is everything we need to run our particular type of Big Tests. Our acceptance tests send in documents through a Soap Service and verify that some data appeared using a REST service. The Soap and REST clients required to do this verification using Java code constitutes our Framework code. This same code was required for our Big Little Tests, so it needed to be somewhere that both the Cucumber and Big Little tests projects could access them.

Once this was all done we got onto writing up our JUnit Big Little tests. What we ended up with was pretty interesting.

Our JUnit tests were concise and readable. The intent of each test could easily be read from the method name and test variables could easily be traced from input to output. These Big tests did everything a Cucumber Test would otherwise do but in a much prettier, coder-friendly manner.

To attempt to illustrate this difference (without showing you any code :( ), one of our Cucumber tests comprise:

• A Scenario from a Feature File.  You can consider this the text description of a test
• A Step definition class for the Scenario.  We stick with one class per scenario for the relevant step definitions.
• A class for common steps.  A number of our scenarios share certain steps. We decided to stick these steps in one class

On the other hand, each of our JUnit Big Little Tests are single methods at roughly 15 lines long.

Considering the intended use of Cucumber and JUnit are completely different, this isn't a comparison that is intended to sway you to start writing your Big Tests in JUnit all of a sudden. I am a big fan of Cucumber as a collaboration tool.

I suppose what we can distil from this is further evidence that Cucumber should only be used for the purpose of verifying high level acceptance criteria. Cucumber is often shoe-horned as a testing tool.  Not only does this lead to an ugly set of acceptance criteria but also a large set of ugly Cucumber tests.

Putting careful thought into what actually belongs in your Cucumber suite can save you a lot of hassle in having to deal with Cucumber's rather unfriendly nature. In most cases, granular edge case testing should be pushed into unit tests. In our unique case, shoe-horning JUnit for Big Testing gave us a more concise and readable set of tests than what we would have had if we shoe-horned Cucumber.

Let me know your thoughts below!

Cheers,

Shrek