Big Testing in Cucumber vs. JUnit

Hey all,

My team and I recently found ourselves in the interesting position of having to Big Test Rhapsody Routes written by another team.  Rhapsody is Orion Health's Integration Engine, used to create customized data workflows in the Clinical setting.  In this case it was being used as an alternative for some work we otherwise would have done in Java.  "Big Test" is a way of saying "End to End test of a live, running application".

On top of the usual Acceptance Criteria tests that you'd expect to perform at a "Big" level, we needed to have more granular, edge case type tests. In Java land, you get this kind of coverage with Unit level tests, but we don't have the ability to write these with Rhapsody routes.  As far as I know, there is no automated way of testing how every piece of Rhapsody configuration strings together or how an embedded Javascript behaves.

As a result, we had to Big Test/Black box test granular edge cases.  We decided to call these type of tests Big Little Tests.

There's a general association drawn between Big Tests and Cucumber so naturally, we considered writing these Big Little Tests in Cucumber. I insisted we do this type of test using Junit instead for a number of reasons:

1. Tests that don't directly back a high level business criteria have no place in a Cucumber test suite.
2. As much as I see the value Cucumber provides in enabling BDD, I actually dislike it a lot as a Testing Framework

To elaborate on point 2, I find Cucumber tests incredibly ugly. Readability of tests is absolutely critical for me, and because using Cucumber requires splitting up scenarios into one method per step, reading your tests becomes difficult. The step definitions for a single scenario are often dispersed across a number of classes. You can't look at one method and immediately tell the intent of a test. Figuring out what's going on usually involves starting at your feature file and systematically drilling into each step to read the code. It works, but it's not fun. On top of this, the idea of passing test context between steps has always made me nauseous.

In my mind, I have always pictured our Big Tests looking a lot prettier as JUnit tests and our decision to do pursue edge case testing in the form of Big JUnit tests became the opportunity I had been looking for to actually compare Cucumber and JUnit as Testing Frameworks.

To make this happen, we needed to seperate out "framework" type code from our Cucumber testing Maven project into its own Maven project. The framework code is everything we need to run our particular type of Big Tests. Our acceptance tests send in documents through a Soap Service and verify that some data appeared using a REST service. The Soap and REST clients required to do this verification using Java code constitutes our Framework code. This same code was required for our Big Little Tests, so it needed to be somewhere that both the Cucumber and Big Little tests projects could access them.

Once this was all done we got onto writing up our JUnit Big Little tests. What we ended up with was pretty interesting.

Our JUnit tests were concise and readable. The intent of each test could easily be read from the method name and test variables could easily be traced from input to output. These Big tests did everything a Cucumber Test would otherwise do but in a much prettier, coder-friendly manner.

To attempt to illustrate this difference (without showing you any code :( ), one of our Cucumber tests comprise:

• A Scenario from a Feature File.  You can consider this the text description of a test
• A Step definition class for the Scenario.  We stick with one class per scenario for the relevant step definitions.
• A class for common steps.  A number of our scenarios share certain steps. We decided to stick these steps in one class

On the other hand, each of our JUnit Big Little Tests are single methods at roughly 15 lines long.

Considering the intended use of Cucumber and JUnit are completely different, this isn't a comparison that is intended to sway you to start writing your Big Tests in JUnit all of a sudden. I am a big fan of Cucumber as a collaboration tool.

I suppose what we can distil from this is further evidence that Cucumber should only be used for the purpose of verifying high level acceptance criteria. Cucumber is often shoe-horned as a testing tool.  Not only does this lead to an ugly set of acceptance criteria but also a large set of ugly Cucumber tests.

Putting careful thought into what actually belongs in your Cucumber suite can save you a lot of hassle in having to deal with Cucumber's rather unfriendly nature. In most cases, granular edge case testing should be pushed into unit tests. In our unique case, shoe-horning JUnit for Big Testing gave us a more concise and readable set of tests than what we would have had if we shoe-horned Cucumber.

Let me know your thoughts below!

Cheers,

Shrek

BDD the Testing Paradigm?

Dear Readers,

I've come to realise that many developers see BDD with Cucumber solely as a means to achieve an automated testing framework.  Sure, the advantages in this regard are clear.  Automated testing frees testers from the burden of regression testing and allows them to focus on exploratory testing.  It also helps to facilitate a Continuous Delivery model with features being pushed out rapidly and over all functionality being guarded by the automated testing suite.  Sounds great.

However, this particular drive for adopting BDD presents the danger of misinterpreting its original intention.Lets start with a definition of BDD:

"Behaviour Driven development is about implementing an application by describing its behaviour from the perspective of its stakeholder"

The primary issue that BDD was created to address is poor communication between primary stakeholders and developers.  It is influenced by the idea that creating a conversational, common language between developers and stakeholders and using this language to specify requirements is the best way of developing the right product. 

Developing automated tests based on this conversational set of specifications is a secondary concern of BDD.

Unfortunately, the drive to create a replacement for regression testing suites has blinded many developers, myself included, from the original purpose of BDD.

The result is a view that does not consider primary stakeholders.  Since the automated tests simply replace regression tests, only developers and testers need to be able to understand their descriptions in the Cucumber Scenario files.  The original purpose of Behaviour Driven Development, as stated in its very name, is lost.

My team at work has suffered from this very issue.  Development had a particular understanding of a feature we were writing, our Business Analyst had another.  It was only during a review of our feature files that we realised that our view of the product had diverged.  If the very first thing we had done was to define a common language between us and to define the behaviour of the product as a result, these divergences would have been addressed in step 1.

So what's my solution?  Firstly, read up on your BDD theory.  Writing up notes on this lecture by Gordon Force was a great start for me.

Before starting the development of a feature: Start with your product's stakeholders.

1. Define who your stakeholders are
2. Define the interests of your stakeholders
3. Defined a common language between you and your stakeholders that you can use to write a set of requirements
4. Using this, figure out a template for your Feature files and Scenarios
5. Figure out your development process around BDD.  Write your scenarios and decide their priority.  Have developers pull these scenarios in priority order to develop and your managers/BA decide which of these scenarios comprise your minimally viable product.  Feed this into your sprint planning

My team have recently implemented this approach and so far it is working out quite well!  I first performed the above analysis for a feature we planned to develop.  The following sprint was spent fleshing out a complete set of Scenario descriptions.  This gave us a granular set of sub-features that we could sensibly pick and choose from for the planning of the next sprint.

Although BDD recommends an outside-in approach, a decision to develop the automated tests in parallel to the code was made for the sake of getting the work done efficiently.  I haven't decided how cynical I should be to this approach, but in any case, since both the developers of the functionality and the developer of the automated tests were working directly from the Scenario descriptions, they ended up marrying quite happily.

My take on all of this is that yes indeed Automated Regression suites are important and that BDD is a great way of achieving them.  However, testing coverage should come as a result of requirement definitions that can be communicated easily with stakeholders, so always start with those!

Cheers,

Shrek

Swt Cucumber Eclipse Plugin!

Hello hello, readers!

I've been finding Cucumber development with Eclipse a bit painful, namely with regards to the lack of linkage between Feature files and Step Definitions as well as executed tests and Step Definitions.  Having to navigate to your Step Definition Java classes and perform text searches for the scenario step you're interested in is just plain inefficient.  So what's the solution?

Luckily for me, my colleague, Tom Whitmore had recently dug up a Cucumber plugin from the internet which seems to work brilliantly.  It does not link executed tests to step definitions :'(  However it does link feature files to step definitions quite beautifully.  All you have to do is hold the 'Ctrl' key and mouse over your feature steps.  If there's a matching step, click to go to it!  It also has all sorts of cute syntax highlighting of feature files.  Makes life easier :)

Anyway, it's called Natural, by Roberto Lo Giacco.  Find it here: Natural on Github.

Happy BDD-ing, people,

Shrek