Hspec: A Testing Framework for Haskell

Contents

Setting expectations

Hspec provides several combinators that can be used to set expectations about the outcome of code examples.

Expecting equality

A common expectation is that two values are equal. shouldBe can be used here:

x `shouldBe` 23

shouldBe can be used in combination with IO actions. Here is an example:

launchMissiles >>= (`shouldBe` Left "permission error")

shouldReturn is a shortcut for that. The above code can be simplified to:

launchMissiles `shouldReturn` Left "permission error"

Require that a predicate holds

shouldSatisfy requires that some predicate holds for a given value.

xs `shouldSatisfy` (not . null)

It is similar to HUnit's assertBool, but it gives a more detailed error message if the expectation fails.

Expecting exceptions

Hspec provides a mechanism to state that an action throws an exception. Here is a basic example:

launchMissiles `shouldThrow` anyException

Expecting exceptions of a specific type

The type of shouldThrow is:

shouldThrow :: Exception e => IO a -> Selector e -> Expectation

It takes an IO action and a Selector. The Selector describes the precise nature of the expected exception.

There are several predefined selectors:

anyException      :: Selector SomeException
anyErrorCall      :: Selector ErrorCall
anyIOException    :: Selector IOException
anyArithException :: Selector ArithException

The selector anyException can be used to expect an arbitrary exception. Likewise anyErrorCall, anyArithException and anyIOException can be used to expect arbitrary exceptions of type ErrorCall, ArithException and IOException respectively.

Using predicates as selectors

A Selector is just a predicate. It can simultaneously constrain the type and value of an exception. This can be used in various ways, e.g. you can expect a specific exception value:

launchMissiles `shouldThrow` (== ExitFailure 1)

Expecting specific `IOException`s

The module System.IO.Error exports predicates to classify IOExceptions. Those can be used in combination with shouldThrow to expect specific IOExceptions. Here is an example that uses isPermissionError to require our hypothetic launchMissiles action to fail with a permission error:

launchMissiles `shouldThrow` isPermissionError
Example code:
-- file Spec.hs
import Test.Hspec
import System.IO.Error (isDoesNotExistError)

main :: IO ()
main = hspec $ do
  describe "readFile" $ do
    context "when specified file does not exist" $ do
      it "throws an exception" $ do
        readFile "none-existing-file" `shouldThrow` isDoesNotExistError
runhaskell Spec.hs

readFile
  when specified file does not exist
    throws an exception [✔]

Finished in 0.0005 seconds
1 example, 0 failures

Dealing with `error` and `undefined`

Both error and undefined throw exceptions of type ErrorCall. There is no Eq instance for ErrorCall, hence it is not possible to use == to expect a specific exception value of type ErrorCall. The following won't work:

evaluate (error "foo") `shouldThrow` (== ErrorCall "foo")  -- This won't work!

Pattern matching can be used instead, but Hspec provides a combinator, errorCall, to make this more convenient. Here is how it's used:

evaluate (error "foo") `shouldThrow` errorCall "foo"
Example code:
-- file Spec.hs
import Test.Hspec
import Control.Exception (evaluate)

main :: IO ()
main = hspec $ do
  describe "Prelude.head" $ do
    it "throws an exception if used with an empty list" $ do
      evaluate (head []) `shouldThrow` errorCall "Prelude.head: empty list"
runhaskell Spec.hs

Prelude.head
  throws an exception if used with an empty list [✔]

Finished in 0.0005 seconds
1 example, 0 failures

Expecting exceptions from pure code

evaluate can be used to expect exceptions from pure code:

evaluate (1 `div` 0) `shouldThrow` anyArithException

However, evaluate only forces its argument to weak head normal form. To better understand what that means, let's look at an other example:

evaluate ('a' : undefined) `shouldThrow` anyErrorCall

Here evaluate does not force the exception. It only evaluates its argument until it encounters the first constructor. Here the constructor is :, as soon as evaluate sees : it's done. It does not look at the arguments of that constructor.

force can be used to force the exception:

(evaluate . force) ('a' : undefined) `shouldThrow` anyErrorCall

Note that return $!! (or any other mechanism that relies solely on seq) does not work reliably! (see the discussion at #5129)

Example code:
-- file Spec.hs
import Test.Hspec
import Control.Exception (evaluate)
import Control.DeepSeq

main :: IO ()
main = hspec $ do
  describe "div" $ do
    it "throws an exception if the second argument is 0" $ do
      evaluate (1 `div` 0 :: Int) `shouldThrow` anyArithException

  describe "evaluate" $ do
    it "forces exceptions" $ do
      evaluate ('a' : undefined) `shouldThrow` anyErrorCall

  describe "evaluate . force" $ do
    it "forces exceptions" $ do
      (evaluate . force) ('a' : undefined) `shouldThrow` anyErrorCall
runhaskell Spec.hs

div
  throws an exception if the second argument is 0 [✔]
evaluate
  forces exceptions [✘]
evaluate . force
  forces exceptions [✔]

Failures:

  Spec.hs:13:34:
  1) evaluate forces exceptions
       did not get expected exception: ErrorCall

  To rerun use: --match "/evaluate/forces exceptions/"

Randomized with seed 921447365

Finished in 0.0005 seconds
3 examples, 1 failure

Beware of GHC's semantics for imprecise exceptions

GHC's semantics for imprecise exceptions can be tricky. But simple examples are evident. If we look at the exceptional value error "foo" + error "bar" it may be tempting to assume that the following expectation holds:

evaluate (error "foo" + error "bar" :: Int) `shouldThrow` errorCall "foo"

There is a pretty good chance that this will indeed hold, but it may equally well fail. The reason is that + does not give any guarantees about the evaluation order of its arguments.

Semantically an exceptional value contains a set of exceptions. When we "look" at the value, one representative from the set is chosen in a non-deterministic way.

For our simple example the set of exceptions is {ErrorCall "foo", ErrorCall "bar"}, so we have to be prepared to encounter any of those. We can accommodate for this by combining errorCall "foo" and errorCall "bar" with ||:

evaluate (error "foo" + error "bar" :: Int)
  `shouldThrow` (||) <$> errorCall "foo" <*> errorCall "bar"

The details of imprecise exceptions are described in the paper A semantics for imprecise exceptions. But beware that GHC does not fully adhere to those semantics (see #1171, #2273, #5561, #5129).

Example code:
-- file Spec.hs
import Test.Hspec
import Control.Exception (evaluate)

main :: IO ()
main = hspec $ do
  describe "an exceptional value" $ do
    it "contains a set of exceptions (semantically)" $ do
      evaluate (error "foo" + error "bar" :: Int)
        `shouldThrow` (||) <$> errorCall "foo" <*> errorCall "bar"
runhaskell Spec.hs

an exceptional value
  contains a set of exceptions (semantically) [✔]

Finished in 0.0005 seconds
1 example, 0 failures