Module Hegel

Introduction

Property-based testing for OCaml, powered by the native Hegel engine based on Hypothesis.

Hegel runs the test function on many generated inputs. You generate data inline, drawing values with draw as the test runs, rather than generating the data then running the property body. Each draw returns an ordinary OCaml value that you bind with let, compute with, and branch on, so a later draw can depend on an earlier generated value or a value from the system under test.

Because Hegel uses integrated shrinking, shrinking comes for free.

Getting started

Install Hegel

To install Hegel for OCaml:

  opam install hegel

The version of Hegel in OPAM sometimes lags behind the version in Github. To pin the version in Github:

  opam pin add hegel "git+https://github.com/hegeldev/hegel-ocaml.git"

Hegel for OCaml supports Linux (amd64/arm64) and macOS (Apple Silicon). macOS amd64 (Intel) has no published libhegel artifact, so on that platform point HEGEL_LIBHEGEL_PATH at a locally built libhegel.dylib.

Hegel works with whatever test framework your project already uses. The examples below use Alcotest.

Add hegel and alcotest to your dune test stanza:

  (test
   (name my_tests)
   (libraries hegel alcotest)
   (preprocess (pps ppx_hegel_test)))

Write your first test

Write a property test using let%hegel_test:

  open Hegel

  let%hegel_test commutative_addition tc =
    let a = draw tc (Generators.integers ~min_value:(-1000) ~max_value:1000 ()) in
    let b = draw tc (Generators.integers ~min_value:(-1000) ~max_value:1000 ()) in
    require_equal tc Core.Int.sexp_of_t (a + b) (b + a)

  let () =
    Alcotest.run
      "my_tests"
      [ "properties", [ Alcotest.test_case "commutative addition" `Quick commutative_addition ] ]

We check the property with require_equal rather than assert (a + b = b + a). It takes a printer for the values and, when they differ, shows a structural diff of the two sides in the failure report instead of a bare "assertion failed". Use require for a boolean check with a custom message. A plain assert can be used as well, but it does not provide as much information as require_equal and require.

Run dune runtest. You should see Alcotest report the test as passing. Hegel generates up to 100 random input pairs and reports the minimal counterexample if it finds one. When a test fails, Hegel prints each value you drew from the failing case, named after the let binding it was bound to (a = …, b = …).

The rest of the examples in the documentation assume you have open Hegel at the top of the test file like in the example above, and refer to generators as Generators.foo.

Next, let's try a test that fails.

  let%hegel_test every_int_is_small tc =
    let n = draw tc (Generators.integers ()) in
    assert (n < 50)

This test asserts that any integer is less than 50, which is obviously incorrect. Hegel finds a test case that makes the assertion fail, then shrinks it to the smallest counterexample (n = 50). The final replay prints the drawn values, the exception, and a rerun with: line that replays the exact case:

  --- Failure: every_int_is_small (my_tests.ml:3) ------------------
  Falsified after 1 test case (0 discarded):

    n = 50

  Exception: File "my_tests.ml", line 5, characters 2-8: Assertion failed
  rerun with: [@@failure_blobs [ "AAEAAAAACgEAAAAy" ]]

To fix this test, you can constrain the integers you generate with min_value and max_value:

  let%hegel_test every_int_is_small tc =
    let n = draw tc (Generators.integers ~min_value:0 ~max_value:49 ()) in
    assert (n < 50)

Use generators

Hegel provides a rich library of generators that you can use out of the box. There are primitive generators, such as Generators.integers, Generators.floats, and Generators.text, and combinators that build generators out of other generators, such as Generators.lists and Generators.tuples2.

For instance, you can use Generators.lists to construct a list of integers:

  let%hegel_test append_increases_length tc =
    let xs = draw tc (Generators.lists (Generators.integers ()) ()) in
    let initial_length = List.length xs in
    let xs = draw tc (Generators.integers ()) :: xs in
    require tc ~msg:"prepending an element must grow the list"
      (List.length xs > initial_length)

Custom generators are also supported. Suppose you have a person record that requires generation. Build a generator for it with Generators.composite, drawing each field in sequence:

  type person =
    { age : int
    ; name : string
    }

  let person =
    Generators.composite (fun tc ->
      let age = draw_silent tc (Generators.integers ()) in
      let name = draw_silent tc (Generators.text ()) in
      { age; name })

You can chain drawing operations together, so a later draw depends on an earlier one. For instance, extending person with a driving_license field that can only be true once age is at least 18:

  type person =
    { age : int
    ; name : string
    ; driving_license : bool
    }

  let person =
    Generators.composite (fun tc ->
      let age = draw_silent tc (Generators.integers ()) in
      let name = draw_silent tc (Generators.text ()) in
      let driving_license =
        if age >= 18 then draw_silent tc (Generators.booleans ()) else false
      in
      { age; name; driving_license })

Changing test settings

To override the default settings, attach a [@@settings ...] attribute:

  let%hegel_test commutative_addition tc =
    let a = draw tc (Generators.integers ()) in
    let b = draw tc (Generators.integers ()) in
    require_equal tc Core.Int.sexp_of_t (a + b) (b + a)
  [@@settings settings ~test_cases:500 ()]

This increases the number of test cases run from 100 to 500.

You can also update settings using the with_* functions:

  let%hegel_test commutative_addition tc =
    let a = draw tc (Generators.integers ()) in
    let b = draw tc (Generators.integers ()) in
    require_equal tc Core.Int.sexp_of_t (a + b) (b + a)
  [@@settings settings ~test_cases:500 () |> with_seed 5 |> with_verbosity Verbose]

Debugging failing test cases

Use note to attach debug information:

  let%hegel_test every_int_is_small tc =
    let n = draw tc (Generators.integers ()) in
    note tc (Printf.sprintf "n is %d" n);
    assert (n < 50)

A failing run prints a framed report: the shrunk counterexample's draws and notes, the exception, and a copy-pasteable rerun with: line whose base64 blob encodes the choice sequence that caused the failure (disable it with with_print_blob false). On a terminal the report headers (and require_equal diffs) print in color; set HEGEL_COLOR to 1 or 0 to force colors on or off.

For an equality property, prefer require_equal over assert (x = y): it adds a structural diff of the two values to this report, so you see exactly how they differ. require is the message-carrying boolean variant.

  let%hegel_test every_int_is_small tc =
    let n = draw tc (Generators.integers ()) in
    assert (n < 50)

  --- Failure: every_int_is_small (my_tests.ml:3) ------------------
  Falsified after 2 test cases (0 discarded):

    n = 50

  Exception: File "my_tests.ml", line 5, characters 2-8: Assertion failed
  rerun with: [@@failure_blobs [ "AAEAAAAACgEAAAAy" ]]

The blob can then be used to replay the failing test case:

  let%hegel_test every_int_is_small tc =
    let n = draw tc (Generators.integers ()) in
    assert (n < 50)
  [@@failure_blobs [ "AAEAAAAACgEAAAAy" ]]

The blob is only meant to reproduce the failure within a specific version of Hegel, since the choice sequence leading to a failure can change from version to version.

Learning more

See Generators for the generators and Stateful for state-machine testing.

Hegel module documentation

Submodules

Settings

Build a settings value with default_settings or settings, refine it with the with_* functions, and attach it to a let%hegel_test with the [@@settings ...] attribute:

  let%hegel_test many_cases tc =
    let n = draw tc (Generators.integers ~min_value:0 ~max_value:99 ()) in
    assert (n < 100)
  [@@settings settings ~test_cases:500 () |> with_verbosity Verbose]

  let s = settings ~test_cases:500 ~seed:42 ()

  let s = default_settings () |> with_verbosity Verbose

  let s = default_settings () |> with_database (Path "_hegel_db")

  let s = default_settings () |> with_suppress_health_check [ Filter_too_much; Too_slow ]

  let s = default_settings () |> with_phases [ Generate; Shrink ]

Running tests

  let my_settings = settings ~test_cases:50 ~seed:5 () in
  let () =
    run_hegel_test ~settings:my_settings (fun tc ->
      let n = draw tc (integers ~min_value:0 ~max_value:9 ()) in
      assert (n >= 0 && n <= 9))

Drawing values

  let%hegel_test draw_example tc =
    let n = draw tc (integers ~min_value:0 ~max_value:100 ()) in
    assert (n >= 0)

  let%hegel_test draw_silent_example tc =
    let n = draw_silent tc (map (fun x -> x * 2) (integers ~min_value:0 ~max_value:9 ())) in
    assert (n >= 0)

Guiding generation

  let%hegel_test head_cons_tail_reconstructs tc =
    let xs = draw tc (lists (integers ()) ()) in
    (* The property is only meaningful for non-empty lists. *)
    assume tc (xs <> []);
    assert (List.hd xs :: List.tl xs = xs)

  let%hegel_test grow_size tc =
    let v = draw tc (integers ~min_value:0 ~max_value:1000 ()) in
    target tc (float_of_int v) "size";
    assert (v <= 1000)

Debugging tests

  let%hegel_test note_value tc =
    let n = draw tc (integers ~min_value:0 ~max_value:99 ()) in
    note tc (Printf.sprintf "n is %d" n);
    assert (n < 100)

  let%hegel_test balanced tc =
    let l = draw tc (lists (integers ()) ()) in
    require tc ~msg:"sum must stay non-negative" (running_sum l >= 0)

  let%hegel_test sort_is_stable tc =
    let l = draw tc (lists (integers ()) ()) in
    require_equal
      tc
      (Core.List.sexp_of_t Core.Int.sexp_of_t)
      (List.sort compare l)
      (stable_sort l)

  let%hegel_test with_printer_example tc =
    let doubled = map (fun x -> x * 2) (integers ~min_value:0 ~max_value:9 ()) in
    let n = draw tc (with_printer Core.Int.sexp_of_t doubled) in
    assert (n >= 0)

Concurrency and parallelism

Hegel can drive generation from more than one thread or domain within a single test. Two rules govern it.

First, test-case handles may not be shared. A single handle must be drawn from by one thread at a time, so give each thread its own clone using clone. A clone has its own choice sequence. Drawing from one shared handle on multiple threads throws a concurrent-use error. Concurrently driving one shared collection, pool, or state machine will likely produce flaky results, so always make a new one per unit of concurrency/parallelism.

Second, a draw is a synchronous engine call that holds its domain's runtime lock and never yields, so it cannot cooperate with an event loop or overlap another draw on the same domain.

As long as you follow these two rules and your code is deterministic, you will be able to replay failures.

Some advice for common concurrency/parallelism libraries:

Use Threads for interleaving of concurrent operations and overlapping blocking work, not parallel generation, since draws serialize under the runtime lock. You should use spawn / join rather than Thread.create. Thread.join drops a worker's exception, whereas join re-raises it into the runner.

  let%hegel_test concurrent_workers tc =
    let w = spawn tc (fun worker -> draw_silent worker gen) in
    let mine = draw_silent tc gen in
    ignore (mine, join w)

Use Domainslib or any domain pool for when you need true parallelism, such as higher generation throughput. We strongly recommend that you do not use domains directly, as they are expensive to create and destruct. Set up the pool once and reuse it. Clone up front then Task.async each clone and Task.await it.

  (* the pool is created once and reused across cases *)
  let pool = Domainslib.Task.setup_pool ~num_domains:2 ()

  let%hegel_test parallel_generation tc =
    Domainslib.Task.run pool (fun () ->
      let worker = clone tc in
      let p = Domainslib.Task.async pool (fun () -> draw_silent worker gen) in
      let mine = draw_silent tc gen in
      ignore (mine, Domainslib.Task.await pool p))

Use Eio for concurrent generation with structured concurrency or if your code already uses Eio. Each fiber should draw its own data from its own clone. Since a draw does not yield, only separate domains make draws truly parallel. Here two workers race increments onto a shared atomic. The property is that no update is lost.

  Eio_main.run @@ fun env ->
  let dmgr = Eio.Stdenv.domain_mgr env in
  run_hegel_test (fun tc ->
    let counter = Atomic.make 0 in
    let amounts = integers ~min_value:0 ~max_value:100 () in
    let worker g () =
      Eio.Domain_manager.run dmgr (fun () ->
        let n = draw_silent g amounts in
        ignore (Atomic.fetch_and_add counter n : int);
        n)
    in
    let worker_b = clone tc in
    let sum_a, sum_b = Eio.Fiber.pair (worker tc) (worker worker_b) in
    require_equal tc Core.Int.sexp_of_t (sum_a + sum_b) (Atomic.get counter))

  let%hegel_test two_hands_two_dice_manual tc =
    let die = integers ~min_value:1 ~max_value:6 () in
    let other_hand = clone tc in
    let out = ref (Error (Failure "unset")) in
    let rolling =
      Thread.create
        (fun () -> out := (try Ok (draw_silent other_hand die) with e -> Error e))
        ()
    in
    let right_hand = draw_silent tc die in
    Thread.join rolling;
    match !out with
    | Ok left_hand -> assert (right_hand + left_hand >= 2)
    | Error e -> raise e

  let%hegel_test two_hands_two_dice tc =
    let die = integers ~min_value:1 ~max_value:6 () in
    let other_hand = spawn tc (fun worker -> draw_silent worker die) in
    let this_hand = draw_silent tc die in
    assert (this_hand + join other_hand >= 2)