Using Hamcrest to assert asynchronous behaviour

In order to write a passing acceptance test, I often find the need to poll Selenium for a given situation to be true.  Such polling tests often look like this:

endTime = getCurrentTime() + duration;
failed = true;
while( failed = hasCheckFailed() ){
  sleep( intervalInMillis )
  now = getCurrentTime();
  if( now > endTime ) {
    fail();
  }
}
return !failed

This works, but it has 3 pieces of information embedded in it – the check, the duration, and the interval.

This check could be replaced with a hamcrest matcher…

assertThat( thing, poll(duration, interval, delegateMatcher) )

where the poll() method returns  Matcher<Thing>.

However, the implementation of the poll() matcher becomes pretty complex – it’s delegating to a matcher, AND handling the timing concerns.

So, we played with another approach, which is to run the matcher against a finite time series:


assertThat( sample(thing), retry( delegateMatcher ))

When the assertion fails, it generates a message like:

Expected: retrying( delegate matcher description )
Got: sampling every 1 SECONDS for 10 SECONDS(
thing.toString()
)
The sample method returns an Iterable<Thing> whose iterator repeated returns thing until the time runs out. It’s next() method sleeps for the duration. This is where the timing concerns are handled (and nothing else).

The retry matcher just iterates through the iterable until the sequence completes, or the test passes. It doesn’t have any time related logic.

The retry matcher is now easy…:


new TypeSafeMatcher<Iterable<T>>(){
  private Matcher<T> delegate;
  public boolean matchesSafely(Iterable<T> ts) {
    for( T t : ts ){
      if( delegate.matches(t)) return true;
    }
    return false
  }
}

The sample() method is a bit trickier, but essentially it returns a RealTimeSeries object that has one method: iterator(). This returns a SampleIterator – here is the code:


private class SampleIterator implements Iterator {
   private boolean firstSample = true;
   private long expectedEnd;
   public boolean hasNext() {
     if (firstSample) {
       return true;
     }
     final long endOfNextSample = clock.timeInMillis()
      + interval.toMillis();
     return endOfNextSample <= expectedEnd;
   }
   public T next() {
     if (firstSample) {
       expectedEnd = clock.timeInMillis()
         + max.toMillis();
       firstSample = false;
     } else {
       try {
         interval.sleep(clock);
       } catch (InterruptedException e) {
         return sampledThing;
       }
     }
     return sampledThing;
   }
   public void remove() {
     throw new UnsupportedOperationException();
   }
}

In practice, we can now write code like:


assertThat( sample(selenium).duration(120), retry(elementIsPresent('element-id')))
assertThat( sample(page).duration(10), retry(hasMessageCount(5)))

and then remove the duplication here

assertThatWithin( duration(120, SECONDS), selenium, elementIsPresent('element-id'))
assertThatWithin( every(10, SECONDS), page, hasMessageCount(5))

Advertisements

3 Responses to Using Hamcrest to assert asynchronous behaviour

  1. Thom says:

    Are Selenium’s waitForCondition and waitForElement etc no use to you here?

    • nickdrew says:

      Absolutely. I think they are appropriate, and happily use them where it’s appropriate.

      I wrote the above from the point of view of making it easy to add asynchronous checking if there are already hamcrest matchers in use in the codebase. Hamcrest gives great diagnostics on test failure, and allows the composition of complex matchers.

      My issue with waitForCondition is that by using arbitrary javascript to implement the check, you get poor diagnostics, and it’s harder to combine complex tests.

      So if I start with a synchronous check:

      assertFalse( sel.isTextPresent( "helloThere from user Me") );

      to make this asynchronous, I might have to do this:

      assertTrue(
      nbsp;sel.waitForCondition(
      nbsp;nbsp;"var allText = selenium.page().bodyText(); var unexpectedText = \"hello there from user${var}\" allText.indexOf(unexpectedText) == -1;"
      nbsp;)
      )

      Where as if I started with:

      assertThat(sel, not(hasText("hello There from user Me")))

      to go to asynchronous checking is much quicker:

      assertThat( sample(sel), retry(not(hasText("hello There from user Me"))))

      Adding conjunctions make it easy to scale the expression:

      assertThat( sample(sel), retry(not(anyOf( hasText("hello"), hasElement("some-id"))))

      At some point my acceptance test may become slow, and then I’ll look into optimising. There are plenty of ways to do that, though, and I’ll try and avoid losing the diagnostics just to speed up the build.

  2. nickdrew says:

    I’ve realised that sample( T ) and retry( T ) are poor names for what we’re testing.

    So, instead, I think this might flow better:

    assertThat( overTime( t ), eventuallyIt( is(expected) ) )

    and when parameterizing the sampling rate:

    assertThat( overTime( t ).
    sampledEvery(500, MILLISECONDS).
    within( 5, SECONDS),
    eventuallyIt( is( expected ) )

    NB: I’m sure in ruby this would look even better, but I’m impressed by how much can be done with the use of static imports and generics in Java.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: