Writing Thread Safety tests for instance data

In my last post, I wrote about Implementing a simple multi-threaded TasksQueue. This post will concentrate in how to test for Thread Safety of the queue. Reminder: our queue is used by multiple consumers which means that I must make sure that before each Enqueue\Dequeue\Count, a lock will be obtained on the queue. Imagine that I have 1 item in the queue and 2 consumers trying to dequeue this item at the same time from different threads: The first dequeue will work just fine but the second will throw an exception (dequeue from an empty queue). We’re actually trying to make sure that this queue works as expected in multi-threaded environment. So far about our goal.

So how can we test it?
Testing for the queue’s thread safety through testing of TasksQueue, the way it’s written now, can be quite hard and misleading. The ConsumeTask method calls dequeue inside a lock but what if we had a thread-safety-related-bug there? do we test only that the dequeue works as expected? not really. ConsumeTask (1) dequeue an item and then (2) “consume it”. We’re actually testing 2 behaviors\logics – this way, it’s really hard to test only for the queue’s thread safety. We should always test a single method for a specific behavior and eliminate dependencies. Only when we cover our basis, we can check for integration between multiple components (the underlying queue and the TasksQueue).

One way of allegedly achieving this goal is to create a decorator around the queue, let’s call it SafeQueue, which will encapsulate a queue and wrap it with thread-safe forwarding of the calls (it will lock some inner object and call the original queue). The SafeQueue could be tested then by its own and used by our TasksQueue. This will “enable” us to remove the locking in the TasksQueue and use Set\WaitOne instead of Pulse\Wait in order to notify our consumers on arrival of a new task:

while (_safeQueue.Count == 0)
Monitor.WaitOne();

// NOTICE: by the time we get here, someone could have pulled the last item from the queue on another thread!
string item = _safeQueue.Dequeue();

WATCH OUT: This is a deadly solution that will make our TasksQueue break in a multi-threaded environment. Just like that, our code is not thread-safe anymore although we’re using a SafeQueue that expose (atomic) thread-safe methods\properties. This is exactly why instance state should not be thread-safe by default (more details at Joe Duffy’s post).

The locking of the queue should remain in our TasksQueue, but we should separate the dequeue part from the handling part and check each one by its own. We’ll check the dequeue part for thread-safety(assuming that the underlying queue was tested by itself) and the handling part for pure logic. We can now test that for X calls for enqueue we get the same X calls for dequeue.

Here is the refactored code:

private void ConsumeTask()
{
   while (true)
   {
      string task = WaitForTask();

      if (task == null) return; // This signals our exit

      try
      {
         // work on the task
      }
      catch (Exception err)
      {
        // log err & eat the exception, we still want to resume consuming other tasks.
      }
   }
}

protected virtual string WaitForTask()
{
   lock (_locker)
   {
      // if no tasks available, wait up till we’ll get something.
      while (_queue.Count == 0)
         Monitor.Wait(_locker);

      // try to put it outside of the lock statement and run the test(bellow)
      return _queue.Dequeue();
   }
}

public virtual void EnqueueTask(string task)
{
   lock (_locker)
   {
      _queue.Enqueue(task);
      Monitor.Pulse(_locker);
   }
}

Now we can create a simple test for the thread safety by overriding both of the enqueue\dequeue methods:

internal class TestableTasksQueue : TasksQueue
{
   private static int _dequeueCount = 0;
   private static int _enqueueCount = 0;

   public TestableTasksQueue(int workerCount) : base(workerCount) {}

   protected override string WaitForTask()
   {
      string item = base.WaitForTask();
      Interlocked.Increment(ref _dequeueCount);
      return item;
   }

   public override void EnqueueTask(string task)
   {
      base.EnqueueTask(task);
      Interlocked.Increment(ref _enqueueCount);
   }

   public static int DequeueCount
   {
      get { return _dequeueCount; }
   }

   public static int EnqueueCount
   {
      get { return _enqueueCount; }
   }
}

The tricky part here is the test itself. Because of subtle multi-threading issues, we can’t actually know when 2 (or more) threads will try to dequeue on the same time, so we have to run this test enough times in order to detect bugs. Here is a little sample:

[TestFixture]
public class TasksQueueTests
{
   [Test]
   public void Counting_DequeueAndEnqueueCountsShouldBeEqual()
   {
      for (int j = 0; j < 1000; j++)
      {
         using (TestableTasksQueue queue = new TestableTasksQueue(5))
         {
            for (int i = 0; i < 100; i++)
               queue.EnqueueTask(“test” + i);
         }

         Assert.AreEqual(TestableTasksQueue.DequeueCount, TestableTasksQueue.EnqueueCount);
      }
   }
}