FsCheck 3: Property-based testing in C#
I would assume that we can agree on the fact that unit testing is good and helpful. At least I hope that this is the common opinion. But…
I would assume that we can agree on the fact that unit testing is good and helpful. At least I hope that this is the common opinion. But what if I told you that there is a way to level up your testing? FsCheck is a library that allows you to easily create property-based tests in C# (and F#, but I don’t cover it in this article). In this article, I’ll show you how to use FsCheck 3 to write property-based tests in C#. We’ll start with simple generators and move to more complex ones. I got inspired to write this article after I discovered that FsCheck 3 has absolutely no documentation, so it’s as useful for me as it is for you. Let’s start with a short intro to property-based testing.
All the code for this post is available on my GitHub.
What is property-based testing?
Let’s say that you have a system that allows you to order fruits and tools. Strange combination, I know, but bear with me. An order has some properties that need to be fulfilled to be valid. For example, items ordered need to have count of at least one. Or the total price has to be non-negative. Or the name needs to be one of the predefined values of items present in the system. To unit test order handling, you would have to come up with a bunch of test cases. But what if I told you that all you need is to define a set of rules that the order needs to follow, and you will have auto-generated data, each time being different? This is what property-based testing is about. You create generators that will create data for you, and then in your test, you just say “Give me a list of orders”, and generators will create them for you. And it will run (by default) 100 times, each time with different input data. This way you can test more cases with less effort (usually). I have to admit that not always it’s easier to write property-based tests, as they require a lot of upfront thinking, but they are worth it when you have a complex calculations system on a big set of data with many different combinations and it would be hard to cover all the possible cases with unit tests. Now let’s get to the actual code.
Our first property-based test
[Fact]
public void SimpleTest()
{
Prop.ForAll<int>(x => x >= 0).QuickCheckThrowOnFailure();
}Let’s try running the test:
System.Exception
Falsifiable, after 1 test (0 shrinks) (10979290545183390790,6994218916298689381)
Last step was invoked with size of 2 and seed of (11283820414104960607,2916118026088292885):
Original:
-1
with exception:
System.Exception: Expected true, got false.This test is failing. Why? Because we use the default generator for integers, which is not limited to non-negative numbers. Let’s break down what we can see here:
Falsifiable, after 1 test- this means that our test failed after only one try(0 shrinks)- shrinking is an interesting, however advanced topic. It's a process of trying to find the smallest input data that will still make the test fail. For simple types it works pretty well out of the box: for numbers, it tries to find the case closest to zero, for lists, it will try to find the shortest list that still makes the test fail.(10979290545183390790,6994218916298689381)- this is the seed that was used to generate the data. This one's super useful if a test fails after 60 runs. Using the seed you can run the test again with the failing data right away, skipping the successful runs.Original: -1- this is the input data for which the test failed
Simple type generator
Okay, so we have a failing test, but how to make it pass? We need to write a generator that will generate only non-negative numbers.
public static class SimpleTypesGenerators
{
// this is a generator that will generate only non-negative numbers
public static Arbitrary<int> OverrideIntArb() =>
ArbMap.Default.GeneratorFor<int>().Where(x => x >= 0).ToArbitrary();
}public class SimpleTypesTests
{
// Another way of running PBT, also specifying the generator
[Property(Arbitrary = new[] { typeof(SimpleTypesGenerators) })]
public Property Generator_OverridingInt(int p)
{
return (p >= 0).ToProperty();
}
}Let’s run the test and see the output:
Ok, passed 100 tests.As you can see, not only did the test pass, but it also ran 100 times. This is the default number of runs for FsCheck.
Now let’s try changing the condition in the test to p <= 50 and see what happens:
FsCheck.Xunit.PropertyFailedException
Falsifiable, after 61 tests (0 shrinks) (7331799476358367170,1508067056380619747)
Last step was invoked with size of 61 and seed of (6419148108620199268,14754055774596855233):
Original:
57As you can see, the test failed after 61 runs, and the input data failing the test was 57.
Generators of custom types
Let’s try to create a generator for a custom record type:
public record PositiveInt(int Value);
public static class SimpleTypesGenerators
{
// Generator, used to generate the values
public static Gen<PositiveInt> GetPositiveInt() =>
ArbMap.Default.GeneratorFor<int>().Where(x => x >= 0).Select(x => new PositiveInt(x));
}This is the generator for the PositiveInt record. To be able to use it in tests, we need to create an Arbitrary from it:
public static class SimpleTypesGenerators
{
public static Arbitrary<PositiveInt> PositiveIntArb() =>
Arb.From(GetPositiveInt());}Now we can use it in our tests:
[Fact]
public void AnotherWayOfRunningTests()
{
var prop = Prop.ForAll<PositiveInt>(p => p.Value >= 0);
prop.Check(Config.Default);
}And the test is passing.
More complex types
Now let’s try to create a generator for order. First of all, here’s the order object:
public record Order(Guid Id, String Name, OrderStatus Status, int Quantity, decimal Price);
// enum with possible statuses
public enum OrderStatus
{
New,
InProgress,
Done
}
// list of possible names for orders - names from outside of the list are illegal and should not be generated
private static readonly List<string> FruitNames = new List<string> { "Apple", "Banana", "Cherry", "Elderberry" };
private static readonly List<string> ToolNames = new List<string> { "Axe", "Hammer", "Screwdriver", "Wrench" };Let’s try to quickly create a generator to generate orders:
public static class ComplexTypesGenerators
{
// Choose one of the statuses - you can specify only a subset of the statuses you want to generate
// if needed, you can split it into multiple generators, having different subsets of statuses
public static Gen<OrderStatus> OrderStatusGenerator()
{
return Gen.Elements<OrderStatus>(OrderStatus.New, OrderStatus.InProgress, OrderStatus.Done);
}
// Generator returning a name from two lists, with weight attached.
// This means that 1/4 of the time it will generate a name from the list of tools,
// and 3/4 it will be the name from the list of fruits
public static Gen<String> NameGenerator()
{
return Gen.Frequency(
(3, Gen.Elements<String>(FruitNames)),
(1, Gen.Elements<String>(ToolNames))
);
}
// Generator for price - generates a number between 1 and 100, and then converts it to decimal
public static Gen<decimal> PriceGenerator()
{
return Gen.Choose(1, 100).Select(x => x * 1.0m);
}
// I have found this old Linq syntax to be the most readable.
// As an alternative, you can also use the syntax from the function below
public static Gen<Order> OrderGenerator()
{
return
from name in NameGenerator()
from qty in Gen.Choose(1, 10)
from price in PriceGenerator()
from status in OrderStatusGenerator()
select new Order(Guid.NewGuid(), name, status, qty, qty * price);
}
// This is an alternative way of writing the same generator as above,
// but chaining selects gets unreadable quickly
public static Gen<Order> AnotherWayOfOrderGeneration()
{
return NameGenerator().SelectMany(name =>
{
return Gen.Choose(1, 10).Select(qty =>
new Order(Guid.NewGuid(),
name,
OrderStatusGenerator().Sample(1, 1).Single(),
qty,
qty * PriceGenerator().Sample(1, 1).Single())
);
});
public static Arbitrary<Order> OrderArb() =>
Arb.From(OrderGenerator());And now let’s write some tests using the above generator:
public class ComplexTypesTests
{
private readonly ITestOutputHelper _testOutputHelper;
public ComplexTypesTests(ITestOutputHelper testOutputHelper)
{
_testOutputHelper = testOutputHelper;
}
[Fact]
public void SingleOrderTest()
{
var prop = Prop.ForAll<Order>(order =>
{
_testOutputHelper.WriteLine(order.ToString());
Assert.True(order.Price >= 1);
});
prop.Check(Config.QuickThrowOnFailure
// Specify the generator class for the test
.WithArbitrary(new[] { typeof(ComplexTypesGenerators) }));
}
[Fact]
public void OrdersListTest()
{
// Having a generator for a complex type, we have a generation of a list of such types for free.
Prop.ForAll<List<Order>>(orders =>
{
_testOutputHelper.WriteLine(orders.Count.ToString());
// Some asserts
})
.Check(Config.QuickThrowOnFailure
.WithArbitrary(new[] { typeof(ComplexTypesGenerators) }));
}
}A few first outputs of the tests:
SingleOrderTest, writing the single object to the console:
Order { Id = ec0b5817-d124-4211-ace9-7a52253a6766, Name = Banana, Status = InProgress, Quantity = 10, Price = 280,0 }
Order { Id = be828369-ff32-4b5a-b6b1-c9dfc17650bb, Name = Cherry, Status = Done, Quantity = 1, Price = 11,0 }
Order { Id = 93d6c498-851d-4118-b727-a0f1f812004c, Name = Cherry, Status = New, Quantity = 9, Price = 81,0 }
Order { Id = 86df039e-3975-4200-b7c4-e68443218584, Name = Banana, Status = Done, Quantity = 1, Price = 54,0 }
Order { Id = b28c0be2-d9fb-4e9c-9b4e-0a0e7ed48c76, Name = Banana, Status = InProgress, Quantity = 10, Price = 60,0 }
Order { Id = de99993a-bc4c-4c51-826e-99e88f30beec, Name = Banana, Status = New, Quantity = 7, Price = 273,0 }
Order { Id = 17d2b4c9-87be-40bb-92f2-e6be6bcca2d5, Name = Axe, Status = New, Quantity = 1, Price = 18,0 }
Order { Id = 1b49b225-fa60-4424-8e93-923b63846d3e, Name = Apple, Status = InProgress, Quantity = 4, Price = 228,0 }
Order { Id = 87a52d4d-9ca8-448c-b643-76751a23c671, Name = Elderberry, Status = InProgress, Quantity = 5, Price = 50,0 }
Order { Id = 0581488e-499c-4cc5-8928-d5ffab089b79, Name = Banana, Status = Done, Quantity = 5, Price = 195,0 }
Order { Id = 923d5fce-3ea8-46d4-8bc9-b634b6f9f296, Name = Axe, Status = New, Quantity = 5, Price = 100,0 }
(...)OrdersListTest, writing the count of the input list to the console:
2
3
1
1
5
2
6
0
5
4
11
11
5
9
14
4
19
0
5
4
2
21
19
24
(...)As you can see, we have a lot of data generated, and the more data we need to generate, the more useful it becomes.
Using generators outside of property tests
It is possible to use generators in regular unit tests. It might be useful if you already have a generator and you don’t want to run it as a PBT (for any reason). However, I feel like there is no reason to do that, because why not just use PBT?
public class GeneratorInNonPbtTests
{
[Fact]
public void NonPbtUnitTest()
{
var positiveInt = SimpleTypesGenerators.GetPositiveInt().Sample(numberOfSamples: 1, size: 1).First();
Assert.True(positiveInt.Value >= 0);
}
}Properties of tests
There are a few properties of the tests that you can use to control the behavior of your tests. I’ll focus on the two most interesting ones: WithArbitrary and WithReplay:
prop.Check(Config.QuickThrowOnFailure
// Specify the generator class for the test
.WithArbitrary(new[] { typeof(ComplexTypesGenerators) })
.WithReplay("(6419148108620199268,14754055774596855233)"));WithArbitrary is used to specify the generator class for the test.
WithReplay is used to replay the test with the same seed that was used to generate the failing data. You can find the seed in the output of the failing test.
Using attribute on the test method it looks similar:
[Property(Arbitrary = new[] { typeof(SimpleTypesGenerators) }, Replay = "(6419148108620199268,14754055774596855233)")]Summary
I am not using property-based testing daily. For simple functions and simple inputs, it’s usually easier to write unit tests. But when I have complex functions in which I know that I might have a log of edge cases, here’s where PBT comes in handy.
The best part about PBT is that it’s repeatable randomness. You can always re-run the failed test with the same seed, getting the same input data. However, there are also some downsides. The biggest one I’ve found is that there is no documentation for FsCheck 3. For me, it was trial and error, and I hope that this article will help you to get started with FsCheck. If you have any troubles, I can recommend looking at the tests of the project itself and playing around with the code — I found it helpful.
Enjoy testing!