Performance testing using XCTest

Photo by Aron Visuals / Unsplash

I think we all have heard the phrase by Sir Tony Hoare:

Premature optimization is the root of all evil

Many of us, me included, at some point visited this dark place where we wanted to make things super-optimized right from the start and this dream ended in a nightmare where we needed to take two steps back and redesign our system in a simpler way.

This doesn't mean we should stop optimizing at all. Even though nowadays our phones are much more powerful and less constrained than they used to be.

Consider the case where we observed stuttering in the application and used Time Profiler instrument to identify a potential issue and then made changes to make it better.
How do we reliably measure the change? We can print processing times to the console but it's cumbersome, could be biased by external factors, and it's not possible to track performance changes over time i.e. in a month from now.

This is where the performance tests come into play.

We found out that the stuttering is caused by contains function in our item container. The core functionality of this function is to check whether an item is contained in the container. Currently, our container implementation is based on an array:

struct ArrayBasedContainer<T: Equatable> {
    let array: [T]
    
    func contains(item: T) -> Bool {
        array.contains(item)
    }
    
    /* Other needed, but not important for the article, code */
}

It was working fine for a long time but recently users started complaining about noticeable stuttering in the application. Time Profiler beyond any doubt marked our contains function as responsible for the situation.

The problem is that it was working fine and now it's not and there are no, obvious, breaking changes. It's hard.
What we know is that there is a significant downgrade in the speed of the contains function and our application recently become popular and we noticed huge user growth.

It would be nice to be able to experiment with the input and to see how the function performs. Doing this in our application code is doable but we would need to add a code for calculating the time, code to, temporary, mock the input, and so on.

Doing this in unit tests context is much better. We have unit tests for logic in place:

func test_WhenItemIsPresent_ContainsReportsTrue() {
    let sut = ArrayBasedContainer(array: [1,2,3])
    
    XCTAssertTrue(sut.contains(item: 2))
}

func test_WhenItemIsMissing_ContainsReportsFalse() {
    let sut = ArrayBasedContainer(array: [1,2,3])
    
    XCTAssertFalse(sut.contains(item: 4))
}

These tests verify whether contains function returns the correct value. We need to check how it's performing now.

Enabling performance testing in a unit test is as easy as adding:

measure { /* The code we want to measure performance for */ }

Let's write our test for the performance of the contains function:

func test_ThousandItems_ContainsPerformance() throws {
    let items = Array(0..<1000)
    let sut = ArrayBasedContainer(array: items)
    
    measure {
        _ = sut.contains(item: 1000)
    }
}

We prepare ArrayBasedContainer containing 1000 items. Then in the measure block, we call the contains function to check how fast this code is. We are using an item that is not available in the container to make sure the test won't be biased. Performance of validating first vs the last item may not be identical. Later I will explain why.

When we run this test the outcome will differ from the regular unit test:

When we tap on a gray diamond we will see a detailed report:

On the bottom, we see all the results. Performance tests are run multiple times to minimize the bias and ensure the results are less affected by external factors. It's possible to configure how many times tests are run if needed - check XCTMeasureOptions.
The Average is 0,000287s which is not bad and shouldn't cause any issues. Let's add three zeros and this time add test for one million items:

This test Average is 0,162s. This is not great. Depending on the number of items speed of our contains function drops. We identified the root cause of our bug and it's a problem with an array and not our code.

How to fix it?

To make the fix we need to understand what happened first. Our container is built on top of an array and is using the contains function. Let's read the documentation of this function:

Returns a Boolean value indicating whether the sequence contains the given element.

This doesn't help us. We know this. It's why we were using this function. But if you scroll down to the "Discussion" section you will see something interesting:

Complexity: O(n), where n is the length of the sequence.

This explains a lot. If you didn't encounter Big O notation before check Big O notation on wikipedia:

Big O notation is used to classify algorithms according to how their run time or space requirements grow as the input size grows.

Our contains function is using contains of the array which has O(n) complexity. This means our function has the same complexity. Let's see what does this mean:

Graphs of functions commonly used in the analysis of algorithms, showing the number of operations N versus input size n for each function

From Big O notation on wikipedia

Our function is the green one in the middle which is:

Not great, not terrible.

In many cases, arrays are fine for using contains checks but we need to remember that the speed of this function drops with the number of elements in the array. Our performance tests confirmed that.

Knowing this we need to find a solution for our performance problem. We remember that array is not the only option in swift. There is, among others, set. Let's check set in the documentation:

An unordered collection of unique elements.

This isn't a problem for us but it's good to remember that set has limitations. Now we check contains function complexity in the documentation:

Complexity: O(1)

Set contains function complexity is a constant time. This means that input size won't impact processing time. Now we will verify whether it's true. First, we will create a new container but this time it will be based on set:

struct SetBasedContainer<T: Hashable> {
    let set: Set<T>
    
    func contains(item: T) -> Bool {
        set.contains(item)
    }
}

Now we can write tests for it:

func test_ThousandItems_ContainsPerformance() throws {
    let items = Array(0..<1000)
    let sut = SetBasedContainer(set: Set<Int>(items))
    
    measure {
        _ = sut.contains(item: 1000)
    }
}

func test_MilionItems_ContainsPerformance() throws {
    let items = Array(0..<1000000)
    let sut = SetBasedContainer(set: Set<Int>(items))
    
    measure {
        _ = sut.contains(item: 1000000)
    }
}

The time has come to run the tests and compare the results. Open Report navigator either by clicking or using CMD + 9 and tap on the latest Test:

The documentation didn't lie. Set contains function performance didn't change much between thousand and million elements. We can replace the array with a set and the application will work fine again.

This fixes our current problem but what to do to make sure this won't become an issue again in the future?

We can set the baselines.

When you open test details tap on the Set Baseline:

Next time you run the tests it will additionally provide information on whether this test performed better or worse:

It's useful when working on optimizing the code. Each test run provides quick feedback.

Baselines can be committed to the repository. Note - it doesn't make sense to compare baselines calculated on different devices. That's why they are stored separately per configuration and only compared when the current configuration has baselines set. Do a diff after setting the baselines if you want to know what and how is stored.

If you have any feedback, or just want to say hi, you are more than welcome to write me an e-mail or tweet to @tustanowskik

If you want to be up to date and always be first to know what I'm working on tap follow @tustanowskik on Twitter

Thank you for reading!

P.S. The Vision Series is not over yet. Stay tuned!

This article was featured in Awesome Swift #277 ūüéČ

Kamil Tustanowski

Kamil Tustanowski

I'm an iOS developer dinosaur who remembers times when Objective-C was "the only way", we did memory management by hand and whole iPhones were smaller than screens in current models.
Poland