Testing concurrent order placement in Magento 2

Triggered by a recent bug report for our Order Number Customiser extension I had to work out a way to run concurrent tests. Essentially for orders placed the very same second a duplicate order number could have been created while not getting caught by Magento's inbuilt duplicate protection.

While I initially tested concurrency manually (with multiple open browsers) and also tried to measure the performance of the code using the Magento performance profiling it wasn't automated. Even though it took 3 years for this issue to surface something in the code clearly had room for improvement.

The hard part to testing concurrency is that there aren't that many tools suited/available for Magento 2. The extension itself already had unit, integration, functional and acceptance tests. Looking through them none of which would run concurrently. The unit tests are run without a database. Mocking the database would also mock away where the failure triggers (a unique constraint of a db column) - so these were out. Using functional/acceptance tests might work but this would end up requiring a lot of infrastructure to expose our behaviour vs testing the scalability of Magento itself.

With that, I settled on trying to create an integration test that would initially fail in a multi concurrency situation.

Phpunit does not provide means to test concurrently out of the box. However php, with the PCNTL extension enabled, is able to run code in parallel. For me this was the first time that I tried to use concurrency in a php class and the general structure is something like the below:

//Code before
$pid = pcntl_fork();
if ($pid == -1) {
    throw new \RuntimeException('Couldn\'t fork process');
}

if ($pid) {
    //Code in main thread
} else {
   //Code in child process
   exit;
}

//In main thread wait for all child processes to finish
//Continue

With the above you end up with 2 concurrent threads. The general idea of my test was then to fork the thread many more times and create orders in the child processes as these would all run in parallel.

//Place all the orders at the same time
for ($count = 0; $count < self::NUM_ORDERS; ++$count) {
    $pid = pcntl_fork();
    if ($pid == -1) {
        throw new \RuntimeException('Couldn\'t fork process');
    }

    if ($pid) {
        $pidsWithChildren[] = $pid;
    } else {
        $this->createOrder();
        exit;
    }
}

//Wait for all child processes to finish
while (count($pidsWithChildren) > 0) {
    foreach ($pidsWithChildren as $key => $pid) {
        $res = pcntl_waitpid($pid, $status, WNOHANG);

        // If the process has already exited
        if ($res === -1 || $res > 0) {
            unset($pidsWithChildren[$key]);
        }
    }

    sleep(1);
}
//Confirm NUM_ORDERS orders have been created

Challenges

The final solution ended up being a bit trickier than the above. The first challenge was the exit; call in the child process to indicate that the process had finished its work. Unfortunately when exit; gets called the database connection was also terminated. This was problematic as the connection was shared from its parent when it was forked. The internets fortunately had a solution to this which involved registering a custom shutdown function to gracefully exit. The child process part then looked like this:

} else {
    $this->createOrder();
    register_shutdown_function(
        function () {
            posix_kill(getmypid(), SIGKILL);
        }
    );
    exit;
}

With the database connection surviving that part of the code, I still ran into database related issues. The testing framework tried to run its clean up tasks prematurely - the child processes are testcases themselves after all. I worked around this issue by adding a @magentoDbIsolation disabled annotation to the test method itself and then running some manual clean up tasks in the testcase's tearDown() method.

Once I had a reliable way to show the issue I was able to tweak the main database query to perform the operation atomically (in other words read and increase the counter in the same query). With my final solution I was pleased to see that I was then able to successfully generate 25 orders per second on my non-optimised dev machine from inside a docker container. This number would already translate to 90.000 orders per hour which is up there with even the biggest Magento Commerce builds.

The changes are available in the latest release of Order Number Customiser and apply to all formats that include {ID} (the default).

Surprises

Once a process is forked communication between the threads requires additional means. The easiest approach I came across involves creating a temporary file $tempFile = tmpfile(); that all processes log output to and that can then be read out by the main process at the end again.

Another surprise for me was that adding a database index does not always help and in this case would have been counterproductive as the extra index would have created a deadlock.