PHP Benchmarking framework

I'd like to suggest a couple of usability issues that came up using PHPBench for the first time:

1. There should be a setUp() and tearDown() etc. by default - I don't like to add @BeforeMethods({"setUp"}) to every docblock
2. The syntax of the data-provider was confusing to me - you should stick to PHPUnit and provide $param1, $param2 and $param3 instead of the single $params array
3. Is there a way to use PHPBench with a continuous integration service like Travis CI? If not, I suggest you add something like a threshold that returns an error code if the benchmarks failed
4. Tabular is great but you should truncate values that are longer than 20 characters: http://i.snag.gy/UNsRd.jpg
5. Please use semantic versioning... 0.6.0 instead of 0.6

Keep in mind, this is just a discussion - please create new issues for each one of the issues

Thank you

I am trying to initialize some hash contexts (resources) as a #BeforeMethods, but it won't let me use it, because something is changes them.

I attached a file and both bench functions should actually work, because they are using exactly the same code, except that one of them is using @BeforeMethods to initialize the hash, the other one is not.

PhpHashBench.php.txt

Hey. Die festen bindungen an eine php version neven total weil 0.16 grundsätzlich auch unter php8 laufen kann. "php": "^7.2", (composer.json) ist in diversen tags hinterlegt und composer will nur mit gewallt installieren :-(

also: php >= Version mit der es laufen muss oder "php": "^7.2 || ^7.3 || ^8.0" funktioniert wunderbar wenn man composer mit phpbench/phpbench: "*" configuriert (auf user seite) und composer --prefer-source nutzt um dessen unterstützung der dependeny checks erfolgreich zu erhalten. ich brauch eine version die laufen kann, nicht die stricte reduktion nichts mehr nutzen zu können :-) LG aus HH

I try to figure out if I can replace my own benchmark micro-framework with phpbench. I did a cursory search on documentation and the benchmarks-examples repository but failed to find what would be the best way to reimplement sanitize/calibrate functionality within phpbench (for example, check this class to see what I mean). Any suggestions?

To reproduce:

phpbench --report=aggregate --dump-file=fileA.xml --context="A"
phpbench --report=aggregate --dump-file=fileB.xml --context="B"

phpbench report --file=fileA.xml --file=fileB.xml --report=compare

right from the docs.

[InvalidArgumentException]
  Column "vcs_branch" does not exist, valid columns: "suite", "date", "stime", "benchmark_full", "groups", "params", "revs", "its", "mem_real", "mem_final", "mem_peak", "min", "max", "sum", "stdev", "mean", "mode", "variance", "rstdev", "best", "worst", "uname_os", "uname_host", "uname_release", "uname_version", "un
  ame_machine", "php_version", "php_xdebug", "baseline_nothing", "baseline_md5", "baseline_file_rw"  	
Exception trace:
 PhpBench\Report\Generator\Table\Row->offsetGet() at C:\project\vendor\phpbench\phpbench\lib\Report\Generator\TableGenerator.php:261
 PhpBench\Report\Generator\TableGenerator->PhpBench\Report\Generator\{closure}() at C:\project\vendor\lstrojny\functional-php\src\Functional\Map.php:43
 Functional\map() at C:\project\vendor\phpbench\phpbench\lib\Report\Generator\TableGenerator.php:265
 PhpBench\Report\Generator\TableGenerator->PhpBench\Report\Generator\{closure}() at C:\project\vendor\lstrojny\functional-php\src\Functional\Map.php:43
 Functional\map() at C:\project\vendor\phpbench\phpbench\lib\Report\Generator\TableGenerator.php:266
 PhpBench\Report\Generator\TableGenerator->processCols() at C:\project\vendor\phpbench\phpbench\lib\Report\Generator\TableGenerator.php:106
 PhpBench\Report\Generator\TableGenerator->generate() at C:\project\vendor\phpbench\phpbench\lib\Report\ReportManager.php:56
 PhpBench\Report\ReportManager->generateReports() at C:\project\vendor\phpbench\phpbench\lib\Report\ReportManager.php:84
 PhpBench\Report\ReportManager->renderReports() at C:\project\vendor\phpbench\phpbench\lib\Console\Command\Handler\ReportHandler.php:41
 PhpBench\Console\Command\Handler\ReportHandler->reportsFromInput() at C:\project\vendor\phpbench\phpbench\lib\Console\Command\ReportCommand.php:87
 PhpBench\Console\Command\ReportCommand->execute() at C:\project\vendor\symfony\console\Command\Command.php:252
 Symfony\Component\Console\Command\Command->run() at C:\project\vendor\symfony\console\Application.php:936
 Symfony\Component\Console\Application->doRunCommand() at C:\project\vendor\symfony\console\Application.php:240
 Symfony\Component\Console\Application->doRun() at C:\project\vendor\symfony\console\Application.php:148
 Symfony\Component\Console\Application->run() at C:\project\vendor\phpbench\phpbench\lib\PhpBench.php:58
 PhpBench\PhpBench::run() at C:\project\vendor\phpbench\phpbench\bin\phpbench.php:24
 require() at C:\project\vendor\phpbench\phpbench\bin\phpbench:4

I get this Exception then i try to run phpbench on windows.

[RuntimeException]
  Could not launch script:

the command I am executing

vendor\bin\phpbench.bat run benchmarks/StringifyBench.php

Windows 10 1511 php 5.5.8

please help me, I would love to use phpbench :)

Hi!

I want to be able to export a specific tag to store it in git so I can compare future revisions of the code against it.

One possibility I considered was storing the storage.xml_storage_path in git, but then my local runs would also be stored there and I would need to check if I accidentally added one to git every time. I find this too tedious and think there is a better way.

Another example of what I have in mind is the way psalm handles baselines: you can provide a baseline file via command line or config and all errors it finds will be checked against it. I can also store the baseline file in git and it will be used by the CI to check if new errors where added by the changed code.

Any idea if this is possible and desirable for enough people?

When using c35e70b3a6ee7cdaf1a7048809b8e3dcb9f0f10f I'm seeing the following error:

\Relay\Benchmarks\MgetBench::benchMgetUsingPredis

    PhpBench\Executor\Exception\ExecutionError
    Fatal error: Uncaught TypeError: {closure}(): Argument #1 ($value) must be of type string, array given in /private/var/folders/cw/70xgjbbx7ps2mlhyqx24snbc0000gn/T/PhpBenchPz3t08:27
    Stack trace:
    #0 [internal function]: {closure}(Array)
    #1 /private/var/folders/cw/70xgjbbx7ps2mlhyqx24snbc0000gn/T/PhpBenchPz3t08(34): array_map(Object(Closure), Array)
    #2 {main}
      thrown in /private/var/folders/cw/70xgjbbx7ps2mlhyqx24snbc0000gn/T/PhpBenchPz3t08 on line 27


    #0 /Users/Till/Development/Repositories/relay/src/PhpBench/RedisExecutor.php(50): PhpBench\Executor\Benchmark\TemplateExecutor->execute(Object(PhpBench\Executor\ExecutionContext), Object(PhpBench\Registry\Config))
    #1 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/Benchmark/Runner.php(366): Relay\PhpBench\RedisExecutor->execute(Object(PhpBench\Executor\ExecutionContext), Object(PhpBench\Registry\Config))
    #2 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/Benchmark/Runner.php(308): PhpBench\Benchmark\Runner->runIteration(Object(Relay\PhpBench\RedisExecutor), Object(PhpBench\Registry\Config), Object(PhpBench\Model\Iteration), Object(PhpBench\Benchmark\Metadata\SubjectMetadata))
    #3 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/Benchmark/Runner.php(279): PhpBench\Benchmark\Runner->runVariant(Object(Relay\PhpBench\RedisExecutor), Object(PhpBench\Registry\Config), Object(PhpBench\Benchmark\RunnerConfig), Object(PhpBench\Benchmark\Metadata\SubjectMetadata), Object(PhpBench\Model\Variant))
    #4 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/Benchmark/Runner.php(197): PhpBench\Benchmark\Runner->runSubject(Object(Relay\PhpBench\RedisExecutor), Object(PhpBench\Benchmark\RunnerConfig), Object(PhpBench\Model\Subject), Object(PhpBench\Benchmark\Metadata\SubjectMetadata))
    #5 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/Benchmark/Runner.php(123): PhpBench\Benchmark\Runner->runBenchmark(Object(PhpBench\Benchmark\RunnerConfig), Object(PhpBench\Model\Benchmark), Object(PhpBench\Benchmark\Metadata\BenchmarkMetadata))
    #6 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/Console/Command/Handler/RunnerHandler.php(141): PhpBench\Benchmark\Runner->run(Object(Generator), Object(PhpBench\Benchmark\RunnerConfig))
    #7 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/Console/Command/RunCommand.php(134): PhpBench\Console\Command\Handler\RunnerHandler->runFromInput(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\StreamOutput), Object(PhpBench\Benchmark\RunnerConfig))
    #8 /Users/Till/Development/Repositories/relay/vendor/symfony/console/Command/Command.php(299): PhpBench\Console\Command\RunCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\StreamOutput))
    #9 /Users/Till/Development/Repositories/relay/vendor/symfony/console/Application.php(978): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\StreamOutput))
    #10 /Users/Till/Development/Repositories/relay/vendor/symfony/console/Application.php(295): Symfony\Component\Console\Application->doRunCommand(Object(PhpBench\Console\Command\RunCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\StreamOutput))
    #11 /Users/Till/Development/Repositories/relay/vendor/symfony/console/Application.php(167): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\StreamOutput))
    #12 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/lib/PhpBench.php(51): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\StreamOutput))
    #13 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/bin/phpbench.php(25): PhpBench\PhpBench::run()
    #14 /Users/Till/Development/Repositories/relay/vendor/phpbench/phpbench/bin/phpbench(4): require('/Users/Till/Dev...')
    #15 {main}

This is how the @ParamProviders used to work:

class GetThroughputBench extends BenchCase
{
    /**
     * @ParamProviders("provideKeys")
     */
    public function benchGetThroughputOfRelayWarmed($params)
    {
        foreach ($params['keys'] as $key) {
            $this->relay->get($key);
        }
    }

    /**
     * Provides the keys for each benchmark.
     *
     * @return Generator
     */
    public function provideKeys()
    {
        $keys = [];

        foreach (static::loadJson('dataset-meteorites.json') as $landing) {
            $keys[] = "meteorite:{$landing->id}";
        }

        yield 'dataset' => [
            'keys' => $keys,
        ];
    }
}

Xdebug 3.1 enables compression of the cachegrind files automatically so the extension of the output by default is not .cachegrind but .cachegrind.gz.

This patch adds a check to see if compression is enabled and then uses .cachegrind.gz instead of .cachegrind as extension.

I have put a simple example benchark into benchmarks/CustomerSoapBench.php:

<?php

namespace POM\benchmarks;

use WebConnector\Customer\CustomerClientFactory;
use WebConnector\Customer\Type\Read;

class CustomerSoapBench {
    /**
     * @return void
     */
    public function benchRead() {
        $client = CustomerClientFactory::factory('https://'. WEBCONNECTOR_HOST .'/test_1094/CustomerIISWebService.svc?wsdl');
        $read = new Read('D83756');
        $client->read($read);
    }
}

placed my phpbench.json config beside it in benchmarks/ and invoked it from the project root like

$ php vendor/bin/phpbench run benchmarks/CustomerSoapBench.php --report=default --config=benchmarks/phpbench.json  -vvv
[DEBUG] Spawning process: C:\tools\php74\php.exe  "C:\Users\mstaab\AppData\Local\Temp\Php5FFA.tmp"

[DEBUG] Spawning process: C:\tools\php74\php.exe  "C:\Users\mstaab\AppData\Local\Temp\Php60D6.tmp"

PHPBench (dev-master) running benchmarks...
with configuration file: C:/dvl/GitHub/philipp/benchmarks/phpbench.json
with PHP version 7.4.13, xdebug ❌, opcache ❌

[DEBUG] Spawning process: C:\tools\php74\php.exe  "C:\Users\mstaab\AppData\Local\Temp\Php629D.tmp"

\POM\benchmarks\CustomerSoapBench

    benchRead...............................I0 [DEBUG] Spawning process: C:\tools\php74\php.exe -dmax_execution_time=0 "C:\Users\mstaab\AppData\Local\Temp\Php
6389.tmp"


In SyntaxError.php line 30:

  [PhpBench\Expression\Exception\SyntaxError]
  Could not find parselet for "none" token:

 " (" ~ rstdev(variant.time.avg) ~ ")"  mode) ~
        ----------------------------------------^


Exception trace:
  at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Exception\SyntaxError.php:30
 PhpBench\Expression\Exception\SyntaxError::forToken() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Parser.php:89
 PhpBench\Expression\Parser->parseExpression() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Parselet\ConcatParselet.php:23
 PhpBench\Expression\Parselet\ConcatParselet->parse() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Parser.php:105
 PhpBench\Expression\Parser->parseExpression() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Parselet\ConcatParselet.php:23
 PhpBench\Expression\Parselet\ConcatParselet->parse() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Parser.php:105
 PhpBench\Expression\Parser->parseExpression() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Parser.php:64
 PhpBench\Expression\Parser->parseList() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\Parser.php:49
 PhpBench\Expression\Parser->parse() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\ExpressionLanguage\RealExpressionLanguage.php:29
 PhpBench\Expression\ExpressionLanguage\RealExpressionLanguage->parse() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Expression\ExpressionLanguage\Me
moisedExpressionLanguage.php:31
 PhpBench\Expression\ExpressionLanguage\MemoisedExpressionLanguage->parse() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Progress\VariantSummaryForma
tter.php:73
 PhpBench\Progress\VariantSummaryFormatter->formatVariant() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Progress\Logger\PhpBenchLogger.php:157
 PhpBench\Progress\Logger\PhpBenchLogger->formatIterationsFullSummary() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Progress\Logger\VerboseLogger.ph
p:87
 PhpBench\Progress\Logger\VerboseLogger->variantEnd() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Benchmark\Runner.php:348
 PhpBench\Benchmark\Runner->endVariant() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Benchmark\Runner.php:310
 PhpBench\Benchmark\Runner->runVariant() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Benchmark\Runner.php:268
 PhpBench\Benchmark\Runner->runSubject() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Benchmark\Runner.php:196
 PhpBench\Benchmark\Runner->runBenchmark() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Benchmark\Runner.php:122
 PhpBench\Benchmark\Runner->run() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Console\Command\Handler\RunnerHandler.php:136
 PhpBench\Console\Command\Handler\RunnerHandler->runFromInput() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\Console\Command\RunCommand.php:134
 PhpBench\Console\Command\RunCommand->execute() at C:\dvl\GitHub\philipp\vendor\symfony\console\Command\Command.php:256
 Symfony\Component\Console\Command\Command->run() at C:\dvl\GitHub\philipp\vendor\symfony\console\Application.php:971
 Symfony\Component\Console\Application->doRunCommand() at C:\dvl\GitHub\philipp\vendor\symfony\console\Application.php:290
 Symfony\Component\Console\Application->doRun() at C:\dvl\GitHub\philipp\vendor\symfony\console\Application.php:166
 Symfony\Component\Console\Application->run() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\lib\PhpBench.php:51
 PhpBench\PhpBench::run() at C:\dvl\GitHub\philipp\vendor\phpbench\phpbench\bin\phpbench.php:25
 require() at C:\dvl\GitHub\philipp\vendor\bin\phpbench(21) : eval()'d code:3
 eval() at C:\dvl\GitHub\philipp\vendor\bin\phpbench:21

run [--filter FILTER] [--group GROUP] [--parameters PARAMETERS] [--assert ASSERT] [--format FORMAT] [--revs REVS] [-l|--progress PROGRESS] [-b|--bootstrap BOO
TSTRAP] [--executor EXECUTOR] [--stop-on-error] [--php-binary PHP-BINARY] [--php-config PHP-CONFIG] [--php-wrapper PHP-WRAPPER] [--php-disable-ini] [--report
REPORT] [-o|--output OUTPUT] [--ref REF] [--file FILE] [--time-unit TIME-UNIT] [--precision PRECISION] [--mode MODE] [-d|--dump-file [DUMP-FILE]] [--dump] [--
iterations ITERATIONS] [--warmup WARMUP] [-r|--retry-threshold RETRY-THRESHOLD] [--sleep SLEEP] [--tag TAG] [--store] [--tolerate-failure] [--] [<path>]

my benchmarks/phpbench.json file looks like

{
  "runner.bootstrap": "../app/portal/phpstan-bootstrap.php"
}

the phpstan-bootstrap.php is the same one I use successfully for phpstan and similar tools:

<?php

define('APP_ROOT', dirname(__DIR__, 2));
define('APP_NAME', basename(__DIR__));
define('APP_PATH', APP_ROOT.'/app/'.APP_NAME);
define('PUBLIC_PATH', APP_ROOT.'/public/'.APP_NAME);

$_SERVER['APP_MODE'] = 'development';
define('APP_MODE', $_SERVER['APP_MODE']);

// \cms\ViewTypeProvider requires a rocket path via vhost
$_SERVER['ROCKET_PATH'] = APP_ROOT.'/vendor/plugins/rocket/';

require_once APP_ROOT.'/vendor/autoload.php';
include_once APP_ROOT.'/config/app.php';

running on phpbench@dev-master from today at 8a5da09d52f44d223b467f98974b0bc5113f4c46

$ php -v
PHP 7.4.13 (cli) (built: Nov 24 2020 12:43:30) ( NTS Visual C++ 2017 x64 )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies

on windows bash

Related: https://wiki.php.net/rfc/opcache.no_cache

In practice, most benchmarks run with the CLI SAPI, which has no opcache by default.

I was wondering if phpbench should (at all) enforce opcache to be enabled, or warn when it isn't.

Raised from https://github.com/lyrixx/php-bench/pull/5#issuecomment-637418989

Bumps ramsey/composer-install from 1 to 2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

When using the throughput output the relative difference isn't working.

{
  "expressions": {
    "diff": "format(\"%.1fx (%.1f%%)\", mode(suite[subject_groups = [\"predis\"]][\"result_time_avg\"]) / mode(result_time_avg), percent_diff(mode(result_time_avg), mode(suite[subject_groups = [\"predis\"]][\"result_time_avg\"])))"
  }
}

When I run: ./vendor/bin/phpbench run --report=redis --filter=GetThroughputBench it works:

./vendor/bin/phpbench run --report=redis --filter=GetThroughputBench
PHPBench (1.2.6) running benchmarks... #standwithukraine
with configuration file: /Users/Till/Development/Repositories/relay/phpbench.json
with PHP version 8.1.11, xdebug ❌, opcache ❌

\CacheWerk\Relay\Benchmarks\Throughput\GetThroughputBench

    GET_Throughput_Predis...................I1 - Mo54,546ops/s (±2.24%)
    GET_Throughput_Credis...................I1 - Mo31,919ops/s (±1.16%)
    GET_Throughput_PhpRedis.................I1 - Mo33,001ops/s (±1.08%)
    GET_Throughput_Relay_NoCache............I1 - Mo59,426ops/s (±0.99%)
    GET_Throughput_Relay_WarmCache..........I1 - Mo4,934,810ops/s (±1.15%)

Subjects: 5, Assertions: 0, Failures: 0, Errors: 0
GetThroughputBench
+--------------------------------+---------+-----+--------+-----------+----------------+-----------------+
| subject                        | revs    | its | rstdev | mem_peak  | mode           | diff            |
+--------------------------------+---------+-----+--------+-----------+----------------+-----------------+
| GET_Throughput_Predis          | 50000   | 2   | ±2.24% | 1.073mb   | 54,546ops/s    | 1.0x (0.0%)     |
| GET_Throughput_Credis          | 50000   | 2   | ±1.16% | 1.208mb   | 31,919ops/s    | 0.6x (-41.5%)   |
| GET_Throughput_PhpRedis        | 50000   | 2   | ±1.08% | 808.544kb | 33,001ops/s    | 0.6x (-39.5%)   |
| GET_Throughput_Relay_NoCache   | 50000   | 2   | ±0.99% | 808.560kb | 59,426ops/s    | 1.1x (8.9%)     |
| GET_Throughput_Relay_WarmCache | 5000000 | 2   | ±1.15% | 808.560kb | 4,934,810ops/s | 90.5x (8947.1%) |
+--------------------------------+---------+-----+--------+-----------+----------------+-----------------+

But just running ./vendor/bin/phpbench run --report=redis skews it up:

./vendor/bin/phpbench run --report=redis
PHPBench (1.2.6) running benchmarks... #standwithukraine
with configuration file: /Users/Till/Development/Repositories/relay/phpbench.json
with PHP version 8.1.11, xdebug ❌, opcache ❌

\CacheWerk\Relay\Benchmarks\MgetBench

    MGET_Predis.............................I1 - Mo4.953ms (±1.01%)
    MGET_Credis.............................I1 - Mo7.148ms (±0.11%)
    MGET_PhpRedis...........................I1 - Mo6.127ms (±1.48%)
    MGET_Relay_NoCache......................I1 - Mo3.678ms (±0.54%)
    MGET_Relay_WarmCache....................I1 - Mo0.290ms (±3.29%)

\CacheWerk\Relay\Benchmarks\Get\GetStringsBench

    GET_Strings_Predis......................I1 - Mo38.305ms (±0.60%)
    GET_Strings_Credis......................I1 - Mo66.388ms (±1.17%)
    GET_Strings_PhpRedis....................I1 - Mo64.149ms (±0.56%)
    GET_Strings_Relay_NoCache...............I1 - Mo34.167ms (±0.36%)
    GET_Strings_Relay_WarmCache.............I1 - Mo0.301ms (±0.58%)

\CacheWerk\Relay\Benchmarks\Get\GetUnserializeBench

    GET_Unserialize_Predis..................I1 - Mo39.697ms (±1.13%)
    GET_Unserialize_Credis..................I1 - Mo69.264ms (±1.65%)
    GET_Unserialize_PhpRedis................I1 - Mo70.536ms (±0.89%)
    GET_Unserialize_Relay_NoCache...........I1 - Mo36.880ms (±1.91%)
    GET_Unserialize_Relay_WarmCache.........I1 - Mo0.484ms (±2.52%)

\CacheWerk\Relay\Benchmarks\Throughput\GetThroughputBench

    GET_Throughput_Predis...................I1 - Mo54,231ops/s (±3.38%)
    GET_Throughput_Credis...................I1 - Mo30,230ops/s (±1.04%)
    GET_Throughput_PhpRedis.................I1 - Mo33,483ops/s (±0.64%)
    GET_Throughput_Relay_NoCache............I1 - Mo59,335ops/s (±0.47%)
    GET_Throughput_Relay_WarmCache..........I1 - Mo4,958,762ops/s (±1.01%)

Subjects: 20, Assertions: 0, Failures: 0, Errors: 0
MgetBench
+----------------------+------+-----+--------+----------+---------+-------------------+
| subject              | revs | its | rstdev | mem_peak | mode    | diff              |
+----------------------+------+-----+--------+----------+---------+-------------------+
| MGET_Predis          | 50   | 2   | ±1.01% | 1.342mb  | 4.953ms | 7.7x (669.3%)     |
| MGET_Credis          | 50   | 2   | ±0.11% | 1.476mb  | 7.148ms | 5.3x (433.1%)     |
| MGET_PhpRedis        | 50   | 2   | ±1.48% | 1.074mb  | 6.127ms | 6.2x (522.0%)     |
| MGET_Relay_NoCache   | 50   | 2   | ±0.54% | 1.074mb  | 3.678ms | 10.4x (936.1%)    |
| MGET_Relay_WarmCache | 50   | 2   | ±3.29% | 1.074mb  | 0.290ms | 131.3x (13026.5%) |
+----------------------+------+-----+--------+----------+---------+-------------------+

GetStringsBench
+-----------------------------+------+-----+--------+----------+----------+-------------------+
| subject                     | revs | its | rstdev | mem_peak | mode     | diff              |
+-----------------------------+------+-----+--------+----------+----------+-------------------+
| GET_Strings_Predis          | 10   | 2   | ±0.60% | 1.330mb  | 38.305ms | 1.0x (-0.5%)      |
| GET_Strings_Credis          | 10   | 2   | ±1.17% | 1.465mb  | 66.388ms | 0.6x (-42.6%)     |
| GET_Strings_PhpRedis        | 10   | 2   | ±0.56% | 1.065mb  | 64.149ms | 0.6x (-40.6%)     |
| GET_Strings_Relay_NoCache   | 10   | 2   | ±0.36% | 1.065mb  | 34.167ms | 1.1x (11.5%)      |
| GET_Strings_Relay_WarmCache | 10   | 2   | ±0.58% | 1.065mb  | 0.301ms  | 126.5x (12549.1%) |
+-----------------------------+------+-----+--------+----------+----------+-------------------+

GetUnserializeBench
+---------------------------------+------+-----+--------+----------+----------+-----------------+
| subject                         | revs | its | rstdev | mem_peak | mode     | diff            |
+---------------------------------+------+-----+--------+----------+----------+-----------------+
| GET_Unserialize_Predis          | 10   | 2   | ±1.13% | 1.331mb  | 39.697ms | 1.0x (-4.0%)    |
| GET_Unserialize_Credis          | 10   | 2   | ±1.65% | 1.467mb  | 69.264ms | 0.6x (-45.0%)   |
| GET_Unserialize_PhpRedis        | 10   | 2   | ±0.89% | 1.065mb  | 70.536ms | 0.5x (-46.0%)   |
| GET_Unserialize_Relay_NoCache   | 10   | 2   | ±1.91% | 1.065mb  | 36.880ms | 1.0x (3.3%)     |
| GET_Unserialize_Relay_WarmCache | 10   | 2   | ±2.52% | 1.065mb  | 0.484ms  | 78.8x (7775.8%) |
+---------------------------------+------+-----+--------+----------+----------+-----------------+

GetThroughputBench
+--------------------------------+---------+-----+--------+-----------+----------------+-------------------------+
| subject                        | revs    | its | rstdev | mem_peak  | mode           | diff                    |
+--------------------------------+---------+-----+--------+-----------+----------------+-------------------------+
| GET_Throughput_Predis          | 50000   | 2   | ±3.38% | 1.073mb   | 54,231ops/s    | 2066.5x (206546.3%)     |
| GET_Throughput_Credis          | 50000   | 2   | ±1.04% | 1.208mb   | 30,230ops/s    | 1151.9x (115091.6%)     |
| GET_Throughput_PhpRedis        | 50000   | 2   | ±0.64% | 808.544kb | 33,483ops/s    | 1275.9x (127485.1%)     |
| GET_Throughput_Relay_NoCache   | 50000   | 2   | ±0.47% | 808.560kb | 59,335ops/s    | 2261.0x (225995.5%)     |
| GET_Throughput_Relay_WarmCache | 5000000 | 2   | ±1.01% | 808.560kb | 4,958,762ops/s | 188953.7x (18895269.2%) |
+--------------------------------+---------+-----+--------+-----------+----------------+-------------------------+

Maybe related to #974. The code: https://github.com/cachewerk/relay

Bumps actions/setup-python from 3 to 4.1.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

In a cli report table, to properly compare numbers, columns with numbers should:

• be aligned right (\STR_PAD_LEFT).
• use the same measuring unit (e.g. not ms in one record, but µs in another record).
• ideally, a unit should be used where all values are > 1, so that big numbers visually stick out with additional digits to the left. (for these last points I currently use --mode=throughput --time-unit=s as cli parameters)

For the first point, I found that the following addition in TableRenderer::render() has the desired effect for the main table:

        $style = (new TableStyle())->setPadType(\STR_PAD_LEFT);
        $nColumns = \count($object->columnNames());
        for ($iCol = 2; $iCol < $nColumns; ++$iCol) {
            $consoleTable->setColumnStyle($iCol, $style);
        }

However, this is obviously the wrong place to do this. I am hard-coding the rule that every column except the first two should be aligned right, which is true for the final overview table but not universally for every table.

What the code example shows is that the symfony console table component can align numbers right.

Problem

I have a test case like this:

class MyBench {
  public function benchA1() {...}
  public function benchB1() {...}
  public function benchA1() {
    ... // Exact copy of benchA1().
  }
  public function benchB1() {
    ... // Exact copy of benchB1().
  }
}

When running all of them, the later (copied) version is notoriously slower than the previous version. E.g. A2 will be slower than A1, and B2 will be slower than B1. I saw differences higher than 10%, and this was reproducible.

I assume this is because the processor gets heated and will throttle when it hits the later benchmark cases.

This can lead to misleading results when comparing the performance of different algorithms.

Solution 1: Sleep

We can use the --sleep parameter or @Sleep annotribute, hoping that the processor will cool down. I tried with --revs=1 --iterations=50 --sleep=10000, it seems to improve stability sometimes, but the problem does not fully go away. Perhaps I need to use higher sleep numbers?

One problem with this is that a user won't see if the sleep number was "high enough", because usually there are no 1:1 copied benchmark methods.

Solution 2: Mix it up

In the past, I created a custom benchmark tool where I would break the strict ordering of benchmark cases. E.g. I would run A, B, A, B, A, B instead of A, A, A, B, B, B.

While this makes a more "fair" comparison of A vs B in the current run, it does not help with differences to earlier runs, where the CPU might have been less heated.

Solution 3: Pre-stress the processor

Heat up the processor so that all the benchmark cases suffer from the same throttling.

I don't know if a processor will reach a fixed throttling level, or if it will throttle more and more. I assume it depends on many factors.

Solution 4: Measure current raw CPU speed?

A reference operation could be used to measure the current CPU speed. A report could show benchmark times divided by the duration for the reference operation.

Problems with this:

• The divided number will be an imaginary unit, because the reference operation is completely arbitrary.
• There might be high variation in the reference operation measurements.
• Resources used for the reference operation might be different than for the benchmark operation, so the division might distort the results.