Athletic
Athletic is a benchmarking framework. It allows developers to benchmark their code without littering microtime() calls everywhere.
Athletic was inspired by the annotation format that PHPUnit uses. Benchmark tests extend the AthleticEvent class and are annotated with specific docblock parameters. The benchmark suite is then run with the Athletic command-line tool.
| Branch | Unit Tests | Coverage |
|---|---|---|
 |
 |
 |
WARNING: No longer maintained
Athletic is not currently being maintained, so it may have bugs or other problems. I would recommend using PhpBench, which was inspired by Athletic. It is far more capable, and actively maintained
Why Benchmark?
Because fast code is good! While premature optimization is certainly evil, optimization is always an important component of software development. And sometimes you just really need to see if one solution to a problem is faster than an alternative.
Why Use Athletic?
Because it makes benchmarking easy! Athletic is built around annotations. Simply create a benchmarking class and annotate a few methods:
/**
* @iterations 1000
*/
public function fastIndexingAlgo()
{
$this->fast->index($this->data);
}
Without Athletic, you have to litter your code with microtime() calls and build timing metrics yourself. Or you end up building a benchmark harness remarkably similar to Athletic (but probably with less syntactic sugar...because who builds throw-away code to read test annotations and fancy output?)
Why can't I use xDebug?
xDebug is an excellent profiling tool, but it is not a benchmarking tool. xdebug (and by extension, cachegrind) will show you what is fast/slow inside your method, and is indispensable for actually optimizing your code. But it is not useful for running 1000 iterations of a particular function and determining average execution time.
Quick Installation via Composer
You can easily install Athletic through Composer in two steps:
# Install Composer
curl -sS https://getcomposer.org/installer | php
# Add Athletic as a dev dependency
php composer.phar require athletic/athletic:~0.1 --dev
You can find out more on how to install Composer, configure autoloading, and other best-practices for defining dependencies at getcomposer.org.
Usage
To begin using Athletic, you must create an Event. This is the analog of a PHPUnit Test Case. An Event will benchmark one or more functions related to your code, compile the results and output them to the command line.
Here is a sample Event:
<?php
namespace Vendor\Package\Benchmarks\Indexing;
use Vendor\Package\Indexing;
use Athletic\AthleticEvent;
class IndexingEvent extends AthleticEvent
{
private $fast;
private $slow;
private $data;
public function setUp()
{
$this->fast = new Indexing\FastIndexer();
$this->slow = new Indexing\SlowIndexer();
$this->data = array('field' => 'value');
}
/**
* @iterations 1000
*/
public function fastIndexingAlgo()
{
$this->fast->index($this->data);
}
/**
* @iterations 1000
*/
public function slowIndexingAlgo()
{
$this->slow->index($this->data);
}
}
Let's look at how this works.
<?php
namespace Vendor\Package\Benchmarks\Indexing;
use Vendor\Package\Indexing;
use Athletic\AthleticEvent;
First, we have a PHP file that is included in your project's repo, just like unit tests. In this example, the Event is saved under the Vendor\Package\Benchmarks\Indexing namespace. It uses classes from your project located at Vendor\Package\Indexing. It also uses a class from the Athletic framework.
class IndexingEvent extends AthleticEvent
{
Next, we declare an indexing class that extends \Athletic\AthleticEvent. This is important because it tells Athletic that this class should be benchmarked. AthleticEvent is an abstract class that provides code to inspect your class and actually run the benchmarks.
private $fast;
private $slow;
private $data;
public function setUp()
{
$this->fast = new Indexing\FastIndexer();
$this->slow = new Indexing\SlowIndexer();
$this->data = array('field' => 'value');
}
Next, we have some private variables and a setUp() method. The setUp() method is invoked once at the beginning of each benchmark iteration. This is a good place to instantiate variables that are important to the benchmark itself, populate data, build database connections, etc. In this example, we are building two "Indexing" classes and a sample piece of data. (More details about setup and tear down are further down in this document)
/**
* @iterations 1000
*/
public function fastIndexingAlgo()
{
$this->fast->index($this->data);
}
/**
* @iterations 1000
*/
public function slowIndexingAlgo()
{
$this->slow->index($this->data);
}
Finally, we get to the meat of the benchmark. Here we have two methods that are annotated with @iterations in the docblock. The @iterations annotation tells Athletic how many times to repeat the method. If a method does not have an iterations annotation, it will not be benchmarked.
That's it! Now you are ready to run the benchmark.
Running Athletic
A benchmark test is run from the command line:
$ php ./vendor/bin/athletic -p /home/ProjectDir/benchmarks/ -b /home/ProjectDir/vendor/autoload.php
The tool has a few flags that can be set:
| Flag | Long Form | Required | Description |
|---|---|---|---|
| -p | --path | Yes | Specifies the path to the Events to benchmark. Will recursively load all files/classes that extend AthleticEvent |
| -b | --bootstrap | Sets the path to an optional bootstrap file which is included before anything else. This is often used to include an autoloader for your project. | |
| -f | --formatter | User-configured formatter to use instead of DefaultFormatter | |
| -h | --help | Help screen with options and their descriptions |
Note: Athletic is intended to be used as a single Phar archive, but that process has not been built yet. Soon!
Output
So what does the output of a benchmark look like?
$ php ./vendor/bin/athletic -p /home/ProjectDir/benchmarks/ -b /home/ProjectDir/vendor/autoload.php
Vendor\Package\Benchmarks\Indexing\IndexingEvent
Method Name Iterations Average Time Ops/second
--------------------- ------------ -------------- -------------
fastIndexingAlgo: [1000 ] [0.0020904064178] [478.37588]
slowIndexingAlgo: [1000 ] [0.0048114223480] [177.59184]
The default formatter outputs the Event class name, each method name, the number of iterations, average time and operations per second. More advanced formatters will be created in the near future (CSVFormatter, database export, advanced statistics, etc).
Further Information
SetUps and TearDowns
Athletic offers several methods to setup and tear down data/variables.
| Method | Description |
|---|---|
| classSetUp() | Invoked at the beginning of the Event before anything else has occurred |
| setUp() | Invoked once before each iteration of the method being benchmark. |
| classTearDown() | Invoked at the end of the event after everything else has occurred. |
| tearDown() | Invoked after each iteration of a benchmark has completed. |
There are two levels of setup and tear down to prevent "state leakage" between benchmarks. For example, an object that caches calculations will perform faster on subsequent calls to the method.
If the goal is to benchmark the initial calculation, it makes sense to place the instantiation of the object in setUp().
If the goal, however, is to benchmark the entire process (initial calculation and subsequent caching), then it makes more sense to instantiate the object in classSetUp() so that it is only built once.
Calibration
Athletic uses Reflection and variable functions to invoke the methods in your Event. Because there is some internal overhead to variable functions, Athletic performs a "calibration" step before each iteration. This step calls an empty calibration method and times how long it takes. This time is then subtracted from the iterations total time, providing a more accurate total time.
![[Feature request] tests comparison](https://avatars.githubusercontent.com/u/154256?v=4)
2.1k Jan 8, 2023
15.9k Jan 3, 2023
801 Dec 25, 2022
765 Dec 30, 2022
6.5k Jan 3, 2023
22 Nov 24, 2022
192 Dec 10, 2022
11.6k Jan 3, 2023
9.9k Jan 4, 2023
11.6k Dec 30, 2022
5.4k Jan 7, 2023
3.9k Jan 1, 2023
2.3k Dec 22, 2022
1.1k Dec 15, 2022
562 Dec 30, 2022
554 Nov 25, 2022
542 Dec 24, 2022