Event driven BDD test framework for PHP

After a composer update moved us up to the new version today, our test that were working in 1.14 just did nothing. Command line would report a fail on the command.

The peridot.php:

<?php //peridot.php
//For using prophecy in peridot in testing.  Allows it to be used globally
require(dirname(__FILE__).'/../../vendor/autoload.php');
ini_set('display_errors',0);
error_reporting(0);
use Evenement\EventEmitterInterface;
use Peridot\Plugin\Prophecy\ProphecyPlugin;
use Peridot\Plugin\Watcher\WatcherPlugin;

return function(EventEmitterInterface $emitter) {
    $plugin = new ProphecyPlugin($emitter);
    $watcher = new WatcherPlugin($emitter);
    $watcher->track(__DIR__ . '/../../OS');
};
?>

Hey there! Since I'm a big fan of testing with peridot, I can't help but notice there weren't that many updates as of late. I need Symfony Console >v4 in some of my projects, but can't, since Peridot still requires the v2.
Hence I'd like to propose updating all dependencies to more recent versions, as well as raising the PHP version requirement to >= 7.1.

I know, that shuns backwards compatibility. However, the definite EOL of PHP 5.6 is just ahead (Dec. 31st I think), so I'd say it's definitely okay to cut loose ends, too. Maybe in the shape of a new major version?

Let me know what you think.

Forgot to mention it - I ran the full test suite, which is all green with some minor modifications.

Continuing from #175, this PR further improves the error handling in Peridot.

The biggest change is that when an error occurs in a test, that test will now fail. This new behavior will honor the error_reporting setting, meaning that systems that typically ignore errors of a given severity can still be tested under Peridot. This is the same underlying mechanism used by the error control operator (@), so that will also work as intended.

Regardless of the error_reporting level, a Peridot error event will still be emitted. It will be up to individual listener implementations to decide whether to honor the error_reporting level.

Another change is to the behavior of tests in which multiple failures occur. Previously, if a test failed AND its teardown also failed, the teardown failure would "overwrite" the test failure. This behavior has changed so that the first failure is always the failure that is reported. This has particular significance given that a test can now fail because of an error, and then continue on to fail because of an assertion failure or other exception.

I also added support for PHP 7 engine exceptions. Note that I have polyfilled the existence of Throwable and Error, so that the tests may also be run under PHP 5.

This PR should also solve #158 and #132.

I have a multi-language project, so I want to organize my tests inside a main test folder.

test/
  - php/
    - peridot.php
    - spec/
  - nodejs/
    - mocha.js
    - spec/

I know that I can use --configuration=test/php/peridot.php to tell it where to look for the config. But how do I configure the folder containing tests?

I suspect this issue may be related to issue 174, but I didn't want to hijack their issue, since it may not be related at all.

Error: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 3072 bytes).

Recently, I created new test file and was running fine locally, but when I created the pull request, the tests failed on circleci. I believe circleci's memory limit is 128 (locally is 512), because when I lower local's memory limit to 128, the tests run out of memory as well.

That was when I discovered how scopes were leaking as described in issue 174. We suspected that may be the issue and added an suite.end event that removed the properties from the scope. However, the issue still persists. I var_dumped the scope after unsetting the properties and noticed there is a ProphecyScope and it still has methodProphecies created by our tests.

I'm wondering if perhaps the ProphecyScope is just getting bigger over the course of the tests and finally runs out of memory. Is there a way to reset/clear a ProphecyScope? Perhaps another suggestion to what the issue is?

Below is the code for our peridot.php file.

return function(EventEmitterInterface $emitter) {
    $plugin = new ProphecyPlugin($emitter);
    $watcher = new WatcherPlugin($emitter);
    $watcher->track(__DIR__ . '/../../OS');

    $testingScope = new TestingScope();

    $emitter->on('suite.start', function($test)use ($testingScope){
        $test->getScope()->peridotAddChildScope($testingScope);
    });

    $emitter->on('suite.end', function($test){
        $scope = $test->getScope();

        foreach($scope as $key => $value){
            unset($scope->$key);
        }
    });
};

Thanks for any suggestions.

This adds the test alias for it and the other variants. This is similar to facebook's Jest. I think there are some cases where test can be more suiting fo the test description.

Specifically, I've created a package which generates markdown documentation from your specs, and I've found that using it can make the wording weird, but test is more natural.

Btw, thank you so much for making this package, it's such a great tool!

Signed-off-by: RJ Garcia [email protected]

I feel like I have to be missing something painfully simple here, but can't for the life of me figure it out. Started out by pulling in Peridot via Composer (along with a few plugins):

"require-dev": {
    "mockery/mockery": "dev-master",
    "squizlabs/php_codesniffer": "~1.5",
    "peridot-php/peridot": "~1.15",
    "peridot-php/peridot-list-reporter": "^1.0",
    "peridot-php/peridot-code-coverage-reporters": "^2.0"
}

I then created a specs/ folder and within it a Sample.spec.php file containing the sample code from the GitHub page:

describe('ArrayObject', function() {
    beforeEach(function() {
        $this->arrayObject = new ArrayObject(['one', 'two', 'three']);
    });

    describe('->count()', function() {
        it("should return the number of items", function() {
            $count = $this->arrayObject->count();
            assert($count === 3, "expected 3");
        });
    });
});

All looks peachy:

$ vendor/bin/peridot specs/Sample.spec.php

  ArrayObject
      ->count()
        ✓ should return the number of items


  1 passing (0 ms)

However, if I change the assertion in attempt to make the test fail...

describe('ArrayObject', function() {
    beforeEach(function() {
        $this->arrayObject = new ArrayObject(['one', 'two', 'three']);
    });

    describe('->count()', function() {
        it("should return the number of items", function() {
            $count = $this->arrayObject->count();
            assert($count === 1, "expected 3");
        });
    });
});

...the test still passes:

$ vendor/bin/peridot specs/Sample.spec.php

  ArrayObject
      ->count()
        ✓ should return the number of items


  1 passing (0 ms)

Using the List reporter also shows the test as passing:

$ vendor/bin/peridot specs/Sample.spec.php -r list
  ArrayObject ->count() should return the number of items


  1 passing (0 ms)

System info (Linux Mint 18):

$ vendor/bin/peridot --version
Peridot version 1.18.1
$ php --version
PHP 7.0.4-7ubuntu2.1 (cli) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
$ composer show
evenement/evenement                         v2.0.0             Événement is a very simp...
hamcrest/hamcrest-php                       v2.0.0             This is the PHP port of Ha...
mockery/mockery                             dev-master ee06e7b Mockery is a simple yet fl...
peridot-php/peridot                         1.18.1             Event driven BDD test fram...
peridot-php/peridot-code-coverage-reporters 2.0.1              Code coverage reporters fo...
peridot-php/peridot-list-reporter           1.0.0              A list reporter for the Pe...
peridot-php/peridot-scope                   1.3.0              Scopes for function bindin...
phpunit/php-code-coverage                   2.2.4              Library that provides coll...
phpunit/php-file-iterator                   1.4.1              FilterIterator implementat...
phpunit/php-text-template                   1.2.1              Simple template engine.
phpunit/php-timer                           1.0.8              Utility class for timing
phpunit/php-token-stream                    1.4.8              Wrapper around PHP's token...
sebastian/environment                       1.3.7              Provides functionality to ...
sebastian/version                           1.0.6              Library that helps with ma...
squizlabs/php_codesniffer                   1.5.6              PHP_CodeSniffer tokenises ...
symfony/console                             v3.1.2             Symfony Console Component
symfony/polyfill-mbstring                   v1.2.0             Symfony polyfill for the M...

I use Asplode in my peridot.php to set up an error handler that converts errors into error exceptions. I only recently realised that this was not working, and that my tests were passing silently despite errors.

I see two problems here:

1. Peridot is currently overwriting and ignoring any existing error handler.
2. Peridot is currently silencing errors, because the error handler doesn't actually do anything except to raise a Peridot "error" event. That seems like a dangerous choice. Shouldn't a test fail if an error occurs during its execution, and shouldn't that happen without having to write a custom "error" event handler?

This PR addresses these two problems in the following ways:

1. If there is an existing error handler, Peridot will now "forward on" the error to the existing handler. This includes using the existing error handler's return value, which is important (see below).
2. If there is no existing error handler, Peridot's error handler will now return false to indicate that the error should propagate to PHP's inbuilt error handler. This means that the error will at least be displayed to the user, instead of being silenced completely.

This does not address the issue of whether tests should fail due to errors, but that's probably another discussion / PR.

This PR adds support for multiple reporters.

Summary/justification of changes:

• This is a not a BC break, unless you count setting the default value of the reporter command-line option in peridot.php.
• Multiple reporters can now be specified on the command line with multiple --reporter options.
  • The docs will need a minor update to reflect this.
• When multiple reporters are specified, they are composed into a 'composite' reporter.
  • The first reporter gets the "real" output interface, and subsequent reporters get a buffered output interface. Once the runner.end event fires, any buffered output is displayed after the primary reporter's output, separated by a single newline. This prevents interlaced output, while retaining streaming output for the primary reporter.
• Retains support for getReporter() and setReporter().
  • We can deprecate them with the next major release, and remove them in the one after that. I'm not sure how we want to handle deprecations though.
• I'm not sure what the PERIDOT_REPORTER environment variable is used for, but I also retained support for it, and added a similar PERIDOT_REPORTERS environment variable, which contains all the reporter names, separated by comma.

Closes #186.

Switches us over to using codecov

@ezzatron - I took a cue from you and introduced a Makefile. codecov says it aggregates reports across builds, so I thought it would be good to let code coverage run for all build targets in the matrix (with the exception of allowed failures)

The Makefile currently only defines a coverage target, but we can define more for other build tasks as we need them.

The coverage target is conditionally defined to skip nightly builds and to use phpdbg for PHP 7.0 and 7.1 (as it is the only driver supported for those versions with our version of phpunit/php-code-coverage)

Currently peridot provide only ability to filter by file. What I propose is additional --filter options for CLI which will allows to filter by test case description. For example:

describe('Some Component', function () {
    it('It works in most common cases (critical)', function () {
         // test case 1
    });

    it('It works in other cases', function () {
        // test case 2
    });
});

So by running

peridot --filter "\(critical\)" specs/ 

We will get only test case 1 executed. I could prepare a PR but I'm not sure about implementation and maybe this should be just plugin and not built in feature.

After a while of thinking about what a peridot 2.0 should look like, I don't think i want to try and compete with Kahlan and PHPUnit. Peridot 1.0 is basically a lite version of kahlan with a smaller community and resources behind it. I don't really see any area that peridot can try to tap into that Kahlan userbase in a way that Kahlan isn't already doing.

The describe-it syntax is very popular in other languages and personally, I do like how expressive the system can be. That being said however, PHPUnit by far is the defacto testing framework and majority of the PHP community resources and integrations revolve around PHPUnit. If kahlan hasn't taken off by now, then it's very likely that the php community won't be ready to rally around a new framework.

Personally, I actually like that the PHP community largely pools our resources into one testing framework. It's very attractive for new comers and also means that the community can focus our efforts on growing the PHPUnit ecosystem instead of dilluting our efforts among competing frameworks.

All that said, I do think there might a space for a new frontend for PHPUnit. A library that piggy backs off the entire phpunit framework but provides a different interface for writing tests with the describe-it syntax that the peridot/kahlan community enjoys.

I think peridot 2.0 would be a good place to explore this integration with phpunit. It'd allow us to focus on the parts of testing framework that are interesting and leave the heavy lifting to phpunit all the while piggy backing off of the wealth of documentation and third party plugins that phpunit provides.

In a similar manner, we'd probably make peridot/leo be just a thin wrapper around the PHPUnit assertions library and include in the peridot 2.0 core.

In the long run, I'd love to see peridot's new phpunit frontend be brought under the fold of PHPUnit with first class support and apart of the phpunit ecosystem instead of a separate library.

Let me know what you all think about this idea, welcome to suggestions :)

Is it possible to implement these with current API and add it to current DSL?

beforeAll Runs a function before any of the tests in this file run. Example in Jest https://jestjs.io/docs/en/setup-teardown#one-time-setup

Repeating Setup For Many Tests Run setup for each test, for ex. to cleanup database and cache storage. So it's a global beforeEach. Example in Jest https://jestjs.io/docs/en/setup-teardown#repeating-setup-for-many-tests

I think the concept of data providers is something that lots of people will look for when switching from PHPUnit to Peridot. I think I brought it up with @brianium during our first get-together, and there's #183 and peridot-php/peridot-core#16 related to this issue already.

I had a conversation with my friend @koden-km about how to migrate some PHPUnit tests that use Data Providers over to Peridot.

We came up with a solution that utilized an associative array and a foreach loop. Here's some sample code:

describe('should produce a double double when there are double digit statistics for', function () {
    /**
     * Builds data for every combination of stats that can be a double double.
     */
    $doubleDoubleDataProvider = function () {
        $statistics = ['points', 'rebounds_defensive', 'assists', 'steals', 'blocks'];
        $statisticsCount = count($statistics);
        $data = [];

        for ($i = 0; $i < $statisticsCount; ++$i) {
            for ($j = $i + 1; $j < $statisticsCount; ++$j) {
                $subStatistics = [$statistics[$i], $statistics[$j]];
                $data[implode(', ', $subStatistics)] = $subStatistics;
            }
        }

        return $data;
    };

    foreach ($doubleDoubleDataProvider() as $label => $statistics) {
        it($label, function () use ($statistics) {
            $statistics = array_fill_keys($statistics, 10);
            $statistics['games_played'] = 1;
            $actual = $this->subject->calculateStatistics($statistics);

            expect($actual->doubleDoubles)->to->equal(1);
            expect($actual->tripleDoubles)->to->equal(0);
        });
    }
});

This produces output like this:

should produce a double double when there are double digit statistics for
  ✓ points, rebounds_defensive
  ✓ points, assists
  ✓ points, steals
  ✓ points, blocks
  ✓ rebounds_defensive, assists
  ✓ rebounds_defensive, steals
  ✓ rebounds_defensive, blocks
  ✓ assists, steals
  ✓ assists, blocks
  ✓ steals, blocks

This approach isn't too bad, but I think that's partly because this scenario only has one input to each test ($statistics). If each test required multiple inputs, either the inner closure would have to expand its list of use'd variables, or it would have to accept all inputs in a single use'd array/object, then access each by index/property.

I looked for some prior art to guide us in implementing this feature. It seems as though the typical JavaScript approach is similar to this:

[1,2,3].forEach(function (itemNumber) {
    describe("Test # " + itemNumber, function () {
        it("should be a number", function (done) {
            expect(itemNumber).to.be.a('number')
            expect(itemNumber).to.be(itemNumber) 
        });
    });
});

Unfortunately, this doesn't map to PHP very well because of the fact that PHP's closures don't automatically inherit variables from the parent scope. That is, at least until the Arrow Functions RFC is implemented.

Maybe this calls for a new Peridot feature that could somehow exploit setDefinitionArguments(). There's a definite challenge though, as plugins like peridot-phony also use this method for injecting other arguments.

Even if we make no code changes, I think having some solid documentation for how to achieve this kind of setup would help newcomers to justify the switch to Peridot.

Currently in some parts of the code, such as the new composite reporter, we assume that a reporter will have the setEventEmitter() method, which comes from HasEventEmitterTrait that we add in AbstractTest:

        array_map(function (ReporterInterface $reporter) use ($eventEmitter) {
            $reporter->setEventEmitter($eventEmitter);
        }, $this->reporters);

There's been some other related stuff where we typehint on abstract classes when we should really use an interface instead. I think we need to audit our typehints, and interfaces.