Full-stack testing PHP framework

Codeception Testing Framework

Last update: Jan 7, 2023

Related tags

Testing testing php unit-testing acceptance-testing bdd end-to-end integration-testing phpunit codeception

Overview

Codeception

Modern PHP Testing for everyone

Codeception is a modern full-stack testing framework for PHP. Inspired by BDD, it provides an absolutely new way of writing acceptance, functional and even unit tests. Powered by PHPUnit.

Build	Webdriver

Contributions

At Codeception we are glad to receive contributions from the community. If you want to send additions or fixes to the code or the documentation please check the Contributing guide.

At a Glance

Describe what you test and how you test it. Use PHP to write descriptions faster.

Run tests and see what actions were taken and what results were seen.

Sample test

$I->amOnPage('/');
$I->click('Pages');
$I->click('New');
$I->see('New Page');
$I->submitForm('form#new_page', ['title' => 'Movie Review']);
$I->see('page created'); // notice generated
$I->see('Movie Review','h1'); // head of page of is our title
$I->seeInCurrentUrl('pages/movie-review'); // slug is generated
$I->seeInDatabase('pages', ['title' => 'Movie Review']); // data is stored in database

For unit testing you can stay on classic PHPUnit tests, as Codeception can run them too.

Installation

Composer

php composer.phar require "codeception/codeception"

TODO: Document how to install the modules, e.g.

php composer.phar require "codeception/module-phpbrowser"

Phar

Download codecept.phar

Copy it into your project.

You can also make Codeception an executable and it put it into your $PATH, for instance:

wget http://codeception.com/codecept.phar

chmod +x codecept.phar

sudo mv codecept.phar /usr/local/bin/codecept

You can then run Codecept in the command line using: codecept bootstrap, codecept run, etc

Run CLI utility:

php codecept.phar

Getting Started

After you successfully installed Codeception, run this command:

codecept bootstrap

This will create a default directory structure and default test suites.

Documentation

View Documentation

The documentation source files can be found at https://github.com/Codeception/codeception.github.com/tree/master/docs/.

License

MIT

Comments

Suggestion: Generate namespaced Cest files by default
I'm guessing that the majority of framework users wants their tests to be namespaced properly. Right?

Documentation is poor (not to say: inexistant), with https://codeception.com/docs/08-Customization#Namespaces being the only reference I found.

So here's what I did, and I'm suggesting to make this the default behavior of codecept g:cest functional First, to save people from having to copy-paste it into their Cest files later on:

Add this to composer.json (this might already be implemented in recent versions):
"autoload-dev": { "psr-4": { "Tests\\": "tests/" } },

Add this to codeception.yml:
namespace: Tests

Add this on top of every (functional) Cest file:
namespace Tests\functional; use Tests\FunctionalTester;

I'm not 100% sure if this is the best way to do it, so please doublecheck.

This would also solve https://github.com/Codeception/Codeception/issues/4874
opened by ThomasLandauer 43
[Yii2] Used database connections are not getting closed
What are you trying to achieve?

Trying to run suite of tests

What do you get instead?

Failures

Provide console output if related. Use -vvv mode for more details.

- ManagerTest: Check oem company monitor Destroying application Starting application [Fixtures] Loading fixtures [yii\db\Connection::open] 'Opening DB connection: pgsql:host=localhost;port=5432;dbname=front_test' [Fixtures] Opened database connection: pgsql:host=localhost;port=5432;dbname=front_test [yii\db\Connection::open] 'Opening DB connection: pgsql:host=localhost;port=5432;dbname=back_test' [Fixtures] Opened database connection: pgsql:host=localhost;port=5432;dbname=back_test [Fixtures] Closing database connection: pgsql:host=localhost;port=5432;dbname=front_test [Fixtures] Closing database connection: pgsql:host=localhost;port=5432;dbname=back_test [Fixtures] Done [yii\db\Connection::open] 'Opening DB connection: pgsql:host=localhost;port=5432;dbname=front_test' [yii\web\Session::open] 'Session started' [yii\web\User::login] 'User \'1256\' logged in from with duration 86400.'

Eventually get this

--------- 300) ApiUserTesterCest: Test the forgot password request Test tests/api/ApiUserTesterCest.php:ForgotPasswordRequest [yii\db\Exception] SQLSTATE[08006] [7] FATAL: sorry, too many clients already /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Connection.php:624 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Connection.php:687 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Connection.php:613 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Connection.php:996 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Connection.php:983 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Schema.php:463 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Connection.php:881 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Command.php:209 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Command.php:1099 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Command.php:1120 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Command.php:442 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/pgsql/Schema.php:182 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Schema.php:237 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/pgsql/QueryBuilder.php:200 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/db/Command.php:959 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/test/InitDbFixture.php:96 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/test/InitDbFixture.php:78 /srv/www/xyzweb/src/private/protected/vendor/yiisoft/yii2/test/FixtureTrait.php:117 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Module/Yii2.php:467 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Module/Yii2.php:285 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Module/Yii2.php:259 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Subscriber/Module.php:56 /srv/www/xyzweb/src/private/protected/vendor/symfony/event-dispatcher/EventDispatcher.php:212 /srv/www/xyzweb/src/private/protected/vendor/symfony/event-dispatcher/EventDispatcher.php:44 /srv/www/xyzweb/src/private/protected/vendor/codeception/phpunit-wrapper/src/Listener.php:133 /srv/www/xyzweb/src/private/protected/vendor/codeception/phpunit-wrapper/src/Listener.php:102 /srv/www/xyzweb/src/private/protected/vendor/phpunit/phpunit/src/Framework/TestResult.php:395 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Test/Test.php:75 /srv/www/xyzweb/src/private/protected/vendor/phpunit/phpunit/src/Framework/TestSuite.php:755 /srv/www/xyzweb/src/private/protected/vendor/codeception/phpunit-wrapper/src/Runner.php:106 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/SuiteManager.php:157 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Codecept.php:189 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Codecept.php:158 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Command/Run.php:466 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Command/Run.php:361 /srv/www/xyzweb/src/private/protected/vendor/symfony/console/Command/Command.php:252 /srv/www/xyzweb/src/private/protected/vendor/symfony/console/Application.php:946 /srv/www/xyzweb/src/private/protected/vendor/symfony/console/Application.php:248 /srv/www/xyzweb/src/private/protected/vendor/symfony/console/Application.php:148 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/src/Codeception/Application.php:108 /srv/www/xyzweb/src/private/protected/vendor/codeception/codeception/codecept:42

Details

Codeception version: 2.4.1

PHP Version: 7.0

Operating System: Ubuntu 16.04

Installation type: Composer

List of installed packages (composer show) alexandernst/yii2-device-detect 0.0.12 Yii2 extension for Mobile-Detect library behat/gherkin v4.5.1 Gherkin DSL parser for PHP 5.3 bower-asset/bootstrap v3.3.7 The most popular front-end framework for developing responsive, mobile first projects on the web. bower-asset/inputmask 3.3.11 Inputmask is a javascript library which creates an input mask. Inputmask can run against vanilla javascript, jQuery and jql... bower-asset/jquery 3.2.1
bower-asset/jquery-ui 1.12.1
bower-asset/punycode v1.3.2
bower-asset/yii2-pjax 2.0.7.1
braintree/braintree_php 3.30.0 Braintree PHP Client Library cebe/markdown 1.1.2 A super fast, highly extensible markdown parser for PHP codeception/codeception 2.4.1 BDD-style testing framework codeception/phpunit-wrapper 6.0.9 PHPUnit classes used by Codeception codeception/specify 1.0 BDD code blocks for PHPUnit and Codeception codeception/stub 1.0.2 Flexible Stub wrapper for PHPUnit's Mock Builder codeception/verify 1.0.0 BDD assertion library for PHPUnit crossjoin/css v1.0.3 CSS reader and writer with full CSS3 support, already supporting huge parts of the current CSS4 spec. It supports media quer... crossjoin/pre-mailer v1.0.5 Crossjoin\PreMailer converts CSS in a given HTML source to inline styles and optimizes it for sending it via e-mail. It also... doctrine/instantiator 1.0.5 A small, lightweight utility to instantiate objects in PHP without invoking their constructors ezyang/htmlpurifier v4.10.0 Standards compliant HTML filter written in PHP facebook/webdriver 1.5.0 A PHP client for Selenium WebDriver firebase/php-jwt v5.0.0 A simple library to encode and decode JSON Web Tokens (JWT) in PHP. Should conform to the current spec. flow/jsonpath 0.4.0 JSONPath implementation for parsing, searching and flattening arrays guzzlehttp/guzzle 6.3.2 Guzzle is a PHP HTTP client library guzzlehttp/promises v1.3.1 Guzzle promises library guzzlehttp/psr7 1.4.2 PSR-7 message implementation that also provides common utility methods imagine/imagine v0.7.1 Image processing for PHP 5.3 kamranahmedse/php-geocode v2.0 A wrapper around the Google Geocoding API to get different details such as latitude, longitude, country, city, district, pos... mobiledetect/mobiledetectlib 2.8.31 Mobile_Detect is a lightweight PHP class for detecting mobile devices. It uses the User-Agent string combined with specific ... myclabs/deep-copy 1.7.0 Create deep copies (clones) of your objects phar-io/manifest 1.0.1 Component for reading phar.io manifest information from a PHP Archive (PHAR) phar-io/version 1.0.1 Library for handling version information and constraints phpdocumentor/reflection-common 1.0.1 Common reflection classes used by phpdocumentor to reflect the code structure phpdocumentor/reflection-docblock 4.3.0 With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information that is e... phpdocumentor/type-resolver 0.4.0
phpmailer/phpmailer v5.2.26 PHPMailer is a full-featured email creation and transfer class for PHP phpoffice/phpexcel 1.8.1 PHPExcel - OpenXML - Read, Create and Write Spreadsheet documents in PHP - Spreadsheet engine phpspec/prophecy 1.7.5 Highly opinionated mocking framework for PHP 5.3+ phpunit/php-code-coverage 5.3.2 Library that provides collection, processing, and rendering functionality for PHP code coverage information. phpunit/php-file-iterator 1.4.5 FilterIterator implementation that filters files based on a list of suffixes. phpunit/php-text-template 1.2.1 Simple template engine. phpunit/php-timer 1.0.9 Utility class for timing phpunit/php-token-stream 2.0.2 Wrapper around PHP's tokenizer extension. phpunit/phpunit 6.5.7 The PHP Unit Testing framework. phpunit/phpunit-mock-objects 5.0.6 Mock Object library for PHPUnit piwik/piwik-php-tracker 1.2.0 PHP Client for Piwik Analytics Tracking API psr/http-message 1.0.1 Common interface for HTTP messages psr/log 1.0.2 Common interface for logging libraries sebastian/code-unit-reverse-lookup 1.0.1 Looks up which function or method a line of code belongs to sebastian/comparator 2.1.3 Provides the functionality to compare PHP values for equality sebastian/diff 2.0.1 Diff implementation sebastian/environment 3.1.0 Provides functionality to handle HHVM/PHP environments sebastian/exporter 3.1.0 Provides the functionality to export PHP variables for visualization sebastian/global-state 2.0.0 Snapshotting of global state sebastian/object-enumerator 3.0.3 Traverses array structures and object graphs to enumerate all referenced objects sebastian/object-reflector 1.1.1 Allows reflection of object attributes, including inherited and non-public ones sebastian/recursion-context 3.0.0 Provides functionality to recursively process PHP variables sebastian/resource-operations 1.0.0 Provides a list of PHP built-in functions that operate on resources sebastian/version 2.0.1 Library that helps with managing the version number of Git-hosted PHP projects symfony/browser-kit v3.4.8 Symfony BrowserKit Component symfony/console v3.4.8 Symfony Console Component symfony/css-selector v2.8.38 Symfony CssSelector Component symfony/debug v3.4.8 Symfony Debug Component symfony/dom-crawler v3.4.8 Symfony DomCrawler Component symfony/event-dispatcher v3.4.8 Symfony EventDispatcher Component symfony/finder v3.4.8 Symfony Finder Component symfony/polyfill-mbstring v1.7.0 Symfony polyfill for the Mbstring extension symfony/process v3.4.8 Symfony Process Component symfony/yaml v3.4.8 Symfony Yaml Component theseer/tokenizer 1.1.0 A small library for converting tokenized PHP source code into XML and potentially other formats webmozart/assert 1.3.0 Assertions to validate method input/output with nice error messages. yiidoc/yii2-redactor 2.0.1 Extension redactor for Yii2 Framework. yiisoft/yii2 2.0.15.1 Yii PHP Framework Version 2 yiisoft/yii2-bootstrap 2.0.8 The Twitter Bootstrap extension for the Yii framework yiisoft/yii2-composer 2.0.6 The composer plugin for Yii extension installer yiisoft/yii2-imagine 2.1.1 The Imagine integration for the Yii framework yiisoft/yii2-jui 2.0.7 The Jquery UI extension for the Yii framework yiisoft/yii2-redis 2.0.8 Redis Cache, Session and ActiveRecord for the Yii framework yurkinx/yii2-image dev-master 2737b1e Yii2 extension for image manipulating using Kohana Image Library.

Suite configuration:

class_name: UnitTester modules: enabled: - Yii2: part: [orm, fixtures] cleanup: true transaction: false coverage: enabled: true remote: false include: - models/* - components/* - forms/*
Bug
opened by scottix 42
Test code or tested code did not (only) close its own output buffers

I am getting this error now and then in combination with Yii2. I drilled the issue down to the occurence of an error NOTICE.

When I change error_reporting(E_NOTICE) to error_reporting(E_ERROR), the code continues.

It took quite a while to traverse through the code to get to this error. Isnt there a better response than "Test code or tested code did not (only) close its own output buffers"?

I prefer maximal insights. Why not outputting errors to the log area? Or to the console?

thanx

opened by dynasource 40
PHP 7.0 / PHPunit 5.1 support
I couldn't find an issue tracking progress on making Codeception compatible with PHP7.0. What's missing to make it run with PHPunit 5.1?

codeception/codeception 2.1.5 requires phpunit/phpunit ~4.8.0 -> no matching package found.

Related issue: #2730
opened by oparoz 37
Adds gitlab CI configuration

Gitlab CI configuration was "work in progress". This Pull Request gives a sample .gitlab-ci.yml file that will run codecept with the built-in php webserver.

opened by richardbrinkman 35
Flash message and session error messages

Hi, I am unable to see session error message using see function nor seefrorm erro or session error fuction returns true, the output html donesn't show flash messages

opened by umang-ranium 35
Fatal Error when testing form.
I am new to codeception so I apologize if I have any misunderstanding with the usage.
I got the following error when I try to run a form I made.

$I->submitForm('#apply-online', array( 'apply_name' => 'foobar', 'apply_tel' => '800-123-1111', 'apply_email' => '[email protected]', 'apply_msg' => 'A Test', 'agree' => 'on', ));

Fatal error: Call to undefined method Symfony\Component\DomCrawler\Field\InputFormField::addChoice() in phar:///Applications/MAMP/htdocs/myform/codecept.phar/vendor/symfony/dom-crawler/Symfony/Component/DomCrawler/Form.php on line 374 FATAL ERROR OCCURRED. TESTS NOT FINISHED.

In addition to that I tried to following what the site said @ http://codeception.com/docs/04-AcceptanceTests I got errors when I use $I->press('Update'); following in the example. But I got problem with $I->click('Update');

This is from the example.

$I = new WebGuy($scenario); $I->amOnPage('testform.html'); $I->fillField('Name', 'Miles'); // we can use input name, or id $I->fillField('user[email]','[email protected]'); $I->selectOption('Gender','Male'); $I->press('Update');
opened by awsp 35
[WIP] Add alpine base for Docker

Add php:7.0-alpine instead of php:7.0-cli as a base, so it could have a lot smaller image.

Image size comparison:

| Base | Docker Hub | HDD | |----------------|-----------:|-------:| | php:7.0-cli | 187 MB | 456 MB | | php:7.0-alpine | 42 MB | 102 MB |

opened by herloct 34
Yii2 rework, new PR same changes.
No more static variables

No more database connection caching (that didn't actually work)

Support for multiple database connections / database services with names other than 'db'

This needs more testing; I've tested it in a few of my production apps.

Also somewhere in the docs we should update the "contract" about application state:

When a test starts you get a fresh application.

Just before a request is executed the Request and Response component are reloaded.

Application state is persisted after the request and reused for the next request (as long as it is not in the Request or Response component.

All PDO based connections are supported as long as they are subclasses of yii\db\Connection and fire the AFTER_OPEN event.

Database handling (for transactions)

All caches are cleared at the end of a test.

At the start of a test we register an event handler for AFTER_OPEN.

When the event is fired we generate a cache key based on connection settings.

We store the PDO object in the cache.

We store an entry in the dsn cache, this is used to identify unsupported cases (think connecting to the same database with different credentials, in which case the transaction creates 2 world views which could cause issues).

If we detect a duplicate PDO object we replace the new PDO with the existing one, this causes the new connection to be closed immediately and guarantees us that the new Connection object uses the same PDO object and thus has the same world view.
opened by SamMousa 32

$Class 'PHPUnit\Framework\ExceptionWrapper' not found$

Class 'PHPUnit\Framework\ExceptionWrapper' not found

What are you trying to achieve?

Trying to run my unit tests. If they pass, everything is well. If the tests fail, an exception is thrown

What do you get instead?

$ sudo vendor/bin/codecept run unit Services/Html/MyTest.php -vvv
PHP Warning:  Class 'PHPUnit\Framework\ExceptionWrapper' not found in /var/www/vagrant/htdocs/vendor/codeception/codeception/shim.php on line 25
PHP Stack trace:
PHP   1. {main}() /var/www/vagrant/htdocs/vendor/codeception/codeception/codecept:0
PHP   2. require_once() /var/www/vagrant/htdocs/vendor/codeception/codeception/codecept:7
PHP   3. include_once() /var/www/vagrant/htdocs/vendor/codeception/codeception/autoload.php:45
PHP   4. class_alias() /var/www/vagrant/htdocs/vendor/codeception/codeception/shim.php:25

Warning: Class 'PHPUnit\Framework\ExceptionWrapper' not found in /var/www/vagrant/htdocs/vendor/codeception/codeception/shim.php on line 25

Call Stack:
    0.0001     365608   1. {main}() /var/www/vagrant/htdocs/vendor/codeception/codeception/codecept:0
    0.0018     393048   2. require_once('/var/www/vagrant/htdocs/vendor/codeception/codeception/autoload.php') /var/www/vagrant/htdocs/vendor/codeception/codeception/codecept:7
    0.0753    2958504   3. include_once('/var/www/vagrant/htdocs/vendor/codeception/codeception/shim.php') /var/www/vagrant/htdocs/vendor/codeception/codeception/autoload.php:45
    0.0929    3738248   4. class_alias() /var/www/vagrant/htdocs/vendor/codeception/codeception/shim.php:25

Codeception PHP Testing Framework v2.3.6
Powered by PHPUnit 6.4.4 by Sebastian Bergmann and contributors.

Unit Tests (1) 
- MyTest: Index
  [Database] Transaction started


  [Symfony\Component\Debug\Exception\FatalThrowableError]
  Class 'PHPUnit\Framework\ExceptionWrapper' not found

Test details are irrelevant, any test failure will cause the exception. If the tests do not fail, everything works as expected

Details

Codeception version: 2.3.6
PHPUnit version: 6.4.4
PHP Version: 7
Operating System: Ubuntu 16
Installation type: Composer
Suite configuration: irrelevant, no changes were made

opened by elimentz 32

PHP 7 + coverage tests using C3 on Travis = Segmentation fault
Extracted from #2712

I launch 2 commands in a row on Travis

$ php vendor/bin/codecept run acceptance --env firefox $ php vendor/bin/codecept run api,integration,unit -vvv --coverage --coverage-xml --coverage-html

The second one will fail with this error message

[GuzzleHttp\Exception\ConnectException]
cURL error 7: couldn't connect to host

in tests/_output/phpbuiltinserver.errors.txt, I get:

[Wed Jan 13 14:36:32 2016] Failed to listen on localhost:8000 (reason: Address already in use) [Wed Jan 13 14:36:40 2016] ::1:48142 [200]: /c3/report/clear

or

[Wed Jan 20 11:28:02 2016] ::1:50413 [200]: /index.php/apps/gallery/files/list?location=&mediatypes=image%2Fpng%3Bimage%2Fjpeg%3Bimage%2Fgif%3Bapplication%2Fpostscript&features=&etag [Wed Jan 20 11:28:10 2016] ::1:50425 [200]: /c3/report/clear

This does not happen on older version of PHP
opened by oparoz 32
Fix failed step output

Fixed #5820 and probably many other reports about incorrect failed step output

The problem with $failedStep = (string)array_shift($this->failedStep); in src/Codeception/Subscriber/Console.php was that all failed steps are added to that array in order of test execution, but errors are printed before failures and order of printing doesn't match order of execution.

opened by Naktibalda 0
Don't override test feature at runtime

Because it causes issues for custom reporters and hides data provider parameters in the output.

$I->wantTo('text') is still parsed and applied before execution by https://github.com/Codeception/Codeception/blob/994ca6c553eb126fefda6eb5375d2ad1b93d4795/src/Codeception/Lib/Parser.php#L33-L40

Fixes #4123 Fixes #4124

opened by Naktibalda 0
Disable phpcs checks in generated action files

Disabling in the first 2 lines is much more efficient because it avoids tokenization of entire file.

@codingStandards annotations are deprecated and will be removed in phpcs v4, so I replaced them with phpcs:disable comments.

Closes #5516

opened by Naktibalda 0

Can not override a config option from the command line in a suite

What are you trying to achieve?

I want to override a config option from the command line in a suite. In my case actually the base-URL for the WebDriver.

What do you get instead?

codecept config:validate -o 'foo: bar'

root@de43a6b7a5fc:/project/tests# codecept config:validate -o 'foo: bar'
Validating global config... foo: bar
Ok
------------------------------

Codeception Config:

Array
(
    foo => bar
    actor => Test
[...]

The above works, I can see foo: bar in the output, it's also shown in the first line.

codecept config:validate -o 'foo: bar' idp

root@de43a6b7a5fc:/project/tests# codecept config:validate -o 'foo: bar' idp
Validating idp config... Ok
------------------------------

idp Suite Config:
[...]

No override is applied, I tried several different formats eg. modules: config: ..., env: chrome: modules: ...

Details

Codeception version: tested latest version for v3, v4, v5 (Docker image) - output above is from 5.0.5
Suite configuration:

class_name: XyzTester
modules:
    enabled:
        - WebDriver
        - Asserts
        - Helper\Xyz
    config:
        WebDriver:
            host: chrome
            url: https://www-test.mydomain.de
            browser: chrome
            port: 4444
            window_size: 1024x768
            restart: true
env:
    chrome:
        modules:
            config:
                WebDriver:
                    browser: 'chrome'
                    host: chrome
                    window_size: 1440x960

    firefox:
    # nothing changed

Related issues:

https://github.com/Codeception/Codeception/issues/4102
https://github.com/Codeception/Codeception/issues/5915
https://github.com/Codeception/Codeception/pull/6536
https://github.com/Codeception/Codeception/pull/6452

opened by schmunk42 0

Launch failed tests with signature or regex
What are you trying to achieve?

It's possible to launch a single test with its signature or regex

vendor/bin/codecept run --env myenv tests/Api/MyCest.php:testWithDataprovider#0 vendor/bin/codecept run --env myenv tests/Api/MyCest.php:testByName$ (if you have also testByNameAndId)

But when I want rerun failed tests, and in failed file I have this naming convention

tests/Api/MyCest.php:testWithDataprovider#0 tests/Api/MyCest.php:testByName$

It doesn't work

What do you get instead?

How to run tests with a signature or a regex from the failed file, with the command -g failed

The Filter Class allows to launch the tests with the signature or the regex, but when we pass by the failed file, we don't launch explicitly the test, but a group of tests

This is to "fix" the problem encountered, when more than one failed test matches the test name or more than one test of a test with dataprovider fails, we have in the failed file

tests/Api/MyCest.php:testByName tests/Api/MyCest.php:testByNameAndId tests/Api/MyCest.php:testWithDataprovider tests/Api/MyCest.php:testWithDataprovider tests/Api/MyCest.php:testWithDataprovider tests/Api/MyCest.php:testWithDataprovider

And so if we re-run the failed tests, we run we run tests/Api/MyCest.php:testByName, it launch tests/Api/MyCest.php:testByName and tests/Api/MyCest.php:testByNameAndId, after tests/Api/MyCest.php:testByNameAndId again and x times the test with dataprovider and they re-run them all x times

Details

Codeception version: 5.0.4

PHP Version: 8.1
opened by ccsuperstar 1

Releases(5.0.6)

5.0.6(Dec 28, 2022)
Fixed canSee assertions in Unit format #6610

tryTo methods must return boolean result #6614

Fixed various issues with handling of @skip and @incomplete annotations and attributes in Cest format #6617

Stopped adding __mocked field to mocks created by Stub library #6620

Fixed deprecated string syntax in Run command #6618 by @shtiher-pp

Source code(tar.gz)
Source code(zip)
5.0.5(Nov 20, 2022)
Don't disable test shuffling when --shard option is used (#6605)

Provided typed signatures for attributes (#6600) by @mdoelker

Support for Attributes in generated Actions files (#6593) by @yesdevnull

Fixed expectNotToPerformAssertions in unit tests (#6602) by @yesdevnull

Source code(tar.gz)
Source code(zip)
5.0.4(Oct 30, 2022)
Execute FailFast subscriber before module _failed hooks #6586 by @yesdevnull

Fixed parsing of @skip annotation #6596

Undeprecated untyped method arguments in Cest format #6591

Removed unnecessary overrides of $resultAggregator property in Unit format and TestCaseWrapper #6590

Print failure/error/warning/skipped/incomplete messages in HTML reports #6595

Fixed counting of successful tests #6595

Source code(tar.gz)
Source code(zip)
5.0.3(Sep 30, 2022)
Fixed passing test result to dependent tests in unit tests (#6580)

Fixed TypeError when @coversNothing annotation is used by Slamdunk (#6582)

codecept init unit creates tests/Support directory (#6578)

Fixed phar file url in self-update command by @voku (#6563)

Added message how to exit Codeception console by @ThomasLandauer (#6561)

Improved compatibility with PHPUnit 10

Source code(tar.gz)
Source code(zip)
5.0.2(Aug 20, 2022)
Fixed remote code coverage for namespaced suites (#6533)

Fixed data provider in abstract Cest class @p-golovin (#6560)

Fixed issue when include groups and test groups empty @geega (#6557)

Source code(tar.gz)
Source code(zip)
5.0.1(Aug 13, 2022)
Propagate --ext and --override parameters to included test suites by @calvinalkan (#6536)

Fixed false negative message about stecman/symfony-console-completion package by @geega (#6541)

Fixed a number of issues in template functionality (#6552)

Fixed DataProvider, to properly load dataProviders from abstract classes by @Basster (#6549)

Source code(tar.gz)
Source code(zip)
4.2.2(Aug 13, 2022)
Propagate --ext and --override parameters to included test suites (#6536)

Fixed false negative message about stecman/symfony-console-completion package (#6541)

Source code(tar.gz)
Source code(zip)
5.0.0(Jul 28, 2022)
5.0.0

Blog post

Summary of all differences from Codeception 4

Added

Basic attribute support

--shard, --grep, --filter options

Test can be filtered by data provider case number or name

Tests of all formats are reported as useless if they perform no assertions and reports_useless_tests setting is enabled

Array of variables can be passed to all pause functions/methods

Dynamic configuration with parameters can use arrays and other non-string types (#6409)

codecept_pause function and $this->pause() in unit tests (#6387)

Interactive console is executed in the scope of paused test class.

New code coverage settings:

path_coverage - enables path and branch coverage

strict_covers_annotation - marks test as risky if it has @covers annotation but executes some other code

ignore_deprecated_code - doesn't collect code coverage for code having @deprecated annotation

disable_code_coverage_ignore - ignores @codeCoverageIgnore, @codeCoverageIgnoreStart and @codeCoverageIgnoreEnd annotations

Optional value to fail-fast option

Dynamic configuration with parameters can use arrays and other non-string types

Changed

PHPUnit is no longer the engine of Codeception, but TestCase format is still supported and assertions are still used

Generators create namespaced test suites by default

Replaced Hoa Console with PsySH in codecept console

Used Symfony VarDumper in codecept_debug

Fixed DotReporter output format

Module after and failed hooks are executed in reverse order

Introduced strict typing and other features of PHP 7 and 8.

Cest format can use data providers from other classes

Fixed injecting dependencies to actor in Cest and Gherkin formats #6506

Variadic parameters can be skipped in dependency injection #6505

Deprecated untyped method arguments in Cest format #6521

Removed

JSON and TAP report formats

Code coverage blacklist functionality

generate:cept command (Cept format itself is deprecated)

Deprecated class aliases:

Codeception\TestCase\Test

Codeception\Platform\Group

Codeception\Platform\Group

Codeception\TestCase

Settings

log_incomplete_skipped

paths.log (replaced by paths.output)

Suite setting class_name (replaced by actor)

Global setting actor (replaced by actor_prefix)

Configuration::logDir method (replaced by Configuration::outputDir in 2.0)

Custom reporters implementing TestListener are no longer supported and must be converted to Extensions

Supported versions

PHP 8

PHPUnit 9 (prepared for upcoming PHPUnit 10)

Symfony 4.4 - 6.x

Source code(tar.gz)
Source code(zip)
5.0.0-RC8(Jul 28, 2022)
Deprecated untyped method arguments in Cest format #6521

Improved code style of generated files #6522

Removed "Powered by PHPUnit" message #6520

Source code(tar.gz)
Source code(zip)
5.0.0-RC7(Jul 22, 2022)
Fixed injecting dependencies to actor in Cest and Gherkin formats #6506

Variadic parameters can be skipped in dependency injection #6505

Source code(tar.gz)
Source code(zip)
5.0.0-RC6(Jul 12, 2022)
Added new attributes (Prepare, Env, BeforeClass,AfterClass, Given, When, Then)

Class level attributes are applied to all methods

Codeception attributes are supported in unit tests

Cest format can use data providers from other classes

Source code(tar.gz)
Source code(zip)
5.0.0-RC5(Jun 28, 2022)
Substitute invalid UTF-8 characters in debug and step output by @SamoylenkoSU (#6483)

Source code(tar.gz)
Source code(zip)
5.0.0-RC4(Jun 22, 2022)
Implemented basic attribute support (#6449) by @DavertMik

Significantly reduced dependencies on PHPUnit

Replaced PHPUnit\Framework\TestResult with ResultAggregator

Added assertionCount method to ResultAggregator

DotReporter prints standard result summary (#6441) by @Orchestrator404

Fixed DotReporter output format

Fixed fetching remote code coverage data (#6448)

Loading .env file must not override existing environment variables (#6477)

All changes from 4.2.0 and 4.2.1:

Improved multi-application experience, allow filtering suites by name (#6435) by @calvinalkan

Configuration override is passed to included suites (#5978) by @calvinalkan

Made dry-run command work with module methods having return types (#6470)

Support for expectError/Warning/Notice/Deprecation methods in unit tests (Requires PHPUnit 8.4+)

Implemented new setting convert_deprecations_to_exceptions (#6469)

Action file generator: Do not return when return type is never (#6462)

Execute setupBeforeClass/tearDownAfterClass only once (#6481)

Source code(tar.gz)
Source code(zip)
4.2.1(Jun 22, 2022)
Execute setupBeforeClass/tearDownAfterClass only once (#6481)

Handle action with intersection return type correctly in dry-run command

Source code(tar.gz)
Source code(zip)
4.2.0(Jun 16, 2022)
Improved multi-application experience, allow filtering suites by name (#6435) by @calvinalkan

Configuration override is passed to included suites (#5978) by @calvinalkan

Made dry-run command work with module methods having return types (#6470)

Support for expectError/Warning/Notice/Deprecation methods in unit tests (Requires PHPUnit 8.4+)

Implemented new setting convert_deprecations_to_exceptions (#6469)

Action file generator: Do not return when return type is never (#6462)

Backported test.useless event from Codeception 5.0 (#6459)

Source code(tar.gz)
Source code(zip)
5.0.0-RC3(Apr 22, 2022)
Fix incorrect type declaration in InitTemplate by @ziadoz

Stricter check for phpdotenv v5 (older versions are not supported)

Throw exception if actor setting is missing in suite configuration

Use correct types in ParamsLoader and Configuration classes, avoid type errors

Source code(tar.gz)
Source code(zip)
5.0.0-RC2(Apr 8, 2022)
Added --shard, --grep, --filter options (#6399)

Added new code coverage settings (#6423)

Dynamic configuration with parameters can use arrays and other non-string types (#6409)

Introduced codecept_pause function and $this->pause() in unit tests (#6387)

Interactive console is executed in the scope of paused test class.

Array of variables can be passed to all pause functions/methods

Replaced Hoa Console with PsySH in codecept console

Used Symfony VarDumper in codecept_debug (#6406)

Fixed type error in code coverage exclude filter by @W0rma

Fix type error in Recorder extension

Source code(tar.gz)
Source code(zip)
5.0.0-RC1(Mar 13, 2022)
Use PHPUnit 9 until PHPUnit 10 is released.

Read about earlier change in CHANGELOG.

Source code(tar.gz)
Source code(zip)
4.1.31(Mar 13, 2022)
RunBefore extension prints error output and stops execution if command failed

Action file generator: Fixed handling of intersection types

Action file generator: Fixed handling of self and parent types

Source code(tar.gz)
Source code(zip)
5.0.0-alpha3(Mar 11, 2022)
Support intersection types in actions

Introduced PSR-12 code style

Extracted some code to modules and shared libs

Fixed new incompatibilities with PHPUnit 10

Source code(tar.gz)
Source code(zip)
4.1.30(Mar 5, 2022)
Fix handling of previous exception in ExtensionException

Improved parser fix for PHP keywords as named parameters

Add link to https://helpukrainewin.org

Source code(tar.gz)
Source code(zip)
5.0.0-alpha2(Feb 19, 2022)
Generators create namespaced test suites by default (#6071) by @DavertMik

Test can be filtered by data provider case number or name (#6363) by @Naktibalda

Removed generate:cept command (Cept format is deprecated)

Removed settings disallow_test_output and log_incomplete_skipped.

Removed setting paths.log (it was replaced by paths.output in Codeception 2.3)

Removed suite setting class_name (replaced by actor in Codeception 2.3)

Removed global setting actor (replaced by actor_prefix in Codeception 2.3)

Removed Configuration::logDir method (replaced by Configuration::logDir in 2.0)

ParamLoader throws exception if xml file is used but simplexml extension is missing (#6346) by @mdoelker

Updated codebase to use PHP 8.0 features by @TavoNiievez

Don't report test as useless if it failed (fixed bug introduced in alpha1)

Don't report local test coverage for remote suites (fixed bug introduced in alpha1)

Moved XmlBuilder class to module-soap

Source code(tar.gz)
Source code(zip)
5.0.0-alpha1(Jan 30, 2022)
Decoupled test execution and reporting from PHPUnit

Custom reporters implementing TestListener are no longer supported and must be converted to Extensions

Tests of all formats are reported as useless if they perform no assertions and reports_useless_tests setting is enabled

Added path_coverage setting to enable path and branch coverage #6158 by @s0lus

Added optional value to fail-fast option (#6275) by @Verest

Removed JSON and TAP loggers

Removed code coverage blacklist functionality

Removed deprecated class aliases

Codeception\TestCase\Test

Codeception\Platform\Group

Codeception\Platform\Group

Codeception\TestCase

Removed settings disallow_test_output, log_incomplete_skipped and report_useless_tests

Introduced strict types in the code base by @TavoNiievez

Compatible with PHPUnit 10 only

Compatible with Symfony 4.4 - 6.0

Requires PHP 8.0 or higher

Source code(tar.gz)
Source code(zip)
4.1.29(Jan 29, 2022)
Fixed duplicate test runs when codeception.yml and codeception.dist.yml are present in multi-app setup by @calvinalkan

Action generator handles mixed type correctly

Parser fix to allow named parameters named class and namespace

Source code(tar.gz)
Source code(zip)
4.1.28(Jan 5, 2022)
Strictly compare test hashes to avoid skipping tests #6320 by @michel-cetina

Fixed deprecation message in codecept build #6311 by @barmax

Source code(tar.gz)
Source code(zip)
4.1.27(Dec 22, 2021)
Renamed editorUrl setting to editor_url for consistency

Fixed wildcard matching of group files (#6302) by @DavertMik

Source code(tar.gz)
Source code(zip)
4.1.26(Dec 21, 2021)
Added editorUrl setting to codeception.yml (#6261) by @ThomasLandauer

Reverted optional value to fail-fast option because it was breaking change (#6290)

Fixed E_DEPRECATED warnings in Example class on PHP 8.1 (#6298) by @fabacino

Source code(tar.gz)
Source code(zip)
4.1.25(Dec 21, 2021)
Update dependencies (https://github.com/Codeception/Codeception/pull/6296)

Source code(tar.gz)
Source code(zip)
4.1.24(Dec 16, 2021)
Fixed running tests from group files in included configs (#6292) by @DavertMik

In this example, tests listed in slow.txt will be executed from backend/tests and frontend/tests:

# included configs include: - backend/tests - frontend/tests # groups defined in files groups: slow: slow.txt
Source code(tar.gz)
Source code(zip)
4.1.23(Dec 11, 2021)
Compatibility with PHP 8.1 (#6252)

Added optional value to fail-fast option (#6275) by #Verest

Code coverage covers .php files only (#6265)

Functions are autoloaded by composer (#6263) by @StuTheWebGuy

Fixed broken URL in bootstrap deprecation message by @p810

codecept init api adds AsJson decorators to suite configuration (See https://github.com/Codeception/module-rest/releases/tag/1.4.1)

Dockerfile upgraded to use xdebug 3.0.4 (#6250) by @PavelBulat

Source code(tar.gz)
Source code(zip)