Introduction
This PR creates an initial test framework to allow for creating and running integration/end-to-end tests for this plugin.
The framework as I have set it up:
- Allows for running tests both as a "project local" as well as a Composer
global
install without screwing up the Composer global environment of the user running the tests.
Tests are run in the system temp directory and the tests clean up after themselves.
- Will ensure that the tests will run with the plugin as it is in the branch being tested.
- As the tests start new PHP processes, the framework ensures that the PHP version used to initiate the test run will be used in those processes. This prevents a test run on a dev machine using the system default PHP version, which would skew the test results.
- Similarly, the framework can be told to use a specific Composer phar file to run the tests with, rather than the system default version.
- The framework also allows for setting up "fake" PHPCS standards for use in the tests, which will make writing the tests simpler as we don't need to keep track of changes in external standards and don't need to take those into account in the tests.
- Tests can be run against multiple PHPCS versions using data providers and a helper class is included to get a specific or random) selection of versions to use compatible with the PHP version against which the tests are being run.
- And to aid in debugging failing tests, it includes a
TestListener
which will spit out all the CLI output seen for a particular test when the test fails.
The tests included in this PR are largely the tests which were already previously being run via the GH Actions workflow, though they are now encapsuled within a PHPUnit based framework and will be run via GH Actions against all supported PHP version, multiple versions of Composer and on multiple OSes.
A lot more tests are still needed, but those should be relatively straight-forward to add once this initial framework is in place and I have already started preparing some follow-up PRs to be pulled once this PR has been merged.
⚠️ This PR depends on PRs #147 and #152 both being accepted and merged.
👉🏻 As this is a huge PR, I have split it up into atomic commits telling the "story" and I would strongly recommend reviewing this PR by looking at each commit separately.
Proposed Changes
Tests: create initial setup
- Require the PHPUnit Polyfills as a basis for the tests.
The PHPUnit Polyfills will ensure a compatible PHPUnit version will be available and will allow for writing tests for the latest version of PHPUnit (9.x), while still being able to run the tests on PHPUnit 4-8.
- Include a basic test bootstrap file which will allow for running the tests both via a Composer installed version of PHPUnit as well as via a PHPUnit PHAR file.
Note: the bootstrap file will try to determine where the
composer.phar
file is located. For running the tests locally and/or overruling the default version of Composer, a COMPOSER_PHAR
environment variable can be set in a local phpunit.xml
overload file.
- Include information about the test setup in the
CONTRIBUTING
documentation.
- Includes adding new jobs to the "Quick test" and "Integration test" GH Actions workflows.
Once the existing tests in the old workflows have been converted to the new test setup, the old workflows can be removed.
- Minor tweaks to various repository config files.
Note: this test setup presumes that the proposal to drop support for PHP 5.3 - see issue #145 - will be accepted as the PHPUnit Polyfills require PHP >= 5.4.
Refs:
- https://github.com/Yoast/PHPUnit-Polyfills/
TestCase: add helper methods for setup/teardown
As most tests will need to run a composer install
, tests should be run in separate, temporary directories.
With that in mind, I'm adding two helper methods to the base TestCase
class:
createTestEnvironment()
- this method will create a PHPCSPluginTest/TestClassName_uniqueID/local
and a PHPCSPluginTest/TestClassName_uniqueID/global
directory in the system temp directory and set the Composer global HOME
directory to the created global
directory.
removeTestEnvironment()
- this method will remove the created directories and reset the Composer global HOME
directory to its original location.
Notes:
- The paths to the created directories will be added to the
$tempDir
, $tempGlobalPath
and $tempLocalPath
properties of the class and can be used in the tests.
- The methods (and associated properties) have been made
static
on purpose to allow for these methods to be used both in setUp()
/tearDown()
fixtures as well as in setUpBeforeClass()
/tearDownAfterClass()
fixtures.
The latter would be useful for a test class needing only one environment with various test methods depending on each other and building on each other.
This is also the reason that these methods are set up as helpers instead of fixtures, as this way, the decision what fixtures are most appropriate can be made for each individual concrete test class.
- A hard cleanup of the
PHPCSPluginTest
directory in the system temp directory has been added to the test bootstrap to make sure that any stray "old" test directories will be removed before a new test run.
Includes a static
TestCase::onWindows()
helper method to determine whether or not the tests are being run on Windows.
TestCase: add helper method for on-the-fly composer.json creation
This adds a test helper method, which, given an array with a Composer configuration and a target location, will on-the-fly create a composer.json
file for use in a test.
The method will automatically add two additional settings to the composer.json
file:
- It will add a
install-codestandards
script to run the plugin on demand.
- It will add the
allow-plugins
permission, which is needed for Composer 2.2.
By adding these keys automatically, this kind of "noise" is not needed in the actual test classes.
The method is static
so as to allow for it to be used in setUpBeforeClass()
/tearDownAfterClass()
fixtures.
Tests: create zip package artifact for the current plugin code
In normal circumstances, a composer install
will download a version of the plugin from Packagist.
This will not work for the tests however, as the development version of the plugin being tested will generally not be available on Packagist.
With this in mind, we need a work-around to ensure that the tests are run with the current version of the plugin as per the last commit in the current branch.
I've implemented this by using the repositories - artifact
feature of Composer.
This works as follow:
- In the test bootstrap, ahead of the test run, the plugin is zipped up as a package and placed in the
tests/artifact
directory.
- Whenever a
composer install/update
command is run within the test suite, the repositories-artifact
key is injected into the composer.json
when the composer.json
file is written to the temp directory used for the test.
- With this in place, Composer will install the plugin from the zipped file in the
tests/artifact
directory instead of downloading from Packagist.
Notes:
- The version number to be used for the zip artifact of the plugin is defined in the test bootstrap file.
This number may need updating once in a while to ensure it is higher than the latest released version of the plugin.
- In the
CreateComposerZipArtifacts
class, three properties are used to determine which files to exclude from the package - $excludedFiles
, $excludedExtensions
and $excludedDirs
.
Depending on new files/directories being introduced in the package, these lists may need to be updated once in a while.
Ref: https://getcomposer.org/doc/05-repositories.md#artifact
TestCase: add helper methods to run CLI commands and custom assertions
This commit adds six additional methods to the base TestCase
class:
executeCliCommand()
: a helper method to execute a CLI command and return the exit code, the contents of stdout
and the content of stderr
for use in assertions.
stabilizeCommand()
: a helper method which is automatically called from the executeCliCommand()
method. This method will:
- Ensure that the PHP version which was used to initiate the test run will also be used in the CLI command.
Without this, the system default PHP version would be used for CLI commands, which may be a different version from the PHP version used to run the tests and would skew the test results.
- Ensure that the Composer version as set up using the
env
variable is used in the tests.
Again, this will prevent the system default version of Composer being used, which may skew the results when the tests are intended to be run against a specific version of Composer.
- Adds the
--no-interaction
flag to all Composer CLI commands to prevent tests hanging when interaction would be expected.
- Adds quotes around a command when the tests are run on Windows in combination with PHP < 8 to ensure cross-platform compatibility of the CLI commands.
getPhpcsCommand()
: a helper method to get the path to the vendor installed PHPCS entry point file.
maybeStripColors()
: a helper method to compare CLI output. This method is still WIP and will probably still need some tweaking.
This method will take an output expectation and an actual output and will attempt to strip CLI colour codes from the actual output when the output expectation does not contain colour codes. This way, expected colour codes can still be tested by including them in the output expectation.
Note: until the bugs in this method have been smoothed out, it is recommended to use --no-ansi
CLI arguments in CLI commands not testing for colour codes.
assertExecute()
: a custom assertion which will run a given CLI command, optionally in a specific working directory and will subsequently assert that the exit code is the same as expected and the stdout
and stderr
outputs contain the expected output.
For more complex assertions against the output of CLI commands, the executeCliCommand()
can be used directly in tests.
The StringContainsString
assertions used for stdout
and stderr
will attempt to strip color codes from the output before doing a compare using the maybeStripColors()
method.
assertComposerValidates()
: a custom assertion to verify that a given composer.json
file validates for use in the tests.
A number of these methods have been made static
to allow for them to be used in conjunction with setUpBeforeClass()
fixtures.
Tests: add TestListener to log and display CLI output for failing tests
While not necessarily the most pretty solution (output is displayed in the middle of the progress report), this allows for access to the composer.json
content, the CLI commands and their output for any test which failed, which will be helpful when debugging test failures, especially in CI.
Tests: add initial baseline test
These tests verify:
- That the plugin can be installed and functions correctly with the full range of supported PHPCS versions.
While the test is not run against the full range of supported PHPCS version each time, due to the random PHPCS version selection, all supported PHPCS versions will be tested over time in CI.
- That the plugin runs when installed.
- That the PHPCS native standards are the only recognized standards when no external standards are available.
- That no `CodeSniffer.conf``file gets created when no external standards are found.
These tests will be run in both a Composer global as well as a Composer project local environment.
The tests use a dataprovider which will randomly select two PHPCS versions compatible with the PHP version on which the test is being run + PHPCS dev-master
+ PHPCS 4.x
(dev version for the next major).
As which PHPCS versions are compatible with which PHP versions needs quite some logic, a separate helper class PHPCSVersions
is introduced, which encapsulates that knowledge.
The PHPCSVersions
class contains a number of (static
) methods as helpers. These can be used in any test class:
get()
: retrieves an array with a specific number of PHPCS versions valid for the current PHP version.
getHighLow()
: retrieves an array of the highest and lowest PHPCS versions valid for the current PHP version.
getHighLowEachMajor()
: retrieves an array of the highest and lowest supported PHPCS versions for each PHPCS major (and valid for the current PHP version).
getRandom()
: gets a random PHPCS version which is valid for the current PHP version.
toDataprovider()
: converts a version array to an array suitable for use as a PHPUnit dataprovider.
getSupportedVersions()
: retrieves an array with all PHPCS versions valid for the current PHP version.
isNextMajorSupported()
: determines whether the current PHP version is supported on the "next major" branch of PHPCS.
getStandards()
: retrieves an array of the names of the PHPCS native standards for a particular PHPCS version.
Generally speaking, the dev-master
and 4.x
branch will not be included in the retrieved ranges, but can be specifically requested by passing optional parameters.
This PHPCSVersions
class will need semi-regular updates when new versions of PHPCS are released and new PHP versions are released.
Tests: move test with external standard from CI to test class
This moves the test which was being run via GH Actions to a test class in the integration test suite.
Differences between the previous and new setup:
- The previous test via GH Actions would only fail on the exit code not being
0
for any of the steps.
The new PHPUnit based tests executes more detailed assertions on the results of the commands being run in the tests.
- The previous test would execute the test run of PHPCS against the
src/Plugin.php
file in this package.
The new PHPUnit based tests will execute against a simple PHP file created on the fly.
- The previous test via GH Actions would only run for a Composer
local
situation.
The new PHPUnit based tests cover both a Composer global
as well as a local
setup.
- The matrix against which the tests are run is different.
In the old situation, the test would only be run on a fixed set of PHP/PHPCS combinations with Composer 2.x on Ubuntu.
In the new situation, the tests will run against both Composer 1.x, as well as 2.x on both Ubuntu and Windows.
Same as before, it will be run against all supported PHP versions (in the extended
integrationtest
workflow), but the PHPCS versions to test against are more varied and - in part - randomly selected out of the PHPCS version compatible with the current PHP version, which results in more comprehensive testing of these scenarios.
Includes:
- Adding a new
createFile()
helper method to the base TestCase
to generate a (PHP) file to do a test run against.
- Removing the "old" integration test from the GH Actions script.
Tests: set up a fake PHPCS standard as a fixture and create an artifact of it
The tests in the RegisterExternalStandardsTest
class contain quite some work-arounds to handle changes made over time to the external PHPCompatibility standard.
The PHPCompatibility
standard was chosen for use in the tests as it is one of the rare standards which still supports both PHPCS 2.x as well as 3.x.
All the same, to improve the stability of the tests and to simplify the actual test code, using "fake" PHPCS standards set up as fixtures as part of this test suite will make life easier.
This commit creates the initial setup for that, by:
- Adding code to the
CreateComposerZipArtifacts
class to handle creating Composer package zip artifacts for each subdirectory of the tests/fixtures
directory.
- Calling that code from the test bootstrap file to ensure all test artifacts are zipped up and ready for use ahead of running the tests.
- Adding an initial "fake"
DummySubDir
standard as a test fixture.
- Adding a
README
file to the tests/fixtures
directory with the rationale and a list of available fixtures.
RegisterExternalStandardsTest: switch out PHPCompatibility for the test fixture
This replaces the use of the PHPCompatibility standard in the RegisterExternalStandardsTest
tests with the use of the dummy-subdir
fake standard fixture.
RegisterExternalStandardsTest: simplify check whether PHPCS can run with the standard
To check whether PHPCS can run with the standard, previously, a scan on a simple file was done, with this file being created on the fly for each test.
This commit replaces that check with running the "explain" command.
Advantages:
- The "explain" command does not need a file to scan and can still confirm that the standard can be read correctly by PHPCS.
- The output for the "explain" command has fewer differences across PHPCS versions, so is simpler to verify.
:new: BaseLineTest: selectively skip the tests
... in a very specific combination of circumstances - Windows + Composer 1.x + PHP 5.5 + PHPCS dev-master
+ no external standards -, as the Composer logs in that case, don't always show the "Running PHPCodeSniffer Composer Installer" message.
As it is such a specific combination, which will probably be rare to encounter in real life, I'm proposing to skip the tests for that combination instead of spending lots of time trying to debug this (for now).
🆕 TestCase: add willPluginOutputShow()
method
In very select circumstances (PHP 5.5 with Composer 1.x on Windows) and sometimes even only with select PHPCS versions, the plugin output will not be shown in the transcript from Composer, even though the plugin does actually run.
Instead of skipping those tests completely, I'm suggestion to selectively skip the output expectation for the composer install/update
run instead.
That way we can still verify that the plugin has actually done its job by checking that paths have been registered with PHPCS.
I would recommend to only implement the use of this method for those select tests where we have actually seen failures due to this Composer bug and to just leave the other tests alone.
Related Issues
- https://github.com/Dealerdirect/phpcodesniffer-composer-installer/issues/9
- https://github.com/Dealerdirect/phpcodesniffer-composer-installer/issues/92
Closes #9
builds / deploys / releases