Humbug - a Mutation Testing framework for PHP

Last update: May 19, 2022

Humbug: Mutation Testing for PHP

🚨 This package is deprecated, check out Infection instead.

Humbug is a Mutation Testing framework for PHP. It is currently in development and so, while it does actually work quite well, it will have rough edges that a team of minions are working hard to hammer out. If it falls out of the gate, you have been warned ;).

Build Status Build status Scrutinizer Code Quality StyleCI Total Downloads Slack

⚠️ ️ Update your remotes! Humbug has transferred to a new location. While your existing repositories will redirect transparently for any operations, take some time to transition to the new URL.

$ git remote set-url upstream https://github.com/humbug/humbug.git

Replace upstream with the name of the remote you use locally; upstream is commonly used but you may be using something else. You may also using a different URL (e.g. [email protected]:mockery/mockery.git). Run git remote -v to see what you're actually using.

Table of Contents

Introduction

Mutation Testing is, in a nutshell, giving your unit tests a run for their money. It involves injecting small defects into source code and then checking if the unit tests noticed. If they do, then your unit tests have "killed" the mutation. If not, the mutation has escaped detection. As unit tests are intended to prevent regressions, having a real regression pass unnoticed would be a bad thing!

Whereas Code Coverage can tell you what code your tests are executing, Mutation Testing is intended to help you judge how well your unit tests actually perform and where they could be improved.

I've written in more detail about why Mutation Testing is worth having: Lies, Damned Lies and Code Coverage: Towards Mutation Testing

Contributing

Humbug is an open source project that welcomes pull requests and issues from anyone. Before opening pull requests, please read our short Contribution Guide.

Installation

Git

You can clone and install Humbug's dependencies using Composer:

git clone https://github.com/humbug/humbug.git
cd humbug
/path/to/composer.phar install

The humbug command is now at bin/humbug.

Phar

If you don't want to track the master branch directly, you can install the Humbug phar as follows:

wget https://padraic.github.io/humbug/downloads/humbug.phar
wget https://padraic.github.io/humbug/downloads/humbug.phar.pubkey
# If you wish to make humbug.phar directly executable
chmod +x humbug.phar

On Windows, you can just download using a browser or from Powershell v3 using the following commands where wget is an alias for Invoke-WebRequest:

wget https://padraic.github.io/humbug/downloads/humbug.phar -OutFile humbug.phar
wget https://padraic.github.io/humbug/downloads/humbug.phar.pubkey -OutFile humbug.phar.pubkey

If you're stuck with Powershell v2:

$client = new-object System.Net.WebClient
$client.DownloadFile("https://padraic.github.io/humbug/downloads/humbug.phar", "humbug.phar")
$client.DownloadFile("https://padraic.github.io/humbug/downloads/humbug.phar.pubkey", "humbug.phar.pubkey")
PHAR Updates

The phar is signed with an openssl private key. You will need the pubkey file to be stored beside the phar file at all times in order to use it. If you rename humbug.phar to humbug, for example, then also rename the key from humbug.phar.pubkey to humbug.pubkey.

The phar releases are currently done manually so they will not be updated with the same frequency as git master. To update your current phar, just run:

./humbug.phar self-update

Note: Using a phar means that fixes may take longer to reach your version, but there's more assurance of having a stable development version. The public key is downloaded only once. It is re-used by self-update to verify future phar releases.

Once releases commence towards stable, there will be an alpha, beta, RC and a final release. Your development track phar file will self-update automatically until it reaches a stable release. If you wish to continue tracking the development level phars, you will need to indicate this using one of the stability flags:

./humbug.phar self-update --dev
Self-Update Request Debugging

If you experience any issues self-updating with unexpected openssl or SSL errors, please ensure that you have enabled the openssl extension. On Windows, you can do this by adding or uncommenting the following line in the php.ini file for PHP on the command line (if different than the file for your http server):

extension=php_openssl.dll

Certain other SSL errors may arise due missing certificates. You can rectify this by finding their location on your system (e.g. C:/xampp/php/ext/cacert.pem), or alternatively downloading a copy from http://curl.haxx.se/ca/cacert.pem. Then ensure the following option is correctly pointing to this file:

openssl.cafile=C:/path/to/cacert.pem

Composer

Due to Humbug's dependencies being pegged to recent versions, adding Humbug to composer.json may give rise to conflicts. The above two methods of installation are preferred where this occurs. You can however install it globally as any other general purpose tool:

composer global require 'humbug/[email protected]'

And if you haven't done so previously...add this to ~/.bash_profile (or ~/.bashrc):

export PATH=~/.composer/vendor/bin:$PATH

Humbug currently works on PHP 5.4 or greater.

Usage

Configuration

Humbug is still under development so, to repeat, beware of rough edges.

Configure command

To configure humbug in your project you may run:

humbug configure

This tool will ask some questions required to create the Humbug configuration file (humbug.json.dist).

Manual Configuration

In the base directory of your project create a humbug.json.dist file:

{
    "timeout": 10,
    "source": {
        "directories": [
            "src"
        ]
    },
    "logs": {
        "text": "humbuglog.txt",
        "json": "humbuglog.json"
    }
}

You can commit the humbug.json.dist to your VCS and override it locally with a humbug.json file.

Edit as appropriate. If you do not define at least one log, detailed information about escaped mutants will not be available. The Text log is human readable. If source files exist in the base directory, or files in the source directories must be excluded, you can add exclude patterns (here's one for files in base directory where composer vendor and Tests directories are excluded):

{
    "timeout": 10,
    "source": {
        "directories": [
            "."
        ],
        "excludes": [
            "vendor",
            "Tests"
        ]
    },
    "logs": {
        "text": "humbuglog.txt"
    }
}

If, from your project's base directory, you must run tests from another directory then you can signal this also. You should not need to run Humbug from any directory other than your project's base directory.

{
    "chdir": "tests",
    "timeout": 10,
    "source": {
        "directories": [
            "src"
        ],
    }
}

Running Humbug

Ensure that your tests are all in a passing state (incomplete and skipped tests are allowed). Humbug will quit if any of your tests are failing.

The magic command, while in your project's base directory (using the PHAR download) is:

./humbug.phar

or if you just cloned Humbug:

../humbug/bin/humbug

or if you added Humbug as a composer dependency to your project:

./vendor/bin/humbug

Instead of php with the xdebug extension you may also run Humbug via phpdbg:

phpdbg -qrr humbug.phar

If all went well, you will get something similar to:

 _  _            _
| || |_  _ _ __ | |__ _  _ __ _
| __ | || | '  \| '_ \ || / _` |
|_||_|\_,_|_|_|_|_.__/\_,_\__, |
                          |___/ 
Humbug version 1.0-dev

Humbug running test suite to generate logs and code coverage data...

  361 [==========================================================] 28 secs

Humbug has completed the initial test run successfully.
Tests: 361 Line Coverage: 64.86%

Humbug is analysing source files...

Mutation Testing is commencing on 78 files...
(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)

.....M.M..EMMMMMSSSSMMMMMSMMMMMSSSE.ESSSSSSSSSSSSSSSSSM..M.. |   60 ( 7/78)
...MM.ES..SSSSSSSSSS...MMM.MEMME.SSSS.............SSMMSSSSM. |  120 (12/78)
M.M.M...TT.M...T.MM....S.....SSS..M..SMMSM...........M...... |  180 (17/78)
MM...M...ESSSEM..MMM.M.MM...SSS.SS.M.SMMMMMMM..SMMMMS....... |  240 (24/78)
.........SMMMSMMMM.MM..M.SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  300 (26/78)
SSSSSSSSM..E....S......SS......M.SS..S..M...SSSSSSSS....MMM. |  360 (37/78)
.M....MM..SM..S..SSSSSSSS.EM.S.E.M............M.....M.SM.M.M |  420 (45/78)
..M....MMS...MMSSS................M.....EME....SEMS...SSSSSS |  480 (52/78)
SSSSS.EMSSSSM..M.MMMM...SSE.....MMM.M..MM..MSSSSSSSSSSSSSSSS |  540 (60/78)
SSS....SSSSSSSSMM.SSS..........S..M..MSSMS.SSSSSSSSSSSSSSSSS |  600 (68/78)
......E...M..........SM.....M..MMMMM.MMMMMSSSSSSSM.SS

653 mutations were generated:
     284 mutants were killed
     218 mutants were not covered by tests
     131 covered mutants were not detected
      17 fatal errors were encountered
       3 time outs were encountered

Metrics:
    Mutation Score Indicator (MSI): 47%
    Mutation Code Coverage: 67%
    Covered Code MSI: 70%

Remember that some mutants will inevitably be harmless (i.e. false positives).

Humbug results are being logged as JSON to: log.json
Humbug results are being logged as TEXT to: log.txt

To explain the perhaps cryptic progress output:

  • Killed Mutation (.): A mutation that caused unit tests to fail which is a positive outcome.
  • Escaped Mutation (M): A mutation where the unit tests still passed which is not what we want! Our unit tests should detect any behaviour changes.
  • Uncovered Mutation (S): A mutation which occurs on a line not covered by any unit test. Since there are no unit tests, this is another undesireable result.
  • Fatal Error (E): A mutation created a fatal error. Usually a positive result since it's obviously going to be noticed. In some cases, however, it might be a Humbug problem that needs fixing.
  • Timeout (T): This is where unit tests exceed the allowed timeout configured for Humbug. Likely a positive result if your timeout is appropriate, and often occurs when a mutation ends up creating an infinite loop.

Kills, errors and timeouts are all counted as detected mutations. We report errors in the logs on the off chance that Humbug itself encountered an internal error, i.e. a bug to be reported as an issue here!

The Metrics

The example summary results reported a number of metric scores:

  • A Mutation Score Indicator (MSI) of 47%. This means that 47% of all generated mutations were detected (i.e. kills, timeouts or fatal errors). The MSI is the primary Mutation Testing metric. Given the code coverage of 65%, there is a 18% discrepancy so Code Coverage was a terrible quality measurement in this example.
  • Mutation Code Coverage is 67%. On average it should be within the same ballpark as your normal code coverage, but code coverage ignores mutation frequency.
  • The Mutation Score Indicator for code that is actually covered by tests was 70% (i.e. ignoring code not even tested). This gives you some idea of how effective the tests that do exist really are.

If you examine these metrics, the standout issue is that the MSI of 47% is 18 points lower than the reported Code Coverage at 65%. These unit tests are far less effective than Code Coverage alone could detect.

Interpreting these results requires some context. The logs will list all undetected mutations as diffs against the original source code. Examining these will provide further insight as to what specific mutations went undetected.

Command Line Options

Humbug has a few command line options of note, other than those normally associated with any Symfony Console application.

Overriding The Configured Timeout

You can manually set the timeout threshold for any single test:

humbug --timeout=10

Restricting Files To Mutate

If you're only interested in mutating a subset of your files, you can pass any number of --file options containing simple file names, globs or regular expressions. Basically, these are all passed to the Symfony Finder's name() method.

humbug --file=NewClass.php --file=*Driver.php

This in no way restricts the initial Humbug check on the overall test suite which is still executed in full to ensure all tests are passing correctly before proceeding.

Mutate specific files

If you want to mutate only a few specific files, you can pass any number of --path options containing full path file names. This option will be passed to a filter \Closure that will intersect files found using the config and/or --file option with the files provided by you using the --path option.

humbug --path=src/Data/NewClass.php --path=src/Driver/Driver.php

Note: This in no way restricts the initial Humbug check on the overall test suite which is still executed in full to ensure all tests are passing correctly before proceeding.

Incremental Analysis

Incremental Analysis (IA) is an experimental unfinished mode of operation where results are cached locally between runs and reused where it makes sense. At present, this mode operates very naively by eliminating test runs where both the immediate file being mutated and the relevant tests for a mutated line have not been modified since the last run (as determined by comparing the SHA1 of the files involved).

humbug --incremental

The IA mode offers a significant performance increase for relatively stable code bases, and you're free to test it and see how it fares in real life. In the future, it does need to take into accounts changes in files which contain parent classes, imported traits and the classes of its immediate dependencies, all of which have an impact on the behaviour of any given object.

IA utilises a local permanent cache, e.g. /home/padraic/.humbug.

Performance

Mutation Testing has traditionally been slow. The concept being to re-run your test suite for each mutation generated. To speed things up significantly, Humbug does the following:

  • On each test run, it only uses those test classes which cover the specific file and line on which the mutation was inserted.
  • It orders test classes to run so that the slowest go last (hopefully the faster tests will detect mutations early!).
  • If a mutation falls on a line not covered by any tests, well, we don't bother running any tests.
  • Performance may, depending on the source code, be significantly impacted by timeouts. The default of 60s may be far too high for smaller codebases, and far too low for larger ones. As a rule of thumb, it shouldn't exceed the seconds needed to normally run the tests being mutated (and can be set lower).

While all of this speeds up Humbug, do be aware that a Humbug run will be slower than unit testing. A 2 second test suite may require 30 seconds for mutation testing. Or 5 minutes. It all depends on the interplay between lines of code, number of tests, level of code coverage, and the performance of both code and tests.

Mutators

Humbug implements a basic suite of Mutators, which essentially tells us when a particular PHP token can be mutated, and also apply that mutation to an array of tokens.

Note: Source code held within functions (rather than class methods) is not mutated at this time.

Binary Arithmetic:

Original Mutated Original Mutated
+ - /= *=
- + %= *=
* / **= /=
/ * & |
% * | &
** / ^ &
+= -= ~
-= += >> <<
*= /= << >>

Boolean Substitution:

This temporarily encompasses logical mutators.

Original Mutated
true false
false true
&& ||
|| &&
and or
or and
!

Conditional Boundaries:

Original Mutated
> >=
< <=
>= >
<= <

Negated Conditionals:

Original Mutated Original Mutated
== != > <=
!= == < >=
<> == >= <
=== !== <= >
!== ===

Increments:

Original Mutated
++ --
-- ++

Return Values:

Original Mutated Original Mutated
return true; return false; return 1.0>; return -( + 1);
return false; return true; return $this; return null;
return 0; return 1; return function(); function(); return null;
return ; return 0; return new Class; new Class; return null;
return 0.0; return 1.0; return (Anything); (Anything); return null;
return 1.0; return 0.0;

Literal Numbers:

Original Mutated
0 1
1 0
Int > 1 Int + 1
Float >= 1 / <= 2 Float + 1
Float > 2 1

If Statements:

All if statements are covered largely by previous mutators, but there are special cases such as using native functions or class methods without any compares or operations, e.g. is_int() or in_array(). This would not cover functions defined in files since they don't exist until runtime (something else to work on!).

Original Mutated
if(is_int(1)) if(!is_int(1))

More Mutators will be added over time.

JSON Log Stats

bin/humbug stats ../my-project/humbuglog.json ../my-project/list-of-classes.txt --skip-killed=yes [-vvv]

Parses stats from humbuglog.json or your custom named JSON log.

CLI reference:

humbug stats [humbuglog.json location] [class list location] [--skip-killed=yes] [-vvv]
humbuglog.json location, defaults to ./humbuglog.json

class list location, a path to a text file containing full class names, one per line.

only this files-related stats would be shown
--skip-killed=yes is used to completely skip output of "killed" section
various verbosity levels define amount of info to be displayed:
    by default, there's one line per class with amount of mutants killed/escaped/errored/timed out (depending on output section)
    -v adds one line per each mutant with line number and method name
    -vv adds extra line for each mutant, displaying diff view of line mutant is detected in
    -vvv shows full diff with several lines before and after

This can be tested on humbug itself, by running in humbug's dir:

bin/humbug bin/humbug stats [-vvv]

Did I Say Rough Edges?

This is a short list of known issues:

  • Humbug does initial test runs, logging and code coverage. Should allow user to do that optionally.
  • Test classes (not tests) are run in a specific order, fastest first. Interdependent test classes may therefore fail regularly which will skew the results.
  • Currently 100% PHPUnit specific, well 98.237%. There is an adapter where PHPUnit code is being shovelled.
  • Certain test suite may make assumptions about having sole access to resources like /tmp which will cause errors when Humbug tries using same.
  • Fine grained test ordering by speed (as opposed to large grained test class ordering) is awaiting implementation.
  • Should test classes be used to carry non-PHPUnit dependent testing code (e.g. register_shutdown_function()), it may create issues when combined with one or more of Humbugs optimisations which assume a finished test really is finished.

Bah, Humbug!

Courtesy of Craig Davis who saw potential in a once empty repository :P.

                    .:::::::::::...
                  .::::::::::::::::::::.
                .::::::::::::::::::::::::.
               ::::::::::::::::::::::::::::.
              :::::::::::::::::::::::::::::::  .,uuu   ...
             :::::::::::::::::::::::::::::::: dHHHHHLdHHHHb
       ....:::::::'`    ::::::::::::::::::' uHHHHHHHHHHHHHF
   .uHHHHHHHHH'         ::::::::::::::`.  uHHHHHHHHHHHHHP"
   HHHHHHHHHHH          `:::::::::::',dHHuHHHHHHHHP"[email protected]@g
  J"HHHHHHHHHP        4H ::::::::'  u$$$.
  ".HHHHHHHHP"     .,uHP :::::' uHHHHHHHHHHP"",e$$$$$c
   HHHHHHHF'      dHHHHf `````.HHHHHHHHHHP",d$$$$$$$P%C
 .dHHHP""         JHHHHbuuuu,JHHHHHHHHP",d$$$$$$$$$e=,z$$$$$$$$ee..
 ""              .HHHHHHHHHHHHHHHHHP",gdP"  ..3$$$Jd$$$$$$$$$$$$$$e.
                 dHHHHHHHHHHHHHHP".edP    " .zd$$$$$$$$$$$"3$$$$$$$$c
                 `???""??HHHHP",e$$F" .d$,?$$$$$$$$$$$$$F d$$$$$$$$F"
                       ?be.eze$$$$$".d$$$$ $$$E$$$$P".,ede`?$$$$$$$$
                      4."?$$$$$$$  z$$$$$$ $$$$r.,.e ?$$$$ $$$$$$$$$
                      '$c  "$$$$ .d$$$$$$$ 3$$$.$$$$ 4$$$ d$$$$P"`,,
                       """- "$$".`$$"    " $$f,d$$P".$$P zeee.zd$$$$$.
                     ze.    .C$C"=^"    ..$$$$$$P".$$$'e$$$$$P?$$$$$$
                 .e$$$$$$$"="$f",c,3eee$$$$$$$$P $$$P'd$$$$"..::.."?$%
                4d$$$P d$$$dF.d$$$$$$$$$$$$$$$$f $$$ d$$$" :::::::::.
               $$$$$$ d$$$$$ $$$$$$$$$$$$$$$$$$ J$$",$$$'.::::::::::::
              "$$$$$$ ?$$$$ d$$$$$$$$$$$$$$$P".dP'e$$$$':::::::::::::::
              4$$$$$$c $$$$b`$$$$$$$$$$$P"",e$$",$$$$$' ::::::::::::::::
              ' ?"?$$$b."$$$$.?$$$$$$P".e$$$$F,d$$$$$F ::::::::::::::::::
                    "?$$bc."$b.$$$$F z$$P?$$",$$$$$$$ ::::::::::::::::::::
                        `"$$c"?$$$".$$$)e$$F,$$$$$$$' ::::::::::::::::::::
                        ':. "$b...d$$P4$$$",$$$$$$$" :::::::::::::::::::::
                        ':::: "$$$$$".,"".d$$$$$$$F ::::::::::::::::::::::
                         :::: be."".d$$$4$$$$$$$$F :::::::::::::::::::::::
                          :::: "??$$$$$$$$$$?$P" :::::::::::::::::::::::::
                           :::::: ?$$$$$$$$f .::::::::::::::::::::::::::::
                            :::::::`"????"".::::::::::::::::::::::::::::::

GitHub

https://github.com/humbug/humbug
Comments
  • 1. Humbug stops, saying tests must pass, but they do

    I'm getting a weird bug in the latest version, Humbug just stops and tell me my test must pass for it to run properly, but when I run the tests they all pass properly:

    $ humbug
    Humbug running test suite to generate logs and code coverage data...
    
       21 [==========================================================] 33 secs
    
    <warning>Tests must be in a fully passing state before Humbug is run.</warning>
    <warning>Incomplete, skipped or risky tests are allowed.</warning>
    <warning>The testing framework reported an exit code of 143.</warning>
    <warning>The testing framework ran into a failure or error. Refer to out below.</warning>
    <warning>Stdout: \n    > ok 22 - Rocketeer\Abstracts\AbstractTaskTest::testCanGetOptionsViaCommandOrSetters
        > not ok 23 - Error: Rocketeer\Abstracts\Strategies\AbstractDependenciesStrategyTest::testCanShareDependenciesFolder
        > \n</warning>
    
    $ phpunit
    PHPUnit 4.4.1 by Sebastian Bergmann.
    
    Configuration read from /Users/anahkiasen/Sites/rocketeers/rocketeer/phpunit.xml
    
    ...............................................................  63 / 323 ( 19%)
    ............................................................... 126 / 323 ( 39%)
    ............................................................... 189 / 323 ( 58%)
    ............................................................... 252 / 323 ( 78%)
    ............................................................... 315 / 323 ( 97%)
    ........
    
    Time: 38.27 seconds, Memory: 389.75Mb
    
    OK (323 tests, 400 assertions)
    

    The progress bar is complete so I assumed this was just a warning but the humbug log is empty and just contains the error above

    Reviewed by Anahkiasen at 2015-01-17 21:07
  • 2. Humbug's Future (Discussion)

    Apologies all, it has been a hectic year or so, and I've allowed ALL of my online activities to come to a stop. I'm currently creeping back onto the open source scene, so I'm getting close to sorting out the Humbug situation. Here's basically where I see things going in brief, and I'm intent on implementing it in the coming weeks:

    • I've created a "Humbugged" organisation: https://github.com/humbugged (Yes, it's odd. Yes, the humbug user name was taken. Humbug!)

    • I hope to transfer ALL Humbug related repositories, included key supporting dependencies, to the new organisation.

    • Organisations need members! If you have emailed, or tweeted, me previously about having a greater involvement in Humbug, I'll reach out to you by email to see if you are still interested (and apologise for the lengthy response wait!). Those who have written PRs, short of having auditions, will be preferred. Ideally, a small team of 2-3 to start with, and I will open it to the floor here if needed to make up numbers.

    • While remaining a BDFL figure, I'm generally very relaxed about where my projects go. Members WILL have scope to act independently, champion new features, manage issues, peer review and merge PRs, etc. I'll remain involved, but mostly to ensure things run smoothly as a glorified moderator. Governance can be elaborated on in time. While I expect to be back actively contributing, that's a goal for the medium term. In the meantime, there are issues, PRs, and lots of scope to improve Humbug.

    • Humbug logo? I think the NWN avatar I use falls a bit short ;).

    The most urgent priorities would, in some basic order:

    1. Getting the existing code running error free under Travis (so we can recheck PRs).
    2. Merging PRs wherever it makes sense.
    3. Issue triage, and solving those where not covered by PRs (within reason).
    4. Put out a new release.
    5. Have a good think about where Humbug is now, and where it should be for a stable release.

    The project will not, at this time, be transferred to a new owner. However, ideally that will become redundant as the project becomes sustainable whether I'm around or not. If I ever do transfer ownership, I would strongly prefer it to be a matter of transitioning administration of the organisation to existing member(s) at that time.

    By all means, please discuss below.

    Reviewed by padraic at 2017-04-12 21:00
  • 3. Refactor result arrays (for testsuite and mutants) into classes

    As mentionned by @rollenes on #115, this removes the dependency of multiple classes on a correct array structure.

    Additionally, this moves the creation logic of mutant results inside of the Mutant class to reduce the responsibility of the Humbug command class.

    Reviewed by fabre-thibaud at 2015-03-05 17:01
  • 4. Cannot run humbug on a Symfony project

    I am trying to use humbug on my Symfony project and I have this weird behavior. Humbug does not find my test suite.

    Here is my humbug configuration:

    {
        "chdir": "app",
        "timeout": 10,
        "source": {
            "directories": [
                "src"
        ]
        },
        "logs": {
            "text": "humbuglog.txt",
            "json": "humbuglog.json"
        }
    }
    

    Here is my phpunit configuration file:

    <?xml version="1.0" encoding="UTF-8"?>
    
    <!-- http://phpunit.de/manual/current/en/appendixes.configuration.html -->
    <phpunit
        backupGlobals               = "false"
        backupStaticAttributes      = "false"
        colors                      = "true"
        convertErrorsToExceptions   = "true"
        convertNoticesToExceptions  = "true"
        convertWarningsToExceptions = "true"
        processIsolation            = "false"
        stopOnFailure               = "false"
        syntaxCheck                 = "false"
        bootstrap                   = "bootstrap.php.cache" >
    
        <testsuites>
            <testsuite name="Test Suite">
                <directory>../src/*/*Bundle/Tests</directory>
            </testsuite>
        </testsuites>
    
        <filter>
            <whitelist>
                <directory>../src</directory>
                <exclude>
                    <directory>../src/*/*Bundle/Resources</directory>
                    <directory>../src/*/*Bundle/Tests</directory>
                </exclude>
            </whitelist>
        </filter>
    </phpunit>
    

    I can run my test suite directly from the command line or from netbeans, but it seems that there is something blocking the use of it from humbug. Here is humbug output:

    
     _  _            _
    | || |_  _ _ __ | |__ _  _ __ _
    | __ | || | '  \| '_ \ || / _` |
    |_||_|\_,_|_|_|_|_.__/\_,_\__, |
                              |___/
    Humbug version 1.0-dev
    
    Humbug running test suite to generate logs and code coverage data...
    
        0 [>---------------------------------------------------------]  1 sec
    
    Humbug has completed the initial test run successfully.
    Tests: 0 Line Coverage: 0.00%
    
    Humbug is analysing source files...
    
    Mutation Testing is commencing on 91 files...
    (.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)
    
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |   60 (13/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  120 (13/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  180 (16/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  240 (17/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  300 (48/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  360 (60/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  420 (65/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  480 (65/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  540 (66/91)
    SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS |  600 (77/91)
    SSSSSSSSSSSSSSSSSSSSSSS
    
    623 mutations were generated:
           0 mutants were killed
         623 mutants were not covered by tests
           0 covered mutants were not detected
           0 fatal errors were encountered
           0 time outs were encountered
    
    Out of 0 test covered mutations, 0% were detected.
    Out of 623 total mutations, 0% were detected.
    Out of 623 total mutations, 0% were covered by tests.
    
    Remember that some mutants will inevitably be harmless (i.e. false positives).
    
    Humbug results are being logged as JSON to: humbuglog.json
    Humbug results are being logged as TEXT to: humbuglog.txt
    
    Time: 36.42 seconds Memory: 7.25MB
    

    And phpunit output:

    PHPUnit 4.3.5 by Sebastian Bergmann.
    
    Configuration read from C:\Work\Sources\app\phpunit.xml.dist
    
    ...............................................................  63 / 677 (  9%)
    ............................................................... 126 / 677 ( 18%)
    ............................................................... 189 / 677 ( 27%)
    ............................................................... 252 / 677 ( 37%)
    ............................................................... 315 / 677 ( 46%)
    ............................................................... 378 / 677 ( 55%)
    ............................................................... 441 / 677 ( 65%)
    ............................................................... 504 / 677 ( 74%)
    ............................................................... 567 / 677 ( 83%)
    ............................................................... 630 / 677 ( 93%)
    ...............................................
    
    Time: 44.55 seconds, Memory: 30.25Mb
    
    ←[30;42mOK (677 tests, 1057 assertions)←[0m
    
    Generating code coverage report in Clover XML format ... done
    

    I am running that on W7 in git bash CLI interface.

    Do you have any ideas why it is not working?

    Reviewed by aledeg at 2015-02-27 15:21
  • 5. Weird escaped mutations

    I tried running Humbung on Mink. It reports me a bunch of escapes, which I was expecting (though we are catching 84% of them on first run, which could have been worse). However, some of the escaped mutations look really suspicious to me. Here is the first reported escaped mutation:

    1) \Humbug\Mutator\Number\Integer
    Diff on \Behat\Mink\Selector\NamedSelector::translateToXPath() in /home/stof/Code/mink/Mink/src/Behat/Mink/Selector/NamedSelector.php:
    --- Original
    +++ New
    @@ @@
         {
    -        if (2 < count($locator)) {
    +        if (3 < count($locator)) {
                 throw new \InvalidArgumentException('NamedSelector expects array(name, locator) as argument');
             }
    
             if (2 == count($locator)) {
                 $selector   = $locator[0];
                 $locator    = $locator[1];
    
            {
                "file": "\/home\/stof\/Code\/mink\/Mink\/src\/Behat\/Mink\/Selector\/NamedSelector.php",
                "mutator": "\\Humbug\\Mutator\\Number\\Integer",
                "class": "\\Behat\\Mink\\Selector\\NamedSelector",
                "method": "translateToXPath",
                "line": 197,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        if (2 < count($locator)) {\n+        if (3 < count($locator)) {\n             throw new \\InvalidArgumentException('NamedSelector expects array(name, locator) as argument');\n         }\n \n         if (2 == count($locator)) {\n             $selector   = $locator[0];\n             $locator    = $locator[1];",
                "stdout": "",
                "stderr": null,
                "tests": [
                    {
                        "class": "Behat\\Mink\\Tests\\Selector\\PartialNamedSelectorTest",
                        "file": "\/home\/stof\/Code\/mink\/Mink\/tests\/Selector\/PartialNamedSelectorTest.php",
                        "time": 0.009642
                    },
                    {
                        "class": "Behat\\Mink\\Tests\\Selector\\ExactNamedSelectorTest",
                        "file": "\/home\/stof\/Code\/mink\/Mink\/tests\/Selector\/ExactNamedSelectorTest.php",
                        "time": 0.009797
                    }
                ]
            },
    

    However, running phpunit --tap tests/Selector/PartialNamedSelectorTest.php gives me this output (I truncated the end as we have more tests in it):

    TAP version 13
    ok 1 - Behat\Mink\Tests\Selector\PartialNamedSelectorTest::testRegisterXpath
    not ok 2 - Error: Behat\Mink\Tests\Selector\PartialNamedSelectorTest::testInvalidLocator
    ok 3 - Behat\Mink\Tests\Selector\PartialNamedSelectorTest::testSelectors with data set "fieldset" ('test.html', 'fieldset', 'fieldset-text', 2, 3)
    ...
    

    I don't see how this would qualify as an escaped mutation.

    Reviewed by stof at 2015-01-24 01:35
  • 6. Getting insta-failure on running

    Humbug running test suite to generate logs and code coverage data...
    
        0 [>---------------------------------------------------------]  1 sec
    
     Tests must be in a fully passing state before Humbug is run.
     Incomplete, skipped or risky tests are allowed.
     The testing framework reported an exit code of 255.
     Stdout:
        >
        > Fatal error: Cannot redeclare class PhpParser\Autoloader in /Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/nikic/php-parser/lib/PhpParser/Autoloader.php on line 9
        >
        > Call Stack:
        >     0.0002     225536   1. {main}() -:0
        >     0.0050     807768   2. Humbug\Adapter\Phpunit::main() -:6
        >     0.0068     979536   3. PHPUnit_TextUI_Command->run() /Users/anahkiasen/.composer/vendor/humbug/humbug/src/Humbug/Adapter/Phpunit.php:157
        >     0.0068     982256   4. PHPUnit_TextUI_Command->handleArguments() /Users/anahkiasen/.composer/vendor/phpunit/phpunit/src/TextUI/Command.php:114
        >     0.0131    1577768   5. PHPUnit_TextUI_Command->handleBootstrap() /Users/anahkiasen/.composer/vendor/phpunit/phpunit/src/TextUI/Command.php:622
        >     0.0132    1588200   6. PHPUnit_Util_Fileloader::checkAndLoad() /Users/anahkiasen/.composer/vendor/phpunit/phpunit/src/TextUI/Command.php:792
        >     0.0133    1588624   7. PHPUnit_Util_Fileloader::load() /Users/anahkiasen/.composer/vendor/phpunit/phpunit/src/Util/Fileloader.php:42
        >     0.0133    1591248   8. include_once('/private/var/folders/fs/grc6ws5d37jfjbl27s73w8n40000gn/T/humbug.phpunit.bootstrap.php') /Users/anahkiasen/.composer/vendor/phpunit/phpunit/src/Util/Fileloader.php:58
        >     0.0133    1593496   9. require_once('/Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/autoload.php') /private/var/folders/fs/grc6ws5d37jfjbl27s73w8n40000gn/T/humbug.phpunit.bootstrap.php:2
        >     0.0135    1609088  10. ComposerAutoloaderInit53a0ac4fb954920498b64b9da42c33b0::getLoader() /Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/autoload.php:7
        >     0.0172    2049720  11. composerRequire53a0ac4fb954920498b64b9da42c33b0() /Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/composer/autoload_real.php:49
        >     0.0173    2052392  12. require('/Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/nikic/php-parser/lib/bootstrap.php') /Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/composer/autoload_real.php:58
        >     0.0177    2143496  13. require('/Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/nikic/php-parser/lib/PhpParser/Autoloader.php') /Users/anahkiasen/Sites/rocketeers/rocketeer/vendor/nikic/php-parser/lib/bootstrap.php:3
        >
        >
    
    Reviewed by Anahkiasen at 2015-01-23 10:06
  • 7. Mutation for functions

    I have the following piece of code in my codebase:

    // src/helpers.php
    
    if (false === function_exists('deep_clone')) {
        /**
         * Deep clone the given value.
         *
         * @param mixed $value
         *
         * @return mixed
         */
        function deep_clone($value)
        {
            return (new \DeepCopy\DeepCopy())->copy($value);
        }
    }
    

    And I have a test for it. Humbug mutate this function into:

    if (false === function_exists('deep_clone')) {
        /**
         * Deep clone the given value.
         *
         * @param mixed $value
         *
         * @return mixed
         */
        function deep_clone($value)
        {
            (new \DeepCopy\DeepCopy())->copy($value); return null;
        }
    }
    

    Which should fail (if I change my source code into that, it does fail) but is not.

    Reviewed by theofidry at 2016-10-28 17:54
  • 8. Logs don't show mutations that weren't covered by tests

    Unless I'm completely inept, I can't find anywhere in the logs that shows what mutations were performed that the tests did not catch.

    I'm seeing a list of mutations that were killed, as well as those that caused fatal errors, but nowhere can I find the ones that didn't cause a test to fail. Naturally these are the ones I'm most interested in! :)

    How can I see what mutations were performed that did not cause a failing test?

    Here's the log that was generated, says 6 "notests" but no details about them:

    {
        "summary": {
            "total": 35,
            "kills": 24,
            "escapes": 0,
            "errors": 5,
            "timeouts": 0,
            "notests": 6,
            "covered_score": 100,
            "combined_score": 83,
            "mutation_coverage": 83
        },
        "escaped": [],
        "errored": [
            {
                "file": "src\/Relationship\/HasMany.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\This",
                "class": "\\AdamWathan\\Faktory\\Relationship\\HasMany",
                "method": "quantity",
                "line": 26,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $this->quantity = $quantity;\n-        return $this;\n+        return null;\n     }\n }\n \n",
                "stdout": "",
                "stderr": "PHP Fatal error:  Call to a member function attributes() on null in \/home\/vagrant\/Code\/Personal\/faktory\/tests\/FaktoryCreateTest.php on line 395",
                "tests": [
                    "FaktoryTest",
                    "FaktoryCreateTest"
                ]
            },
            {
                "file": "src\/Relationship\/Relationship.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\This",
                "class": "\\AdamWathan\\Faktory\\Relationship\\Relationship",
                "method": "foreignKey",
                "line": 21,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $this->foreign_key = $key;\n-        return $this;\n+        return null;\n     }\n \n     public function getForeignKey()\n     {\n         if (! is_null($this->foreign_key)) {\n             return $this->foreign_key;",
                "stdout": "",
                "stderr": "PHP Fatal error:  Call to a member function attributes() on null in \/home\/vagrant\/Code\/Personal\/faktory\/tests\/FaktoryCreateTest.php on line 583",
                "tests": {
                    "0": "FaktoryCreateTest",
                    "3": "BelongsToTest",
                    "4": "RelationshipTest"
                }
            },
            {
                "file": "src\/Relationship\/Relationship.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\This",
                "class": "\\AdamWathan\\Faktory\\Relationship\\Relationship",
                "method": "attributes",
                "line": 61,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $this->attributes = array_merge($this->attributes, $attributes);\n-        return $this;\n+        return null;\n     }\n \n     public function __set($key, $value)\n     {\n         $this->attributes[$key] = $value;\n     }",
                "stdout": "",
                "stderr": "PHP Fatal error:  Call to a member function quantity() on null in \/home\/vagrant\/Code\/Personal\/faktory\/tests\/FaktoryCreateTest.php on line 406",
                "tests": {
                    "0": "FaktoryTest",
                    "2": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Strategy\/Build.php",
                "mutator": "\\Humbug\\Mutator\\Boolean\\LogicalNot",
                "class": "\\AdamWathan\\Faktory\\Strategy\\Build",
                "method": "buildRelationships",
                "line": 20,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         foreach ($this->attributes as $attribute => $value) {\n-            if (! $value instanceof Relationship) {\n+            if ( $value instanceof Relationship) {\n                 continue;\n             }\n             $relationship = $this->buildRelationship($value);\n             $this->setAttribute($attribute, $relationship);\n         }\n     }",
                "stdout": "",
                "stderr": "PHP Fatal error:  Call to a member function build() on string in \/home\/vagrant\/Code\/Personal\/faktory\/src\/Strategy\/Build.php on line 30",
                "tests": [
                    "FaktoryTest"
                ]
            },
            {
                "file": "src\/Strategy\/Strategy.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\NewObject",
                "class": "\\AdamWathan\\Faktory\\Strategy\\Strategy",
                "method": "make",
                "line": 17,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        return new static($model, $sequence);\n+        new static($model, $sequence); return null;\n     }\n \n     public function attributes($attributes)\n     {\n         $this->attributes = $attributes;\n     }",
                "stdout": "",
                "stderr": "PHP Fatal error:  Call to a member function attributes() on null in \/home\/vagrant\/Code\/Personal\/faktory\/src\/Factory\/Factory.php on line 74",
                "tests": {
                    "0": "FaktoryTest",
                    "26": "FaktoryCreateTest"
                }
            }
        ],
        "timeouts": [],
        "killed": [
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\NewObject",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "make",
                "line": 24,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        return new static($model, $factory_repository);\n+        new static($model, $factory_repository); return null;\n     }\n \n     public function getModel()\n     {\n         return $this->model;\n     }",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "26": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Increment\\Increment",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "newInstance",
                "line": 76,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $instance = $strategy->newInstance();\n-        $this->sequence++;\n+        $this->sequence--;\n         return $instance;\n     }\n \n     protected function mergeAttributes($override_attributes)\n     {\n         if (is_callable($override_attributes)) {",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "26": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\FunctionCall",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "mergeAttributes",
                "line": 85,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         }\n-        return array_merge($this->attributes, $override_attributes);\n+        array_merge($this->attributes, $override_attributes); return null;\n     }\n \n     protected function getOverridesFromClosure($closure)\n     {\n         $that = clone $this;\n         $closure($that);",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "26": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Number\\Integer",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "buildMany",
                "line": 100,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->build($override_attributes[$i]);\n-        }, range(0, $count - 1));\n+        }, range(1, $count - 1));\n     }\n \n     protected function expandAttributesForList($attributes, $count)\n     {\n         return array_map(function ($i) use ($attributes) {\n             return $this->extractAttributesForIndex($i, $attributes);",
                "stdout": "",
                "stderr": "",
                "tests": [
                    "FaktoryTest"
                ]
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Arithmetic\\Subtraction",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "buildMany",
                "line": 100,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->build($override_attributes[$i]);\n-        }, range(0, $count - 1));\n+        }, range(0, $count + 1));\n     }\n \n     protected function expandAttributesForList($attributes, $count)\n     {\n         return array_map(function ($i) use ($attributes) {\n             return $this->extractAttributesForIndex($i, $attributes);",
                "stdout": "",
                "stderr": "",
                "tests": [
                    "FaktoryTest"
                ]
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Number\\Integer",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "buildMany",
                "line": 100,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->build($override_attributes[$i]);\n-        }, range(0, $count - 1));\n+        }, range(0, $count - 0));\n     }\n \n     protected function expandAttributesForList($attributes, $count)\n     {\n         return array_map(function ($i) use ($attributes) {\n             return $this->extractAttributesForIndex($i, $attributes);",
                "stdout": "",
                "stderr": "",
                "tests": [
                    "FaktoryTest"
                ]
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Number\\Integer",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "expandAttributesForList",
                "line": 107,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->extractAttributesForIndex($i, $attributes);\n-        }, range(0, $count - 1));\n+        }, range(1, $count - 1));\n     }\n \n     protected function extractAttributesForIndex($i, $attributes)\n     {\n         return array_map(function ($value) use ($i) {\n             return is_array($value) ? $value[$i] : $value;",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "6": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Arithmetic\\Subtraction",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "expandAttributesForList",
                "line": 107,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->extractAttributesForIndex($i, $attributes);\n-        }, range(0, $count - 1));\n+        }, range(0, $count + 1));\n     }\n \n     protected function extractAttributesForIndex($i, $attributes)\n     {\n         return array_map(function ($value) use ($i) {\n             return is_array($value) ? $value[$i] : $value;",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "6": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Number\\Integer",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "expandAttributesForList",
                "line": 107,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->extractAttributesForIndex($i, $attributes);\n-        }, range(0, $count - 1));\n+        }, range(0, $count - 0));\n     }\n \n     protected function extractAttributesForIndex($i, $attributes)\n     {\n         return array_map(function ($value) use ($i) {\n             return is_array($value) ? $value[$i] : $value;",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "6": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\FunctionCall",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "extractAttributesForIndex",
                "line": 113,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         return array_map(function ($value) use ($i) {\n-            return is_array($value) ? $value[$i] : $value;\n+            is_array($value) ? $value[$i] : $value; return null;\n         }, $attributes);\n     }\n \n     public function createMany($count, $override_attributes)\n     {\n         $override_attributes = $this->expandAttributesForList($override_attributes, $count);",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "3": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Number\\Integer",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "createMany",
                "line": 122,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->create($override_attributes[$i]);\n-        }, range(0, $count - 1));\n+        }, range(1, $count - 1));\n     }\n \n     public function define($name, $definitionCallback)\n     {\n         $callback = function ($f) use ($definitionCallback) {\n             $f->setAttributes($this->attributes);",
                "stdout": "",
                "stderr": "",
                "tests": [
                    "FaktoryCreateTest"
                ]
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Arithmetic\\Subtraction",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "createMany",
                "line": 122,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->create($override_attributes[$i]);\n-        }, range(0, $count - 1));\n+        }, range(0, $count + 1));\n     }\n \n     public function define($name, $definitionCallback)\n     {\n         $callback = function ($f) use ($definitionCallback) {\n             $f->setAttributes($this->attributes);",
                "stdout": "",
                "stderr": "",
                "tests": [
                    "FaktoryCreateTest"
                ]
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\Number\\Integer",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "createMany",
                "line": 122,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n             return $this->create($override_attributes[$i]);\n-        }, range(0, $count - 1));\n+        }, range(0, $count - 0));\n     }\n \n     public function define($name, $definitionCallback)\n     {\n         $callback = function ($f) use ($definitionCallback) {\n             $f->setAttributes($this->attributes);",
                "stdout": "",
                "stderr": "",
                "tests": [
                    "FaktoryCreateTest"
                ]
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\NewObject",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "belongsTo",
                "line": 137,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $factory = $this->factory_repository->getFactory($name);\n-        return new BelongsTo($this->model, $factory, $foreign_key, $attributes);\n+        new BelongsTo($this->model, $factory, $foreign_key, $attributes); return null;\n     }\n \n     public function hasMany($name, $count, $foreign_key = null, $attributes = [])\n     {\n         $factory = $this->factory_repository->getFactory($name);\n         return new HasMany($this->model, $factory, $count, $foreign_key, $attributes);",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "2": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\NewObject",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "hasMany",
                "line": 143,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $factory = $this->factory_repository->getFactory($name);\n-        return new HasMany($this->model, $factory, $count, $foreign_key, $attributes);\n+        new HasMany($this->model, $factory, $count, $foreign_key, $attributes); return null;\n     }\n \n     public function hasOne($name, $foreign_key = null, $attributes = [])\n     {\n         $factory = $this->factory_repository->getFactory($name);\n         return new HasOne($this->model, $factory, $foreign_key, $attributes);",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "2": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/Factory.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\NewObject",
                "class": "\\AdamWathan\\Faktory\\Factory\\Factory",
                "method": "hasOne",
                "line": 149,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $factory = $this->factory_repository->getFactory($name);\n-        return new HasOne($this->model, $factory, $foreign_key, $attributes);\n+        new HasOne($this->model, $factory, $foreign_key, $attributes); return null;\n     }\n }\n \n",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "3": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/FactoryProxy.php",
                "mutator": "\\Humbug\\Mutator\\Boolean\\LogicalNot",
                "class": "\\AdamWathan\\Faktory\\Factory\\FactoryProxy",
                "method": "getInstance",
                "line": 15,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        if (! isset($this->instance)) {\n+        if ( isset($this->instance)) {\n             $this->instance = $this->factory_loader->__invoke();\n         }\n         return $this->instance;\n     }\n \n     public function __call($method, $parameters)",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "27": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Factory\/FactoryProxy.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\FunctionCall",
                "class": "\\AdamWathan\\Faktory\\Factory\\FactoryProxy",
                "method": "__call",
                "line": 24,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $instance = $this->getInstance();\n-        return call_user_func_array([$instance, $method], $parameters);\n+        call_user_func_array([$instance, $method], $parameters); return null;\n     }\n }\n \n",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "26": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Faktory.php",
                "mutator": "\\Humbug\\Mutator\\Boolean\\LogicalNot",
                "class": "\\AdamWathan\\Faktory\\Faktory",
                "method": "fetchFactory",
                "line": 62,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        if (! isset($this->factories[$name])) {\n+        if ( isset($this->factories[$name])) {\n             throw new FactoryNotRegisteredException(\"'{$name}' is not a registered factory.\");\n         }\n         return $this->factories[$name];\n     }\n }\n ",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "27": "FaktoryCreateTest"
                }
            },
            {
                "file": "src\/Relationship\/Relationship.php",
                "mutator": "\\Humbug\\Mutator\\Boolean\\LogicalNot",
                "class": "\\AdamWathan\\Faktory\\Relationship\\Relationship",
                "method": "getForeignKey",
                "line": 26,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        if (! is_null($this->foreign_key)) {\n+        if ( is_null($this->foreign_key)) {\n             return $this->foreign_key;\n         }\n         return $this->guessForeignKey();\n     }\n \n     protected function guessForeignKey()",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryCreateTest",
                    "24": "BelongsToTest",
                    "28": "RelationshipTest"
                }
            },
            {
                "file": "src\/Relationship\/Relationship.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\FunctionCall",
                "class": "\\AdamWathan\\Faktory\\Relationship\\Relationship",
                "method": "snakeCase",
                "line": 39,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        return ctype_lower($value) ? $value : strtolower(preg_replace('\/(.)([A-Z])\/', '$1_$2', $value));\n+        ctype_lower($value) ? $value : strtolower(preg_replace('\/(.)([A-Z])\/', '$1_$2', $value)); return null;\n     }\n \n     protected function relatedModelBase()\n     {\n         return $this->extractClassBase($this->getRelatedModel());\n     }",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryCreateTest",
                    "4": "BelongsToTest",
                    "6": "RelationshipTest"
                }
            },
            {
                "file": "src\/Relationship\/Relationship.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\FunctionCall",
                "class": "\\AdamWathan\\Faktory\\Relationship\\Relationship",
                "method": "extractClassBase",
                "line": 55,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         $class_pieces = explode('\\\\', $class);\n-        return array_pop($class_pieces);\n+        array_pop($class_pieces); return null;\n     }\n \n     public function attributes($attributes)\n     {\n         $this->attributes = array_merge($this->attributes, $attributes);\n         return $this;",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryCreateTest",
                    "4": "BelongsToTest",
                    "6": "RelationshipTest"
                }
            },
            {
                "file": "src\/Strategy\/Create.php",
                "mutator": "\\Humbug\\Mutator\\Boolean\\LogicalNot",
                "class": "\\AdamWathan\\Faktory\\Strategy\\Create",
                "method": "independentAttributes",
                "line": 43,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n         foreach ($this->attributes as $attribute => $value) {\n-            if (! $value instanceof Relationship) {\n+            if ( $value instanceof Relationship) {\n                 $result[$attribute] = $value;\n             }\n         }\n         return $result;\n     }\n ",
                "stdout": "",
                "stderr": "",
                "tests": [
                    "FaktoryCreateTest"
                ]
            },
            {
                "file": "src\/Strategy\/Strategy.php",
                "mutator": "\\Humbug\\Mutator\\ReturnValue\\NewObject",
                "class": "\\AdamWathan\\Faktory\\Strategy\\Strategy",
                "method": "newModel",
                "line": 27,
                "diff": "--- Original\n+++ New\[email protected]@ @@\n     {\n-        return new $this->model;\n+        new $this->model; return null;\n     }\n \n     protected function setAttribute($attribute, $value)\n     {\n         $this->attributes[$attribute] = $value;\n     }",
                "stdout": "",
                "stderr": "",
                "tests": {
                    "0": "FaktoryTest",
                    "26": "FaktoryCreateTest"
                }
            }
        ]
    }
    
    Reviewed by adamwathan at 2015-03-14 03:25
  • 9. Notice: Uninitialized string offset: 0 in ~\humbug\src\Adapter\Locator.php on line 27

    Ran humbug (added as composer require --dev humbug/humbug) without having tests and got a lot of Notices. Added !empty($name) && to start of if statement in ~\humbug\src\Adapter\Locator.php line 27 and some ( and ) to keep the if statement in working order and all is peachy again.

    I'm unable to provide a PR for this, so this is just a heads up.

    Reviewed by johanneskonst at 2016-02-25 18:56
  • 10. The testing framework reported an exit code of 143

    I installed PHPUnit 4.8.10 and Humbug 1.0-dev with Composer. My phpunit.xml looks the following:

    <?xml version="1.0" encoding="UTF-8"?>
    <phpunit>
        <testsuites>
            <testsuite name="All Unit tests">
                <directory suffix="Test.php">./test/lib</directory>
                <exclude>./test/lib/processes/actions/EditLaboratories/groups/EditGroupsTreeControllerTest.php</exclude>
            </testsuite>
        </testsuites>
    </phpunit>
    

    When I run "vendor/bin/phpunit" all tests run fine:

    .......................................
    
    Time: 249 ms, Memory: 6.75Mb
    
    OK (39 tests, 163 assertions)
    

    But when I start Humbug with "vendor/bin/humbug" I get:

    
     _  _            _
    | || |_  _ _ __ | |__ _  _ __ _
    | __ | || | '  \| '_ \ || / _` |
    |_||_|\_,_|_|_|_|_.__/\_,_\__, |
                              |___/ 
    Humbug version 1.0-dev
    
    Humbug running test suite to generate logs and code coverage data...
    
        0 [>---------------------------------------------------------]  1 sec
    
    Tests must be in a fully passing state before Humbug is run.
    Incomplete, skipped or risky tests are allowed.
    The testing framework reported an exit code of 143.
    The testing framework ran into a failure or error. Refer to output below.
    Stdout:
       > TAP version 13
       > 
    Stderr:
       > PHP Fatal error:  Uncaught exception 'ErrorException' with message 'Trying to get property of non-object' in /home/jan/PhpstormProjects/labaccess/vendor/phpunit/phpunit/src/Util/XML.php:224
       > Stack trace:
       > #0 /home/jan/PhpstormProjects/labaccess/vendor/phpunit/phpunit/src/Util/XML.php(224): lib\common\classes\Logger->phpErrorHandler(8, 'Trying to get p...', '/home/jan/Phpst...', 224, Array)
       > #1 /home/jan/PhpstormProjects/labaccess/vendor/phpunit/phpunit/src/Util/Configuration.php(336): PHPUnit_Util_XML::xmlToVariable(Object(DOMElement))
       > #2 /home/jan/PhpstormProjects/labaccess/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(761): PHPUnit_Util_Configuration->getListenerConfiguration()
       > #3 /home/jan/PhpstormProjects/labaccess/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(145): PHPUnit_TextUI_TestRunner->handleConfiguration(Array)
       > #4 /home/jan/PhpstormProjects/labaccess/vendor/phpunit/phpunit/src/TextUI/Command.php(148): PHPUnit_TextUI_TestRunner->doRun(Object(PHPUnit_Framework_TestSuite), Array)
       > #5 /home/jan/PhpstormProjects/labac in /home/jan/PhpstormProjects/labaccess/vendor/phpunit/phpunit/src/Util/XML.php on line 224
       > 
    

    My humbug.json.dist looks like this:

    {
        "source": {
            "directories": [
                "lib"
            ]
        },
        "timeout": 10,
        "logs": {
            "text": "humbuglog.txt"
        }
    }
    

    Have I configured something wrong or is there an error in Humbug / PHPUnit?

    Reviewed by limetec at 2015-10-01 10:20
  • 11. make humbug installable via composer require

    currently, humbug can't be installed directly via composer require :

    $ composer require humbug/humbug 1.0.*@dev
    ./composer.json has been created
    Loading composer repositories with package information
    Updating dependencies (including require-dev)
    Your requirements could not be resolved to an installable set of packages.
    
      Problem 1
        - Installation request for humbug/humbug 1.0.*@dev -> satisfiable by humbug/humbug[1.0.x-dev].
        - humbug/humbug 1.0.x-dev requires padraic/phpunit-extensions [email protected] -> no matching package found.
    
    Potential causes:
     - A typo in the package name
     - The package is not available in a stable-enough version according to your minimum-stability setting
       see <https://groups.google.com/d/topic/composer-dev/_g3ASeIFlrc/discussion> for more details.
    
    Read <http://getcomposer.org/doc/articles/troubleshooting.md> for further common problems.
    
    Installation failed, deleting ./composer.json.
    
    Reviewed by samsonasik at 2015-02-15 16:54
  • 12. TImeouts - how to prevent them

    | Question | Answer | ------------| --------------- | Humbug ver. |1.0.0-rc.0 | Adapter ver.| PHPUnit 5.7 | Platform | Ubuntu (Bash) | PHP ver. | 7.1.9

    How do you prevent timeouts ? for example

    -        while (($line = fgets($handle)) !== false) {
    +        while (($line = fgets($handle)) !== true) {
    

    Causes a timeout, though I have no idea how to prevent that happening ? Do I just accept it as a result value ?

    Other examples include

    -        for ($i = $start; $i <= $end; $i++) {
    +        for ($i = $start; $i <= $end; $i--) {
    

    Thanks

    Reviewed by exussum12 at 2017-10-08 12:17
  • 13. Incompatibility with php-vcr (or other libraries using stream_wrapper_register)

    | Question | Answer | ------------| --------------- | Humbug ver. | 1.0-dev | Adapter ver.| PHPUnit 5.7.1 | Platform | Ubuntu (Docker) | PHP ver. | 7.1

    php-vcr uses stream_wrapper_register for the file protocol. If you are turn it on at the bootstrap file humbug will stop working. A option of replacing the original file with the mutated one instead of using a stream wrapper could solve the issue, but isn't that elegant.

    Reviewed by GM-Alex at 2017-09-07 20:55
  • 14. Humbug fails with PHPUnit stderr setting

    | Question | Answer | --------------| --------------- | Humbug ver. | 1.0-dev | Adapter ver. | PHPUnit 5.7.21 | Platform | Windows 10 Pro (conemu x64) | PHP ver. | 7.1.8 w/ Xdebug 2.5.5

    Ran into an issue with humbug starting with stderr="true" in phpunit.xml config despite passing tests. PHPUnit --tap didn't generate any errors either.

    Humbug output

    Tests must be in a fully passing state before Humbug is run.
    Incomplete, skipped or risky tests are allowed.
    The testing framework ran into a failure or error. Refer to output below.
    Stderr:
       > PHPUnit 5.7.21 by Sebastian Bergmann and contributors.
       >
       > Runtime:       PHP 7.1.8 with Xdebug 2.5.5
       > Configuration: Y:\Apache24\temp\humbug\phpunit.humbug.xml
    
    ...
    
       > OK (125 tests, 304 assertions)
       >
       > Generating code coverage report in PHP format ... done
    

    phpunit.xml

    <phpunit bootstrap="bootstrap.php"
        colors="true"
        verbose="true"
        stderr="true">
        <testsuites>
    ...
        </testsuites>
        <filter>
            <whitelist processUncoveredFilesFromWhitelist="true">
                <directory suffix=".php">../src/</directory>
            </whitelist>
        </filter>
        <logging>
            <log type="coverage-html" target="./tmp/Report" lowUpperBound="50" highLowerBound="100"/>
        </logging>
    </phpunit>
    
    

    Removing the setting solved it.

    Reviewed by ccwebdesign at 2017-08-08 11:57
  • 15. Stripping PHPUnit configuration of elements

    Humbug currently, as a first step towards building an XML configuration, strips all filter, listener and logging elements. Assess for potential problems. Do we really need to stripping out code coverage excludes, etc.? Reduce the scope for unexpected problems due to configuration tampering.

    Reviewed by padraic at 2017-05-11 19:34
SimpleTest is a framework for unit testing, web site testing and mock objects for PHP

SimpleTest SimpleTest is a framework for unit testing, web site testing and mock objects for PHP. Installation Downloads All downloads are stored on G

Apr 21, 2022
The modern, simple and intuitive PHP unit testing framework.

atoum PHP version atoum version 5.3 -> 5.6 1.x -> 3.x 7.2 -> 8.x 4.x (current) A simple, modern and intuitive unit testing framework for PHP! Just lik

May 18, 2022
Full-stack testing PHP framework

Codeception Modern PHP Testing for everyone Codeception is a modern full-stack testing framework for PHP. Inspired by BDD, it provides an absolutely n

May 19, 2022
The PHP Unit Testing framework.

PHPUnit PHPUnit is a programmer-oriented testing framework for PHP. It is an instance of the xUnit architecture for unit testing frameworks. Installat

May 26, 2022
PHP unit testing framework with built in mocks and stubs. Runs in the browser, or via the command line.

Enhance PHP A unit testing framework with mocks and stubs. Built for PHP, in PHP! Quick Start: Just add EnhanceTestFramework.php and you are ready to

May 2, 2022
Pest is an elegant PHP Testing Framework with a focus on simplicity
Pest is an elegant PHP Testing Framework with a focus on simplicity

Pest is an elegant PHP Testing Framework with a focus on simplicity. It was carefully crafted to bring the joy of testing to PHP. Explore the docs: pe

May 19, 2022
A drop in fake logger for testing with the Laravel framework.
A drop in fake logger for testing with the Laravel framework.

Log fake for Laravel A bunch of Laravel facades / services are able to be faked, such as the Dispatcher with Bus::fake(), to help with testing and ass

May 22, 2022
Unit testing tips by examples in PHP

Unit testing tips by examples in PHP Introduction In these times, the benefits of writing unit tests are huge. I think that most of the recently start

May 9, 2022
PHP libraries that makes Selenium WebDriver + PHPUnit functional testing easy and robust
PHP libraries that makes Selenium WebDriver + PHPUnit functional testing easy and robust

Steward: easy and robust testing with Selenium WebDriver + PHPUnit Steward is a set of libraries made to simplify writing and running robust functiona

Apr 29, 2022
PHPArch is a work in progress architectural testing library for PHP projects

PHPArch What is this? Installation Simple Namespace validation Available Validators Defining an architecture Syntactic sugar: Bulk definition of compo

May 18, 2022
:computer: Parallel testing for PHPUnit

ParaTest The objective of ParaTest is to support parallel testing in PHPUnit. Provided you have well-written PHPUnit tests, you can drop paratest in y

May 22, 2022
Few additional testing assertions for Laravel views

Laravel View Test Assertions Few additional assertions for testing Laravel views. Why Laravel has well established and documented way of testing reque

May 14, 2022
Real-world Project to learning about Unit Testing/TDD with Laravel for everybody

KivaNote - a Laravel TDD Sample Project Let me introduce you to KivaNote, a simple real-world application using Laravel to show you how the TDD & Unit

May 14, 2022
Magic Test allows you to write browser tests by simply clicking around on the application being tested, all without the slowness of constantly restarting the testing environment.

Magic Test for Laravel Magic Test allows you to write browser tests by simply clicking around on the application being tested, all without the slownes

May 10, 2022
Package for unit testing Laravel form request classes
Package for unit testing Laravel form request classes

Package for unit testing Laravel form request classes. Why Colin DeCarlo gave a talk on Laracon online 21 about unit testing Laravel form requests cla

May 11, 2022
A video course for laravel artisan to learn creating API using testing

About Laravel Laravel is a web application framework with expressive, elegant syntax. We believe development must be an enjoyable and creative experie

Apr 1, 2022
PHPUnit extension for database interaction testing.

This extension is no longer maintained DbUnit PHPUnit extension for database interaction testing. Installation Composer If you use Composer to manage

May 7, 2022
Very simple mock HTTP Server for testing Restful API, running via Docker.

httpdock Very simple mock HTTP Server for testing Restful API, running via Docker. Start Server Starting this server via command: docker run -ti -d -p

Dec 24, 2021