Hello,
we noticed that our phpunit tests started to freeze before execution for up to 1 minute. After debugging, I found that the bottclneck is in the $fileIteratorFacade->getFilesAsArray(...)
method.
It foreach
es over every file in the folders recursevely, even if the folders are excluded. Let me explain it with the example:
Consider we have the following structure:
first
├── exclude
│ ├── exclude1.php
│ └── exclude2.php
├── first1.php
└── second
├── second1.php
└── second2.php
and the following simple script:
<?php
require 'vendor/autoload.php';
use SebastianBergmann\FileIterator\Factory;
$factory = new Factory();
$iterator = $factory->getFileIterator('first', '', '', ['first/exclude']);
foreach ($iterator as $file) {
echo $file->getRealPath(), "\n\n";
}
When I add a debugging code to the vendor here:
https://github.com/sebastianbergmann/php-file-iterator/blob/7f0f29702170e2786b2df813af970135765de6fc/src/Iterator.php#L73-L77
I get the following result:
/tmp/phpunit-filter/first/first1.php
>> Excluding /tmp/phpunit-filter/first/exclude/exclude2.php
>> Excluding /tmp/phpunit-filter/first/exclude/exclude1.php
/tmp/phpunit-filter/first/second/second2.php
/Users/maksrafalko/tmp/phpunit-filter/first/second/second1.php
Please note that while first/exclude
folder is excluded, Iterator keeps traversing over all files from excluded folder.
Imagine, we have hundred of thousands of files (vendor
folders) inside excluded folder, but each file is anyway processed.
What is the right way to do it?
From my opinion, folders that are excluded
must not be traversed. Let's see how symfony/finder
does it. Again, create a simple script for the same folder structure:
<?php
require 'vendor/autoload.php';
use Symfony\Component\Finder\Finder;
$finder = new Finder();
$finder->files()->in('first')->exclude('exclude');
foreach ($finder as $file) {
echo $file->getRealPath(), "\n\n";
}
and let's see the result:
/tmp/phpunit-filter/first/first1.php
>> Excluding exclude
/tmp/phpunit-filter/first/second/second2.php
/tmp/phpunit-filter/first/second/second1.php
As you can see, symfony/finder
meets first/excluded
folder, sees that it is excluded and does not traverse it farther.
This is done thanks to FilterIterator
and its implementation in symfony/finder
: ExcludeDirectoryFilterIterator
- Do you think we need to implement something similar in
php-file-iterator
?
- Do you have any objections or know something that is incompatible with the filter iterator?
Thank you.
https://github.com/symfony/finder/blob/1d4d30533fa8e343a85f6c51b0cba1ef5d041929/Iterator/ExcludeDirectoryFilterIterator.php#L57-L59