Fixes problem with executable in #53

To test it out:

1. Create a new directory and add following composer.json file to it

   {
       "minimum-stability": "dev",
       "require": {
           "sebastian/phpcpd": "dev-feature/composer-support@dev"
       },
       "repositories": [
           {
               "type": "vcs",
               "url": "https://github.com/Ocramius/phpcpd.git"
           }
       ]
   }
   

2. Run following:

   curl -s https://getcomposer.org/installer | php
   php composer.phar install
   

3. Run the cli tools:

    ./vendor/bin/phpcpd vendor/sebastian/phpcpd/src
   

   should produce

    phpcpd 1.4.0-11-g458de68 by Sebastian Bergmann.
   
    0.00% duplicated lines out of 1597 total lines of code.
   
    Time: 0 seconds, Memory: 2.25Mb
   

Same works without defining the composer.json file and using the cloned repository instead.

Could you add an annotation option to make it possible to disable CPD for:

• a whole file (@SuppressWarnings(PHPCPD) over class body) -a whole method (@SuppressWarnings(PHPCPD) over method body)
• several lines of code (@SuppressWarnings(PHPCPD-START) and @SuppressWarnings(PHPCPD-END))

Full description of algorithm here: https://www.cqse.eu/fileadmin/content/news/publications/2009-do-code-clones-matter.pdf

Original source available here: https://www.cqse.eu/en/news/blog/conqat-end-of-life/

Possible to detect Type 3 clones.

Example run:

php phpcpd --algorithm suffixtree --min-tokens 25 QuestionTheme.php

Question:

• I need to add new options, which are not supported by the abstract strategy. How about replacing the input args with a new class, StrategyConfiguration? Needed options are: --edit-distance (int), and --head-equality (int).
• ~~To be able to analyze more than one file, the input should not be "file" but a list of tokens. Maybe there's a memory saving mechanism in place right now, where only the hashes are saved between analysis? Which works for the Rabin-Karp default algorithm, but maybe not for the new one. Have to double check.~~

This PR adds a new --regexps-exclude option to counter a limitation with other options.

The --names-exclude option refers only to base name, which forbids to exclude a given path. For instance, if we want to exclude web/app_*.php it will not work. We have to exclude app_*.php through the --names-exclude. But it will also exclude src/app_test.php for instance.

The --regexps-exclude allows to specify a regexp which will then be inserted into the notPath method of the finder (a PR on finder-facade has been created). This way, we can exclude previous patterns with a command line such:

phpcpd.phar --regexps-exclude '~web/app_.*\.php~' myProject

This PR can probably helps to fix some issues with the --exclude option, and should be an answer to #84.

Version 1.3.1; who knows what steps to reproduce.

In some cases, the phpcpd textui can produce e_notices: PHP Notice: Undefined offset: 27498 in /usr/local/lib/php/pear/PHPCPD/TextUI/Command.php on line 260

Greetings. I am still new to phpcpd, but I think it could really become one of my favorite tools.

I am trying trying to detect duplicate code on different files, belonging to the same project. I am currently working on old stuff to renew, and I can find lots of cut-paste issues.

Nevertheless I have the impression that phpcpd allows only to find duplicate code inside each file, not comparing the contents of a file to another different one. Can you confirm this?

Is there a way to find any possible duplicate code in a project (no problem of memory nor time), examining all existing files, with your tool?

Thank you for your feedback. Cheers Emanuele

The following functions are detected as copy-pasted but their signatures are different:

<?php

function foo(?string $foo): ?string
{
    return $foo;
}

<?php

function foo(string $foo): string
{
    return $foo;
}

First off -- thanks for making this!

Not sure if anyone else has run into this issue, but can't install via Composer:

$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for sebastian/phpcpd * -> satisfiable by sebastian/phpcpd[1.4.1].
    - sebastian/phpcpd 1.4.1 requires sebastian/finder-facade dev-master -> no matching package found.

Looked through #58, but didn't see anything about this.

Hi,

i had some trouble running phpcpd in an environment where memory_limit was not big enough to run phpcpd over a big codebase. So i stored all hashes within a sqlite file to reduce the memory usage.

With the provided changes it is possible to switch the default storage (internal array) to a temporary sqlite database. Of course this makes the detection slower by using a sqlite database on the filesystem, but you can run phpcpd with a very small memory usage.

Here are some examples / comparisons running phpcpd on zendframework2 code:

Running with memory (default) storage (fast but a lot of memory used):

$ php phpcpd --progress --min-tokens=70 --min-lines=5 vendor/zendframework/
phpcpd 2.0.1-3-ge862208 by Sebastian Bergmann.

[...]

2.49% duplicated lines out of 303394 total lines of code.

Time: 12.92 seconds, Memory: 177.25Mb

Running with sqlite storage (slower but less memory used):

$ php phpcpd --progress --min-tokens=70 --min-lines=5 --hash-storage=SQLite vendor/zendframework/
phpcpd 2.0.1-3-ge862208 by Sebastian Bergmann.

[...]

2.49% duplicated lines out of 303394 total lines of code.

Time: 36.47 seconds, Memory: 16.25Mb

Additionally it is now very easy to add another HashStorageAdapter by providing a class that implements HashStorageInterface.

Best regards, Matthias

phpcpd --log-pmd ./build/logs/phploc.csv ./ phpcpd 1.4.0 by Sebastian Bergmann.

PHP Fatal error: Class 'Symfony\Component\Finder\Adapter\GnuFindAdapter' not found in /usr/share/pear/Symfony/Component/Finder/Finder.php on line 69 PHP Stack trace: PHP 1. {main}() /usr/bin/phpcpd:0 PHP 2. SebastianBergmann\PHPCPD\TextUI\Command->main() /usr/bin/phpcpd:56 PHP 3. SebastianBergmann\FinderFacade\FinderFacade->findFiles() /usr/share/pear/SebastianBergmann/PHPCPD/TextUI/Command.php:221 PHP 4. Symfony\Component\Finder\Finder->__construct() /usr/share/pear/SebastianBergmann/FinderFacade/FinderFacade.php:93

I don't know why ? how to fix it , thank you

Such option will be useful in a report stage during a CI process. Currently each copy/paste detection will trigger a error exit code. For example such option in phpmd is called --ignore-violations-on-exit.

In our projects we use the new PHP attributes for documentation on each controller. Theese are quite a few lines per endpoint.

As it is totaly fine that two endpoints might have the exact same documentation (for example an create and an update endpoint being called the same way) this should not be detected as a duplicate.

My suggestion would be to exclude PHP attributes #[...] in general from the detection, like it is already done for comments /** */

Hi Guys,

I am having a wired duplication on a single file, which I am sure there is no duplication and I can't get that's wrong with it. By removing --fuzzy parameter seems to be working, but I would like to use it. Can you please help?

• phpcpd 6.03
• php 7.4.29

Command:

phpcpd --fuzzy '/var/www/test/Brand.php'

Expected result:

No clones found.

Actual result:

Found 1 clones with 251 duplicated lines in 1 files:

  - /var/www/test/Brand.php:31-282 (251 lines)
    /var/www/test/Brand.php:40-291

86.25% duplicated lines out of 291 total lines of code.
Average size of duplication is 251 lines, largest clone has 251 of lines

Time: 00:00, Memory: 4.00 MB

File Content:

<?php

namespace CoreBundle\Entity\Company;

use Doctrine\Common\Collections\ArrayCollection;
use Doctrine\ORM\Mapping as ORM;

use JMS\Serializer\Annotation as JMS;
use Gedmo\Mapping\Annotation as Gedmo;

/**
 * Class Brand
 *
 * @ORM\Entity(repositoryClass="CoreBundle\Repository\Company\BrandRepository")
 * @ORM\Table(name="brand")
 *
 * @package CoreBundle\Entity\Company
 */
class Brand
{
    /* Algolia index name */
    const ALGOLIA_INDEX_NAME = 'BRAND';

    /**
     * @var int $id
     *
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;

    /**
     * @var string $name
     *
     * @ORM\Column(name="name", type="string", length=32)
     *
     * @Gedmo\Translatable(fallback=true)
     */
    protected $name;

    /**
     * @var string $nameAr
     *
     * @JMS\SerializedName("name_ar")
     * @JMS\Groups({"searchable"})
     */
    protected $nameAr;

    /**
     * @var string $slug
     *
     * @ORM\Column(name="slug", type="string")
     *
     * @Gedmo\Slug(fields={"name"}, updatable=false)
     */
    protected $slug;

    /**
     * @var bool $enabled
     *
     * @ORM\Column(name="enabled", type="boolean", options={"default":1})
     */
    protected $enabled;

    /**
     * @var bool $autoConfirmOrder
     *
     * @ORM\Column(name="auto_confirm_order", type="boolean", options={"default":0})
     */
    protected $autoConfirmOrder;

    /**
     * @var bool $cashPaymentAllowed
     *
     * @ORM\Column(name="cash_payment_allowed", type="boolean", options={"default":0})
     */
    protected $cashPaymentAllowed;

    /**
     * @var bool $nutritionalEnabled
     *
     * @ORM\Column(name="nutritional_enabled", type="boolean", options={"default":0})
     */
    protected $nutritionalEnabled;

    /**
     * @var bool $loyaltyEnabled
     *
     * @ORM\Column(name="loyalty_enabled", type="boolean", options={"default":0})
     */
    protected $loyaltyEnabled;

    /**
     * @var int $invoicePrintCopies
     *
     * @ORM\Column(name="invoice_print_copies", type="smallint", options={"default":2})
     */
    protected $invoicePrintCopies;

    /**
     * @var bool $aggregate
     *
     * @ORM\Column(name="aggregate", type="boolean", options={"default":0})
     */
    protected $aggregate;

    /**
     * @var \DateTime $aggregateDate
     *
     * @ORM\Column(name="aggregate_date", type="date", nullable=true)
     */
    protected $aggregateDate;

    /**
     * @var \DateTime $firstOrderDate
     *
     * @ORM\Column(name="first_order_date", type="date", nullable=true)
     */
    protected $firstOrderDate;

    /**
     * @var \DateTime $lastOrderDate
     *
     * @ORM\Column(name="last_order_date", type="date", nullable=true)
     */
    protected $lastOrderDate;

    /**
     * @var bool $supermarket
     *
     * @ORM\Column(name="supermarket", type="boolean", options={"default":0})
     */
    protected $supermarket;

    /**
     * @var MediaEntity $logo
     *
     * @ORM\OneToOne(targetEntity="CoreBundle\Entity\Media", cascade={"persist"})
     * @ORM\JoinColumn(name="logo_media_id", referencedColumnName="id")
     */
    protected $logo;

    /**
     * @var string $publicImageUrl
     *
     * @JMS\SerializedName("image_url")
     * @JMS\Groups({"searchable"})
     */
    protected $publicImageUrl;

    /**
     * @var MediaEntity $photo
     *
     * @ORM\OneToOne(targetEntity="CoreBundle\Entity\Media", cascade={"persist"})
     * @ORM\JoinColumn(name="photo_media_id", referencedColumnName="id")
     */
    protected $photo;

    /**
     * @var MediaEntity $productPhoto
     *
     * @ORM\ManyToOne(targetEntity="CoreBundle\Entity\Media", cascade={"persist"})
     * @ORM\JoinColumn(name="product_photo_media_id", referencedColumnName="id")
     */
    protected $productPhoto;

    /**
     * @var \DateTime $closingHour
     *
     * @ORM\Column(name="closing_hour", type="time")
     */
    protected $closingHour;

    /**
     * @var bool $vThruTwilioTaskQueue
     *
     * @ORM\Column(name="vthru_twilio_task_queue", type="boolean")
     */
    protected $vThruTwilioTaskQueue;

    /**
     * @var string $twilioTaskQueueSid
     *
     * @ORM\Column(name="twilio_task_queue_sid", type="text", nullable=true)
     */
    protected $twilioTaskQueueSid;

    /**
     * @var string $compensationEmails
     *
     * @ORM\Column(name="compensation_emails", type="string", nullable=true)
     */
    protected $compensationEmails;

    /**
     * @deprecated
     * @var string
     *
     * @ORM\Column(name="gateway_destination_id", type="string", nullable=true)
     */
    protected $gatewayDestinationId;

    /**
     * @var FeeEntity $fee
     *
     * @ORM\OneToOne(targetEntity="CoreBundle\Entity\Company\Brand\Fee", inversedBy="brand", cascade={"persist"})
     */
    protected $fee;

    /**
     * @var BrandEntity\Financial $financial
     *
     * @ORM\OneToOne(targetEntity="CoreBundle\Entity\Company\Brand\Financial", inversedBy="brand", cascade={"persist"})
     */
    protected $financial;

    /**
     * @var LoyaltyEntity $loyalty
     *
     * @ORM\OneToOne(targetEntity="CoreBundle\Entity\Company\Brand\Loyalty", cascade={"persist"})
     * @ORM\JoinColumn(name="loyalty_id", referencedColumnName="id", nullable=true)
     */
    protected $loyalty;

    /**
     * @var ArrayCollection $loyalties
     *
     * @ORM\OneToMany(targetEntity="CoreBundle\Entity\Company\Brand\Loyalty", mappedBy="brand", cascade={"persist"})
     */
    protected $loyalties;

    /**
     * @var CompanyEntity $company
     *
     * @ORM\ManyToOne(targetEntity="CoreBundle\Entity\Company", inversedBy="brands")
     * @ORM\JoinColumn(name="company_id", referencedColumnName="id")
     */
    protected $company;

    /**
     * @var ArrayCollection $stores
     *
     * @ORM\OneToMany(targetEntity="CoreBundle\Entity\Company\Brand\Store", mappedBy="brand")
     */
    protected $stores;

    /**
     * @var ArrayCollection $products
     *
     * @ORM\OneToMany(targetEntity="CoreBundle\Entity\Product", mappedBy="brand")
     */
    protected $products;

    /**
     * @var ArrayCollection $categories
     *
     * @ORM\OneToMany(targetEntity="CoreBundle\Entity\Product\Category", mappedBy="brand")
     */
    protected $categories;

    /**
     * @var ArrayCollection $tags
     *
     * @ORM\ManyToMany(targetEntity="CoreBundle\Entity\Company\Brand\Tag", mappedBy="brands")
     */
    protected $tags;

    /**
     * @var WebsiteEntity $website
     *
     * @ORM\OneToOne(targetEntity="CoreBundle\Entity\Company\Brand\Website", inversedBy="brand", cascade={"persist"})
     */
    protected $website;

    /**
     * @var IntegrationEntity $integration
     *
     * @ORM\OneToOne(targetEntity="IntegrationBundle\Entity\Integration")
     * @ORM\JoinColumn(name="integration_id", referencedColumnName="id")
     */
    protected $integration;

    /**
     * @deprecated
     * @var string $tapDestResponse
     *
     * @ORM\Column(name="tap_dest_response", type="text", nullable=true)
     */
    protected $tapDestResponse;
}

With version 6 the "--exclude" option is no longer working as expected:

Command: bin/phpcpd --exclude TCA --fuzzy packages

This should ignore every file which has "TCA" in its path. With version 5.0 this works great and I only get outputs from files without "TCA" in it's path. Using 6.0 it will also lead to messages about files in a folder called "TCA".

Output with 5.0.2:

bin/phpcpd --exclude TCA --fuzzy packages
phpcpd 5.0.2 by Sebastian Bergmann.
Found 2 clones with 79 duplicated lines in 4 files:
  - /my-project-path/packages/mypackage/Classes/Domain/Model/Entity.php:248-302 (54 lines)
    /my-project-path/packages/mypackage/Classes/Domain/Model/AnotherEntity.php:140-194
  - /my-project-path/packages/mypackage/Classes/Controller/ExampleController.php:102-127 (25 lines)
    /my-project-path/packages/mypackage/Classes/Controller/AnotherExampleController.php:77-102
1.56% duplicated lines out of 5049 total lines of code.
Average size of duplication is 39 lines, largest clone has 54 of lines
Time: 00:00.051, Memory: 6.00 MB

Output with 6.0.2 (also tested 6.0.3):

bin/phpcpd --exclude TCA --fuzzy packages
phpcpd 6.0.2 by Sebastian Bergmann.
Found 7 clones with 655 duplicated lines in 14 files:
  - /my-project-path/packages/mypackage/Configuration/TCA/some_php.php:10-93 (83 lines)
    /my-project-path/packages/mypackage/Configuration/TCA/some_more_php.php:10-93
  - /my-project-path/packages/mypackage/Configuration/TCA/even_more_php.php:26-105 (79 lines)
    [...]
  - /my-project-path/packages/mypackage/Classes/Domain/Model/Entity.php:248-302 (54 lines)
    /my-project-path/packages/mypackage/Classes/Domain/Model/AnotherEntity.php:140-194
  - /my-project-path/packages/mypackage/Configuration/TCA/some_php2.php:26-69 (43 lines)
    /my-project-path/packages/mypackage/Configuration/TCA/some_more_php2.php:26-69
    /my-project-path/packages/mypackage/Configuration/TCA/even_more_php2.php:26-69
  - /my-project-path/packages/mypackage/Classes/Controller/ExampleController.php:102-127 (25 lines)
    /my-project-path/packages/mypackage/Classes/Controller/AnotherExampleController.php:77-102
7.90% duplicated lines out of 8288 total lines of code.
Average size of duplication is 93 lines, largest clone has 83 of lines
Time: 00:00.023, Memory: 6.00 MB

Hope this description helps a bit.

Thanks and have a nice day! Alex

Hi folks! Using phpcpd in a large project with many cp's found, a whitelist parameter would be helpful.

I would like to use it like this to only output the c+p from modified files compared to origin/master: phpcpd src --whitelist $(git diff origin/master -G. --diff-filter=AM --name-only | grep src/ | paste -sd "," -)

This way I can utilize my CI to make sure that new commits never contain C+P code, even though some old code may have c+p parts... at least until someone makes a change in some of the old files, then they have to fix that anyway, which is fine for me.

I tried using phpcpd --whitelist $(git diff origin/master -G. --diff-filter=AM --name-only | grep src/ | paste -sd " " -) but that only executes phpcpd on the changed files, therefore the files that contain the copied source will most certainly never be in the changelist and there will never be any issues reported by phpcpd.

A whitelist would work just like phpcpd src with the tiny difference, that the issues found are filtered and only respected when one of the files matches the whitelisted files.

I hope I managed to write that down somewhat understandable.