roach-php-bundle
Symfony bundle for Roach PHP.
Roach is a complete web scraping toolkit for PHP. It is
a shameless cloneheavily inspired by the popular Scrapy package for Python.
The Symfony bundle mostly provides the necessary container bindings for the various services Roach uses, as well as making certain configuration options available via a config file. To learn about how to actually start using Roach itself, check out the rest of the documentation.
Installing the Symfony bundle
Add nelexa/roach-php-bundle to your composer.json file:
composer require nelexa/roach-php-bundle
Register the bundle:
Register bundle into config/bundles.php (Flex did it automatically):
return [
//...
\Nelexa\RoachPhpBundle\RoachPhpBundle::class => ['all' => true],
];
Available Commands
The Symfony bundle of Roach registers a few console commands to make out development experience as pleasant as possible.
Run spider
php bin/console roach:run
After that, you will get the entire list of available spiders.
Choose a spider class:
[0] App\Spider\GoogleSpider
[1] App\Spider\FacebookSpider
[2] App\Spider\TwitterSpider
Simply select the desired spider (▼ or ▲) or enter its number and press Enter.
You can pass as the first argument the name spider class name to run or its alias. For example, if you have a class App\Spider\GoogleSpider, then you can pass the following aliases: GoogleSpider, google_spider or google.
php bin/console roach:run google
Sometimes it is useful to override the number of concurrent requests and the pre-request delay. To do this, you can pass the --concurrency and --delay options.
php bin/console roach:php google --concurrency 8 --delay 2
These options override the $concurrency and $requestDelay public properties of your spider.
Starting the REPL
Roach ships with an interactive shell (often called Read-Evaluate-Print-Loop, or Repl for short) which makes prototyping our spiders a breeze. We can use the provided roach:shell command to launch a new Repl session.
php bin/console roach:shell "https://roach-php.dev/docs/introduction"
Generator classes
First install Symfony MakerBundle.
composer require --dev symfony/maker-bundle
Create a new roach spider class
php bin/console make:roach:spider
Create a new roach extension class
php bin/console make:roach:extension
Create a new roach item processor class
php bin/console make:roach:item:processor
Create a new roach downloader request middleware class
php bin/console make:roach:middleware:downloader:request
Create a new roach downloader response middleware class
php bin/console make:roach:middleware:downloader:response
Create a new roach spider item middleware class
php bin/console make:roach:middleware:spider:item
Create a new roach spider request middleware class
php bin/console make:roach:middleware:spider:request
Create a new roach spider response middleware class
php bin/console make:roach:middleware:spider:response
Screencast
Credits
Changelog
Changes are documented in the releases page.
License
The MIT License (MIT). Please see LICENSE for more information.
1.2k Dec 19, 2022
1 Mar 24, 2022
5 Dec 18, 2021
1.1k Jan 3, 2023
11 Dec 15, 2022
8 May 6, 2022
11 Sep 17, 2022
2 Nov 5, 2021
283 Dec 15, 2022
720 Dec 25, 2022
199 Jan 3, 2023
35 Nov 17, 2022
7 Oct 20, 2022
27 Jul 7, 2022
14 May 23, 2022
98 Jan 4, 2023
5 Aug 20, 2022
10 Jan 8, 2023
1.6k Dec 30, 2022
![[:question:] Как правильно передать option для middleware ?](https://avatars.githubusercontent.com/u/23055853?v=4)
2.7k Dec 31, 2022
327 Dec 30, 2022
9.1k Jan 1, 2023
1.3k Dec 28, 2022
1.7k Dec 30, 2022
2.5k Dec 27, 2022
46 Sep 21, 2022
1 Nov 26, 2021
1 Dec 22, 2021