I'm trying to integrate this into an app that is already locked to psr/log:^3.

jeremykendall/php-domain-parser version 5 is locked to psr/log:^1

There are 2 tests that fail when bumping to the new version. I can fix and submit a PR.

Hi

Its a great project, we use that library, its great :+1: thanks for all the work!

We are not sure what the versioning concept of this library looks like, also we could not find information about what has changed (a changelog), there are only tags: https://github.com/spekulatius/PHPScraper/tags. So it would be nice to read something about what is the versioning strategy and when are breaks expected.

Of course semver would be nice with an 1.0.0 release and a CHANGELOG.md and UPGRADE.md. My personal example would look like this https://github.com/nadar/quill-delta-parser/blob/master/CHANGELOG.md and this https://github.com/nadar/quill-delta-parser/blob/master/UPGRADE.md

Thanks and keep up to great work!

I'm working on a project that needs to be constantly changing proxies and taking a look in the Goute client, the underlying implementation already supports it.

Research

1. HttpBrowser, which Goute is based on, supports a custom client, implementing the HttpClientInterface (source code)

...
class HttpBrowser extends AbstractBrowser
{
    private $client;

    public function __construct(HttpClientInterface $client = null, History $history = null, CookieJar $cookieJar = null)
...
        $this->client = $client ?? HttpClient::create();
...

2. The HttpClient::create() supports proxy using the $defaultOptions parameter, that gets passed to the selected HttpClient (source code)

        public static function create(array $defaultOptions = [], int $maxHostConnections = 6, int $maxPendingPushes = 50): HttpClientInterface

Implementation details

The idea is to expose this functionality though a setProxy function in the core class. The library will continue to dynamically select the httpClient accordingly.

I just made a POC in my fork with all necessary code to make this work (a6589da).

public function setProxy(string $proxy)
{
    $httpClient = HttpClient::create([
        'proxy' => $proxy
    ]);

    $this->client = new Client($httpClient);

    return $this;
}

How to use

$web = new phpscraper;
$web->__call('setProxy', [
    'http://user:[email protected]:3128',
]);

If this feature gets approved, I will open a PR with it. If anything needs to be changed, let me know.

I am using this example https://phpscraper.de/examples/extract-keywords.html with PHP 8.1.8

Deprecated: strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in /Users/khanakia/D1/www/php/scrap_php/vendor/spekulatius/phpscraper/src/phpscraper.php on line 858
PHP Deprecated:  strpos(): Passing null to parameter #1 ($haystack) of type string is deprecated in /Users/khanakia/D1/www/php/scrap_php/vendor/spekulatius/phpscraper/src/phpscraper.php on line 859

I wanted to know the response code of the url but could not get access.

Looking at Goutte under the hood, I think I could get it if I could do something like:

$web->getClient()->getInternalResponse()->getStatusCode()

So, I would like a function to get access to the $client, like:

public function getClient()
{
   return $this->client;
}

Hi

Im getting an

SSL connect error for "https://bbc.in/3cNMnkw".

when connecting to some websites. I'm writing a tweetdeck style system and so will scrape the site that the tweet links to to pick up the details so that a nicer link can be generated. Any ideas on how to solve this ? Most work fine, its just a few that fail.

Also Im sometimes getting a "This browser is no longer supported" coming through as a description from the page. I notice you are passing through 'Mozilla/5.0 (compatible; PHP Scraper/0.x; +https://phpscraper.de)' as the user agent. Is it worth altering the user agent ? I dont think I can pass it in as a parameter anywhere from what I can see?

Thanks.

When i run the sample code:

$web = new \Spekulatius\PHPScraper\PHPScraper();
$web->go('https://www.google.com/');
echo $web->title;

It return:

Spekulatius\PHPScraper\Core::setHttpClient(): Argument #1 ($httpClient) must be of type Symfony\Component\HttpClient\CurlHttpClient, Symfony\Component\HttpClient\NativeHttpClient given, called in C:\www\web-crawer\vendor\spekulatius\phpscraper\src\PHPScraper.php on line 108

Environments

PHP: 8.1.13 PHPScraper: 1.0.1

In my branch I've removed the magic __get and __call methods, and moved what was core into the phpscraper, so now there is only one class.

After a while I got tired of find/replace, so I created a rector rule to change the properties to method.

Is there a demo repository that uses I can use to test my branch? Tests are passing, except for the ones related to internal/external links, which I'll address in another issue.

Is this method used? Where is charset() defined?

    /**
     * Get the header collected as an array
     *
     * @return array
     */
    public function headers()
    {
        return [
            'charset' => $this->charset(),
            'contentType' => $this->contentType(),
            'viewport' => $this->viewport(),
            'canonical' => $this->canonical(),
            'csrfToken' => $this->csrfToken(),
        ];
    }

Symfony\Component\HttpClient\Exception\TransportException: SSL certificate problem: unable to get local issuer certificate for ***

Getting an error when trying to get data from a link that doesn't have ssl

Problem

Running composer require spekulatius/phpscraper fails on macOS 10.15.5 with the following:

[RuntimeException]                                                                                                                                                                                                          
 Failed to extract spekulatius/phpscraper: (50) '/usr/bin/unzip' -qq '/xxx/xxx/vendor/composer/tmp-123dcc14bbc3272649fdd489b0ecc9a3' -d '/xxx/xxx/vendor/composer/74030a62'  
                                                                                                                                                                                                                            
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/tests/resources/assets/katze-+?-++-+?.jpg                                                                    
        Illegal byte sequence                                                                                                                                                                                               
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/tests/resources/assets/???.jpg                                                                               
        Illegal byte sequence                                                                                                                                                                                               
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/websites/test-pages/assets/katze-+?-++-+?.jpg                                                                
        Illegal byte sequence                                                                                                                                                                                               
error:  cannot create /xxx/xxx/vendor/composer/74030a62/spekulatius-PHPScraper-fb4f590/websites/test-pages/assets/???.jpg                                                                           
        Illegal byte sequence                                                                                                                                                                                               

It seems like the unzip command within composer fails due to the strange characters.

Hi @spekulatius,

Some user using my https://github.com/datlechin/flarum-link-preview that used PHPScrapper, They see the scrap content is in other language, is the website content based on where the website is hosted?

https://discuss.flarum.org/d/30011-link-preview/178

Thanks, so it actually depends on the header. So far headers haven't been processed much. Exposing them would be beneficial in general. What do you think?

Originally posted by @spekulatius in https://github.com/spekulatius/PHPScraper/pull/164#discussion_r1045676088

E.g.

// GET request
$web->get('https://...');

// POST request
$response = $web->post('https://...', [
  'param' => 'first param',
]);

// ...

This could be done either directly in PHPScraper or built upon another specialized lib such as Symfony HTTP. Exposing the functionality of the existing dependency sounds like a reasonable way to go, if the idea is of interest.

#16 proposes adding support for JSONLD.

There isn't only JSONLD - structured data can be provided also in the microdata notation, and: good news - there is a project which parses microdata and converts it to the same data structure as JSONLD: https://github.com/yusufkandemir/microdata-parser

So it should be possible to use both and treat it just like an additional JSONLD block!

A first test:

$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromHTML($web->client->getResponse()->getContent(), $web->currentUrl())->toArray();

Internally this project uses an own DOM document class derived from DOMDocument. It has a function to import a DOMDocument - but Symphonys response class doesn't allow to access the DOMDocument.

I did a small test, but didn't fiddle it out how to pass the DOMDocument without reparsing - my try which didn't work:

$dom = new \DOMDocument('1.0', 'UTF-8');
$dom->importNode($web->filterFirst('//*')->getNode(0), true);
$jsonlddata = \YusufKandemir\MicrodataParser\Microdata::fromDOMDocument($dom);

What makes sense? Adding separate PHPScraper functions for JSONLD and microdata? Or mixing both automatically? (my opinion: mixing)

How should support for microdata look like? Adding the other project to PHPScraper? Extending the existing classes or porting the whole functionality to PHPScraper?

How do I check if the scraped page is an error page? It would be helpful to have the http status for that - something like

$web->go('https://httpstat.us/404');
echo $web->status;     // prints 404

Just before posting, I found the solution myself:

$web->client->getResponse()->getStatusCode()

It would be great to have this added to the documentation. Or maybe add $web->status as shortcut?