🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Hi friends,

First, sorry for not following the right process to add a new crawler. I'm a little newbie in this world.

I just found that facebook is using a new structure to crawle ads links.

Look the user-agents that I got here:

Mozilla\/5.0 (Linux; Android 6.0.1; SM-J500M Build\/MMB29M; wv) AppleWebKit\/537.36 (KHTML, like Gecko) Version\/4.0 Chrome\/62.0.3202.84 Mobile Safari\/537.36 [FB_IAB\/FB4A;FBAV\/152.0.0.42.136;]

Mozilla\/5.0 (Linux; Android 6.0.1; SM-J700M Build\/MMB29K; wv) AppleWebKit\/537.36 (KHTML, like Gecko) Version\/4.0 Chrome\/62.0.3202.84 Mobile Safari\/537.36 [FB_IAB\/FB4A;FBAV\/152.0.0.42.136;]

Mozilla\/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit\/603.3.8 (KHTML, like Gecko) Mobile\/14G60 [FBAN\/FBIOS;FBAV\/150.0.0.32.132;FBBV\/80278251;FBDV\/iPhone6,2;FBMD\/iPhone;FBSN\/iOS;FBSV\/10.3.3;FBSS\/2;FBCR\/O2;FBID\/phone;FBLC\/en_GB;FBOP\/5;FBRV\/0]

Mozilla\/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit\/604.3.5 (KHTML, like Gecko) Mobile\/15B202 [FBAN\/FBIOS;FBAV\/151.0.0.61.202;FBBV\/82156572;FBDV\/iPhone6,2;FBMD\/iPhone;FBSN\/iOS;FBSV\/11.1.2;FBSS\/2;FBCR\/TIM;FBID\/phone;FBLC\/pt_BR;FBOP\/5;FBRV\/83160404]

I just attached a XLS file with all user-agents I got here.

facebook-ads-crawlers.xlsx

Can someone help me to add it to the crawler list? =)

Hi,

Still parsing my logs, hits by Google googlebot are not detected. The trick is that it uses a valid User-agent - Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/28.0.1500.71 Safari\/537.36 - but you should read from HTTP From header, whose value is googlebot(at)googlebot.com. It is a big modification, but is it considered?

This allows, for instance, a custom app to whitelist / blacklist custom agents.

For example, this is usercode from an app which needs to manually whitelist wechat.

    /**
     * Check if a request is a crowler, with a wechat exemption
     *
     * @param HTTPRequest $request
     * @return bool
     */
    protected function checkIsCrawler(HTTPRequest $request): bool
    {
        // Add wechat exemption to crawler detection
        $exclusions = new Exclusions();
        $exclusionList = array_merge(
            $exclusions->getAll(),
            ['MicroMessenger\/']
        );
        $exclusions->setAll($exclusionList);

        // Set custom exclusions to new detector
        $detect = new CrawlerDetect();
        $detect->setExclusions($exclusions);

        // Detect bots
        $userAgent = $request->getHeader('User-Agent');
        return $detect->isCrawler($userAgent);
    }

Setters return $this to support chaining. E.g. $crawler->setUaHttpHeaders($headers)->setHeaders($appHeaders)

Hi Guys,

Thanks for the amazing script , it works amazing for us .

But today we can see that Googlebot with IP 66.249.73.157 has been detected and blocked! When looking the IP up , it seems to be the real one for Googlebot .

Any way to exclude the well known crawlers :) ? Thanks again

Hi!

This seems to be a bot, see also https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/

Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/43.0.2357.121 Mobile Safari/537.36 LieBaoFast/4.51.3
Mozilla/5.0(Linux;Android 5.1.1;OPPO A33 Build/LMY47V;wv) AppleWebKit/537.36(KHTML,link Gecko) Version/4.0 Chrome/42.0.2311.138 Mobile Safari/537.36 Mb2345Browser/9.0
Mozilla/5.0(Linux;U;Android 5.1.1;zh-CN;OPPO A33 Build/LMY47V) AppleWebKit/537.36(KHTML,like Gecko) Version/4.0 Chrome/40.0.2214.89 UCBrowser/11.7.0.953 Mobile Safari/537.36

Hi, really appreciate your work, guys, the package is great.

But why is there no way to mannualy extend the default crawlers list? For example i need to add "Sendsay.Ru/1.0; https://Sendsay.Ru/; [email protected]" (it's a niche Russian crawler), but i have no options but to write an issue here. It would be really great if we could have some kind of a configuration. Or at least have Crawlers injected in CrawlerDetect class via constructor so we could extend it and inject our custom Crawlers implementation without rewriting the whole constructor.

Thanks)

Because of issue #427 I think we shouldn't remove single characters in the exclusions list.

The exclusions were partly to improve performance. We see much greater returns (by using these exclusions) when running over multiple user agents, so on a single request this should make very little to no difference.

UserAgent

Mozilla/5.0 (Linux; Android 7.0; FRD-AL00 Build/HUAWEIFRD-AL00; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/53.0.2785.49 Mobile MQQBrowser/6.2 TBS/043602 Safari/537.36 MicroMessenger/6.5.16.1120 NetType/WIFI Language/zh_CN

Hello,

We use crawler-detect to detect social networks bots and we've noticed that some bot user agents passed the tests. There they are : Twitter :

• Mozilla/5.0 (compatible; TrendsmapResolver/0.1)
• Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)
• Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
• Mozilla/5.0
• Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)

Facebook :

• Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_3 like Mac OS X) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.0 Mobile/14G60 Safari/602.1
• Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1
• Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1
• Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1
• Mozilla/5.0 (iPhone; CPU iPhone OS 12_0_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1

I've put them in the tests/crawlers.txt but we cannot differenciate these user agents from ordinary user agents (except for the first user agent) so I just add the TrendsmapResolver to the Fixtures/Crawlers.php.

Could you please let me know how to recognize that tey're bots ?

Yours sincerely,

Mathilde

Mozilla/5.0 (Linux; Android 6.0.1; LG-K100 Build/MXB48T; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/50.0.2661.86 Mobile Safari/537.36 YandexSearch/5.75

https://github.com/JayBizzle/Crawler-Detect/blob/master/raw/Crawlers.txt#L800

bunkerized-nginx uses these lists, and it blocks access to home-assistant when trying to log in as:

2021/02/16 12:55:07 [warn] 24951#24951: *35 [lua] access_by_lua(main-lua.conf:166):80: [BLOCK] User-Agent okhttp/4.9.0 is blacklisted, client: 10.0.2.100, server: www.example.com, request: "POST /auth/token HTTP/2.0", host: "www.example.com"

Additional information

https://github.com/home-assistant/android/search?q=okhttp&type=code

Hello guys.

First of all, thank you for your package. It is really helpful for projects I've been doing.

Literally, I'm referencing existing issue #309 but from a little different point of view. My goal is not to simply add a few exotic bots to the list. I aim to exclude the custom-configured monitoring systems that we use like Zabbix, Munin, etc. They perform health checks using HTTP requests with specific UA suffix, which we are configuring by ourselves. It can be like, ${PROJECT_NAME}Monitoring, ${DOMAIN}Robot, ${NODE_NAME}Zabbix, etc.

And at this point, we're stuck. Cause of:

• the code of this package doesn't imply extensibility of the crawlers list
• adding our-specific UA suffixes to the crawlers list will not make any benefit to the community

So my question: is any chance that you'll review your position about code extensibility or maybe you have any workaround?

As for now, I see the one and a little dumb way to get the goal (excluding project fork) is to make custom classes extend from Crawlers and CrawlerDetect, so CustomCrawlers will contain extra lines for custom monitoring systems and CustomCrawlerDetect will use it.

This is not a firm decision, just exploring the possibility only supporting officially supported PHP versions - https://www.php.net/supported-versions.php

Hi,

I've install Crawler-Detect, it work fine but urlscan.io can crawl. For now, I still have not succeeded in obtaining their fingerprints but I prefer to warn, maybe you will.

Cordially

Lines extracts from a few websites in Russia with high traffic. Each suggested line checked with «User agent is NOT a bot» result.

Sample sources.

@JayBizzle please, approve list and after your «OK» i'll send PR.

Confirmed

• [x] HTTP Tester/1.0 https://apps.apple.com/us/app/httpbot/id1232603544

• [x] SimonWebHelper/1.0 CFNetwork/1179.0.1 Darwin/20.0.0 https://www.dejal.com/simon

• [x] saelmon https://selectel.ru/services/additional/monitoring

• [x] Zadarma API https://zadarma.com/ru/support/api

• [x] Mozilla/5.0 (compatible; Artax) https://github.com/amphp/artax

• [x] iOS/14.0 (18A5319i) dataaccessd/1.0 «...process responsible for background syncing of Exchange, iCloud, CalDAV, and other calendar data...»

• [x] Atlassian Webhook HTTP Client Manual verified with private Atlassian JIRA instance. I can't find any public mention in docs, blog, etc.

• [x] macOS/11.0 (20A5323l) CalendarAgent/949

• [x] Mac+OS+X/10.15.6 (19G46c) CalendarAgent/930.5.1

• [x] TLS tester from https://testssl.sh/dev/

• [x] Orpho.Ru - v. 1.5.5

• [x] Corax - [email protected]

• [x] Mozilla/5.0 (compatible; Domains Project/1.1.0; +https://domainsproject.org)

Suspicious

• [ ] AppleCoreMedia/1.0.0.15E216 (iPhone; U; CPU OS 11_3 like Mac OS X; ru_ru)

• [ ] U7IA53O8WRQLP5HTN36O 520895322210219123 178783837217128874 128152866426197781 170646164760074160 897891597049094717 10798953746823634 865980663587406199 83960315279034295 50879782656773070 425389085694464584 520832197366222982 566888253833316094 497618096526395537 425659151411788951 494971665505918016 870530337619880183 349139421128410017 627636913921908195 757052309832051516 923961866520554027 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

• [ ] P5IFYN4 466155777352526040 187582494976457628 205329424338575162 290476900380796124 777992341037289367 321980071372304533 644559670148391318 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36

• [ ] Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; MRSPUTNIK 2, 4, 0, 386; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; BRI/2; AskTbFXTV5/5.14.1.20007)

• [ ] Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Pivim Multibar; MRSPUTNIK 2, 4, 0, 171; MRA 5.7 (build 03796))