Public Suffix List based domain parsing implemented in PHP

Jeremy Kendall

Last update: Jan 1, 2023

Related tags

URL php-domain-parser

Overview

PHP Domain Parser

PHP Domain Parser is a resource based domain parser implemented in PHP.

Motivation

While there are plenty of excellent URL parsers and builders available, there are very few projects that can accurately parse a domain into its component subdomain, registrable domain, second level domain and public suffix parts.

Consider the domain www.pref.okinawa.jp. In this domain, the public suffix portion is okinawa.jp, the registrable domain is pref.okinawa.jp, the subdomain is www and the second level domain is pref.
You can't regex that.

PHP Domain Parser is compliant around:

accurate Public Suffix List based parsing.
accurate IANA Top Level Domain List parsing.

Installation

Composer

$ composer require jeremykendall/php-domain-parser

System Requirements

You need:

PHP >= 7.4 but the latest stable version of PHP is recommended
the intl extension

Usage

If you are upgrading from version 5 please check the upgrading guide for known issues.

Resolving Domains

This library can resolve a domain against:

The Public Suffix List
The IANA Top Level Domain List

In both cases this is done using the resolve method implemented on the resource instance. The method returns a Pdp\ResolvedDomain object which represents the result of that process.

For the Public Suffix List you need to use the Pdp\Rules class as shown below:

<?php 
use Pdp\Rules;
use Pdp\Domain;

$publicSuffixList = Rules::fromPath('/path/to/cache/public-suffix-list.dat');
$domain = Domain::fromIDNA2008('www.PreF.OkiNawA.jP');

$result = $publicSuffixList->resolve($domain);
echo $result->domain()->toString();            //display 'www.pref.okinawa.jp';
echo $result->subDomain()->toString();         //display 'www';
echo $result->secondLevelDomain()->toString(); //display 'pref';
echo $result->registrableDomain()->toString(); //display 'pref.okinawa.jp';
echo $result->suffix()->toString();            //display 'okinawa.jp';
$result->suffix()->isICANN();                  //return true;

For the IANA Top Level Domain List, the Pdp\TopLevelDomains class is use instead:

<?php

use Pdp\Domain;
use Pdp\TopLevelDomains;

$topLevelDomains = TopLevelDomains::fromPath('/path/to/cache/tlds-alpha-by-domain.txt');
$domain = Domain::fromIDNA2008('www.PreF.OkiNawA.jP');

$result = $topLevelDomains->resolve($domain);
echo $result->domain()->toString();            //display 'www.pref.okinawa.jp';
echo $result->suffix()->toString();            //display 'jp';
echo $result->secondLevelDomain()->toString(); //display 'okinawa';
echo $result->registrableDomain()->toString(); //display 'okinawa.jp';
echo $result->subDomain()->toString();         //display 'www.pref';
echo $result->suffix()->isIANA();              //return true

In case of an error an exception which extends Pdp\CannotProcessHost is thrown.

The resolve method will always return a ResolvedDomain even if the domain syntax is invalid or if there is no match found in the resource data. To work around this limitation, the library exposes more strict methods, namely:

Rules::getCookieDomain
Rules::getICANNDomain
Rules::getPrivateDomain

for the Public Suffix List and the following method for the Top Level Domain List:

TopLevelDomains::getIANADomain

These methods resolve the domain against their respective data source using the same rules as the resolve method but will instead throw an exception if no valid effective TLD is found or if the submitted domain is invalid.

All these methods expect as their sole argument a Pdp\Host implementing object, but other types (ie: string, null and stringable objects) are supported with predefined conditions as explained in the remaining document.

<?php

use Pdp\Domain;
use Pdp\Rules;
use Pdp\TopLevelDomains;

$publicSuffixList = Rules::fromPath('/path/to/cache/public-suffix-list.dat');
$domain = Domain::fromIDNA2008('qfdsf.unknownTLD');

$publicSuffixList->getICANNDomain($domain);
// will throw because `.unknownTLD` is not part of the ICANN section

$result = $publicSuffixList->getCookieDomain($domain);
$result->suffix()->value();   // returns 'unknownTLD'
$result->suffix()->isKnown(); // returns false
// will not throw because the domain syntax is correct.

$publicSuffixList->getCookieDomain(Domain::fromIDNA2008('com'));
// will not throw because the domain syntax is invalid (ie: does not support public suffix)

$result = $publicSuffixList->resolve(Domain::fromIDNA2008('com'));
$result->suffix()->value();   // returns null
$result->suffix()->isKnown(); // returns false
// will not throw but its public suffix value equal to NULL

$topLevelDomains = TopLevelDomains::fromPath('/path/to/cache/public-suffix-list.dat');
$topLevelDomains->getIANADomain(Domain::fromIDNA2008('com'));
// will not throw because the domain syntax is invalid (ie: does not support public suffix)

To instantiate each domain resolver you can use the following named constructor:

fromString: instantiate the resolver from a inline string representing the data source;
fromPath: instantiate the resolver from a local path or online URL by relying on fopen;

If the instantiation does not work an exception will be thrown.

WARNING:

You should never resolve domain name this way in production, without, at least, a caching mechanism to reduce PSL downloads.

Using the Public Suffix List to determine what is a valid domain name and what isn't is dangerous, particularly in these days when new gTLDs are arriving at a rapid pace.

If you are looking to know the validity of a Top Level Domain, the IANA Top Level Domain List is the proper source for this information or alternatively consider using directly the DNS.

If you must use this library for any of the above purposes, please consider integrating an updating mechanism into your software.

For more information go to the Managing external data source section

Resolved domain information.

Whichever methods chosen to resolve the domain on success, the package will return a Pdp\ResolvedDomain instance.

The Pdp\ResolvedDomain decorates the Pdp\Domain class resolved but also gives access as separate methods to the domain different components.

use Pdp\Domain;
use Pdp\TopLevelDomains;

$domain = Domain::fromIDNA2008('www.PreF.OkiNawA.jP');
/** @var TopLevelDomains $topLevelDomains */
$result = $topLevelDomains->resolve($domain);
echo $result->domain()->toString();            //display 'www.pref.okinawa.jp';
echo $result->suffix()->toString();            //display 'jp';
echo $result->secondLevelDomain()->toString(); //display 'okinawa';
echo $result->registrableDomain()->toString(); //display 'okinawa.jp';
echo $result->subDomain()->toString();         //display 'www.pref';
echo $result->suffix()->isIANA();              //return true

You can modify the returned Pdp\ResolvedDomain instance using the following methods:

<?php 

use Pdp\Domain;
use Pdp\Rules;

/** @var Rules $publicSuffixList */
$result = $publicSuffixList->resolve(Domain::fromIDNA2008('shop.example.com'));
$altResult = $result
    ->withSubDomain(Domain::fromIDNA2008('foo.bar'))
    ->withSecondLevelDomain(Domain::fromIDNA2008('test'))
    ->withSuffix(Domain::fromIDNA2008('example'));

echo $result->domain()->toString(); //display 'shop.example.com';
$result->suffix()->isKnown();       //return true;

echo $altResult->domain()->toString(); //display 'foo.bar.test.example';
$altResult->suffix()->isKnown();       //return false;

TIP: Always favor submitting a Pdp\Suffix object rather that any other supported type to avoid unexpected results. By default, if the input is not a Pdp\Suffix instance, the resulting public suffix will be labelled as being unknown. For more information go to the Public Suffix section

Domain Suffix

The domain effective TLD is represented using the Pdp\Suffix. Depending on the data source the object exposes different information regarding its origin.

<?php 
use Pdp\Domain;
use Pdp\Rules;

/** @var Rules $publicSuffixList */
$suffix = $publicSuffixList->resolve(Domain::fromIDNA2008('example.github.io'))->suffix();

echo $suffix->domain()->toString(); //display 'github.io';
$suffix->isICANN();                 //will return false
$suffix->isPrivate();               //will return true
$suffix->isPublicSuffix();          //will return true
$suffix->isIANA();                  //will return false
$suffix->isKnown();                 //will return true

The public suffix state depends on its origin:

isKnown returns true if the value is present in the data resource.
isIANA returns true if the value is present in the IANA Top Level Domain List.
isPublicSuffix returns true if the value is present in the Public Suffix List.
isICANN returns true if the value is present in the Public Suffix List ICANN section.
isPrivate returns true if the value is present in the Public Suffix List private section.

The same information is used when Pdp\Suffix object is instantiate via its named constructors:

<?php 
use Pdp\Suffix;

$iana = Suffix::fromIANA('ac.be');
$icann = Suffix::fromICANN('ac.be');
$private = Suffix::fromPrivate('ac.be');
$unknown = Suffix::fromUnknown('ac.be');

Using a Suffix object instead of a string or null with ResolvedDomain::withSuffix will ensure that the returned value will always contain the correct information regarding the public suffix resolution.

Using a Domain object instead of a string or null with the named constructor ensure a better instantiation of the Public Suffix object for more information go to the ASCII and Unicode format section

Accessing and processing Domain labels

If you are interested into manipulating the domain labels without taking into account the Effective TLD, the library provides a Domain object tailored for manipulating domain labels. You can access the object using the following methods:

the ResolvedDomain::domain method
the ResolvedDomain::subDomain method
the ResolvedDomain::registrableDomain method
the ResolvedDomain::secondLevelDomain method
the Suffix::domain method

Domain objects usage are explain in the next section.

<?php
use Pdp\Domain;
use Pdp\Rules;

/** @var Rules $publicSuffixList */
$result = $publicSuffixList->resolve(Domain::from2008('www.bbc.co.uk'));
$domain = $result->domain();
echo $domain->toString(); // display 'www.bbc.co.uk'
count($domain);           // returns 4
$domain->labels();        // returns ['uk', 'co', 'bbc', 'www'];
$domain->label(-1);       // returns 'www'
$domain->label(0);        // returns 'uk'
foreach ($domain as $label) {
   echo $label, PHP_EOL;
}
// display 
// uk
// co
// bbc
// www

$publicSuffixDomain = $result->suffix()->domain();
$publicSuffixDomain->labels(); // returns ['uk', 'co']

You can also add or remove labels according to their key index using the following methods:

<?php 
use Pdp\Domain;
use Pdp\Rules;

/** @var Rules $publicSuffixList */
$domain = $publicSuffixList->resolve(Domain::from2008('www.ExAmpLE.cOM'))->domain();

$newDomain = $domain
    ->withLabel(1, 'com')  //replace 'example' by 'com'
    ->withoutLabel(0, -1)  //remove the first and last labels
    ->append('www')
    ->prepend('docs.example');

echo $domain->toString();           //display 'www.example.com'
echo $newDomain->toString();        //display 'docs.example.com.www'
$newDomain->clear()->labels();      //return []
echo $domain->slice(2)->toString(); //display 'www'

WARNING: Because of its definition, a domain name can be null or a string.

To distinguish this possibility the object exposes two (2) formatting methods Domain::value which can be null or a string and Domain::toString which will always cast the domain value to a string.

use Pdp\Domain;

$nullDomain = Domain::fromIDNA2008(null);
$nullDomain->value();    // returns null;
$nullDomain->toString(); // returns '';

$emptyDomain = Domain::fromIDNA2008('');
$emptyDomain->value();    // returns '';
$emptyDomain->toString(); // returns '';

ASCII and Unicode formats.

Domain names originally only supported ASCII characters. Nowadays, they can also be presented under a UNICODE representation. The conversion between both formats is done using the compliant implementation of UTS#46, otherwise known as Unicode IDNA Compatibility Processing. Domain objects expose a toAscii and a toUnicode methods which returns a new instance in the converted format.

<?php 
use Pdp\Rules;

/** @var Rules $publicSuffixList */
$unicodeDomain = $publicSuffixList->resolve('bébé.be')->domain();
echo $unicodeDomain->toString(); // returns 'bébé.be'

$asciiDomain = $publicSuffixList->resolve('xn--bb-bjab.be')->domain();
echo $asciiDomain->toString();  // returns 'xn--bb-bjab.be'

$asciiDomain->toUnicode()->toString() === $unicodeDomain->toString(); //returns true
$unicodeDomain->toAscii()->toString() === $asciiDomain->toString();   //returns true

By default, the library uses IDNA2008 algorithm to convert domain name between both formats. It is still possible to use the legacy conversion algorithm known as IDNA2003.

Since direct conversion between both algorithms is not possible you need to explicitly specific on construction which algorithm you will use when creating a new domain instance via the Pdp\Domain object. This is done via two (2) named constructors:

Pdp\Domain::fromIDNA2008
Pdp\Domain::fromIDNA2003

At any given moment the Pdp\Domain instance can tell you whether it is in ASCII mode or not.

Once instantiated there's no way to tell which algorithm is used to convert the object from ascii to unicode and vice-versa

use Pdp\Domain;

$domain = Domain::fromIDNA2008('faß.de');
echo $domain->value(); // display 'faß.de'
$domain->isAscii();    // return false

$asciiDomain = $domain->toAscii(); 
echo $asciiDomain->value(); // display 'xn--fa-hia.de'
$asciiDomain->isAscii();    // returns true

$domain = Domain::fromIDNA2003('faß.de');
echo $domain->value(); // display 'fass.de'
$domain->isAscii();    // returns true

$asciiDomain = $domain->toAscii();
echo $asciiDomain->value(); // display 'fass.de'
$asciiDomain->isAscii();    // returns true

TIP: Always favor submitting a Pdp\Domain object for resolution rather that a string or an object that can be cast to a string to avoid unexpected format conversion errors/results. By default, and with lack of information conversion is done using IDNA 2008 rules.

Managing the package external resources

Depending on your application, the mechanism to store your resources may differ, nevertheless, the library comes bundle with a optional service which enables resolving domain name without the constant network overhead of continuously downloading the remote databases.

The interfaces and classes defined under the Pdp\Storage namespace enable integrating a resource managing system and provide an implementation example using PHP-FIG PSR interfaces.

Using PHP-FIG interfaces

The Pdp\Storage\PsrStorageFactory enables returning storage instances that retrieve, convert and cache the Public Suffix List and the IANA Top Level Domain List using standard interfaces published by the PHP-FIG.

To work as intended, the Pdp\Storage\PsrStorageFactory constructor requires:

a PSR-16 Simple Cache implementing library.
a PSR-17 HTTP Factory implementing library.
a PSR-18 HTTP Client implementing library.

When creating a new storage instance you will require:

a $cachePrefix argument to optionally add a prefix to your cache index, default to the empty string;
a $ttl argument if you need to set the default $ttl, default to null to use the underlying caching default TTL;

The $ttl argument can be:

an int representing time in second (see PSR-16);
a DateInterval object (see PSR-16);
a DateTimeInterface object representing the date and time when the item will expire;

The package does not provide any implementation of such interfaces as you can find robust and battle tested implementations on packagist.

Refreshing the resource using the provided factories

THIS IS THE RECOMMENDED WAY OF USING THE LIBRARY

For the purpose of this example we will use our PSR powered solution with:

Guzzle HTTP Client as our PSR-18 HTTP client;
Guzzle PSR-7 package which provide factories to create a PSR-7 objects using PSR-17 interfaces;
Symfony Cache Component as our PSR-16 cache implementation provider;

We will cache both external sources for 24 hours in a PostgreSQL database.

You are free to use other libraries/solutions/settings as long as they implement the required PSR interfaces.

<?php 

use GuzzleHttp\Psr7\Request;
use Pdp\Storage\PsrStorageFactory;
use Psr\Http\Message\RequestFactoryInterface;
use Psr\Http\Message\RequestInterface;
use Symfony\Component\Cache\Adapter\PdoAdapter;
use Symfony\Component\Cache\Psr16Cache;

$pdo = new PDO(
    'pgsql:host=localhost;port:5432;dbname=testdb', 
    'user', 
    'password', 
    [PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]
);
$cache = new Psr16Cache(new PdoAdapter($pdo, 'pdp', 43200));
$client = new GuzzleHttp\Client();
$requestFactory = new class implements RequestFactoryInterface {
    public function createRequest(string $method, $uri): RequestInterface
    {
        return new Request($method, $uri);
    }
};

$cachePrefix = 'pdp_';
$cacheTtl = new DateInterval('P1D');
$factory = new PsrStorageFactory($cache, $client, $requestFactory);
$pslStorage = $factory->createPublicSuffixListStorage($cachePrefix, $cacheTtl);
$rzdStorage = $factory->createTopLevelDomainListStorage($cachePrefix, $cacheTtl);

// if you need to force refreshing the rules 
// before calling them (to use in a refresh script)
// uncomment this part or adapt it to you script logic
// $pslStorage->delete(PsrStorageFactory::PUBLIC_SUFFIX_LIST_URI);
$publicSuffixList = $pslStorage->get(PsrStorageFactory::PUBLIC_SUFFIX_LIST_URI);

// if you need to force refreshing the rules 
// before calling them (to use in a refresh script)
// uncomment this part or adapt it to you script logic
// $rzdStorage->delete(PsrStorageFactory::TOP_LEVEL_DOMAIN_LIST_URI);
$topLevelDomains = $rzdStorage->get(PsrStorageFactory::TOP_LEVEL_DOMAIN_LIST_URI);

Be sure to adapt the following code to your own application. The following code is an example given without warranty of it working out of the box.

You should use your dependency injection container to avoid repeating this code in your application.

Automatic Updates

It is important to always have an up to date Public Suffix List and Top Level Domain List.
This library no longer provide an out of the box script to do so as implementing such a job heavily depends on your application setup. You can use the above example script as a starting point to implement such a job.

Changelog

Please see CHANGELOG for more information about what has been changed since version 5.0.0 was released.

Contributing

Contributions are welcome and will be fully credited. Please see CONTRIBUTING for details.

Testing

pdp-domain-parser has:

a PHPUnit test suite
a code analysis compliance test suite using PHPStan.
a code analysis compliance test suite using Psalm.
a coding style compliance test suite using PHP CS Fixer.

To run the tests, run the following command from the project folder.

$ composer test

Security

If you discover any security related issues, please email [email protected] instead of using the issue tracker.

Credits

License

The MIT License (MIT). Please see License File for more information.

Attribution

Portions of the Pdp\Rules class are derivative works of the PHP registered-domain-libs. I've included a copy of the Apache Software Foundation License 2.0 in this project.

Comments

Allows an optional path parameter in the cache update script

The cache class allows for you to use a custom path which is very helpful, but the installer currently doesn't support this. So now you can use an optional parameter path when calling the bin/update-psl to specify the cache location. This seems like a preferable alternative to having people copy and edit the installer file as it's an optional improvement getting the installer up to date with the cache's already existing functionality.
enhancement version 5

opened by Shardj 15
Overall modification prompted by magic functions breaking phpspec tests.
Thanks for being open-minded about this one, really appreciate it.

Mainly replaced magic FNs, and corrected some typos along the way. Tests all fixed. Recommend a major version increment since this'll be a BC if folks are using versions with typos or pseudo-public access.

Fixed typo in 'isMultiLabelDomain' (was isMutliLabelDomain).

Standardized use of Registrable vs. Registerable (registrable is proper spelling)

Removed magic getters, substituted with proper accessors

Made private members previous accessed by accessors, protected to permit inheritance

Changed all tests and docs to reflect changes
opened by Saeven 15
Websites which use valid tlds for their domain don't seem to be treated correctly

"https://nhs.uk", "https://s3.amazonaws.com", "https://platform.sh"

I wasn't able to get the parts for domains such as the above, where the whole domain is a valid tld, but so is a subpart of the domain which in reality is the tld that the domain is using.
question wontfix

opened by Shardj 14
Type missmatch
Hello, I think there is something wrong with the typing:

Issue summary

Rules implements PublicSuffixList

its has this method
public function resolve($host): ResolvedDomainName

But the interface signature is different:
public function resolve(Host $host): ResolvedDomainName;

It might not be a big deal, but when we use the PsRStorageFactory:

public function createPublicSuffixListStorage(string $cachePrefix = '', $cacheTtl = null): PublicSuffixListStorage

and PublicSuffixListStorage:

public function get(string $uri): PublicSuffixList;

So, since we don't use Rules directly but an interfaces, there are an error when calling with a string as stated in the README.

System informations

| Information | Description | |--------------|---------| | Pdp version | 6|
question version 6
opened by lyrixx 13
The host `องค๜กร` is invalid : a label or domain name contains disallowed characters.
I have this errors with symfony any suggestions pls?

` use Pdp\Manager; use Pdp\Cache;

$guzzleAdapter = new GuzzleHttpClientAdapter(new GuzzleClient()); $cache = new Cache(); $manager = new Manager($cache, $guzzleAdapter); $manager->refreshRules(); $rules = $manager->getRules(); $domain = $rules->resolve('www.github.com');

`

[Pdp\Exception] The host องค๜กร is invalid : a label or domain name contains disallowed characters.
question version 5
opened by tuxado 12

Some questions, confusions

First of all, please pardon my ignorance. I wanted to post this in Rewrite the documentation #275 as comment but thought to create a new post / issue.

Some suggestions.

1. Kindly do briefly explain in documentation

what functions / properties of objects do. E.g. Like isResolvable() and isKnown() (common between Pdp\Domain and Pdp\PublicSuffix Objects).
and what their return or values means. E.g. What is meant by isResolvable or isKnown of Pdp\Domain if their value is true.

2. If I search some domain with TLD that does not exists in PSL, the parser still parse the domain and even publicSuffix index / key is populated. Is there a way to tell the parser to throw exception or return a null Pdp\Domain if TLD is not part of PSL (both ICANN and PRIVATE) in given domain.

Code sample

$res = $rules->resolve('paki.appri');
print_r($res);
var_dump($res);

Result

// Pdp\Domain Object
// (
//     [domain] => paki.appri
//     [registrableDomain] => paki.appri
//     [subDomain] => 
//     [publicSuffix] => appri
//     [isKnown] => 
//     [isICANN] => 
//     [isPrivate] => 
// )
// /var/www/html/tests.php:10:
// object(Pdp\Domain)[104]
//   private 'domain' => string 'paki.appri' (length=10)
//   private 'labels' => 
//     array (size=2)
//       0 => string 'appri' (length=5)
//       1 => string 'paki' (length=4)
//   private 'publicSuffix' => 
//     object(Pdp\PublicSuffix)[109]
//       private 'publicSuffix' => string 'appri' (length=5)
//       private 'section' => string '' (length=0)
//       private 'labels' => 
//         array (size=1)
//           0 => string 'appri' (length=5)
//       private 'asciiIDNAOption' => int 0
//       private 'unicodeIDNAOption' => int 0
//       private 'isTransitionalDifferent' => boolean false
//   private 'registrableDomain' => string 'paki.appri' (length=10)
//   private 'subDomain' => null
//   private 'asciiIDNAOption' => int 0
//   private 'unicodeIDNAOption' => int 0
//   private 'isTransitionalDifferent' => boolean false

3. Please provide a single concise full example consisting of major parts of this parser. Like you insists upon setting some scenario to update PSL (or RZD) cache. Here is an example of my test usage.

require_once(__DIR__ . DIRECTORY_SEPARATOR . 'vendor/autoload.php');
try {
    $manager = new \Pdp\Manager(new \Pdp\Cache(), new \Pdp\CurlHttpClient(), new DateInterval('P1D')); // Using Manager to make, refresh, update cache. new DateInterval('P1D') defines that cache is updated daily day.
    $rules = $manager->getRules(); //  Rules are fetched from cache and cache files are updated if needed as per DateInterval provide in Manager object creation.
    $domain = $rules->resolve('www.abc.enfield.sch.uk'); // Resolve provided domain and returns Pdp\Domain object.
    print_r($domain);
    var_dump($domain);
    var_dump($domain->getPublicSuffix()); // Returns PublicSuffix part of provided domain as string.
    // $ps = \Pdp\PublicSuffix::createFromDomain($domain); // Make and return Pdp\PublicSuffix object in reference to Pdp\Domain object.
    $ps = $rules->getPublicSuffix($domain); // Make and return Pdp\PublicSuffix object in reference to Pdp\Domain object.
    print_r($ps);
    var_dump($ps);
} catch (Exception $e) {
    var_dump($e);
}

Environment.

Ubuntu 18.04.
LAMP with PHP 7.4
jeremykendall/php-domain-parser 5.7.0 through Composer

I hope that I am able to explain my points properly.

Also I have a question that is not related to this Parser but as you have more knowledge of PSL and TLDs. Is there a way I can check which TLD is available for public registration and which are not? E.g. A normal person can register a .com domain but not .google because .google does not allow public registration. Some kind of list like PSL defining which TLDs are available for public registration.

enhancement version 5 documentation version 6

opened by umairkhan-dev 11

Implement file locking within PublicSuffixListManager class

Addresses issue #78

To test concurrency, I inserted the following at line 249:

echo "lock obtained for '$filepath'\n\n";
sleep(5);

Then ran the following commands within 5 seconds of eachother:

$ time bin/update-psl 
Updating Public Suffix List.
lock obtained for /Users/mike/Projects/php-domain-parser/data/public-suffix-list.txt

lock obtained for /Users/mike/Projects/php-domain-parser/data/public-suffix-list.php

Update complete.

real    0m11.799s
user    0m0.388s
sys 0m0.028s

$ time bin/parse
Array
(
    [scheme] => http
    [user] => user
    [pass] => pass
    [host] => www.pref.okinawa.jp
    [subdomain] => www
    [registrableDomain] => pref.okinawa.jp
    [publicSuffix] => okinawa.jp
    [port] => 8080
    [path] => /path/to/page.html
    [query] => query=string
    [fragment] => fragment
)
Host: http://user:[email protected]:8080/path/to/page.html?query=string#fragment
'okinawa.jp' IS a valid public suffix.

real    0m4.535s
user    0m0.071s
sys 0m0.013s

The blocking appears to be effective.

PHPUnit passes, but I did not achieve complete code coverage as I am not sure how to simulate a file which opens but cannot be written, a file which opens which cannot be read, or a file which opens but cannot be locked. I also do not know how to properly test asynchronous I/O within PHPUnit, so these file locking tests were performed manually as above. If you have a solution, please let me know.

version 3

opened by mikegreiling 10

Incorrectly parses long suffixes with missing levels
Issue summary

There are a few long suffixes defined by publicsuffix.org which do not list smaller suffixes. Specifically:

users.scale.virtualcloud.com.br

ui.nabu.casa

es-1.axarnet.cloud

it1.eur.aruba.jenv-aruba.cloud

s3.cn-north-1.amazonaws.com.cn

s3.dualstack.ap-northeast-1.amazonaws.com

s3.dualstack.eu-central-1.amazonaws.com

s3.dualstack.us-east-1.amazonaws.com

git-pages.rit.edu

user.party.eus

cust.dev.thingdust.io

cloud.jelastic.open.tim.it

user.aseinet.ne.jp

forgot.her.name

cdn.prod.atlassian-dev.net

fr-1.paas.massivegrid.net

As an example, users.scale.virtualcloud.com.br but virtualcloud.com.br is not. So the suffix for it should be parsed as just com.br.

It looks like to me (per my brief look at the cached data) that the data is parsed as a simple label map where each array contains a nested array of labels at a deeper level.

If things should work the way I expect, then I think the way to approach this is temporarily hold a flat array of values from PSL, and for each suffix look for it's parent level suffix via shifting off each label; the resulting array for the above case would look something like:

[ 'com' => [ 'br' => [ 'users.scale.virtualcloud' ]]]

Although that might complicate the actual comparison logic, so it might be better to map the entire suffix rather than just the labels which would result in something like this:

[ 'com' => [ 'com.br' => [ 'users.scale.virtualcloud.com.br' ]]]

System informations

| Information | Description | |-------------------|-----------------| | Pdp version | 6.1 | | PHP version | 8.1 | | OS Platform | Alpine |

Standalone code, or other way to reproduce the problem

(Please complete the text below to help us fix the issue)

Expected result

Actual result
bug documentation version 6
opened by ash-m 9
$Return value of Pdp\Converter::idnToAscii() must be of the type string, boolean returned$

Return value of Pdp\Converter::idnToAscii() must be of the type string, boolean returned

Issue summary

does not work

System informations

(In case of a bug report Please complete the table below)

| Information | Description | |--------------|---------| | Pdp version | 5.3.0 | | PHP version | 7.2 | | OS Platform | BSD |

Standalone code, or other way to reproduce the problem

$manager = new Manager(new Cache(), new CurlHttpClient()); $rules = $manager->getRules();

Expected result

no error

Actual result

Type: TypeError Message: Return value of Pdp\Converter::idnToAscii() must be of the type string, boolean returned File: /project/vendor/jeremykendall/php-domain-parser/src/IDNAConverterTrait.php Line: 88
bug question version 5

opened by spagu 9
Add a way to validate TLDs

Add a new function Pdp\Parser::getRawPublicSuffix() that can be used to check a host’s TLD for validity. The method returns false for invalid TLDs. Move most logic from Pdp\Parser::getPublicSuffix() into this new method. Ensure existing unit tests pass. Add unit tests for validating the one new logic rule introduced by the new method.
enhancement

opened by SmellyFish 9
cannot get info for sa.gov.au
Issue summary

We use pdp to determine whether a given url uses a valid domain name.

As per issue #251 the library does not allow to determine whether or not sa.gov.au is a valid hostname, when it is actually. Please visit it at https://sa.gov.au/

Standalone code, or other way to reproduce the problem

$manager = new Manager(new Cache(), new CurlHttpClient()); var_dump($manager->getRules()->resolve('sa.gov.au'));

Expected result

anything that helps to tell that sa.gov.au is a valid hostname

Actual result

object(Pdp\Domain)#66 (7) { ["domain"]=> string(9) "sa.gov.au" ["registrableDomain"]=> NULL ["subDomain"]=> NULL ["publicSuffix"]=> NULL ["isKnown"]=> bool(false) ["isICANN"]=> bool(false) ["isPrivate"]=> bool(false) }

Happy to give contributions if required.

Thanks
question wontfix version 5 version 6
opened by gsouf 8

Releases(6.2.0)

6.2.0(Nov 5, 2022)
Added

None

Fixed

Internal code to make services readonly

Internal code to make VO properties readonly

Internal code improved typehinting

Deprecated

None

Removed

PHP7 and PHP8.0 support

Source code(tar.gz)
Source code(zip)
6.1.2(Sep 29, 2022)
Added

None

Fixed

#321 remove regression when resolving private domain suffix.

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
6.1.1(Feb 18, 2022)
Added

None

Fixed

#321 improve resolving private domain suffix. Rules::getPrivateDomain is more restrictive It will throw if the domain name does not contain a valid "private" TLD.

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
6.1.0(Jun 19, 2021)
Added

TimeToLive::until

TimeToLive::fromDurationString

Fixed

.gitattributes files to be filter out.

TimeToLive marked as internal

Host::toUnicode method MUST never throw exceptions on conversion according to RFC3490.

UnableToResolveDomain typo in the exception message

Deprecated

TimeToLive::fromDateTimeInterface use TimeToLive::fromNow

TimeToLive::fromScalar use TimeToLive::convert

Removed

None

Source code(tar.gz)
Source code(zip)
6.0.0(Dec 13, 2020)
Added

Adding proper Interfaces

Added Domain::clear to easily initialize an empty domain object

Added Domain::slice to easily slice a domain object

Added ResolvedDomain object to normalize Resolver results.

Added Suffix object to replace the PublicSuffix object from v5.

Public Suffix List and IANA Top Level Domain List are fully decoupled

Added Idna, IDN support has been completely revamped

Added internal Stream class, to improve path/stream resolution

Resolver uses by default UTS#46 IDNA2008 algorithm to convert domain names

Storage capability is now optional and can be based on PHP-FIG related interfaces to improve interoperability

Pdp\TopLevelDomains::getIANADomain which throws on syntax error and if no effective TLD is found (behave like Pdp\TopLevelDomains::resolve in v5).

Fixed

The Pdp\Domain class not longer directly exposes Effective TLD status.

Effective TLD resolver (Pdp\Rules::resolve and Pdp\TopLevelDomains::resolve) no longer accept IDNA options.

Rules::getICANNDomain, Rules::getPrivateDomain will throw even if a PublicSuffix is found but does not belong to the correct PSL section.

Pdp\TopLevelDomains::resolve acts like Pdp\Rules::resolve and only throw on TypeError

Deprecated

None

Removed

__toString and __debugInfo usage

Support for PHP7.4-

Composer script for automatic updates of the remote databases

CLI command bin/update-psl

Pdp\Cache, Pdp\CacheException: The package PSR-16 Cache implementation using the underlying filesystem.

Pdp\HttpClient, Pdp\CurlHttpClient and Pdp\HttpClientException: The package Http Client.

Pdp\Installer, Pdp\Manager: The package source manager and installer

Pdp\Logger, The package logger implementation

Pdp\Rules::ALL_DOMAINS constant deprecated in version 5.3

Pdp\Domain::getDomain deprecated in version 5.3

Pdp\Domain::resolve

Pdp\Domain::getPublicSuffix replaced by Pdp\ResolvedDomain::suffix

Pdp\Domain::getRegistrableDomain replaced by Pdp\ResolvedDomain::registrableDomain

Pdp\Domain::getSubDomain replaced by Pdp\ResolvedDomain::subDomain

Pdp\Domain::withPublicSuffix replaced by Pdp\ResolvedDomain::withSuffix

Pdp\Domain::getLabel replaced by Pdp\Domain::label

Pdp\Domain::isTransitionalDifferent replaced by Pdp\IdnaInfo::isTransitionalDifferent

Pdp\PublicSuffix replaced by Pdp\Suffix

Accessing suffix information from the Pdp\Domain object is no longer possible you need to do it from Pdp\Suffix

Pdp\TopLevelDomains::contains without replacement

Internal Converter classes (implementation details are no longer exposed).

Source code(tar.gz)
Source code(zip)
5.7.2(Oct 26, 2020)
Added

None

Fixed

Added support for PHP8 see #289 based on works by @szepeviktor

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
5.7.1(Aug 24, 2020)
Added

None

Fixed

Cache throws when cache directory doesn't exist #280

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
5.7.0(Aug 3, 2020)
Added

Rules::getCookieDomain

Rules::getICANNDomain

Rules::getPrivateDomain

CouldNotResolvePublicSuffix::dueToUnresolvableDomain

Fixed

Improve type hinting and return type by dropping EOL PHP versions support.

Improve development environment by dropping EOL PHP versions support.

Fix composer script

Deprecated

None

Removed

Support for PHP7.0 and PHP7.1

The external data from IANA and mozilla is no longer part of the package and will be downloaded only on demand on composer update/install.

Source code(tar.gz)
Source code(zip)
5.6.0(Dec 29, 2019)
Added

A simple Psr3 compatible logger class which output the logs to you cli console.

Fixed

composer.json updated to be composer 2.0 ready

package bundle installer is rewritten to improve its usage see #249 and #250

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
5.5.0(Apr 14, 2019)
Added

Support for IDNA options see #236 thanks to Insolita.

PublicSuffix::labels and Domain::labels to return the VO labels see #241

IDNAConverterTrait::parse (internal)

Fixed

Don't swallow cache errors #232

Update travis settings to allow testing against future version of PHP.

Deprecated

IDNAConverterTrait::setLabels replaced by IDNAConverterTrait::parse (internal)

Removed

None

Source code(tar.gz)
Source code(zip)
5.4.0(Nov 22, 2018)
Added

Pdp\TopLevelDomains to allow resolving domain againts IANA Root zone database

Pdp\TLDConverter converts the IANA Root Zones database into an associative array

Pdp\Manager::getTLDs a service to return a cache version of the IANA Root zone database

Pdp\Manager::refreshTLDs a service to refresh the cache version of the IANA Root zone database

added a new $ttl parameter to improve PSR-16 supports to

Pdp\Manager::__construct

Pdp\Manager::getRules

Pdp\Manager::refreshRules

Pdp\Exception\CouldNotLoadTLDs exception

Fixed

Pdp\IDNAConverterTrait::setLabels improve IDN domain handling

Pdp\IDNAConverterTrait throws a UnexpectedValueException if the Intl extension is misconfigured see #230

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
5.3.0(May 22, 2018)
Added

Pdp\PublicSuffixListSection interface implemented by Pdp\Rules and Pdp\PublicSuffix

Pdp\DomainInterface interface implemented by Pdp\Domain and Pdp\PublicSuffix

Pdp\Domain::getContent replaces Pdp\Domain::getDomain

Pdp\Domain::withLabel adds a new label to the Pdp\Domain.

Pdp\Domain::withoutLabel removes labels from the Pdp\Domain.

Pdp\Domain::withPublicSuffix updates the Pdp\Domain public suffix part.

Pdp\Domain::withSubDomain updates the Pdp\Domain sub domain part.

Pdp\Domain::append appends a label to Pdp\Domain.

Pdp\Domain::prepend prepends a label to Pdp\Domain.

Pdp\Domain::resolve attach a public suffix to the Pdp\Domain.

Pdp\Domain::isResolvable tells whether the current Pdp\Domain can have a public suffix attached to it or not.

Pdp\PublicSuffix::createFromDomain returns a new Pdp\PublicSuffix object from a Pdp\Domainobject

Pdp\Exception sub namespace to organize exception. All exception extends the Pdp\Exception class to prevent BC break.

Fixed

Pdp\Domain domain part computation (public suffix, registrable domain and sub domain)

Pdp\Domain and Pdp\PublicSuffix host validation compliance to RFC improved

Improve Pdp\Converter and Pdp\Manager class to better report error on IDN conversion.

Improve Pdp\Installer vendor directory resolution see PR #222

Pdp\Exception nows extends InvalidArgumentException instead of RuntimeException

Deprecated

Pdp\Domain::getDomain use instead Pdp\Domain::getContent

Pdp\Rules::ALL_DOMAINS use the empty string instead

Removed

None

Source code(tar.gz)
Source code(zip)
5.2.0(Feb 23, 2018)
Added

Pdp\Rules::getPublicSuffix returns a Pdp\PublicSuffix value object

Pdp\Rules::__set_state is implemented

Pdp\Domain::toUnicode returns a Pdp\Domain with its value converted to its Unicode form

Pdp\Domain::toAscii returns a Pdp\Domain with its value converted to its AScii form

Pdp\PublicSuffix::toUnicode returns a Pdp\PublicSuffix with its value converted to its Unicode form

Pdp\PublicSuffix::toAscii returns a Pdp\PublicSuffix with its value converted to its AScii form

Fixed

Pdp\Domain::getDomain returns the normalized form of the domain name

Pdp\PublicSuffix is no longer internal.

Normalizes IDN conversion using a internal IDNConverterTrait

Internal code improved by requiring PHPStan for development

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
5.1.0(Dec 18, 2017)
Added

Pdp\Rules::createFromPath named constructor to returns a new instance from a path

Pdp\Rules::createFromString named constructor to returns a new instance from a string

Fixed

Pdp\CurlHttpClient default curl options improved

Deprecated

None

Removed

None

Source code(tar.gz)
Source code(zip)
5.0.0(Dec 13, 2017)
Added

Pdp\Exception a base exception for the library

Pdp\Rules a class to resolve domain name against the public suffix list

Pdp\Domain an immutable value object to represents a parsed domain name

Pdp\Installer a class to enable improve PSL maintenance

Pdp\Cache a PSR-16 file cache implementation to cache a local copy of the PSL

Pdp\Manager a class to enable managing PSL sources and Rules objects creation

Pdp\Converter a class to convert the PSL into a PHP array

Fixed

invalid domain names improved supported

idn_* conversion error better handled

domain name with RFC3986 encoded string improved supported

Deprecated

None

Removed

PHP5 support

URL Parsing capabilities and domain name validation

Pdp\PublicSuffixList class replaced by the Pdp\Rules class

Pdp\PublicSuffixManager class replaced by the Pdp\Manager class

Pdp\HttpAdapter\HttpAdapterInterface interface replaced by the Pdp\HttpClient interface

Pdp\HttpAdapter\CurlHttpAdapter class replaced by the Pdp\CurlHttpClient class

Source code(tar.gz)
Source code(zip)
4.0.3-alpha(Sep 28, 2017)

Source code(tar.gz)
Source code(zip)
4.0.2-alpha(Sep 28, 2017)

Source code(tar.gz)
Source code(zip)
4.0.1-alpha(Sep 27, 2017)

Release to include all commits to master before new-new version (BC breaking v5). Totally pooched semantic versioning so v4 will be kind of an interim version. Apologies ☹️
Source code(tar.gz)
Source code(zip)
4.0.0-alpha(Feb 12, 2016)
Many Improvements, Many BC Breaks

The Most Significant BC Breaks

All misspellings of registrable have been corrected (registerable -> registrable) (see #61)

All magic methods have been replaced with getters.

Other Updates and BC Breaks

#63: Ensure scheme returns null rather than an empty string

#64 and #65: Cleans up Exception message

Source code(tar.gz)
Source code(zip)
3.0.0(Mar 30, 2015)
New Features and BC Breaks

IMPORTANT: These are all BC breaking changes (some more significant than others).

Adds RFC 3986 scheme support

http(s) and ftp(s) are no longer the only supported schemes

All RFC 3986 valid schemes are parsed

All schemes are lowercased on output regardless of the case passed to the parser

The http scheme is no longer prepended to URLs which do not include schemes

Renames Public Suffix List update script from ./bin/pdp-psl. to ./bin/update-psl

Adds a post-install script to composer.json to update the Public Suffix list when calling composer install

Backports PHP 5.4.7 parse_url() fix to pdp_parse_url() and supported PHP 5.3 versions: "Fixed host recognition when scheme is omitted and a leading component separator is present."

Source code(tar.gz)
Source code(zip)
2.0.3(Feb 16, 2015)
Fixes #43

Source code(tar.gz)
Source code(zip)
2.0.2(Feb 12, 2015)
Fixes bug in bin/parse

Updates Public Suffix List

Source code(tar.gz)
Source code(zip)
2.0.1(Jan 27, 2015)
Corrects mistake in documentation

Updates bin/parse to include suffix validity

Source code(tar.gz)
Source code(zip)
2.0.0(Jan 2, 2015)
Features

Adds IDNA support

Removes Parser::mbParseUrl, a UTF-8 compatible version of PHP's parse_url

Adds global function pdp_parse_url, a UTF-8 compatible version of PHP's parse_url

Adds Parser:: isSuffixValid to test validity of domain suffix (thanks to @SmellyFish)

See https://github.com/jeremykendall/php-domain-parser/compare/1.4.6...2.0.0 for detailed version diff

BC Breaks

Library now requires ext-intl and ext-mbstring.
Source code(tar.gz)
Source code(zip)
1.4.6(Jul 28, 2014)

Adds IPv6 support, courtesy of a bugfix submitted by @geekwright.
Source code(tar.gz)
Source code(zip)
1.4.5(Jul 22, 2014)

Documentation updates (removes TODO)
Source code(tar.gz)
Source code(zip)
1.4.4(Jul 21, 2014)

Adds "ext-mbstring" to composer require key
Source code(tar.gz)
Source code(zip)
1.4.3(Jul 21, 2014)

Fixes #32.
Source code(tar.gz)
Source code(zip)
1.4.2(Jul 21, 2014)
Deprecates Parser::mbParseUrl, a UTF-8 compatible version of PHP's parse_url

Adds global function mb_parse_url, a UTF-8 compatible version of PHP's parse_url

Source code(tar.gz)
Source code(zip)
1.4.1(Jul 21, 2014)

Adds IDNA documentation to README
Source code(tar.gz)
Source code(zip)