This is the plan for how module registration for phpList 4 should work. Comments and feedback welcome.

Principles

• All phpList modules are Composer packages that are recognized by the package type phplist-module defined in the composer.json.
• A phpList module package must only contain code, assets and configuration related to this package. The module-related files must also not be "piggy-backing" in some other package. Packages still can depend on other packages and make use of their code, though (by adding them to the require section of the composer.json).
• From the view of the administrator/user, a module is installed by requiring it in the root package composer.json and then running a composer update. It will not be possible for a package to be in an "installed, but not activated" state.
• It must be possible to test module packages on their own, i.e., without the base-distribution package, both with automated tests as well as manual tests. (The module package usually will depend on at least the phplist4-core package, though.)
• Modules that provide the same capabilities (e.g., the same routes) can be configured via their composer.json to conflict with each (or more specifically: with other packages that provide the same capabilities) (TODO: document how to do this)

Technical approach

• For the installed application, each bundle can bring its own configuration that will be auto-included once the bundle is included.
• There will be an application-wide configuration file that lists all module-provided bundles (i.e., their class names).
• Each module package must provide the class names of its provided bundles in the extra section of its composer.json.
• The phplist4-core package will provide a composer script that lists all installed modules (i.e., all installed composer packages that have the corresponding type). (The core might need to read vendor/composer/installed.json for this. I'll need to check this.) This script will be there mainly for convenience, and also to keep the PRs relatively small. This script will also be accessible via a bin/console command.
• The phplist4-core package will provide a composer script that runs through all installed module packages, reads their composer.json file, and generates the root application bundle classes file. This script will listen to the update and install events of all phplist 4 packages (including the core). It will also listen to the package creation event of the base-distribution package. (NB: Scripts will always only be executed for the current root package, not for any dependencies.)
• The "empty start page" will be moved to a separate module package (for which we still need to find a good name). This package will be included by default in the base-distribution package (as long as the web-frontend package does not provide any usable content). This package will conflict with the web-frontend package as both provide the / route (i.e., the root route). This package will also be used by the automated tests for module registration in the phplist4-core package.

(Planned) Pull requests

• [x] #153 Add a function for provided the path to the application root.
• [x] #155, #156 Add a composer script that lists all installed module packages to the phplist4-core package. This will include a PHP class for this that can also be used for other things.
• [x] #157 Add the not-yet-autogenerated bundles classes configuration file to the phplist4-core package, move the REST API configuration and the "empty start page" configuration there, and add it to Git. (At this stage, the phplist4-core still has hard-coded references to the rest-api package.) (This config file will be removed once auto-generation of the config file is added.)
• [x] #158 Add the script that auto-generates the root configuration file to the phplist4-core package and delete the old manual configuration file. Add composer listeners for update and install to the phplist4-core package and add the file to the .gitignore.
• [x] Change the "empty start page" bundle to auto-include its configuration.
• [x] For all module packages and the base-distribution package, add composer listeners for update and install to the phplist4-core package and add the file to the .gitignore.
• [x] #168 Add a script that auto-generates a routes configuration file.
• [x] https://github.com/phpList/rest-api/pull/40 Add the configuration file to the rest-api package.
• [x] #183 Clean up the code for the empty start page in the phplist4-core package.
• [x] #179, #205, #207 Use dependency injection.
• [x] #203 Allow modules to provide their own (non-service) configuration
• [x] Use the FOSRestBundle for our REST module.

We have to define a coding style. This will provide a clean and uniform workflow. Code review will be easier.

http://www.php-fig.org/psr/psr-1/ http://www.php-fig.org/psr/psr-2/

Useful links but we can define our custom coding style.

This is a domain model in the domain-driven-design sense (i.e., mostly a UML class diagram). It'd not an ER diagram. The model names usually are similar to the corresponding DB table names, but are technically independent.

The graphical domain model is best viewed by using the generated SVG and then using GitHub's "View" button.

@samtuke @michield I'd appreciate feedback on these questions:

• Do the domain contexts make sense? Are the context names perfect? Are the entities in the correct context? Should we have any more contexts? Which ones?
• Are the associations correct?
• Are the directions of the associations correct?
• Are the descriptions of the domain models (in the markdown file) correct? Is anything missing, or could anything be improved?
• Comments on my "what is this, and what does it do" questions. :-)
• For which m:n associations should we have dedicated named association models? (We probably need dedicated association models for those models that have properties.)
• Which * must actually be 1..n?
• Which entities are not in use anymore and can be ignored (and should be removed for phpList 3)?

I know this is quite a huge deal for Michiel and maybe for others, but I think it will be confusing when not changing the database at all. Now that messages have become campaigns and users have become subscribers, it would make a lot of sense to change the naming in the database as well. It will be confusing to put subscribers in the users database and do queries referencing user_ids etc.

I would even want to suggest another change, not to use mailing-lists anymore, but switch over to a more general filter system (so mailing lists can be compared to user attributes). We can have categories of filters like:

• Mailing lists
  • News
  • Development
• Favorite color
  • Red
  • Green And use regular attributes as filters as well
• Name
• State
• Country
• ...

The GUI could easily split this up and still pretend like we use lists, but it might be easier to build a general filter for which subscribers need to receive a given campaign.

PHP 5.4 and PHP 5.5 have reached their end of life and thus are not supported anymore (not even with security updates). So PHPList will not support it anymore, either.

Now Travis import the database schema before each build.

The database schema has been taken from the development demo database (but without the data).

The existing version 3 of phplist has a somehow worrying issue. It "phones home" via several methods that can't be disabled via a config option.

The one that worries me most is the rss feed on the dashboard, which is a security risk. In the past, we've seen some serious exploits in the unserialize() function of PHP. Unfortunately, the rss on the phplist dashboard uses exactly that, to load remote memory arrays directly to the PHP interpreter.

This can be exploited by someone first getting access to the remote phplist server, or by using a man-in-the-middle attack (like DNS injection) to send a malicious string to thousands of phplist installations.

Since version 4 is still in development, I propose a few quick solutions to avoid any potential security issues in the future:

• Add a config parameter to disable the rss calls on the dashboard.
• Stop using unserialize(), instead use a proper XML or JSON reply from the phplist server and add some checks and limits on the strings contained within that reply.
• Verify the remote reply via secure keys or similar method, to make sure the reply is being generated by the phplist server.

Please let me know what you think about the issue.

Thank you.

• [ ] Admin Class (which is not User Class).
• [ ] Subscriber Class
• [ ] List/Campaign Class
• [ ] ...

Up to us to decide on which components we will keep on core.

Namespace Refactoring

Currently we include version in our namespace:

namespace PhpList\PhpList4\Core

I think we will have trouble when relasing a next major version for example 5.0.0. Contributor will get confused whether the same class can be used or not in 5.0.0 version. Please take a look at symfony. Symfony have a different version from 2.0 to 4.0. Some components act differently in each version. To make maintenance easier we don't see any version included on symfony namespace.

We can use git branch feature to maintenance spesific version of phplist, so we don't have to stick a namespace into a spesific namespace. For example:

• branch 3.0: for phplist version 3.0 (yes, we can even use same git repository to maintenance 3.0 version)
• branch 4.0: for phplist version 4.0

Thanks to psr4 autoloading. This namespace will create an unnecessary sub directory in psr0 autoload:

// path/to/phplist4-core
src/
----- PhpList
----------- PhpList4
------------------ Core

I suggest we just use a more simple also a more common practice used by an opensource project:

namespace PHPList\Core;

I know this refactoring can produce a lot of error. With find and replace (php editor feature), and phpunit tests to detect any error; I think we can refactor namespace easily.

Testing Configuration

1. phpunit.xml.dist File

By default phpunit will search phpunit.xml in the current working directory (path/to/phplist4-core), if this file is not exists then phpunit try to find default configuration file which is phpunit.xml.dist. I think it's more simple to just use phpunit without having to pass --configuration in command:

# path/to/phplist4-core

./vendor/bin/phpunit
# command above is more simple then:
./vendor/bin/phpunit  -c Configuration/PHPUnit/phpunit.xml

So I suggest we need to put our configuration file in path/to/phplist4-core/phpunit.xml.dist, and git ignore the path/to/phplist4-core/phpunit.xml file because it should be a custom configuration for every developer.

2. Testing Configuration

We should separate testing configuration from project configuration. so we need to move all tests configuration from Configuration directory to tests/config.

Directory Naming Conventions

I suggest that we use lowercase for non namespace directory. So when we see Configuration directory we will know that it will contain a class, while config dir only contain a configuration file. Here's my sugestion:

• tests directory: Since there are no namespace called Integration, Support, and Unit, we just use lowercase name for this directories
• Configuration directory: we just use simply use config because it is a common name used by symfony based project to store configuration files.
• Database directory: we need to rename this directory to data.

Here's what I hope phplist4-core project directory like:

path/to/phplist4-core
├── config
│   ├── config.yml
│   └── # other config files...
├── data
│   ├── phplist-db.mwb
│   └── Schema.sql
├── src
│   ├── Core
│   ├── Routing
│   └── System
└── tests
    ├── config
    │   ├── code-sniffer-ruleset.xml
    │   ├── phpmd-ruleset.xml
    │   └── phpunit-travis.xml
    ├── integration
    │   ├── Composer
    │   └── Core
    ├── system
    └── unit
        ├── Core
        └── Domain

Some References

Here's some sample of good symfony based project that we can implement in phplist:

• Sylius
• OroInc
• API Platform Core

The unit tests still do not pass completely as they cannot connect to any DB on Travis yet. This will be part of a separate PR.

(And unit tests should not connect to a real database.)

Hi!

As soon as you are already using some Symfony2 components, why not using the Symfony2 Fullstack to bootstrap application development? I think this will make code more consistent and incapsulated i.e. with using container instead of global constants and static classes and ini config. Also it's provides essential libraries, like Doctrine2 ORM and Swiftmailer which allows you to focus on business logic and fits better to OO style of this version

Update symfony and affected packages.

phpList/core currently depends on symfony 3.4.37 which is several versions behind the latest version and is not supported.

For this to be done, phpunit/dbunit first has to be removed as mentioned in #324 as it requires various packages that have been replaced by newer versions of symfony/symfony see;

  Problem 1
    - phpunit/dbunit[3.0.2, ..., 3.0.3] require symfony/yaml ^3.0 || ^4.0 -> satisfiable by symfony/yaml[v3.0.0, ..., v3.4.47, v4.0.0, ..., v4.4.20].
    - phpunit/dbunit[3.0.0, ..., 3.0.1] require symfony/yaml ^3.0 -> satisfiable by symfony/yaml[v3.0.0, ..., v3.4.47].
    - Only one of these can be installed: symfony/yaml[v3.0.0, ..., v3.4.47, v4.0.0, ..., v4.4.20], symfony/symfony[v5.2.5]. symfony/symfony replaces symfony/yaml and thus cannot coexist with it.
    - Root composer.json requires symfony/symfony ^5.2.5 -> satisfiable by symfony/symfony[v5.2.5].
    - Root composer.json requires phpunit/dbunit ^3.0.0 -> satisfiable by phpunit/dbunit[3.0.0, 3.0.1, 3.0.2, 3.0.3].

DBUnit abandoned

PHPUnit/DBUnit has been abandoned, as a result no longer maintained, it has apparently become less useful over time as explained by its maintainers in the new maintainer request "New Maintainer #217".

Test cases have to now be rewritten without using the dbunit

Description

Currently dependencies cannot resolve non-conflicting installable versions on php8.0. Failing build commands are marked with continue-on-error: ${{matrix.php-versions == '8.0' }} # [temp-php8] to ignore php8.0 errors in the .github/workflows/ci.yml. It's the required to also ignore build errors on php8.0 for phpList/rest-api which depends on phpList/core.

Once we are fully able to support php8.0 on core, build error for php8.0 ignored in phpList/rest-api may no longer be ignored.

To reduce download size, we should have a .gitattributes file that makes sure that development-only files (tests, CI stuff and non-production configuration) will not end up in the Composer packages and the ZIPs generated by GitHub.

Like this: https://github.com/MyIntervals/emogrifier/commit/80266bb671b4aab6c8d496230546b902ec0d3e1a#diff-fc723d30b02a4cca7a534518111c1a66

…adding changing invertedBy to mappedBy in Subscriber

Summary

Doctrine bidirectional many to many relationship requires an invertedBy mapping and a mappedBy one.

Fixes validation error as per issue #311

Short problem description

Entities define a schema that is invalid according to doctrine schema validation tool. As a result the schema cannot be created

Steps to reproduce

1. bin/console doctrine:schema:validate
2. shows validation errors

[Mapping]  FAIL - The entity-class 'PhpList\Core\Domain\Model\Messaging\SubscriberList' mapping is invalid:
* The field PhpList\Core\Domain\Model\Messaging\SubscriberList#subscribers is on the owning side of a bi-directional relationship, but the specified mappedBy association on the target-entity PhpList\Core\Domain\Model\Subscription\Subscriber# does not contain the required 'inversedBy' attribute.

[Mapping]  FAIL - The entity-class 'PhpList\Core\Domain\Model\Subscription\Subscriber' mapping is invalid:
* The field PhpList\Core\Domain\Model\Subscription\Subscriber#subscribedLists is on the owning side of a bi-directional relationship, but the specified mappedBy association on the target-entity PhpList\Core\Domain\Model\Messaging\SubscriberList# does not contain the required 'inversedBy' attribute.


In SchemaException.php line 109:
                                                                  
  The table with name 'phplist.phplist_listuser' already exists.  

3. As a result also the schema cannot be generated

Expected behaviour

no errors when running bin/console doctrine:schema:create and a consistent db structure as a result

Actual behaviour

As per description in steps

System configuration

docker, php7.2, mysql 5.7 Extra info at https://github.com/bizmate/phplist-benchmark

Package version

dev-master tried both hashes 9f890f2 and d4dc77b

PHP and Composer version

php - php7.2 ( https://github.com/bizmate/phplist-benchmark/blob/master/docker/php/Dockerfile#L1 ) Composer version 1.9.3 2020-02-04 12:58:49