Throws error without symbiote/silverstripe-queuedjobs module installed.

Fatal error: Class 'Symbiote\QueuedJobs\Services\AbstractQueuedJob' not found

Is this module supposed to be required?

VersionedCollector->getTableNameForClass() uses DataObjectSchema->baseDataTable the base table not DataObjectSchema->tableName.

So it ends up adding duplicate values to the class list in VersionedCollector->getTablesListForClass() when it's creating the VersionedCollector->deleteVersionsQuery() which in turn gets passed on to the SQLExpressionProcessor which causes an error.

The best example of this is if you have a record with the class of SilverStripe\Assets\Image it will add two SilverStripe\Assets\File tables.

When running GarbageCollectorService::inst()->process();, it collects all collectors and processors as defined by the config.

In https://github.com/brettt89/silverstripe-garbage-collector/blob/master/src/GarbageCollectorService.php#L92 it tries to create singletons of each processor to get their implementor class, but since they define a mandatory constructor param, this fails, e.g. SQLExpressionProcessor (https://github.com/brettt89/silverstripe-garbage-collector/blob/master/src/Processors/SQLExpressionProcessor.php#L19) requires an SQLExpression to be provided and similar for others.

There's a couple of options how this can be handled:

1. set the implementor class as a config variable (can't be enforced via interface)
2. change the method to be static so that it doesn't require an object instance
3. use reflection to create an instance without running the constructor method

I can submit a PR if one of the options above is chosen (or a different one is suggested).

As raised, the recent minor enhancement changed the way how the list of tables was collected and caused duplication of base tables instead of using the class inheritance-based approach as it was before.

This addresses the issue and adds a test.

Fix for #16

New collector and processor for removal of _obsolete_ tables from the database.

• Uses a new Model type RawSQL as identification does not work for non-classes objects (e.g. Arrays / Strings / etc).

To Do:

• [x] Write Tests.
• [x] Documentation

This issue was raised to track the Production Ready issues that may be blocking users from using this module within Production for their Applications.

Tasks

• [x] Remove "Warning" from README.md
• [x] Test on actual applications thoroughly
• [x] Determine minimum requirements and define
• [x] Update composer.json to have version control that allows for earliest possibly dependencies
• [x] Performance Testing

BUG: Collector job hydration fix

The collector job needs to have a constructor which is able to run without any params. This is because once job is stored in DB as a job descriptor it will be picked up by the queue runner which will create a new instance of the job by calling the constructor without any params. Any job data assignments in such constructor call are irrelevant as the job data is populated right after that. The main thing is that the code needs to be executable.

Changes

• constructor is executable without any params
• docblock update

Related issues

https://github.com/brettt89/silverstripe-garbage-collector/issues/28

Silverstripe Framework only added modify method to DateTime class in Silverstripe 4.5. These changes support earlier versions of Silverstripe 4 by using strtotime for modification and manually setting the value.

Fixes: #21

Hi

Composer requirements are "silverstripe/framework": "^4.2" But it actually doesn't work on 4.4 do to this error:

[Emergency] Uncaught BadMethodCallException: Object->__call(): the method 'modify' does not exist on 'SilverStripe\ORM\FieldType\DBDatetime' Trace SilverStripe\View\ViewableData->__call(modify, Array) VersionedCollector.php:166 SilverStripe\GarbageCollector\Collectors\VersionedCollector->getRecordsForDeletion(Array) VersionedCollector.php:98 SilverStripe\GarbageCollector\Collectors\VersionedCollector->getCollections() GarbageCollectorServiceTask.php:36 GarbageCollectorServiceTask->processCollector(SilverStripe\GarbageCollector\Collectors\VersionedCollector) GarbageCollectorServiceTask.php:26 GarbageCollectorServiceTask->process() GarbageCollectorServiceTask.php:15 GarbageCollectorServiceTask->run(SilverStripe\Control\HTTPRequest) TaskRunner.php:104

Modify was added to DBDateTime in 4.5 https://github.com/silverstripe/silverstripe-framework/commit/844d2ef134a344bca0e39ecc4c5d8bdd75de4171

Thanks

VersionedCollector stops collecting records when one of classes has no versions to delete causing it to not continue looping through all the classes.

https://github.com/brettt89/silverstripe-garbage-collector/blob/d9a539e92ea8428b87b157f22b032d2e5beaf25f/src/Collectors/VersionedCollector.php#L102

FluentVersionedCollectorExtension added for collecting data that is localised using Fluent.

Reference: https://github.com/tractorcow-farm/silverstripe-fluent/

Collector job hydration error

The collector job needs to have a constructor which is able to run without any params. This is because once job is stored in DB as a job descriptor it will be picked up by the queue runner which will create a new instance of the job by calling the constructor without any params. Any job data assignments in such constructor call are irrelevant as the job data is populated right after that. The main thing is that the core needs to be executable.

Fixed in https://github.com/brettt89/silverstripe-garbage-collector/pull/27

After creating a task that calls GarbageCollectorService::inst()->process() in it's run() method, the following SQL error occurs both for mysqld Ver 10.5.9-MariaDB for osx10.16 on x86_64 (Homebrew) as well as on Silverstripe Cloud. I can't recall the MySQL version running on our stack but will work to track that down.

Config

---
Name: website-GarbageCollectors
---
SilverStripe\GarbageCollector\Collectors\VersionedCollector:
  # Increase Versioned keep limit to 10 records
  keep_limit: 10
  # Define base classes to collect versions for
  base_classes:
    - SilverStripe\CMS\Model\SiteTree

SilverStripe\GarbageCollector\Collectors\ChangeSetCollector:
  # Reduce Changeset Lifetime to 10 days
  deletion_lifetime: 10

# Register collectors with service
SilverStripe\GarbageCollector\GarbageCollectorService:
  collectors:
    - 'SilverStripe\GarbageCollector\Collectors\VersionedCollector'
    - 'SilverStripe\GarbageCollector\Collectors\ChangeSetCollector'

Task

class GarbageCollectionTask extends BuildTask
{
    /**
     * @var string
     */
    private static $segment = 'GarbageCollectionTask';

    /**
     * {@inheritDoc}
     * @return string
     */
    public function getDescription()
    {
        return _t(
            __CLASS__ . '.Description',
            'A task used to trim Change Sets, Versions, and any other configured objects'
        );
    }

    /**
     * @param HTTPRequest $request
     * @throws \Exception
     */
    public function run($request)
    {
        GarbageCollectorService::inst()->process();
    }
}

MariaDB

2021-10-05 09:53:37][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) 42000-1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID"...' at line 2" [] []
[2021-10-05 09:53:37][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) 42000-1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID"...' at line 2" [] []
[2021-10-05 09:53:37][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) 42000-1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID"...' at line 2" [] []
[2021-10-05 09:53:37][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) 42000-1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID"...' at line 2" [] []
[2021-10-05 09:53:37][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) 42000-1064: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID"...' at line 2" [] []

MySQL

[2021-10-05 09:57:46][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LE' at line 1" [] []
[2021-10-05 09:57:46][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LE' at line 1" [] []
[2021-10-05 09:57:46][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LE' at line 1" [] []
[2021-10-05 09:57:46][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LE' at line 1" [] []
[2021-10-05 09:57:46][ERROR] Unable to process records: "Couldn't run query: DELETE FROM "ChangeSet" LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LEFT JOIN "ChangeSetItem_ReferencedBy" ON "ChangeSetItem"."ID" = "ChangeSetItem_ReferencedBy"."ChangeSetItemID" WHERE ("ChangeSet"."ID" IN (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)) You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'LEFT JOIN "ChangeSetItem" ON "ChangeSet"."ID" = "ChangeSetItem"."ChangeSetID" LE' at line 1" [] []

What is the reasoning behind this? I see that you mentioned isPublishedInLocale() as a reason but I don't see why.

I have SiteTree objects that have hundreds of different versions but as some of the versions were published they haven't been deleted even if they are hundreds of versions behind.

This is the line of code with the comment I mentioned above.

https://github.com/brettt89/silverstripe-garbage-collector/blob/d9a539e92ea8428b87b157f22b032d2e5beaf25f/src/Collectors/VersionedCollector.php#L282-L284

As per the title, currently, the configuration is global for all base classes defined for VersionedCollector.

An enhancement to that can be one where base_classes config would also accept nested array indexed by the class with config overrides for that class.