Possible Bug Report

After upgrading two pimcore installations on two different servers, I got errors in the Pimcore backend. I updated from 4.1.3 to 4.2.0 (stable branch) without errors.

The frontend keeps working fine but the backend will show the loading image for one second and after that a completely white page. There are no entries in debug or php error log. Firebug shows this:

I thought it'd be a damaged file somewhere and replaced all files with new ones from the pimcore release without success. I guess the update itself has been tested so it might be the configuration of our servers - but the two servers are configured slightly different and both - independent - pimcore installations had the same issue so this might be a bug.

File permissions are correct, database is okay as far as I can see. Frontend is working.

Based on complexity and data-amount of an Object Class, memory increases heavily on Importing/updating thousands of items. I have an import with around 20k Products, with a lot of informations like classifications, images, relations, etc. Around every 10th product, memory increases about 2MB. So, 20k products means a memory consumption of around:

100mb base usage (large import file needs to be kept in memory + symfony container) + every 10th item consumes 2 mb = more than 4gb of ram usage.

To make this easier test-able, I created a new Pimcore installation (latest stable release, 5.2.3). Created a very simple class:

And ran imports with different amounts (used PHP7.1):

Amount | Time | Start Memory Usage | End Memory Usage | Blackfire Report -------- | -----| ---------------------| --------------------| ----------------- 1000 | ~2min | 16mib | 94 mib | https://blackfire.io/profiles/2b2cbe9a-9eba-4de3-ac60-b1515f5db389/graph 10000 | ~21min | 16mib | 734 mib | https://blackfire.io/profiles/e814e0ac-3a0d-4c4e-9e8c-a952883f12e2/graph

So I guess you can imagine the memory consumption for 20k or more items.

Command used for importing data:

<?php

namespace AppBundle\Command;

use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            $progress->advance();
        }
    }

}

Wrapping import in a nested transactions brings following results. A bit better but not much.

Amount | Time | Start Memory Usage | End Memory Usage | Blackfire Report -------- | -----| ---------------------| --------------------| ----------------- 1000 | ~1min | 16mib | 84mib | https://blackfire.io/profiles/2600b39a-b6e3-41b9-9ba3-0ce2142d123d/graph 10000 | ~20min | 16mib | 698mib | https://blackfire.io/profiles/78176e0b-7afe-477d-982c-387a9cc9b3d0/graph

Command used for nested transactions

<?php

namespace AppBundle\Command;

use Pimcore\Db;
use Pimcore\Model\DataObject\Service;
use Pimcore\Model\DataObject\Test;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Helper\ProgressBar;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class AppTestCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('app:test')
            ->addArgument('itemsToCreate', InputArgument::REQUIRED);
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $itemsToCreate = $input->getArgument('itemsToCreate');

        $progress = new ProgressBar($output, $itemsToCreate);
        $progress->setFormat(
            '%current%/%max% [%bar%] %percent:3s%% (%elapsed:6s%/%estimated:-6s%) %memory:6s%: %message%'
        );

        Db::get()->beginTransaction();

        for ($i = 0; $i < $itemsToCreate; $i++) {
            $progress->setMessage('Create Item with Number '.$i);

            $item = new Test();
            $item->setBlub(uniqid());
            $item->setBlub2(uniqid());
            $item->setBlubLocalized(uniqid());
            $item->setKey($i.'-'.uniqid());
            $item->setParent(Service::createFolderByPath('/blub-'.$itemsToCreate.'/'.$i % 100));
            $item->save();

            if ($i % 100 === 0) {
                Db::get()->commit();
                Db::get()->beginTransaction();
            }

            $progress->advance();
        }

        Db::get()->commit();
    }

}

TL;DR

Pimcore uses a lot of memory on persisting a lot of items. I am not sure why and how to optimize. But importing thousands of items in one import is not possible at the moment.

• Introduces FOSJS Routing Bundle
• Replace most of the routes with actual Route Generation
• Replace the other by adding at least the Base-Path, wasn't sure how to handle those properly
• Replace all static route uses in PHP and Templates

This requires precise review and testing. I already tested most parts, but I am pretty sure I missed something.

Expected behavior

When clicking on the Versions tab on DataObjects it should be possible to view and restore from versions created prior to Pimcore X which contains Classification Stores.

Actual behavior

[2021-10-20T09:56:31.636336+02:00] php.CRITICAL: Uncaught Error: 
Cannot assign __PHP_Incomplete_Class to property Pimcore\Model\DataObject\Classificationstore::$object of type ?Pimcore\Model\DataObject\Concrete {"exception":"[object] (TypeError(code: 0): 
Cannot assign __PHP_Incomplete_Class to property Pimcore\\Model\\DataObject\\Classificationstore::$object of type ?Pimcore\\Model\\DataObject\\Concrete at /var/www/pimcore/vendor/pimcore/pimcore/lib/Tool/Serialize.php:43)"} []
[2021-10-20T09:56:31.636655+02:00] request.CRITICAL: Uncaught PHP Exception TypeError: 
"Cannot assign __PHP_Incomplete_Class to property Pimcore\Model\DataObject\Classificationstore::$object of type ?Pimcore\Model\DataObject\Concrete" at /var/www/pimcore/vendor/pimcore/pimcore/lib/Tool/Serialize.php line 43 {"exception":"[object] (TypeError(code: 0): 
Cannot assign __PHP_Incomplete_Class to property Pimcore\\Model\\DataObject\\Classificationstore::$object of type ?Pimcore\\Model\\DataObject\\Concrete at /var/www/pimcore/vendor/pimcore/pimcore/lib/Tool/Serialize.php:43)"} []

Steps to reproduce

Save a version in Pimcore 6.9.6 Upgrade to Pimcore 10.1+ Try to restore the object

edit example version file (renamed with .txt to allow upload here on Github) 255796.txt

Expected behavior

All fields should be visible in the Ext JS interface.

Actual behavior

The domain-specific URL Slug field (and its button is hidden:

By manually adjusting the height of the elements in the DOM it is possible to show that both fields are available, just hidden behind some inline height attributes:

Steps to reproduce

Putting a URL Slug in a fieldset which in turn is in a Localized fields

When you update an asset file (also when you update a new one), all thumbnails of this asset get deleted, see https://github.com/pimcore/pimcore/blob/6d6774a139e7cadd3ed446ef300c50523838a4c8/models/Asset/Image.php#L52

This calls https://github.com/pimcore/pimcore/blob/6d6774a139e7cadd3ed446ef300c50523838a4c8/models/Asset.php#L1987-L2006 The folder structure for thumbnail storage is currently: <asset storage>/<path to asset>/image-thumb__<asset id>__<thumbnail name>/<asset filename> To find all thumbnails which belong to the just saved asset, above function needs to loop through all the thumbnail files in <asset storage>/<path to asset>/ and filter for those whose path contains image-thumb__<asset id>__.

Depending on the asset folder structure and storage type this may need a lot of time. For example when you have 50000 images in one asset folder and you have 20 thumbnail definitions (not regarding the automatically or dynamically created thumbnail definitions, e.g. tree preview or custom size download), this folder will contain up to 1M files. In addition when the thumbnails are not stored locally but on a cloud storage provider, the problem gets even worse. In our case uploading one asset to such a folder needed several minutes to complete.

This PR changes the folder structure for storing thumbnails to <asset storage>/<path to asset>/image-thumb__<asset id>/<thumbnail name>/<asset filename>. This way in case of updating an asset file, we only need to delete the directory <asset storage>/<path to asset>/image-thumb__<asset id>. No loop necessary anymore.

With #5594 saving field's default value to database was introduced. The problem is that these default values do not get saved to versions currently. This is especially bad because when you publish this version, you will again have null in the respective fields.

Steps to reproduce:

1. Create class with input field abc, default value test
2. Create object of above class and save it
3. In the database tables now there is test in column abc
4. Go to versions tab, select jsut saved version -> there abc is empty
5. Restore the just saved version -> The value is now also null in the database column -> this is really bad because you can not restore the exact state the object had at this point (what is the main purpose of versioning beside tracking changes)

With this PR the default values are also saved to the database (and also to the PHP object being saved - which is important, if you have further operations with it after saving).

This PR adds getters and setters for reverse many-to-many relations. For example when you have: Product

• categories (many-to-many-object-relation)

Category

• products (reverse many-to-many relation)

$products = $category->getProducts();
$products[] = Product::getById(123);
$category->setProducts($products);
$category->save();

Partly resolves #3920

Hi,

It seems the recursive DeepCopy used when saving a new version has a serious performance impact when you have a lot of relationships. The following is a blackfire profile for a batch edit of 25 objects:

https://blackfire.io/profiles/9902f36c-0857-4511-ac3a-54bf25e10a29/graph

When I make a shallow copy it's significantly faster. Is there any reason for traversing all of the related objects? Of course direct relationships have to be included because they are shown in the version tab but any deeper than this is not included in the version data anyway.

When an element gets opened in Pimcore admin UI, it gets locked so a message gets shown to other users that this element is already open.

When the element tabs get closed, those locks get released.

But when a user simply closes the browser, the locks do not get released. On the one hand this causes the funny effect that you get messages that the current user himself has the object currently open (even when using only one browser window). And on the other hand this irritates other users because they may not dare to change anything although the other user actually already closed his browser (and with it all the elements).

This is caused by the CoreHandler in the latest version. If i revert it to the state before the error is gone. I think it relates to this: https://github.com/pimcore/pimcore/issues/7039

Expected behavior

If we have a lot of tabs open (more than fits on the screen), when the next tab is opened, they move to the currently selected object.

Actual behavior

If we have a lot of tabs open (more than fits on the screen), when we open another tab, we have to move the tabs yourself using the arrow.

Steps to reproduce

Open a lot of tabs (more than fits on the screen) and then open next and next tab.

Expected behavior

Upload new image at imageGallery directly.

Actual behavior

Pimcore throws an error:

{"success":false,"message":"Mime type does not match with asset type: hotspotimage"}

Steps to reproduce

Add a new element in object imageGallery. Click on Hamburger-Menu and upload a new image. This happens on official pimcore demo site, too.

Changes in this pull request

Resolves #14001

Additional info

Just a quick edit, didn't checked if there are any other new features (had no time)

Expected behavior

Documentation should define why the descriped way is the "best way".

Actual behavior

At least for me it's unclear why this is the best way.

Steps to reproduce

https://pimcore.com/docs/pimcore/current/Development_Documentation/Objects/Object_Classes/Data_Types/Reverse_Object_Relation_Type.html#page_Working-with-PHP-API

Why do we have to use:

$def = $object->getClass()->getFieldDefinition("myNonOwnerObjectField");
$refKey = $def->getOwnerFieldName();
$refId = $def->getOwnerClassId();
$nonOwnerRelations = $object->getRelationData($refKey, false, $refId);

which returns an array with the object id's instead of

$object->getMyNonOwnerObjectField()

which returns an array with the objects?

Fixes PhpStan level 6 errors:

• Property * type has no value type specified in iterable type array.
• Method * return type has no value type specified in iterable type array.
• Method * has parameter * with no value type specified in iterable type array.