While executing a job, the job is removed even if the config mention 'queues' => array( array('mails' => array( 'table_name' => 'queue_default', 'delete_lifetime' => 1440, 'buried_lifetime' => 5760, 'sleep_when_idle' => 2, ) ) ),

I've read the following in the documentation:

"Warning : In previous versions of SlmQueueDoctrine the worker would quit if there where no jobs available for processing. That meant you could safely create a cronjob that would start a worker every minute. If you do that now you will quickly run out of available resources."

I'm curious as to why you would abandon this particular feature. Most/all of the queued jobs are pushed automatically, why then should someone manual run/process the queue every time? Could you perhaps add this feature back or at least explain why you abandoned it in the first place.

By the way if I'm missing something, maybe a new way to set up Cronjobs for SlimQueueDoctrine, could you share an example. Thank you.

I think we don't need the dependency to Doctrine ORM Module. Doctrine Module should be enough (we mostly rely on DBAL here).

Could somebody make the change ? @basz

EDIT : we should completely remove the dependency to DoctrineModule at all I think, and mimic the ZfcRbac configuration.

Is there a way to make the worker faster?

For example, I add 10 actions each second but the worker execute only 5 of them per second... in this way the queue_default table increase exponentially and it will be never empty.

How can I do?

Thanks

To the SlmQueueDoctrine users (ping @stefankleff, @basz) is anyone willing to write tests ? SlmQueueDoctrine is the only one not to have tests, and I'd like to have them so I can tag first version (0.2.0).

Thanks a lot !

It's still need test if Doctrine works if both dasprid/container-interop-doctrine and doctrine/doctrine-orm-module are used.

Please, see https://github.com/juriansluiman/SlmQueueDoctrine/issues/145#issuecomment-474785002.

Worker should catch PDOExceptions to avoid the worker from dying.

My 'supervised' workers are restarted three times in case of unexpected halting this however isn't enough to survive a simple mysql restart.

The application has thrown an exception!
PDOException
SQLSTATE[HY000] [2002] No such file or directory
.../vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:40

Hi,

Currently I'm migrating an Zend MVC application to Zend Expressive, which use this module.

SlmQueue is already Expressive "ready". Anyone has used SlmQueueDoctrine with Zend Expressive? Or you are using another adapter (Sqs)?

Thanks.

IM

Hey,

I added two queues in my configuration. The default queue worker works fine. but if I start the worker for the email queue, I get the following result. But in my database table email_queue is only one job, which does not get executed. If I remove the MaxRunsStrategy the worker processes more and more jobs, which do not exist. What is wrong?

Console output by running php index.php queue doctrine default --start Finished worker for queue 'default':

• 50 seconds passed
• 22.95MB memory usage
• 0 jobs processed
• lifetime of 50 seconds reached

Console output by running php index.php queue doctrine email --start Finished worker for queue 'email':

• 30 jobs processed
• 0 seconds passed
• 22.75MB memory usage
• 30 jobs processed
• maximum of 30 jobs processed

Configuration

[
    'queues' => [
        'default' => [
            'table_name' => 'queue',
            'buried_lifetime' => 60 * 24 * 3,
            'deleted_lifetime' => 60
        ],

        'email' => [
            'table_name' => 'email_queue',
            'buried_lifetime' => 60 * 24 * 3,
            'deleted_lifetime' => 60
        ]
    ],

    'queue_manager' => [
        'factories' => [
            'default' => \SlmQueueDoctrine\Factory\DoctrineQueueFactory::class,
            'email' => \SlmQueueDoctrine\Factory\DoctrineQueueFactory::class,
        ]
    ],
        
    'worker_strategies' => [
        'queues' => [
            'default' => [
                \SlmQueue\Strategy\WorkerLifetimeStrategy::class => [
                    'lifetime' => 50
                ]
            ],
 
            'email' => [
                \SlmQueue\Strategy\WorkerLifetimeStrategy::class => [
                    'lifetime' => 50
                ],

                \SlmQueue\Strategy\MaxRunsStrategy::class => [
                    'max_runs' => 30
                ]
            ]
        ]
    ]
];

When adding job to queue with $queue->push($job); There is an SQL exception : SQLSTATE[23502]: Not null violation: 7 ERROR: null value in column "id" violates not-null constraint

We already decided to transferhttps://github.com/juriansluiman/SlmQueue to the JouwWeb organization. We also actively use SlmQueueDoctrine ourselves. I think it makes sense to also transfer this one to us.

@basz and @juriansluiman Are you okay with this?

While on development, it would be handy if the job queue reloads as soon as a file was changed (but only just after a job finishes obviously).

Inspiration: https://github.com/spatie/laravel-horizon-watcher.

Hi there

Fetching the ObjectManager in onClear can never be negated since ObjectManagerAwareInterface->getObjectManager() can't return null.

This should be wrapped in a try/catch

ClearObjectManagerStrategy

There is currently no way to hook into an event that can tell you whenever a job failed reliably. When a job crashes / exits / dies during executing a requests, it will remain on status busy.

Ideally there would be a system in place that can detect this, and that should broadcast an error event.

Maybe this should / could be fixed on the SlmQueue level.

Once per week a cronjob creates puts approx. 2000 messages into the queue during one transaction in our system. We have 4 workers processing messages from this doctrine queue. Unfortunately, several workers usually end up in deadlocks when fetching these messags from the queue. I tried looking or solutions but did not manage to come up with anything, also did not find anything similar in the already existing issues, so I'll leave the exception information below.

Caught exception while processing queue; An exception occurred while executing 'SELECT * FROM queue_default WHERE (status = ?) AND (queue = ?) AND (scheduled <= ?) ORDER BY priority ASC, scheduled ASC LIMIT 1 FOR UPDATE' with params [1, "my_queue", "2020-07-13 04:00:25.690387"]:

SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock try restarting transaction; An exception occurred while executing 'SELECT * FROM queue_default WHERE (status = ?) AND (queue = ?) AND (scheduled <= ?) ORDER BY priority ASC, scheduled ASC LIMIT 1 FOR UPDATE' with params [1, "my_queue", "2020-07-13 04:00:25.690387"]:

This PR extends the DoctrineQueue to publish scheduled and priority. Previously only id got exposed to the outside world.

My use-case is that I would like to add a strategy that can send the time it took before the job started to an outside monitoring system.

Sometimes it can happen, that script will allocate too much memory (eg. pdf generation) and then it throws uncatchable fatal error.

SlmQueueDoctrine recovers those jobs and runs it over and over, not firing up any event, and I can't find any way to handle those errors. I can set executionTime for recover, and they will not be recovered, but those jobs are still left unprocessed in database, and they just die silently.

What I need to do, is to not recover jobs, and set some things, eg. using Strategy. When memory is out, I can't do anything besides registering new register_shutdown_function, rising memory limit and do what I want. But this is reduntant, because I am already using Strategies to catch errors, send them to raygun or log them.

In this case nothing seems to work, script is stopped, slmqueue is trying to recover it, and it is stopped again. No event fired.

Is there any way to mark those jobs as finished with error, unrecoverable?