Dashboard and code-driven configuration for Laravel queues.

I'm running horizon on a latest forge machine as documented... the daemon with php artisan horizon and on deployments php artisan horizon:terminate but time to time i need to manually run php artisan horizon:purge.

This is the output i just got after after hours of last release:

$ php artisan horizon:purge
Observed Orphan: 17043
Observed Orphan: 30634
Observed Orphan: 31084
Observed Orphan: 31085

I can confirm it's orphans by running htop on tree mode (press f5) and i see these process as root process, not inside the master php artisan horizon process.

And also, every time i run purge, it ALWAYS wrongly see 1 process as orphan:

$ php artisan horizon:purge
Observed Orphan: 17059

I haven't found a pattern yet why this is happening.

• Horizon Version: v3.4.4
• Laravel Version: v6.9.0
• PHP Version: 7.4.0
• Redis Driver & Version: phpredis 5.1.1

Description:

This issue has been haunting us a long time now. Initially we did not know why this was randomly happening (Issue #708 ) later we found out that the re-connecting in general was not working (Issue #716 ) which was fixed by taylor (thank you!) but this only is now working correctly for redis instances that are not password protected.

In a situation where horizon is running against a redis instances that is password protected, and redis is shortly (or long) unavailable, even after redis is back, horizon will not recover spitting out NOAUTH errors like this one:

ERROR: NOAUTH Authentication required. {"exception":"[object] (RedisException(code: 0): NOAUTH Authentication required. at /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php:111)
[stacktrace]
#0 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(111): Redis->lLen('commands:master...')
#1 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(440): Illuminate\\Redis\\Connections\\Connection->command('llen', Array)
#2 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/Connection.php(211): Illuminate\\Redis\\Connections\\PhpRedisConnection->command('llen', Array)
#3 /var/www/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(482): Illuminate\\Redis\\Connections\\Connection->__call('llen', Array)
#4 /var/www/vendor/laravel/horizon/src/RedisHorizonCommandQueue.php(51): Illuminate\\Redis\\Connections\\PhpRedisConnection->__call('llen', Array)
#5 /var/www/vendor/laravel/horizon/src/MasterSupervisor.php(263): Laravel\\Horizon\\RedisHorizonCommandQueue->pending('master:d4711be9...')
#6 /var/www/vendor/laravel/horizon/src/MasterSupervisor.php(240): Laravel\\Horizon\\MasterSupervisor->processPendingCommands()
#7 /var/www/vendor/laravel/horizon/src/MasterSupervisor.php(213): Laravel\\Horizon\\MasterSupervisor->loop()
#8 /var/www/vendor/laravel/horizon/src/Console/HorizonCommand.php(63): Laravel\\Horizon\\MasterSupervisor->monitor()
#9 [internal function]: Laravel\\Horizon\\Console\\HorizonCommand->handle(Object(Laravel\\Horizon\\Repositories\\RedisMasterSupervisorRepository))
#10 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#11 /var/www/vendor/laravel/framework/src/Illuminate/Container/Util.php(36): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#12 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#13 /var/www/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(App\\Extensions\\Illuminate\\Foundation\\Application), Array, Object(Closure))
#14 /var/www/vendor/laravel/framework/src/Illuminate/Container/Container.php(590): Illuminate\\Container\\BoundMethod::call(Object(App\\Extensions\\Illuminate\\Foundation\\Application), Array, Array, NULL)
#15 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(202): Illuminate\\Container\\Container->call(Array)
#16 /var/www/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#17 /var/www/vendor/laravel/framework/src/Illuminate/Console/Command.php(189): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#18 /var/www/vendor/symfony/console/Application.php(1011): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#19 /var/www/vendor/symfony/console/Application.php(272): Symfony\\Component\\Console\\Application->doRunCommand(Object(Laravel\\Horizon\\Console\\HorizonCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#20 /var/www/vendor/symfony/console/Application.php(148): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#21 /var/www/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#22 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(131): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#23 /var/www/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#24 {main}

Important. This ONLY happens with the phpredis driver, predis is not affected.

Steps To Reproduce:

• Have a fresh (or not fresh) laravel installation with horizon.
• Have a redis instance that is password protected (for example running it from this dockerfile)
• configure your database.php in laravel accordingly to have correct redis password set

FROM redis:5.0.2
COPY redis.conf /usr/local/etc/redis/redis.conf
CMD [ "redis-server", "/usr/local/etc/redis/redis.conf" ]

where the redis.conf looks like this

requirepass testpassword

• Then start horizon, let it warm up
• turn off your redis instance or break the connection between horizon & redis somehow.
• turn on redis instance again (or enable connection between horizon & redis again)
• horizon will continuously spit out errors as specified above (NOAUTH)

I have a server with 4 horizon instances (all running under another prefix), Job insertion always goes to the right horizon instance, but it is like a lottery which worker handles the job, which is really confusing to work with.

I am running horizon on horizontal scale setup with multiple server running a same instance. I found that once in a while there are jobs that's being processed multiple times and somehow passed my constraint checking.

I suspect this could be cause by that job being processed by multiple workers at the same time, hence it manage to pass the constraint check I have on the job handle() function.

So my question, is horizon are fully compatible to run on horizontal scaling setup?

I have a job that gets dispatched to a queue inside a supervisor that has 20 workers assigned to it to handle large concurrency. There seems to be an issue (race condition maybe?) whereby a single dispatched job gets picked up by multiple horizon workers and processed multiple times. This causes problems because the job can only be processed once or it will fail.

I added a bunch of info logs in various places, including the job itself, the code that dispatches the job, and Repositories/RedisJobRepository.php inside Horizon. Here is what I found.

First the logs:

1. "dispatching job" right before the call to dispatch(new MyJob())
2. "enter job constructor" inside the job's constructor
3. "executing job " inside the job's execute method
4. "returning from job " at the very end of the job's execute method
5. "reserving job " inside RedisJobRepository::reserved
6. "completed job " inside RedisJobRepository::completed

Then the output:

[2017-10-17 20:15:38] pre.INFO: dispatching job
[2017-10-17 20:15:38] pre.INFO: enter job constructor 27262
[2017-10-17 20:15:38] pre.INFO: reserving job 202 3316
[2017-10-17 20:15:38] pre.INFO: executing job 3316
[2017-10-17 20:15:43] pre.INFO: reserving job 202 3307
[2017-10-17 20:15:43] pre.INFO: executing job 3307
[2017-10-17 20:15:45] pre.INFO: returning from job 3316
[2017-10-17 20:15:45] pre.INFO: completed job 202 3316
[2017-10-17 20:15:48] pre.INFO: reserving job 202 3323
[2017-10-17 20:15:48] pre.INFO: executing job 3323
[2017-10-17 20:15:48] pre.INFO: completed job 202 3323
[2017-10-17 20:15:48] pre.ERROR: <exception thrown from job because it already ran>
[2017-10-17 20:15:50] pre.INFO: returning from job 3307
[2017-10-17 20:15:50] pre.INFO: completed job 202 3307

Note that only a single job was constructed and dispatched, but the same job got reserved 3 times by 3 different horizon workers, and executed 3 times. I'm not quite sure how 2 jobs managed to succeed while the 3rd one failed; according to the code, it is supposed to clean up a temp file that is required in the job, but perhaps the jobs from processes 3316 and 3307 both managed to load the file for processing before one or the other deleted it.

The above output is quite consistent and reproducible, obviously with different ids each time. I tried to set 'tries' => 1 in my horizon config for this supervisor and ended up with a different set of consistently reproducible logs:

[2017-10-17 20:20:55] pre.INFO: dispatching job  
[2017-10-17 20:20:55] pre.INFO: enter job constructor 28229  
[2017-10-17 20:20:55] pre.INFO: reserving job 204 4358  
[2017-10-17 20:20:55] pre.INFO: executing job 4358  
[2017-10-17 20:21:00] pre.INFO: reserving job 204 4369  
[2017-10-17 20:21:00] pre.INFO: completed job 204 4369  
[2017-10-17 20:21:01] pre.ERROR: A queued job has been attempted too many times or run too long. The job may have previously timed out. {"exception":"[object] (Illuminate\\Queue\\MaxAttemptsExceededException(code: 0): A queued job has been attempted too many times or run too long. The job may have previously timed out. at /opt/www/pre/site/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:394)
[2017-10-17 20:21:02] pre.INFO: returning from job 4358  
[2017-10-17 20:21:02] pre.INFO: completed job 204 4358

Now, it only reserves the job twice on two workers, and instead of getting my "file not found" exception inside the job, it throws the MaxAttemptsExceededException.

I would be happy to try anything and provide more details as needed!

Laravel: 5.5.14 Horizon: 1.0.4

Horizon supervisor config:

            'high-concurrency' => [
                'connection' => 'redis',
                'queue' => ['a', 'bunch', 'of', 'queues''],
                'balance' => 'false',
                'processes' => 20, 
                'tries' => 1, // or 3
                'timeout' => 300,
            ],

config/queue.php:

        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue'  => env('REDIS_QUEUE', 'default'), // set to "pre"
            'retry_after' => 5,
        ],

config/database.php:

    'redis' => [

        'client' => 'predis',

        'default' => [
            'host' => env('REDIS_HOST', 'localhost'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => env('REDIS_DATABASE', 0), // not set in the .env
        ],

    ],

• Horizon Version: 5.4.0
• Laravel Version: 8.13.0
• PHP Version: 7.4.3
• Redis Driver & Version: phpredis 5.3.0
• Database Driver & Version: MySQL 8.0.22

Description:

Horizon is working nicely for few days, and next I woke up in the morning and see all my Jobs are failed. When I check the logs, I got these:

[2020-11-09 10:27:37] production.ERROR: Trying to access array offset on value of type null {"exception":"[object] (ErrorException(code: 0): Trying to access array offset on value of type null at /var/www/laravel-api/production/vendor/laravel/horizon/src/JobPayload.php:48)
[stacktrace]
#0 /var/www/laravel-api/production/vendor/laravel/horizon/src/JobPayload.php(48): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /var/www/laravel-api/production/vendor/laravel/horizon/src/Repositories/RedisJobRepository.php(428): Laravel\\Horizon\\JobPayload->id()
#2 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Support/helpers.php(263): Laravel\\Horizon\\Repositories\\RedisJobRepository->Laravel\\Horizon\\Repositories\\{closure}()
#3 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Redis/Connections/PhpRedisConnection.php(406): tap()
#4 /var/www/laravel-api/production/vendor/laravel/horizon/src/Repositories/RedisJobRepository.php(435): Illuminate\\Redis\\Connections\\PhpRedisConnection->pipeline()
#5 /var/www/laravel-api/production/vendor/laravel/horizon/src/Listeners/MarkJobsAsMigrated.php(36): Laravel\\Horizon\\Repositories\\RedisJobRepository->migrated()
#6 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(411): Laravel\\Horizon\\Listeners\\MarkJobsAsMigrated->handle()
#7 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(236): Illuminate\\Events\\Dispatcher->Illuminate\\Events\\{closure}()
#8 /var/www/laravel-api/production/vendor/laravel/horizon/src/RedisQueue.php(174): Illuminate\\Events\\Dispatcher->dispatch()
#9 /var/www/laravel-api/production/vendor/laravel/horizon/src/RedisQueue.php(128): Laravel\\Horizon\\RedisQueue->event()
#10 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Support/helpers.php(263): Laravel\\Horizon\\RedisQueue->Laravel\\Horizon\\{closure}()
#11 /var/www/laravel-api/production/vendor/laravel/horizon/src/RedisQueue.php(129): tap()
#12 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(211): Laravel\\Horizon\\RedisQueue->migrateExpiredJobs()
#13 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(187): Illuminate\\Queue\\RedisQueue->migrate()
#14 /var/www/laravel-api/production/vendor/laravel/horizon/src/RedisQueue.php(111): Illuminate\\Queue\\RedisQueue->pop()
#15 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(323): Laravel\\Horizon\\RedisQueue->pop()
#16 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(332): Illuminate\\Queue\\Worker->Illuminate\\Queue\\{closure}()
#17 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(145): Illuminate\\Queue\\Worker->getNextJob()
#18 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(116): Illuminate\\Queue\\Worker->daemon()
#19 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\\Queue\\Console\\WorkCommand->runWorker()
#20 /var/www/laravel-api/production/vendor/laravel/horizon/src/Console/WorkCommand.php(50): Illuminate\\Queue\\Console\\WorkCommand->handle()
#21 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\\Horizon\\Console\\WorkCommand->handle()
#22 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#23 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#24 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#25 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/Container.php(596): Illuminate\\Container\\BoundMethod::call()
#26 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call()
#27 /var/www/laravel-api/production/vendor/symfony/console/Command/Command.php(258): Illuminate\\Console\\Command->execute()
#28 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run()
#29 /var/www/laravel-api/production/vendor/symfony/console/Application.php(920): Illuminate\\Console\\Command->run()
#30 /var/www/laravel-api/production/vendor/symfony/console/Application.php(266): Symfony\\Component\\Console\\Application->doRunCommand()
#31 /var/www/laravel-api/production/vendor/symfony/console/Application.php(142): Symfony\\Component\\Console\\Application->doRun()
#32 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\\Component\\Console\\Application->run()
#33 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run()
#34 /var/www/laravel-api/production/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle()
#35 {main}

[2020-11-09 10:27:38] production.ERROR: Undefined offset: 1 {"exception":"[object] (ErrorException(code: 0): Undefined offset: 1 at /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php:250)
[stacktrace]
#0 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(250): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/RedisQueue.php(189): Illuminate\\Queue\\RedisQueue->retrieveNextJob()
#2 /var/www/laravel-api/production/vendor/laravel/horizon/src/RedisQueue.php(111): Illuminate\\Queue\\RedisQueue->pop()
#3 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(323): Laravel\\Horizon\\RedisQueue->pop()
#4 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(332): Illuminate\\Queue\\Worker->Illuminate\\Queue\\{closure}()
#5 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(145): Illuminate\\Queue\\Worker->getNextJob()
#6 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(116): Illuminate\\Queue\\Worker->daemon()
#7 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\\Queue\\Console\\WorkCommand->runWorker()
#8 /var/www/laravel-api/production/vendor/laravel/horizon/src/Console/WorkCommand.php(50): Illuminate\\Queue\\Console\\WorkCommand->handle()
#9 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\\Horizon\\Console\\WorkCommand->handle()
#10 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#11 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#12 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#13 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Container/Container.php(596): Illuminate\\Container\\BoundMethod::call()
#14 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call()
#15 /var/www/laravel-api/production/vendor/symfony/console/Command/Command.php(258): Illuminate\\Console\\Command->execute()
#16 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run()
#17 /var/www/laravel-api/production/vendor/symfony/console/Application.php(920): Illuminate\\Console\\Command->run()
#18 /var/www/laravel-api/production/vendor/symfony/console/Application.php(266): Symfony\\Component\\Console\\Application->doRunCommand()
#19 /var/www/laravel-api/production/vendor/symfony/console/Application.php(142): Symfony\\Component\\Console\\Application->doRun()
#20 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\\Component\\Console\\Application->run()
#21 /var/www/laravel-api/production/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run()
#22 /var/www/laravel-api/production/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle()
#23 {main}
"} 

config/horizon.php:

<?php

use Illuminate\Support\Str;

return [
    'domain' => null,
    'path' => 'admin/horizon',
    'use' => 'default',
    'prefix' => env(
        'HORIZON_PREFIX',
        Str::slug(env('APP_NAME', 'laravel'), '_').'@'.env('APP_ENV').'_horizon:'
    ),
    'middleware' => ['web'],
    'waits' => [
        'redis:default' => 60,
    ],
    'trim' => [
        'recent' => 60,
        'pending' => 60,
        'completed' => 60,
        'recent_failed' => 10080,
        'failed' => 10080,
        'monitored' => 10080,
    ],
    'metrics' => [
        'trim_snapshots' => [
            'job' => 24,
            'queue' => 24,
        ],
    ],
    'fast_termination' => false,
    'memory_limit' => 64,
    'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'simple',
                'processes' => 10,
                'tries' => 1,
                'timeout' => 600,
            ],
            'scrapper' => [
                'connection' => 'redis',
                'queue' => ['price'],
                'balance' => 'simple',
                'processes' => 1,
                'tries' => 1,
                'timeout' => 600,
            ],
            'scout' => [
                'connection' => 'redis',
                'queue' => ['scout'],
                'balance' => 'simple',
                'processes' => 1,
                'tries' => 3,
                'timeout' => 600,
            ],
        ],

        'development' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'simple',
                'processes' => 2,
                'tries' => 1,
                'timeout' => 600,
            ],
            'scrapper' => [
                'connection' => 'redis',
                'queue' => ['price'],
                'balance' => 'simple',
                'processes' => 1,
                'tries' => 1,
                'timeout' => 600,
            ],
            'scout' => [
                'connection' => 'redis',
                'queue' => ['scout'],
                'balance' => 'simple',
                'processes' => 1,
                'tries' => 1,
                'timeout' => 600,
            ],
        ],

        'local' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'simple',
                'processes' => 2,
                'tries' => 1,
            ],
            'scrapper' => [
                'connection' => 'redis',
                'queue' => ['price'],
                'balance' => 'simple',
                'processes' => 1,
                'tries' => 1,
            ],
            'scout' => [
                'connection' => 'redis',
                'queue' => ['scout'],
                'balance' => 'simple',
                'processes' => 1,
                'tries' => 3,
            ],
        ],
    ],
];

I had also this error on Laravel 7 / Horizon 4, so I upgrated to Laravel 8 / Horizon 5 with the hope that this error will disappear, in vain.

I have a few queue jobs that needs some time to be run because they are encoding videos. They typically last a couple minutes.

When I fire the job, it runs correctly and I can see that it's being executed in the "Recent Jobs" section of Horizon. I can see it running, then it gets a red cross icon as if it failed, and then immediately after it becomes green marking it as completed. The problem is that I can see it also in the "Failed" section, where it says that the job has failed throwing a "Illuminate\Queue\MaxAttemptsExceededException".

At the moment I have 2 processes for the queue. If I set it to just 1 process, then the problem is not happening anymore.

I also tried setting the "timeout" property for the job to 1800 seconds, but it looks like it doesn't care.

Could it be that if I have 2 processes the second one tries to run the job even if the first one is still running it? Is there something I need to consider for especially long jobs? I have another queue with 10 processes running small tasks and that is not giving any problem at all.

Thanks a lot for your help and congrats for the amazing library!!!

• Horizon Version: 3.4.3
• Laravel Version: 6.6.0
• PHP Version: 7.2.24
• Redis Driver & Version: phpredis 5.1.1
• Database Driver & Version: mysql

Description:

When running

Steps To Reproduce:

• Create a job that needs more than 60 seconds and to run and uses a lot of memory (I generated 3,000,000 unique strings using $faker->unique()->regexify() for the sake of testing). I'll attach the job file.
• (Re)start horizon
• Queue the job

What happens:

• Despite the memory limit being specified as 64MB in config/horizon.php the worker threads are started with the option --memory=128 which is also ignored and the processes are allowed to consume nearly a whole gigabyte of resident memory without failing

• Despite the job completing without issue, Laravel logs an exception (local.ERROR: ... attempted too many times or run too long. The job may have previously timed out. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): App\Jobs\SendOrderEmail ... at /work/jsong/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:612...)

• A job that took 1 minute and 36 seconds to complete is displayed in the horizon recent jobs list as having taken 3.63 seconds. Another that took 1m32s displays as 0.68s. Another that also took 1m32s displays as 0.18s.

Looking at the timing of the exceptions and when the job was completed it seems that Horizon is logging the difference between when the exception was thrown and when the job was completed. I thought that perhaps this is more of a laravel issue than a horizon issue. But when I work the queue manually with php artisan queue:work or php artisan queue:work --deamon the exception is not thrown and the horizon interface reports time correctly

SendOrderEmail.php.txt

I know Horizon currently does not support the Database driver and that there are currently no plans for this. But are you open to a PR to add this? In that case, I would consider creating a proof-of-concept for the database.

I can get Horizon working fine, by just adding the readyNow method to the driver, but would probably need to invest some more time for the actual jobs to show up.

(Yes I know Redis is probably better, but I have a use-case with extra filtering using relations in the jobs which I can't really do with Redis)

• Horizon Version: 3.7.1
• Laravel Version: 5.8
• PHP Version: 7.3
• Redis Driver & Version: predis 5.2
• Database Driver & Version: Mysql8.0

Description:

When we hover the mouse on a delayed tag to a delayed job, and when we click to see the detail of the job, the "Delayed Until" time is wrong.

• Redis time is ok
• Host time is ok
• Laravel timezone is ok

Steps To Reproduce:

Launch job Go to horizon and check

Screen : https://anopetia.tinytake.com/tt/NDA3NTQ4Nl8xMjUzNTMxMw

I'm using Horizon on multiple projects, I had to make them play nice so they all work side-by-side

Perhaps as a result of this, I've never been able to get any of them to display proper stats. I have a cron job running that triggers the snapshot every 5 minutes.

The metrics are empty:

The dashboard shows no data for most of the fields, on top of that I know the number of queued jobs to be way off

Even odder! There's nothing showing in the Recent Jobs.. even though it's processing ~20 jobs per minute for hours now

What could be causing all this?

Bumps json5 from 1.0.1 to 1.0.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

• Horizon Version: v5.10.2
• Laravel Version: v9.35.1
• PHP Version: 8.1.10
• Redis Driver & Version: ext-redis 5.3.7
• Database Driver & Version: mysqlnd 81.10

Description:

• Laravel Queue works can be assigned to multiple queues and will work on them in the configured priority.
• Laravel Horizon also has a queue option where multiple queues can be configured.
• My assumption is that, in auto balance mode, Horizon would assign most workers to the highest priority queue if it is not empty in order to empty it as fast as possible.
• Instead Horizon will assign most workers to the queue with the most pending jobs, regardless of the queue's priority.

Steps To Reproduce:

1. In horizon.php configure Horizon like so:

      'defaults' => [
       'supervisor-1' => [
           'connection' => 'redis',
           'queue' => ['default', 'low'],
           'balance' => 'auto',
           'maxProcesses' => 10,
       ],
   ]
   

2. Create 10 jobs on the default queue and 1.000 jobs on the low queue.

Expected:

Horizon assigns most workers to the default queue because it has the higher priority.

Actual:

Horizon assigns most workers to the low queue because it has more pending jobs: