Beanstalk is a simple, fast work queue.

Last update: Jun 28, 2022

Build Status codecov

beanstalkd

Simple and fast general purpose work queue.

https://beanstalkd.github.io/

See doc/protocol.txt for details of the network protocol.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms. See CodeOfConduct.txt for details.

Quick Start

$ make
$ ./beanstalkd

also try,

$ ./beanstalkd -h
$ ./beanstalkd -VVV
$ make CFLAGS=-O2
$ make CC=clang
$ make check
$ make install
$ make install PREFIX=/usr

Requires Linux (2.6.17 or later), Mac OS X, FreeBSD, or Illumos.

Currently beanstalkd is tested with GCC and clang, but it should work with any compiler that supports C99.

Uses ronn to generate the manual. See http://github.com/rtomayko/ronn.

Subdirectories

  • adm - files useful for system administrators
  • ct - testing tool; vendored from https://github.com/kr/ct
  • doc - documentation
  • pkg - scripts to make releases

Tests

Unit tests are in test*.c. See https://github.com/kr/ct for information on how to write them.

GitHub

https://github.com/kr/beanstalkd
Comments
  • 1. AF_UNIX support for beanstalkd

    Add the posibility of accepting connections from a local socket (bound to a filesystem path).

    To minimize portability issues, socket paths longer than 100 bytes are rejected. Systemd support has been added too, but not tested as I don't have access to any Fedora or similar.

    The behaviour of the new command line switch may be strange, but it has been implemented in a way to avoid script breakage when this option is not used.

    The automatic socket file replacement may be a bit surprising too. However, I found that implementing an automatic removal/cleanup on exit was a bit more intrusive.

    Any feedback will be appreciated.

    Reviewed by etanol at 2012-10-20 14:45
  • 2. create a 1.11 release to include the commits since 1.10

    Hi @JensRantil I wondered if it was possible to create a 1.11 release of beanstalkd, to include the commits already in master since 1.10.

    https://github.com/beanstalkd/beanstalkd/compare/v1.10...master

    Reviewed by tomponline at 2018-09-03 16:02
  • 3. Beanstalk not purging old binlogs?

    I know this was previously an issue, that was supposed to be fixed in 1.5, However, we're still seeing this happen.

    We're currently using beanstalk 1.10, and noticed that our binlog directory has grown to over 265GB in usage.

    When looking at beanstalk stats, we have roughly 10K jobs in a delayed state. Even with our maximum job size of 10MB (99.9% of jobs are going to be less than 1 or 2MB each), that only puts us at 100GB of storage.

    It appears as though jobs that have been purged, are not being removed from disk.

    Is there a safe way to clean up after this? How can we prevent it from happening?

    EDITING THIS FOR SOLUTION

    The solution to this was to update to beanstalkd 1.11. This seems to have resolved it for us.

    Reviewed by laszlof at 2019-12-27 02:56
  • 4. heap invariant compromised in connsched

    Function connsched was modifying field tickat while a Conn was still inside the heap. Wacky hijinks ensued.

    Original report follows.


    Reported in version 1.5:

    We are running beanstalkd on Ubuntu. Yesterday we received alerts saying the beanstalkd process stopped and our queues began backing up. This is the only error message I could find. Any idea what it means?

    May 13 11:44:39 bns6 kernel: [4658098.511165] beanstalkd[16793] general protection ip:40978b sp:7fff307194c0 error:0 in beanstalkd[400000+12000]
    

    https://groups.google.com/d/topic/beanstalk-talk/3uRiRBonVuE

    Reviewed by kr at 2012-05-18 01:51
  • 5. Beanstalk does not free memory

    Hi,

    We're using beanstalk in a high-throughput application, in which tens if not hundreds of jobs per second are processed in multiple queues in beanstalkd.

    Inevitably, the beanstalkd process takes over the entire memory of the server it's running on. The rate at which this happens varies, it can take days or, just like now, half an hour.

    This looks an awful lot like issue #54, only that's closed now. Since this behaviour is affecting our application, I would like to readdress the issue.

    I was wondering, has any headway been made with issue #54? Perhaps you can advise me as to how to avoid the problem. Please tell what what other info I can supply.

    Thanks,

    Wim.

    Reviewed by wimrijnders at 2013-09-23 08:05
  • 6. core dump with beanstalkd 1.8

    Tom Guevin [email protected] wrote:

    We recently upgraded to version 1.8 from 1.4.3 and saw this core file under a high load situation:

    # /tmp/gdb /usr/bin/beanstalkd /var/core/12791.beanstalkd.core
    GNU gdb (GDB) 7.1-ubuntu
    Copyright (C) 2010 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "i486-linux-gnu".
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>...
    Reading symbols from /usr/bin/beanstalkd...(no debugging symbols found)...done.
    [New Thread 12791]
    
    warning: Can't read pathname for load map: Input/output error.
    Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found)...done.
    Loaded symbols for /lib/tls/i686/cmov/libc.so.6
    Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
    Loaded symbols for /lib/ld-linux.so.2
    Core was generated by `/usr/bin/beanstalkd'.
    Program terminated with signal 11, Segmentation fault.
    #0  0x080519d2 in h_conn ()
    (gdb) bt
    #0  0x080519d2 in h_conn ()
    #1  0x08051a3e in prothandle ()
    #2  0x08052e88 in srvserve ()
    #3  0x08054a38 in main ()
    

    Any suggestions?

    Reviewed by kr at 2012-12-13 06:54
  • 7. add sunos.c for solaris/illumos support

    I added preliminary solaris/illumos support with event ports in sunos.c as mentioned in #112. The other changes outside sunos.c were related to portability issues between BSD/Linux and SunOS.

    This branch was tested with make bench on the following platforms:

    • SmartOS joyent_20130629T040542Z with GCC 4.7.2
    • CentOS+OpenVZ 2.6.32-042stab076.7 with GCC 4.7.1
    • OS X 10.8.4 with clang 4.2

    Here's the related documentation for event ports:

    Reviewed by potatosalad at 2013-08-12 23:34
  • 8. improve tests coverage

    codecov

    Right now tests coverage is at 32%. What is worse, some files are not tested at all: conn.c, file.c, linux.c, prot.c, walg.c

    https://codecov.io/gh/beanstalkd/beanstalkd/list/master/

    We might want to pick low hanging fruit by adding integration tests that cover wide area.

    Reviewed by ysmolsky at 2019-06-28 11:01
  • 9. beanstalkd is compiled without optimisations

    I noticed beanstalkd is compiled without optimisations. Adding -O3 to CFLAGS gives a new warning that is converted to an error, but otherwise it compiles fine.

    Should optimisations be enabled?

    Reviewed by Minnozz at 2016-07-12 14:54
  • 10. consider adding active maintainers

    @kr, would you consider the idea of adding more maintainers to the project?

    There are lots of requests for a release and there are many outstanding bugs in the beanstalkd. Some use it in production because it's fast and capable, but there are critical bugs for which people has to use workarounds and crutches. Users, me included, love this server and want to make it even better.

    I understand that you are busy and maybe lost interest, but please find some time to pass the torch to somebody who still cares.

    Thank you.

    CC: @JensRantil

    Reviewed by ysmolsky at 2019-06-17 08:29
  • 11. Update protocol.txt

    I've wrapped each command into quotes (except for those which had already been wrapped). So now each command declaration starts with:

    The "<command_name>" command
    
    Reviewed by sergeyklay at 2016-04-05 18:46
  • 12. Error on building beanstalkd 1.12 source from spec file

    Hi ,

    I need beanstalkd 1.12 rpm file , so I am trying to build beanstalkd 1.12 source code from spec provided in source, but on build command i am getting error as- image

    If we remove buildconf.sh , it still gives error for /Configure file not found.

    Please suggest me to solve the build errors

    Reviewed by premagavarawad at 2022-07-01 04:52
  • 13. binlogs continually increase when delaying jobs

    Beanstalk versions: 1.11, 1.12, current master (4c275d5945299e4562389f9f2ca7c326173d6335)

    Steps to reproduce:

    1. Launch beanstalkd with binlogs enabled
    2. Fill a tube with 100k jobs
    3. Continually reserve and release the jobs with a delay
    4. Observe that over time, the binlog size increases

    I would expect that for a fixed size of jobs, the size of the binlogs remains fixed.

    Example perl code:

    Create jobs

    use strict;
    use warnings;
    use Beanstalk::Client;
    
    my $c = Beanstalk::Client->new();
    $c->use('test');
    
    for (1 .. 100_000) {
        $c->put({}, {});
    }
    

    Worker:

    use strict;
    use warnings;
    use Beanstalk::Client;
    
    my $c = Beanstalk::Client->new();
    $c->watch('test');
    
    while (1) {
        my $job = $c->reserve;
        $job->release({ delay => 20 });
    }
    

    Cause:

    I believe it's due to filewrjobshort being called every time the job becomes ready, these Jobrec dumps are never cleaned up until the job is deleted. https://github.com/beanstalkd/beanstalkd/blob/4c275d5945299e4562389f9f2ca7c326173d6335/walg.c#L186

    Reviewed by simon816 at 2022-05-02 21:56
  • 14. Test suite regression

    A change introduced in this commit https://github.com/beanstalkd/beanstalkd/commit/4c275d5945299e4562389f9f2ca7c326173d6335 broke the test suite:

    ct/_ctcheck
    ....................................................................F...............................
    
    cttest_stats_tube: failure
    start server port=38419 pid=284
    >72 use tubea
    <72 USING tubea
    >72 put 0 0 0 1
    >72 x
    <72 INSERTED 1
    >72 delete 1
    <72 DELETED
    >72 stats-tube tubea
    <72 OK 265
    <72 ---
    name: "tubea"
    current-jobs-urgent: 0
    current-jobs-ready: 0
    current-jobs-reserved: 0
    current-jobs-delayed: 0
    current-jobs-buried: 0
    total-jobs: 1
    current-using: 1
    current-watching: 0
    current-waiting: 0
    cmd-delete: 1
    cmd-pause-tube: 0
    pause: 0
    pause-time-left: 0
    
    testserv.c:330: test: strstr(line, sub)
    testserv.c:330: "
    name: tubea
    " not in "---
    name: "tubea"
    current-jobs-urgent: 0
    current-jobs-ready: 0
    current-jobs-reserved: 0
    current-jobs-delayed: 0
    current-jobs-buried: 0
    total-jobs: 1
    current-using: 1
    current-watching: 0
    current-waiting: 0
    cmd-delete: 1
    cmd-pause-tube: 0
    pause: 0
    pause-time-left: 0
    
    "
    
    1 failures; 0 errors.
    make: *** [Makefile:95: check] Error 1
    

    It would probably be worthwhile to clean up the extant Travis configuration and set up a new CI pipeline elsewhere, like GH Actions.

    Reviewed by 1ma at 2022-01-08 12:27
  • 15. Document pipelining and its limitations

    This issue was raised in @266 before where @JensRantil proposed following solution to the problem:

    Before considering this, have you first tested using a client that supports pipelining?

    @meox

    Originally posted by @ysmolsky in https://github.com/beanstalkd/beanstalkd/issues/427#issuecomment-507016097

    We should document somewhere that pipelining is supported and that it has limitations. Currently because of the protocol not using IDs in all responses pipelining can only be reliably implemented for reserve commands. Commands like bury or stats-job don't return an id and thus matching a response with its command is hard. (For stats-job you could look in the data, but that's a hack)

    Reviewed by SamMousa at 2022-01-04 10:06
  • 16. Format for lists and dictionary responses

    I think a somewhat simpler/easier way forward would be to restrict the output format of the existing commands from YAML to JSON. This ought to be a backwards-compatible change, since all YAML parsers are supposed to be able to read JSON. However, it would be worth testing the major client languages to double-check that they actually do the right thing in that case.

    Originally posted by @kr in https://github.com/beanstalkd/beanstalkd/issues/610#issuecomment-1003193205

    I've thought some more on this and I think going either the YAML or the JSON road is troublesome unless you want to pull in a proper library into beanstalkd.

    We could actually simplify the formats as well, since the beanstalkd protocol is a text based protocol and requires a custom parser / interpreter it's not much work to define the lists and dictionaries.

    Lists:

    • All entries are strings
    • Each entry is separate by a \n

    This gives you all you need to know, no parsing and more importantly no escaping required.

    Dictionary:

    • Each entry has a key and a value
    • Since we control the keys we can just put strict definitions on them; keys only consist of the alphabet a-z and the character -.
    • Values should be interpreted as string unless we specify that they are some other type explicitly.
    • Keys and values are separated by a :

    These could have trivial implementations in beanstalkd code and to not break existing clients we could implement these as new commands. For example stats-tubes-2 or stats-2.

    For me this has no priority by the way, I'm happy parsing the YAML in it's current state and since I'm assuming developers create tube names in a reasonable way (not containing user input) it should cause no issues.

    Reviewed by SamMousa at 2022-01-04 08:52
A PHP Library To Make Your Work Work Easier/Faster

This Is A Php Library To Make Your Work Easier/Faster,

Jun 22, 2022
Use php-fpm as a simple built-in async queue

PHP-FPM Async Queue Use php-fpm as a simple built-in async queue. Based on interoperable queue interfaces Queue Interop. Usage composer makasim/php-fp

Feb 1, 2022
Magento 2 Message Queue Open Source Module

Magento 2 Message Queue Module Lightweight implementation of message queue for Magento 2 Community Edition. System requirements This extension support

Sep 30, 2021
Magento 2 Message Queue OS AMQP Broker Implementation

Magento 2 Message Queue AMQP Backend AMQP message queue backend implementation for Rcason_Mq. Installation Require the module via Composer $ composer

Nov 7, 2020
Queue Management Systems for LPG vendor agencies of Sri Lanka, for the LPG shortages in 2022

gas-queue-mgt Queue Management Systems for LPG vendor agencies of Sri Lanka, for the LPG shortages in 2022 Installation Requirements PHP 7.4 or later

Mar 30, 2022
The swiss army knife for Magento developers, sysadmins and devops. The tool provides a huge set of well tested command line commands which save hours of work time. All commands are extendable by a module API.

netz98 magerun CLI tools for Magento 2 The n98 magerun cli tools provides some handy tools to work with Magento from command line. Build Status Latest

Jun 24, 2022
Easily create and work with code snippets from PHP

code-snippets Easily create and work with code snippets from source code files of any type in PHP. The original code this package is based on was borr

Jun 2, 2022
"Whatever you do, work at it with all your heart." - Apostole Paul, probably, maybe.

Apostolos Whatever you do, work at it with all your heart. - Apostole Paul Also remember: Sigmund Freud says in Civilization and Its Discontents that

Jan 14, 2022
Make Laravel and Storyblok work together beautifully.

Use Storyblok’s amazing headless CMS in way that feels familiar to Laravel developers This package allows you to use fantastic Storyblok headless CMS

Mar 14, 2022
PHP 7+ Payment processing library. It offers everything you need to work with payments: Credit card & offsite purchasing, subscriptions, payouts etc. - provided by Forma-Pro

Supporting Payum Payum is an MIT-licensed open source project with its ongoing development made possible entirely by the support of community and our

Jun 28, 2022
When the player is killed and dies, the sound will work

KillDeathSound | v1.0.0 ✔️ When the player is killed and dies, the sound will work ✔️ ☝ Features When the player is killed and dies, the sound will wo

Apr 23, 2022
This composer plugin is a temporary implementation of using symbolic links to local packages as dependencies to allow a parallel work process

Composer symlinker A Composer plugin to install packages as local symbolic links. This plugin is a temporary implementation of using symbolic links to

Nov 9, 2021
Firebird-PHP: A library created to meet a work need when handling a Firebird database
Firebird-PHP: A library created to meet a work need when handling a Firebird database

Firebird-PHP: A library created to meet a work need when handling a Firebird database

Jun 27, 2022
Laminas\Text is a component to work on text strings

laminas-text This package is considered feature-complete, and is now in security-only maintenance mode, following a decision by the Technical Steering

Mar 8, 2022
Joole Reflector - used to work with the properties of objects, their changes and merges

Joole Reflector allows you to change protected, as well as private properties of an object.

May 7, 2022
GDPR compliant TYPO3 content elements which work great with PIWIK Consent Manager.
GDPR compliant TYPO3 content elements which work great with PIWIK Consent Manager.

PIWIK Consent Manager TYPO3 extension PIWIK Consent Manager integration in order to make TYPO3 content elements GDPR compliant. You can click on the i

Jun 21, 2022
Opinionated version of Wikimedia composer-merge-plugin to work in pair with Bamarni composer-bin-plugin.

Composer Inheritance Plugin Opinionated version of Wikimedia composer-merge-plugin to work in pair with bamarni/composer-bin-plugin. Usage If you are

Apr 27, 2022
A simple and fast pmmp js es6 promise virion

Promises A simple and fast pmmp js es6 promise virion Api usage these examples are in base injectable php these should be easily transferable. Example

Dec 14, 2021
Simple, fast, isolated way to download+assimilate ZIP/TAR files

Composer Downloads Plugin The "Downloads" plugin allows you to download extra files (*.zip or *.tar.gz) and extract them within your package. This is

May 20, 2021