Add the posibility of accepting connections from a local socket (bound to a filesystem path).

To minimize portability issues, socket paths longer than 100 bytes are rejected. Systemd support has been added too, but not tested as I don't have access to any Fedora or similar.

The behaviour of the new command line switch may be strange, but it has been implemented in a way to avoid script breakage when this option is not used.

The automatic socket file replacement may be a bit surprising too. However, I found that implementing an automatic removal/cleanup on exit was a bit more intrusive.

Any feedback will be appreciated.

Hi @JensRantil I wondered if it was possible to create a 1.11 release of beanstalkd, to include the commits already in master since 1.10.

https://github.com/beanstalkd/beanstalkd/compare/v1.10...master

I know this was previously an issue, that was supposed to be fixed in 1.5, However, we're still seeing this happen.

We're currently using beanstalk 1.10, and noticed that our binlog directory has grown to over 265GB in usage.

When looking at beanstalk stats, we have roughly 10K jobs in a delayed state. Even with our maximum job size of 10MB (99.9% of jobs are going to be less than 1 or 2MB each), that only puts us at 100GB of storage.

It appears as though jobs that have been purged, are not being removed from disk.

Is there a safe way to clean up after this? How can we prevent it from happening?

EDITING THIS FOR SOLUTION

The solution to this was to update to beanstalkd 1.11. This seems to have resolved it for us.

Function connsched was modifying field tickat while a Conn was still inside the heap. Wacky hijinks ensued.

Original report follows.

Reported in version 1.5:

We are running beanstalkd on Ubuntu. Yesterday we received alerts saying the beanstalkd process stopped and our queues began backing up. This is the only error message I could find. Any idea what it means?

May 13 11:44:39 bns6 kernel: [4658098.511165] beanstalkd[16793] general protection ip:40978b sp:7fff307194c0 error:0 in beanstalkd[400000+12000]

https://groups.google.com/d/topic/beanstalk-talk/3uRiRBonVuE

Hi,

We're using beanstalk in a high-throughput application, in which tens if not hundreds of jobs per second are processed in multiple queues in beanstalkd.

Inevitably, the beanstalkd process takes over the entire memory of the server it's running on. The rate at which this happens varies, it can take days or, just like now, half an hour.

This looks an awful lot like issue #54, only that's closed now. Since this behaviour is affecting our application, I would like to readdress the issue.

I was wondering, has any headway been made with issue #54? Perhaps you can advise me as to how to avoid the problem. Please tell what what other info I can supply.

Thanks,

Wim.

Tom Guevin [email protected] wrote:

We recently upgraded to version 1.8 from 1.4.3 and saw this core file under a high load situation:

# /tmp/gdb /usr/bin/beanstalkd /var/core/12791.beanstalkd.core
GNU gdb (GDB) 7.1-ubuntu
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/beanstalkd...(no debugging symbols found)...done.
[New Thread 12791]

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/tls/i686/cmov/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Core was generated by `/usr/bin/beanstalkd'.
Program terminated with signal 11, Segmentation fault.
#0  0x080519d2 in h_conn ()
(gdb) bt
#0  0x080519d2 in h_conn ()
#1  0x08051a3e in prothandle ()
#2  0x08052e88 in srvserve ()
#3  0x08054a38 in main ()

Any suggestions?

I added preliminary solaris/illumos support with event ports in sunos.c as mentioned in #112. The other changes outside sunos.c were related to portability issues between BSD/Linux and SunOS.

This branch was tested with make bench on the following platforms:

• SmartOS joyent_20130629T040542Z with GCC 4.7.2
• CentOS+OpenVZ 2.6.32-042stab076.7 with GCC 4.7.1
• OS X 10.8.4 with clang 4.2

Here's the related documentation for event ports:

• port_create(3c)
• port_associate(3c)
• port_getn(3c)
• poll(2)

Right now tests coverage is at 32%. What is worse, some files are not tested at all: conn.c, file.c, linux.c, prot.c, walg.c

https://codecov.io/gh/beanstalkd/beanstalkd/list/master/

We might want to pick low hanging fruit by adding integration tests that cover wide area.

I noticed beanstalkd is compiled without optimisations. Adding -O3 to CFLAGS gives a new warning that is converted to an error, but otherwise it compiles fine.

Should optimisations be enabled?

@kr, would you consider the idea of adding more maintainers to the project?

There are lots of requests for a release and there are many outstanding bugs in the beanstalkd. Some use it in production because it's fast and capable, but there are critical bugs for which people has to use workarounds and crutches. Users, me included, love this server and want to make it even better.

I understand that you are busy and maybe lost interest, but please find some time to pass the torch to somebody who still cares.

Thank you.

CC: @JensRantil

I've wrapped each command into quotes (except for those which had already been wrapped). So now each command declaration starts with:

The "<command_name>" command

Hello there!

I have been testing coolbeans but it does not have all the commands implemented for beanstalkd, so using that project is not an option yet, and I was wondering which is the best way to replicate the data between beanstalkd servers.

The idea is to have a Haproxy in front of it and only send the the traffic to one available beanstalkd, and if it fails, then send it to the next one, like (Redis + Sentinel + Haproxy) pattern, but the problem is that, to be able to do it replication need to be done, at least pasive replication. So my question is, do you have some idea or best practise to do this with beanstalkd?

Thank you in advance!

Hi ,

I need beanstalkd 1.12 rpm file , so I am trying to build beanstalkd 1.12 source code from spec provided in source, but on build command i am getting error as-

If we remove buildconf.sh , it still gives error for /Configure file not found.

Please suggest me to solve the build errors

Beanstalk versions: 1.11, 1.12, current master (4c275d5945299e4562389f9f2ca7c326173d6335)

Steps to reproduce:

1. Launch beanstalkd with binlogs enabled
2. Fill a tube with 100k jobs
3. Continually reserve and release the jobs with a delay
4. Observe that over time, the binlog size increases

I would expect that for a fixed size of jobs, the size of the binlogs remains fixed.

Example perl code:

Create jobs

use strict;
use warnings;
use Beanstalk::Client;

my $c = Beanstalk::Client->new();
$c->use('test');

for (1 .. 100_000) {
    $c->put({}, {});
}

Worker:

use strict;
use warnings;
use Beanstalk::Client;

my $c = Beanstalk::Client->new();
$c->watch('test');

while (1) {
    my $job = $c->reserve;
    $job->release({ delay => 20 });
}

Cause:

I believe it's due to filewrjobshort being called every time the job becomes ready, these Jobrec dumps are never cleaned up until the job is deleted. https://github.com/beanstalkd/beanstalkd/blob/4c275d5945299e4562389f9f2ca7c326173d6335/walg.c#L186

This issue was raised in @266 before where @JensRantil proposed following solution to the problem:

Before considering this, have you first tested using a client that supports pipelining?

@meox

Originally posted by @ysmolsky in https://github.com/beanstalkd/beanstalkd/issues/427#issuecomment-507016097

We should document somewhere that pipelining is supported and that it has limitations. Currently because of the protocol not using IDs in all responses pipelining can only be reliably implemented for reserve commands. Commands like bury or stats-job don't return an id and thus matching a response with its command is hard. (For stats-job you could look in the data, but that's a hack)

I think a somewhat simpler/easier way forward would be to restrict the output format of the existing commands from YAML to JSON. This ought to be a backwards-compatible change, since all YAML parsers are supposed to be able to read JSON. However, it would be worth testing the major client languages to double-check that they actually do the right thing in that case.

Originally posted by @kr in https://github.com/beanstalkd/beanstalkd/issues/610#issuecomment-1003193205

I've thought some more on this and I think going either the YAML or the JSON road is troublesome unless you want to pull in a proper library into beanstalkd.

We could actually simplify the formats as well, since the beanstalkd protocol is a text based protocol and requires a custom parser / interpreter it's not much work to define the lists and dictionaries.

Lists:

• All entries are strings
• Each entry is separate by a \n

This gives you all you need to know, no parsing and more importantly no escaping required.

Dictionary:

• Each entry has a key and a value
• Since we control the keys we can just put strict definitions on them; keys only consist of the alphabet a-z and the character -.
• Values should be interpreted as string unless we specify that they are some other type explicitly.
• Keys and values are separated by a :

These could have trivial implementations in beanstalkd code and to not break existing clients we could implement these as new commands. For example stats-tubes-2 or stats-2.

For me this has no priority by the way, I'm happy parsing the YAML in it's current state and since I'm assuming developers create tube names in a reasonable way (not containing user input) it should cause no issues.