Bridge to use Symfony Messenger on AWS Lambda with Bref.
This bridge allows messages to be dispatched to SQS, SNS or EventBridge, while workers handle those messages on AWS Lambda.
Installation
This guide assumes that:
- Symfony is installed
- Symfony Messenger is installed
- Bref is installed and configured to deploy Symfony
First, install this package:
composer require bref/symfony-messenger
Next, register the bundle in config/bundles.php:
return [
// ...
Bref\Symfony\Messenger\BrefMessengerBundle::class => ['all' => true],
];
SQS, SNS and EventBridge can now be used with Symfony Messenger.
Usage
Symfony Messenger dispatches messages. To create a message, follow the Symfony Messenger documentation.
To configure where messages are dispatched, all the examples in this documentation are based on the example from the Symfony documentation:
# config/packages/messenger.yaml
framework:
messenger:
transports:
async: '%env(MESSENGER_TRANSPORT_DSN)%'
routing:
'App\Message\MyMessage': async
SQS
The SQS service is a queue that works similar to RabbitMQ. To use it, set its URL in the environment variable MESSENGER_TRANSPORT_DSN:
MESSENGER_TRANSPORT_DSN=https://sqs.us-east-1.amazonaws.com/123456789/my-queue
That's it, messages will be dispatched to that queue.
The implementation uses the SQS transport provided by Symfony Amazon SQS Messenger, so all those features are supported. If you already use that transport, the transition to AWS Lamdba is very easy and should not require any change for dispatching messages.
Create the SQS queue
You can create the Queue yourself in the Console, write custom Cloudformation or use Lift's Queue construct that will handle that for you.
Here is a simple example with Lift, make sure to install the plugin first and check out the full documentation for more details.
# serverless.yml
service: my-app
provider:
name: aws
environment:
...
MESSENGER_TRANSPORT_DSN: ${construct:jobs.queueUrl}
constructs:
jobs:
type: queue
worker:
handler: bin/consumer.php
timeout: 20 # in seconds
reservedConcurrency: 5 # max. 5 messages processed in parallel
layers:
- ${bref:layer.php-80}
plugins:
- serverless-lift
In all cases, you would want to disable auto_setup to avoid extra requests and permission issues.
# config/packages/messenger.yaml
framework:
messenger:
transports:
async:
dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
options:
auto_setup: false
Add permissions
When running Symfony on AWS Lambda, it is not necessary to configure credentials. The AWS client will read them from environment variables automatically.
You just have to provide the correct role statements in serverless.yml and Lambda will take care of the rest. The required IAM permission to publish to SQS using Messenger is sqs:SendMessage on the given queue.
If you use Lift, this is done automatically for you.
Consume messages from SQS
- If you don't use Lift, create the function that will be invoked by SQS in
serverless.yml:
functions:
worker:
handler: bin/consumer.php
timeout: 20 # in seconds
reservedConcurrency: 5 # max. 5 messages processed in parallel
layers:
- ${bref:layer.php-80}
events:
# Read more at https://www.serverless.com/framework/docs/providers/aws/events/sqs/
- sqs:
arn: arn:aws:sqs:us-east-1:1234567890:my_sqs_queue
# Only 1 item at a time to simplify error handling
batchSize: 1
- Create the handler script (for example
bin/consumer.php):
<?php declare(strict_types=1);
use Bref\Symfony\Messenger\Service\Sqs\SqsConsumer;
require dirname(__DIR__) . '/config/bootstrap.php';
$kernel = new \App\Kernel($_SERVER['APP_ENV'], (bool) $_SERVER['APP_DEBUG']);
$kernel->boot();
// Return the Bref consumer service
return $kernel->getContainer()->get(SqsConsumer::class);
If you are using Symfony 5.1, use this instead:
<?php declare(strict_types=1);
use Bref\Symfony\Messenger\Service\Sqs\SqsConsumer;
use Symfony\Component\Dotenv\Dotenv;
require dirname(__DIR__).'/vendor/autoload.php';
(new Dotenv())->bootEnv(dirname(__DIR__).'/.env');
$kernel = new \App\Kernel($_SERVER['APP_ENV'], (bool)$_SERVER['APP_DEBUG']);
$kernel->boot();
// Return the Bref consumer service
return $kernel->getContainer()->get(SqsConsumer::class);
- Register and configure the
SqsConsumerservice:
# config/services.yaml
services:
Bref\Symfony\Messenger\Service\Sqs\SqsConsumer:
public: true
autowire: true
arguments:
# Pass the transport name used in config/packages/messenger.yaml
$transportName: 'async'
Now, anytime a message is dispatched to SQS, the Lambda function will be called. The Bref consumer class will put back the message into Symfony Messenger to be processed.
FIFO Queue
The FIFO queue guarantees exactly once delivery, and has a mandatory queue name suffix .fifo:
# config/packages/messenger.yaml
framework:
messenger:
transports:
async:
dsn: 'https://sqs.us-east-1.amazonaws.com/123456789/my-queue.fifo'
# serverless.yml
resources:
Resources:
Queue:
Type: AWS::SQS::Queue
Properties:
QueueName: my-queue.fifo
FifoQueue: true
Symfony Amazon SQS Messenger will automatically calculate/set the MessageGroupId and MessageDeduplicationId parameters required for FIFO queues, but you can set them explicitly:
use Symfony\Component\Messenger\MessageBus;
use Symfony\Component\Messenger\Bridge\AmazonSqs\Transport\AmazonSqsFifoStamp;
/* @var MessageBus $messageBus */
$messageBus->dispatch(new MyAsyncMessage(), [new AmazonSqsFifoStamp('my-group-message-id', 'my-deduplication-id')]);
Everything else is identical to the normal SQS queue.
SNS
AWS SNS is "notification" instead of "queues". Messages may not arrive in the same order as sent, and they might arrive all at once. To use it, create a SNS topic and set it as the DSN:
MESSENGER_TRANSPORT_DSN=sns://arn:aws:sns:us-east-1:1234567890:foobar
That's it, messages will be dispatched to that topic.
Note: when running Symfony on AWS Lambda, it is not necessary to configure credentials. The AWS client will read them from environment variables automatically.
To consume messages from SNS:
- Create the function that will be invoked by SNS in
serverless.yml:
functions:
worker:
handler: bin/consumer.php
timeout: 20 # in seconds
reservedConcurrency: 5 # max. 5 messages processed in parallel
layers:
- ${bref:layer.php-80}
events:
# Read more at https://www.serverless.com/framework/docs/providers/aws/events/sns/
- sns:
arn: arn:aws:sns:us-east-1:1234567890:my_sns_topic
- Create the handler script (for example
bin/consumer.php):
<?php declare(strict_types=1);
use Bref\Symfony\Messenger\Service\Sns\SnsConsumer;
require dirname(__DIR__) . '/config/bootstrap.php';
$kernel = new \App\Kernel($_SERVER['APP_ENV'], (bool) $_SERVER['APP_DEBUG']);
$kernel->boot();
// Return the Bref consumer service
return $kernel->getContainer()->get(SnsConsumer::class);
- Register and configure the
SnsConsumerservice:
# config/services.yaml
services:
Bref\Symfony\Messenger\Service\Sns\SnsConsumer:
public: true
autowire: true
arguments:
# Pass the transport name used in config/packages/messenger.yaml
$transportName: 'async'
Now, anytime a message is dispatched to SNS, the Lambda function will be called. The Bref consumer class will put back the message into Symfony Messenger to be processed.
EventBridge
AWS EventBridge is a message routing service. It is similar to SNS, but more powerful. To use it, configure the DSN like so:
# "myapp" is the EventBridge "source", i.e. a namespace for your application's messages
# This source name will be reused in `serverless.yml` later.
MESSENGER_TRANSPORT_DSN=eventbridge://myapp
Optionally you can add set the EventBusName via a event_bus_name query parameter, either the name or the ARN:
MESSENGER_TRANSPORT_DSN=eventbridge://myapp?event_bus_name=custom-bus
MESSENGER_TRANSPORT_DSN=eventbridge://myapp?event_bus_name=arn:aws:events:us-east-1:123456780912:event-bus/custom-bus
That's it, messages will be dispatched to EventBridge.
Note: when running Symfony on AWS Lambda, it is not necessary to configure credentials. The AWS client will read them from environment variables automatically.
To consume messages from EventBridge:
- Create the function that will be invoked by EventBridge in
serverless.yml:
functions:
worker:
handler: bin/consumer.php
timeout: 20 # in seconds
reservedConcurrency: 5 # max. 5 messages processed in parallel
layers:
- ${bref:layer.php-80}
events:
# Read more at https://www.serverless.com/framework/docs/providers/aws/events/event-bridge/
- eventBridge:
# This filters events we listen to: only events from the "myapp" source.
# This should be the same source defined in config/packages/messenger.yaml
pattern:
source:
- myapp
- Create the handler script (for example
bin/consumer.php):
<?php declare(strict_types=1);
use Bref\Symfony\Messenger\Service\EventBridge\EventBridgeConsumer;
require dirname(__DIR__) . '/config/bootstrap.php';
$kernel = new \App\Kernel($_SERVER['APP_ENV'], (bool) $_SERVER['APP_DEBUG']);
$kernel->boot();
// Return the Bref consumer service
return $kernel->getContainer()->get(EventBridgeConsumer::class);
- Register and configure the
EventBridgeConsumerservice:
# config/services.yaml
services:
Bref\Symfony\Messenger\Service\EventBridge\EventBridgeConsumer:
public: true
autowire: true
arguments:
# Pass the transport name used in config/packages/messenger.yaml
$transportName: 'async'
Now, anytime a message is dispatched to EventBridge for that source, the Lambda function will be called. The Bref consumer class will put back the message into Symfony Messenger to be processed.
Error handling
AWS Lambda has error handling mechanisms (retrying and handling failed messages). Because of that, this package does not integrates Symfony Messenger's retry mechanism. Instead, it works with Lambda's retry mechanism.
This section is work in progress, feel free to contribute to improve it.
When a message fails with SQS, by default it will go back to the SQS queue. It will be retried until the message expires. Here is an example to setup retries and "dead letter queue" with SQS:
# serverless.yml
resources:
Resources:
Queue:
Type: AWS::SQS::Queue
Properties:
# This needs to be at least 6 times the lambda function's timeout
# See https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
VisibilityTimeout: '960'
RedrivePolicy:
deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn
# Jobs will be retried 5 times
# The number needs to be at least 5 per https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
maxReceiveCount: 5
# The dead letter queue is a SQS queue that receives messages that failed to be processed
DeadLetterQueue:
Type: AWS::SQS::Queue
Properties:
# Messages are stored up to 14 days (the max)
MessageRetentionPeriod: 1209600
When using SNS and EventBridge, messages will be retried by default 2 times.
Configuration
Configuring AWS clients
By default, AWS clients (SQS, SNS, EventBridge) are preconfigured to work on AWS Lambda (thanks to environment variables populated by AWS Lambda).
However, it is possible customize the AWS clients, for example to use them outside of AWS Lambda (locally, on EC2…) or to mock them in tests. These clients are registered as Symfony services under the keys:
bref.messenger.sqs_clientbref.messenger.sns_clientbref.messenger.eventbridge_client
For example to customize the SQS client:
services:
bref.messenger.sqs_client:
class: AsyncAws\Sqs\SqsClient
public: true # the AWS clients must be public
arguments:
# Apply your own config here
-
region: us-east-1
Disabling transports
By default, this package registers Symfony Messenger transports for SQS, SNS and EventBridge.
If you want to disable some transports (for example in case of conflict), you can remove BrefMessengerBundle from config/bundles.php and reconfigure the transports you want in your application's config. Take a look at Resources/config/services.yaml to copy the part that you want.
Customizing the serializer
If you want to change how messages are serialized, for example to use Happyr message serializer, you need to add the serializer on both the transport and the consumer. For example:
# config/packages/messenger.yaml
framework:
messenger:
transports:
async:
dsn: 'https://sqs.us-east-1.amazonaws.com/123456789/my-queue'
serializer: 'Happyr\MessageSerializer\Serializer'
# config/services.yaml
services:
Bref\Symfony\Messenger\Service\Sqs\SqsConsumer:
public: true
autowire: true
arguments:
$transportName: 'async'
$serializer: '@Happyr\MessageSerializer\Serializer'
319 Nov 30, 2022
0 Jul 13, 2022
2 Jun 19, 2022
22 Feb 28, 2022
13 Jan 29, 2022
1 Oct 29, 2021
3 Aug 21, 2022
1 Jan 23, 2022
15 Nov 9, 2022
160 Dec 2, 2022
1 Mar 25, 2022
48 Sep 1, 2021
9 Jan 28, 2022
218 Sep 25, 2022
5 Apr 6, 2017
143 Nov 26, 2022
11 Apr 26, 2022
28 Nov 15, 2022