I integrated your code in my Lumen project that is essentially Queues that do asynchronous jobs on the server. Since Lumen (or Laravel) keeps launching them repeatedly, your code isn't uploading the log buffer to AWS. Nothing shows up. I downgraded to version 0.1.1 and it works. The buffer mecanism isn't working for Laravel.

Hi,

I am using your cloud watch handler for symfony. I rolled out the cwh in multiple environments (dev, staging, demo, prod, .. ) and are now experiencing something odd after a couple of successfull pushes:

  [Aws\CloudWatchLogs\Exception\CloudWatchLogsException]
  Error executing "PutLogEvents" on "https://logs.eu-central-1.amazonaws.com"; AWS HTTP error: Client error response [url] https://logs.eu-central-
  1.amazonaws.com [status code] 400 [reason phrase] Bad Request InvalidSequenceTokenException (client): The given sequenceToken is invalid. The nex
  t expected sequenceToken is: 49578481559872967104658335510065862421249324983057912274 - {"__type":"InvalidSequenceTokenException","expectedSequen
  ceToken":"49578481559872967104658335510065862421249324983057912274","message":"The given sequenceToken is invalid. The next expected sequenceToke
  n is: 49578481559872967104658335510065862421249324983057912274"}

This is my configuration:

        class: Maxbanton\Cwh\Handler\CloudWatch
        arguments:
            - "@cloudwatch_client"
            - "%elasticbeanstalk_app%"              # groupName
            - "%kernel.environment%" # streamName
            - 30                     # retentionDays
            - 5                  # logsInBatch
            - { mytag: "tag" }       # tags
            - NOTICE                # logLevel

Is it maybe an issue, that I chose batchsize of 5?

It is neccesary to use use different access key / secret for different environment?

Hello folks,

On a Symfony application, after I enabled this library in my production environment I started to get a lot of Exceptions such as the one below:

[2018-02-26 11:44:55] prod.EMERGENCY: Message: Error executing "PutLogEvents" on "https://logs.us-west-2.amazonaws.com"; AWS HTTP error: Client error: POST https://logs.us-west-2.amazonaws.com resulted in a 400 Bad Request response: {"__type":"InvalidParameterException","message":"Log events in a single PutLogEvents request must be in chronological or (truncated...) InvalidParameterException (client): Log events in a single PutLogEvents request must be in chronological order. - {"__type":"InvalidParameterException","message":"Log events in a single PutLogEvents request must be in chronological order."} []

Below is my current configuration:

services:
    monolog.formatter.api:
        class: Monolog\Formatter\LineFormatter
        arguments:
            - "[%%extra.route%%] [%%extra.server_hostname%%] [%%extra.request%%] %%level_name%%: %%message%% %%context%% %%extra%%\n"

    cloudwatchlogs.api:
        class: Maxbanton\Cwh\Handler\CloudWatch
        arguments:
            - '@aws.cloudwatchlogs'
            - '/path'
            - 'api'
            - 30
            - 10000
            - { mytag: 'api' }
            - DEBUG
        calls:
            - ['setFormatter', ['@monolog.formatter.api']]

monolog:
    handlers:
        api:
            type:         service
            id:           cloudwatchlogs.api
            formatter:    ~

Any ideas on why would this happen? One of my channels generates a lot of DEBUG messages but I presume this wouldn't really be a problem?

Thank you in advance.

Renato.

Hi there,

We would like to use this package in our Symfony project. Is there a chance that you have an example configuration for Symfony?

Many thanks!

Hi, thanks for writing this monolog handler, it's been really useful. We've hit one issue with it - Cloudwatch logs has a limit of 5 log events/second.

When our application exceeds this limit it causes an exception to be thrown:

Exception in run: Error executing "PutLogEvents" on "https://logs.us-east-1.amazonaws.com"; AWS HTTP error: Client error: `POST https://logs.us-east-1.amazonaws.com` resulted in a `400 Bad Request` response:
{
    "__type": "ThrottlingException",
    "message": "Rate exceeded"
}
ThrottlingException (client): Rate exceeded -
{
    "__type": "ThrottlingException",
    "message": "Rate exceeded"
}
[] []

Other cloudwatch agents get around this by internally buffering log events and then periodically sending batches of to the PutLogEvents API. It would be great if this handler could do the same.

Justin.

Lumen version: 5.6 package version: v1.1.14

Recently I started getting this error while running my lumen application: The batch of log events in a single PutLogEvents request cannot span more than 24 hours.

Eexception 'Aws\CloudWatchLogs\Exception\CloudWatchLogsException' with message 'Error executing "PutLogEvents" on "https://logs.us-east-1.amazonaws.com"; AWS HTTP error: Client error: `POST https://logs.us-east-1.amazonaws.com` resulted in a `400 Bad Request` response:
{"__type":"InvalidParameterException","message":"The batch of log events in a single PutLogEvents request cannot span mo (truncated...)
 InvalidParameterException (client): The batch of log events in a single PutLogEvents request cannot span more than 24 hours. - {"__type":"InvalidParameterException","message":"The batch of log events in a single PutLogEvents request cannot span more than 24 hours."}'

GuzzleHttp\Exception\ClientException: Client error: `POST https://logs.us-east-1.amazonaws.com` resulted in a `400 Bad Request` response:
{"__type":"InvalidParameterException","message":"The batch of log events in a single PutLogEvents request cannot span mo (truncated...)
 in /application/path/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113
Stack trace:
...

Is anyone familiar with this type of error? Does this error mean the log stream was pushing logs for more than 24 hours? Can this package handle log streams that old?

Thanks for your help.

• What kind of change does this PR introduce? (Bug fix, feature, docs update, ...) feature

• What is the current behavior? (You can also link to an open issue here) The handler checks for existence of the given log group, if it doesn't exist, it created the log group

• What is the new behavior (if this is a feature change)? By enabling the option, the handler doesn't check for the group's existence and goes on with initializing the stream.

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?) No, existing behavior remains unchanged. User has to explicitly trigger the option

• Other information: This change was made in an effort to reduce the security boundaries of the application (I create the log group beforehand and don't need the app to have logs:DescribeLogGroups and logs:CreateLogGroup permissions. Also, not checking if the group exists improves the performance by removing the HTTP call.

Please let me know what you think about it.

Alternatively, I propose to make the initialize and refreshSequenceToken methods protected, because currently there's no way to achieve my change by extending the class, so copy-pasting the whole class is the only solution.

I am currently researching whether it is possible to asynchronously send over log entries as a "send and forget" approach. The rationale of this is that during profiling with Blackfire.io, it turned out that dispatching the logs (and waiting for the response) currently takes around 25% of the time of a complete request cycle. Instead of reducing / disabling dispatching log entries, I thought maybe use a fire and forget approach... and just hope the entries arrive at Amazon🤞 .

Realizing this seems to be possible by using CurlMultiHandler as HTTP Handler of the CloudWatchLogClient and manually invoking the tick() as described here:

https://github.com/aws/aws-sdk-php/issues/621#issuecomment-111232418

In that case instead of $this->client->putLogEvents($data);, we should invoke $this->client->putLogEventsAsync($data);

@maxbanton have you already considered this?

Whether this is completely possible is still something which seems debatable when looking at https://github.com/guzzle/guzzle/issues/1127 https://github.com/guzzle/guzzle/issues/1425 https://github.com/guzzle/guzzle/issues/1429#issuecomment-197152914

I already tried to do this for the AwsCloudWatch client when sending over metrics. But it seems the request is still being "blocked". That might be covered by https://github.com/guzzle/guzzle/pull/1924/files

Another, 'simpler' way might be to set the response timeout really low: https://github.com/guzzle/guzzle/issues/1429#issuecomment-304449743

Describe the bug I am trying to use this package in my laravel application. My steps:

1. I created new group in IAM and added CloudwatchFullAccess policy
2. I created a user and added it to this group
3. After this I got Access key ID and Secret Access key
4. I installed package according to package documentation:

• composer require maxbanton/cwh:^2.0
• next I added cloudwatch setings to my logging.php like here:

        'cloudwatch' => [
            'driver' => 'custom',
            'via' => \App\Logging\CloudWatchLoggerFactory::class,
            'sdk' => [
                'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
                'version' => 'latest',
                'credentials' => [
                    'key' => env('CLOUDWATCH_LOG_KEY'),
                    'secret' => env('CLOUDWATCH_LOG_SECRET')
                ]
            ],
            'retention' => env('CLOUDWATCH_LOG_RETENTION',7),
            'level' => env('CLOUDWATCH_LOG_LEVEL','info'),
            'stream' => env('CLOUDWATCH_STREAM', 'backend-log'),
        ],

• created factory class here: App/Logging/CloudWatchLoggerFactory.php

<?php

namespace App\Logging;

use Aws\CloudWatchLogs\CloudWatchLogsClient;
use Maxbanton\Cwh\Handler\CloudWatch;
use Monolog\Logger;

class CloudWatchLoggerFactory
{
    /**
     * Create a custom Monolog instance.
     *
     * @param  array  $config
     * @return \Monolog\Logger
     */
    public function __invoke(array $config)
    {
        $sdkParams = $config["sdk"];
        $tags = $config["tags"] ?? [ ];
        $name = $config["name"] ?? 'cloudwatch';

        // Instantiate AWS SDK CloudWatch Logs Client
        $client = new CloudWatchLogsClient($sdkParams);

        // Log group name, will be created if none
        $groupName = config('app.name') . '-' . config('app.env');

        // Log stream name, will be created if none
        $streamName = config('app.hostname');

        // Days to keep logs, 14 by default. Set to `null` to allow indefinite retention.
        $retentionDays = $config["retention"];

        // Instantiate handler (tags are optional)
        $handler = new CloudWatch($client, $groupName, $streamName, $retentionDays, 10000, $tags);

        // Create a log channel
        $logger = new Logger($name);
        // Set handler
        $logger->pushHandler($handler);

        return $logger;
    }
}

• Next I used this in my index controller: Log::info('Test CloudWatch');
• added LOG_CHANNEL=cloudwatchto my .env I get no errors, but no data is transferred to the Cloudwatch. Also a Log group is created on Cloudwatch side, but the log streams is empty. What I did wrong?

Expected behavior

1. create new group in IAM and added CloudwatchFullAccess policy
2. create new user and add into this group
3. install package according to package documentation
4. switch to cloudwatch channel in your .env
5. when you run Log::info() in your code logging data should be transmitted to Cloudwatch

Please tell about your environment: Laravel Version: v7.19.1 PHP Version: v7.2.24 Ubuntu 18.04.3 LTS

• Application mode: Web app

• What kind of change does this PR introduce? feature

• What is the current behavior? not supporting monolog

• What is the new behavior (if this is a feature change)? it will support monolog 2.0

• Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?) yes, thei will need to update application with monolog 2.0

• Other information:

• I'm submitting a ...

  • [ ] bug report
  • [ ] feature request
  • [x] support request

• Do you want to request a feature or report a bug?

• What is the current behavior?

• If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

• What is the expected behavior?

• What is the motivation / use case for changing the behavior?

• Please tell about your environment:

  • PHP Version:
  • Operating system (distro):
  • Application mode (web app / cli app / daemon cli app):

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links to have context, eg. stackoverflow, etc)

Getting Instances of Aws\CloudWatchLogs\CloudWatchLogsClient cannot be serialized Error File: /var/www/newhtml/breathe-v2/vendor/aws/aws-sdk-php/src/AwsClient.php

Hi,

Infra guy here, not familliar with php, symfony, composer ,etc... Recently, I encountered a "Throttling Exeption: Rate exceeded" in our php legacy code that use this library with version v1.1.1.

My question is about the fix introduced in the v1.1.4. Does the sleep will pause the execution of the HTTP request ? Or is this library handled in another thread or in an asynchronous way and will log when it can without pausing the execution of the request ?

Our legacy application use Symfony 2.8 with php 7.2. It's not possible to update these two.

Thanks in advance.

Describe the bug Support for Monolog v3 has been added in the composer file in v2.0.3: https://github.com/maxbanton/cwh/pull/106

However it seems like the methods are not compatible with the Monolog v3 interfaces: Fatal error: Declaration of Maxbanton\Cwh\Handler\CloudWatch::write(array $record): void must be compatible with Monolog\Handler\AbstractProcessingHandler::write(Monolog\LogRecord $record): void in .../vendor/maxbanton/cwh/src/Handler/CloudWatch.php on line 164

Please tell about your environment:

• PHP Version: 8.1.12
• Operating system (distro): debian
• Application mode:
  • [X] Web app
  • [ ] CLI app
  • [ ] Daemon worker

When using this component with static analyzers (such as phpstan) we are not able to pass null as retention because the parameter is marked as int. This fixes it.

Is your feature request related to a problem? Please describe. The documentation today is nice and comprehensive as long as you're using the standard credential provider (ENV variables / etc.). However when placed into an ECS task using IAM Task Roles for credential and authorization it seems that the CloudwatchLogsClient doesn't seem to pick the credentials up as expected. As far as I can see, this is by no fault of this library.

Failing to provide credentials in other instances of the PHP SDK (at least in our other implementations) simply "works" with the SDK but for some reason CW likes to fail with the following error:

Credentials must be an instance of Aws\\Credentials\\CredentialsInterface, an associative array that contains \"key\", \"secret\", and an optional \"token\" key-value pairs, a credentials provider function, or false."

Of course, being tricksy you could help manually resolve this scenario by leveraging the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI env / etc. but we found that longer queue workers would eventually end up failing with cached tokens that would expire before the job executed (might be solved in a newer version of Laravel - we're on 6).

For our solution, we leveraged this documentation from AWS and their examples, I was able to resolve issues with being unable to push CW logs after a certain period of time:

$provider = CredentialProvider::ecsCredentials();

$sdkParams = [
    'region' => env('AWS_DEFAULT_REGION', 'us-west-2'),
    'version' => 'latest',
    'credentials' => CredentialProvider::memoize($provider)
];

$client = new CloudWatchLogsClient($sdkParams);

Describe the solution you'd like Include this information in the README

Describe alternatives you've considered Provide an alternative that is better suited?