Exception: Log events in a single PutLogEvents request must be in chronological order

[timkelty]

Got the following Aws\CloudWatchLogs\Exception\CloudWatchLogsException:

"Log events in a single PutLogEvents request must be in chronological order."

[timkelty]

Similar issue: maxbanton/cwh#70

[kaushik-manish]

How did you solve this issue? @timkelty

[timkelty]

@manish020195 I bailed on this package, and used maxbandon/cwh directly :)

Here's an example of my log compontent (Craft is an instance of Yii):

        'log' => [
            'targets' => [
                function () {
                    $logger = new \Monolog\Logger('craftcms');
                    $logger->pushHandler(new \Monolog\Handler\StreamHandler('php://stderr', \Monolog\Logger::WARNING));

                    if (!Craft::$app->getRequest()->isConsoleRequest) {
                        $logger->pushHandler(new \Monolog\Handler\StreamHandler('php://stdout', \Monolog\Logger::DEBUG));
                    }

                    return Craft::createObject([
                        'class' => \samdark\log\PsrTarget::class,
                        'except' => ['yii\web\HttpException:40*'],
                        'logVars' => [],
                        'logger' => $logger,
                    ]);
                },
                function () {
                    $handler = (new \ReflectionClass(\Maxbanton\Cwh\Handler\CloudWatch::class))->newInstanceArgs([
                        'client' => new \Aws\CloudWatchLogs\CloudWatchLogsClient([
                            'region' => getenv('AWS_DEFAULT_REGION'),
                            'version' => 'latest',
                            'credentials' => [
                                'key' => getenv('AWS_ACCESS_KEY_ID'),
                                'secret' => getenv('AWS_SECRET_ACCESS_KEY'),
                            ]
                        ]),
                        'group' => '/web/' . CRAFT_ENVIRONMENT,
                        'stream' => @file_get_contents("http://instance-data/latest/meta-data/instance-id") ?: gethostname(),
                        'retention' => 14,
                        'batchSize' => 10000,
                        'tags' => [],
                        'level' => \Monolog\Logger::NOTICE,
                        'bubble' => true,
                    ])->setFormatter(new \Monolog\Formatter\JsonFormatter());

                    return Craft::createObject([
                        'class' => \samdark\log\PsrTarget::class,
                        'except' => ['yii\web\HttpException:40*'],
                        'logger' => (new \Monolog\Logger('craftcms'))->pushHandler($handler),
                        'enabled' => CRAFT_ENVIRONMENT !== 'local',
                    ]);
                }
            ]
        ]

This give me stdout stream logging (for docker logs), and AWS cloudwatch logs remotely.

[devarda]

Here's a working solution:

1. No need to install this package, you can remove it

2. Put this in components/AwsLogTarget.php file. This is the PR fix sequence order #5 (fixed version of this composer package):

<?php
namespace app\components;

use yii\log\Target as BaseTarget;
use yii\base\InvalidConfigException;
use Aws\CloudWatchLogs\CloudWatchLogsClient;
use yii\log\Logger;
use yii\helpers\VarDumper;

class AwsLogTarget extends BaseTarget
{
    /**
     * @var string The name of the log group.
     */
    public $logGroup;

    /**
     * @var string The AWS region to use e.g. eu-west-1.
     */
    public $region;

    /**
     * @var string Your AWS access key.
     */
    public $key;

    /**
     * @var string The name of the log stream. When not set, we try to get the ID of your EC2 instance.
     */
    public $logStream;

    /**
     * @var string Your AWS secret.
     */
    public $secret;

    /**
     * @var CloudWatchLogsClient
     */
    private $client;

    /**
     * @var string
     */
    private $sequenceToken;

    /**
     * @inheritdoc
     */
    public function init()
    {
        if (empty($this->logGroup)) {
            throw new InvalidConfigException("A log group must be set.");
        }

        if (empty($this->region)) {
            throw new InvalidConfigException("The AWS region must be set.");
        }

        if (empty($this->logStream)) {
            if (empty($this->key)) {
                $instanceId = @file_get_contents("http://instance-data/latest/meta-data/instance-id");
                if ($instanceId !== false) {
                    $this->logStream = $instanceId;
                } else {
                    throw new InvalidConfigException("Cannot identify instance ID and no log stream name is set.");
                }
            } else {
                throw new InvalidConfigException("No log stream name is set.");
            }
        }

        $params = [
            'region' => $this->region,
            'version' => 'latest',
        ];

        if (!empty($this->key) && !empty($this->secret)) {
            $params['credentials'] = [
                'key' => $this->key,
                'secret' => $this->secret,
            ];
        }

        $this->client = new CloudWatchLogsClient($params);
    }

    /**
     * @inheritdoc
     */
    public function export()
    {
        $this->ensureLogGroupExists();

        $this->refreshSequenceToken();

        $data = [
            'logEvents' => array_map([$this, 'formatMessage'], $this->messages),
            'logGroupName' => $this->logGroup,
            'logStreamName' => $this->logStream,
        ];

        if (!empty($this->sequenceToken)) {
            $data['sequenceToken'] = $this->sequenceToken;
        }
        $logEvents = $data['logEvents'];

        usort($logEvents, function (array $a, array $b) {
            return $a['timestamp'] > $b['timestamp'] ? 1 : -1;
        });
        $data['logEvents'] = $logEvents;

        $response = $this->client->putLogEvents($data);

        $this->sequenceToken = $response->get('nextSequenceToken');
    }

    /**
     * @inheritdoc
     */
    public function formatMessage($message)
    {
        list($text, $level, $category, $timestamp) = $message;
        $level = Logger::getLevelName($level);
        if (!is_string($text)) {
            // exceptions may not be serializable if in the call stack somewhere is a Closure
            if ($text instanceof \Throwable || $text instanceof \Exception) {
                $text = (string) $text;
            } else {
                $text = VarDumper::export($text);
            }
        }
        $traces = [];
        if (isset($message[4])) {
            foreach ($message[4] as $trace) {
                $traces[] = "in {$trace['file']}:{$trace['line']}";
            }
        }

        $prefix = $this->getMessagePrefix($message);

        return [
            'timestamp' => $timestamp*1000,
            'message' => "{$prefix}[$level][$category] $text" . (empty($traces) ? '' : "\n    " . implode("\n    ", $traces))
        ];
    }

    /**
     * Get the sequence token for the selected log stream.
     *
     * @return void
     */
    private function refreshSequenceToken()
    {
        $existingStreams = $this->client->describeLogStreams([
            'logGroupName' => $this->logGroup,
            'logStreamNamePrefix' => $this->logStream,
        ])->get('logStreams');

        $exists = false;

        foreach ($existingStreams as $stream) {
            if ($stream['logStreamName'] === $this->logStream) {
                $exists = true;
                if (isset($stream['uploadSequenceToken'])) {
                    $this->sequenceToken = $stream['uploadSequenceToken'];
                }
            }
        }

        if (!$exists) {
            $this->client->createLogStream([
                'logGroupName' => $this->logGroup,
                'logStreamName' => $this->logStream,
            ]);
        }
    }

    /**
     * Ensures that the selected log group exists or create it
     *
     * @return void
     */
    private function ensureLogGroupExists()
    {
        $existingGroups = $this->client->describeLogGroups([
            'logGroupNamePrefix' => $this->logGroup,
        ])->get('logGroups');

        $exists = false;

        foreach ($existingGroups as $group) {
            if ($group['logGroupName'] === $this->logGroup) {
                $exists = true;
            }
        }

        if (!$exists) {
            $this->client->createLogGroup([
                'logGroupName' => $this->logGroup,
            ]);
        }
    }
}

3. Go to your log settings. Mine is at config/params.php and add this in log settings. Make sure to adjust the keys and such.:

	'log' => [
			'traceLevel' => YII_DEBUG ? 3 : 0,
			'targets' => [
                [
                    'class' => 'app\components\AwsLogTarget',
                    'region' => 'XXX',
                    'logGroup' => 'XXX',
                    'logStream' => 'XXX', // omit for automatic instance ID
                    'levels' => ['error', 'warning'],
                    'except' => ['yii\web\HttpException:404'],
                    'logVars' => ['_GET', '_POST', '_FILES', '_COOKIE', '_SESSION', '_SERVER'],
                    'key' => 'XXX', // omit for instance role
                    'secret' => 'XXX', // omit for instance role
                ],
				 
			],
		]