DiaryFolio

 AWS Lambda: Function to stream logs via SQS Summary As part of logging and monitoring strategy it is quite important to emit data from AWS services to another service or system or aggregation layer. AWS SQS (Amazon Simple Queue Service) is a great tool to communicate between such micro-services with real-time & between software components at any volume. Aim of this article is a snippet to stream logs via SQS to an external service (like logstash or siem tools) using Lambda Function Pre-Reqs Permission to pull data from specific SQS queue Steps Ensure lambda function can reach the SQS queue Below is a snippet of code to push the data in real-time import gzip import json import base64 import boto3 import time def lambda_handler(event, context): sqs = boto3.client('sqs') account = boto3.client('sts').get_caller_identity()['Account'] queue_url = "https://sqs.eu-west-1.amazonaws.com/12345567928/my-app-{}.fifo".format(account) cw_da