Quick AWS SQS SNS
Posted on October 2, 2021
Tags: aws
1 Broad Summary
- SQS: ONE-TO-ONE bridge between lambda + service/db/s3
- Just throw it in front of everything if cost allows
- SNS: FANOUT pubsub
2 SQS
- SQS
- Functionality
- lambda poll Queue: by configuring SQS lambda trigger
- lambda push Queue: using boto3
- Options
- unOrdered Queue
- FIFO Queue
- Functionality
Max message size is 256 KB
Typically middlemans one-to-one, NOT FANOUT
2.1 Mechanism
- Producer Send Msg to SQS
- SQS Send Msg to Consumer
- Consumer initiate Polling 4a. Consumer Process Msg 4b. Consumer Delete Msg
Message Details:
ID Size MD5 of message body
ABC-CDE-FE32-EG12-ZZZZZZZZZZ 9 bytes 4901d08f14cacf11650dadaac7e22067
Sender account ID Sent First received
ABCDEFFFFFFFFFFFFFFF 7/8/2022, 12:26:26 EDT 7/8/2022, 12:26:50 EDT
Receive count Message attributes count Message attributes size MD5 of message attributes
2 - - -
- Note that it is possible to poll for message but not delete it. This will make the *Receive count” go up.
2.2 Common pattern
- A very common pattern is throwing SQS infront of a lambda or db or microservice.(really just throw it in front of anything important)
- Retrys if fail
- If message processing lambda needs to be taken down for service, means messages can be left in queue to be later processed.
- Loosen tight dependencies by giving an agnostic message queue between services
2.3 Examples
2.3.1 Ex1
import json
import boto3
def lambda_handler(event, context):
# TODO implement
= boto3.client('sqs')
sqs
sqs.send_message(= "https://sqs.us-east-1.amazonaws.com/637189393063/myqueue",
QueueUrl = "hi"
MessageBody
)return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
3 Counter example: SQS vs SNS
- Imagine client order message needs to be processed by Accounting Service AND Analytic Service
- SQS :(BAD) Only one (Either Accounting or Analytic Service ) wins out and takes the message
- SNS :(GOOD) Both service sub to the topic and can process the message
3.1 Usecase
REMEMBER: How you throw SQS in-front of anything important?
- SQS-SNS fanout pattern
- SNS is typically used in conjunction with SQS
- SQS fans out to multiple SNS queues infront of individual microservices
- This way when one microservice goes down, data isn’t lost and instead kept in queue