SQS ¶

Tldr

Using Amazon SQS, you can send, store, and receive messages between software components, without losing messages or requiring other services to be available. In Amazon SQS, an application sends messages into a queue.

A user or service retrieves a message from the queue, processes it, and then deletes it from the queue (as its a pull-based system).

SQS pull-based architecture

SQS will guarantee that a message is delivered at least once, but that message may be redelivered.

SQS queues only make an “attempt” to deliver messages in order (more or less a FIFO approach) but do not guarantee FIFO. If strict FIFO is needed, that option can be selected.

SNS vs SQS

SNS manages notifications and SQS manages messages
SNS is a push-based system while SQS is a pull-based system

Long/Short Polling¶

Amazon SQS offers short and long polling options for receiving messages from a queue. Consider your application's requirements for responsiveness and cost efficiency when choosing between these two polling options:

Short polling (default) – The ReceiveMessage request queries a subset of servers (based on a weighted random distribution) to find available messages and sends an immediate response, even if no messages are found.

The following diagram shows the short-polling behavior of messages returned from a standard queue after one of your system components makes a receive request. Amazon SQS samples several of its servers (in gray) and returns messages A, C, D, and B from these servers. Message E isn't returned for this request, but is returned for a subsequent request.

Long polling – ReceiveMessage queries all servers for messages, sending a response once at least one message is available, up to the specified maximum. An empty response is sent only if the polling wait time expires. This option can reduce the number of empty responses and potentially lower costs.

Types of SQS¶

Standard (Default)¶

Amazon SQS offers standard as the default queue type. Standard queues support a nearly unlimited number of API calls per second, per API action (SendMessage, ReceiveMessage, or DeleteMessage). Standard queues support at-least-once message delivery.

Messages delivered out of order

However, occasionally (because of the highly distributed architecture that allows nearly unlimited throughput), more than one copy of a message might be delivered out-of-order. Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they're sent.

FIFO¶

FIFO (First-In-First-Out) queues have all the capabilities of the standard queues, but are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated.

There are some key distinctions between SNS and SQS:

SNS is a pub/sub system, while SQS is a queueing system. You'd typically use SNS to send the same message to multiple consumers via topics. In comparison, in most scenarios, each message in an SQS queue is processed by only one consumer.
With SQS, messages are delivered through a long polling (pull) mechanism, while SNS uses a push mechanism to immediately deliver(push) messages to subscribed endpoints.
SNS is typically used for applications that need realtime notifications, while SQS is more suited for message processing use cases.
SNS does not persist messages - it delivers them to subscribers that are present, and then deletes them. In comparison, SQS can persist messages (from 1 minute to 14 days).

Features¶

DLQ¶

Visibility Timeout¶

Best Practices¶

It is a best practice to set the visibility timeout on your SQS queue to 6x the Lambda function timeout to allow for retries if the function is getting throttled.
Retries – Use the maxReceiveCount on the queue's policy to limit the number of times Lambda will retry to process a failed execution.