This post is the handout for the SQS Lab I ran as a part of hands-on AWS training to the Engineering + DevOps team at Marqeta.

In this lab we will use AWS CLI to work with SQS. I’ve replaced the account ID with 111111111111.

Prepare the CLI (aws-shell)

In the previous lab we configured aws-shell on our laptops with an access key, secret, and default region. However, the user only had AmazonS3FullAccess via a managed policy, and AmazonEC2FullAccess via the developers group.

We will fire up the aws-shell and use basic SQS commands. Autocomplete feature of the shell will come in very handy as you’ll see.

First, we will try to list the queues we have in our account.

aws> sqs list-queues

An error occurred (AccessDenied) when calling the ListQueues operation: Access to the resource https://queue.amazonaws.com/ is denied.
aws>

Since the access key and secret we’re using is not tied to SQS policy, we go ahead and add that to the developers group. It is called AmazonSQSFullAccess.

Once done, you should see an empty list for the same command.

aws> sqs list-queues
aws>

Next, lets create a queue.

aws> sqs create-queue --queue-name my_test_queue
{
    "QueueUrl": "https://queue.amazonaws.com/111111111111/my_test_queue"
}
aws>

We can go on the console in SQS section, and see this queue being created. You will notice that the Queue URL on the console is a bit richer, with region in there as well.

https://sqs.us-east-1.amazonaws.com/111111111111/my_test_queue Now that we’ve created a queue, we will go ahead and send a message to it. As you do this, you will notice that aws-shell will autocomplete the queue-url for us.

aws> sqs send-message --queue-url https://queue.amazonaws.com/111111111111/my_test_queue --message-body "Hello!"
{
    "MD5OfMessageBody": "952d2c56d0485958336747bcdd98590d",
    "MessageId": "31e40b96-c61c-4b6a-8682-67ebb7badc36"
}
aws>

The max message size is 256KB, with a default visibility timeout of 30 seconds. This means whoever consumes this message has 30 seconds to process it, otherwise it will become available to other consumers for processing, i.e. back in the queue. Lets give that a try.

aws> sqs receive-message --queue-url https://queue.amazonaws.com/111111111111/my_test_queue
{
    "Messages": [
        {
            "Body": "Hello!",
            "ReceiptHandle": "AQEBD9ziSgXsnzTdBNzjNDksLpLU/ASfSdxdeFT1MkzCezLdnKL9HljPpsRaz97eSdjRCm27c3/iNcaITv2PEFgpNs62zezc4jpzWh3ysPYVwVgBjqbhobVQiF8JBRxuHSfqgQ8qSsgF2QOpU2Bn6qE/wC/LSqwxNqd/aQRab5EUA9zgqHx9L15FIlbmdGrz8Qz6ebPyFi4Lr5gb+S79t+wN9s0yE9+kEQvKjeweF0OHRyGeBWybwEFVlc3qPeo0QT9WLdkQ68PKBCY8s7djSJoD7npv9j2YeFS4+EcNzJ9ql5x2wpzcLOF672exXivigcjGMUdP9JT32Xju61ixWYnIDgd7KXWo12f+VgSc7/s143kZGPNwpkicvoNRlmzx/urI",
            "MD5OfBody": "952d2c56d0485958336747bcdd98590d",
            "MessageId": "31e40b96-c61c-4b6a-8682-67ebb7badc36"
        }
    ]
}
aws>

We immediately fire the same command, and notice nothing comes back.

aws> sqs receive-message --queue-url https://queue.amazonaws.com/111111111111/my_test_queue
aws>

This is due to the visibility timeout of 30 seconds. Lets wait 30 seconds, and you’ll see the same message comes back on the queue.

aws> sqs receive-message --queue-url https://queue.amazonaws.com/111111111111/my_test_queue
{
    "Messages": [
        {
            "Body": "Hello!",
            "ReceiptHandle": "AQEBU5MiEmzKdpwJ705Ehe6brf5fmkx0sa+m2ibZxbVsmdsKiHX+RyDu4s+T2yBNP9MS7NjSmEA9l/dLgwn55cN6qDCTCxxOrvVEp1MVMqximPCjzFgkmYTYR5w32TBqamDxKqadAGPUgjLyLlvD/M5tc+MuyvrFL2WaWovK7cpMRk5VGlW+pSaC+TNfhtt2QeLpNW4GUnLxtAu96BrY54ll3jelHtt3jnuAqNOaJ6nzDwKvfonyrpdd4sVS+cKPo4LzS47kJyX8p/gZdi/7cY5Oty+1RhfEkfmkEUKm2Eg4DK8Rm+E9jyUjYK+VmxWc/0JAeKwGRkIuZEuYW1kWav7y3/PkJNaXHJvVQtyLDQiSy8zf8wrjyoyGknaonmUIHtyP",
            "MD5OfBody": "952d2c56d0485958336747bcdd98590d",
            "MessageId": "31e40b96-c61c-4b6a-8682-67ebb7badc36"
        }
    ]
}
aws>

This is where the application developers have to pick a right visibility timeout, which should allow the consumer enough time to process the message and delete it from the queue, so multiple processing does not occur. Let us delete the message after processing (reading) it.

aws> sqs delete-message --queue-url https://queue.amazonaws.com/111111111111/my_test_queue --receipt-handle "AQEBHh4KsuIv0QABOD49rbmmu6V0r0I34ENhrQqT5me0pPMfM2KOBWX+NvqPVV8eaB3T7zujcJj0qgQlvvqbzgCMKeKYSlExbVgLs6tFHKZgR6wUVH/X6RhN579u4LHFo3SBu9IExDpz226sEoVAn0MOhOlvArU72xJQm+pNPwRjEicQxu6MQPakfiq8yHGICYSUPyBhz+/OwxLx4iDe8exWY2Esvz2zt1q7kD/u2PqzPXiL0uIBsW18tyslMfftGSEV3cB19HFJEHNJ9olZtstNMrX1MAGJZqDHxS1rllSCrNCiqD4264Ou/eyf7duPwhvNT97mmUS0NbCuhTX5P3Kt4BytNFwKY3Qu13vkgb1rciO/1NNifwNbJ+gjOI7xS246"
aws>

Notice that we need to provide the receipt-handle that we got after receiving the message in order to delete it.

Next we can go on the console and look at cloudwatch monitoring. We will also create an alarm to notify us if a message has not been picked up for N minutes. This is something that can alert us of dead consumers, or an issue with processing the messages.

More on visibility timeout

In order to change visibility timeout, which can be done when the consumer does not want the message to go back in the queue, like it wants to retry processing the message itself instead of passing it on to another consumer, it can be done by providing the queue-url and the receipt-handle like so - note that visibility-timeout is specified in seconds.

Lets push a message, consume it, and then increase the visibility-timeout to 2 minutes (120 seconds).

aws> sqs send-message --queue-url https://queue.amazonaws.com/111111111111/my_test_queue --message-body "Gonna take long"
{
    "MD5OfMessageBody": "de80b2f443dc0a15382a40a625b74d3a",
    "MessageId": "8efcd759-eb0f-48a3-92fe-5b83e0dd8740"
}
aws> sqs receive-message --queue-url https://queue.amazonaws.com/111111111111/my_test_queue
{
    "Messages": [
        {
            "Body": "Gonna take long",
            "ReceiptHandle": "AQEBYkUuIUjSEnzSI7ymjk/oRB0K9osJ2Nyud1+pij4nVAJ8taZ+FHq1kvWIm4LGrjZEVveMPOjsxP/nmmaUCEXAZKYQHATnQDT2Hqt+NJiMQG+EFd6HBTXtT5gH29UN/ieAOGHAV6zWR2IJH+h4eRA8U7tgciatz5P/KGrHgH1qmxe9EoSkXOulikG82cs22gymNBeqNe+TZiNgwH6Kfd6CwTcCh8oowk1YXMqXaVCBso1n1R7JtMkDWd4R1CnEVCtTt1ZCfbVgHrKADPFkuwUYIp0lEXXYeDD5GqvJBaR3+INFW2t7qEMb0uzPr1s9n42W2uMEC7u45dqYss9mWlNhrI7+YG6IxCL0jx7VxIIrkOj1ihsHgwj+bxMCH7Eiu+87",
            "MD5OfBody": "de80b2f443dc0a15382a40a625b74d3a",
            "MessageId": "8efcd759-eb0f-48a3-92fe-5b83e0dd8740"
        }
    ]
}

aws> sqs change-message-visibility --queue-url https://queue.amazonaws.com/111111111111/my_test_queue --receipt-handle "AQEBYkUuIUjSEnzSI7ymjk/oRB0K9osJ2Nyud1+pij4nVAJ8taZ+FHq1kvWIm4LGrjZEVveMPOjsxP/nmmaUCEXAZKYQHATnQDT2Hqt+NJiMQG+EFd6HBTXtT5gH29UN/ieAOGHAV6zWR2IJH+h4eRA8U7tgciatz5P/KGrHgH1qmxe9EoSkXOulikG82cs22gymNBeqNe+TZiNgwH6Kfd6CwTcCh8oowk1YXMqXaVCBso1n1R7JtMkDWd4R1CnEVCtTt1ZCfbVgHrKADPFkuwUYIp0lEXXYeDD5GqvJBaR3+INFW2t7qEMb0uzPr1s9n42W2uMEC7u45dqYss9mWlNhrI7+YG6IxCL0jx7VxIIrkOj1ihsHgwj+bxMCH7Eiu+87" --visibility-timeout 120

Now even if we try to get message after 30 seconds, this message will not show up. However, it will after 120 seconds unless we delete it. Remember that this value is absolute, which means it changes the timeout from default (30 seconds) to 120 seconds and not to 30+120 seconds.