What is SQS??
SQS stands for Simple Queue Service.
SQS was the first service available in AWS.
Amazon SQS is a web service that gives you access to a message queue that can be used to store messages while waiting for a computer to process them.
Amazon SQS is a distributed queue system that enables web service applications to quickly and reliably queue messages that one component in the application generates to be consumed by another component where a queue is a temporary repository for messages that are awaiting processing.
With the help of SQS, you can send, store and receive messages between software components at any volume without losing messages.
Using Amazon sqs, you can separate the components of an application so that they can run independently, easing message management between components.
Any component of a distributed application can store the messages in the queue.Messages can contain up to 256 KB of text in any format such as json, xml, etc.
Any component of an application can later retrieve the messages programmatically using the Amazon SQS API.
The queue acts as a buffer between the component producing and saving data, and the component receives the data for processing. This means that the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer is only intermittently connected to the network.
If you got two EC2 instances which are pulling the SQS Queue. You can configure the autoscaling group if a number of messages go over a certain limit. Suppose the number of messages exceeds 10, then you can add additional EC2 instance to process the job faster. In this way, SQS provides elasticity.
Let’s understand through an example.
Let’s look at a website that generates a Meme. Suppose the user wants to upload a photo and wants to convert into Meme. User uploads a photo on a website and website might store a photo in s3. As soon as it finished uploads, it triggers a Lambda function. Lambda analyzes the data about this particular image to SQS, and this data can be “what the top of the meme should say”, “what the bottom of the meme should say”, the location of the S3 bucket, etc. The data sits inside the SQS as a message. An EC2 instance looks at the message and performs its job. An EC2 instance creates a Meme and stores it in S3 bucket. Once the EC2 instance completed its job, it moves back to the SQS. The best thing is that if you lose your EC2 instance, then also you would not lose the job as the job sits inside the S3 bucket.
Let’s look at another example of SQS, i.e., Travel Website.
Suppose the user wants to look for a package holiday and wants to look at the best possible flight. A User types a query in a browser, it then hits the EC2 instance. An EC2 instance looks “What the user is looking for?”, it then puts the message in a queue to the SQS. An EC2 instance pulls queue. An EC2 instance continuously pulling the queue and looking for the jobs to do. Once it gets the job, it then processes it. It interrogates the Airline service to get all the best possible flights. It sends the result to the web server, and the web server sends back the result to the user. A User then selects the best flight according to his or her budget.
If we didn’t have SQS, then what happened?
A web server passes the information to an application server and then application server queried an Airline service. If an Application server crashes, then a user loses its query. One of the great thing about SQS is that data is queued in the SQS even if the application server crashes, the message in the queue is marked as an invisible in a timeout interval window. When the timeout runs out, message reappears in the queue; then a new EC2 instance can use this message to perform its job. Therefore, we can say that SQS removes the application server dependency.
Queue Types
There are two types of Queue:
*Standard Queues (default)
*FIFO Queues (First-In-First-Out)
Standard Queue
SQS Visibility Timeout
The visibility timeout is the amount of time that the message is invisible in the SQS Queue after a reader picks up that message.
If the provided job is processed before the visibility time out expires, the message will then be deleted from the Queue. If the job is not processed within that time, the message will become visible again and another reader will process it. This could result in the same message being delivered twice.
The Default Visibility Timeout is 30 seconds.Visibility Timeout can be increased if your task takes more than 30 seconds.
The maximum Visibility Timeout is 12 hours.
Important points to remember:
*SQS is pull-based, not push-based.
*Messages are 256 KB in size.
*Messages are kept in a queue from 1 minute to 14 days.
*The default retention period is 4 days.
*It guarantees that your messages will be processed at least once.
redBus Case Study
redBus is an Indian travel agency that specializes in bus travel throughout India by selling bus tickets throughout the country. Tickets are purchased through the company’s Website or through the Web services of its agents and partners. The company also offers software, on a Software as a Service (SaaS) basis, which gives bus operators the option of handling their own ticketing and managing their own inventories. To date, the company says they have sold over 30 million bus tickets and has more than 1750 bus operators using the software to manage their operations.
The Challenge
The company previously ran its operations from a traditional data center by purchasing and renting its systems and infrastructure. In addition to the expense, several logistical problems evolved from this arrangement. The biggest problem was that the infrastructure could not effectively handle processing fluctuations, which had a negative impact on productivity. Additionally, the procurement of servers or upgrading the server configuration was an extremely time-consuming endeavor. Over time, redBus realized that a better solution was imperative — a solution that offered scalability to handle the company’s processing fluctuations. redBus looked to Amazon Web Services (AWS) for a solution.
Why Amazon Web Services??
After testing the AWS solution on a small application for several months, the travel agency determined that it was very workable and convenient. Although redBus was quite enthusiastic about the on-demand instances and variety of instance types, several other features cemented the company’s decision to migrate completely to AWS. These features included the ability to easily manage access to servers through security groups, the easy-to-use, self-service management console, the concept of Elastic IPs, and superior support.
The company has incorporated many of the AWS products into its solution, including Amazon Elastic Compute Cloud (Amazon EC2),Elastic Load Balancing , Amazon Relational Database Service(Amazon RDS), Amazon Simple Storage Service (Amazon S3),Amazon Elastic Block Storage(Amazon EBS), and Amazon CloudWatch. Charan Padmaraju, Chief Technology Officer believes that “with features like Elastic Load Balancing and multiple availability zones, AWS provides the required infrastructure to build for redundancy and auto-failover. When you incorporate these in your system/application design, you can achieve high reliability and scale.”
The Benefits:
Since migrating to AWS, redBus has seen measurable improvements in the bottom line. Padmaraju says, “By scaling up and down dynamically based on the load, we maintain performance as well as minimize cost. With the time savings that the IT and development staffs obtain from the AWS solution, AWS gives us an overall cost benefit of about 30–40%.” He adds, “By hosting at [the AWS Asia Pacific (Singapore) region], redBus.in gained significantly in terms of website performance by way of reduced latency (about 4x). This is a great advantage when the customers are from India.”
Of the many excellent characteristics of AWS, perhaps the most significant to redBus is the ability to “instantly replicate the whole setup on demand for testing by creating and destroying instances on demand for experimentation, thereby reducing the time to market.” Less time to market translates to increased profitability and success.
The travel agency anticipates expanding the AWS solution to include Amazon Simple Notification Service (Amazon SNS) and Amazon Simple queue Service (Amazon SQS) for monitoring, alerts, and intercommunication. “Amazon SQS is an especially good solution for enabling messaging between external applications and our applications,” says Padmaraju.
Since joining forces with AWS, redBus has gained the freedom to experiment on new solutions and applications at minimal cost, increased the efficiency of its operations, and improved its profitability.
