Luke Curtis

Luke Curtis

Software Developer and Project Manager

Contact Me

Rate Limit your Laravel Redis Queue

07/02/2019 (4 months ago) | Luke Curtis

I recently came across a problem whereby an API service only allowed us to send 2 requests to their service every second. However the client was, at times, approving over 50 of these requests in a single second

Before we continue, I am going to assume you are comfortable using Laravel Queues and also using Redis as your queue driver

A not uncommon scenario and I'm sure many people come up with all kinds of ways to solve the problem, from using the PHP own sleep command to even creating an intermediary table and batch processing from there. Which are all solutions that although work, are pretty awkward and purposefully slow down your application.

The solution to this problem was to create a rate limited throttle request in my job. What I will mention is you must be using Redis as your queue driver for this to work (and ideally Laravel Horizon for a better indication of what's happening). The reason for this simply down to Redis being able to use atomic operations as opposed to the usual Sync driver

TL;DR: Check out the gist here

Take for example the following code:

The customer query

The customer query

Here we are getting all customers that currently are not linked to MYOB - which an accounting software for tracking invoices and the like. For each of them we are dispatching a Job called CreateMyobCustomer

Now we can go ahead and check out what happens in the actual Job (note I am using my own package called Laramyob - currently WIP - to interact with MYOB).

Create the MYOB customer for the customer

Create the MYOB customer for the customer

Here's where things get tricky, we define a throttle key called myob which is what Redis will use to match against to determine if the closure should run.

Next we determine how many jobs under this key are 'allow'ed to run every specified interval under the 'every' key, in MYOB's case it's 2 requests per 1 second. Once we've done this we can save the request, and continue as normal.

The second part of the 'then' function allows you to handle a callback should the request have failed and you can 'release' that job to be attempted later if appropriate.

Edit 20/06/19: Thanks David Lloyd for the correction regarding Redis atomic operations