vasupcalifornia.blogg.se

#REDIS WINDOWS CLIENT INSTALL#

In PHP this might look like this: // composer require predis/predis We do this because this is being used in a distributed context (we have many workers performing API calls), and we don't want to write when we already exceeded our limits. Also note that, in the same transaction as reading the list of timestamps, we add a new timestamp to the list (the ZADD command). It serves as the key for our Redis sorted set. Instead of using the actual OAuth access tokens (and duplicating them to Redis), you might want to use an identifier or hash of the token instead as $accessToken. EXEC will execute all previously queued commands and restore the connection state to normal.EXPIRE $accessToken $slidingWindow to reset the expiry date for this sorted set of timestamps (for the current OAuth Token).ZADD $accessToken $now $now to add a log for the current API call that we're about to do.ZRANGE $accessToken 0 -1 to get a list of all API call timestamps that happened during the window.ZREMRANGEBYSCORE $accessToken 0 ($now - $slidingWindow) to remove API call timestamps that were done before the start of the window.

Subsequent commands will be queued for atomic execution using EXEC.

MULTI to mark the start of a transaction block.

> ZREMRANGEBYSCORE $accessToken 0 ($now - $slidingWindow) We're going to use a sorted set to hold our timestamps because it fits our needs.

#REDIS WINDOWS CLIENT INSTALL#

If you're on a mac, you can just $ brew install redis to get it. Assuming you already have Redis installed, start a server: $ redis-server.

We're ready to translate our algorithm to Redis commands. Redis is a key/value store that supports lists, sets, sorted sets and more. What about Redis? Yes, I like where this is going. Sadly Memcached doesn't have the concept of an array or list (we could serialize an array using our favourite programming language). What about Memcached? Yes, that's also an option.

Maybe we could use MySQL or PostgreSQL? Yes, we could but then we would need a system that periodically removes outdated timestamps since neither MySQL or PostgreSQL allow us to set a time to life on a row. Now we need some kind of database to store a list of timestamps grouped per access token. Let's say we did 4413 API calls in the last hour then we're allowed to do 587 more at this moment.

How many calls did we do in the last hour?ĬallsInTheLastHour = timestamps.filter(timestamp => timestamp > now - slidingWindow) Īfter we calculated the amount of API calls we did in the last hour we can calculate the remaining API calls: remaining = maxCallsPerHour - count.

When we're about to do an API call in our crawler we need to check if we're allowed to do one: In pseudocode: timestamps.push(Date.now()) Every time we do an API call we add the timestamp (in microseconds) of the API call to a list. The sliding window is an hour in this case. We're allowed to do 5000 API calls per access token each hour.Įvery point on the axis represents an API call. We could of course perform our HTTP requests without checking rate limits upfront, and wait until we get a 429 OAuthRateLimitException from Instagram, but that would exhaust our tokens and block us from talking efficiently to their API.Īll rate limits on the Instagram Platform are controlled separately for each access token, and on a sliding 1-hour window. For our Instagram crawler we needed a system to keep track of the amount of API calls we did to prevent us from hitting the rate limits.