How Instagram Hundreds of Millions simple key-value in Redis – It was 4 years ago post but the trick is still valid. The trick here is to leverage redis hash (ie. dictionary) to construct buckets that each holds around 1000 2nd level keys. This way you can cut the first level keys by 1000 folds. Also the memory footprint of the dictionary is quite compact that eventually drops the memory usage from 21G to 5G. The 1000 bucket size is like a magic number according to the test result. If you increase the size further, it will increase your CPU activity.
How Hootsuite switch to Redis for session management – This article talked about how Hootsuite switched to use Redis for session management from Memcache + MySQL setup. They were at ~5000 operations per second and it was well below a single tuned Redis instance can handle. What I like about this article is that it detailed the steps of their migration process. The “dark launch” process that allows them to taste the power of Redis while helping them to identify the set-up issues in a safe way. They faced “packet drop” network issue due to too many network traffics and tcp open connections. It was resolved by using connection pool in Twemproxy. They also needed to turn off the logging of Twemproxy due to disk IO speed on AWS not able to catch up.
Near realtime analytics
Light Speed Analytics With Redis Bitmaps And Lua – This article showed you how you can use Redis Bitmap for unique user calculation over ad hoc date range. Interesting way to use Redis for fast analytics. The memory usage for 128 million of users will cost around you 16 MB in memory. If you keep who visits your site per day for 128 million users, you will need cost to 0.5G per month and 6G for an year. Even your site luckily reach 10x up to 1B of users, you can still fit all unique visitors for an year in a single node’s memory. You can certainly create monthly/ year aggregate like this for time longer than 1 year. So, it is a sound solution. The interesting thing you can do is to produce 1 bitmap per user data that models as 0 or 1 in value like sex then you can use bit operation to get unique males for a date range. Pretty cool!