Getting Started: Redis and Python

Posted: March 22nd, 2010 | Author: | Filed under: NoSQL, programming, Python, Redis | 25 Comments »

So if you have been following NoSQL movement, the migration of some types of data to non-relational datastores has recently picked up speed. For web (and other developers) this has lead to some impressive engineering resources developing some amazing tools being open sourced for the world to use. One that caught my eye recently has been Salvatore Sanfilippo’s redis which has been taken under the wing of VMWare, solidifying and validating the great work that Salavatore has done making redis an amazing tool in any developers arsenal.

A very simplified explanation of redis is that it is an in memory key-value store like memcached but, it is persistent on disk, unlike memcached which is volatile. Along with being disk-persistent redis also supports some basic data structures like lists, sets, ordered sets, hashes, and of course basic key-value string storage like memcached. Redis, with disk-persistence and basic data structures, remains blazing fast with published benchmarks of 110,000 SETs per second, about 81,000 GETs per second.

This post is the start of a series of articles on redis for Python programmers. A prerequisite for this is going to be some basic Python knowledge, which if you haven’t used before I highly recommend the free web book Dive Into Python. This is going to be a simple overview of the basic data types and usage for Python programmers and we will slowly progress into more complex usages, so if you haven’t done anything with redis before this is a perfect start.

Installing Redis

Redis is extremely easy to install on POSIX systems (Linux, OSX, Cygwin for Windows). Simply download either a release version or checkout the trunk version of redis from the project home page. Unpackage, make, and run. That’s it!

gettingstarted $ wget http://redis.googlecode.com/files/redis-1.2.5.tar.gz
gettingstarted $ gunzip redis-1.2.5.tar.gz
gettingstarted $ tar -xvf redis-1.2.5.tar
gettingstarted $ cd redis-1.2.5
redis-1.2.5 $ make
redis-1.2.5 $ ./redis-server
21 Mar 17:37:11 * Warning: no config file specified, using the default config. In order to specify a config file use 'redis-server /path/to/redis.conf'
21 Mar 17:37:11 - Server started, Redis version 1.2.5
21 Mar 17:37:11 - The server is now ready to accept connections on port 6379
21 Mar 17:37:11 . 0 clients connected (0 slaves), 832496 bytes in use, 0 shared objects

6 commands and redis is running and ready for a connection. Obviously this isn’t best practice for running redis in production but, it works well for just getting it up and running for some experimentation. Let’s leave that window open and open another for a Python console.

In every example below we only are covering the basic operations of each type. If you want to learn about all of the possible operations for redis read up on the redis Command Reference. This has all of the commands available to you.

Redis and Python

First thing we need is the redis-py module for Python. You can install this manually or just use easy_install.

easy_install redis

Everything has a place: keys

Redis is a key-value store just like a Python dictionary. So everything that goes into redis no matter what it is, has a key (or location, or address, whatever makes sense to you). You can’t store something without a place for it to go

Now let’s open up a Python console and start to play with the basic data structures

Data structure: String

The most basic data structure in redis is the string. Simply, you have a key and it stores a string.

>>> import redis
>>> r_server = redis.Redis("localhost")
>>> r_server.set("name", "DeGizmo")
True
>>> r_server.get("name")
'DeGizmo'

So, line by line let’s walk through what we’ve done here.

  1. import the redis module
  2. create an object called “r_server” of redis.Redis type with the parameter “localhost” which is the network location of our running redis server
  3. call the function “set” on the “r_server” object with the parameters “name” and “DeGizmo”. The first parameter “name” is the key and “DeGizmo” is the value. So in redis we created a key-value pair of “name” -> “DeGizmo”
  4. call the function “get” which will retrieve the value from the key which is passed as the parameter. We are getting the key “name” and redis returns the value “DeGizmo”

That’s it. If you go and stop the redis server and restart it and run r_server.get(“name”) command again, it will return “DeGizmo” again due to redis’s persistent store.

Data structure: Integer

The second data type available is an integer.

>>> r_server.set("hit_counter", 1)
True
>>> r_server.incr("hit_counter")
2
>>> r_server.get("hit_counter")
'2'
>>> r_server.decr("hit_counter")
1
>>> r_server.get("hit_counter")
'1'

Again using the “r_server” Redis object we run the following commands

  1. We set the redis key “hit_counter” to a value of 1 just like in the example above
  2. Then we run the command “incr” on the key “hit_counter”. This will increment (incr) the key by 1
  3. We get the value for the key “hit_counter” to verify that it was been incremented
  4. Then we run the command “decr” on the key “hit_counter”. This will decrement (decr) the key by 1
  5. We get the value for the key “hit_counter” to verify that it was been decremented

Simple huh? Well remember that redis is a persistent store that can be accessed by multiple machines (or programs) simultaneously. If you have a horizontally scaled web farm with many different web servers running on separate machines, your web app can all call on and reference the same redis instance and all operate on the “hit_counter” key at the same time and increment it just like how you would use a MySQL database. You can kind of think of redis like a “shared” storage across different applications.

Data structure: Lists

Very similar to the built in Python list type, the redis list type has a few basic methods that combined can quite powerful. We are only covering a tiny portion of the commands available, you can find all the commands in the redis Command Reference.

>>> r_server.rpush("members", "Adam")
True
>>> r_server.rpush("members", "Bob")
True
>>> r_server.rpush("members", "Carol")
True
>>> r_server.lrange("members", 0, -1)
['Adam', 'Bob', 'Carol']
>>> r_server.llen("members")
3
>>> r_server.lindex("members", 1)
'Bob'

We don’t actually have to create or declare anything when using redis, we just use it and redis handles everything else for us. So we just go ahead and tell redis at the key “members” we are going to add (rpush, right push) the value “Adam”. Redis creates the list for us at key “members” and adds the value “Adam” to the list.

  1. With the r_server object again we call the method “rpush” which will add the value “Adam” to the newly created list “members”
  2. We add “Bob” to the same list
  3. Finally we’ll add “Carol”
  4. With the lrange method we are asking redis to return all the objects in “members”. lrange takes 3 arguments: key, start index in list, end index in list. We are getting the objects from the key “members” starting at index 0 and ending at -1 (which is technically the -1, or last index in the list, which will return everything)
  5. The llen method asks redis to return the length of the list at the key “members” which now has 3 objects
  6. lindex method tells redis that we want the object from the key “members” at the index position of 1 (remember lists start at index 0), so redis returns “Bob”

We’ve got some elements in the list at the key “members”; now lets get remove some elements.

>>> r_server.rpop("members")
'Carol'
>>> r_server.lrange("members", 0, -1)
['Adam', 'Bob']
>>> r_server.lpop("members")
'Adam'
>>> r_server.lrange("members", 0, -1)
['Bob']
  1. With the method rpop (right pop) we remove the element in the list on the right side (tail), which is “Carol”
  2. Now when we ask for the list “members” from redis again (from the start of 0, and the end of -1 which returns everything) we see our list now doesn’t have “Carol”
  3. We now lpop (left pop) an element from the list “members”. This will remove the far left element from the list, “Adam”
  4. Now the entire list only contains “Bob”

Data structure: Set

Again, sets perform identically to the built in Python set type. Simply, sets are lists but, can only have unique values. In the above example if we added the value Adam (r_server.lpush(“members”, “Adam”) ) 20 times our list would contain 20 values all containing the value “Adam”. In a set, all elements are unique.

>>> r_server.delete("members")
True
>>> r_server.sadd("members", "Adam")
True
>>> r_server.sadd("members", "Bob")
True
>>> r_server.sadd("members", "Carol")
True
>>> r_server.sadd("members", "Adam")
False
>>> r_server.smembers("members")
set(['Bob', 'Adam', 'Carol'])
  1. First off we delete the old list in the key “members” so we can use it as a set
  2. Then we sadd (set add) the value “Adam” to the key “members”
  3. Do the same for the value “Bob”
  4. Do the same for the value “Carol”
  5. Now we try to add the value “Adam” to the key “members” again but, this time it returns “False” since we are working on a set, and a set only has unique values. There already is a value “Adam” present in this set
  6. The method smembers returns all the members of the set in the Python type set

An example of a use of sets in a web application would be for “upvotes” on a reddit, or hacker news type website. We want to keep track of who up votes a story but, you should only be able to up vote a story once.

>>> r_server.sadd("story:5419:upvotes", "userid:9102")
True
>>> r_server.sadd("story:5419:upvotes", "userid:12981")
True
>>> r_server.sadd("story:5419:upvotes", "userid:1233")
True
>>> r_server.sadd("story:5419:upvotes", "userid:9102")
False
>>> r_server.scard("story:5419:upvotes")
3
>>> r_server.smembers("story:5419:upvotes")
set(['userid:12981', 'userid:1233', 'userid:9102'])

I added a little twist in here with the name of the key: “story:5419:upvotes” but, it’s easy to explain. Redis is “flat” with it’s keyspace. So if we have many different stories we use a fixed key naming convention for easy reference in redis. For this example our key is broken down like this: “object type” : “id” : “attribute”. So, we have an object of type “story” with an ID of “5419″ with an attribute “upvotes’. Redis has no idea what any of this means it just knows the key is “story:5419:upvotes” but, it doesn’t matter, we know what it means and we can divide up our objects into this name space to make it easier to work with and keep from “losing” things. The value being added to the key is divided up in the same way. “userid” is the type and “9102″ is the value or the ID for that user voting on the story.

  1. Just like before we are adding the value “userid:9102″ to the key “story:5419:upvotes”
  2. Now we are adding the value “userid:12981″
  3. Finally adding the valud “userid:1233″
  4. Now, the user with the ID 9102 tries to up vote the story with the ID 5419 again, and redis returns False since that user has already up votes this story before and you can’t up vote a story twice!
  5. The method “scard” is asking redis for the cardinality of the set at key “story:5419:upvotes” or how many things are in this set, and redis returns 3.
  6. Finally we return the list of userid’s that we have stored in the set.

Data structure: Ordered sets

The last data structure we are going to talk about today is an ordered (or sorted) set. This operates just like a set but, has an extra attribute when we add something to a set called a “score”. This score determines the order of the elements in the set. We will stick with the concept for this final example

>>> r_server.zadd("stories:frontpage", "storyid:3123", 34)
True
>>> r_server.zadd("stories:frontpage", "storyid:9001", 3)
True
>>> r_server.zadd("stories:frontpage", "storyid:2134", 127)
True
>>> r_server.zadd("stories:frontpage", "storyid:2134", 127)
False
>>> r_server.zrange("stories:frontpage", 0, -1, withscores=True)
[('storyid:9001', 3.0), ('storyid:3123', 34.0), ('storyid:2134', 127.0)]
>>> frontpage = r_server.zrange("stories:frontpage", 0, -1, withscores=True)
>>> frontpage.reverse()
>>> frontpage
[('storyid:2134', 127.0), ('storyid:3123', 34.0), ('storyid:9001', 3.0)]

Quick namespace explanation like before. For the key we are going to be referring to “stories:frontpage” which is going to be a set of stories slated for the front page of our website. We are storing in that key the value of “storyid:3123″ which is the ID of some story on the site and then a score, which in our case is going to be the number of votes on a story.

  1. First we add the value “storyid:3123″ to “stories:frontpage”, and “storyid:3123″ in our example is going to have 34 votes.
  2. Then add “storyid:9001″ with 3 votes
  3. Then add “storyid:2134″ with 127 votes
  4. We are going to try to add “story:2134″ to the set again but, we can’t since it already exists.
  5. Now we are going to ask redis for all the elements in “stories:frontpage” from index 0 to index -1 (the end of the list) with all associated scores (withscores=True)
  6. We’ve got the scores but, they are in ascending order, we want them in descending order for our website, so we are going to store the results in the variable “frontpage”
  7. Then reverse it (which is an in place operation in Python)
  8. Now print out the front page!

In conclusion let’s do a quick example of a “view” in an application in which a user will vote of a story using redis as a storage engine

#given variables
#r_server   = our redis server
#user_id    = the user who voted on the story
#story_id   = the story which the user voted on
if r_server.sadd("story:%s" % story_id, "userid:%s" % user_id):
    r_server.zincrby("stories:frontpage", "storyid:%s" % story_id, 1)

2 lines of code? This is might compact but, once we unravel it we can see how it makes sense and how powerful redis can be. Let’s start with the if statement.

if r_server.sadd("story:%s" % story_id, "userid:%s" % user_id):

We know the command “sadd” already. This will add an element to a set at a key. The key in this case is

"story:%s" % story_id

If story_id is 3211, then the resulting string will be “story:3211″. This is the key in redis which contains the list of users that has voted on the story.

The value to be inserted at this key is

"userid:%s" % user_id

Just like with story, if the user_id is 9481 then the string to be inserted into the set at “story:3211″ will be “user_id:9481″

Now the redis command “sadd” will return False if that element is already present in the set. So if a user has already voted on this story before we don’t execute the statement under the if. But, if it is added, then we have a brand new vote and we have to increment the votes for the front page.

r_server.zincrby("stories:frontpage", "storyid:%s" % story_id, 1)

We have an ordered set at the key “stories:frontpage” and we are going to increment the element “storyid:%s” % story_id (“story:3211″) by 1.

And now we’re done! We’ve made sure the user hasn’t voted on this story before and then we’ve incremented the number of votes for this story on the front page!


25 Comments on “Getting Started: Redis and Python”

  1. 1 DeGizmo » Blog Archive » Redis: Relations in a NoSQL world said at 1:01 pm on March 23rd, 2010:

    [...] the first article in our series on Redis we talked about how to get started and the basics of the simple data [...]

  2. 2 Matt said at 10:30 pm on March 23rd, 2010:

    Good series of articles. Redis got my attention after your posts but I can’t say that I understand it completely.

    Let’s say I’ve a Cassandra-Python or PostgreSQL-Python website. Where does Redis fit in this setup?

  3. 3 Alex Popescu said at 9:03 am on March 24th, 2010:

    Matt, you can start thinking of Redis as a highly optimized, persistent, data smart map/dict/hashmap.

    Map/dict/hashmap => everything is a key-value pair

    Data smart => values are not completely opaque. Redis offers support for integers, strings, list and recently maps. There is support for additional operations on values depending on their type.

    Persistent => the key – value pairs are not in memory only (nb: there are more details about how Redis is working.)

    Highly optimized => see the numbers mentioned in the article. Plus you could consult the docs for additional operations and see their O.

    You can find quite a few Redis use cases on myNoSQL: http://nosql.mypopescu.com/tagged/redis, a blog that is focused on NoSQL only.

  4. 4 Adam said at 9:04 am on March 24th, 2010:

    @Matt
    If you are familiar with Python, think about redis as a was to persist Python dictionaries. It’s in the name redis itself (REmote DIctionary Server).

    Anything that can be stored in a Python dictionary can be stored in redis.

    You might want to look the next two articles in the series, they go into more practical examples

    http://degizmo.com/2010/03/23/redis-relations-in-a-nosql-world/

    http://degizmo.com/2010/03/24/redis-relations-in-a-nosql-world-using-hashes/

  5. 5 Adam said at 9:06 am on March 24th, 2010:

    Clarifying something in my previous post.

    “Anything that can be stored in a Python dictionary can be stored in redis.”

    If you pickle an object down to a string, you can store it in redis.

    That actually might make a fun article itself.

  6. 6 foeniz said at 12:21 am on April 5th, 2010:

    Wow, I have been looking into this more and more, I really like what one can do with this sort of schema.

    Here’s something I messed around with, it saved my app a good 5 lines of mysqlalchemy:

    r.zadd(“site:%s” % id, (r.lrange(“post:%s:comment” % id, 0, 0)), r.get(“global:%s:score” % id))

  7. 7 Bishwendu Kundu said at 5:52 pm on March 31st, 2011:

    thnx………just the article i needed to get started with redis in python……

  8. 8 fu said at 2:18 am on April 18th, 2011:

    I wondered how can I use python interface for sorting in redis…

  9. 9 arf said at 2:34 am on June 6th, 2011:

    that’s cool explaination Mr.Adams

    I’m trying to store python object in redis, how to call it out

    class Person:
    def __init__(self,name):
    self.name = name
    def whatsyourname(self):
    print self.name

    a = Person(‘arf’)
    b = Person(‘bob’)

    r_server.rpush(“list”,a)
    r_server.rpush(“list”,b)

    r_server

    >>> r_server.lindex(“list”,1)

    >>> z = r_server.lindex(“list”,1)
    >>> z.whatsyourname
    Traceback (most recent call last):
    File “”, line 1, in
    AttributeError: ‘str’ object has no attribute ‘whatsyourname’

    I don’t know how to call whatsyourname function…

  10. 10 saurav verma said at 2:48 am on December 13th, 2012:

    Awesome post, Helped me a lot to understand redis, I thin python with redis makes an awesome couple :)

  11. 11 bgteydtnbix said at 6:46 pm on May 16th, 2013:

    Enter comment here.

  12. 12 however it|but rather|also|so|remember said at 10:28 pm on August 25th, 2013:

    […] nevertheless 23-4-48 but nevertheless pearly white's while|as|with|on top of that|in addition the key|nonetheless the|however yet somehow|yet unfortunately|except|but […]

  13. 13 redis: getting started | The Ubuntu Incident said at 7:23 am on September 29th, 2013:

    […] Getting Started: Redis and Python […]

  14. 14 xiaobing said at 2:17 am on November 1st, 2013:

    It’s very good, but the contents seems to little.

  15. 15 riegie said at 6:36 pm on December 9th, 2013:

    Definitely thinking of implementing Redis into my current stack. Thanks for the intro.

  16. 16 AronDuh said at 12:24 pm on December 13th, 2013:

    Enter comment here.

  17. 17 hate said at 8:57 am on May 16th, 2014:

    We’re a goup of volunteers and opening a brand neww scheme in our
    community. Your site provvided us with useful info to work on.
    You’ve done an impressive activity and our whole group
    can bbe grateful too you.

  18. 18 Alex said at 11:54 am on July 5th, 2014:

    It’s hard to find your website in google. I found it on 17
    spot, you should build quality backlinks , it will help
    you to get more visitors. I know how to help you, just type in google – k2 seo tricks

  19. 19 Efrain said at 10:40 pm on July 7th, 2014:

    Its like you read my mind! You seem to know a lot approximately this, like you wrote the ebook in it or something.
    I believe that you could do with a few p.c. to drive the message home
    a bit, but other than that, this is wonderful blog.
    A great read. I’ll definitely be back.

    my web page webpage (Efrain)

  20. 20 domain said at 7:14 pm on August 19th, 2014:

    Very good write-up. I certainly love this website.
    Continue the good work!

  21. 21 Nate said at 5:33 am on August 22nd, 2014:

    My web site: web site – Nate,

  22. 22 Bruce said at 3:22 pm on August 29th, 2014:

    I read a lot of interesting posts here. Probably you
    spend a lot of time writing, i know how to save you a lot
    of work, there is an online tool that creates readable, SEO friendly posts in minutes,
    just type in google – laranitas free content source

  23. 23 Aravind said at 10:19 pm on September 6th, 2014:

    Do you need to call bgsave() for it to persist to disk?

    Before this: “That’s it. If you go and stop the redis server and restart it and run r_server.get(“name”) command again, it will return “DeGizmo” again due to redis’s persistent store.”

  24. 24 Arleen said at 6:00 am on September 10th, 2014:

    Here is my web site: site (Arleen)

  25. 25 Carlos Rivera said at 11:14 pm on September 10th, 2014:

    good examples of List and Ordered sets


Leave a Reply