If you’re having caching problems, I feel bad for you son, I’ve got 99 problems, but caching ain’t one.

How did I manage to setup a distributed caching store for Drupal Memcache.

Drupal installations can become pretty heavy when loaded with all kinds of functionality and modules. One of the reasons is that it also stores page structure in database (think blocks, views, also panels and panes if you are using CTools). One way to mitigate this is to use a distributed caching system like Memcached or Reddis. Luckily, one can download the Drupal Memcache module and with some tweaking set it up for your production environment.

Our Drupal installation is deployed on AWS. AWS offers an awesome ElasticCache service which makes it relatively easy to run a cluster of cache server devices and manage them. We will be looking in AWS ElasticCache for Memcached.

AWS has an awesome tutorial on how to setup an ElastiCache cluster. One gotcha here is that if you will be running multiple devices on that cluster, in order for your client (server running Drupal) to find them, you need to install the AWS ElastiCache Cluster Client for PHP. Of course you can assign individual cluster nodes addresses to different cache bins, but that’s just cumbersome and hard to manage.

Once your AWS setup is ready, install the Drupal Memcache module and enable it. Also make sure Memcached is installed on your box and the agent is running. For ec2 linux this is how you do it:

Step 1: Install Memcached

sudo yum install memcached

Step 2: Install php memcached

sudo yum install php-memcached

Step 3: Configure memcached and start the service. Memcached is configured to run on port 11211 by default. You can change that here:

sudo vim /etc/sysconfig/memcached

Now start the service

sudo service memcached start
sudo chkconfig memcached on

The memcached tool will output statistics about the service

sudo memcached-tool localhost:11211 stats

Step 4: Open settings.php and enter the following settings for memcached. By this time you should have the connection string for your memcached cluster

 $conf['cache_backends'][] = 'sites/all/modules/memcache/memcache.inc';
 $conf['cache_default_class'] = 'MemCacheDrupal';
 $conf['cache_class_cache_form'] = 'DrupalDatabaseCache';
 $conf['memcache_servers'] = array('YOUR_CONNECTION_STRING:11211' => 'default');

If you do not specify ‘memcache_servers’ your instance will automatically connect to localhost:11211. In case you are connecting to AWS ElastiCache, you don’t really need memcached service running on your front end instance. However it is nice to have for testing purposes and using the memcached-tool.

Drupal Memcache module provides settings for Stampede Protection. I had stampede protection enabled initially, but page load times went up to 15 secs. Then I discovered that there is a bug in the module. When stampede protection is enabled, MemcachedDrupal::set() does not release the lock once it is done updating the cache (line 263 of memcache.inc)

 if (variable_get('memcache_stampede_protection', FALSE) && isset($GLOBALS['locks'][$lock])) {
      lock_release("$lock");
    }

So the process waits the time passed to it in stampede_semaphore even after the update is finished and never releases the lock during this time. You can tweak the values in wait_time, wait_limit and stampede_semaphore, and even if you set them to 1 sec, 3 times and 3 secs you will still see page loads of at least 3 secs which is unacceptable. As of version 7.x-1.6 I don’t think this has been fixed.