DAVE'S LIFE ON HOLD

Varnish Cache + Riak: Part II

A few days ago, I wrote up how to implement a simple Varnish cache Riak key value store setup for development use. Not only did I use Varnish as a cache, but I also used it to rewrite request paths to point different paths to different buckets. I also added some basic access control, using the acl feature to restrict access to the /admin/ interfaces.

Today, I'm going to address the question:

"How does this setup account for inconsistencies between multiple varnish nodes after requests that cause a purge?"

Sort answer is the development server where you're running varnish on one box on port 80, and riak running on 3 ports on the same box does not. It assumes that you are going through a single varnish node in development. When you go to production, however, this is not only unadvisable but also highly unlikely. Managing cache state across multiple varnish caches requires that you execute the purge.url directive on all of the nodes that have access to that resource on the backend.

There are three basic approaches you can take:


  1. Have a service monitor PUT, POST, DELETE requests from the Varnish logs and propagate the changes

  2. Add hooks to Riak to relay the purge requests to each varnish node upon successful update

  3. Have varnish relay the request to its peers



For the purposes of this article, I'm going to focus on #3, and demonstrate how you can embed C code in your .vcl file to invoke arbitrary web requests to other servers within varnish. But first a disclaimer: NB: I do not recommend you do this, and don't blame me when you DDOS your own site!.

Ok, that out of the way, Varnish requires GCC be installed on your production boxes. The reason Varnish requires this is that it actually compiles your .vcl file into a shared library and loads it via dlopen. It then uses the sub routine hooks to extend the functionality of the server at run time via native code. This is why .vcl files can have C C directives embedded inside of them, which compile down to native code, and perform their tasks as quickly as anything that can be done on your server.

In our previous example, we were already catching the PUT, POST, and DELETE http methods as follows:


if (req.request == "PUT" || req.request == "DELETE" || req.request == "POST") return(pass);
" alt="
purge("req.url ~ " req.url);
return(pass);
" />


This was done to purge the cache before passing through to the backend Riak servers to process the PUT, DELETE, or POST. So if we are going to support a number of other varnish nodes in our cluster, we probably want to propagate the purging to all of the nodes in the cluster. So that the next time the client requests the newly written data, and hits a different box, they will not be fed a stale piece out of cache. To do this we can change the config to read as follows:


if (req.request == "PUT" || req.request == "DELETE" || req.request == "POST") " alt="
set req.http.X-Purge-Path = req.url;
C{
purge_remotes(VRT_GetHdr(sp,HDR_REQ,"\\015X-Purge-Path:"));
" />C
purge("req.url ~ " req.url);
return(pass);
}


Here we are appending a new header entry, "X-Purge-Path:" to our request header, which will be passed on to Riak, as well, which we then use to call a native C function purge_remotes. We then purge the local cache and pass it on through to Riak. To implement purge_remotes we will embed some more C at the beginning of the file:


C<br />#include <stdio.h><br />#include <stdlib.h><br />#include <dlfcn.h><br />#include <syslog.h>l<br />#include <curl/curl.h><br />C


Doing this will provide access to syslog, dlopen, dlsym, and curl. In order to purge the remotes, we will use libcurl to make PURGE requests to the peers, and allow each of them to clear their caches. To setup that bit, we can expose the peer configuration as a C string array and include the source file for our purge_remotes function:


C<br />char* peers[] = { ;
#include "/Users/dave/Servers/purge_remotes.c"
}C


For this example, we're only going to run one peer running on a different port on the same machine, but you can easily add any number of machines to the lists of peers. The trailing NULL is important, as we are going to make one PURGE request for each string in this array, until we encounter a null pointer value. The second line simply includes the C source file directly into our VCL which is then compiled inline. This allows us to save the bulk of our C code in an external file and edit it independently.

The final bit of configuration change we need for this to work is to add the handler for our PURGE method, so that if one of our peers contacts us about an update they did, we can handle it gracefully:


if (req.request == "PURGE" ) error 200 "OK";
" alt="
purge("req.url ~ " req.url);
error 200 "OK";
" />


This is all one needs to handle the PURGE request, and since we don't really care that much about the response, we're just going to error out with a 200 OK. You could go to some length to minimize the payload here, and you could synthesize a response, but for now this will be sufficient to demonstrate the technique.

The implementation of void purge_remotes(char* path) is fairly straight forward and consists of 3 general parts, a preamble, an initialization section, and the request dispatch. The preamble is trivial:


void purge_remotes(char* path) if (!libcurl) {
syslog(LOG_ERR,"Could not open libcurl");
return;
" alt="
CURL *curl;
CURLcode res;
int i;
char* url = NULL;


Here we declare the function prototype and the local variables. The path is passed to us from the embedded C code in vcl_recv. After this we need to dynamically load the functions from libcurl into our current process space, and lookup each of the symbols that we will need to use the Curl API:


void* libcurl = dlopen("/opt/local/lib/libcurl.dylib", RTLD_LAZY|RTLD_LOCAL);
if (!libcurl) {
syslog(LOG_ERR,"Could not open libcurl");
return;
" />

CURLcode (curl_easy_setopt_p)() = dlsym(libcurl,"curl_easy_setopt");
CURLcode (
curl_easy_perform_p)() = dlsym(libcurl,"curl_easy_perform");
CURL* (curl_easy_init_p)() = dlsym(libcurl,"curl_easy_init");
void (
curl_easy_cleanup_p)() =dlsym(libcurl,"curl_easy_cleanup");
if (! curl_easy_setopt_p || ! curl_easy_perform_p || ! curl_easy_init_p || ! curl_easy_cleanup_p) return;
" alt="
syslog(LOG_ERR, "Could not find symbol for some curl function");
return;
" />


Dlopen takes a path, and a set of flags which in this case tell it to open my copy of libcurl from MacPorts, and set it to lazy load the symbol references and to not export any symbols into the processes global symbol table. This is the safest option from a linking standpoint, as dlopen will automatically handle not loading the library more than once, and only our code will have access to the function pointers. The following lines then load: curl_easy_setopt, curl_easy_perform, curl_easy_init, and curl_easy_cleanup. The function pointers (all ending in _p) allow us to call those shared library functions. If any of these pointers fail to be found, or the library fails to load, we quit out and don't proceed with processing the purge.

The final bit is the bit that actually performs the remote purge requests:

curl = curl_easy_init_p();
if(curl) curl_easy_setopt_p(curl, CURLOPT_CUSTOMREQUEST, "PURGE");
curl_easy_setopt_p(curl, CURLOPT_URL, url);
res = curl_easy_perform_p(curl);
syslog(LOG_INFO,"Purged %s",url);
free(url);
url = NULL;
" alt="
for (i = 0; peers[i]; i) {
asprintf(&url,"http://%s%s",peers[i],path);
curl_easy_setopt_p(curl, CURLOPT_CUSTOMREQUEST, "PURGE");
curl_easy_setopt_p(curl, CURLOPT_URL, url);
res = curl_easy_perform_p(curl);
syslog(LOG_INFO,"Purged %s",url);
free(url);
url = NULL;
" />
curl_easy_cleanup_p(curl);
}


All that we do is init curl, create a new request for each peer in peers, with the appropriate path. We then modify the method using the CURLOPT_CUSTOMREQUEST field, and perform the request. Finally we clean up after ourselves, and free all of the memory we allocated for this process. And we're done! If you then ran 2 varnishes on different ports, and had the same config for each you can test this by using curl to put a new index.html page in your site bucket:


curl -X PUT http://localhost/riak/site/index.html -H "Content-type: text/html" --data-binary @Sites/index.html


You can also use curl to purge your caches by hand by doing things like:


curl -X PURGE http://localhost/


Which varnish will map to the appropriate backend asset and purge its cache internally.

In this example configuration, there is a sever performance bottleneck. In order to make it easy to understand, I used the easy curl interface. This blocks each request before continuing onto the next one. This means that all of the remote caches will be purged prior to the contents being updated. Not only does this introduce a new race condition, but it also doesn't scale very well. However that said, using the curl multi interface for full asynchronous purging is also possible. The application of that is almost identical, but presents interesting challenges when debugging across multiple machines.

Once again, if you have a resource that has a high write volume, you are probably better off not caching it at all, and instead making those a straight pass through to Riak. Riak tends to perform as well as varnish in serving most static files. But for resources that are expensive like map-reduce queries or full key listings, having a cache in front can make your life much easier.

Stay tuned for more.