Feeds:
Posts
Comments

Posts Tagged ‘caching’

Where to Cache Stuff?

A fairly common task I need to do is:

  • query a remote service which returns a large amount of data
  • extract just the bits I need from that data
  • do something with the extracted bits of data

Often the initial query will take a few seconds to run and I’ll be thinking I can’t be bothered to wait for this, why don’t I just cache the data.

If I decide it is worth it, the next question is where and how to cache.

Perl AnyEvent

And what I generally think of first is an AnyEvent-based Proxy Server. Just as quickly, I discard that option as I can’t be bothered to figure out how to ensure the proxy is up when I need it. For example, what happens when the physical host where the proxy is running reboots? What happens if the sysadmin kills my process? Etc.

Storable and Freeze / Thaw

So, next I think: the filesystem is always available (hopefully!), I’ll use that. So then I’m thinking about Cache::File, Storable and Freeze/Thaw.

This often leads to a couple of issues too – where should I store my datafiles. Should I store it somewhere in $HOME and then if multiple people want to cache the data, they are each hitting the remote service, or should I find I have a shared area? Then, should this shared area be on a network drive, or local to the box?

Database

So, finally, I get to thinking about caching the data in the database. I know for sure that will always be available.

Can you guess which option I generally choose? Does anyone else have any thoughts and where and how to cache data?

Read Full Post »

Follow

Get every new post delivered to your Inbox.