DiskCache is an Apache2 licensed disk and file backed cache library, written in pure-Python, and compatible with Django.

The cloud-based computing of 2018 puts a premium on memory. Gigabytes of empty space is left on disks as processes vie for memory. Among these processes is Memcached (and sometimes Redis) which is used as a cache. Wouldn't it be nice to leverage empty disk space for caching?

Django is Python's most popular web framework and ships with several caching backends. Unfortunately the file-based cache in Django is essentially broken. The culling method is random and large caches repeatedly scan a cache directory which slows linearly with growth. Can you really allow it to take sixty milliseconds to store a key in a cache with a thousand items?

In Python, we can do better. And we can do it in pure-Python!

In [1]: import pylibmc
In [2]: client = pylibmc.Client(['127.0.0.1'], binary=True)
In [3]: client[b'key'] = b'value'
In [4]: %timeit client[b'key']

10000 loops, best of 3: 25.4 µs per loop

In [5]: import diskcache as dc
In [6]: cache = dc.Cache('tmp')
In [7]: cache[b'key'] = b'value'
In [8]: %timeit cache[b'key']

100000 loops, best of 3: 11.8 µs per loop

Note: Micro-benchmarks have their place but are not a substitute for real measurements. DiskCache offers cache benchmarks to defend its performance claims. Micro-optimizations are avoided but your mileage may vary.

DiskCache efficiently makes gigabytes of storage space available for caching. By leveraging rock-solid database libraries and memory-mapped files, cache performance can match and exceed industry-standard solutions. There's no need for a C compiler or running another process. Performance is a feature and testing has 100% coverage with unit tests and hours of stress.

Does your company or website use DiskCache? Send us a message and let us know.

• TODO: update with Comparison below
• Pure-Python
• Fully Documented
• Benchmark comparisons (alternatives, Django cache backends)
• 100% test coverage
• Hours of stress testing
• Performance matters
• Django compatible API
• Thread-safe and process-safe
• Supports multiple eviction policies (LRU and LFU included)
• Keys support "tag" metadata and eviction
• Developed on Python 2.7
• Tested on CPython 2.7, 3.4, 3.5, 3.6 and PyPy
• Tested on Linux, Mac OS X, and Windows
• Tested using Travis CI and AppVeyor CI

Installing DiskCache is simple with pip:

$ pip install diskcache

You can access documentation in the interpreter with Python's built-in help function:

>>> from diskcache import Cache, FanoutCache, DjangoCache
>>> help(FanoutCache)
>>> cache = FanoutCache('path/to/cache/directory')
>>> help(cache)
>>> help(DjangoCache)

For those wanting more details, this part of the documentation describes tutorial, benchmarks, API, and development.

• DiskCache Tutorial
• DiskCache Cache Benchmarks
• DiskCache DjangoCache Benchmarks
• Case Study: Web Crawler
• Talk: All Things Cached - SF Python 2017 Meetup
• DiskCache API Reference
• DiskCache Development

Features

• Atomic? Yes/No
• Persistent? Yes/No
• Thread-safe? Yes/No
• Process-safe? Yes/No
• Backend? Memory/Filesystem/SQLite/Redis/SQLAlchemy/etc.
• Serialization? Pickle/JSON/etc.
• Data Types? Mapping (Ordered, Sorted)/Deque/etc.
• Eviction Policies? LRU/LFU/MRU/LRS/Random/Dummy/Infinite
• Vacuum? Automatic/Manual
• Can be pickled? (Important for multiprocessing module)
• Supports process forking? (Important for multiprocessing)
• Supports metadata?
• Tests? Yes/No
• Coverage? Yes/No
• Stress? Yes/No
• CI Tests? None/Travis/AppVeyor/etc.
• Python? 2/3/PyPy/Jython
• Platforms? *nix/Windows
• License? Apache2/GPL/etc.
• Docs? None/Readme/Extensive
• Benchmarks? Yes/No
• Timings for get/set/delete
• Sources? GitHub/SourceForge
• Structure? Document/Relational/Key-Value
• Pure-Python? Yes/No
• Server? Yes/No
• Goal? Database/ORM/KV Store/etc.
• Web Framework Support? Django, Flask, Pyramid, Plone

• dbm -- https://docs.python.org/library/dbm.html
• shelve -- https://docs.python.org/library/shelve.html
• sqlitedict -- https://github.com/RaRe-Technologies/sqlitedict
• pickleDB -- https://pythonhosted.org/pickleDB/

• Timings for get/set/delete

```python import dbm import diskcache import pickledb import shelve import sqlitedict

print('diskcache') dc = diskcache.FanoutCache('/tmp/diskcache') %timeit dc['key'] = 'value'

print('dbm') %timeit d = dbm.open('/tmp/dbm', 'c'); d['key'] = 'value'; d.close()

print('shelve') %timeit s = shelve.open('/tmp/shelve'); s['key'] = 'value'; s.close()

print('sqlitedict') sd = sqlitedict.SqliteDict('/tmp/sqlitedict', autocommit=True) %timeit sd['key'] = 'value'

print('pickledb') p = pickledb.load('/tmp/pickledb', True) %timeit p['key'] = 'value' ```

Caching Libraries

• klepto -- https://pypi.org/project/klepto/
• joblib.Memory -- https://joblib.readthedocs.io/en/latest/memory.html

In-Memory Data Structures

• dict -- https://docs.python.org/3/library/stdtypes.html#typesmapping
• pandas -- https://pandas.pydata.org/
• Sorted Containers -- http://www.grantjenks.com/docs/sortedcontainers/
• ldtable -- https://github.com/Jwink3101/ldtable

Pure-Python Databases

• ZODB -- http://www.zodb.org/
• CodernityDB -- http://labs.codernity.com/codernitydb/
• TinyDB -- https://tinydb.readthedocs.io/

Object Relational Mappings (ORM)

• Django ORM -- https://docs.djangoproject.com/en/dev/topics/db/
• SQLAlchemy -- https://www.sqlalchemy.org/
• Peewee -- http://docs.peewee-orm.com/
• dataset -- https://dataset.readthedocs.io/
• SQLObject -- http://sqlobject.org/
• Pony ORM -- https://ponyorm.com/

SQL Databases

• SQLite -- https://docs.python.org/library/sqlite3.html
• MySQL -- https://dev.mysql.com/downloads/connector/python/
• PostgreSQL -- http://initd.org/psycopg/
• Oracle -- https://pypi.org/project/cx_Oracle/
• Microsoft SQL Server -- https://pypi.org/project/pyodbc/

Other Databases

• Memcached -- https://pypi.org/project/python-memcached/
• MongoDB -- https://api.mongodb.com/python/current/
• Redis -- https://redis.io/clients#python
• LMDB -- https://lmdb.readthedocs.io/
• BerkeleyDB -- https://pypi.org/project/bsddb3/
• LevelDB -- https://plyvel.readthedocs.io/

• DiskCache Documentation
• DiskCache at PyPI
• DiskCache at GitHub
• DiskCache Issue Tracker

Copyright 2016-2018 Grant Jenks

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.