dwarfs

DwarFS is a read-only file system with a focus on achieving very high compression ratios in particular for very redundant data.

DwarFS also doesn't compromise on speed and for some cases it is on par with or performs better than SquashFS. For the primary use case, DwarFS compression is an order of magnitude better than SquashFS compression, it's 6 times faster to build the file system, it's typically faster to access files on DwarFS and it uses less CPU resources.

Distinct features of DwarFS are:

• Clustering of files by similarity using a similarity hash function. This makes it easier to exploit the redundancy across file boundaries.

• Segmentation analysis across file system blocks in order to reduce the size of the uncompressed file system. This saves memory when using the compressed file system and thus potentially allows for higher cache hit rates as more data can be kept in the cache.

• Highly multi-threaded implementation. Both the file system creation tool as well as the FUSE driver are able to make good use of the many cores of your system.

• Optional experimental Python scripting support to provide custom filtering and ordering functionality.