Deduplication (or “dedupe”) has been a major buzzword in the storage and data protection sectors for quite some time now, and 
for good reason. It has the potential to be a game-changer because it can increase the capacity and physical density of storage devices, thereby reducing their overall cost. For a discussion of the basics of deduplication, see my earlier blog post entitled Deduplication: What It Is, And Why It’s Cool.
When NOT to Deduplicate
There are, however, instances where deduplication is not a good idea, and these instances should be taken into consideration when thinking about acquiring new dedupe-enabled storage or software. Herewith are a few instances to consider NOT using dedupe:
You Feel The Need For Speed. Deduplication, because it is a complex series of algorithms, can eat up a lot of CPU cycles. There are three types of deduplication: inline, source-side,and post-process, and the first two can exact a high cost in system and/or storage speed.
- Inline dedupe is the process whereby data is deduplicated while it is being stored. The dedupe-enabled storage device looks at each chunk or block of data as it arrives, comparing it to an index of data that is already stored. If any chunks or blocks are redundant, they are not stored. This is a CPU-intensive process and can result in a slow storage device. (An exception to this is the dedupe capability of storage arrays from NexGen Storage, which actually get faster because of inline dedupe. As an old-line storage guy, I can tell you this is a head-scratcher for me, but it does actually work.)
- Source-side dedupe is sometimes a feature of enterprise backup software. The dedupe process occurs on a production server before shipping the data off to a backup server, thereby reducing the amount of data that must be transported over the network before being backed up. This sounds great, but the CPU of the production server, which may otherwise be doing important things like processing email or managing databases, must divert precious CPU cycles for the dedupe operation. This can cause the server to slow down, which is the last thing you want most production servers to do.
- Post-process dedupe is the non-offender in the dedupe vs. speed discussion, because post-process dedupe happens, as you might expect, after the data arrives and has been stored. As examples, this type of dedupe is utilized on the backup storage arrays from ExaGrid and the backup appliances from Unitrends.
Some Data Won’t Dedupe Very Well. Certain kinds of data are stored in hyper-efficient formats that simply do not deduplicate very well at all, at least at the block or chunk level. These are mostly media files such as MP3, MP4, GIF, TIFF and JPEG. If you store large directories of media files, such as you would with a video-editing platform, digital music system or other media-centric systems, you will see very little benefit from dedupe, and will be wasting your money.
So, to summarize, deduplication can be a game-changer when applied in the proper way, i.e. post process (with, again, the exception of the geniuses at NexGen Storage) and to the proper types of files, i.e. non-media files.
For a further discussion of whether or not to deduplicate, please call me a 888-259-6889 or email me at mark@southerndatastorage.com.