data deduplication hardware

What is data deduplication hardware?

Data deduplication hardware is disk storage that eliminates redundant copies of data and retains one instance to be stored. Hardware-based deduplication products perform deduplication at the target rather than the source, or server. Data deduplication features can be found in data protection hardware such as virtual tape libraries, backup appliances and network-attached storage systems.

Data deduplication recognizes redundant data and only stores one copy on disk. It replaces the redundant data with a pointer to the unique copy. Subsequent backups only copy changed data blocks. Data deduplication, also called dedupe, increases efficiency by eliminating unnecessary copies of information from the backup process. Data reduction techniques reduce the amount of capacity required for backups.

Because the reduction takes place at the target, hardware-based dedupe does not reduce the amount of data being sent across the network during a backup. It does not tax the server's CPU, however. While source deduplication allows dedupe software to use less storage and bandwidth, it requires more processing power. By emphasizing reduction at the target, deduplication hardware can provide faster performance for large data sets. Because of this, it is often used by companies that work with bigger data sets and smaller bandwidth constraints. Smaller businesses and remote offices may steer clear of the hardware approach.

Data deduplication hardware is known for its high performance, as well as relatively non-disruptive deployments. Because the appliances are purpose-built, data deduplication hardware can provide a quick deployment and simple integration with existing backup software. Dedicated appliances are typically seen as generic disk systems by the backup software, which may not even recognize that deduplication is taking place. Deduplication hardware often requires less maintenance than a software approach, and may see better compression levels.

While the plug-and-play experience of using hardware for dedupe can mean easy deployment, it could affect flexibility and scalability. Unless the systems can perform global deduplication, they will not recognize redundancies across multiple appliances. This limits deduplication to individual appliances and can add management complexity.

Global deduplication recognizes redundancies of data sent from one node to another and does not make an additional copy.

When evaluating data deduplication hardware products, important factors to consider are dedupe ratios, storage infrastructure costs, performance requirements and existing data protection protocols. While software is another deduplication option, it may not support protocols already in place, as a dedicated deduplication appliance would.

Although deduplication originated as a backup technology, it is now also found in primary storage systems. Solid-state storage systems often use dedupe to reduce the amount of data stored on expensive solid-state drives (SSDs).