| News | Features | Interviews |
| Blog | Contact | Editorials |
| ZFS Gets Deduplication |
| By Thom Holwerda on 2009-11-02 23:20:34 |
| ZFS has received built-in deduplication. "Deduplication is the process of eliminating duplicate copies of data. Dedup is generally either file-level, block-level, or byte-level. Chunks of data - files, blocks, or byte ranges - are checksummed using some hash function that uniquely identifies data with very high probability. Chunks of data are remembered in a table of some sort that maps the data's checksum to its storage location and reference count. When you store another copy of existing data, instead of allocating new space on disk, the dedup code just increments the reference count on the existing data. When data is highly replicated, which is typical of backup servers, virtual machine images, and source code repositories, deduplication can reduce space consumption not just by percentages, but by multiples." |
|
| Wrong link |
| By diegocg on 2009-11-02 23:48:51 |
| The link doesn't work, it has extra characters. |
 - Score: 3 |
| RE: Wrong link |
| By TObYv on 2009-11-03 00:45:00 |
| Perhaps the link needs to be fscked? |
| - Score: 18 |
| ZFS Compression and Deduplication |
| By Jondice on 2009-11-03 02:37:24 |
|
I'm aware that zfs compression will be compatible with deduplication, but I wonder if it will generally give improved storage capacity over simply using dedup. The only way it would be worse is if compressing larger blocks will give fewer identical, compressed, smaller blocks. This seems unlikely although I haven't gone over the details of the compression algorithm used. |
| - Score: 1 |
| I skimmed the article... |
| By Tuishimi on 2009-11-03 06:13:50 |
| ...so I probably missed this but I assume this feature is meant for servers, hopefully with (is this irony?) data replication (mirror set, backups) - what happens if the disk sustains physical damage? That could affect (theoretically - unless I misunderstand the concept) a lot more data than it would have without the dedup feature? |
| - Score: 3 |
| RE: I skimmed the article... |
| By Jondice on 2009-11-03 06:35:42 |
|
A good point, but I think most people using this in servers will have already taken appropriate (RAID, backup) precautions. ZFS compression and dedup should be nice for home users of Solaris as well (of which I am one). Still have a ways to go before my workstation's pool is full, even w/o dedup: [brandon@barker]:[8]:[~]:$ zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT rpool 149G 64.0G 85.0G 42% ONLINE - storage 6.31T 863G 5.47T 13% ONLINE - |
| - Score: 1 |
| RE: I skimmed the article... |
| By c0t0d0s0 on 2009-11-03 08:07:35 |
| That's easy: The RAID1, RAID0, RAIDZ, RAIDZ1, RAIDZ2, RAIDZ3 takes care of ensuring the availability of your data. The deduplication takes care of the fact, that you store a block only once for example when you have a dozen similar VMware/Virtualbox images with a Windows in it for example. By the way: Dedup creates a second copy of the block when more than hundred blocks (default, it's adjustable) are just refering to a single block due to the dedup. |
| - Score: 2 |
| RE: ZFS Compression and Deduplication |
| By Laurence on 2009-11-03 10:36:04 |
|
> I'm aware that zfs compression will be compatible with deduplication, but I wonder if it will generally give improved storage capacity over simply using dedup. The only way it would be worse is if compressing larger blocks will give fewer identical, compressed, smaller blocks. This seems unlikely although I haven't gone over the details of the compression algorithm used. You can choose the compression algorithm to use. A quick look on wikipedia details: LZJB and gzip as supported algorithms but I'm pretty sure theres since been a 3rd supported format. |
| - Score: 2 |
| RE: I skimmed the article... |
| By Laurence on 2009-11-03 10:53:49 |
|
> ...so I probably missed this but I assume this feature is meant for servers, hopefully with (is this irony?) data replication (mirror set, backups) - what happens if the disk sustains physical damage? That could affect (theoretically - unless I misunderstand the concept) a lot more data than it would have without the dedup feature? This feature will be better suited for servers than home PCs, but that doesn't mean that such facility couldn't be useful for some home users: HTPC / media servers: if you have lot's of DVD rips of TV shows, then you could save several hundred MBs with the intro/outro credits being duduped alone. media professionals: Granted ZFS isn't coming to OS X now, but if you're a media professional (music, graphics, etc) and want to keep back ups of your projects then you may well have several files with similar contents as the art took shape (much like backed up lines of code in a CVS repository) that all said, I'd be a touch cautious about jumping in and dedup'ing your file system on consumer grade hardware unless you were confident with your hardware and I'd still recommend weekly scrubs to highlight data degradation before it rots your data completely |
| - Score: 2 |
| RE[2]: I skimmed the article... |
| By Tuxie on 2009-11-03 14:15:56 |
|
> HTPC / media servers: if you have lot's of DVD rips of TV shows, then you could save several hundred MBs with the intro/outro credits being duduped alone. Err, no. There is no way the intro/outro scenes are going to be byte-by-byte-identical in the encoded data for different episodes even if they look identical to the eye. Even if nothing else is, the timestamp metadata for each frame is going to differ. |
| - Score: 2 |
| RE[3]: I skimmed the article... |
| By Laurence on 2009-11-03 14:31:31 |
|
> Err, no. There is no way the intro/outro scenes are going to be byte-by-byte-identical in the encoded data for different episodes even if they look identical to the eye. Even if nothing else is, the timestamp metadata for each frame is going to differ. I guess that depends on the codec used. I thought many MPEG codecs didn't have a timestamp as such and used a form of encoding that allowed an MPEG file (be it a video container file or an MP3 audio file) to be chopped in to parts at any random point and each of the parts can still play individually (much like the myth about worms ability to be chopped up and each part becoming alive) Besides, your point is only valid for shows that have a pre-opening credits teaser rather than those (typically older) shows that always opened with music and credits. |
| - Score: 2 |
| News | Features | Interviews |
| Blog | Contact | Editorials |