Selfhosted

59939 readers

528 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam.
Posts here are to be centered around self-hosting. Please ensure it is clear in your post how it relates to self-hosting.
Don't duplicate the full text of your blog or git here. Just post the link for folks to click.
Submission headline should match the article title.
No trolling.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

Experiences with zfs deduplication? (feddit.org)

submitted 1 year ago by needanke@feddit.org to c/selfhosted@lemmy.world

16 comments fedilink hide all child comments

I recently moved my files to a new zfs-pool and used that chance to properly configure my datasets.

This led me to discovering zfs-deduplication.

As most of my storage is used by my jellyfin library (~7-8Tb), which is mostly uncompressed bluray rips I thought I might be able to save some storage using deduplication in addition to compression.

Has anyone here used that for similar files before? What was your experience with it?

I am not too worried about performance. The dataset in question is rarely changed. Basically only when I add more media every couple of months. I also have overshot my cpu-target when originally configuring my server so there is a lot of headroom there. I have 32Gb of ram which is not really fully utilized either (but I also would not mind upgrading to 64 too much).

My main concern is that I am unsure it is useful. I suspect just because of the amount of data and similarity in type there would statistically be a lot of block-level duplication but I could not find any real world data or experiences on that.

you are viewing a single comment's thread
view the rest of the comments

[–] Moonrise2473@feddit.it 13 points 1 year ago (1 children)

Uncompressed Blu-ray rips are almost the same size when compressed with lossless compression. The binary content of h264 files are almost random bits so deduplication is almost a waste of CPU time. Maybe you can save space from the useless repeated media like trailers and other ads in the bluray isos

[–] needanke@feddit.org 1 points 1 year ago

Thank you, I didn't know that, but that was the type of answer I was looking for :D.

I only ripped the relevant media anyways, so no trailers to remove.