Selfhosted

60588 readers

1445 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Detailed Rules Post

Be civil.
No spam.
Posts are to be related to self-hosting.
Don't duplicate the full text of your blog or readme if you're providing a link.
Submission headline should match the article title.
No trolling.
Promotion posts require active participation, with an account that is at least 30 days old. F/LOSS without a paywall has exceptions, with requirements. See the rules link for details. Tags [CBH] or [AIP] are required, see the links in Rule 8 for details.
AI-related discussions and AI-involved promotional posts have additional requirements for tagging, as noted in Rule 7 and the AI & Promotional Post Expanded Rules post, and find example disclosures here.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 3 years ago

MODERATORS

curbstickle@anarchist.nexus

curbstickle_lw@lemmy.world

Experiences with zfs deduplication? (feddit.org)

submitted 2 years ago by needanke@feddit.org to c/selfhosted@lemmy.world

16 comments fedilink hide all child comments

I recently moved my files to a new zfs-pool and used that chance to properly configure my datasets.

This led me to discovering zfs-deduplication.

As most of my storage is used by my jellyfin library (~7-8Tb), which is mostly uncompressed bluray rips I thought I might be able to save some storage using deduplication in addition to compression.

Has anyone here used that for similar files before? What was your experience with it?

I am not too worried about performance. The dataset in question is rarely changed. Basically only when I add more media every couple of months. I also have overshot my cpu-target when originally configuring my server so there is a lot of headroom there. I have 32Gb of ram which is not really fully utilized either (but I also would not mind upgrading to 64 too much).

My main concern is that I am unsure it is useful. I suspect just because of the amount of data and similarity in type there would statistically be a lot of block-level duplication but I could not find any real world data or experiences on that.

you are viewing a single comment's thread
view the rest of the comments

[–] emptiestplace@lemmy.ml 1 points 2 years ago (1 children)

Just adjust it if you actually need the RAM and it isn't relinquishing quickly enough.

options zfs zfs_arc_max=17179869184 in /etc/modprobe.d/zfs.conf, update-initramfs -u, reboot - this will limit ZFS ARC to 16GiB.

arc_summary to see what it's using now.

As for using a simple fs on LVM, do you not care about data integrity?

[–] IsoKiero@sopuli.xyz 1 points 2 years ago

this will limit ZFS ARC to 16GiB.

But if I have 32GB to start with, that's still quite a lot and, as mentioned, my current usage pattern doesn't really benefit from zfs over any other common filesystem.

As for using a simple fs on LVM, do you not care about data integrity?

Where you get that from? LVM has options to create raid volumes and, again as mentioned, I can mix and match those with software raid however I like. Also, single host, no matter how sophisticated filesystems and raid setups, doesn't really matter when talking about keeping data safe, that's what backups are for and it's a whole another discussion.