I think something might be wrong with your Neovim if it aggregated 70 gigs of log files.
this post was submitted on 29 Oct 2023
215 points (95.4% liked)
Linux
48031 readers
1166 users here now
From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Rules
- Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
- No misinformation
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon by AlpΓ‘r-Etele MΓ©der, licensed under CC BY 3.0
founded 5 years ago
MODERATORS
don't worry, they've just been using neovim for 700 years, it'll be alright
Sure, that's also a possibility. I'd be interested in their time machine though.
So I found out that qbittorrent generates errors in a log whenever it tries to write to a disk that is full...
Everytime my disk was full I would clear out some old torrents, then all the pending log entries would write and the disk would be full again. The log was well over 50gb by the time I figured out that i'm an idiot. Hooray for having dedicated machines.
I once did something even dumber. When I was new to Linux and the CLI, I added a recursive line to my shell config that would add it self to the shell config. So I pretty much had exponential growth of my shell config and my shell would take ~20 seconds to start up before I found the broken code snippet.
If you have ideas please let me know. I'm preparing to hop distros so I'm very tempted to ignore the problem, blame the old distro, and hope it doesn't happen again :)
I would have to look at the log file. Some plugin probably has an issue and writes massive amounts of data to the log every time you use Neovim. Monitor the growth of the log file and contact me via DM if it goes crazy again, I'm gonna try to figure out what's going on.
ncdu is the best utility for this type of thing. I use it all the time.
I install ncdu on any machine I set up, because installing it when it's needed may be tricky
load more comments
(10 replies)
Try ncdu as well. No instructions needed, just run ncdu /path/to/your/directory.
I usually use something like du -sh * | sort -hr | less, so you don't need to install anything on your machine.
Same, but when it's real bad sort fails π for some reason my root is always hitting 100%
I usually go for du -hx | sort -h and rely on my terminal scroll back.
dust does more than what this script does, its a whole new tool. I find dust more human readable by default.
Maybe, but I need it one time per year or so. It is not a task for which I want to install a separate tool.
load more comments
(1 replies)
Almost the same here. Well, du -shc *|sort -hr
I admin around three hundred linux servers and this is one of my most common tasks - although I use -shc as I like the total too, and don't bother with less as it's only the biggest files and dirs that I'm interested in and they show up last, so no need to scrollback.
When managing a lot of servers, the storage requirements when installing extra software is never trivial. (Although our storage does do very clever compression and it might recognise the duplication of the file even across many vm filesystems, I'm never quite sure that works as advertised on small files)
I admin around three hundred linux servers
What do you use for management? Ansible? Puppet? Chef? Something else entirely?
Main tool is Uyuni, but we use Ansible and AWX for building new vms, and adhoc ansible for some changes.
Interesting; I hadn't heard of Uyuni before. Thanks for the info!
Seems it just runs Salt/Saltstack?
Suse forked Redhat's Spacewalk just before it turned into Foreman + Katello.
Then worked an absolute crapload on it to turn it into a modern orchestrator. Part of that was to adopt salt as the agent interface, gradually getting rid of the creaking EL traditional client.
To say "it just runs salt" is to rather miss all the other stuff Uyuni does. Full repo and patch management, remote control, config management, builds, ansible playbook support, salt support, and just about everything else you need to manage hundreds of machines. Oh, and it does that for Rocky, RHEL, Alma, Suse, Ubuntu, Debian and probably a bunch more too, by now. Has a very rich webui, a full API and you can do a bunch more from the cli as well. And if your estate gets too big to manage with one machine, there are proxy agents, as many as you want. I only run a couple of hundred vms through it, but there are estates running thousands.
And it's free and foss.
Honestly, it's pretty awesome and I'm amazed it's not more widely known.
Oh that's pretty nifty, thanks for the comment. Sorry wasn't trying to minimize the tool, I was simply referring to the orchestration/config management aspect of it when I looked it up real quick.
I used to be responsible for configurations of 40,000 (yes forty thousand) VMs for a large company using puppet and then later using Ansible and that was an interesting challenge. I've been out of the configuration management game for a few years now though so I'm pretty out of the loop. Was familiar with spacewalk back in the day too.
I'll have to check Uyuni out, thanks for sharing!
We'd use du -xh --max-depth=1|sort -hr
du -xh --max-depth=1|sort -hr
Interesting. Do you often deal with dirs on different filesystems?
Yeah, I was a Linux System Admin/Engineering for MLB/Disney+ for 5 years. When I was an admin, one of our tasks was clearing out filled filesystems on hosts that alerted.
Sounds pretty similar to what I do now - but never needed the -x. Guess that might be quicker when you're nested somewhere there is a bunch of nfs/smb stuff mounted in.
We'd do it from root (/) and drill down from there, it was usually /var/lib or /var/logs that was filling up, but occasionally someone would upload a 4.5 GB file to their home folder which has a quota of 5 GB.
Using ncdu would have been the best way, but that would require it being installed on about 7 thousand machines.
I'd say head -n25 instead of less since the offending files are probably near the top anyway
So like filelight?
I really like ncdu
Yeah I got turned onto ncdu recently and Iβve been installing it on every vm I work on now
A 70gb log file?? Am I misunderstanding something or wouldn't that be hundreds of millions of lines
I've definitely had to handle 30gb plain text files before so I am inclined to believe twice as much should be just as possible
You guys aren't using du -sh ./{dir1,dir2} | sort -nh | head?
I miss WinDirStat for seeing where all my hard drive space went. You can spot enormous files and folders full of ISOs at a glance.
For bit-for-bit duplicates (thanks, modern DownThemAll), use fdupes.
If WizTree is available on Linux then I highly recommend it over all other alternatives.
It reads straight from the table and is done within a couple of seconds.
Filelight on linux
Squirreldisk on windows
Both libre
Maybe other tools support this too but one thing I like about xdiskusage is that you can pipe regular du output into it. That means that I can run du on some remote host that doesn't have anything fancy installed, scp it back to my desktop and analyze it there. I can also pre-process the du output before feeding it into xdiskusage.
I also often work with textual du output directly, just sorting it by size is very often all I need to see.
I use gdu and never had any issues like that with it
Yeah, it helped me unblock my server where I ran out of space
view more: next βΊ