Ask Lemmy

INC-224, never forget.

I am an infra engineer at a fairly large scale (not like Amazon, but we have some BIG customers) SaaS company; despite our scale, we are only like 250 people and of them only about 90 engineers. We store a bunch of data in MySQL.

15:30:00, I get a page “MySQL table is full.” I immediately know my day is ruined, since I’ve never heard of this error before, but know it ain’t great.

15:30:10, every Pagerduty escalation policy in the entire company gets bombarded with pages.

I look at the database instance. The table size is “only” 16TiB, so it’s a bit confusing.

We are hard down for several hours as we scramble to delete data or somehow free up space. Turns out, google backs ClpudSQL MySQL instances with ext4 disks instead of zfs, and the max file size on ext4 is… you guessed it, 16TiB.

We learned a LOT of lessons from this, and are now offloading a shitload of json into either MongoDB or gcs, depending on the requirements. The largest table is down to 3TiB now :D

My company called all lab staff "pandemic heroes" for coming in every day during the pandemic and taking on extra work to compensate for management and office staff who stayed home for years.

Then shortly after return to office, they closed the lab and laid off all lab staff.

An IT contractor at my government job was one of the people that tried to kidnap Michigan governor Gretchen Whitmer.

https://www.nbcnews.com/politics/politics-news/wisconsin-man-pleads-guilty-michigan-gov-whitmer-kidnap-plot-rcna75191

Worked at a place where our CIO was completely unqualified to be a leader, much less a leader in IT. She was a micromanager who took the position of "telling stakeholders" instead of "working with stakeholders" so any project she was on was really her pushing through whatever agenda she had at the time. Meanwhile her deputy CIO was stealing computer equipment from the server room but I digress....

April fools one year and I decide to prank it up. I moved the hinges (not the door handles) of the freezer/fridge in the breakroom so that the handle and hinges were on the same side. It's a fifteen minute job to move everything so I did it the night before the 1st.

The next morning our hungover CIO stumbles into the breakroom and cannot get the fridge to open. After a few seconds of futile tugging on the handle, she gave up and took her lunch to her office.

Others in the office figured it out pretty quickly and had a good chuckle.

Later on that day CIO sends out a nastygram about pranks being unprofessional, property damage, someone was going to be in huge trouble, yadda yadda....

But she's not the director. The director tells her to basically fuck off, it was a funny prank, and perhaps she needed to lighten up.

She never found out it was me.

Girl did dabs on break with her gf came back zonked out since she'd never smoked weed before.

Ended up slapping manager and getting taken away by ems

Cook got arrested at work one time when cops came to pick her up at her job. She was 4 feet tall so we joked they picked her up and carried her away. She had to use a step stool to make the soup and someone would hide the stool from her so she'd be pissed the next morning.

Same place had a cook drinking lean and offering it to people.

Retirement home btw

Software company before git. The source server corrupted and the product code was lost. 5 guys had to get together and figure out the latest version between them (everybody had different changesets) and produce a new "current" version. At the end we lost all history prior and ever since all changes prior to 2008 have been attributed to 1 guy.