Chaos space marine from wathammer 40k

Updated post with link

Updated post with link

It’s not. I’m literally wearing it as we speak.

That’s not really… possible at this point. We have thousands of customers (some very large ones, like A——n and G—-e and Wal___t) with tens or hundreds of millions of users, and even at lowest traffic periods do 60k+ queries per second.

This is the same MySQL instance I wrote about a while ago that hit the 16TiB table size limit (due to ext4 file system limitations) and caused a massive outage; worst I’ve been involved in during my 26 year career.

Every day I am shocked at our scale, considering my company is only like 90 engineers.

Just had to restart our main MySQL instance today. Had to do it at 6am since that’s the lowest traffic point, and boy howdy this resonates.

2 solid minutes of the stack throwing 500 errors until the db was back up.

“Too fucking bad”

Rofl. My dad is WHY I do this.

Fair point. I usually just correct people when they are talking about cement concrete. I’ve never actually heard of “asphalt concrete”

I just inherited this quirk from my dad is all

Well technically the tanks do heat the water that is already hot, to maintain the temp.

But I do get your point; I myself say that sometimes by accident.

It’s concrete, not cement. (Sidewalks for example, or foundations of buildings, etc)

Cement is an ingredient in concrete.

INC-224, never forget.

I am an infra engineer at a fairly large scale (not like Amazon, but we have some BIG customers) SaaS company; despite our scale, we are only like 250 people and of them only about 90 engineers. We store a bunch of data in MySQL.

15:30:00, I get a page “MySQL table is full.” I immediately know my day is ruined, since I’ve never heard of this error before, but know it ain’t great.

15:30:10, every Pagerduty escalation policy in the entire company gets bombarded with pages.

I look at the database instance. The table size is “only” 16TiB, so it’s a bit confusing.

We are hard down for several hours as we scramble to delete data or somehow free up space. Turns out, google backs ClpudSQL MySQL instances with ext4 disks instead of zfs, and the max file size on ext4 is… you guessed it, 16TiB.

We learned a LOT of lessons from this, and are now offloading a shitload of json into either MongoDB or gcs, depending on the requirements. The largest table is down to 3TiB now :D

Fucked around, found out.

Yeah I’ve been using it for years. Works fine!

There are nicer ways to say this.

I’ll pass

Jesus fucking Christ.

100% possible, it’s how adguard and piholes work

Yes, duh.

No fuckin’ thanks.