DevOps

2099 readers

6 users here now

DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means for improving and shortening the systems development life cycle.

Rules:

Posts must be relevant to DevOps
No NSFW content
No hate speech, bigotry, etc
Try to keep discussions on topic
No spam of tools/companies/advertisements
It’s OK to post your own stuff part of the time, but the primary use of the community should not be promotional content.

Icon base by Lorc under CC BY 3.0 with modifications to add a gradient

founded 2 years ago

MODERATORS

Ategon@programming.dev

RandomDevOpsDude@programming.dev

Hard to achieve memory efficiency in Kubernetes clusters? (lemmy.world)

submitted 2 months ago by weastie@lemmy.world to c/devops@programming.dev

4 comments fedilink hide all child comments

When setting resources on Kubernetes pods, I'm finding it very difficult to achieve good memory efficiency.

I'm using the "no CPU limits, set memory=limits" philosophy that I hear heavily recommended on the internet.

The problem is, that many pods will have random memory spikes. In order for them not to be OOM Killed, you have to set the memory requests for them above their highest spike, which means most of the time they're only using like 25% or so of their memory allocation.

I've been trying to optimize this one cluster, and on average I'm only getting 33% of the total memory requested for all the pods in the cluster actually being used. Whenever I try decreasing some pod's memory requests, I eventually get OOMs. I was hoping I could reach closer to 50%, considering that this particular cluster has a stable workload.

I'm sure that I could optimize it a bit better, but not by much.

Is this a shared experience in Kubernetes? That you ultimately have to sacrifice a lot of memory efficiency.

you are viewing a single comment's thread
view the rest of the comments

[–] devtoolkit_api@discuss.tchncs.de 1 points 1 hour ago

Memory efficiency in K8s is really about getting your requests and limits right. A few things that helped us:

Use VPA (Vertical Pod Autoscaler) in recommend mode first to see what pods actually use vs what they request
Set requests based on P99 usage, not peak theoretical usage
Goldilocks is a great open-source tool that automates VPA recommendations across namespaces
For JVM workloads, -XX:MaxRAMPercentage=75 with container-aware GC flags makes a big difference

The biggest waste we found was pods requesting 1Gi but using 200Mi on average. Multiply that by 100 pods and you are wasting a lot of cluster capacity.