I’ve been switching my company over to Prometheus, and I’ve come across a few things that need discussion and opinions.<\/p>\n
First, concrete advice:
\nDon’t just write an alert like
\n“`
\nalert: foo
\nexpr: sum(rate(bar[5m])) > 5
\n“`
\nWrite it so you record the rate, and then alert on that metric:
\n“`
\nrecord: bar:rate
\nexpr: sum(rate(bar[5m]))
\nalert: foo
\nexpr: bar:rate > 5
\n“`<\/p>\n
From my Google days, I can say I should probably specify what the time is on that rate.<\/p>\n
Questions:
\n1) How long should the rate window be? [5m]? [2m]? 3? 10?
\n* I’ve adopted 5m as standard across my company, being a compromise between being fast-moving and not overly smoothed
\n2) How long should alert `for`s be?
\n3) Metric naming
\n* I’m using `A_Metric_Name`; Not sure if this is right
\n4) Recorded rule naming
\n* I like `product:metric[:submetric]:unit` ; eg. houseparty:websockets_open:byDeviceType:sum<\/p>\n","protected":false},"excerpt":{"rendered":"
I’ve been switching my company over to Prometheus, and I’ve come across a few things that need discussion and opinions. First, concrete advice: Don’t just write an alert like “` alert: foo expr: sum(rate(bar[5m])) > 5 “` Write it so you record the rate, and then alert on that metric: “` record: bar:rate expr: sum(rate(bar[5m])) … Continue reading Prometheus alerting and questions<\/span>