On-call & monitoring

The state of prod, before anyone pages.

#monitoring-prodprivate channel2 tool calls

No alert, just a question: what is erroring in production right now? It groups the last hours of the production logs by error and namespace, says which of it is the application and which is infrastructure noise, and hands over the query for each so the next person can re-run it.

Try it now Deploy it yourself GCP presets

monitoring-prodthread

Dan11:02 AM

@attest_tag what's erroring in prod right now?

attest_tagApp11:03 AM

gcp_query_logs · severity>=ERROR · last 6h · 200 lines
run_js · group by namespace, message · 4 groups

No single application failure in the last six hours. What is erroring is infrastructure:

Prometheus, monitoring namespace — “out of order sample” on remote write, by far the most frequent: samples arriving with timestamps older than the series already has. Noise unless the metrics themselves are missing.
A controller retrying a reconcile every few minutes — one object, the same error each time.
Two application errors, both one-offs, both retried successfully.

The log query for each group is below, so it can be re-run.

google/gemini-flash8.4k in530 out$0.0046Configure

Names, figures, repositories and customers are changed; token counts and cost are illustrative. The ask, and the shape of the answer, are as they were.

How the turn went

One broad query, grouped in the sandbox, with the noise named.

01
One query, broad on purpose
Everything at ERROR and above for six hours, across the cluster. Two hundred lines is a lot to read and nothing to compute over, which is what the next step is for.
02
Grouping happens in the sandbox
The lines go into run_js as JSON and come back as four groups with counts. No arithmetic in the model’s head, and no second query to work out what the first one said.
03
The answer names what is noise
A monitoring stack complaining about itself is the most common thing in a production log. It says so, and keeps the two real errors visible.

What it needs

The setup, in the console.

Connection

The GCP logs preset, read-only, attached to the channel — or to the workspace, if every channel may read the logs. The Sentry and Datadog presets serve the same ask from their own stores.

Credential

A service account key with the Viewer role for logs or monitoring on the project. Sealed with AES-256-GCM in the database; the proxy signs each request, and the key is never in a prompt.

Tools it used

gcp_query_logs from the GCP preset's tool pack, and run_js, which every channel has.

What it will not do

Enforced outside the model.

It will not silence, acknowledge or change anything: the presets are read-only, and the bot has no tool that touches the cluster.
Reads never wait for a Confirm. A channel that may reach the logs may look at them; holding reads would train everyone to press Confirm without reading it.

Try it

Say it in a channel it is in.

The sentence as it was typed. It works in any channel the bot has been invited to that carries the connection above — no command syntax, no form, and follow-ups in the thread need no mention.

@attest_tag what's erroring in prod right now?

gcp_query_logs
run_js

The alert lands. So does the answer.

Bring the channel this would live in.

Sign up, connect the service above in one channel, and type this sentence yourself — or run the same binary on your own infrastructure.

Try it now Deploy it yourself

All solutions