The pager goes off. A dashboard is timing out, memory on one node is pinned, and QPS looks fine, so this is not a flood of traffic. It is one query eating the cluster. You are not at your desk. The laptop is in the other room. The fix is usually small: find the offending query and stop it. This is a walkthrough of doing that from an iPhone with ProbeDeck, find it, judge it, kill it, and an honest account of what KILL does once you tap the button.
Step 1: Find the query in system.processes
Everything running right now lives in system.processes. It is the first place to look when something is on fire, and reading it is free: no unlock, no account, only a connection.
Each active query shows up with a query_id, the user who launched it, elapsed (how long it has been running, colour-graded by duration so the old ones jump out), memory_usage, and rows read. On a phone you sort by elapsed and the culprit is usually right at the top: one query that has been running for minutes while everything else is measured in milliseconds, holding a lot of memory and grinding through a large number of rows.
That colour grading matters more than it sounds. Half-awake, parsing a table of microsecond timings is the last thing you want. You want the one red row that says this one has been going for four minutes. That is your starting point, not yet your verdict.
Step 2: Judge before you kill
A long-running query is not automatically a wrong query. This is the step people skip at 3 a.m., and it is the one that saves you from making the incident worse.
Before you touch anything, read the row properly:
- Elapsed. Four minutes on an analytical
SELECTmight be normal. Four minutes on something that usually returns instantly is not. - Memory. A query climbing toward the node’s limit is a threat to everything else on the box.
- The user. An ETL service account running a heavy job may be a legitimate scheduled backfill. Your own ad-hoc session from an hour ago that you forgot about is a very different story.
- The SQL. This is the tell. A
SELECTwith noLIMITscanning a huge table, aJOINthat blew up, an aggregation across every partition: that is a runaway. A tidyINSERT ... SELECTdoing a planned migration is not.
The distinction that matters is legitimate backfill versus runaway ad-hoc query. Killing a backfill that was three hours into rebuilding a table is its own incident. So spend the ten seconds: the SQL preview plus the user usually tells you which one you are looking at. If you cannot tell, that is a signal to reach for the laptop or ping whoever owns that service, not to kill blindly.
Step 3: KILL QUERY, with a confirmation you can trust
Once you have decided, the action itself is deliberately a few taps, not one.
In ProbeDeck you tap the query, and a confirmation sheet comes up showing the query_id, the user, how long it has run, and a preview of the SQL: the same facts you used to make the call, amplified so you cannot misread them, on a connection carrying a PROD badge if that is where you are pointed. Production connections are badged amber so you never confuse them with staging. Behind the sheet, the app runs a plain KILL QUERY WHERE query_id='…' against exactly the query you picked.
Destructive actions sit behind a type-to-confirm step: a hard confirm where you type to proceed, not a swipe you can trigger with your thumb by accident. That guard covers KILL and the other operations that change the cluster. It exists because a phone is easy to fumble, and a mis-tap against production should be hard, not free.
A note on pricing so it is not a surprise mid-incident: killing a query is part of the one-time Pro unlock ($19.99, no subscription). The KILL button is visible on the free tier with a PRO badge, so you discover it while reading the screen, not at the exact moment you need it. Monitoring stays free: opening the app, reading system.processes, watching the cluster.
What KILL does (and does not)
This is the part most tools gloss over, and getting it wrong leads to bad decisions during an incident. ProbeDeck is deliberate about this: it never shows a fake “Undo,” because for these operations there is nothing honest to undo.
KILL QUERYasks the query to stop. It is a request, not a switch. The query winds down at a safe point and releases its resources, usually quickly, but not always instant. If it does not vanish the moment you confirm, that is expected; give it a few seconds before assuming something is wrong.KILL MUTATIONcancels a mutation but does not roll back rows already changed. If anALTER ... UPDATE/DELETEhad already rewritten part of a table, cancelling stops further work. It does not un-change what it already changed. There is no rewind.DROPandTRUNCATEare irreversible. They are not on the “stop a query” path, but the same honesty applies: once run, the data is gone, and no app can bring it back.
The practical takeaway: KILL QUERY is the safe, reversible-in-spirit action. You are stopping work, not destroying data, which is why it is the right first move against a runaway SELECT. The mutation and drop cases are where you slow down, because “stop” and “undo” are not the same thing.
After the kill: it is a tourniquet, not a cure
Killing a query frees its resources. It does not fix why the query went bad. The missing LIMIT is still missing. The cardinality blowup in that JOIN will happen again the next time someone runs it. The dashboard that generated a monster query will generate it again on the next refresh.
So treat the kill as what it is, the action that stops the bleeding so the rest of the cluster recovers, and then follow up. Note the query_id and the SQL while you have them on screen. When you are back at a laptop, that is your thread to pull: add the LIMIT, fix the join, put a per-user memory or time cap in place so the next runaway trips a guardrail instead of your pager. The phone bought you the ten minutes; the real fix still lives on the desktop.
ProbeDeck is a native iOS ClickHouse monitoring and on-call client: free monitoring for reading system.processes and watching your cluster, with a one-time Pro unlock ($19.99) for operations like killing queries. No subscription, no account, no backend. Credentials stay in the iOS Keychain, and it connects over TLS or an SSH bastion.
Related: Kill a ClickHouse query from iOS · How to monitor ClickHouse from your iPhone · Why is my ClickHouse replica lagging?