Understanding alerts

An alert is a message sent because an incident opened, or because one ended. It is not the monitoring thinking out loud: everything that reaches you was confirmed first.

What raises one#

A check fails. That alone raises nothing — a single lost packet is not an outage. The check is run again a few seconds later, and again after that; by default the re-checks happen 15 and 45 seconds after the first failure, and two failures in a row are needed before an incident is opened. The admin can set those numbers otherwise.

When the failure holds, the incident is confirmed, and that is the moment the alert leaves.

An incident carries one of two levels of attention:

• critical — a host's core service stopped. The machine, or something the machine's other work rests on. This is the level that is meant to wake somebody.
• warning — a site failed, or an advisory check found something. Attention, not a fire.

That second point surprises people, so it is worth saying plainly: a site of yours being down is a warning, not a critical. On the overview, it is counted under Open warnings and not under Open alerts. That is not a judgement about your site. It is a judgement about the difference between one site being unreachable and the machine carrying two hundred of them being unreachable.

Why an alert can arrive after the fault#

Four reasons, all deliberate.

A failure is confirmed before it is announced. That is the delay described above: seconds, not minutes, and it is what keeps a lost packet or one bad second from becoming a message on your phone.

A check runs on an interval. Nothing is watched continuously — that would hammer the very machines being watched. By default a site's page is fetched every five minutes, a host is pinged every ten seconds, DNS is read every quarter of an hour, a certificate every six hours, and a domain's registration once a day. So the worst case for a site's outage is roughly one interval plus the confirmation. For a certificate this matters not at all: what is being watched there is a date, weeks away.

One alert per cause. When a host stops answering its ping — nothing at all comes back, which is not the same as a few packets lost — every site on it, and every service of its own (web server, database, ports), fails at the same second. You get one message that names the host and how many sites it carries; the other incidents still open and are still recorded, listed underneath the host's, and they send nothing. Before a check on a machine is confirmed, the machine is pinged first if it has not answered in the last half minute: a few seconds, so that the cause is known before its consequences are announced. When the host answers again, everything that folded under it is checked again at once: what is back closes without a message, as it folded; what still fails is announced then, as a fault of its own. In the history, the host's incident carries how many folded under it; with folded lists them on their own lines. The same holds inside a site: a name has to resolve before a page can answer, and a page has to answer before its certificate means anything, so one falling site sends one alert and not five. If you ever receive nothing while your site is clearly down, look at the host: the message went out about it.

Nothing is sent while the monitoring is blind. If the machine Webmonit runs on loses its own network, everything looks down from where it sits. It says so, once, on its own pages, and sends no alert at all until it can see again.

What the message says#

The first word of the title is the news:

• DOWN — a site or a check has failed and it is confirmed.
• HOST DOWN — a machine answers nothing at all: its ping gets no reply. One of its services failing — a mail port, its web server — is DOWN with that service named, critical all the same.
• WARNING — something that is not a failure needs attention: a certificate nearing its expiry, a drive ageing, an array rebuilding. Nothing stopped answering.
• UP — it is back. It says how long it was gone.
• RESOLVED — the warning is over. Nothing had stopped, so nothing "came back up".
• STILL DOWN — a reminder about a fault nobody has acknowledged yet, when reminders are turned on.
• CHANGED — a fault somebody had acknowledged has turned into a different fault, so the acknowledgement was lifted and this needs looking at again.
• CLOSED — and this one is not good news. It means the check stopped being run — replaced, paused or removed — so its incident was closed. Nothing recovered; the question stopped being asked.

Under the title, one fact per line: which check, what it does, the site, the host, the account, the error in the checker's own words, how many sites the host carries when a whole machine is down, and since when. When the admin has set a public address for the interface, the message also carries a link straight to the incident.

Repeating an alert#

By default, a fault is said once, when it happens, and once more when it ends. An admin who wants an unacknowledged critical repeated can turn it on in Channels — every thirty minutes, every hour or every four hours — and it is then repeated for as long as it is neither acknowledged nor resolved. Those are the STILL DOWN messages, and they carry how long it has been down and that nobody has taken it in hand: worth it when one message at three in the morning may go unheard, noise when the person reading is already on it.

A warning does not repeat. It sends one message when it opens and one when it ends.

Acknowledging#

To acknowledge an incident is to write down: I have seen this, I have it in hand. It records who, when, and a note — what was seen, what is being done.

It does three things and no more. It stops the reminders, when they are on. It moves the fault out of the "waiting to be noticed" count and into the "in hand" one. And it leaves the note where the next person to look will find it.

It does not resolve the incident. The incident stays open until the thing actually recovers, and the site stays down until it is up. And if what is broken changes into a different fault, the acknowledgement is lifted on its own and a CHANGED message goes out — because what was acknowledged is not what is happening now.

Acknowledging needs an account that is allowed to write. A read-only account cannot acknowledge, but sees on every open incident who acknowledged it, when, and what they wrote. That is usually the answer to "does anyone know about this yet".

Maintenance windows#

A maintenance window is a period declared in advance during which failures are expected: a migration, a nightly backup that slows a machine, a version upgrade.

While a window is open, the checks keep running and the results keep being recorded — the history is not falsified — but the covered targets read as maintenance rather than down, and no alert goes out. A window can cover one site, a whole host and its sites, a customer's whole tree, or everything. It can be one-off or repeat daily at a set hour, which is the right answer for a backup that loads a machine every night.

A window declared after the fact works the same way: a host that fell three minutes before you declared the window reads as maintenance from then on, nothing rings for it again while the window is open, and if its outage had been announced, its recovery is announced too — a channel that heard "down" hears "up". Two details are worth knowing. If a fault outlives its window, the alert that was held back is delivered when the window closes: a site that fell during the maintenance and never came back is an outage nobody has been told about, and Webmonit will not leave it that way. And a window can be marked public, in which case its title and its note are announced on the status pages, before it starts and while it runs.

Declaring a window needs an account allowed to write. Windows are visible in the interface either way, and a site covered by one shows blue and says so.

What a window covered counts as maintenance, not downtime: on the status pages its hours show blue and stay out of the availability figure, and so do the days rolled up. This is decided when the history is read, so a window declared after the fact — the provider's intervention you learned about when it started — or put on the right day afterwards counts just the same. A window's dates, title, description and whether it is public can be edited at any time from the windows page; what it covers cannot: a window on another machine is another window. Alerts already sent before the window was declared or corrected cannot be called back.

Choosing what you are told about#

Under My alerts, an account decides what reaches it and where. Two pieces:

A channel is a place a message can be sent — an email address, a phone through ntfy, a Discord or Matrix room, a webhook into something of your own. Your own channels are yours whatever your access: a read-only account may add them, test them and remove them, because they change nothing about the monitoring itself. If your account has an email address on it, that works with no channel at all.

A subscription says which subject goes to which channel, and from which level up: everything you can see, one host, or one site; and warnings and worse, outages only, critical only, or everything, notices included. With no subscription you are told nothing — the page says so.

None of this touches the admin's own alerting. What you subscribe to is in addition to what they already receive.

What to do when one arrives#

Read the first word. It tells you whether something is broken (DOWN, HOST DOWN), whether something needs attention before it breaks (WARNING), or whether the news is good (UP, RESOLVED).

Open the link. The incident page carries the cause in the checker's own words, the results around the failure, and the evidence — what the site actually answered while it was failing: the status line, the headers, the first kilobytes of the page. That is where "why" lives.

Look at whether it is folded under another incident. If the page says the incident is folded under a host's, the machine is what broke, the admin already has one message about it naming every site affected, and the thing to do is wait for the host.

Look at whether somebody acknowledged it, and read their note.

If it is a WARNING about a date — a certificate running out, a domain registration approaching — nothing is broken yet and there is time. That is exactly what the warning is for. But it will become an outage on the day named if nobody acts, and the person who can act is sometimes you rather than the admin.

If you cannot act on it yourself, tell the admin, and give them the incident number. Every incident has one, it is in the message and at the top of its page, and it saves both of you the hunt.