Site downtime alerts

Graphs in the control panel show the state of the machine, but they don't answer the question “is the site up?” The application can hang while the CPU is idle, and without an external check you'll learn about it from your clients.

External checks

The main rule: don't check from the machine you're monitoring. Use any external uptime monitoring service — there are many, including free plans for several addresses with a one-minute interval. The settings that really matter:

• Check address. Not the home page, but a separate route like /healthz that checks both the database and the queue. Otherwise, a cached home page will be served even when the backend is dead.
• Expected response. A 200 code plus a substring in the body. Many applications return 200 with an error message when they break.
• Threshold. Two or three failed attempts in a row, otherwise any network hiccup will wake you up.
• Channel. Telegram or email. Make sure the alert actually arrives: disable the service for a minute and wait for the message.

Don't forget a separate certificate expiry check — an expired TLS certificate looks like an outage and happens regularly.

Watchdog on the machine itself

External monitoring will tell you that the site is down. To make it come back up on its own, configure systemd. In the service unit:

[Service]
Restart=always
RestartSec=5
StartLimitBurst=5
StartLimitIntervalSec=120

This way the service will survive an accidental crash, but won't loop endlessly if the configuration is broken. After editing: systemctl daemon-reload && systemctl restart myapp.

For applications that don't crash but hang, add an HTTP check. A pair of units — a timer and a service:

cat >/etc/systemd/system/healthcheck.service <<'EOF'
[Service]
Type=oneshot
ExecStart=/usr/local/bin/healthcheck.sh
EOF

cat >/etc/systemd/system/healthcheck.timer <<'EOF'
[Timer]
OnUnitActiveSec=60
[Install]
WantedBy=timers.target
EOF

systemctl enable --now healthcheck.timer

The script itself:

#!/bin/sh
curl -fsS --max-time 10 http://127.0.0.1:8080/healthz >/dev/null || systemctl restart myapp

Machine health alerts

It's also useful to learn about running out of disk space and memory before an outage. The simplest option is a cron job that sends a Telegram message when a threshold is exceeded:

0 * * * * root [ "$(df --output=pcent / | tr -dc 0-9)" -gt 85 ] && curl -s -X POST "https://api.telegram.org/bot$TOKEN/sendMessage" -d chat_id=$CHAT -d text="VDS disk usage is over 85%"

A more advanced option with ready-made thresholds is provided by Netdata.

Common mistakes

• Monitoring on the same machine. If the machine goes down, both the watchdog and the alerting die.
• Too sensitive a threshold. A stream of false alarms means you'll scroll past the real one.
• Restart=always instead of investigating the cause. Restarting masks a memory leak: the service crashes every half hour, and you don't see it. Check systemctl show myapp -p NRestarts and the journal.
• Checking only via ping. The machine responds to ICMP even with a completely dead web server.

The first thing to check when an alert fires is “Overview” in the control panel /hosting/vds/m/<id>. A break in the graphs means a problem with the machine; flat graphs when the site is unavailable mean a problem in the application.