Notification Infrastructure
It’s important for significant security events to be made visible to administrators.
NTFY
I use NTFY to receive push notifications – although, as described below, I mirror those notifications to Signal.
NTFY deployment in Kubernetes: ntfy.yaml
As an example, here is how I monitor all SSH login attempts on my NixOS servers with NTFY: sshd-monitor.nix
NTFY Signal Mirror
I wrote the following tool to mirror notifications I get in specific ntfy topics to Signal: github:heywoodlh/signal-ntfy-mirror
I won’t describe how it all works, but here’s the NixOS implementation for those willing to figure it out themselves: ntfy-signal.nix
The resulting Signal alert messages look like this:
Logging
Good logging infrastructure is a critical security component.
Syslog-ng
Syslog-ng, in my opinion, is the underrated GOAT of logging. It’s minimal, yet can do so much.
Syslog-ng deployment in Kubernetes: syslog.yaml
The deployment also contains configurations for the following two tools to quickly search the logs:
Here is how looking up failed SSH login attempts with my logbash deployment looks:
❯ logbash linux ssh-failed-login
/logs/linux/2025_07_23.log:Jul 23 21:30:56 ts-syslog-6d6kv-0 <86>1 2025-07-23T15:30:56-06:00 nix-nvidia sshd-session 2354768 - - Failed keyboard-interactive/pam for invalid user fakeuser from 192.168.1.48 port 63139 ssh2
/logs/linux/2025_07_23.log:Jul 23 21:30:56 ts-syslog-6d6kv-0 <86>1 2025-07-23T15:30:56-06:00 nix-nvidia sshd-session 2354768 - - Connection closed by invalid user fakeuser 192.168.1.48 port 63139 [preauth]
/logs/linux/2025_07_23.log:Jul 23 21:30:56 ts-syslog-6d6kv-0 <30>1 2025-07-23T15:30:56-06:00 nix-nvidia s7wypw6qzlva5z7m58zl2szjq7p0c1ks-sshd-monitor 2354781 - - {"id":"xvFYOYQpzn53","time":1753306256,"expires":1753349456,"event":"message","topic":"ssh-notifications","message":"Jul 23 15:30:56 nix-nvidia sshd-session[2354768]: Failed keyboard-interactive/pam for invalid user fakeuser from 192.168.1.48 port 63139 ssh2"}
/logs/linux/2025_07_23.log:Jul 23 21:30:56 ts-syslog-6d6kv-0 <30>1 2025-07-23T15:30:56-06:00 nix-nvidia s7wypw6qzlva5z7m58zl2szjq7p0c1ks-sshd-monitor 2354785 - - {"id":"hOCAbts9lJ6X","time":1753306256,"expires":1753349456,"event":"message","topic":"ssh-notifications","message":"Jul 23 15:30:56 nix-nvidia sshd-session[2354768]: Connection closed by invalid user fakeuser 192.168.1.48 port 63139 [preauth]"}
Syslog-ng to NTFY alert/notification
I want to draw special attention to the fact that syslog-ng can run actions on specific matches. This is especially powerful when you want certain log messages to trigger events.
In my case, I want specific logs to trigger NTFY alerts. I couldn’t find any reference examples when trying to set this up, so I hope this saves anybody wanting to do this some time.
Here’s a snippet from my syslog-ng configuration that triggers an NTFY notification on a match:
source unifi_remote {
udp(ip(0.0.0.0) port(514));
};
destination ntfy {
http(
url("http://ntfy.default.svc.cluster.local/security-notifications")
method("POST")
user-agent("syslog-ng User Agent")
headers("Title: syslog-ng alert ${HOST}")
body("${ISODATE} ${MESSAGE}")
);
};
# All noteworthy ssh events
filter ssh_events {
message("(Failed password|Invalid verification code|Invalid user|Accepted publickey|Accepted password|Accepted keyboard-interactive|Failed keyboard-interactive).*");
};
# unifi ssh notifications
log {
source(unifi_remote);
filter(ssh_events);
destination(ntfy);
};
Endpoint monitoring
I use a handful of tools for monitoring my servers and workstations.
Using FleetDM Policies for determining endpoint posture
FleetDM is an open source endpoint monitoring solution built around OSQuery. OSQuery is perhaps the only solution I know of wherein one can get the state of a machine without administrative shell access and with minimal invasion of privacy.
FleetDM deployment in Kubernetes: fleetdm.yaml
MacOS FleetDM configuration
Installation of the fleetctl-generated installer for MacOS, hosted on a web server in my Tailnet: base.nix
As a Nix-Darwin and Home-Manager here’s where I configure my Mac relevant to Fleet:
- Fleet installation: github:heywoodlh/nixos-configs – darwin/roles/base.nix#L62-77
- Defaults implementation: github:heywoodlh/nixos-configs – home/modules/darwin-defaults.nix
And, here’s how the Macs show up on Fleet after they are configured:
Linux configuration
NixOS configuration for OSQuery: osquery.nix
Ubuntu servers installing the FleetDM generated package with Ansible: security.yml
Vulnerability scanning
I have at least two solutions for vulnerability scanning at the time of writing.
Nuclei for web scanning
I use Nuclei from Project Discovery for automated vulnerability scanning in my cluster.
Nuclei deployment in Kubernetes: nuclei.yaml
The features in my deployment are the following:
- Automatic enumeration of Kubernetes services as targets for Nuclei to scan
- Upload of results to projectdiscovery.io’s dashboard
Flan Scan (nmap + vulners) for network vulnerability scanning
For a simple, less web-focused vulnerability scan, I use Cloudflare’s Flan scanner which is as a wrapper around nmap and the vulners script that produces a report of the findings.
Flan scan deployment in Kubernetes: flan-scan.yaml
