Multiple instances often need different ports, release tracks and waivers. This guide shows when to use a shared command, how to keep a configuration file per instance and how to schedule the resulting checks.
- Checking a fleet of instances
- One command, several hosts
- One configuration file per instance
- A loop over the files
- Where the checks should run from
- Keeping the waivers honest
- Only alerting on what changed
- One dashboard for the whole fleet
- Scheduling the whole thing
One command, several hosts
--host takes a comma-separated list. The plugin scans them in turn, prints a
one-line summary followed by a block per host, and exits with the worst state
it found - see Checking multiple hosts.
check-opencloud-security --check-hardening \
--host opencloud1.example.com,opencloud2.example.com:9200,[2001:db8::1]
This is the right answer when the instances are alike. Everything after
--host applies to all of them, so the moment one instance needs --insecure
or a waiver the others do not, you have outgrown it.
Aggregating also loses the per-host history your monitoring system would otherwise keep. If you want one red service per broken instance rather than one red service for the group, use one check per host instead.
Add --webhook-digest here and the webhook fires at most once for the whole
--host list, instead of once per host that meets --webhook-on - useful
when a webhook receiver is a person who would rather get one message about
three broken instances than three separate pings. It only combines what
happens inside this one process: since it is one flag on one ScanContext,
it has no effect on the "configuration file per instance" and "loop over the
files" patterns below, where each instance runs as its own separate process
with nothing to combine across - each of those still sends its own webhook
per instance, exactly as without the flag.
One configuration file per instance
Each instance gets a file, and the file carries everything that makes it different. Nothing is repeated on the command line, so a change is a change in one place.
# /etc/check-opencloud-security/prod-eu.yml
host: opencloud-eu.example.com
check_hardening: true
update_warning: true
scanner:
target_port: 9200
release_track: production
ignore_hardenings:
# The reverse proxy owns this header; the instance cannot set it.
- hstsPreload
releases:
mode: auto
token: secret://releases_token
check-opencloud-security --config /etc/check-opencloud-security/prod-eu.yml
Precedence is command line > environment variable > configuration file >
default, so a per-instance file can still be overridden for one run without
editing it. The full syntax, including secret://, is in
Configuration file and secrets.
Write the first file with check-opencloud-security --configure --config
/etc/check-opencloud-security/prod-eu.yml and copy it for the rest.
A loop over the files
With one file per instance, scanning the fleet is a for loop, and the exit
codes are what you report on:
#!/bin/sh
# Scan every configured instance; exit with the worst state seen.
set -u
worst=0
rank() { case "$1" in 2) echo 3 ;; 1) echo 2 ;; 3) echo 1 ;; *) echo 0 ;; esac; }
for config in /etc/check-opencloud-security/*.yml; do
check-opencloud-security --config "$config" || state=$?
state="${state:-0}"
[ "$(rank "$state")" -gt "$(rank "$worst")" ] && worst="$state"
unset state
done
exit "$worst"
rank exists because Nagios exit codes are not ordered by severity:
CRITICAL (2) outranks WARNING (1), which outranks UNKNOWN (3), which
outranks OK (0). Sorting numerically would report a host that could not be
reached as worse than a host that is end-of-life.
Where the checks should run from
The plugin scans over the network, from wherever it runs, so where you run it decides what it can see:
- An instance behind a firewall needs a check running inside it, not a check on the monitoring server with a hole punched through.
- Whether HTTPS is enforced and whether the certificate is trusted depend on the path taken to the instance. Scanning through a load balancer that terminates TLS measures the load balancer.
- Debug-port probes only mean anything from a network that is supposed not to reach them. From inside the instance's own host they will find ports that no outsider could.
If several monitoring consumers need the same result, run the scan service close to the instances and let them share its cache. The plugin itself never talks to it - it always scans in process - so the service is for dashboards and scripts.
Keeping the waivers honest
Review ignore_hardenings entries regularly so accepted findings do not hide
new problems. Waivers have two safeguards:
- A waiver only ever suppresses the alert. The finding stays in the result
document with
"ignored": true, and--debugstill explains it - see Accepting a finding you are not going to fix. - Only a check that actually failed can be waived, so a waiver cannot silently cover a measure that later regresses into a different finding.
Review them by scanning with the waivers off and diffing:
for config in /etc/check-opencloud-security/*.yml; do
echo "== $config"
COS_SCANNER_IGNORE_HARDENINGS="" check-opencloud-security --config "$config" --debug \
| grep -E 'ignored|waived|FAIL'
done
One identifier will never appear in that list, however many instances you
run: publicLinkExpirationEnforced is hardcoded by OpenCloud and fails on
every instance, so it is recorded but excluded from
the alert, the hardenings_missing metric and the webhook. It does not need
a waiver; see
Measures that are not settings.
Only alerting on what changed
Repeated alerts for unchanged findings can make new problems easy to miss. Give each host a baseline to report regressions:
for config in /etc/check-opencloud-security/*.yml; do
check-opencloud-security --config "$config" \
--baseline /var/lib/check_opencloud/baseline.json \
--warn-on-new
done
One file is enough for the whole fleet: it stores one entry per host, keyed by
the host as it was given on the command line. A comma-separated --host list
works the same way.
Two things to get right:
- The monitoring user must own the directory. The file is written atomically with owner-only permissions, and a baseline that cannot be written is reported as a line of output and nothing more - it never changes the verdict.
- Use the same spelling of the host everywhere.
opencloud.example.comandhttps://opencloud.example.com/normalise to the same host, but10.0.0.5does not match the name that resolves to it, and the check would treat it as a host it has never seen.
A release past its end of life keeps alerting on every run regardless of the baseline, which is the point: it receives no security fixes, so it gets worse every day it stays up. See Reporting only what changed.
Add --self-update-check on one host in the fleet - not all of them - to be
told when a newer plugin version is published. It is cached for a day and
never changes the exit code.
One dashboard for the whole fleet
Monitoring answers "is this instance broken right now", one host at a time.
A fleet review asks something else: which instances run a release nobody
patches any more, which waivers run out this month, which finding is broken
everywhere, and which instance nobody has looked at lately. Keep the result
documents scan writes, and fleet answers all four from the files alone:
dir="/var/lib/opencloud-reports/$(date +%F)"
mkdir -p "$dir"
for host in opencloud1.example.com opencloud2.example.com; do
check-opencloud-scanner scan "$host" > "$dir/$host.json"
done
check-opencloud-scanner fleet /var/lib/opencloud-reports \
--inventory /etc/check-opencloud-security/hosts.txt --format html > fleet.html
It scans nothing and stores nothing; the newest report of each host is the
one that counts, and the release and the waiver deadlines are measured
against today rather than the day the report was written. An inventory
makes a host with no report at all show up as missing. See
fleet - a dashboard from saved results.
Scheduling the whole thing
- With Icinga2: one
Serviceper host, applied from a host group - see Icinga Director or Automated deployment with Ansible. - Without it: a systemd timer or cron entry running the loop above, see Scheduling.
- On a cluster: a
CronJob, see Kubernetes.
Stagger the schedules. Twenty instances scanned at 0 6 * * * means twenty
simultaneous scans from one address, and the update check will meet GitHub's
anonymous rate limit on the way.