# dotmesh-monitor

Ansible playbooks for deploying MeshCore monitoring nodes (Raspberry Pi Zero W / Zero 2 W).

## Hosts

| Host | Hardware | Group |
|---|---|---|
| dm-baldock | Pi Zero W (armv6) | zero_w |
| dm-ashwell | Pi Zero 2 W (armv7) | zero2_w |
| dm-edworth | Pi Zero 2 W (armv7) | zero2_w |

There's also a fourth capture node, **dotminipc** (device name `dm-stotfold`) —
not managed by this repo (it's a Docker container on the shared dotnetwork
host, not a dedicated Pi), but documented below since it's the same upstream
software and shares config with the Pi fleet. See "dotminipc" below.

## Prerequisites

**Local machine:**

```bash
pip install ansible
# or: sudo apt install ansible
```

**New Pi node checklist:**

1. Flash Raspberry Pi OS Lite (Trixie) with Raspberry Pi Imager, using "Edit Settings" (OS customisation) to set hostname, the `david` user + password, and the `dotnetwork` wifi (SSID/password only — the imager only supports one network at flash time; the deployed-location and `dotmobile` networks get added later by the `wifi` role, see below).
2. **Before ejecting the card**, verify the customisation actually got written — mount the boot partition and check `network-config`/`user-data` aren't just the commented-out stock template (see "Imager gotcha" below). Only `cmdline.txt`'s regdomain getting a fresh timestamp while the rest stay at the image's build date is the tell that it silently failed.
3. Boot the Pi, find its LAN IP (e.g. from the router's DHCP leases), confirm SSH access with the password you set — no key is seeded at flash time.
4. Run `ssh-copy-id david@<ip>` **from an interactive terminal** (not through a non-interactive shell/script — it needs a real TTY to prompt for the password) so ansible can connect with a key.
5. Connect the MeshCore device via USB, then find its serial ID: `ls /dev/serial/by-id/`.
6. Add `serial_port` (and `wifi_ssid_location`) to `ansible/host_vars/<hostname>/vars.yml`, and the location wifi password to `ansible/host_vars/<hostname>/vault.yml` (see "Vault" below).
7. Run `site.yml` against just that host, overriding the host's address since Tailscale/DNS won't resolve it yet: `ansible-playbook site.yml --limit <hostname> -e "ansible_host=<ip>"`. This authorizes your SSH keys, joins the deployed-location + dotmobile wifi networks, installs (but does not authenticate) Tailscale, and deploys everything else in one pass.
8. Tailscale needs one manual step: SSH in and run `sudo tailscale up`, then open the printed URL in a browser to approve the device on your tailnet. (You *can* pass `-e tailscale_auth_key=tskey-...` — from the [admin console](https://login.tailscale.com/admin/settings/keys) — to authenticate non-interactively instead, but there's no stored key anywhere for this repo, so the interactive route is simplest for a one-off node.)
9. The meshcore-packet-capture installer also needs one manual step (see "meshcore-packet-capture install is interactive" below): SSH in and run `sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/agessaman/meshcore-packet-capture/main/install.sh)"`, answering its ~3 prompts (service account, install method — pick **1**, IATA/broker config — defaults are fine, `site.yml` overwrites `.env.local` afterward anyway).
10. Once Tailscale is up, re-run `site.yml` without the `ansible_host` override — it'll resolve via the Tailscale hostname from here on, and will now just write `.env.local` + enable the service since the installer step is already satisfied.

### meshcore-packet-capture install is interactive

The `install.sh` bootstrap has no real non-interactive path for a *fresh* install — its `--update` flag only changes behavior when an installation already exists. It also refuses to run at all with piped stdin (`curl | sudo bash` errors out asking you to download the script first) and its Python layer explicitly opens `/dev/tty` for prompts, which just hangs forever over plain SSH/ansible (no human there to answer). We tried feeding it scripted answers via `script`/a pty and it's not worth the fragility — just run it manually once per node (step 9 above); `ansible/roles/meshcore_capture/tasks/main.yml`'s `creates:` guard means ansible never touches it again afterward.

### Imager gotcha (2026-07)

The Raspberry Pi Imager available via Flathub (`org.raspberrypi.rpi-imager`) is stuck on **1.9.6** and there's no newer `.deb`/Flatpak in the Ubuntu or Flathub repos either — Flathub hasn't published the 2.0.x rewrite. 1.9.6 silently fails to apply OS customisation (hostname, user, SSH, wifi) on newer Raspberry Pi OS Trixie images: it only writes the kernel `cfg80211.ieee80211_regdom=` cmdline parameter and leaves `user-data`/`network-config` as the stock commented-out template, with no error. The result looks exactly like a wifi problem (Pi never appears on the network) but is actually "the card has no credentials on it at all."

Fix: grab the real `.deb` from the [GitHub releases page](https://github.com/raspberrypi/rpi-imager/releases) (e.g. `rpi-imager_2.0.10_amd64.deb`) and `sudo dpkg -i` it — that version writes the customisation correctly.

## Usage

**Deploy to a single host (recommended for first run / testing):**

```bash
cd ansible
ansible-playbook -i inventory.yml site.yml --limit dm-edworth
```

**Deploy to all nodes:**

```bash
ansible-playbook -i inventory.yml site.yml
```

**Dry run:**

```bash
ansible-playbook -i inventory.yml site.yml --limit dm-edworth --check
```

If sudo requires a password, add `--ask-become-pass`.

You'll be prompted for a Tailscale auth key — leave blank if the node is already authenticated.

## What it does

1. **wifi** — configures NetworkManager connections for `dotnetwork` (home), `dotmobile` (phone hotspot, field troubleshooting fallback), and the host's deployed-location network
2. **base** — apt upgrade, installs screen/pipx/vnstat/git, sets MOTD, authorizes SSH keys for both laptop partitions, installs Tailscale (always) and authenticates it (only if `tailscale_auth_key` is set — otherwise run `sudo tailscale up` manually once, see checklist above)
3. **meshcore_cli** — installs `meshcore-cli` via pipx
4. **meshcore_capture** — runs the agessaman/meshcore-packet-capture install script (skipped once already installed — see "meshcore-packet-capture install is interactive" above, this needs a manual first run), writes config (TOML `config.d/99-user.toml` on current "system"-layout nodes, legacy `.env.local` on the two nodes still on the old layout — see `meshcore_capture_layout` in `group_vars/all/vars.yml`), enables the capture service, deploys update/log helper scripts
5. **scripts** — deploys `voltage.sh` and `bandwidth.sh`

## Config

Shared MQTT config lives in `group_vars/meshcore.yml`. Per-host serial port and wifi SSID are in `host_vars/<hostname>/vars.yml`.

Running the playbook again re-applies `.env.local` and restarts the service if it changed — safe to run on already-deployed nodes.

## Vault

`group_vars/all/vault.yml` (shared wifi/SSH secrets) and `host_vars/<hostname>/vault.yml` (per-host deployed-location wifi password) are encrypted with Ansible Vault. `ansible.cfg` points at `../.vault_pass` (gitignored, not committed) for the password — ask David for a copy, or generate a fresh one and re-encrypt if starting over:
```bash
ansible-vault view --vault-password-file ../.vault_pass group_vars/all/vault.yml
ansible-vault edit --vault-password-file ../.vault_pass host_vars/dm-edworth/vault.yml
```
`*/vault.yml.example` shows the expected keys.

`group_vars/meshcore.yml` (MQTT credentials) is still plaintext — consider moving it into the vault too if this repo is shared further.

## dotminipc (Docker node, not managed by this repo)

A fourth capture point, device name `dm-stotfold`, runs as a Docker container
(`ghcr.io/agessaman/meshcore-packet-capture:latest`) on **dotminipc**
(172.16.31.92), the shared Docker/HA host documented in the sibling
`dotnetwork` repo. It isn't part of this repo's inventory — no Ansible role
here touches it — but it's the same upstream software as the Pi fleet, so
config drift between the two is worth knowing about.

- Compose file: `dotnetwork/docker/dotminipc/meshcore-packet-capture/docker-compose.yml`
- Config: `dotnetwork/docker/dotminipc/meshcore-packet-capture/config.d/99-user.toml`
  — host-only, gitignored (contains MQTT credentials), same TOML shape as
  this repo's `meshcore_capture` role template
  (`ansible/roles/meshcore_capture/templates/99-user.toml.j2`); keep the
  broker list in sync between the two if either changes. The committed
  `config.d/99-user.toml.example` in the dotnetwork repo is the sanitized
  reference copy.
- Deploy/restart: `ssh david@172.16.31.92`, then from `/opt/docker`:
  `docker compose up -d meshcore-capture --force-recreate`. Logs:
  `docker logs -f meshcore-packet-capture`.

**2026-07-27 outage, for reference**: the `:latest` image was rebuilt
2026-07-25 and switched its config format from `.env.local` to TOML
(`/etc/meshcore-packet-capture/config.toml` + `config.d/*.toml` —
the same "system" layout `meshcore_capture_layout` already models for the Pi
fleet). It silently stopped reading `.env.local` at all — despite upstream's
own README still describing that as a supported "legacy" path, in practice it
was just ignored — so the container fell back to the image's baked-in
defaults (`connection_type = "ble"`, `serial.ports = ["/dev/ttyUSB0"]`, no
owner key, only 2 of 5 brokers) and crash-looped on
`could not open port /dev/ttyUSB0`. Fix was to mount a full TOML override at
`config.d/99-user.toml` instead (see above) — same pattern as the Pi
`meshcore_capture` role already uses for `system`-layout nodes. Also found in
the process: the old `.env.local` had `PACKETCAPTURE_MQTT5_SERVER` set to
`mqtt.meshmapper.cc`, which fails TLS handshake — the correct domain is
`mqtt.meshmapper.net` (confirmed via `openssl s_client`; `.net` presents a
valid Let's Encrypt cert, `.cc` returns a TLS alert). If the Pi fleet's
`99-user.toml.j2` broker list is ever regenerated from a stale copy of this
node's old config, watch out for that typo resurfacing.