davidandClaude Sonnet 5
6173bf3ac2
The upstream image switched from .env.local to TOML config, which silently broke the dotminipc container; record what happened and where its config lives now, since it shares the same upstream software as the Pi fleet. Co-Authored-By: Claude Sonnet 5 <[email protected]>
139 lines
10 KiB
Markdown
139 lines
10 KiB
Markdown
# dotmesh-monitor
|
|
|
|
Ansible playbooks for deploying MeshCore monitoring nodes (Raspberry Pi Zero W / Zero 2 W).
|
|
|
|
## Hosts
|
|
|
|
| Host | Hardware | Group |
|
|
|---|---|---|
|
|
| dm-baldock | Pi Zero W (armv6) | zero_w |
|
|
| dm-ashwell | Pi Zero 2 W (armv7) | zero2_w |
|
|
| dm-edworth | Pi Zero 2 W (armv7) | zero2_w |
|
|
|
|
There's also a fourth capture node, **dotminipc** (device name `dm-stotfold`) —
|
|
not managed by this repo (it's a Docker container on the shared dotnetwork
|
|
host, not a dedicated Pi), but documented below since it's the same upstream
|
|
software and shares config with the Pi fleet. See "dotminipc" below.
|
|
|
|
## Prerequisites
|
|
|
|
**Local machine:**
|
|
|
|
```bash
|
|
pip install ansible
|
|
# or: sudo apt install ansible
|
|
```
|
|
|
|
**New Pi node checklist:**
|
|
|
|
1. Flash Raspberry Pi OS Lite (Trixie) with Raspberry Pi Imager, using "Edit Settings" (OS customisation) to set hostname, the `david` user + password, and the `dotnetwork` wifi (SSID/password only — the imager only supports one network at flash time; the deployed-location and `dotmobile` networks get added later by the `wifi` role, see below).
|
|
2. **Before ejecting the card**, verify the customisation actually got written — mount the boot partition and check `network-config`/`user-data` aren't just the commented-out stock template (see "Imager gotcha" below). Only `cmdline.txt`'s regdomain getting a fresh timestamp while the rest stay at the image's build date is the tell that it silently failed.
|
|
3. Boot the Pi, find its LAN IP (e.g. from the router's DHCP leases), confirm SSH access with the password you set — no key is seeded at flash time.
|
|
4. Run `ssh-copy-id david@<ip>` **from an interactive terminal** (not through a non-interactive shell/script — it needs a real TTY to prompt for the password) so ansible can connect with a key.
|
|
5. Connect the MeshCore device via USB, then find its serial ID: `ls /dev/serial/by-id/`.
|
|
6. Add `serial_port` (and `wifi_ssid_location`) to `ansible/host_vars/<hostname>/vars.yml`, and the location wifi password to `ansible/host_vars/<hostname>/vault.yml` (see "Vault" below).
|
|
7. Run `site.yml` against just that host, overriding the host's address since Tailscale/DNS won't resolve it yet: `ansible-playbook site.yml --limit <hostname> -e "ansible_host=<ip>"`. This authorizes your SSH keys, joins the deployed-location + dotmobile wifi networks, installs (but does not authenticate) Tailscale, and deploys everything else in one pass.
|
|
8. Tailscale needs one manual step: SSH in and run `sudo tailscale up`, then open the printed URL in a browser to approve the device on your tailnet. (You *can* pass `-e tailscale_auth_key=tskey-...` — from the [admin console](https://login.tailscale.com/admin/settings/keys) — to authenticate non-interactively instead, but there's no stored key anywhere for this repo, so the interactive route is simplest for a one-off node.)
|
|
9. The meshcore-packet-capture installer also needs one manual step (see "meshcore-packet-capture install is interactive" below): SSH in and run `sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/agessaman/meshcore-packet-capture/main/install.sh)"`, answering its ~3 prompts (service account, install method — pick **1**, IATA/broker config — defaults are fine, `site.yml` overwrites `.env.local` afterward anyway).
|
|
10. Once Tailscale is up, re-run `site.yml` without the `ansible_host` override — it'll resolve via the Tailscale hostname from here on, and will now just write `.env.local` + enable the service since the installer step is already satisfied.
|
|
|
|
### meshcore-packet-capture install is interactive
|
|
|
|
The `install.sh` bootstrap has no real non-interactive path for a *fresh* install — its `--update` flag only changes behavior when an installation already exists. It also refuses to run at all with piped stdin (`curl | sudo bash` errors out asking you to download the script first) and its Python layer explicitly opens `/dev/tty` for prompts, which just hangs forever over plain SSH/ansible (no human there to answer). We tried feeding it scripted answers via `script`/a pty and it's not worth the fragility — just run it manually once per node (step 9 above); `ansible/roles/meshcore_capture/tasks/main.yml`'s `creates:` guard means ansible never touches it again afterward.
|
|
|
|
### Imager gotcha (2026-07)
|
|
|
|
The Raspberry Pi Imager available via Flathub (`org.raspberrypi.rpi-imager`) is stuck on **1.9.6** and there's no newer `.deb`/Flatpak in the Ubuntu or Flathub repos either — Flathub hasn't published the 2.0.x rewrite. 1.9.6 silently fails to apply OS customisation (hostname, user, SSH, wifi) on newer Raspberry Pi OS Trixie images: it only writes the kernel `cfg80211.ieee80211_regdom=` cmdline parameter and leaves `user-data`/`network-config` as the stock commented-out template, with no error. The result looks exactly like a wifi problem (Pi never appears on the network) but is actually "the card has no credentials on it at all."
|
|
|
|
Fix: grab the real `.deb` from the [GitHub releases page](https://github.com/raspberrypi/rpi-imager/releases) (e.g. `rpi-imager_2.0.10_amd64.deb`) and `sudo dpkg -i` it — that version writes the customisation correctly.
|
|
|
|
## Usage
|
|
|
|
**Deploy to a single host (recommended for first run / testing):**
|
|
|
|
```bash
|
|
cd ansible
|
|
ansible-playbook -i inventory.yml site.yml --limit dm-edworth
|
|
```
|
|
|
|
**Deploy to all nodes:**
|
|
|
|
```bash
|
|
ansible-playbook -i inventory.yml site.yml
|
|
```
|
|
|
|
**Dry run:**
|
|
|
|
```bash
|
|
ansible-playbook -i inventory.yml site.yml --limit dm-edworth --check
|
|
```
|
|
|
|
If sudo requires a password, add `--ask-become-pass`.
|
|
|
|
You'll be prompted for a Tailscale auth key — leave blank if the node is already authenticated.
|
|
|
|
## What it does
|
|
|
|
1. **wifi** — configures NetworkManager connections for `dotnetwork` (home), `dotmobile` (phone hotspot, field troubleshooting fallback), and the host's deployed-location network
|
|
2. **base** — apt upgrade, installs screen/pipx/vnstat/git, sets MOTD, authorizes SSH keys for both laptop partitions, installs Tailscale (always) and authenticates it (only if `tailscale_auth_key` is set — otherwise run `sudo tailscale up` manually once, see checklist above)
|
|
3. **meshcore_cli** — installs `meshcore-cli` via pipx
|
|
4. **meshcore_capture** — runs the agessaman/meshcore-packet-capture install script (skipped once already installed — see "meshcore-packet-capture install is interactive" above, this needs a manual first run), writes config (TOML `config.d/99-user.toml` on current "system"-layout nodes, legacy `.env.local` on the two nodes still on the old layout — see `meshcore_capture_layout` in `group_vars/all/vars.yml`), enables the capture service, deploys update/log helper scripts
|
|
5. **scripts** — deploys `voltage.sh` and `bandwidth.sh`
|
|
|
|
## Config
|
|
|
|
Shared MQTT config lives in `group_vars/meshcore.yml`. Per-host serial port and wifi SSID are in `host_vars/<hostname>/vars.yml`.
|
|
|
|
Running the playbook again re-applies `.env.local` and restarts the service if it changed — safe to run on already-deployed nodes.
|
|
|
|
## Vault
|
|
|
|
`group_vars/all/vault.yml` (shared wifi/SSH secrets) and `host_vars/<hostname>/vault.yml` (per-host deployed-location wifi password) are encrypted with Ansible Vault. `ansible.cfg` points at `../.vault_pass` (gitignored, not committed) for the password — ask David for a copy, or generate a fresh one and re-encrypt if starting over:
|
|
```bash
|
|
ansible-vault view --vault-password-file ../.vault_pass group_vars/all/vault.yml
|
|
ansible-vault edit --vault-password-file ../.vault_pass host_vars/dm-edworth/vault.yml
|
|
```
|
|
`*/vault.yml.example` shows the expected keys.
|
|
|
|
`group_vars/meshcore.yml` (MQTT credentials) is still plaintext — consider moving it into the vault too if this repo is shared further.
|
|
|
|
## dotminipc (Docker node, not managed by this repo)
|
|
|
|
A fourth capture point, device name `dm-stotfold`, runs as a Docker container
|
|
(`ghcr.io/agessaman/meshcore-packet-capture:latest`) on **dotminipc**
|
|
(172.16.31.92), the shared Docker/HA host documented in the sibling
|
|
`dotnetwork` repo. It isn't part of this repo's inventory — no Ansible role
|
|
here touches it — but it's the same upstream software as the Pi fleet, so
|
|
config drift between the two is worth knowing about.
|
|
|
|
- Compose file: `dotnetwork/docker/dotminipc/meshcore-packet-capture/docker-compose.yml`
|
|
- Config: `dotnetwork/docker/dotminipc/meshcore-packet-capture/config.d/99-user.toml`
|
|
— host-only, gitignored (contains MQTT credentials), same TOML shape as
|
|
this repo's `meshcore_capture` role template
|
|
(`ansible/roles/meshcore_capture/templates/99-user.toml.j2`); keep the
|
|
broker list in sync between the two if either changes. The committed
|
|
`config.d/99-user.toml.example` in the dotnetwork repo is the sanitized
|
|
reference copy.
|
|
- Deploy/restart: `ssh [email protected]`, then from `/opt/docker`:
|
|
`docker compose up -d meshcore-capture --force-recreate`. Logs:
|
|
`docker logs -f meshcore-packet-capture`.
|
|
|
|
**2026-07-27 outage, for reference**: the `:latest` image was rebuilt
|
|
2026-07-25 and switched its config format from `.env.local` to TOML
|
|
(`/etc/meshcore-packet-capture/config.toml` + `config.d/*.toml` —
|
|
the same "system" layout `meshcore_capture_layout` already models for the Pi
|
|
fleet). It silently stopped reading `.env.local` at all — despite upstream's
|
|
own README still describing that as a supported "legacy" path, in practice it
|
|
was just ignored — so the container fell back to the image's baked-in
|
|
defaults (`connection_type = "ble"`, `serial.ports = ["/dev/ttyUSB0"]`, no
|
|
owner key, only 2 of 5 brokers) and crash-looped on
|
|
`could not open port /dev/ttyUSB0`. Fix was to mount a full TOML override at
|
|
`config.d/99-user.toml` instead (see above) — same pattern as the Pi
|
|
`meshcore_capture` role already uses for `system`-layout nodes. Also found in
|
|
the process: the old `.env.local` had `PACKETCAPTURE_MQTT5_SERVER` set to
|
|
`mqtt.meshmapper.cc`, which fails TLS handshake — the correct domain is
|
|
`mqtt.meshmapper.net` (confirmed via `openssl s_client`; `.net` presents a
|
|
valid Let's Encrypt cert, `.cc` returns a TLS alert). If the Pi fleet's
|
|
`99-user.toml.j2` broker list is ever regenerated from a stale copy of this
|
|
node's old config, watch out for that typo resurfacing.
|