Document dotminipc Docker capture node and its 2026-07-27 config-format outage

The upstream image switched from .env.local to TOML config, which silently
broke the dotminipc container; record what happened and where its config
lives now, since it shares the same upstream software as the Pi fleet.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

README.md

@@ -10,43 +10,60 @@ Ansible playbooks for deploying MeshCore monitoring nodes (Raspberry Pi Zero W /
 | dm-ashwell | Pi Zero 2 W (armv7) | zero2_w |
 | dm-edworth | Pi Zero 2 W (armv7) | zero2_w |
 
+There's also a fourth capture node, **dotminipc** (device name `dm-stotfold`) —
+not managed by this repo (it's a Docker container on the shared dotnetwork
+host, not a dedicated Pi), but documented below since it's the same upstream
+software and shares config with the Pi fleet. See "dotminipc" below.
+
 ## Prerequisites
 
 **Local machine:**
+
 ```bash
 pip install ansible
 # or: sudo apt install ansible
 ```
 
 **New Pi node checklist:**
-1. Flash Raspberry Pi OS Lite (Trixie), connect to WiFi
-2. Install Tailscale and join the network
-3. Ensure `david` user exists with sudo access
-4. Connect the MeshCore device via USB, then find its serial ID:
-   ```bash
-   ls /dev/serial/by-id/
-   ```
-5. Set `serial_port` in `ansible/host_vars/<hostname>.yml`
 
-SSH key auth is required. From this machine:
-```bash
-ssh-copy-id david@<hostname>.tail740bb.ts.net
-```
+1. Flash Raspberry Pi OS Lite (Trixie) with Raspberry Pi Imager, using "Edit Settings" (OS customisation) to set hostname, the `david` user + password, and the `dotnetwork` wifi (SSID/password only — the imager only supports one network at flash time; the deployed-location and `dotmobile` networks get added later by the `wifi` role, see below).
+2. **Before ejecting the card**, verify the customisation actually got written — mount the boot partition and check `network-config`/`user-data` aren't just the commented-out stock template (see "Imager gotcha" below). Only `cmdline.txt`'s regdomain getting a fresh timestamp while the rest stay at the image's build date is the tell that it silently failed.
+3. Boot the Pi, find its LAN IP (e.g. from the router's DHCP leases), confirm SSH access with the password you set — no key is seeded at flash time.
+4. Run `ssh-copy-id david@<ip>` **from an interactive terminal** (not through a non-interactive shell/script — it needs a real TTY to prompt for the password) so ansible can connect with a key.
+5. Connect the MeshCore device via USB, then find its serial ID: `ls /dev/serial/by-id/`.
+6. Add `serial_port` (and `wifi_ssid_location`) to `ansible/host_vars/<hostname>/vars.yml`, and the location wifi password to `ansible/host_vars/<hostname>/vault.yml` (see "Vault" below).
+7. Run `site.yml` against just that host, overriding the host's address since Tailscale/DNS won't resolve it yet: `ansible-playbook site.yml --limit <hostname> -e "ansible_host=<ip>"`. This authorizes your SSH keys, joins the deployed-location + dotmobile wifi networks, installs (but does not authenticate) Tailscale, and deploys everything else in one pass.
+8. Tailscale needs one manual step: SSH in and run `sudo tailscale up`, then open the printed URL in a browser to approve the device on your tailnet. (You *can* pass `-e tailscale_auth_key=tskey-...` — from the [admin console](https://login.tailscale.com/admin/settings/keys) — to authenticate non-interactively instead, but there's no stored key anywhere for this repo, so the interactive route is simplest for a one-off node.)
+9. The meshcore-packet-capture installer also needs one manual step (see "meshcore-packet-capture install is interactive" below): SSH in and run `sudo bash -c "$(curl -fsSL https://raw.githubusercontent.com/agessaman/meshcore-packet-capture/main/install.sh)"`, answering its ~3 prompts (service account, install method — pick **1**, IATA/broker config — defaults are fine, `site.yml` overwrites `.env.local` afterward anyway).
+10. Once Tailscale is up, re-run `site.yml` without the `ansible_host` override — it'll resolve via the Tailscale hostname from here on, and will now just write `.env.local` + enable the service since the installer step is already satisfied.
+
+### meshcore-packet-capture install is interactive
+
+The `install.sh` bootstrap has no real non-interactive path for a *fresh* install — its `--update` flag only changes behavior when an installation already exists. It also refuses to run at all with piped stdin (`curl | sudo bash` errors out asking you to download the script first) and its Python layer explicitly opens `/dev/tty` for prompts, which just hangs forever over plain SSH/ansible (no human there to answer). We tried feeding it scripted answers via `script`/a pty and it's not worth the fragility — just run it manually once per node (step 9 above); `ansible/roles/meshcore_capture/tasks/main.yml`'s `creates:` guard means ansible never touches it again afterward.
+
+### Imager gotcha (2026-07)
+
+The Raspberry Pi Imager available via Flathub (`org.raspberrypi.rpi-imager`) is stuck on **1.9.6** and there's no newer `.deb`/Flatpak in the Ubuntu or Flathub repos either — Flathub hasn't published the 2.0.x rewrite. 1.9.6 silently fails to apply OS customisation (hostname, user, SSH, wifi) on newer Raspberry Pi OS Trixie images: it only writes the kernel `cfg80211.ieee80211_regdom=` cmdline parameter and leaves `user-data`/`network-config` as the stock commented-out template, with no error. The result looks exactly like a wifi problem (Pi never appears on the network) but is actually "the card has no credentials on it at all."
+
+Fix: grab the real `.deb` from the [GitHub releases page](https://github.com/raspberrypi/rpi-imager/releases) (e.g. `rpi-imager_2.0.10_amd64.deb`) and `sudo dpkg -i` it — that version writes the customisation correctly.
 
 ## Usage
 
 **Deploy to a single host (recommended for first run / testing):**
+
 ```bash
 cd ansible
 ansible-playbook -i inventory.yml site.yml --limit dm-edworth
 ```
 
 **Deploy to all nodes:**
+
 ```bash
 ansible-playbook -i inventory.yml site.yml
 ```
 
 **Dry run:**
+
 ```bash
 ansible-playbook -i inventory.yml site.yml --limit dm-edworth --check
 ```
@@ -57,20 +74,65 @@ You'll be prompted for a Tailscale auth key — leave blank if the node is alrea
 
 ## What it does
 
-1. **base** — apt upgrade, installs screen/pipx/vnstat/git, sets MOTD, installs and authenticates Tailscale
-2. **meshcore_cli** — installs `meshcore-cli` via pipx
-3. **meshcore_capture** — runs the agessaman/meshcore-packet-capture install script, writes `.env.local` config, enables `meshcore-capture.service`, deploys update/log helper scripts
-4. **scripts** — deploys `voltage.sh` and `bandwidth.sh`
+1. **wifi** — configures NetworkManager connections for `dotnetwork` (home), `dotmobile` (phone hotspot, field troubleshooting fallback), and the host's deployed-location network
+2. **base** — apt upgrade, installs screen/pipx/vnstat/git, sets MOTD, authorizes SSH keys for both laptop partitions, installs Tailscale (always) and authenticates it (only if `tailscale_auth_key` is set — otherwise run `sudo tailscale up` manually once, see checklist above)
+3. **meshcore_cli** — installs `meshcore-cli` via pipx
+4. **meshcore_capture** — runs the agessaman/meshcore-packet-capture install script (skipped once already installed — see "meshcore-packet-capture install is interactive" above, this needs a manual first run), writes config (TOML `config.d/99-user.toml` on current "system"-layout nodes, legacy `.env.local` on the two nodes still on the old layout — see `meshcore_capture_layout` in `group_vars/all/vars.yml`), enables the capture service, deploys update/log helper scripts
+5. **scripts** — deploys `voltage.sh` and `bandwidth.sh`
 
 ## Config
 
-Shared MQTT config lives in `group_vars/meshcore.yml`. Per-host serial port is in `host_vars/<hostname>.yml`.
+Shared MQTT config lives in `group_vars/meshcore.yml`. Per-host serial port and wifi SSID are in `host_vars/<hostname>/vars.yml`.
 
 Running the playbook again re-applies `.env.local` and restarts the service if it changed — safe to run on already-deployed nodes.
 
-## Credentials note
+## Vault
 
-`group_vars/meshcore.yml` contains MQTT credentials in plaintext. Consider encrypting with Ansible Vault if this repo is shared:
+`group_vars/all/vault.yml` (shared wifi/SSH secrets) and `host_vars/<hostname>/vault.yml` (per-host deployed-location wifi password) are encrypted with Ansible Vault. `ansible.cfg` points at `../.vault_pass` (gitignored, not committed) for the password — ask David for a copy, or generate a fresh one and re-encrypt if starting over:
 ```bash
-ansible-vault encrypt_string 'yourpassword' --name mqtt_ukmesh_password
+ansible-vault view --vault-password-file ../.vault_pass group_vars/all/vault.yml
+ansible-vault edit --vault-password-file ../.vault_pass host_vars/dm-edworth/vault.yml
 ```
+`*/vault.yml.example` shows the expected keys.
+
+`group_vars/meshcore.yml` (MQTT credentials) is still plaintext — consider moving it into the vault too if this repo is shared further.
+
+## dotminipc (Docker node, not managed by this repo)
+
+A fourth capture point, device name `dm-stotfold`, runs as a Docker container
+(`ghcr.io/agessaman/meshcore-packet-capture:latest`) on **dotminipc**
+(172.16.31.92), the shared Docker/HA host documented in the sibling
+`dotnetwork` repo. It isn't part of this repo's inventory — no Ansible role
+here touches it — but it's the same upstream software as the Pi fleet, so
+config drift between the two is worth knowing about.
+
+- Compose file: `dotnetwork/docker/dotminipc/meshcore-packet-capture/docker-compose.yml`
+- Config: `dotnetwork/docker/dotminipc/meshcore-packet-capture/config.d/99-user.toml`
+  — host-only, gitignored (contains MQTT credentials), same TOML shape as
+  this repo's `meshcore_capture` role template
+  (`ansible/roles/meshcore_capture/templates/99-user.toml.j2`); keep the
+  broker list in sync between the two if either changes. The committed
+  `config.d/99-user.toml.example` in the dotnetwork repo is the sanitized
+  reference copy.
+- Deploy/restart: `ssh [email protected]`, then from `/opt/docker`:
+  `docker compose up -d meshcore-capture --force-recreate`. Logs:
+  `docker logs -f meshcore-packet-capture`.
+
+**2026-07-27 outage, for reference**: the `:latest` image was rebuilt
+2026-07-25 and switched its config format from `.env.local` to TOML
+(`/etc/meshcore-packet-capture/config.toml` + `config.d/*.toml` —
+the same "system" layout `meshcore_capture_layout` already models for the Pi
+fleet). It silently stopped reading `.env.local` at all — despite upstream's
+own README still describing that as a supported "legacy" path, in practice it
+was just ignored — so the container fell back to the image's baked-in
+defaults (`connection_type = "ble"`, `serial.ports = ["/dev/ttyUSB0"]`, no
+owner key, only 2 of 5 brokers) and crash-looped on
+`could not open port /dev/ttyUSB0`. Fix was to mount a full TOML override at
+`config.d/99-user.toml` instead (see above) — same pattern as the Pi
+`meshcore_capture` role already uses for `system`-layout nodes. Also found in
+the process: the old `.env.local` had `PACKETCAPTURE_MQTT5_SERVER` set to
+`mqtt.meshmapper.cc`, which fails TLS handshake — the correct domain is
+`mqtt.meshmapper.net` (confirmed via `openssl s_client`; `.net` presents a
+valid Let's Encrypt cert, `.cc` returns a TLS alert). If the Pi fleet's
+`99-user.toml.j2` broker list is ever regenerated from a stale copy of this
+node's old config, watch out for that typo resurfacing.

ansible/group_vars/all/vars.yml

@@ -1,6 +1,10 @@
 remote_user: david
 ansible_user: "{{ remote_user }}"
-ansible_become: true
+# become is set at the play level in site.yml, NOT here — ansible_become as a
+# connection variable takes precedence over a task's own `become: false`
+# keyword, which silently broke every become:false task in this repo (they
+# all ran as root regardless). Play-level `become:` correctly loses to a
+# task-level override.
 
 meshcore_description: "meshcore management"
 
@@ -16,3 +20,8 @@ wifi_password_dotnetwork: "{{ vault_wifi_dotnetwork }}"
 
 wifi_ssid_dotmobile: "dotmobile"
 wifi_password_dotmobile: "{{ vault_wifi_dotnetwork }}"
+
+# meshcore-packet-capture install generation — see roles/meshcore_capture/tasks/main.yml.
+# New nodes get the current upstream layout by default; older nodes override
+# this to "legacy" in their own host_vars.
+meshcore_capture_layout: "system"

ansible/host_vars/dm-baldock/vars.yml

@@ -3,3 +3,6 @@ serial_port: /dev/serial/by-id/usb-Espressif_USB_JTAG_serial_debug_unit_98:3D:AE
 # Deployed-location wifi
 wifi_ssid_location: "H-NeT"
 wifi_password_location: "{{ vault_wifi_location }}"
+
+# Predates the current upstream installer's system-service layout
+meshcore_capture_layout: "legacy"

ansible/host_vars/dm-edworth/vars.yml

@@ -1,5 +1,4 @@
-# Find this by connecting the MeshCore device then running: ls /dev/serial/by-id/
-serial_port: FILL_IN_SERIAL_PORT
+serial_port: /dev/serial/by-id/usb-Espressif_USB_JTAG_serial_debug_unit_10:B4:1D:E7:FA:C8-if00
 
 # Deployed-location wifi
 wifi_ssid_location: "Ridgeway"

ansible/roles/base/tasks/main.yml

@@ -17,6 +17,15 @@
     state: present
   become: true
 
+- name: Install libffi headers (armv6 has no prebuilt cffi wheel, needs to compile it)
+  apt:
+    name:
+      - libffi-dev
+      - pkg-config
+    state: present
+  become: true
+  when: ansible_architecture == "armv6l"
+
 - name: Set MOTD
   template:
     src: motd.j2
@@ -26,19 +35,17 @@
     mode: "0644"
   become: true
 
-- name: Add Tailscale apt signing key
+- name: Download Tailscale install script
   get_url:
-    url: https://pkgs.tailscale.com/stable/debian/bookworm.nosetup.sh
-    dest: /tmp/tailscale-setup.sh
+    url: https://tailscale.com/install.sh
+    dest: /tmp/tailscale-install.sh
     mode: "0755"
-  when: tailscale_auth_key != ""
 
 - name: Run Tailscale install script
-  shell: sh /tmp/tailscale-setup.sh
+  shell: sh /tmp/tailscale-install.sh
   args:
     creates: /usr/bin/tailscale
   become: true
-  when: tailscale_auth_key != ""
 
 - name: Enable and start tailscaled
   systemd:
@@ -46,7 +53,6 @@
     enabled: true
     state: started
   become: true
-  when: tailscale_auth_key != ""
 
 - name: Authenticate Tailscale
   shell: tailscale up --authkey {{ tailscale_auth_key }}

ansible/roles/meshcore_capture/handlers/main.yml

@@ -1,6 +1,6 @@
 ---
 - name: restart meshcore-capture
   systemd:
-    name: meshcore-capture
+    name: "{{ 'meshcore-capture' if meshcore_capture_layout == 'legacy' else 'meshcore-packet-capture' }}"
     state: restarted
   become: true

ansible/roles/meshcore_capture/tasks/main.yml

@@ -1,23 +1,56 @@
 ---
+# This is a genuine first-run interactive installer (asks for service
+# account, install method, IATA/broker config) with no real non-interactive
+# path for a fresh install — its own --update flag only changes behavior
+# when an install already exists. So: run it manually once per node,
+# answering its ~3 prompts (see README), then ansible's `creates:` guard
+# skips it forever after. Do NOT try to script answers into it — it insists
+# on a real controlling tty and hangs waiting for one over plain SSH/ansible.
+#
+# meshcore_capture_layout distinguishes two generations of this upstream
+# installer:
+#   system (default) — install method 1, current (v2.0.0+) upstream default.
+#     Dedicated meshcore-capture system user, /opt + /etc/meshcore-packet-capture,
+#     config.d/*.toml, service meshcore-packet-capture.service.
+#   legacy — what dm-baldock/dm-ashwell were set up with (older installer):
+#     flat ~/.meshcore-packet-capture, .env.local, service meshcore-capture.service.
+#     Kept only so this role stays a no-op/safe on those two nodes; new nodes
+#     should use "system".
 - name: Run meshcore-packet-capture install script
   shell: |
     bash <(curl -fsSL https://raw.githubusercontent.com/agessaman/meshcore-packet-capture/main/install.sh)
   args:
     executable: /bin/bash
-    creates: /etc/systemd/system/meshcore-capture.service
-  become: false
+    creates: >-
+      {{ '/etc/systemd/system/meshcore-capture.service' if meshcore_capture_layout == 'legacy'
+         else '/etc/systemd/system/meshcore-packet-capture.service' }}
+  become: true
 
-- name: Write .env.local config
+- name: Write .env.local config (legacy layout)
   template:
     src: env.local.j2
-    dest: "{{ ansible_env.HOME }}/.meshcore-packet-capture/.env.local"
+    dest: "/home/{{ remote_user }}/.meshcore-packet-capture/.env.local"
+    owner: "{{ remote_user }}"
+    group: "{{ remote_user }}"
     mode: "0640"
   become: false
+  when: meshcore_capture_layout == 'legacy'
+  notify: restart meshcore-capture
+
+- name: Write user config overrides (system layout)
+  template:
+    src: 99-user.toml.j2
+    dest: /etc/meshcore-packet-capture/config.d/99-user.toml
+    owner: root
+    group: meshcore-capture
+    mode: "0640"
+  become: true
+  when: meshcore_capture_layout != 'legacy'
   notify: restart meshcore-capture
 
 - name: Enable and start meshcore-capture service
   systemd:
-    name: meshcore-capture
+    name: "{{ 'meshcore-capture' if meshcore_capture_layout == 'legacy' else 'meshcore-packet-capture' }}"
     enabled: true
     state: started
     daemon_reload: true
@@ -25,14 +58,22 @@
 
 - name: Deploy meshcore-capture-update script
   copy:
-    content: "#!/usr/bin/env bash\nbash <(curl -fsSL https://raw.githubusercontent.com/agessaman/meshcore-packet-capture/main/install.sh)\n"
-    dest: "{{ ansible_env.HOME }}/meshcore-capture-update.sh"
+    content: |
+      #!/usr/bin/env bash
+      bash <(curl -fsSL https://raw.githubusercontent.com/agessaman/meshcore-packet-capture/main/install.sh){{ ' --update' if meshcore_capture_layout != 'legacy' else '' }}
+    dest: "/home/{{ remote_user }}/meshcore-capture-update.sh"
+    owner: "{{ remote_user }}"
+    group: "{{ remote_user }}"
     mode: "0755"
   become: false
 
 - name: Deploy meshcore-capture-logs script
   copy:
-    content: "#!/usr/bin/env bash\nsudo journalctl -u meshcore-capture -f\n"
-    dest: "{{ ansible_env.HOME }}/meshcore-capture-logs.sh"
+    content: |
+      #!/usr/bin/env bash
+      sudo journalctl -u {{ 'meshcore-capture' if meshcore_capture_layout == 'legacy' else 'meshcore-packet-capture' }} -f
+    dest: "/home/{{ remote_user }}/meshcore-capture-logs.sh"
+    owner: "{{ remote_user }}"
+    group: "{{ remote_user }}"
     mode: "0755"
   become: false

ansible/roles/meshcore_capture/templates/99-user.toml.j2

@@ -0,0 +1,126 @@
+# MeshCore Packet Capture - user overrides
+# Managed by Ansible - local changes will be overwritten on next playbook run
+#
+# Defines every broker fully (not relying on config.d/10-letsmesh.toml being
+# auto-installed) — whether that preset exists depends on which path the
+# installer took (fresh install vs migrate), so a partial override here would
+# silently no-op on nodes where it wasn't created.
+
+[general]
+iata = "{{ packetcapture_iata }}"
+log_level = "{{ packetcapture_log_level }}"
+
+[update]
+repo = "{{ packetcapture_update_repo }}"
+branch = "{{ packetcapture_update_branch }}"
+
+[serial]
+ports = ["{{ serial_port }}"]
+
+[capture]
+connection_type = "serial"
+advert_interval_hours = {{ packetcapture_advert_interval_hours }}
+owner_public_key = "{{ packetcapture_owner_public_key }}"
+owner_email = "{{ packetcapture_owner_email }}"
+
+# LetsMesh.net Packet Analyzer (US)
+[[broker]]
+name = "letsmesh-us"
+enabled = true
+server = "mqtt-us-v1.letsmesh.net"
+port = 443
+transport = "websockets"
+keepalive = 120
+qos = 0
+retain = true
+
+[broker.tls]
+enabled = true
+verify = true
+
+[broker.auth]
+method = "token"
+audience = "mqtt-us-v1.letsmesh.net"
+owner = "{{ packetcapture_owner_public_key }}"
+email = "{{ packetcapture_owner_email }}"
+
+# LetsMesh.net Packet Analyzer (EU)
+[[broker]]
+name = "letsmesh-eu"
+enabled = true
+server = "mqtt-eu-v1.letsmesh.net"
+port = 443
+transport = "websockets"
+keepalive = 120
+qos = 0
+retain = true
+
+[broker.tls]
+enabled = true
+verify = true
+
+[broker.auth]
+method = "token"
+audience = "mqtt-eu-v1.letsmesh.net"
+owner = "{{ packetcapture_owner_public_key }}"
+email = "{{ packetcapture_owner_email }}"
+
+# MeshRank
+[[broker]]
+name = "meshrank"
+enabled = true
+server = "meshrank.net"
+port = 8883
+transport = "tcp"
+qos = 0
+retain = false
+
+[broker.tls]
+enabled = true
+verify = true
+
+[broker.auth]
+method = "none"
+topic_token = "{{ mqtt_meshrank_token }}"
+
+[broker.topics]
+packets = "meshrank/uplink/{TOKEN}/{PUBLIC_KEY}/packets"
+status = "meshrank/uplink/{TOKEN}/{PUBLIC_KEY}/status"
+
+# UKMesh (custom broker, no upstream preset)
+[[broker]]
+name = "ukmesh"
+enabled = true
+server = "mqtt.ukmesh.com"
+port = 443
+transport = "websockets"
+qos = 0
+retain = false
+
+[broker.tls]
+enabled = true
+verify = true
+
+[broker.auth]
+method = "password"
+username = "{{ mqtt_ukmesh_username }}"
+password = "{{ mqtt_ukmesh_password }}"
+
+# MeshMapper
+[[broker]]
+name = "meshmapper"
+enabled = true
+server = "mqtt.meshmapper.net"
+port = 443
+transport = "websockets"
+keepalive = 55
+qos = 0
+retain = true
+
+[broker.tls]
+enabled = true
+verify = true
+
+[broker.auth]
+method = "token"
+audience = "mqtt.meshmapper.net"

ansible/roles/meshcore_cli/tasks/main.yml

@@ -11,5 +11,5 @@
   shell: pipx install meshcore-cli
   args:
     executable: /bin/bash
-    creates: "{{ ansible_env.HOME }}/.local/bin/meshcore-cli"
+    creates: "/home/{{ remote_user }}/.local/bin/meshcore-cli"
   become: false

ansible/roles/scripts/tasks/main.yml

@@ -2,13 +2,17 @@
 - name: Deploy voltage.sh
   copy:
     src: voltage.sh
-    dest: "{{ ansible_env.HOME }}/voltage.sh"
+    dest: "/home/{{ remote_user }}/voltage.sh"
+    owner: "{{ remote_user }}"
+    group: "{{ remote_user }}"
     mode: "0755"
   become: false
 
 - name: Deploy bandwidth.sh
   copy:
     src: bandwidth.sh
-    dest: "{{ ansible_env.HOME }}/bandwidth.sh"
+    dest: "/home/{{ remote_user }}/bandwidth.sh"
+    owner: "{{ remote_user }}"
+    group: "{{ remote_user }}"
     mode: "0755"
   become: false

ansible/site.yml

@@ -1,6 +1,7 @@
 ---
 - name: Deploy MeshCore monitoring nodes
   hosts: meshcore
+  become: true
   vars_prompt:
     - name: tailscale_auth_key
       prompt: "Tailscale auth key (leave blank to skip)"