fetch_ca.py — Download CA certificate bundles for the PicoMite
================================================================

Purpose
-------
fetch_ca.py downloads one or more public Certificate Authority root
certificates from their canonical sources, optionally bundles them
into a single PEM file, optionally validates the bundle against a
real TLS server, and prepares the file for use with the WebMite's
WEB TLS CA command.

The PicoMite needs a CA bundle to verify TLS server certificates
(see the chapter "Secure (TLS) Connections" in the WiFi manual).
This script is the simplest way to produce one.

Requirements
------------
- Python 3.6 or newer (uses only the standard library — no pip
  packages required).
- Internet access for the host PC.
- The host PC's own trust store must be valid, since the downloads
  go to letsencrypt.org / amazontrust.com / digicert.com themselves
  over HTTPS.


Basic usage
-----------
Run the script from the same directory the file is in.  Without any
arguments it downloads ISRG Root X1 (Let's Encrypt) and Amazon Root
CA 1 — a combined ~4 KB bundle that covers the majority of public
HTTPS servers (including httpbin.org, most Let's Encrypt-signed
sites, AWS-hosted services and S3 buckets):

  python fetch_ca.py

This writes ca.pem in the current directory.


Listing what's available
------------------------
  python fetch_ca.py --list

Shows the named roots the script knows about, with a one-line
description of each.  Names are short tags like isrg-x1, amazon-r1,
digicert-g2 — use these on the command line.


Selecting which roots to download
---------------------------------
Pass one or more root names as positional arguments:

  python fetch_ca.py isrg-x1                   ' just Let's Encrypt
  python fetch_ca.py isrg-x1 amazon-r1         ' default mix
  python fetch_ca.py isrg-x1 digicert-g2 amazon-r1
  python fetch_ca.py amazon-r1 amazon-r2 amazon-r3 amazon-r4

The resulting bundle has all selected roots concatenated, each with
a small comment header naming the source URL.  Order does not
matter — mbedtls scans every root looking for a chain.


Custom output filename
----------------------
  python fetch_ca.py -o my-bundle.pem isrg-x1 amazon-r1

Useful when you want multiple bundles on the device (eg, a strict
bundle for production servers and a wider one for development).


Verifying the bundle works
--------------------------
After downloading, --verify connects to a real TLS server with the
new bundle and confirms the server's certificate chain validates
against it.  This catches mistakes (wrong CA selected, bundle
corrupted, etc.) before you transfer the file to the PicoMite and
discover the handshake fails.

  python fetch_ca.py --verify httpbin.org:443
  python fetch_ca.py --verify broker.hivemq.com:8883 isrg-x1

Exit code 0 on success, 2 on verification failure.


Recommended bundles for common scenarios
----------------------------------------
HTTPS to the open web (most public REST APIs, weather services,
news sites, OpenWeatherMap, etc):
  python fetch_ca.py isrg-x1 amazon-r1 -o ca.pem

Internet radio streaming sources:
  python fetch_ca.py isrg-x1 amazon-r1 digicert-g2 -o ca.pem

Public MQTT brokers (HiveMQ, EMQX, mosquitto):
  python fetch_ca.py isrg-x1 -o ca.pem
  (all three use Let's Encrypt)

GitHub API and DigiCert-signed enterprise sites:
  python fetch_ca.py isrg-x1 amazon-r1 digicert-g2 -o ca.pem

For your own private servers (eg, an MQTT broker on your LAN with a
self-signed cert), do NOT use this script — generate your own CA
with openssl, sign your server cert with it, and ship that CA cert
directly to the PicoMite.


Bundle size and the PicoMite's memory limit
-------------------------------------------
The PicoMite reads the bundle into mbedtls's heap at TLS handshake
time.  Each CA consumes roughly 1.5 KB of RAM while a connection is
being established.  Sizing guidance:

  1-3 CAs (~2-5 KB on disk):      Comfortable.
  4-8 CAs (~6-12 KB on disk):     Works but exercises the limit.
  More than 8 CAs:                Don't.
  Full Mozilla CA store (~200 KB) Will NOT load — exhausts heap.

Pick only the CAs your application actually needs.  The script's
default (isrg-x1 + amazon-r1) is the sweet spot for most uses.


Transferring the bundle to the PicoMite
---------------------------------------
The PicoMite has a TFTP server built in (enabled automatically when
WiFi is connected).  From the same PC running fetch_ca.py:

  tftp -i PICOMITE-IP PUT ca.pem

Replace PICOMITE-IP with your device's IP address (shown after
"Connected" on startup, or via MM.Info(IP Address) in BASIC).

Other options: XModem over a serial or Telnet console, or dragging
the file in MMEdit if it supports that for your platform.


Using the bundle on the PicoMite
--------------------------------
Once the file is on the device:

  > WEB NTP                         ' set the system clock first
  > WEB TLS CA "ca.pem"             ' enable peer verification
  TLS: CA bundle loaded, peer verification REQUIRED
  > WEB OPEN TLS CLIENT "host", 443 ' now verifies the cert

The CA stays loaded until WEB TLS NOVERIFY is run or the WebMite is
restarted.  See the WiFi manual chapter for the full workflow.


Troubleshooting
---------------
"Failed to parse CA bundle" on the PicoMite:
  The file was not recognised as PEM.  Check it starts with
  "-----BEGIN CERTIFICATE-----" and contains the matching END line.
  Re-run fetch_ca.py — do not edit the file by hand.

"TLS client error -13" (or similar negative number) on the PicoMite:
  The server's certificate doesn't chain to anything in your bundle.
  Run on the host PC:
    python fetch_ca.py --verify HOST:PORT
  with different root combinations until one validates.  Then
  rebuild the bundle with those roots.

"NTP timeout" preventing WEB TLS CA from being effective:
  Verification needs real time.  Try a different NTP server:
    WEB NTP 0, "time.google.com", 15000
  or "time.cloudflare.com" or "pool.ntp.org".

Verification suddenly stops working after months/years:
  A public CA may have rotated.  Re-run fetch_ca.py to pick up the
  current canonical root and re-transfer the file.


Behaviour notes
---------------
- The script writes LF line endings regardless of OS, so the
  resulting file is byte-identical between Windows and Unix.
- DER input is auto-converted to PEM if a source serves binary.
- A comment header is added per certificate naming the source URL.
  mbedtls's X.509 parser ignores everything outside the BEGIN/END
  markers so the comments cost nothing on the device.
- The script does not pin specific certificates — it downloads
  whatever the authoritative server currently publishes.  This is
  the right behaviour because public CAs rotate occasionally; if you
  need pinning, build the bundle once and store the resulting file
  in version control rather than re-running the script.
