We Stopped Inventing Sync Protocols: Building a Custom MCUmgr Group in Zephyr

Every connected embedded product eventually grows a protocol. You need to get structured data off a device and into a phone, none of the standard BLE profiles fit, so you define a characteristic, invent a framing scheme, and get on with the actual product.

We did that on HealthyPi Move. It worked. Then we deleted it, and implemented the same feature set as a custom MCUmgr management group instead.

This post is the engineering detail behind that decision: how you actually extend MCUmgr in Zephyr, how we sized a command set against the SMP transport, what it bought in reliability and code, and what it looked like doing it a second time on HealthyPi 6 — a completely different SoC that now ships a custom management group of its own.

This is the second post in a series. The first, HealthyPi Move NEXT, covers the release this work shipped in.

1. What we had, and what was wrong with it

The old firmware moved health data over a custom BLE service with hand-rolled framing: start-of-frame bytes 0x0A 0xFA, a length field, a command byte, stop bytes. Sessions came off the filesystem as whole files.

Nothing about that is unusual, and it genuinely worked. The problem is what you own once you’ve written it:

• Framing and reassembly — yours, across every phone, every BLE stack, every MTU negotiation.
• Request/response pairing and sequence matching — yours.
• Error codes — yours, and undocumented until someone asks.
• Reconnect and resume semantics — yours, and the part that’s least tested because it only matters when things go wrong.
• Client implementations — everyone else’s, reimplemented from a README, forever.

None of that is product work. It’s infrastructure you maintain so the product can exist, and every hour spent debugging a dropped-connection edge case is an hour not spent on the thing people bought the device for.

Meanwhile, sitting on the same radio, was a protocol we already trusted with the single most dangerous operation the device performs.

2. The realisation: SMP is already there, and already trusted

Zephyr ships MCUmgr, and its SMP (Simple Management Protocol) transport is what carries firmware updates. If you do DFU on Zephyr, you already run an SMP server — a request/response protocol with CBOR payloads, framing, sequence matching and error handling, exercised by every Zephyr device that has ever taken an update.

Think about the risk asymmetry for a moment. We were trusting SMP to stream a signed firmware image into flash, where a single corrupted byte bricks the device in a customer’s hands. And alongside it we ran a second, home-grown protocol — with a fraction of the testing — to move heart-rate samples, where the worst case is a wrong number on a chart.

That’s backwards. If SMP is good enough for the operation that can brick the device, it is good enough for the operation that can’t.

MCUmgr is also explicitly designed to be extended. The built-in groups (os, img, fs, settings, stat) are just registered handler tables, and Zephyr reserves everything from 64 upward (MGMT_GROUP_ID_PERUSER) for application-defined groups — Zephyr’s own additions grow downward from that boundary specifically to avoid colliding with yours. Adding your own group is a first-class operation, not a hack.

So we deleted the custom service and implemented the Healthy Store as management group 0x1000.

3. How you actually extend MCUmgr in Zephyr

This is the part worth having in front of you, because the mechanics are much smaller than people expect.

3.1 A handler is a function over a CBOR streamer

Each command is a function taking a struct smp_streamer *, which carries the decoded request and the response encoder. A read handler pulls its arguments out with zcbor_bulk, does the work, and encodes a CBOR map back:

static int hs_h_sync(struct smp_streamer *ctxt)
{
    zcbor_state_t *zsd = ctxt->reader->zs;
    zcbor_state_t *zse = ctxt->writer->zs;

    uint32_t since = 0, max = HS_SYNC_MAX_BATCH;
    size_t decoded;
    struct zcbor_map_decode_key_val keys[] = {
        ZCBOR_MAP_DECODE_KEY_DECODER("since", zcbor_uint32_decode, &since),
        ZCBOR_MAP_DECODE_KEY_DECODER("max",   zcbor_uint32_decode, &max),
    };

    if (zcbor_map_decode_bulk(zsd, keys, ARRAY_SIZE(keys), &decoded) != 0) {
        return MGMT_ERR_EINVAL;
    }
    /* ... read samples, encode "recs"/"n"/"next"/"more" ... */
}

There’s no protocol code in there. No framing, no length prefix, no checksum, no state machine — just decode, work, encode.

3.2 Registration is a table and one macro

The entire wiring for the whole group — eleven commands in a release build:

static const struct mgmt_handler hpi_hs_handlers[] = {
    [HPI_HS_CMD_HELLO]   = { hs_h_hello,   NULL },   /* read  */
    [HPI_HS_CMD_TYPES]   = { hs_h_types,   NULL },
    [HPI_HS_CMD_SYNC]    = { hs_h_sync,    NULL },
    [HPI_HS_CMD_SUMMARY] = { hs_h_summary, NULL },
    [HPI_HS_CMD_RECORDS] = { hs_h_records, NULL },
    [HPI_HS_CMD_ACK]     = { NULL,         hs_h_ack },   /* write */
    [HPI_HS_CMD_SET_TZ]  = { NULL,         hs_h_set_tz },
    /* ... BPT calibration commands ... */
};

static struct mgmt_group hpi_hs_group = {
    .mg_handlers       = hpi_hs_handlers,
    .mg_handlers_count = ARRAY_SIZE(hpi_hs_handlers),
    .mg_group_id       = HPI_HS_MGMT_GROUP_ID,   /* 0x1000 */
};

static void hpi_hs_mgmt_register(void)
{
    mgmt_register_group(&hpi_hs_group);
}

MCUMGR_HANDLER_DEFINE(hpi_hs_mgmt, hpi_hs_mgmt_register);

That’s it. MCUMGR_HANDLER_DEFINE puts the registration function in an iterable section that MCUmgr walks at init, so there’s no call to add to main() and no ordering to get right. The designated-initialiser table also makes the read/write split explicit at a glance — the left slot is the read handler, the right is the write handler, and a command that is one and not the other says so in its own line.

The whole file is handlers plus that block. Everything underneath — transport, framing, sequence numbers, error propagation — belongs to Zephyr.

3.3 The constraint that shapes your command set: the netbuf

The one place SMP’s design leans on yours is the buffer. Your whole response has to fit in one MCUmgr netbuf — CONFIG_MCUMGR_TRANSPORT_NETBUF_SIZE, which defaults to a fairly tight 384 bytes. We raise it to 1024, and that number then constrains every command in the group.

That single number drove two design decisions:

Batch size. Samples are an 18-byte packed record — sequence number, UTC timestamp, type id, quality bitmask, fixed-point value. A SYNC batch is capped at 40 records = 720 bytes, leaving comfortable room for the CBOR map around it. Samples go over as one CBOR byte string rather than an array of maps, which is what makes that density possible: 40 samples as CBOR maps with string keys would blow the buffer several times over.

Paging the registry. The type registry has per-metric strings — key, unit, and the HealthKit / Health Connect mapping hints — so it can’t be served in one shot. TYPES is paged at 5 entries per call, and the response carries next and total so a client loops until next == total.

That last one is a trap worth naming, because it bit our own client first: a client that calls TYPES once sees only the first five of twenty-one types. If you page a response, document the loop, because the single-call case looks like it works.

4. Designing the command set

Eleven commands cover the whole surface — twelve if you count the synthetic-data generator, which is Kconfig-gated off in release builds. The interesting ones:

Three things in there are worth stealing.

The cursor is the whole sync design. SYNC {since, max} returns {recs, n, next, more}. The client stores next and resumes from it. Because sequence numbers are monotonic and ingest is idempotent on seq, a dropped link costs you nothing — reconnect, re-request from your cursor, carry on. There is no session state on the device to get out of sync with.

more needs to be defined carefully. It means “another page is worth fetching” — n > 0 && next < head — not simply next < head. The naive version stays true on an empty page, so a client looping on more alone spins forever on an unadvancing cursor. We shipped the naive version first and had to fix it, which is exactly the class of bug you inherit when you own a protocol and don’t when you don’t.

HELLO exposes oldest, not just head. Retention drops the tail, so a client needs to distinguish “the store is empty” from “my cursor is stale and I’ve missed data that’s gone”. Publishing both ends of the valid range makes that a comparison instead of a probe: oldest > head means empty, and a cursor below oldest - 1 means you’ve lost samples and should restart from there.

5. Versioning: the part that decides whether clients rot

Riding a standard protocol doesn’t save you from schema design. We split versioning in two:

• HPI_HS_SCHEMA_VERSION — the meaning, unit and scale of sample types, and the wire sample layout.
• HPI_HS_GROUP_VERSION — the shape of the command set.

Both come back from HELLO, so a client knows what it’s talking to before it talks.

The split has already paid for itself. The group version is at 2 — v2 added the four blood-pressure calibration commands — while the schema version is still 1. An older client that has never heard of those four commands is still completely correct about every sample it reads, because nothing about the data changed. One number moved; the other didn’t have to.

The rule that matters: adding a new metric type id is not breaking. The registry lives on the device, clients cache it by id, and an unknown id is skipped rather than guessed at. Renumbering or reusing an id, or changing a type’s unit or scale, is breaking and bumps the schema version.

That asymmetry is what lets firmware grow new metrics without a coordinated app release — and it only works because the registry is served by the device instead of being compiled into every client.

6. What it bought

Reliability we didn’t write. Sequence matching, request/response pairing, framing, reassembly, error codes — all Zephyr’s, all exercised by every Zephyr DFU ever performed. Our bug surface shrank to our own handlers.

One link, one lock. DFU, sample sync and record downloads now share a single SMP characteristic and arbitrate against each other properly. Under the old design a firmware update and a data sync were two independent services racing for the same radio, and the failure modes were exactly as much fun as that sounds.

Clients for free. Any MCUmgr client library gets you most of the way. The Move app talks to the group through two Dart packages we’ve since published — mcumgr_dart for SMP itself, and healthypi_healthy_store for the Healthy Store commands layered on top — and on the research side, Python’s smpclient lets you define request classes for a custom group in a few lines.

The reuse stopped being hypothetical faster than we expected. HealthyPi Studio, the desktop workstation for HealthyPi 6, is a different application, on a different platform, for a different board, speaking a different command set — and it depends on that same mcumgr_dart, firmware updates included. One SMP client, written for a watch, now drives a benchtop monitor.

Transport independence, for real. It runs over BLE today. The identical command set runs over USB-CDC or UART with no protocol change — a different pipe, not a different protocol. Which brings us to the next board.

7. HealthyPi 6: the same mechanism, a different board

HealthyPi 6 shares no silicon with Move. Move is an nRF5340; HealthyPi 6 is an STM32H757XI — Cortex-M7 at 400 MHz for the application, Cortex-M4 at 200 MHz doing QRS detection and HRV on CMSIS-DSP — alongside an ESP32-C6 network co-processor. Different vendor, different core count, different peripherals, different power envelope, different product.

What they share is Zephyr, and therefore MCUmgr.

HealthyPi 6 runs an SMP server over USB. One cable enumerates two serial ports: CDC 0 emits the sample stream and never answers a command, and CDC 1 carries MCUmgr/SMP — the stock os and settings groups, the datetime hook, the management notification hooks. Any off-the-shelf MCUmgr client can echo and reset it on day one.

And then, above that, a group of its own: group 64 — MGMT_GROUP_ID_PERUSER exactly, where Move took 0x1000 from further up the same reserved space. Around twenty-five commands, registered by the same one-line MCUMGR_HANDLER_DEFINE, covering ground the Healthy Store never needed:

Nothing in that table is a sync cursor, and that is the point. The two boards share no commands at all. What they share is that neither team had to write a protocol to get them — the whole surface is documented in docs/MCUMGR_COMMANDS.md, which opens by telling you the device speaks standard SMP and any client that can send a custom group id will do.

The netbuf constraint from §3.3 shows up here too, at a different number: HealthyPi 6 accepts SMP packets up to 512 bytes where Move raises its netbuf to 1024. Same mechanism, different budget, and so a different set of batch and paging constants falling out of it. That arithmetic is a per-board design input, not something you inherit.

The Healthy Store group itself hasn’t landed on HealthyPi 6 — there is no hpi_hs source in that tree, and a benchtop monitor with an SD card has different storage problems than a watch with a wear-levelled flash budget. When it does, the work is the handlers and the storage backend. Not the protocol, not the framing, not the transport, and not the client.

7.1 The transport we wrote and then turned off

One honest footnote, because it’s the most instructive part.

HealthyPi 6 briefly had a second SMP transport, bound to UART4, so a WiFi client could reach the MCUmgr server by byte-relaying through the ESP32-C6’s TCP↔UART passthrough. Writing it exposed exactly where Zephyr’s seams are. The uart_mcumgr console driver and the smp_uart transport are singletons bound to the single zephyr,uart-mcumgr chosen node, so a second UART transport can’t reuse them — both halves had to be mirrored against UART4 explicitly. But the framing and transport helpers underneath (mcumgr_serial_*, smp_transport_*) are generic and not tied to the chosen node, so the two coexisted, and every management group on the device became reachable over the second pipe at once.

It is not in the shipping build. UART4 became the HealthyBridge host link to the ESP32-C6, and two consumers on one UART is not a degraded mode — both call uart_rx_enable(), the second silently displaces the first, and the symptom is a link that stops carrying data for no visible reason. Rather than document that as a footgun, the Kconfig makes the combination unselectable:

config HPI_WIRELESS_OTA
    bool "MCUmgr SMP transport on UART4 (ESP32 wireless OTA gateway)"
    depends on MCUMGR_TRANSPORT_UART
    depends on !DT_HAS_PROTOCENTRAL_HEALTHYBRIDGE_ESP32_UART_ENABLED
    default n

The right answer is to carry SMP as a frame type inside the HealthyBridge protocol already running on that wire, which is where it’s headed. The lesson generalises: the generic parts of MCUmgr really are reusable across transports, and the parts that pin you are the chosen-node singletons. Knowing which is which before you start is worth an afternoon of reading.

That’s the modularity argument, and it’s the reason this was worth doing twice over. A custom protocol is a per-product cost. A management group is a per-product feature on top of infrastructure you already ship.

8. When you shouldn’t do this

Being fair about the boundaries:

• Streaming live waveforms. SMP is request/response with a bounded buffer. It’s right for bulk sync of stored data and wrong for a continuous high-rate feed — our live ECG view doesn’t go through it.
• You aren’t already running MCUmgr. If you don’t do DFU over SMP, you’re adding a stack to get a protocol, and the calculus is different. (Though “we should do DFU over SMP” is usually the better conclusion.)
• Very tight flash budgets. MCUmgr plus a transport plus zcbor isn’t free. It was trivially worth it for us because DFU had already paid for it.
• Non-Zephyr RTOS. MCUmgr exists outside Zephyr, but the ergonomics above — MCUMGR_HANDLER_DEFINE, the iterable-section registration, zcbor_bulk — are what make it this small.

9. Takeaways

If you’re about to define a BLE characteristic and a framing scheme for structured data on a Zephyr device, check first whether an MCUmgr group does the job:

1. Registration is a handler table, a struct mgmt_group, and one macro. Not a subsystem — a file.
2. Size the command set against the netbuf first. Batch limits and paging are design inputs, not afterthoughts.
3. Pack bulk payloads as a CBOR byte string, not an array of maps, or the buffer arithmetic won’t work.
4. Make sync cursor-based and idempotent, and publish both ends of the valid range so a stale cursor is detectable.
5. Version the schema and the command set separately, and keep the type registry on the device so new metrics don’t break old clients.
6. Document your paging, because the single-call case looks like it works.

The whole thing is open, on both boards.

• Move firmware — github.com/Protocentral/healthypi-move-fw (MIT). The Healthy Store group is under app/src/health/; the wire contract our own app was built from is docs/HPI_HS_API.md.
• HealthyPi 6 firmware — github.com/Protocentral/healthypi-6-fw (MIT). Group 64 is under app_m7/src/control/mcumgr_hpi/; the command reference is docs/MCUMGR_COMMANDS.md.
• Dart clients — mcumgr_dart (SMP) and healthypi_healthy_store (the Healthy Store group), both MIT.
• Python client — protocentral-healthypi, with every group-64 command pre-defined.
• Desktop app — HealthyPi Studio (MIT), mcumgr_dart in production on macOS, Windows and Linux.

Building something against the Healthy Store, or extending MCUmgr for your own device? Open an issue — we’d genuinely like to know what other people do with the group-id space.