mbedTLS P-256 ECDH path passes NULL RNG to `mbedtls_ecp_mul()` and silently produces invalid LESC DHKey

[aussetg]

Describe the bug

When BTstack is built with LE Secure Connections and the mbedTLS P-256 backend:

#define ENABLE_LE_SECURE_CONNECTIONS
#define HAVE_MBEDTLS_ECC_P256

the software DHKey calculation in src/btstack_crypto.c calls mbedtls_ecp_mul() with a null RNG callback and ignores the return value:

mbedtls_ecp_mul(&mbedtls_ec_group, &DH, &d, &Q, NULL, NULL);
mbedtls_mpi_write_binary(&DH.X, btstack_crypto_ec_p192->dhkey, 32);

Current mbedTLS requires f_rng != NULL for mbedtls_ecp_mul(). With NULL, it returns MBEDTLS_ERR_ECP_BAD_INPUT_DATA (-0x4F80). BTstack then writes DH.X anyway; in the reproducer this yields an invalid all-zero DHKey. In real LE Secure Connections pairing this can surface as SM_REASON_DHKEY_CHECK_FAILED (0x0B).

The entire btstack flow leading to the error is:

1. BTstack selects mbedTLS software ECC

   btstack/src/btstack_crypto.c

   Lines 92 to 105 in 5bc5cbd

   	// Software ECC-P256 implementation provided by mbedTLS, allow config via MBEDTLS_CONFIG_FILE
   	#ifdef HAVE_MBEDTLS_ECC_P256
   	#define ENABLE_ECC_P256
   	#define USE_MBEDTLS_ECC_P256
   	#define USE_SOFTWARE_ECC_P256_IMPLEMENTATION
   	#ifdef MBEDTLS_CONFIG_FILE
   	// cppcheck-suppress preprocessorErrorDirective
   	#include MBEDTLS_CONFIG_FILE
   	#else
   	#include "mbedtls/mbedtls_config.h"
   	#endif
   	#include "mbedtls/platform.h"
   	#include "mbedtls/ecp.h"
   	#endif

2. Key generation receives a BTstack-provided RNG wrapper
   BTstack prefetches random bytes using HCI_LE_Rand and feeds them through this wrapper:

   btstack/src/btstack_crypto.c

   Lines 499 to 520 in 5bc5cbd

   	#if (defined(USE_MICRO_ECC_P256) && !defined(WICED_VERSION)) || defined(USE_MBEDTLS_ECC_P256)
   	// @return OK
   	static int sm_generate_f_rng(unsigned char * buffer, unsigned size){
   	if (btstack_crypto_ecc_p256_key_generation_state != ECC_P256_KEY_GENERATION_ACTIVE) return 0;
   	log_info("sm_generate_f_rng: size %u - offset %u", (int) size, btstack_crypto_ecc_p256_random_offset);
   	btstack_assert((btstack_crypto_ecc_p256_random_offset + size) <= btstack_crypto_ecc_p256_random_len);
   	uint16_t remaining_size = size;
   	uint8_t * buffer_ptr = buffer;
   	while (remaining_size) {
   	*buffer_ptr++ = btstack_crypto_ecc_p256_random[btstack_crypto_ecc_p256_random_offset++];
   	remaining_size--;
   	}
   	return 1;
   	}
   	#endif
   	#ifdef USE_MBEDTLS_ECC_P256
   	// @return error - just wrap sm_generate_f_rng
   	static int sm_generate_f_rng_mbedtls(void * context, unsigned char * buffer, size_t size){
   	UNUSED(context);
   	return sm_generate_f_rng(buffer, size) == 0;
   	}
   	#endif /* USE_MBEDTLS_ECC_P256 */

   
   Key generation uses it:

   btstack/src/btstack_crypto.c

   Line 553 in 5bc5cbd

   int res = mbedtls_ecp_gen_keypair(&mbedtls_ec_group, &d, &P, &sm_generate_f_rng_mbedtls, NULL);

3. DHKey calculation passes no RNG and ignores the return value:

   btstack/src/btstack_crypto.c

   Lines 585 to 590 in 5bc5cbd

   	mbedtls_mpi_read_binary(&d, btstack_crypto_ecc_p256_d, 32);
   	mbedtls_mpi_read_binary(&Q.X, &btstack_crypto_ec_p192->public_key[0] , 32);
   	mbedtls_mpi_read_binary(&Q.Y, &btstack_crypto_ec_p192->public_key[32], 32);
   	mbedtls_mpi_lset(&Q.Z, 1);
   	mbedtls_ecp_mul(&mbedtls_ec_group, &DH, &d, &Q, NULL, NULL);
   	mbedtls_mpi_write_binary(&DH.X, btstack_crypto_ec_p192->dhkey, 32);

   
   Problems:
   1. mbedtls_ecp_mul() is called with f_rng == NULL.
   2. The return value of mbedtls_ecp_mul() is ignored.
   3. The return values of the surrounding mbedTLS calls are also ignored.
   4. If multiplication fails, DH.X is not a valid shared secret, but BTstack still writes it into dhkey and continues the SMP state machine.
4. Current mbedTLS requires a non-null RNG for mbedtls_ecp_mul() :
   In mbedtls/ecp.h, mbedtls_ecp_mul() documents: https://github.com/Mbed-TLS/mbedtls/blob/e185d7fd85499c8ce5ca2a54f5cf8fe7dbe3f8df/include/mbedtls/ecp.h#L944-L957
   In library/ecp.c, the implementation enforces that: https://github.com/Mbed-TLS/mbedtls/blob/e185d7fd85499c8ce5ca2a54f5cf8fe7dbe3f8df/library/ecp.c#L2671-L2681
5. btstack surfaces it as DHKEY_CHECK_FAILED

Expected behavior

BTstack should either provide a valid RNG callback to mbedtls_ecp_mul() and check all mbedTLS return values, or abort the DHKey calculation clearly if no RNG is available / mbedTLS returns an error.

It should not silently write DH.X after a failed multiplication and continue pairing with an invalid DHKey.
Controller-based ECDH actually already has such a failure mode:

btstack/src/btstack_crypto.c

Lines 1161 to 1167 in 5bc5cbd

	if (hci_subevent_le_generate_dhkey_complete_get_status(packet)){
	log_error("Generate DHKEY failed -> abort");
	// set DHKEY to 0xff..ff
	memset(btstack_crypto_ec_p192->dhkey, 0xff, 32);
	} else {
	hci_subevent_le_generate_dhkey_complete_get_dhkey(packet, btstack_crypto_ec_p192->dhkey);
	}

HCI Packet Logs

No .pklg packet log is included for this minimal reproduction.

The failure is deterministic in BTstack's local mbedTLS-backed P-256 DHKey calculation path before any peer-specific HCI analysis is needed. The repro execute the same mbedTLS call sequence used by BTstack's HAVE_MBEDTLS_ECC_P256 ECDH path:

 mbedtls_ecp_mul(&group, &DH, &d, &Q, NULL, NULL);
 mbedtls_mpi_write_binary(&DH.X, dhkey, 32);

Logs show mbedtls_ecp_mul(..., NULL, NULL) returning MBEDTLS_ERR_ECP_BAD_INPUT_DATA (-0x4F80), followed by DH.X being written as an all-zero invalid DHKey.

Environment: (please complete the following information):

• Current BTstack branch: master ( 5bc5cbdbeec33be1fdbd0d50e04c0f6deab99d2d )
• Bluetooth Controller: irrelevant but MT7925
• Remote device: None needed
• Tool/Libraries versions:
  • cmake version 4.3.2
  • cc (GCC) 16.1.1 20260430
  • Linux framework 7.0.5-2-cachyos Setting correct length for BTSTACK_EVENT_STATE event. #1 SMP PREEMPT_DYNAMIC Sat, 09 May 2026 11:29:28 +0000 x86_64 GNU/Linux
  • mbedTLS: Mbed TLS 3.6.2 from Pico SDK checkout, commit 107ea89daaefb9867ea9121002fbbdf926780e98 and local Linux package mbedtls 3.6.5-1 (pkg-config --modversion mbedcrypto reports 3.6.5).
  • bluetoothctl: 5.86

Additional context

I reproduced the bug both on an embedded device (Pico 2 W) and locally. It should however not matter as for the repro we don't even need btstack or any bluetooth really.

We actually just need to reproduce this very specific path:

btstack/src/btstack_crypto.c

Lines 578 to 593 in 5bc5cbd

	// da * Pb
	mbedtls_mpi d;
	mbedtls_ecp_point Q;
	mbedtls_ecp_point DH;
	mbedtls_mpi_init(&d);
	mbedtls_ecp_point_init(&Q);
	mbedtls_ecp_point_init(&DH);
	mbedtls_mpi_read_binary(&d, btstack_crypto_ecc_p256_d, 32);
	mbedtls_mpi_read_binary(&Q.X, &btstack_crypto_ec_p192->public_key[0] , 32);
	mbedtls_mpi_read_binary(&Q.Y, &btstack_crypto_ec_p192->public_key[32], 32);
	mbedtls_mpi_lset(&Q.Z, 1);
	mbedtls_ecp_mul(&mbedtls_ec_group, &DH, &d, &Q, NULL, NULL);
	mbedtls_mpi_write_binary(&DH.X, btstack_crypto_ec_p192->dhkey, 32);
	mbedtls_ecp_point_free(&DH);
	mbedtls_mpi_free(&d);
	mbedtls_ecp_point_free(&Q);

with

// Minimal Linux-only source-level repro for BTstack's mbedTLS LESC DHKey bug.
//
// This intentionally mirrors the problematic shape in BTstack's
// HAVE_MBEDTLS_ECC_P256 ECDH path: call mbedtls_ecp_mul() with a NULL RNG,
// ignore the failure, and then write DH.X as if it were a valid DHKey.

#include <stdbool.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>

#include "mbedtls/build_info.h"
#include "mbedtls/ecp.h"

static bool all_zero(const uint8_t *buf, size_t len) {
    uint8_t acc = 0;
    for (size_t i = 0; i < len; i++) acc |= buf[i];
    return acc == 0;
}

static void print_hex(const uint8_t *buf, size_t len) {
    for (size_t i = 0; i < len; i++) printf("%02x", buf[i]);
    printf("\n");
}

int main(void) {
    mbedtls_ecp_group group;
    mbedtls_mpi d;
    mbedtls_ecp_point dh;
    uint8_t dhkey[32];

    mbedtls_ecp_group_init(&group);
    mbedtls_mpi_init(&d);
    mbedtls_ecp_point_init(&dh);
    memset(dhkey, 0xcc, sizeof(dhkey));

    int load_err = mbedtls_ecp_group_load(&group, MBEDTLS_ECP_DP_SECP256R1);
    int scalar_err = mbedtls_mpi_lset(&d, 1);

    // Repro: current mbedTLS requires f_rng != NULL here.
    int mul_err = mbedtls_ecp_mul(&group, &dh, &d, &group.G, NULL, NULL);

    // BTstack's affected path writes DH.X regardless of the mbedtls_ecp_mul result.
    int write_err = mbedtls_mpi_write_binary(&dh.MBEDTLS_PRIVATE(X), dhkey, sizeof(dhkey));

    printf("mbedTLS version: %s\n", MBEDTLS_VERSION_STRING_FULL);
    printf("load_err=%d scalar_err=%d\n", load_err, scalar_err);
    printf("mbedtls_ecp_mul(..., NULL, NULL) err=%d (0x%04x)\n", mul_err, (unsigned)(-mul_err));
    printf("mbedtls_mpi_write_binary(DH.X) err=%d\n", write_err);
    printf("dhkey_after_failed_mul=");
    print_hex(dhkey, sizeof(dhkey));
    printf("dhkey_after_failed_mul_all_zero=%s\n", all_zero(dhkey, sizeof(dhkey)) ? "yes" : "no");

    bool reproduced = load_err == 0 && scalar_err == 0 &&
                      mul_err == MBEDTLS_ERR_ECP_BAD_INPUT_DATA &&
                      write_err == 0 && all_zero(dhkey, sizeof(dhkey));
    printf("RESULT: %s\n", reproduced ? "REPRODUCED" : "NOT_REPRODUCED");

    mbedtls_ecp_point_free(&dh);
    mbedtls_mpi_free(&d);
    mbedtls_ecp_group_free(&group);
    return reproduced ? 0 : 1;
}

You should see something like:

mbedTLS version: Mbed TLS 3.6.2
load_err=0 scalar_err=0
mbedtls_ecp_mul(..., NULL, NULL) err=-20352 (0x4f80)
mbedtls_mpi_write_binary(DH.X) err=0
dhkey_after_failed_mul=0000000000000000000000000000000000000000000000000000000000000000
dhkey_after_failed_mul_all_zero=yes

Disclaimer: I have used AI to isolate the bug from my own initial code but I am the one writing this bug report.