Bluetooth: L2CAP: Hold incoming data until encryption key size is known
Once encryption is enabled on an ACL link, the encryption key size is
read from the controller, using hdev->min_enc_key_size in the meantime.
Data received before the reply is processed with that key size, so a
peer starting SMP over BR/EDR right after encryption, which it may do,
gets its Pairing Request rejected with "Encryption Key Size" whenever
the pending security level is FIPS, as that requires a 16 octet key:
> HCI Event: Encryption Change (0x08) plen 4
> ACL Data RX: Handle 1 flags 0x02 dlen 11
BR/EDR SMP: Pairing Request (0x01) len 6
@ MGMT Event: Authentication Failed (0x0011) plen 8
Status: Authentication Failed (0x05)
< HCI Command: Read Encryption Key Size (0x05|0x0008) plen 2
> HCI Event: Command Complete (0x0e) plen 7
Key size: 16
< ACL Data TX: Handle 1 flags 0x00 dlen 6
BR/EDR SMP: Pairing Failed (0x05) len 1
Reason: Encryption key size (0x06)
Mark the connection with HCI_CONN_ENC_KEY_SIZE_PEND while the key size
is read and hold incoming L2CAP data back in pending_rx meanwhile,
processing it once the reply is handled.
Found with mgmt-tester "Pairing Acceptor - SMP over BR/EDR 2" under User
Mode Linux time travel, where the emulated peer always replies before
the key size is read.
Fixes: e3f6a257a73d ("Bluetooth: Use actual encryption key size for SMP over BR/EDR")
Assisted-by: OpenCode:claude-opus-5.5
Signed-off-by: Luiz Augusto von Dentz <luiz.von.dentz@intel.com>
3 files changed