@emaste wrote:

I am concerned about invalid/illegal sequences and note that errors from mbtowc() are not handled.

Right, esp. that it can return -1, I have to fix this. I guess we can judge about character validity based on its return value.

In theory you have to check for the Extended Capabilities element (127, IEEE80211_ELEMID_EXTCAP) to be present in the same frame that contains the SSID and bit 48 (SSID is UTF-8 encoded) being set.
If that is not there or it is not set the encoding is undefined; I think there was an exception if you knew from previous frames which had the extcap; I'd have to go and look it up.

So the proper solution is slightly more complex and I'd suggest adding that check before doing an UTF-8 decode.

@bz wrote:

In theory you have to check for the Extended Capabilities element (127, IEEE80211_ELEMID_EXTCAP) to be present in the same frame that contains the SSID and bit 48 (SSID is UTF-8 encoded) being set.

So the proper solution is slightly more complex and I'd suggest adding that check before doing an UTF-8 decode.

Very good points, yes we should do that.

@bz wrote:

In theory you have to check for the Extended Capabilities element (127, IEEE80211_ELEMID_EXTCAP) to be present in the same frame that contains the SSID and bit 48 (SSID is UTF-8 encoded) being set.

Hi Bjoern, sorry for not looking into this sooner rather than later, but yesterday I finally did. :-) Apparently we currently don't make any use (and thus keep track) of extended capabilities, per what I see in net80211/ieee80211_input.c. So we first have to add missing bits to wlan.ko. I've modified the code like this:

diff --git a/sys/net80211/ieee80211_input.c b/sys/net80211/ieee80211_input.c
index 66a5ba1c4035..04e03acef652 100644
--- a/sys/net80211/ieee80211_input.c
+++ b/sys/net80211/ieee80211_input.c
@@ -560,6 +560,8 @@ ieee80211_parse_beacon(struct ieee80211_node *ni, struct mbuf *m,
 	scan->ies = frm;
 	scan->ies_len = efrm - frm;
 
+	int utf8_ssid = 0;
+
 	while (efrm - frm > 1) {
 		IEEE80211_VERIFY_LENGTH(efrm - frm, frm[1] + 2,
 		    return (scan->status = IEEE80211_BPARSE_BADIELEN));
@@ -642,8 +644,11 @@ ieee80211_parse_beacon(struct ieee80211_node *ni, struct mbuf *m,
 			scan->meshconf = frm;
 			break;
 #endif
-		/* Extended capabilities; nothing handles it for now */
+		/* Extended capabilities; we only check bit 48 (UTF-8 SSID) */
 		case IEEE80211_ELEMID_EXTCAP:
+			utf8_ssid = 2;
+			if (frm[1] >= 7 && frm[8] & 0x01)
+				utf8_ssid = 3;
 			break;
 		case IEEE80211_ELEMID_VENDOR:
 			if (iswpaoui(frm))
@@ -733,6 +738,8 @@ ieee80211_parse_beacon(struct ieee80211_node *ni, struct mbuf *m,
 		 */
 		IEEE80211_VERIFY_LENGTH(scan->country[1], 3 * sizeof(uint8_t),
 		    scan->country = NULL);
+		if (utf8_ssid > 0)
+			scan->country[utf8_ssid] |= 0x20;
 	}
 	if (scan->csa != NULL) {
 		/*

So if an EXTCAP element is present in the frame, I'd lowercase the first country code letter, and if bit 48 (first bit of the 7th octet) is set, I'd lowercase the second country code letter. This way I can easily see the results in ifconfig -v wlan0 list scan without modifying anything else. To my surprise, while most frames did contain extended capabilities, none of them indicated that SSID is UTF-8 encoded (while it really was) and one AP with non-Latin UTF-8 SSID did not send any EXPCAP element. Am I peeking at the wrong place? Or is it really the case that many APs simply do not set that bit even if the SSID is indeed encoded in UTF-8, per this comment?