wtap: use typed rssi and noise floor values

Adjust the rssi and nf arguments to typed int8_t and adjust the maths
for rssi to be consistant with what net80211 expects.

Sponsored by: The FreeBSD Foundation
MFC after: 3 days
Reviewed by: lwhsu, adrian
Differential Revision: https://reviews.freebsd.org/D57020