tcp: fix conversion of rttvar

A wrong variable and wrong scaling factors were used.

Reported by: Coverity Scan
CID: 1508689
Reviewed by: rscheff
MFC after: 1 week
Sponsored by: Netflix, Inc.
Differential Revision: https://reviews.freebsd.org/D44612