Fix the way 'factor' behaves when using OpenSSL to match the description
of how it works when not compiled with OpenSSL.

Also, allow users to specify a hexadecimal number by using a prefix of
'0x'. Before this, users could only specify a hexadecimal value if that
value included a hex digit ('a'-'f') in the value.

PR: 243136
Submitted by: Steve Kargl
Reviewed by: gad
MFC after: 3 weeks