Fix the conversion to use nan_mix() in r336362. fmod*(x, y),
remainder*(x, y) and remquo*(x, y, quo) were broken for y = 0 by changing
multiplication by y to addition of y. (When y is 0, the result should be
NaN but became 1 for finite x.)

Use a new macro nan_mix_op() to give more control over the mixing, and
expand comments.

Recent re-testing missed finding this bug since I only tested the macro
version on amd64 and i386 and these arches don't use the C versions (they
use either asm versions or builtins).

Reported by: enh via freebsd-numerics