posixshm_test: Fix sign mismatches in ?: results.

GCC 12's -Wsign-compare complains if the two alternative results of
the ?: operator are differently signed. Cast the small, sub-page
off_t values to size_t to quiet the warning.

Reviewed by: imp, kib
Differential Revision: https://reviews.freebsd.org/D37539