arm64: Check if we have a map before checking the flags

This fixes amdgpu on arm64 where linuxkpi is calling id_mapped
and we call might_bounce without a map.