MFC r341523:
ibcore: Don't access invalid port.

The port number in the listen_id_priv has been observed to be zero which
means no port has been selected. The current code lacks a check for invalid
port number.

Sponsored by: Mellanox Technologies