MFC r361502: Do not remove upcall if we haven't yet.
This fixes assertion if we failed to bind listening HA socket.