dmu_objset_hold_flags() should call dsl_dataset_rele_flags() on error

This was caught when doing a manual check to see if #17352 needed to be
improved to catch mismatches across stack frames of the kind that were
first found in #17340.

Reviewed-by: George Amanakis <gamanakis@gmail.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Alexander Motin <mav@FreeBSD.org>
Signed-off-by: Richard Yao <richard@ryao.dev>
Closes #17353
(cherry picked from commit 83fa80a5500ed7baccedfa90ad55ac879af710c9)
(cherry picked from commit 07d815f2a06573514f51c8601aa60db6fe1a76ad)