mlx5: Add a no-append flow insertion mode
(cherry picked from commit 6a6af22b6ec42eb0436381fae8dd1b86bc4c6327)