libpfctl: fix error handling

In two cases we returned E2BIG where it should have been a boolean ('false').

MFC after: 1 week
Sponsored by: Rubicon Communications, LLC ("Netgate")

(cherry picked from commit 9bb1c46b4c38c44565fc24f13acdc19b0cb5a1c4)