pfctl: Revert "pfctl: Another set skip <group> fix"
This reverts commit 0c156a3c32cd0d9168570da5686ddc96abcbbc5a.
This fix broke using '<ifgroup>:network' in tables.
MFC after: 1 week
(cherry picked from commit d5b08e13dd6beb3436e181ff1f3e034cc8186584)