[skip ci] improvements to cap_sysctl.3

• Correct some function prototypes which were documented with the wrong pointer type.
• Clarify return values and requirements for freeing the limit handle.

MFC after: 1 week
Sponsored by: Axcient
Reviewed by: oshogbo
Differential Revision: https://reviews.freebsd.org/D37586