I'm puzzled by this (original) code and that gpioctl don't allow you to clear any flags.
If for any reason I set some inversion flag on a pin, how am I supposed to clear it?

gpioctl -c 40 OUT IO # Set the INVOUT flag and call nct_set_pin_inverted with 0
gpioctl -c 40 IN II # Set the INVIN flag and call nct_set_pin_inverted with 1, i.e., effectively enabling inversion
gpioctl -c 40 OUT # Don't change the flags and don't call nct_set_pin_inverted

It looks like we're missing gpioctl -c 40 -IO to remove the INVOUT flag and call nct_set_pin_inverted with 0.

Note that NCT chips don't seems to distinguish between input inversion and output inversion.

(Replying to myself)
To avoid patching gpioctl, I propose the following code. It solve two problems:

• allow clearing the inversion flag
• and help ensure INVIN and INVOUT are treated as a single flag, i.e., gpioctl -c 40 OUT IO set INVIN too and gpioctl -c 40 OUT clear both.

/* inversion */
pin->gp_flags &= ~(GPIO_PIN_INVIN | GPIO_PIN_INVOUT);
if ((flags & (GPIO_PIN_INVIN | GPIO_PIN_INVOUT)) != 0) {
	nct_set_pin_inverted(sc, pin_num, 1);
	pin->gp_flags |= (GPIO_PIN_INVIN | GPIO_PIN_INVOUT);
} else
	nct_set_pin_inverted(sc, pin_num, 0);