Only set pud settings if this is a pullup or pulldown configuration.
This removes the need to set the MMC pins with pullups in our DTS.
Thanks to jmcneill@ for spotting this.

Tested on Orange Pi One (Allwinner H3).

MFC after: 1 week