Do not re-build ports unneccessarily

The previous patch has not completely fixed the issue, according to some
reports (it did on my system and for my set of installed ports).

This version contains a further change suggested by Walter Schwarzenfeld.

Submitted by: Walter Schwarzenfeld
Approved by: antoine (implicit)