Remove wrong build dependency on m4
The only thing that depends on m4 is for building the testsuite, not the regular program