Simplify both the patches and the Makefile.
Remove various patches and bodges that have since landed upstream.