FFmpeg can handle AAC, FLAC, MP3, MP4, OPUS, VORBIS, WAV making these dependencies unnecessary.
The idea is to minimize the amount of dependencies, try to consolidate overall in the long run and possibly try to deprecate some such as madlib which isn't seen development in years and has known vulns and is quite inefficient. While the default configuration of ffmpeg pulls in opus and vorbis they aren't necessarily needed as ffmpeg supports decoding of both these formats without any external libraries. This will also make it potentially less tedious when doing custom packages/repos and you want to avoid getting unnecessary dependencies pulled in.
If libmad is vulnerable and should be removed in the near future, why disable the other input plugins? What about the other ~80 ports that depend on libmad? I strongly suggest opening a PR (with patch) for marking audio/libmad vulnerable in vuln.xml to inform users about it.
That being said I am going to disable MAD by default in cmus. FFMPEG is the fallback input plugin and seeking in files is slower than with the other plugins, so I'd rather keep them on by default unless there is a more compelling reason than saving a few small dependencies.
It's in the works, I'm slowly working my way through the list so it's on my todo list. Many utilities in /audio have newer counterparts or so but I'm not sure how you would go about deprecating those in a later stage.
Are you experience performance issues with any specific format as it pretty much instantly with indexed formats such as m4a which is to be expected however I noticed a minor difference using MP3s on my ARMv7 box jumping 1h or so however ffmpeg seems to be a bit more accurate in that regard.