loader: handle malloc failures in vbe_init, use consistent naming

Handle malloc failures in vbe_init().

If it should so happen and we do get malloc failure in vbe_init(),
use original mode list.

Replace nitems with nentries to have naming consistency and avoid
confusion with nitems() macro.

Reported by: yuripv, rpokala