This is a little unclear. Maybe:

The password implementation only provides compatibility with VNC clients.
It does not provide any additional security for the VNC protocol.

(Which is still a little weak, probably ought to say something about VNC being unencrypted and all.)

I think this will break building with OpenSSL disabled (WITHOUT_OPENSSL=1).

You could disable the new code if MK_OPENSSL is no.

This comment needs to be adjusted.

Although it is consistent with the rest of this code, it is wrong to use buf without checking how much data was actually read.

This truncates or pads the password to 16 bytes, while the comment says 8.

These constants are not necessary for rfb's clients and can therefore go into rfb.c.

Why not just quote the RFC?

This type of authentication is known to be cryptographically weak and
   is not intended for use on untrusted networks.  Many implementations
   will want to use stronger security, such as running the session over
   an encrypted channel provided by IPsec [RFC4301] or SSH [RFC4254].

Leftover debug ?

type -> CFLAGS

I don't think the RFC references add anything here and could be removed.

May want to warn (or error out) if the password is longer than 8 bytes, since that is length used in the key generation.

The bit reversal can be done with a 1-liner as per the example code I sent

static uint8_t
bit_reverse(uint8_t val)
{
/* From https://graphics.stanford.edu/~seander/bithacks.html */
return (((val * 0x80200802ULL) & 0x0884422110ULL) *

	    0x0101010101ULL >> 32);

}

Might want to see what other VNC serves write back as the error here and copy it.

How do we conditional the man page? password= well not work if its build without OpenSSL, but the man page provides no info about this possible problem.

Reasonable, just strike the RFC 4301 and RFC4254 which are bound to change over time anyway, refering to IPsec and SSH leads the user down the right path.

Warning would be fine, error would be a POLA, many people use vnc passwords longer than the 8 bytes supported by the protocol and every server and client I have run either silently ignores this, or beeps as you type extra characters.

This should probably be a call to usage(), and usage() should be adjusted to have -password or no -password based on #ifdef HAVEW_OPENSSL, at least that is more the unix norm for commands that do not parse from unknown arguments.

Isnt this just the same as strcpy(buf + 4, message)?

Is strlen(message) somehow bounds restricted some place?

Yes, I will remove in the next diff update.

Very bad typo, don't know how that happens :D

As far as I know, we don't have it. I guess we have the same situation with RFB without UEFI.

Done! Will be updated in the next diff.

I'm with @rgrimes, I never saw any other VNC server/client report it, most of those that I saw run silently or just ignores it.

Yes I saw that, but I thought would be better avoid two loops to do the same thing.

What do you think?

RealVNC returns 'Wrong Password'. There is no standard, IMHO, the current message looks understandable.

Actually we just bypass the password in case there is no OpenSSL and print out a message in case user didn't notice that his OS is without OpenSSL.

strncpy() avoid buffer overflow. See at strcpy(3).

The use of strlen(message) makes this have the exact some buffer overflow bug that using strcpy would have.

I can't see that; Do you have a better suggestion?

There's no need to copy the string into a buffer. You can simply stream_write() the byte-swapped size, and follow it with a stream_write() of the string.