Let linuxulator mprotect mask unsupported bits before calling kern_mprotect

[tijl]

Summary

After rS349240 kern_mprotect returns EINVAL for unsupported bits in the prot argument. Linux rtld uses PROT_GROWSDOWN and PROT_GROWS_UP when marking the stack executable. This patch masks these bits like kern_mprotect used to do. For other non-standard bits ENOTSUP is returned.

Here is a ktrace of a Linux program that fails to run:

17134 mserver  CALL  linux_mmap2(0,0x53ccb0,0x5,0x802,0x3,0)
17134 mserver  RET   linux_mmap2 675282944/0x28400000
17134 mserver  CALL  linux_mmap2(0x28879000,0x14000,0x3,0x812,0x3,0x479)
17134 mserver  RET   linux_mmap2 679972864/0x28879000
17134 mserver  CALL  linux_mmap2(0x2888d000,0xafcb0,0x3,0x32,0xffffffff,0)
17134 mserver  RET   linux_mmap2 680054784/0x2888d000
17134 mserver  CALL  linux_mprotect(0xffbfd000,0x1000,0x1000007)
17134 mserver  RET   linux_mprotect -1 errno -22 Invalid argument
17134 mserver  CALL  close(0x3)
17134 mserver  RET   close 0
17134 mserver  CALL  writev(0x2,0xffbfbff4,0xa)
17134 mserver  GIO   fd 2 wrote 137 bytes
      "./mserver: error while loading shared libraries: libmaple.so: cannot e\
       nable executable stack as shared object requires: Invalid argument
      "
17134 mserver  RET   writev 137/0x89
17134 mserver  CALL  linux_exit_group(0x7f)

[trasz]

Looks good. One thing you might want to check, if you have time, is that whether we should just ignore unknown flags? I seem to remember (although I might well be completely confused here) that Linux mmap used to ignore unknown bits; does linux mremap perhaps do the same?

Also, a shameless plug: linux-c7-strace decodes all those flags :-)

[trasz]

(And if you're feeling adventurous, the linux-ltp port contains mprotect tests; you might want to check those as well.)

[tijl]

In D20864#452323, @trasz wrote:

Looks good. One thing you might want to check, if you have time, is that whether we should just ignore unknown flags? I seem to remember (although I might well be completely confused here) that Linux mmap used to ignore unknown bits; does linux mremap perhaps do the same?

Linux mprotect seems to return EINVAL for unsupported bits so I'll change ENOTSUP to EINVAL in the patch:
https://elixir.bootlin.com/linux/v5.1.16/source/mm/mprotect.c#L478
https://elixir.bootlin.com/linux/v5.1.16/source/include/linux/mman.h#L95

[brooks]

It seems deeply weird that a program would set these flags with mprotect, but this change seems fine (modulo changing the error to EINVAL).