headers: Implement _ISOC11_SOURCES macro when __POSIX_C_SOURCE defined

[imp]

Summary

When _ISOC11_SOURCES is defined for glibc at the same time
POSIX_C_SOURCE is defined, it extends the POSIX_C_SOURCE definition
by exaclty what C11 adds to the spec for each system header. We follow
both OpenBSD's and glibc's convention by also C11 or higher compliation
mode is selected.

The Open Group is working on issuing a new version of the POSIX standard
that will realign the standard from C99 to a newer version of C. This
commit is a stop-gap measure for greater compatibility until that
environment has been standardized.

PR: 255290
Sponsored by: Netflix

[brooks]

"definfed" in the title line.

I think this is a better way to resolve the undefined behavior of requesting POSIX and c11. It really doesn't make sense that you could run c11 and not get C11 features.

[imp]

In D29902#671000, @brooks wrote:

"definfed" in the title line.

I think this is a better way to resolve the undefined behavior of requesting POSIX and c11. It really doesn't make sense that you could run c11 and not get C11 features.

It doesn't make sense that there's no way to get it at all. This gives a way that's compatible with other systems to opt-in.

thanks for the title correction. Doh!

[imp]

Update comment to be better worded (hopefully). No change in functionality.

[imp]

Another pass to improve wording...

[imp]

better?

[imp]

final?

[arichardson]

Seems reasonable to me.

sys/sys/cdefs.h

724

725

[olce]

Please see my latest comment in bug #255290.

In particular, POSIX allows an implementation to define any feature test macro they wish. There is no need that they are standardized.

I understand that, so far, you seemed to prefer to keep compatibility for strictly conforming programs compiled with cc by not automatically making C11 additions visible, in particular considering that the default standard for cc will change at some point. But I should point out that:

1. This is not required by the standard (again, see latest comment in the bug).
2. It would be a very surprising behavior that, if C11 is explicitly requested, some expected symbols are not defined.

So I think that, if C11 is explicitly requested, the relevant symbols should _always_ be defined. When C11 becomes the default for cc, you could keep the C99 symbols if _POSIX_C_SOURCE is defined if you absolutely wish, for compatibility with strictly conforming applications. Personally, I would ditch that entirely, because nothing in the standard says so. Strictly conforming programs should be built with c99, not cc.

sys/sys/cdefs.h

724

Some typos:
complaint => compliant
definning => defining

725

Also:
complaint => compliant

732–736

glibc compatibility is good.

What about _C99_SOURCE, used below in the _ANSI_SOURCE block? I could not relate it to any standard. Shouldn't it be taken into account as well?

[olce]

sys/sys/cdefs.h

732–736

Sorry, obviously, I meant _C11_SOURCE.

[imp]

In particular, POSIX allows an implementation to define any feature test macro they wish. There is no need that they are standardized.

This is exactly what we're doing here. We're just choosing to follow the glibc practice as prior art.

So I think that, if C11 is explicitly requested...

Your opinion has been noted. I have a different reading of the standard, though. There's a conflict here. By requesting c11, you're not defining a new feature test macro to request more symbols to be defined. It may be convenient, but goes against the last 30ish years of what POSIX_C_SOURCE has done in the our tree, and what I believe to be the intent of the standard: to request that only certain, well known symbols be defined. The standard is very clear on this: you may only define the symbols specified by the standard. That's what I've done here. The standard also allows additional feature test macros to be defined and those will allow additional symbols to be defined. That's also what I've done here.

Had I done otherwise, there'd be no way to request c11 language semantics from the compiler, but not get the c11 symbols from the headers defined. The bug before was not being an escape hatch to ask for them to be defined also. Now there is. It won't be automatic in this iteration because it mixes up two concepts we've tried very hard to keep separate for a long time. And the standard does not compel it to be automatic. There's a difference of opinion about whether it allows it or not, and I've landed on a scheme that makes it possible for both styles to be used, whereas an automatic scheme does not.

Since these headers are used for all the different compilers that can be installed on FreeBSD, I'm not moved by saying the c99 compiler has to be named something specific.

The next issue will likely realign the posix / open group standard to a newer C version. When that's standardized, it will be implemented. Until then, there will be a diversity of opinion about the standard.

sys/sys/cdefs.h

732–736

It's already handled below. It's the default.

[imp]

typos from the review

[olce]

In particular, POSIX allows an implementation to define any feature test macro they wish. There is no need that they are standardized.

This is exactly what we're doing here. We're just choosing to follow the glibc practice as prior art.

I completely agree. But I don't think this is what you were implying at first. (To quote you from the header: "There's a diversity of opinion about whether this automatic extension is standards conformant, so only enable C11 visibility when specifically requested since the Open Group standard allows this when other feature macros are defined by other standards.".)

So I think that, if C11 is explicitly requested...

Your opinion has been noted. I have a different reading of the standard, though. There's a conflict here. By requesting c11, you're not defining a new feature test macro to request more symbols to be defined. It may be convenient, but goes against the last 30ish years of what POSIX_C_SOURCE has done in the our tree, and what I believe to be the intent of the standard: to request that only certain, well known symbols be defined. The standard is very clear on this: you may only define the symbols specified by the standard. That's what I've done here. The standard also allows additional feature test macros to be defined and those will allow additional symbols to be defined. That's also what I've done here.

So, no, we don't have a different reading of the standard on that point (either I misunderstood you, or it's your view that changed). I never said that the standard required that requesting C11 implied a new feature test macro, sorry if that was confusing (I employed "should" in the bug in the sense that we should do it, not to say that it was required). But the standard simply leaves that open (the application can define feature test macros or not as it wants, and the system may interpret them as it wants; arguments to the compiler on the command-line are arguably part of what constitutes the application). At which point, this is a choice to make on our side.

Had I done otherwise, there'd be no way to request c11 language semantics from the compiler, but not get the c11 symbols from the headers defined.

Surely. But I fail to see why anyone would want that. I've never stumbled on such a use case in 20 years. And the bug you're fixing is not about that, they do want the C11 symbols.

It won't be automatic in this iteration because it mixes up two concepts we've tried very hard to keep separate for a long time.

Does this approach have other (dis)advantages than the one you and I already pointed out? (Pointers to anything clueful welcome.)

And the standard does not compel it to be automatic.

We agree.

But then you get surprising behavior (C11 requested on the command-line but symbols not available). That's why my opinion is to implement the change (define a feature set macro on C11 requested).

And I understand that you (and others) don't have the same opinion, and that you want to leave this out for later (although this was what the bug was about initially; your change instead allows a workaround for them, but indeed they had one already).

There's a difference of opinion about whether it allows it or not (snip)

That's where I would have ideally been interested in seeing technical arguments and quotations. Because I stand again on the fact that the standard allows that, after all the quotations I made. And I'm finding even more material to back it:
XRAT
B.2.2 POSIX.1 Symbols
(snip)
It is further intended that these feature test macros apply only to the headers
specified by POSIX.1-2017. Implementations are expressly permitted to make
visible symbols not specified by POSIX.1-2017, within both POSIX.1 and other
headers, under the control of feature test macros that are not defined by
POSIX.1-2017.
(snip)
For instance, when _POSIX_C_SOURCE is defined as 200809L, the system should
make visible the same name space as permitted and required by the POSIX.1-2017
standard.
(Emphasis is mine.)

Feature macros are here for the application to request symbol (non-)visibility.

Putting an explicit C standard on a compiler command-line is arguably an application request for symbol visibility, among others. Is that where we disagree?

Since these headers are used for all the different compilers that can be installed on FreeBSD, I'm not moved by saying the c99 compiler has to be named something specific.

Since any compiler other than c99 doesn't exist in the eye of the standard, why force C99 on programs defining _POSIX_C_SOURCE to 200809L? If they don't work without C11, they are not compliant anyway.

In your interpretation, I think it would be best that _POSIX_C_SOURCE set to 200809L and C11 requested result in a plain #error interrupting compilation. Which is what I would prefer to avoid.

The next issue will likely realign the posix / open group standard to a newer C version. When that's standardized, it will be implemented. Until then, there will be a diversity of opinion about the standard.

This won't change the fact that defining _POSIX_C_SOURCE alone never required a conforming system to compile and run the program according to the designated version of the standard, which is clear on this topic beyond any doubt (see quotations in the bug). The only system requirement on _POSIX_C_SOURCE is on symbol visibility, nothing more.

I'm surprised to learn that FreeBSD has been interpreting it like that (which doesn't contradict the standard, but is a very strict choice on top of it), and I'm duly noting it.

In any case, thanks for you time, and for this change, which is a step forward.

[imp]

I am waiting to hear what the project's traditional experts say here as well.

A quick survey of other, easy to snag libc shows inconsistent behavior:

NetBSD defines a couple of things based on _ISOC11_SOURCE || __STDC_VERSION >= 201101L, but those are only:
static_assert, alligned_alloc, at_quick_exit, quick_exit. It appears it also defines max_align_t always for C11 mode. But the rest of the C11 standard is basically unimplemented, though the intent is clear.

OpenBSD takes the expansive view that version overrides the specific macros for what's visible, though their comments are inconsistent with the code, but the code is fairly clear of the intent.

musl is a hodge-podge of what's defined and what's not. Mostly it defaults to C11, and POSIX_C_SOURCE doesn't limit things quite like it should. There's some attempt to give a strict namespace, but it's hit or miss, so a lot of C11 functions bleed through, even when compiling with C99.

So from a practical point of view, there can be an expectation that these symbols be defined. I'd be interested in what @wollman has to say, though he may demure and say only this combination isn't well defined since the compilation environment is effectively c99 and the standard is unhelpful about what to do in other environments.

[olce]

sys/sys/cdefs.h

732–736

It's not, because the case below is not parsed when _POSIX_C_SOURCE is defined. I was talking about _POSIX_C_SOURCE and _C11_SOURCE both defined.

[imp]

After consulting with the OpenBSD folk and not hearing from Wollman,
I've reconsidered. I have become convinced that the OpenBSD and glibc
practice is the best way to resolve the ambiguity.

[imp]

sys/sys/cdefs.h

732–736

_C11_SOURCE is a private FreeBSD for the source tree only, and we don't need this there.

[olce]

In D29902#676065, @imp wrote:

After consulting with the OpenBSD folk and not hearing from Wollman,
I've reconsidered. I have become convinced that the OpenBSD and glibc
practice is the best way to resolve the ambiguity.

Glad to hear that! I've just added a typo fix and a suggestion to rephrase. Otherwise good to go for me.

sys/sys/cdefs.h

723

726