Apologies for the long description...
After rS366908, we started seeing errors like this on startup:
$ dtrace -n 'fbt::in6_cksum_pseudo:entry { @[args[0]->ip6_ctlun.ip6_un1.ip6_un1_nxt] = count(); }' dtrace: invalid probe specifier fbt::in6_cksum_pseudo:entry { @[args[0]->ip6_ctlun.ip6_un1.ip6_un1_nxt] = count(); }: "/usr/lib/dtrace/ipfw.d", line 121: failed to copy type of 'ip6p': Conflicting type is already defined
A few things may have led to this being somewhat obscured:
First, on my system, ipfw.d was the first script parsed from /usr/lib/dtrace. I'm not sure whether this would have been different if it had been loaded in a different order.
Second, not everyone will see this. Because ipfw.d depends on the ipfw provider, you will only see this if the ipfw module is loaded or ipfw is compiled into the kernel.
The problem is with the way types are being resolved when importing types from the kernel.
Dtrace prepopulates common C types into a "C" CTF container. It then prepopulates some types into a "D" CTF container. It then opens the kernel and imports types from the kernel into its own CTF container. It then opens each kernel module and imports types from the modules into their own CTF container. When doing symbol resolution for types which appear in .d files, it searches each CTF container in order (C -> D -> kernel -> module 1 -> module 2 -> module n) and chooses the first matching type.
If the type used in the .d file is found in any CTF container other than the "current" CTF container (which, in the case of ipfw.d, appears to be the "D" CTF container), the code calls the ctf_add_type() function to copy the type from the container in which it was found to the "current" CTF container. (The call to ctf_add_type() moves execution from libdtrace to libctf.)
As part of this copy, ctf_add_type() also copies all types necessary to fully resolve the type. In the case of structures, it will copy all the types to which the structure refers. However, in this case, it does not look at all the CTF containers to do type resolution. Instead, it will only look at the CTF container which is the source of the copy. Prior to inserting the types into the destination CTF container, ctf_add_type() checks to see if the new type has the CTF_ADD_ROOT flag set. If so, it will ensure that the code is not about to add a conflicting type with the same name as an existing type. As relevant here, intrinsics have the CTF_ADD_ROOT flag set, while bitfields do not.
Finally, it is relevant to this discussion to understand that libctf builds its name hash table with the assumption that intrinsic types will appear before bitfields. (See ctf_open.c:363-381.)
So, let's look at the ipfw_match_info structure in ipfw.d:
typedef struct ipfw_match_info {
uint32_t flags;
struct mbuf *m;
void *mem;
struct inpcb *inp;
struct ifnet *ifp;
struct ip *ipp;
struct ip6_hdr *ip6p;
...When dtrace starts processing this structure, it tries to resolve the uint32_t type. It does not find it in the "C" CTF container, but it does find it in the "D" CTF container. (It is one of the pre-populated instrinsics.) So, no action is necessary.
Dtrace next moves to resolving "struct mbuf". It does not find that type in the "C" or "D" CTF containers, but does find it in the kernel CTF container. It then calls ctf_add_type() to copy the appropriate types from the kernel CTF container.
When ctf_add_type() copies the "struct mbuf" type, it also recursively copies all of the types of the structures members. The first use of an unsigned integer is this:
uint32_t m_type:8, /* type of data in this mbuf */
ctf_add_type() resolves this within the kernel CTF container to "unsigned int". Probably because this is a bitfield, the CTF_ADD_ROOT flag is not set on the resulting type.
As ctf_add_type() continues to process the structure, it adds a 24-bit uint32_t, and a 32-bit uint32_t. When this is done, the dtrace library calls ctf_update() and ctf_bufopen() on its "D" CTF container. ctf_bufopen() calls init_types() which recreates the hash table for that CTF container. Because of the assumption that intrinsics will appear first in the debugging information, the 8-bit "unsigned int" is chosen as the formal version of "unsigned int".
The dtrace library can resolve "void *" on its own. And, it imported the definitions of "struct inpcb", "struct ifnet", and "struct ip" when it was recursively adding the definition of the "struct mbuf" members. So, it next needs to process struct ip6_hdr, which starts like this:
struct ip6_hdr {
union {
struct ip6_hdrctl {
u_int32_t ip6_un1_flow; /* 20 bits of flow-ID */
...Because it does not already have the definition of "struct ip6_hdr" in its "D" CTF container, it calls ctf_add_type() to import it from the kernel CTF container. Probably because u_int32_t resolves to a well-known intrinsic ("unsigned int"), it does have the CTF_ADD_ROOT flag set. Therefore, before adding this type to the "D" CTF container, ctf_add_type() checks for conflicting types in the "D" CTF container. When it does a hash lookup for "unsigned int", it finds the 8-bit "unsigned int". It recognizes this as a conflict and returns an error, which eventually produces the somewhat obscure error returned to the user.
After pondering this for a while, it seemed like the best way to resolve this was to ensure that we always copy an intrinsic before we copy the bitfield type. This makes the rest of the assumptions in libctf remain true. And, it seems like it has a fairly low risk of unintended consequences.