Page MenuHomeFreeBSD

Fix dtrace symbol resolution in the face of bitfields
ClosedPublic

Authored by jtl on Sat, Nov 14, 12:32 AM.

Details

Summary

Apologies for the long description...

After rS366908, we started seeing errors like this on startup:

$ dtrace -n 'fbt::in6_cksum_pseudo:entry { @[args[0]->ip6_ctlun.ip6_un1.ip6_un1_nxt] = count(); }'
dtrace: invalid probe specifier fbt::in6_cksum_pseudo:entry { @[args[0]->ip6_ctlun.ip6_un1.ip6_un1_nxt] = count(); }: "/usr/lib/dtrace/ipfw.d", line 121: failed to copy type of 'ip6p': Conflicting type is already defined 

A few things may have led to this being somewhat obscured:

First, on my system, ipfw.d was the first script parsed from /usr/lib/dtrace. I'm not sure whether this would have been different if it had been loaded in a different order.

Second, not everyone will see this. Because ipfw.d depends on the ipfw provider, you will only see this if the ipfw module is loaded or ipfw is compiled into the kernel.

The problem is with the way types are being resolved when importing types from the kernel.

Dtrace prepopulates common C types into a "C" CTF container. It then prepopulates some types into a "D" CTF container. It then opens the kernel and imports types from the kernel into its own CTF container. It then opens each kernel module and imports types from the modules into their own CTF container. When doing symbol resolution for types which appear in .d files, it searches each CTF container in order (C -> D -> kernel -> module 1 -> module 2 -> module n) and chooses the first matching type.

If the type used in the .d file is found in any CTF container other than the "current" CTF container (which, in the case of ipfw.d, appears to be the "D" CTF container), the code calls the ctf_add_type() function to copy the type from the container in which it was found to the "current" CTF container. (The call to ctf_add_type() moves execution from libdtrace to libctf.)

As part of this copy, ctf_add_type() also copies all types necessary to fully resolve the type. In the case of structures, it will copy all the types to which the structure refers. However, in this case, it does not look at all the CTF containers to do type resolution. Instead, it will only look at the CTF container which is the source of the copy. Prior to inserting the types into the destination CTF container, ctf_add_type() checks to see if the new type has the CTF_ADD_ROOT flag set. If so, it will ensure that the code is not about to add a conflicting type with the same name as an existing type. As relevant here, intrinsics have the CTF_ADD_ROOT flag set, while bitfields do not.

Finally, it is relevant to this discussion to understand that libctf builds its name hash table with the assumption that intrinsic types will appear before bitfields. (See ctf_open.c:363-381.)

So, let's look at the ipfw_match_info structure in ipfw.d:

typedef struct ipfw_match_info {
    uint32_t        flags;
    struct mbuf     *m;
    void            *mem;
    struct inpcb    *inp;
    struct ifnet    *ifp;
    struct ip       *ipp;
    struct ip6_hdr  *ip6p;
...

When dtrace starts processing this structure, it tries to resolve the uint32_t type. It does not find it in the "C" CTF container, but it does find it in the "D" CTF container. (It is one of the pre-populated instrinsics.) So, no action is necessary.

Dtrace next moves to resolving "struct mbuf". It does not find that type in the "C" or "D" CTF containers, but does find it in the kernel CTF container. It then calls ctf_add_type() to copy the appropriate types from the kernel CTF container.

When ctf_add_type() copies the "struct mbuf" type, it also recursively copies all of the types of the structures members. The first use of an unsigned integer is this:

uint32_t         m_type:8,      /* type of data in this mbuf */

ctf_add_type() resolves this within the kernel CTF container to "unsigned int". Probably because this is a bitfield, the CTF_ADD_ROOT flag is not set on the resulting type.

As ctf_add_type() continues to process the structure, it adds a 24-bit uint32_t, and a 32-bit uint32_t. When this is done, the dtrace library calls ctf_update() and ctf_bufopen() on its "D" CTF container. ctf_bufopen() calls init_types() which recreates the hash table for that CTF container. Because of the assumption that intrinsics will appear first in the debugging information, the 8-bit "unsigned int" is chosen as the formal version of "unsigned int".

The dtrace library can resolve "void *" on its own. And, it imported the definitions of "struct inpcb", "struct ifnet", and "struct ip" when it was recursively adding the definition of the "struct mbuf" members. So, it next needs to process struct ip6_hdr, which starts like this:

struct ip6_hdr {
    union {
        struct ip6_hdrctl {
            u_int32_t ip6_un1_flow; /* 20 bits of flow-ID */
...

Because it does not already have the definition of "struct ip6_hdr" in its "D" CTF container, it calls ctf_add_type() to import it from the kernel CTF container. Probably because u_int32_t resolves to a well-known intrinsic ("unsigned int"), it does have the CTF_ADD_ROOT flag set. Therefore, before adding this type to the "D" CTF container, ctf_add_type() checks for conflicting types in the "D" CTF container. When it does a hash lookup for "unsigned int", it finds the 8-bit "unsigned int". It recognizes this as a conflict and returns an error, which eventually produces the somewhat obscure error returned to the user.

After pondering this for a while, it seemed like the best way to resolve this was to ensure that we always copy an intrinsic before we copy the bitfield type. This makes the rest of the assumptions in libctf remain true. And, it seems like it has a fairly low risk of unintended consequences.

Test Plan

dtrace is now working correctly.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

jtl requested review of this revision.Sat, Nov 14, 12:32 AM

Recently I faced with this problem on some machines:

# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: invalid probe specifier fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }: in action list: m_pkthdr is not a member of struct mbuf

And it seems it is exactly related.

Bitfields seem to introduce quite a few corner cases in CTF code. I wonder if it would have been better to encode the bitfield encoding in struct/union member definitions rather than adding new types.

cddl/contrib/opensolaris/common/ctf/ctf_create.c
1393 ↗(On Diff #79532)

Does this somehow ensure that we don't end up with duplicate definitions of the same intrinsic in the destination container?

cddl/contrib/opensolaris/common/ctf/ctf_create.c
1393 ↗(On Diff #79532)

It should. It looks like intrinsincs have the CTF_ADD_ROOT flag set. If so, when we try to add the intrinsic, it should look for duplicate types in both the destination's hash table (lines 1305-1308) and (if not found in the hash table) the list of types added since we last created the hash table (lines 1336-1354).

The one case which could result in duplicates is if we try to add an instrinsic which does not have the CTF_ADD_ROOT flag. In that case, the code would skip the lookup in the destination's hash table, but would still check for duplicates in the recently-added types. I'm not sure whether it is worth checking for this case, but am open to suggestions for improvement.

In D27213#608081, @ae wrote:

Recently I faced with this problem on some machines:

# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: invalid probe specifier fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }: in action list: m_pkthdr is not a member of struct mbuf

And it seems it is exactly related.

It does appear that problem is likely related to type resolution. However, for what its worth, this patch does not solve that problem on my test machine. So, that problem may have a different cause.

In D27213#608180, @jtl wrote:
In D27213#608081, @ae wrote:

Recently I faced with this problem on some machines:

# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: invalid probe specifier fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }: in action list: m_pkthdr is not a member of struct mbuf

And it seems it is exactly related.

It does appear that problem is likely related to type resolution. However, for what its worth, this patch does not solve that problem on my test machine. So, that problem may have a different cause.

I can't reproduce this problem at all on head. The script appears to work properly. I remember that the use of anonymous unions in struct mbuf caused some problems, at least one of which was fixed by r305055, but that was a long time ago.

cddl/contrib/opensolaris/common/ctf/ctf_create.c
1393 ↗(On Diff #79532)

I think I see. It doesn't seem worth trying to address that corner case in the absence of a test suite. Duplicate definitions of an intrinsic are probably not going to create any visible problems.

This revision is now accepted and ready to land.Mon, Nov 16, 4:59 PM
In D27213#608180, @jtl wrote:
In D27213#608081, @ae wrote:

Recently I faced with this problem on some machines:

# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: invalid probe specifier fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }: in action list: m_pkthdr is not a member of struct mbuf

And it seems it is exactly related.

It does appear that problem is likely related to type resolution. However, for what its worth, this patch does not solve that problem on my test machine. So, that problem may have a different cause.

I can't reproduce this problem at all on head. The script appears to work properly. I remember that the use of anonymous unions in struct mbuf caused some problems, at least one of which was fixed by r305055, but that was a long time ago.

I can reproduce this problem locally. Some others can't reproduce the problem I reported in the main description. I'm not sure why, but there seems to be some non-determinism in the way symbols are loaded/resolved?

In any case, since I can reproduce the problem @ae reported, I'll try to do some debugging on it.

It seems I found how to reproduce it on test system:

  1. Load systemt without any unneeded modules
  2. kldload dtraceall
  3. Run
# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: description 'fbt::ip_input:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  2  49220                   ip_input:entry ix0
  2  49220                   ip_input:entry ix0
  6  49220                   ip_input:entry ix0
^C
# kldunload dtraceall
# kldload ipfw
# kldload dtraceall
# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: invalid probe specifier fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }: in action list: m_pkthdr is not a member of struct mbuf
In D27213#608269, @ae wrote:

It seems I found how to reproduce it on test system:

  1. Load systemt without any unneeded modules
  2. kldload dtraceall
  3. Run
# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: description 'fbt::ip_input:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  2  49220                   ip_input:entry ix0
  2  49220                   ip_input:entry ix0
  6  49220                   ip_input:entry ix0
^C
# kldunload dtraceall
# kldload ipfw
# kldload dtraceall
# dtrace -n 'fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }'
dtrace: invalid probe specifier fbt::ip_input:entry { printf("%s", stringof(args[0]->m_pkthdr.rcvif->if_xname)); }: in action list: m_pkthdr is not a member of struct mbuf

This is probably because the ipfw.d file is skipped unless the ipfw module is loaded (or the functionality is compiled into the kernel).

In D27213#608228, @jtl wrote:

I can't reproduce this problem at all on head. The script appears to work properly. I remember that the use of anonymous unions in struct mbuf caused some problems, at least one of which was fixed by r305055, but that was a long time ago.

I can reproduce this problem locally. Some others can't reproduce the problem I reported in the main description. I'm not sure why, but there seems to be some non-determinism in the way symbols are loaded/resolved?

FWIW, I did verify this today. One of my colleagues gave me access to a machine which was not seeing the problem I initially described in this review. On that machine, psinfo.d was being loaded first, then ipfw.d. On the machines which I had observed to be impacted, ipfw.d was being loaded first. Because ipfw.d is not involved in an explicit dependency chain between libraries, the ordering seems to be based on the order in which readdir() is returning files (which seems to match the order that ls -f lists things. The newest file seems to be read first, in absence of explicit ordering. When I used gdb to change the order the libraries were loaded, my colleague's machine saw the same error I had observed.

In any case, since I can reproduce the problem @ae reported, I'll try to do some debugging on it.

The problem @ae reported seems to be due to the CTF code adding the anonymous unions with a name of "" instead of a 0 name index. This confuses _ctf_member_info(), which expects anonymous unions/structures to have a 0 name index. I have not yet debugged why this is occurring. However, if I block ipfw.d from loading, the problem goes away.