Page MenuHomeFreeBSD

osd: Do not resize other objects' slot arrays on deregistration
Needs ReviewPublic

Authored by rcm on Fri, Oct 16, 12:28 PM.
Tags
None
Referenced Files
F176543300: D60772.id190031.diff
Mon, Oct 19, 5:12 PM
F176524392: D60772.id.diff
Mon, Oct 19, 1:23 PM
Unknown Object (File)
Sun, Oct 18, 8:56 AM
Unknown Object (File)
Sun, Oct 18, 1:42 AM
Unknown Object (File)
Sat, Oct 17, 12:45 AM
Unknown Object (File)
Sat, Oct 17, 12:44 AM
Unknown Object (File)
Sat, Oct 17, 12:36 AM
Unknown Object (File)
Sat, Oct 17, 12:35 AM
Subscribers

Details

Summary

osd_get_unlocked() reads an object's slot array without the OSD lock,
which is safe only while nothing else resizes or frees the array. The
callers that use it, such as mac_do reading its slot of curthread, rely
on being the only writer of that object's slots. osd_deregister()
breaks that: it deletes the departing slot from every object through
do_osd_del(), which shrinks an object's array with realloc() when the
slot was its last, and frees it when no other slot is set. A thread
reading another slot of the same array at that moment reads freed
memory.

Have osd_deregister() only run the destructor and clear the slot,
leaving each array to the operations on its own object, as it already
does for the destructor and method arrays since a76629cb0381.
osd_exit() now frees an array that holds only deregistered slots, which
it used to leave to osd_deregister(). osd_get_unlocked() no longer
asserts on osd_destructors, which osd_register() may reallocate;
osd_get() checks it under the lock instead.

Document osd_get_unlocked() and what its callers must guarantee.

Fixes: 56b274213024 ("Add function to OSD to get values without taking the lock.")
Sponsored by: Rubicon Communications, LLC ("Netgate")

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

rcm requested review of this revision.Fri, Oct 16, 12:28 PM

It seems to me the current model is buggy at a high-level and we should fix it completely, which means going further than this change and actually getting rid of osd_get_unlocked() entirely. Using the osd*get_unlocked() variants in mac_do(4) appears to have been a mistake.

Of course, you must arrange for osd_get_unlocked() not to be called concurrently with operations that could modify the underlying array, like osd_set() and osd_del(), else you risk an access-after-free. It's possible not to change the slot array on osd_deregister() as you're doing here, and you could even do that for osd_del(), but you'll never be able to do it for osd_set() (unless we preallocate and limit the OSD slots from the start, which doesn't look acceptable). When using osd_get_unlocked(), you're at the mercy of another module registering some slot and starting to set it. This makes it impossible to safely compose modules using the same struct osd, basically defeating the whole point of OSD.

Then we could start looking at optimizing the locking if it proves to be a problem in practice.

The main point for old_get_unlocked() was more of a peek at the value to see if something is there without needing to take the lock.
For example, at work we had a case where code was setting the value in the slot only when it needed it (which was a pointer to a structure which itself contains a sxlock.) The code would check the slot to see if there was actually something assigned and only use it when set. The code also only ever sets the value once, therefore if it's non-NULL, then that value should be stable for that object. Also, where it was used, the object must be valid through the whole code path, so there was no risk of the thing it was pointing to going away. However, the overhead of taking the lock (also given the type of lock that OSD uses) was causing unnecessary complications.

(snip) therefore if it's non-NULL, then that value should be stable for that object (snip)

The value can be stable or not, that doesn't change the problem, which is that the underlying storage for that value is not and you may access some freed storage.

True, the original code would be a race potentially if there was any deregister that would happen.

I was just giving a background of the original intent of the osd_get_unlocked().

I was just giving a background of the original intent of the osd_get_unlocked().

I hear you. We could optimize access if needed. E.g., there's currently a single object lock. Also, we could think about splitting osd_slots[] into one array protected by read-mostly locks, and the other by the regular read-write ones. If more is needed, then it is arguable whether the concerned slot should just be a plain field in the encompassing structure rather than an OSD (but still, there would be ways to optimize the access further in OSD, at a slight expense for "normal" accesses). But if there's no pressing need for that, let's just remove the buggy osd_get_unlocked(). I don't think such a development would belong to this revision anyway.

@rcm : So, I'd just replace the changes here with suppressing osd_get_unlocked(). Are you OK with that?