In D57381#1348820, @markj wrote:

Hmm, I missed that the LLVM change has not yet landed. @zishun.yi.dev_gmail.com will you add some tests as requested in the PR? If that's done, I wonder if @dim or @aokblast can help land it?

update the atomic part to revert the previous and fold in markj@'s commit

In D57381#1346389, @markj wrote:

In D57381#1345924, @zishun.yi.dev_gmail.com wrote:

In D57381#1344342, @markj wrote:

In D57381#1330788, @zishun.yi.dev_gmail.com wrote:

In D57381#1330138, @markj wrote:

@zishun.yi.dev_gmail.com could you please rebase this patch so that it applies to the latest main? I don't really like the diff to riscv/include/atomic.h, and would like to explore different approaches. (Really I'd like to get rid of all of these interceptors entirely, but the best way to do that is probably to reimplement atomic.h using C11 atomics...)

Yeah, I agree the current code is ugly, and your approach sounds great.

For now I've moved most of the ugly code into a separate revision(D58037) so it doesn't block the riscv work. That part could well be refactored entirely in the future , so keeping it separate from the KASAN changes seems cleaner.

I finally got back to this, sorry for the delay. I wrote some smaller patches that are sufficient to deal with the atomic issues: https://github.com/markjdb/freebsd/commits/main-riscv-kasan/

The change to subr_asan.c is a bit ugly, but I think it's acceptable, and if we can reimplement atomic(9) on top of C11 atomics most of that code will go away anyway. The atomic_subword.h change seems like a nice cleanup on its own, I submitted for review already.

What do you think?

Yeah, I think the change to subr_asan.c is acceptable. And the other commits seem clean.

I submitted D58680. Once that lands in some form, you can fold the other commits from that branch into your patch, and I think it'll be good to go.

In D57381#1344342, @markj wrote:

In D57381#1330788, @zishun.yi.dev_gmail.com wrote:

In D57381#1330138, @markj wrote:

@zishun.yi.dev_gmail.com could you please rebase this patch so that it applies to the latest main? I don't really like the diff to riscv/include/atomic.h, and would like to explore different approaches. (Really I'd like to get rid of all of these interceptors entirely, but the best way to do that is probably to reimplement atomic.h using C11 atomics...)

Yeah, I agree the current code is ugly, and your approach sounds great.

For now I've moved most of the ugly code into a separate revision(D58037) so it doesn't block the riscv work. That part could well be refactored entirely in the future , so keeping it separate from the KASAN changes seems cleaner.

I finally got back to this, sorry for the delay. I wrote some smaller patches that are sufficient to deal with the atomic issues: https://github.com/markjdb/freebsd/commits/main-riscv-kasan/

The change to subr_asan.c is a bit ugly, but I think it's acceptable, and if we can reimplement atomic(9) on top of C11 atomics most of that code will go away anyway. The atomic_subword.h change seems like a nice cleanup on its own, I submitted for review already.

What do you think?

address aokblast@'s comment

address aokblast@'s comment

address aokblast@'s comment

Update the "recv buffer full" test case to expect NLMSG_OVERRUN.

Implement ENOBUFS and NETLINK_NO_ENOBUFS in the Linux ABI, and send
NLMSG_OVERRUN in the FreeBSD ABI.

In D57381#1330138, @markj wrote:

@zishun.yi.dev_gmail.com could you please rebase this patch so that it applies to the latest main? I don't really like the diff to riscv/include/atomic.h, and would like to explore different approaches. (Really I'd like to get rid of all of these interceptors entirely, but the best way to do that is probably to reimplement atomic.h using C11 atomics...)

• Address mhorne@'s comments, thanks for testing, and it's great to know that KASAN already caught a real bug!
• Rebase to origin/main
• Move the MI interceptor code to a separate parent revision.

clarify cmsg

move NETLINK_GET_STRICT_CHK and NETLINK_MSG_INFO to another revision.

remove NETLINK_SND_SYNC.

In D57807#1326559, @glebius wrote:

What are the actual applications that depend on that?

In D57807#1326440, @glebius wrote:

Can you please elaborate what software expects this and what problems the patch tries to address?

Linux actually has a specific netlink socket flag, NETLINK_NO_ENOBUFS, which suggests that a lot of Linux software depends on the default ENOBUFS behavior.
As obiwac@ mentioned, our initial thought was to align this behavior with Linux, especially in preparation for the synchronous Netlink patch I plan to introduce.

Add a test to check the recv buffer, as suggested by obiwac@.

Include this in the revision where I add this flag.

change the commit order.

• change the commit order.
• include the manual change.

Address some of obiwac@'s comments.

rebase

Based on the code flow I mentioned above, I found that the DMA_TO_DEVICE and
DMA_FROM_DEVICE directions in dma_sync_single_for_device might be reversed.

I re-read the manual today:

BUS_DMASYNC_PREREAD    Perform any synchronization required prior
                       to an update of host memory by the device.

In D55497#1320154, @bz wrote:

Isn't this closer to what I once had and hselasky said I should change?

IMO, hselasky said we cannot pass multiple flags to the non-bidirectional DMA cases. I think that is correct.
But I guess his thought was that some USB controllers might access the DMA buffer after the dma_sync_single_for_cpu, so he added the PREREAD flag. However, under Linux semantics, the device should not access the DMA buffer after the dma_sync_single_for_cpu.

do sync conversion in setsockopt handler, tearing down the taskqueue and returning an error if some messages are pending procesding.

In D57519#1322702, @melifaro wrote:

Do you see a use case for setting sync mode mid-session? If not, i’d suggest doing conversion in setsockopt handler, tearing down the async thread and returning an error if some messages are pending procesding.

I think you're right, it's weird to set sync mid-session. I will change it later

• Move some logic in sync path to a separate function.
• Move the lock acquisition inside the branch

In D57519#1322555, @melifaro wrote:

Thank you for working on this!
Added a bit of comments on the MR itself, also a bit more generic questions:

1. What is the proposed processing mechanism in sync case? Can we clearly state the logic in the description?

For example, if the caller sent multiple nettlink messages on a single send / write call - do we assume the caller should have allocated enough RX buffer for all of the replies? Or put it differently - what should happen when we don't have enough space in the receive buffer to (a) get all of the reply, (b) to send netlink header with error and (c) when we, say, processed and replied to 2 netlink messages out of 5?

Linux is sync. Its behavior is this send() will succeed and process all requests (5, for example). But it drops the replies that exceed the receive buffer(3 replies). And will return ENOBUFS in the next recv() to indicate we have dropped some replies.
In other words, the success of send() does not depend on whether the receive buffer is full. So I align with its behavior, the corresponding part is the nl_dropped_bytes in nl_soreceive().

• Can pass the test
• Add SB_WAIT
• Remove timeout
• Refactor nl_process_nbuf_sync. The difference from nl_process_nbuf is that the sync version removes nlmsg_ignore_limit and returns the actual error.
• Make recv return ENOBUF when it sees nl_dropped_messages.

Add the test mentioned in my previous comment, but without changing the kernel
code yet, so it currently fails as expected.

After investigating the behavior of netlink send when the buffer is full, I found:
FreeBSD async send path behavior:

• Send buffer full: Deadlocks (this indicates the recv buffer is also full).
• Recv buffer full: Send doesn't depend on it, so it will never block. It just keeps filling the send buffer.

Linux send path behavior:

• Send buffer will never fill up because of sync.
• Recv buffer full: Send doesn't depend on it, so it will never block, but it will fail on the next recv.

Therefore, I think the FreeBSD sync send path behavior should be:

• Send buffer full: Deadlock.
• Recv buffer full: Same as linux

Hi, I think this is a good change overall. I actually tried to solve the same problem before in D55328, but I ended up abandoning it because my use case no longer required it.

fix potential deadlock

Fixed hart boot hang on riscv smp caused by KASAN, by initializing the tp register earlier in the assembly stage.

In D57381#1316039, @mhorne wrote:

Hi, glad to see this review.

I will take a detailed look next week, and I hope to test it out as well.

Address review feedback.

As suggested by @jrtc27, KASAN shouldn't intercept unsupported riscv atomic functions.

Prevent KASAN from intercepting atomic functions that are unsupported by the
architecture. The specific changes can be summarized as follows:

• Use ARCH_SUPPORT_ATOMIC_*_WIDTH x-macros to control the generation of interceptor functions in subr_asan.c.
• Use #undef in machine/atomic.h to undefine the macros that are forcibly defined by sys/atomic_san.h but lack actual MD implementations.

In D57379#1314784, @jrtc27 wrote:

In D57379#1314783, @zishun.yi.dev_gmail.com wrote:

In D57379#1314781, @jrtc27 wrote:

Having said that, what is the use case you have for these? Generally speaking you shouldn't be using sub-word atomics unless absolutely needed, since performance will be worse.

Ah, I didn't actually consider the performance before.
I added these to simplify the KASAN porting. KASAN needs to intercept these defines, but riscv doesn't have these definitions, so adding them ensures the KASAN common code won't be polluted by arch-specific code.

KASAN should just not intercept things that don't exist

Fix segment order

In D57379#1314781, @jrtc27 wrote:

Having said that, what is the use case you have for these? Generally speaking you shouldn't be using sub-word atomics unless absolutely needed, since performance will be worse.

In D57379#1314780, @jrtc27 wrote:

Surely these belong in atomic_subword.h or similar? They're not at all specific to RISC-V, and other architectures lack various of these.

In D57380#1314674, @minsoochoo0122_proton.me wrote:

I guess it's okay to merge this into D57381? See f115c0612131d8f939f6f357f57bdd85bd6a59de for amd64.

In D57378#1314670, @minsoochoo0122_proton.me wrote:

I would like to know what this is for. Code looks okay, but I wonder why we need these aliases. Could you elaborate more in the commit message and revision summary?

To support the FAM cmd for udmabuf, I first decided to improve the linuxkpi ioctl through this patch, but due to some hacks in this patch and the suggestion from my GSoC mentor Aymeric, I decided to enhance the native FreeBSD ioctl call stack - to let the native FreeBSD ioctl support FAM.

In D57129#1309087, @markj wrote:

In D57129#1309086, @markj wrote:

Sigh. I realized just after pushing this that the address range is wrong, not the comment:

(kgdb) p/x kva_layout
$1 = {kva_min = 0xffff800000000000, kva_max = 0xfffffffffffff000, dmap_low = 0xfffff80000000000, dmap_high = 0xfffffc0000000000, 
  lm_low = 0xffff808000000000, lm_high = 0xfffff78000000000, km_low = 0xfffffe0000000000, km_high = 0xfffffffffffff000, rec_pt = 0xffff800000000000, 
  kasan_shadow_low = 0xfffff78000000000, kasan_shadow_high = 0xfffff80000000000, kmsan_shadow_low = 0xfffffc0000000000, 
  kmsan_shadow_high = 0xfffffe0000000000, kmsan_origin_low = 0xfffff60000000000, kmsan_origin_high = 0xfffff80000000000}

512GB makes sense, as that's one PML4 entry.

... but only 256GB is used, since the kernel map is 2TB and the shadow map scale is 1:8, as you noted. So this is ok after all, ignore me. :)