In D52166#1195280, @aokblast wrote:

Hello, could you please set the xhci_debug=1 and usb_passthru_debug=1 in bhyve and send me the report? The report maybe very long I think.

Quick question. How can I go about setting those options and generating the report using vm-bhyve? I already patched it using the diff from your github PR.

In D52166#1195391, @kenrap_kennethraplee.com wrote:

In D52166#1195280, @aokblast wrote:

Hello, could you please set the xhci_debug=1 and usb_passthru_debug=1 in bhyve and send me the report? The report maybe very long I think.

Quick question. How can I go about setting those options and generating the report using vm-bhyve? I already patched it using the diff from your github PR.

Hello, sorry it is from the code directly since it works like the normal DPRINTF. xhci_debug is in pci_xhci.c. usb_passthru_debug is in usb_passthru.c

In D52166#1195408, @aokblast wrote:

In D52166#1195391, @kenrap_kennethraplee.com wrote:

In D52166#1195280, @aokblast wrote:

Hello, could you please set the xhci_debug=1 and usb_passthru_debug=1 in bhyve and send me the report? The report maybe very long I think.

Quick question. How can I go about setting those options and generating the report using vm-bhyve? I already patched it using the diff from your github PR.

Hello, sorry it is from the code directly since it works like the normal DPRINTF. xhci_debug is in pci_xhci.c. usb_passthru_debug is in usb_passthru.c

Gotcha, thanks for clarifying: Here is the report (3911 lines): https://gist.github.com/kenrap/ecfc003562bd8717900a46de4903d7bf

Thanks @obiwac!

In D52166#1195426, @kenrap_kennethraplee.com wrote:

In D52166#1195408, @aokblast wrote:

In D52166#1195391, @kenrap_kennethraplee.com wrote:

In D52166#1195280, @aokblast wrote:

Hello, could you please set the xhci_debug=1 and usb_passthru_debug=1 in bhyve and send me the report? The report maybe very long I think.

Quick question. How can I go about setting those options and generating the report using vm-bhyve? I already patched it using the diff from your github PR.

Hello, sorry it is from the code directly since it works like the normal DPRINTF. xhci_debug is in pci_xhci.c. usb_passthru_debug is in usb_passthru.c

Gotcha, thanks for clarifying: Here is the report (3911 lines): https://gist.github.com/kenrap/ecfc003562bd8717900a46de4903d7bf

Could you please check if it works now?

In D52166#1195497, @aokblast wrote:

In D52166#1195426, @kenrap_kennethraplee.com wrote:

In D52166#1195408, @aokblast wrote:

In D52166#1195391, @kenrap_kennethraplee.com wrote:

In D52166#1195280, @aokblast wrote:

Hello, could you please set the xhci_debug=1 and usb_passthru_debug=1 in bhyve and send me the report? The report maybe very long I think.

Quick question. How can I go about setting those options and generating the report using vm-bhyve? I already patched it using the diff from your github PR.

Hello, sorry it is from the code directly since it works like the normal DPRINTF. xhci_debug is in pci_xhci.c. usb_passthru_debug is in usb_passthru.c

Gotcha, thanks for clarifying: Here is the report (3911 lines): https://gist.github.com/kenrap/ecfc003562bd8717900a46de4903d7bf

Could you please check if it works now?

No change in behavior. Here is a new report: https://gist.github.com/kenrap/a39b327616560bfd5edb816960953d4d

Sorry, I know that it is a little bit annoying. But since I don't have such device. Could you please try again?

In D52166#1196056, @aokblast wrote:

Sorry, I know that it is a little bit annoying. But since I don't have such device. Could you please try again?

No worries, I want to make sure this gets done right. I'll be your beta tester for this. 👍

I'm rebuilding my src tree now.

No difference. The report: https://gist.github.com/kenrap/9a1c406d097aa12c7a7b7ebe106cb681

In D52166#1196057, @kenrap_kennethraplee.com wrote:

In D52166#1196056, @aokblast wrote:

Sorry, I know that it is a little bit annoying. But since I don't have such device. Could you please try again?

No worries, I want to make sure this gets done right. I'll be your beta tester for this. 👍

I'm rebuilding my src tree now.

Thanks. I really need to get a device with stream in their descriptor. Could you please try again.

In D52166#1196089, @aokblast wrote:

In D52166#1196057, @kenrap_kennethraplee.com wrote:

In D52166#1196056, @aokblast wrote:

Sorry, I know that it is a little bit annoying. But since I don't have such device. Could you please try again?

No worries, I want to make sure this gets done right. I'll be your beta tester for this. 👍

I'm rebuilding my src tree now.

Thanks. I really need to get a device with stream in their descriptor. Could you please try again.

So, there is good news and bad news.

Good news is, the boot time went down significantly to 6 seconds.

The bad news, the device is still not detected by the guest OS.

Another report: https://gist.github.com/kenrap/2ad9fc1a216e0c6f7eb089c4f3b44c5f

In D52166#1196117, @kenrap_kennethraplee.com wrote:

In D52166#1196089, @aokblast wrote:

In D52166#1196057, @kenrap_kennethraplee.com wrote:

In D52166#1196056, @aokblast wrote:

Sorry, I know that it is a little bit annoying. But since I don't have such device. Could you please try again?

No worries, I want to make sure this gets done right. I'll be your beta tester for this. 👍

I'm rebuilding my src tree now.

Thanks. I really need to get a device with stream in their descriptor. Could you please try again.

So, there is good news and bad news.

Good news is, the boot time went down significantly to 6 seconds.

The bad news, the device is still not detected by the guest OS.

Another report: https://gist.github.com/kenrap/2ad9fc1a216e0c6f7eb089c4f3b44c5f

In D52166#1196117, @kenrap_kennethraplee.com wrote:

In D52166#1196089, @aokblast wrote:

In D52166#1196057, @kenrap_kennethraplee.com wrote:

In D52166#1196056, @aokblast wrote:

Sorry, I know that it is a little bit annoying. But since I don't have such device. Could you please try again?

No worries, I want to make sure this gets done right. I'll be your beta tester for this. 👍

I'm rebuilding my src tree now.

Thanks. I really need to get a device with stream in their descriptor. Could you please try again.

So, there is good news and bad news.

Good news is, the boot time went down significantly to 6 seconds.

The bad news, the device is still not detected by the guest OS.

Another report: https://gist.github.com/kenrap/2ad9fc1a216e0c6f7eb089c4f3b44c5f

Hello: Does your device work on FreeBSD host? I saw a transfer is cancelled so the problem maybe on the libusb side or kernel stack.

In D52166#1196153, @aokblast wrote:

Hello: Does your device work on FreeBSD host? I saw a transfer is cancelled so the problem maybe on the libusb side or kernel stack.

Sorry for the late reply. Yes, I'm able to mount the device on my FreeBSD host.

Hello, I think I need to at least get a usb 3 hub to test this feature since I haven’t had any super speed device. I will take a look at the candidates hub. However, I am occupied by other stuff recently. Maybe you won’t get any feedback very soon (Maybe a months afterI promise that I will be back ASAP since I would like to finish this feature. Sorry for your inconvenience on helping me test this!

In D52166#1196949, @aokblast wrote:

Hello, I think I need to at least get a usb 3 hub to test this feature since I haven’t had any super speed device. I will take a look at the candidates hub. However, I am occupied by other stuff recently. Maybe you won’t get any feedback very soon (Maybe a months afterI promise that I will be back ASAP since I would like to finish this feature. Sorry for your inconvenience on helping me test this!

Appreciate the heads up. I'll be on standby.

Hi, I've tested this with a licensing dongle and a smart card reader on an external USB HUB with Windows 11. Licensing worked, smart card was detected correctly. I am still wondering if param1/2 in the config are the best thing but they work, so be it. Many thanks to you and the FreeBSD Foundation! I'll keep trying and using it for other bits as well the next days on other machines. I just need to disable the debugging as my remote window kept scrolling (have the tablet on for vnc as well in this case) like I was on a 9600 connection downloading Netscape ;-)

It's gotten silent here. May I ask what the plan is? Is there a chance to get this work into 16?

In D52166#1232521, @bz wrote:

It's gotten silent here. May I ask what the plan is? Is there a chance to get this work into 16?

Hello. I received the notice of conscription for 4 month obligatory military service in my country in late Octorber which have started from November. I will continue to work on this after my service.

In D52166#1196957, @kenrap_kennethraplee.com wrote:

In D52166#1196949, @aokblast wrote:

Hello, I think I need to at least get a usb 3 hub to test this feature since I haven’t had any super speed device. I will take a look at the candidates hub. However, I am occupied by other stuff recently. Maybe you won’t get any feedback very soon (Maybe a months afterI promise that I will be back ASAP since I would like to finish this feature. Sorry for your inconvenience on helping me test this!

Appreciate the heads up. I'll be on standby.

Hello, I am back and I suspect that https://reviews.freebsd.org/D55289 fixes the problem.
I have rebased the patch to the HEAD.
Could you please check if it actually works?

In D52166#1273524, @aokblast wrote:

In D52166#1196957, @kenrap_kennethraplee.com wrote:

In D52166#1196949, @aokblast wrote:

Hello, I think I need to at least get a usb 3 hub to test this feature since I haven’t had any super speed device. I will take a look at the candidates hub. However, I am occupied by other stuff recently. Maybe you won’t get any feedback very soon (Maybe a months afterI promise that I will be back ASAP since I would like to finish this feature. Sorry for your inconvenience on helping me test this!

Appreciate the heads up. I'll be on standby.

Hello, I am back and I suspect that https://reviews.freebsd.org/D55289 fixes the problem.
I have rebased the patch to the HEAD.
Could you please check if it actually works?

Welcome back! 👋

And thanks for rebasing too. I upgraded my base for the updated patch and others as well.

Sadly the non-detection problem still persists...

I got question though. Is there a way to passthru with a specific ugen number of the same vendor and product IDs?

I'm using multiple of these adapters and they have the same IDs. Only the earliest ugen device gets passthru'd when I need a later one instead.

I've been out of the loop with this so I'm trying to recall what I've forgotten. 🙂

In D52166#1273749, @kenrap_kennethraplee.com wrote:

In D52166#1273524, @aokblast wrote:

In D52166#1196957, @kenrap_kennethraplee.com wrote:

In D52166#1196949, @aokblast wrote:

Hello, I think I need to at least get a usb 3 hub to test this feature since I haven’t had any super speed device. I will take a look at the candidates hub. However, I am occupied by other stuff recently. Maybe you won’t get any feedback very soon (Maybe a months afterI promise that I will be back ASAP since I would like to finish this feature. Sorry for your inconvenience on helping me test this!

Appreciate the heads up. I'll be on standby.

Hello, I am back and I suspect that https://reviews.freebsd.org/D55289 fixes the problem.
I have rebased the patch to the HEAD.
Could you please check if it actually works?

Welcome back! 👋

And thanks for rebasing too. I upgraded my base for the updated patch and others as well.

Sadly the non-detection problem still persists...

I got question though. Is there a way to passthru with a specific ugen number of the same vendor and product IDs?

I'm using multiple of these adapters and they have the same IDs. Only the earliest ugen device gets passthru'd when I need a later one instead.

I've been out of the loop with this so I'm trying to recall what I've forgotten. 🙂

Sure, it is possible by just specify ugen number. I can work on this.
And I need some time to take a look at what I was doing 4 months ago:)

Woohoo. Happy to see updates here.

I've rebased my main once in between; while my smart card reader still works on Win11 after passthru one of my licensing dongles no longer does. I'll grab an updated stack from here the next days and rebuild to see. I haven't had time to debug it anyway so can as well do it on the latest revision. Thanks for all this work!

In D52166#1273856, @bz wrote:

Woohoo. Happy to see updates here.

I've rebased my main once in between; while my smart card reader still works on Win11 after passthru one of my licensing dongles no longer does. I'll grab an updated stack from here the next days and rebuild to see. I haven't had time to debug it anyway so can as well do it on the latest revision. Thanks for all this work!

Hi, I found a very subtle fails in our xHCI. Please also apply this patch while you are testing.

https://reviews.freebsd.org/D57146

I did a very large regions of bug fixes.

The full patchset is in here https://github.com/aokblast/freebsd-src/tree/bhyve/impl_usbdev_passthru. It should fix many unworking devices.

in general, nice work!

please also add to manpage in this commit

Thanks! note that patch does not apply to main

Hey, any further traction on this? What's left before it can be landed?

Thanks for your review @bnovkov

In D52166#1330369, @adrian wrote:

Hey, any further traction on this? What's left before it can be landed?

There are three things blocking this patch.

1. We need to upstream the libusb capsicum option enum to upstream as we don't want to break compatibility with upstream
2. Pass a isoc xfer based device doesn't work (like UVC)
3. Pass a Hub doesn't work as we can only see the hub device but not its childrens.

I wouldn't over-index on trying to get everything perfect and upstreamed before landing stuff here.

• capsicum is a freebsd-ism here; upstreaming it is going to require a bunch of #ifdef entries which D51865 doesn't have.
• you can just say that ISO transfers aren't yet supported in passthrough.
• you can also just say that hub child device enumeration isn't yet supported in passthrough.

If it's useful now to a bunch of people I think we should focus on what it currently supports and get it landed.
I'd hate to see this languish here because it's not perfect. :-)

In D52166#1334510, @adrian wrote:

If it's useful now to a bunch of people I think we should focus on what it currently supports and get it landed.
I'd hate to see this languish here because it's not perfect. :-)

Is there any way to assist from the outside? Is anyone willing to prepare a proposal to the Foundation to wrap this up?

In D52166#1356953, @editor_callfortesting.org wrote:

In D52166#1334510, @adrian wrote:

If it's useful now to a bunch of people I think we should focus on what it currently supports and get it landed.
I'd hate to see this languish here because it's not perfect. :-)

Is there any way to assist from the outside? Is anyone willing to prepare a proposal to the Foundation to wrap this up?

Kyle E kindly updated us on a Production User Call about progress on the dependencies for this. Thank you!

We all look forward to the ability to pass through our eID cards, software license dongles, Ubikeys, webcams, HDMI adapters, and the like to the degree that it will support them.

sorry if i am testing the diff the wrong way and this is not the right place to report test results,
i am having some difficulties getting it working with usb 2.0 hi speed mass storage devices,i have tried three different USBs
first when i was trying to compile the diff,i got hit with

/usr/src/usr.sbin/bhyve/usb_passthru.c:493:15: error: use of undeclared identifier 'LIBUSB_OPTION_CAPSICUMIZE'
  493 |                 { .option = LIBUSB_OPTION_CAPSICUMIZE }
      |                             ^~~~~~~~~~~~~~~~~~~~~~~~~

then kevans@ informed me that option has been scrapped in favour of libusb doing sandboxing internally
then when i passthrough a usb 2.0 hi speed drive

i was getting umass0: <SanDisk Cruzer Blade, class 0/0, rev 2.00/1.00, addr 2> on usbus8
umass0: at uhub1, port 5, addr 2 (disconnected)
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 3 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 2 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 1 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Retrying command, 0 more tries remain
(probe0:umass-sim0:0:0:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
(probe0:umass-sim0:0:0:0): Error 5, Retries exhausted
umass0: detached
and several other issues such as enumeration and attachment failure during guest startup
performance degradations during bulk transfers
device getting stuck in a reset loop
long(30s) freezes and readahead recoveries during bulk transfer

the problems are in pci_xhci.c and usb_passthru.c mainly

Writes to an unpowered port were dropped entirely, so after a guest power cycle the change bits could never be cleared and the port wedged with the guest hub driver retrying forever. pci_xhci should honor write-1-to-clear of the change bits always, when PP is set, re-initialize the port and post a port status change event so the guest re-enumerates
something like

@@ -358,6 +358,7 @@ static void pci_xhci_dump_trb(struct xhci_trb *trb);
 static void pci_xhci_assert_interrupt(struct pci_xhci_softc *sc);
 static void pci_xhci_reset_slot(struct pci_xhci_softc *sc, int slot);
 static void pci_xhci_reset_port(struct pci_xhci_softc *sc, int portn, int warm);
+static void pci_xhci_init_port(struct pci_xhci_softc *sc, int portn);
 static void pci_xhci_update_ep_ring(struct pci_xhci_softc *sc,
     struct pci_xhci_dev_emu *dev, struct pci_xhci_dev_ep *devep,
     struct xhci_endp_ctx *ep_ctx, uint32_t streamid,
@@ -499,8 +500,24 @@ pci_xhci_portregs_write(struct pci_xhci_softc *sc, uint64_t offset,
 		}
 
 		if ((p->portsc & XHCI_PS_PP) == 0) {
-			WPRINTF(("pci_xhci: portregs_write to unpowered "
-			         "port %d", port));
+			p->portsc &= ~(value &
+			    (XHCI_PS_CSC |
+			     XHCI_PS_PEC |
+			     XHCI_PS_WRC |
+			     XHCI_PS_OCC |
+			     XHCI_PS_PRC |
+			     XHCI_PS_PLC |
+			     XHCI_PS_CEC |
+			     XHCI_PS_CAS));
+
+			if (value & XHCI_PS_PP) {
+				pci_xhci_init_port(sc, port);
+				p->portsc |= XHCI_PS_CSC;
+				pci_xhci_set_evtrb(&evtrb, port,
+				    XHCI_TRB_ERROR_SUCCESS,
+				    XHCI_TRB_EVENT_PORT_STS_CHANGE);
+				pci_xhci_insert_event(sc, &evtrb, 1);
+			}
 			break;
 		}

for example

and another thing is the data path misreported completions three ways. the stall branch set USB_STALL and then overwrote it with processed=1, so a stalled transfer reported as success and the guest never cleared the endpoint halt
Bulk/interrupt completion walked the live ring without stopping at the transfer's USB_LAST_DATA, completing a queued next TD without it reaching the device, the OUT walk stopped early at LINK TRBs or when act_len hit 0, splitting a TD. The errcode byte mixed usb_error_t and enum USB_ERRCODE, the OUT path stored 0/18 and pci_xhci decoded with USB_TO_XHCI_ERR(), so stalls surfaced as wrong codes and failed OUT transfers as success
it should set the stall errcode after the marking loop, bound every completion walk to the submitted TD (captured head/ndata/size, stop at USB_LAST_DATA), store enum USB_ERRCODE values, and decode them explicitly in pci_xhci_xfer_complete. this fixes the mass-storage enumeration reset loop and the throughput degradation.

another issue is that the host kernel re-attaches kernel drivers (umass) to a passed-through device at several points, first the USB explore thread re-probes asynchronously after set-configuration, ugen_set_interface re-attaches synchronously, and libusb_reset_device triggers a full asynchronous re-enumeration. A host driver and the guest then drive the device concurrently and corrupt each other's transfers
i think the right thing would be to remove the libusb_reset_device call from the guest port-reset path since the device keeps its address and configuration, so no host action is needed and add usb_passthru_detach_host_drivers(), called after configuration/interface changes and on port resets, polling until the device stays clean for two consecutive sweeps (the asynchronous re-probe can land between sweeps)

also in usb_passthru.c error codes are stored only via USB_DATA_SET_ERRCODE(data, ...) under if (data) so requests with no data stage (wLength == 0, e.g. a remote-wakeup denial stall, or a control transfer failing before a data block exists) drops the error and the guest sees success. it should report on errblk = data ?: first block instead

additionally libusb_set_configuration cancels all in-flight transfers even when the kernel no-ops the request because the configuration is unchanged
Guests re-issue SET_CONFIGURATION during enumeration and error recovery, killing bulk traffic each time so usb_passthru.c should compare against the active configuration and skip if unchanged

and the last issue i found is that some BOT devices wedge a bulk endpoint (NAK forever, until reset) when a CSW poll arrives long after the data phase and in some scenarios the guest's poll can be delayed past that
The wedged transfer pends until the guest's 30 s SCSI timeout forces a full device reset, causing periodic freezes and throughput collapse from the guest's readahead recovery
so bulk transfers on mass-storage interfaces should be given a 5s timeout and report TIMED_OUT as USB_ERR so the guest runs its fast clear-halt recovery instead(scoped to mass storage only)

please check out the inline comments as well,thank you!

For posterity, I think we agreed that I would rebase this forward since @aokblast is quite busy these days, and I'm going to see if there are functional pieces I can split this further into. (after Euro)