@jhb yes, of course.

Can you please add IOMMU to arm64 NOTES?

Moved to github.

I put all of the suggested changes in https://github.com/freebsd/freebsd-src/pull/1179 sorry for doubling up on phab and gb. I'm new here :)

Let's put all the reviews in the PR for this. It's usually a bad idea to do both and wastes my time having to deal with both unless there's a good reason.

Committed by @stevek in 3bbe8ed1a7 (D44535)

@bz Ok, so it turns out that the issue I had was not with the driver. So there was no issue with the reset function, and the current version of the Diff properly works for me with the ACPI version of the driver. Since the clk* functions are no longer called when sc->mutex is locked, I assume the issue regarding mutex ordering is also fixed for the FDT driver. So this means the driver should also be ready for you to test.

I will do additional tests. There have been quite a few moving parts this week on my end (with the I2C tool I have been working on in addition to the driver), so I could have rushed my tests a bit too much and have drawn the wrong conclusions. I will do systematic tests starting from working versions.

In D44020#1009471, @bz wrote:

In D44020#1009470, @pldrouin_gmail.com wrote:

In D44020#1009469, @bz wrote:

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

My assumption here is that it is the reset function is not called twice at the same time in parallel from the bus.

What you describe is what I do in the latest version, but it does not work, since without the lock, i2c_get_div_val gets executed before the attach function has the chance to set the variables that are used by i2c_get_div_val. So I need to delay the call to i2c_get_div_val until the attach function completes. Without that I end up with the wrong divider value. Note that this problem seems to only occur at boot time and not if I load the driver as a module after booting.

Do you enable interrupts before you need them? Do you need IBIC_BIIE for the initial READ?

In D44020#1009470, @pldrouin_gmail.com wrote:

In D44020#1009469, @bz wrote:

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

My assumption here is that it is the reset function is not called twice at the same time in parallel from the bus.

What you describe is what I do in the latest version, but it does not work, since without the lock, i2c_get_div_val gets executed before the attach function has the chance to set the variables that are used by i2c_get_div_val. So I need to delay the call to i2c_get_div_val until the attach function completes. Without that I end up with the wrong divider value. Note that this problem seems to only occur at boot time and not if I load the driver as a module after booting.

In D44020#1009469, @bz wrote:

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

My assumption here is that it is the reset function is not called twice at the same time in parallel from the bus.

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

In D44020#1009368, @pldrouin_gmail.com wrote:

Ok I think I know how this could be fixed. I should create a sc->initialized_freq flag which is set by the attach function once sc->freq has been set. The reset function should check the status of this flag after locking sc->mutex, and if it is not the case, it should cv_wait for a condition from attach. The attach function should cv_signal this condition after setting the flag. Does it look like the best strategy to you?

Ok I think I know how this could be fixed. I should create a sc->initialized_freq flag which is set by the attach function once sc->freq has been set. The reset function should check the status of this flag after locking sc->mutex, and if it is not the case, it should cv_wait for a condition from attach. The attach function should cv_signal this condition after setting the flag. Does it look like the best strategy to you?

Is there a way to prevent the reset function from being called before the driver is done attaching?

The driver seems to be misbehaving with the latest changes. I think it is due to interference between the attach and the reset functions (the attach and reset functions are called in parallel, and i2c_get_div_val gets called by the reset function before sc->freq is set to UINT32_MAX). This issue combined with the lock restrictions for the clk* functions makes it tricky to get the driver properly initialized at boot time. I will have to take a look at it another day...

Merging changes from D44020

Moving back sc->mutex initialization into vf_i2c_attach_common as before, but now locking the mutex whenever it is needed to communicate with the controller. No longer locking it before calling ofw* and clk* functions in vf_i2c_fdt_attach

I think I will have to make additional changes...

Trying to fix the "lock order reversal: (sleepable after non-sleepable)" error in FDT by moving back to an i2g_get_div_val function that gets called outside the sc->mutex lock with a WRITE call of the divider register inside the lock, that gets only called when the divider register value has been set.

Merging changes from D44020

Please ignore this revision

Based on the last couple of tests, it looks like the clk functions should not be called when a mutex is locked with mtx_lock?

We'll need to adjust that slightly in a different direction ...
@manu can you lend a hand? Probably a lot easier for you than for me.

Sorry the previous diff had been generated using an outdated version of the D44020 diff.

-The previous diff was not generated properly w.r.t. the last version of D44020

Merging changes from D44020

Adding a couple of missing spaces

In D44020#1006425, @markmi_dsl-only.net wrote:

FYI: sys/dev/clk/clk.c reports:

/*
 * Locking - we use three levels of locking:
 * - First, topology lock is taken.  This one protect all lists.
 * - Second level is per clknode lock.  It protects clknode data.
 * - Third level is outside of this file, it protect clock device registers.
 * First two levels use sleepable locks; clock device can use mutex or sx lock.
 */

So clk_get_freq using a sleepable lock in its implementation is expected:

CLK_TOPO_SLOCK();
rv = clknode_get_freq(clknode, freq);
CLK_TOPO_UNLOCK();

Merging changes from D44020

Modifying vf_i2c.c, vf_i2c_fdt.c and vf_i2c_acpi.c so sc->mutex is used for the attach and detach functions as well, in order to prevent race conditions, in particular with the reset function that is called in parallel with the attach function at boot time.

FYI: sys/dev/clk/clk.c reports:

Thank you for all the changes. I think we are getting pretty close to getting this in.

Merging updates from D44020