The gic_page_size change should be split out to a new review.

This is a showstopper for Snapdragon Elite support...

git diff -U999999

Please generate patches with git diff -U99999 or git show -U99999 so that Phab will show context

@jhb yes, of course.

Can you please add IOMMU to arm64 NOTES?

Moved to github.

I put all of the suggested changes in https://github.com/freebsd/freebsd-src/pull/1179 sorry for doubling up on phab and gb. I'm new here :)

Let's put all the reviews in the PR for this. It's usually a bad idea to do both and wastes my time having to deal with both unless there's a good reason.

Committed by @stevek in 3bbe8ed1a7 (D44535)

@bz Ok, so it turns out that the issue I had was not with the driver. So there was no issue with the reset function, and the current version of the Diff properly works for me with the ACPI version of the driver. Since the clk* functions are no longer called when sc->mutex is locked, I assume the issue regarding mutex ordering is also fixed for the FDT driver. So this means the driver should also be ready for you to test.

I will do additional tests. There have been quite a few moving parts this week on my end (with the I2C tool I have been working on in addition to the driver), so I could have rushed my tests a bit too much and have drawn the wrong conclusions. I will do systematic tests starting from working versions.

In D44020#1009471, @bz wrote:

In D44020#1009470, @pldrouin_gmail.com wrote:

In D44020#1009469, @bz wrote:

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

My assumption here is that it is the reset function is not called twice at the same time in parallel from the bus.

What you describe is what I do in the latest version, but it does not work, since without the lock, i2c_get_div_val gets executed before the attach function has the chance to set the variables that are used by i2c_get_div_val. So I need to delay the call to i2c_get_div_val until the attach function completes. Without that I end up with the wrong divider value. Note that this problem seems to only occur at boot time and not if I load the driver as a module after booting.

Do you enable interrupts before you need them? Do you need IBIC_BIIE for the initial READ?

In D44020#1009470, @pldrouin_gmail.com wrote:

In D44020#1009469, @bz wrote:

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

My assumption here is that it is the reset function is not called twice at the same time in parallel from the bus.

What you describe is what I do in the latest version, but it does not work, since without the lock, i2c_get_div_val gets executed before the attach function has the chance to set the variables that are used by i2c_get_div_val. So I need to delay the call to i2c_get_div_val until the attach function completes. Without that I end up with the wrong divider value. Note that this problem seems to only occur at boot time and not if I load the driver as a module after booting.

In D44020#1009469, @bz wrote:

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

My assumption here is that it is the reset function is not called twice at the same time in parallel from the bus.

This may not be based on your latest version but the general idea is: call i2c_update_div_val() before taking the lock, get the div_reg value back (maybe you can avoid passing it as pointer and checking for error by simply checking for sc->freq == UINT32_MAX in the caller (reset function) afterwards to see if you should change anything or not (just came to my mind, haven't checked all code paths). Then do the register writes all together under lock.

In D44020#1009368, @pldrouin_gmail.com wrote:

Ok I think I know how this could be fixed. I should create a sc->initialized_freq flag which is set by the attach function once sc->freq has been set. The reset function should check the status of this flag after locking sc->mutex, and if it is not the case, it should cv_wait for a condition from attach. The attach function should cv_signal this condition after setting the flag. Does it look like the best strategy to you?

Ok I think I know how this could be fixed. I should create a sc->initialized_freq flag which is set by the attach function once sc->freq has been set. The reset function should check the status of this flag after locking sc->mutex, and if it is not the case, it should cv_wait for a condition from attach. The attach function should cv_signal this condition after setting the flag. Does it look like the best strategy to you?

Is there a way to prevent the reset function from being called before the driver is done attaching?

The driver seems to be misbehaving with the latest changes. I think it is due to interference between the attach and the reset functions (the attach and reset functions are called in parallel, and i2c_get_div_val gets called by the reset function before sc->freq is set to UINT32_MAX). This issue combined with the lock restrictions for the clk* functions makes it tricky to get the driver properly initialized at boot time. I will have to take a look at it another day...

Merging changes from D44020

Moving back sc->mutex initialization into vf_i2c_attach_common as before, but now locking the mutex whenever it is needed to communicate with the controller. No longer locking it before calling ofw* and clk* functions in vf_i2c_fdt_attach

I think I will have to make additional changes...

Trying to fix the "lock order reversal: (sleepable after non-sleepable)" error in FDT by moving back to an i2g_get_div_val function that gets called outside the sc->mutex lock with a WRITE call of the divider register inside the lock, that gets only called when the divider register value has been set.

Merging changes from D44020

Please ignore this revision

Based on the last couple of tests, it looks like the clk functions should not be called when a mutex is locked with mtx_lock?

We'll need to adjust that slightly in a different direction ...
@manu can you lend a hand? Probably a lot easier for you than for me.

Sorry the previous diff had been generated using an outdated version of the D44020 diff.

-The previous diff was not generated properly w.r.t. the last version of D44020

Merging changes from D44020

Adding a couple of missing spaces

In D44020#1006425, @markmi_dsl-only.net wrote:

FYI: sys/dev/clk/clk.c reports:

/*
 * Locking - we use three levels of locking:
 * - First, topology lock is taken.  This one protect all lists.
 * - Second level is per clknode lock.  It protects clknode data.
 * - Third level is outside of this file, it protect clock device registers.
 * First two levels use sleepable locks; clock device can use mutex or sx lock.
 */

So clk_get_freq using a sleepable lock in its implementation is expected:

CLK_TOPO_SLOCK();
rv = clknode_get_freq(clknode, freq);
CLK_TOPO_UNLOCK();

Merging changes from D44020

Modifying vf_i2c.c, vf_i2c_fdt.c and vf_i2c_acpi.c so sc->mutex is used for the attach and detach functions as well, in order to prevent race conditions, in particular with the reset function that is called in parallel with the attach function at boot time.

FYI: sys/dev/clk/clk.c reports:

Thank you for all the changes. I think we are getting pretty close to getting this in.

Merging updates from D44020

• Renaming driver's data structure

• Merging readability change from D44020.
• Fixing a bug that was introduced in Diff 134928 (a pointer was not properly declared).

Yes it is equivalent. I updated the code.

Modifying the code to make it more readable

In D44021#1005017, @bz wrote:

WOW. Having looked at that logic probably a year ago this is good work; also good catches on the missing unlock and NOACK for the 1 byte!

Also changing the references for the driver updates, based on recommendations in D44020.

• Merge changes that were done to D44020 into here
• Implementing all bz's recommendations

Implementing all but one of bz's recommendations, pending feedback from comment.

WOW. Having looked at that logic probably a year ago this is good work; also good catches on the missing unlock and NOACK for the 1 byte!

Sorry this looks a lot but is mostly just white space.

In D43811#1004062, @pldrouin_gmail.com wrote:

In D43811#1004020, @bz wrote:

One comment: I tested this on a LS1088 in FDT mode and it makes me super happy as I can scan bits (and flawlessly read) now I had problems with on another board (which I should test as well once I can again).

And that brings me to a request -- and please say "no" if you feel like it's too much hassle. Can we split this into two parts as it is essentially two things: (i) the move of the code and split for the various bus attachments, and (ii) the fixing/improving of the actual I2C bits/logic. It would certainly help commit messages and explanations of the why these changes are done and working so well.

There's a few style bits and some hard coded values we should probably #define in there still but nothing major. I'll wait on commenting on whether we'll split it or not.

At this point in order to split it into two diffs like that I think the easiest would be to reintroduce the old code into the split files. Maybe I can do it manually if you think it is worth it.

In D43811#1004020, @bz wrote:

One comment: I tested this on a LS1088 in FDT mode and it makes me super happy as I can scan bits (and flawlessly read) now I had problems with on another board (which I should test as well once I can again).

And that brings me to a request -- and please say "no" if you feel like it's too much hassle. Can we split this into two parts as it is essentially two things: (i) the move of the code and split for the various bus attachments, and (ii) the fixing/improving of the actual I2C bits/logic. It would certainly help commit messages and explanations of the why these changes are done and working so well.

There's a few style bits and some hard coded values we should probably #define in there still but nothing major. I'll wait on commenting on whether we'll split it or not.

One comment: I tested this on a LS1088 in FDT mode and it makes me super happy as I can scan bits (and flawlessly read) now I had problems with on another board (which I should test as well once I can again).

Updating copyright notice to state copyright for each author and also removing the original vf_i2c author for vf_i2c_acpi

In D43746#1003716, @markmi_dsl-only.net wrote:

In D43746#1003698, @pldrouin_gmail.com wrote:

In D43746#1003688, @bz wrote:

In D43746#999338, @pldrouin_gmail.com wrote:

Merging the code with the existing driver in D43811. This differential is now obsolete.

Can you "abandon" this one so it's no longer open?

Where can I do that? I fail to find such an option...

In the bottom Submit area there is an "Add Action..." button in the top of the subarea. For the author, it should show an option for abandonment that you could select.

In D43746#1003698, @pldrouin_gmail.com wrote:

In D43746#1003688, @bz wrote:

In D43746#999338, @pldrouin_gmail.com wrote:

Merging the code with the existing driver in D43811. This differential is now obsolete.

Can you "abandon" this one so it's no longer open?

Where can I do that? I fail to find such an option...

I looked through src and it looks like either form of quoting are used for the copyright notice...

In D43746#1003688, @bz wrote:

In D43746#999338, @pldrouin_gmail.com wrote:

Merging the code with the existing driver in D43811. This differential is now obsolete.

Can you "abandon" this one so it's no longer open?

I got it from https://www.freebsd.org/internal/software-license/

In D43746#999338, @pldrouin_gmail.com wrote:

Merging the code with the existing driver in D43811. This differential is now obsolete.

In D43811#1003573, @bz wrote:

Really looking forward to this. Also a step towards getting SFP+ support going..

sys/modules/vf_i2c/Makefile: Removing header files that are not required

Really looking forward to this. Also a step towards getting SFP+ support going..

As far as I know, all requests have been addressed. Please let me know if I have missed anything. Thanks!

sys/arm/freescale/vybrid/files.vybrid: Removing ACPI support for vf_i2c

I think the latest diff addresses everything that has been mentioned so far.

Merging vf_i2c_acpi and vf_i2c_fdt devices and modules