Updated with @cem's suggestions. Reposting for mentor approval.

In D22253#486458, @cem wrote:

Thanks!

Some initial tentative feedback and CCing kib@. Caveat, I haven't yet consulted any official documentation nor have I looked at the adjacent code to understand what it is doing, so my comments may just be uninformed. But, provisionally, I don't understand the motivation for some of this changeset.

In D22007#481175, @cem wrote:

Write Combining (WC) ... If the WC buffer is partially filled, the writes may be delayed until the next occurrence of a serializing event; such as, an SFENCE or MFENCE instruction, CPUID execution, a read or write to uncached memory, an interrupt occurrence, or a LOCK instruction execution.

The list is oddly specific and does not include 'mov to control registers,' but the "such as" language and cpuid example suggest to me that setting cr3 may be sufficient (if, as Konstantin points out, we actually did so).

ah sorry, here's the same patch, but with context restored.

This fixes a failure in the test suite. libvmmalloc was using more thread local storage in the inital-exec mode than rtld was reserving space for, causing the library to fail to load. Switching to the local-dynamic model fixes the issue and allows the test suite to pass.

In D21987#480367, @cem wrote:

Do Control Region SPAs have namespaces? I'm still fairly new to NFIT.

Likewise, thanks for the speedy reviews all.

• remove the machdep.intel_speed_shift tunable
• call out to hwpstate_intel's identify from est's
• show the cf_drv_dev's nameunit in dev.cpufreq.%d.freq_driver
• initialize the debug sysctl's sbuf w/o a drain function
• reject percentages outside [0, 100] in epp sysctl

• update cpufreq.4
• create hwpstate_intel.4
• guard intel_speed_shift with amd64 || i386
• undo dev => cf_dev
• some cf_drv_dev => dev
• some style fixes
• SYSCTL_ADD_STRING => SYSCTL_ADD_CONST_STRING
• SYSCTL_OUT thing

Sorry for the long delay here, I was waiting for the pmdk-1.6 release which incorporates some of our fixes upstream, and then went on vacation for the summer.

Index: stable/12

and also:

In D19976#463043, @rgrimes wrote:

The additional note on style(9) on variable order declaration is informative only, you can choose to fix at your option.

@rgrimes I believe I've addressed your comments from earlier. Would you mind taking a look at the new patch? Thanks.

In D21281#462741, @bcran wrote:

In D21281#462739, @scottph wrote:

Is this error case something that happens only sometimes? Fetching multiple files is something that has definitely worked for me before.

Yes, it only happens if a file can't be found. For example if it's asked to open /boot/foo, then it'll try /boot/foo and /boot/foo/ before failing. And the crash is only seen when booting over a secure connection: HTTP works fine, HTTPS crashes.

Is this error case something that happens only sometimes? Fetching multiple files is something that has definitely worked for me before.

Still working on fully reviewing this, but I noticed this one thing in testing.

Looks good to me

gdb_read_mem and gdb_write_mem could conceivably be factored together, though I'm not sure the complexity of that would be worth it. With or without that, this patch looks good to me.

In D19976#430794, @araujo wrote:

@scott.d.phillips_intel.com I'm wondering if sysutils/uefi-edk2-bhyve will work out of the box with these new options, or do we need to update this port as well?

reword comment about GETADDFLAGS macro

In D19975#429429, @rgrimes wrote:

A question to @jhb, should the order of the enum and the code updated to be in alphabetic order, in a separate review, it would only obfuscate the change to do it here.

bhyve.8, update .Dd, start sentences on a new line.

Note that the UEFI firmware used with this needs to have the flash variable driver added in, as in: https://github.com/freebsd/uefi-edk2/pull/8

Aside from Makefile formatting questions, this looks fine to me and makes a working UEFI binary for me.

• fix one more style(9) problem

• rebase patch
• add commit message tags
• Don't use i for non-int loop variables
• Fix some style(9) issues with braces
• Don't use variable names prefixed with underscore
• Handle unaligned accesses in find_matches()
• Fix an off-by-one error in the allocation of ivars

• rebase patch
• add commit message tags

Add Intel copyright, add commit message tags.

Add copyright for Intel, add commit message tags, allocate flush addrs with mallocarray .

In D18346#394764, @kib wrote:

Ok. I think we could do some cross of what you want and what I do not want.

Right now how I wrote nvdimm.c, it directly digs into subordinate devices of the root NVDIMM device and instantiate device_t for each found device in the namespace in probe. What if we add a top-level newbus driver (in nvdimm.c) which attaches to the root ACPI device. This driver duties would be:

• create children for each subordinate device, where the current nvdimm.c driver attaches. Basically, this moves the nvdimm_probe() into the root device driver;
• iterate over the NFIT and create SPAs, basically identical to what you do in the current patch.

The difference with the patch is that root driver is not a hack but a place where we put all the enumeration logic.

In D18346#394755, @kib wrote:

I do not understand your fixation on the newbus attachment for SPA. Now you are trying to hide it under the root NVDIMM device, but again, I do not see any connection between the root and activation of SPA support. To get SPA used, we only need SPA entries from the NFIT table.

rebase diff so that it is no longer against D18346

ah, the other thing was that having SPAs as devices gave me a convenient fix to the ordering problem where the nvdimm devices are not yet attached at SPA initialization time. perhaps a better fix would be to have a device on the acpi bus for the NVDIMM root device (ACPI0012), which control the SPA initialization, outside of anything related to geom.

In D18346#392753, @kib wrote:

But namespaces have no relevance for newbus at all, e.g. we do not create device_t for partitions for the same reason.

In D18346#392735, @kib wrote:

What do we need ? I see that we need a unique unit number (currently punned by reusing the SPA index) and a way to list all SPAs at the unload time, to destruct them. Would be a global list of all SPAs enough for the later ?

In D18346#392701, @kib wrote:

Why do we need newbus attachment for SPAs ?

For ACPI-enumerated NVDIMMs, creating the devices is more or less natural since they represent physical objects that exist in the chassis. But SPAs are purely software constructs, which newbus devices seems to only tunnel the calls to init_ine()/fini_one().

FYI, this diff is on top of D18346

• check for the existence of an spa device before adding a new one to the acpi bus.

@kib Here's an approach to splitting the geom and spa device by making nvdimm_spa devices on the acpi bus. Let me know if this looks like the right direction here.

In D18009#390469, @0mp wrote:

@mat, I believe that this revision is ready to be committed.

The whole Doxygen and Pandoc thing could be dropped all together. We may add them later if @scott.d.phillips_intel.com decides it is useful.

Fix assert to use va loop variable.

• revert to pmdk 1.5
• address @mat's comments on Diff 50890

In D18346#390124, @kib wrote:

So there are really two changes, and the addition of the spa_g_destroy() is rather small comparing to the other part. You stopped allocating spa_mappings[] array at all. Can you extract it into separate change ?

BTW, I am not sure if it is possible to destroy geom forcibly by a user command. If yes, then the tight bind between spa lifetime and geom lifetime is undesirable, I think, because besides geom spa provides usermode access device.

@0mp: sorry for all the errors still in portlint, I was running it without any options and thinking I would see everything important. I've fixed the man pages path. There were warnings about the other documentation files too in the older patch, but those got moved out of pmdk in the 1.5 version. The 1.5 version also added a strangely packaged python module which I've left out of the port altogether for the time being.

Update DISTVERSION to 1.5, update man path, add USE_LDCONFIG