Page MenuHomeFreeBSD

powerpc/pmap: Use dcbz to zero pages in the radix pmap
Needs ReviewPublic

Authored by pkubaj on Tue, Sep 8, 12:49 PM.
Tags
None
Referenced Files
F171149982: D59507.id.diff
Wed, Sep 9, 12:55 AM
F171144455: D59507.id.diff
Tue, Sep 8, 11:59 PM
F171137366: D59507.id186216.diff
Tue, Sep 8, 10:49 PM
F171130232: D59507.id186213.diff
Tue, Sep 8, 9:37 PM
F171122601: D59507.diff
Tue, Sep 8, 8:34 PM
Subscribers

Details

Reviewers
None
Group Reviewers
PowerPC
Summary

pagezero() in the radix pmap was a plain store loop (bzero), which under
the kernel build flags (-mno-vsx -msoft-float) compiles to byte stores.
dcbz establishes a zeroed cache line directly in the cache without a
read-for-ownership fetch from memory, roughly halving the memory
transactions of page zeroing.

dcbz raises an alignment interrupt on caching-inhibited mappings, and
the kernel does not emulate it, so mmu_radix_zero_page() falls back to
bzero() for any page whose memattr is not the write-back default. The
internal pagezero() callers only touch freshly allocated page-table
pages, which are always write-back.

Only page zeroing is changed. dcbz helps solely because zeroing has no
source to read; for page copying it is a pessimization (a redundant
zeroing pass on top of the mandatory source read), so
mmu_radix_copy_page() is deliberately left as a plain bcopy().

Test Plan

Measured on POWER9 (Raptor Blackbird, DD2.3, bare metal), zeroing a cold
256 MB buffer with 128-byte scalar loops (median of 5, tight variance):

byte stores (what libkern memset compiles to)   8.6 GB/s
doubleword (std) stores                         26.8 GB/s
dcbz                                            34.6 GB/s

Cross-checked on two POWER9 pseries guests; both there hit a ~17 GB/s
memory-bandwidth ceiling that flattens std and dcbz, but neither ever
regresses. Fresh anonymous memory verified to read back byte-exact zero.

Diff Detail

Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

Updated the diff: the previous upload accidentally included an unrelated hunk reverting the pmap_invalidate_range() operand order, an artifact of generating the diff against a base branch that already carried that separate fix. This revision now contains only the dcbz page zero/copy change.

sys/powerpc/aim/mmu_radix.c
912

This should be dcbtst ("data cache block touch to store"), since you're writing to the cache line anyway. You only want to allocate the line (mark it as owned), not do anything with it until

pkubaj retitled this revision from powerpc/pmap: Use dcbz to zero and copy pages in the radix pmap to powerpc/pmap: Use dcbz to zero pages in the radix pmap.
pkubaj edited the summary of this revision. (Show Details)
pkubaj edited the test plan for this revision. (Show Details)

Updated: dropped the page-copy change and narrowed this to page zeroing only. Bare-metal benchmarking on POWER9 (Blackbird DD2.3) showed dcbz is a pessimization for copying (it adds a redundant zeroing pass on top of the mandatory source read: ~-13% cold, ~-31% for the COW hot-source case) while remaining a clear win for zeroing (no source to read). mmu_radix_copy_page() is therefore left as plain bcopy(). Retitled accordingly.