Index: user/ngie/make_check/UPDATING =================================================================== --- user/ngie/make_check/UPDATING (revision 295077) +++ user/ngie/make_check/UPDATING (revision 295078) @@ -1,1485 +1,1492 @@ Updating Information for FreeBSD current users. This file is maintained and copyrighted by M. Warner Losh . See end of file for further details. For commonly done items, please see the COMMON ITEMS: section later in the file. These instructions assume that you basically know what you are doing. If not, then please consult the FreeBSD handbook: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html Items affecting the ports and packages system can be found in /usr/ports/UPDATING. Please read that file before running portupgrade. NOTE: FreeBSD has switched from gcc to clang. If you have trouble bootstrapping from older versions of FreeBSD, try WITHOUT_CLANG and WITH_GCC to bootstrap to the tip of head, and then rebuild without this option. The bootstrap process from older version of current across the gcc/clang cutover is a bit fragile. NOTE TO PEOPLE WHO THINK THAT FreeBSD 11.x IS SLOW: FreeBSD 11.x has many debugging features turned on, in both the kernel and userland. These features attempt to detect incorrect use of system primitives, and encourage loud failure through extra sanity checking and fail stop semantics. They also substantially impact system performance. If you want to do performance measurement, benchmarking, and optimization, you'll want to turn them off. This includes various WITNESS- related kernel options, INVARIANTS, malloc debugging flags in userland, and various verbose features in the kernel. Many developers choose to disable these features on build machines to maximize performance. (To completely disable malloc debugging, define MALLOC_PRODUCTION in /etc/make.conf, or to merely disable the most expensive debugging functionality run "ln -s 'abort:false,junk:false' /etc/malloc.conf".) +20160129: + Building ZFS pools on top of zvols is prohibited by default. That + feature has never worked safely; it's always been prone to deadlocks. + Using a zvol as the backing store for a VM guest's virtual disk will + still work, even if the guest is using ZFS. Legacy behavior can be + restored by setting vfs.zfs.vol.recursive=1. + 20160119: The NONE and HPN patches has been removed from OpenSSH. They are still available in the security/openssh-portable port. 20160113: With the addition of ypldap(8), a new _ypldap user is now required during installworld. "mergemaster -p" can be used to add the user prior to installworld, as documented in the handbook. 20151216: The tftp loader (pxeboot) now uses the option root-path directive. As a consequence it no longer looks for a pxeboot.4th file on the tftp server. Instead it uses the regular /boot infrastructure as with the other loaders. 20151211: The code to start recording plug and play data into the modules has been committed. While the old tools will properly build a new kernel, a number of warnings about "unknown metadata record 4" will be produced for an older kldxref. To avoid such warnings, make sure to rebuild the kernel toolchain (or world). Make sure that you have r292078 or later when trying to build 292077 or later before rebuilding. 20151207: Debug data files are now built by default with 'make buildworld' and installed with 'make installworld'. This facilitates debugging but requires more disk space both during the build and for the installed world. Debug files may be disabled by setting WITHOUT_DEBUG_FILES=yes in src.conf(5). 20151130: r291527 changed the internal interface between the nfsd.ko and nfscommon.ko modules. As such, they must both be upgraded to-gether. __FreeBSD_version has been bumped because of this. 20151108: Add support for unicode collation strings leads to a change of order of files listed by ls(1) for example. To get back to the old behaviour, set LC_COLLATE environment variable to "C". Databases administrators will need to reindex their databases given collation results will be different. Due to a bug in install(1) it is recommended to remove the ancient locales before running make installworld. rm -rf /usr/share/locale/* 20151030: The OpenSSL has been upgraded to 1.0.2d. Any binaries requiring libcrypto.so.7 or libssl.so.7 must be recompiled. 20151020: Qlogic 24xx/25xx firmware images were updated from 5.5.0 to 7.3.0. Kernel modules isp_2400_multi and isp_2500_multi were removed and should be replaced with isp_2400 and isp_2500 modules respectively. 20151017: The build previously allowed using 'make -n' to not recurse into sub-directories while showing what commands would be executed, and 'make -n -n' to recursively show commands. Now 'make -n' will recurse and 'make -N' will not. 20151012: If you specify SENDMAIL_MC or SENDMAIL_CF in make.conf, mergemaster and etcupdate will now use this file. A custom sendmail.cf is now updated via this mechanism rather than via installworld. If you had excluded sendmail.cf in mergemaster.rc or etcupdate.conf, you may want to remove the exclusion or change it to "always install". /etc/mail/sendmail.cf is now managed the same way regardless of whether SENDMAIL_MC/SENDMAIL_CF is used. If you are not using SENDMAIL_MC/SENDMAIL_CF there should be no change in behavior. 20151011: Compatibility shims for legacy ATA device names have been removed. It includes ATA_STATIC_ID kernel option, kern.cam.ada.legacy_aliases and kern.geom.raid.legacy_aliases loader tunables, kern.devalias.* environment variables, /dev/ad* and /dev/ar* symbolic links. 20151006: Clang, llvm, lldb, compiler-rt and libc++ have been upgraded to 3.7.0. Please see the 20141231 entry below for information about prerequisites and upgrading, if you are not already using clang 3.5.0 or higher. 20150924: Kernel debug files have been moved to /usr/lib/debug/boot/kernel/, and renamed from .symbols to .debug. This reduces the size requirements on the boot partition or file system and provides consistency with userland debug files. When using the supported kernel installation method the /usr/lib/debug/boot/kernel directory will be renamed (to kernel.old) as is done with /boot/kernel. Developers wishing to maintain the historical behavior of installing debug files in /boot/kernel/ can set KERN_DEBUGDIR="" in src.conf(5). 20150827: The wireless drivers had undergone changes that remove the 'parent interface' from the ifconfig -l output. The rc.d network scripts used to check presence of a parent interface in the list, so old scripts would fail to start wireless networking. Thus, etcupdate(3) or mergemaster(8) run is required after kernel update, to update your rc.d scripts in /etc. 20150827: pf no longer supports 'scrub fragment crop' or 'scrub fragment drop-ovl' These configurations are now automatically interpreted as 'scrub fragment reassemble'. 20150817: Kernel-loadable modules for the random(4) device are back. To use them, the kernel must have device random options RANDOM_LOADABLE kldload(8) can then be used to load random_fortuna.ko or random_yarrow.ko. Please note that due to the indirect function calls that the loadable modules need to provide, the build-in variants will be slightly more efficient. The random(4) kernel option RANDOM_DUMMY has been retired due to unpopularity. It was not all that useful anyway. 20150813: The WITHOUT_ELFTOOLCHAIN_TOOLS src.conf(5) knob has been retired. Control over building the ELF Tool Chain tools is now provided by the WITHOUT_TOOLCHAIN knob. 20150810: The polarity of Pulse Per Second (PPS) capture events with the uart(4) driver has been corrected. Prior to this change the PPS "assert" event corresponded to the trailing edge of a positive PPS pulse and the "clear" event was the leading edge of the next pulse. As the width of a PPS pulse in a typical GPS receiver is on the order of 1 millisecond, most users will not notice any significant difference with this change. Anyone who has compensated for the historical polarity reversal by configuring a negative offset equal to the pulse width will need to remove that workaround. 20150809: The default group assigned to /dev/dri entries has been changed from 'wheel' to 'video' with the id of '44'. If you want to have access to the dri devices please add yourself to the video group with: # pw groupmod video -m $USER 20150806: The menu.rc and loader.rc files will now be replaced during upgrades. Please migrate local changes to menu.rc.local and loader.rc.local instead. 20150805: GNU Binutils versions of addr2line, c++filt, nm, readelf, size, strings and strip have been removed. The src.conf(5) knob WITHOUT_ELFTOOLCHAIN_TOOLS no longer provides the binutils tools. 20150728: As ZFS requires more kernel stack pages than is the default on some architectures e.g. i386, it now warns if KSTACK_PAGES is less than ZFS_MIN_KSTACK_PAGES (which is 4 at the time of writing). Please consider using 'options KSTACK_PAGES=X' where X is greater than or equal to ZFS_MIN_KSTACK_PAGES i.e. 4 in such configurations. 20150706: sendmail has been updated to 8.15.2. Starting with FreeBSD 11.0 and sendmail 8.15, sendmail uses uncompressed IPv6 addresses by default, i.e., they will not contain "::". For example, instead of ::1, it will be 0:0:0:0:0:0:0:1. This permits a zero subnet to have a more specific match, such as different map entries for IPv6:0:0 vs IPv6:0. This change requires that configuration data (including maps, files, classes, custom ruleset, etc.) must use the same format, so make certain such configuration data is upgrading. As a very simple check search for patterns like 'IPv6:[0-9a-fA-F:]*::' and 'IPv6::'. To return to the old behavior, set the m4 option confUSE_COMPRESSED_IPV6_ADDRESSES or the cf option UseCompressedIPv6Addresses. 20150630: The default kernel entropy-processing algorithm is now Fortuna, replacing Yarrow. Assuming you have 'device random' in your kernel config file, the configurations allow a kernel option to override this default. You may choose *ONE* of: options RANDOM_YARROW # Legacy /dev/random algorithm. options RANDOM_DUMMY # Blocking-only driver. If you have neither, you get Fortuna. For most people, read no further, Fortuna will give a /dev/random that works like it always used to, and the difference will be irrelevant. If you remove 'device random', you get *NO* kernel-processed entropy at all. This may be acceptable to folks building embedded systems, but has complications. Carry on reading, and it is assumed you know what you need. *PLEASE* read random(4) and random(9) if you are in the habit of tweaking kernel configs, and/or if you are a member of the embedded community, wanting specific and not-usual behaviour from your security subsystems. NOTE!! If you use RANDOM_DUMMY and/or have no 'device random', you will NOT have a functioning /dev/random, and many cryptographic features will not work, including SSH. You may also find strange behaviour from the random(3) set of library functions, in particular sranddev(3), srandomdev(3) and arc4random(3). The reason for this is that the KERN_ARND sysctl only returns entropy if it thinks it has some to share, and with RANDOM_DUMMY or no 'device random' this will never happen. 20150623: An additional fix for the issue described in the 20150614 sendmail entry below has been been committed in revision 284717. 20150616: FreeBSD's old make (fmake) has been removed from the system. It is available as the devel/fmake port or via pkg install fmake. 20150615: The fix for the issue described in the 20150614 sendmail entry below has been been committed in revision 284436. The work around described in that entry is no longer needed unless the default setting is overridden by a confDH_PARAMETERS configuration setting of '5' or pointing to a 512 bit DH parameter file. 20150614: ALLOW_DEPRECATED_ATF_TOOLS/ATFFILE support has been removed from atf.test.mk (included from bsd.test.mk). Please upgrade devel/atf and devel/kyua to version 0.20+ and adjust any calling code to work with Kyuafile and kyua. 20150614: The import of openssl to address the FreeBSD-SA-15:10.openssl security advisory includes a change which rejects handshakes with DH parameters below 768 bits. sendmail releases prior to 8.15.2 (not yet released), defaulted to a 512 bit DH parameter setting for client connections. To work around this interoperability, sendmail can be configured to use a 2048 bit DH parameter by: 1. Edit /etc/mail/`hostname`.mc 2. If a setting for confDH_PARAMETERS does not exist or exists and is set to a string beginning with '5', replace it with '2'. 3. If a setting for confDH_PARAMETERS exists and is set to a file path, create a new file with: openssl dhparam -out /path/to/file 2048 4. Rebuild the .cf file: cd /etc/mail/; make; make install 5. Restart sendmail: cd /etc/mail/; make restart A sendmail patch is coming, at which time this file will be updated. 20150604: Generation of legacy formatted entries have been disabled by default in pwd_mkdb(8), as all base system consumers of the legacy formatted entries were converted to use the new format by default when the new, machine independent format have been added and supported since FreeBSD 5.x. Please see the pwd_mkdb(8) manual page for further details. 20150525: Clang and llvm have been upgraded to 3.6.1 release. Please see the 20141231 entry below for information about prerequisites and upgrading, if you are not already using 3.5.0 or higher. 20150521: TI platform code switched to using vendor DTS files and this update may break existing systems running on Beaglebone, Beaglebone Black, and Pandaboard: - dtb files should be regenerated/reinstalled. Filenames are the same but content is different now - GPIO addressing was changed, now each GPIO bank (32 pins per bank) has its own /dev/gpiocX device, e.g. pin 121 on /dev/gpioc0 in old addressing scheme is now pin 25 on /dev/gpioc3. - Pandaboard: /etc/ttys should be updated, serial console device is now /dev/ttyu2, not /dev/ttyu0 20150501: soelim(1) from gnu/usr.bin/groff has been replaced by usr.bin/soelim. If you need the GNU extension from groff soelim(1), install groff from package: pkg install groff, or via ports: textproc/groff. 20150423: chmod, chflags, chown and chgrp now affect symlinks in -R mode as defined in symlink(7); previously symlinks were silently ignored. 20150415: The const qualifier has been removed from iconv(3) to comply with POSIX. The ports tree is aware of this from r384038 onwards. 20150416: Libraries specified by LIBADD in Makefiles must have a corresponding DPADD_ variable to ensure correct dependencies. This is now enforced in src.libnames.mk. 20150324: From legacy ata(4) driver was removed support for SATA controllers supported by more functional drivers ahci(4), siis(4) and mvs(4). Kernel modules ataahci and ataadaptec were removed completely, replaced by ahci and mvs modules respectively. 20150315: Clang, llvm and lldb have been upgraded to 3.6.0 release. Please see the 20141231 entry below for information about prerequisites and upgrading, if you are not already using 3.5.0 or higher. 20150307: The 32-bit PowerPC kernel has been changed to a position-independent executable. This can only be booted with a version of loader(8) newer than January 31, 2015, so make sure to update both world and kernel before rebooting. 20150217: If you are running a -CURRENT kernel since r273872 (Oct 30th, 2014), but before r278950, the RNG was not seeded properly. Immediately upgrade the kernel to r278950 or later and regenerate any keys (e.g. ssh keys or openssl keys) that were generated w/ a kernel from that range. This does not affect programs that directly used /dev/random or /dev/urandom. All userland uses of arc4random(3) are affected. 20150210: The autofs(4) ABI was changed in order to restore binary compatibility with 10.1-RELEASE. The automountd(8) daemon needs to be rebuilt to work with the new kernel. 20150131: The powerpc64 kernel has been changed to a position-independent executable. This can only be booted with a new version of loader(8), so make sure to update both world and kernel before rebooting. 20150118: Clang and llvm have been upgraded to 3.5.1 release. This is a bugfix only release, no new features have been added. Please see the 20141231 entry below for information about prerequisites and upgrading, if you are not already using 3.5.0. 20150107: ELF tools addr2line, elfcopy (strip), nm, size, and strings are now taken from the ELF Tool Chain project rather than GNU binutils. They should be drop-in replacements, with the addition of arm64 support. The WITHOUT_ELFTOOLCHAIN_TOOLS= knob may be used to obtain the binutils tools, if necessary. See 20150805 for updated information. 20150105: The default Unbound configuration now enables remote control using a local socket. Users who have already enabled the local_unbound service should regenerate their configuration by running "service local_unbound setup" as root. 20150102: The GNU texinfo and GNU info pages have been removed. To be able to view GNU info pages please install texinfo from ports. 20141231: Clang, llvm and lldb have been upgraded to 3.5.0 release. As of this release, a prerequisite for building clang, llvm and lldb is a C++11 capable compiler and C++11 standard library. This means that to be able to successfully build the cross-tools stage of buildworld, with clang as the bootstrap compiler, your system compiler or cross compiler should either be clang 3.3 or later, or gcc 4.8 or later, and your system C++ library should be libc++, or libdstdc++ from gcc 4.8 or later. On any standard FreeBSD 10.x or 11.x installation, where clang and libc++ are on by default (that is, on x86 or arm), this should work out of the box. On 9.x installations where clang is enabled by default, e.g. on x86 and powerpc, libc++ will not be enabled by default, so libc++ should be built (with clang) and installed first. If both clang and libc++ are missing, build clang first, then use it to build libc++. On 8.x and earlier installations, upgrade to 9.x first, and then follow the instructions for 9.x above. Sparc64 and mips users are unaffected, as they still use gcc 4.2.1 by default, and do not build clang. Many embedded systems are resource constrained, and will not be able to build clang in a reasonable time, or in some cases at all. In those cases, cross building bootable systems on amd64 is a workaround. This new version of clang introduces a number of new warnings, of which the following are most likely to appear: -Wabsolute-value This warns in two cases, for both C and C++: * When the code is trying to take the absolute value of an unsigned quantity, which is effectively a no-op, and almost never what was intended. The code should be fixed, if at all possible. If you are sure that the unsigned quantity can be safely cast to signed, without loss of information or undefined behavior, you can add an explicit cast, or disable the warning. * When the code is trying to take an absolute value, but the called abs() variant is for the wrong type, which can lead to truncation. If you want to disable the warning instead of fixing the code, please make sure that truncation will not occur, or it might lead to unwanted side-effects. -Wtautological-undefined-compare and -Wundefined-bool-conversion These warn when C++ code is trying to compare 'this' against NULL, while 'this' should never be NULL in well-defined C++ code. However, there is some legacy (pre C++11) code out there, which actively abuses this feature, which was less strictly defined in previous C++ versions. Squid and openjdk do this, for example. The warning can be turned off for C++98 and earlier, but compiling the code in C++11 mode might result in unexpected behavior; for example, the parts of the program that are unreachable could be optimized away. 20141222: The old NFS client and server (kernel options NFSCLIENT, NFSSERVER) kernel sources have been removed. The .h files remain, since some utilities include them. This will need to be fixed later. If "mount -t oldnfs ..." is attempted, it will fail. If the "-o" option on mountd(8), nfsd(8) or nfsstat(1) is used, the utilities will report errors. 20141121: The handling of LOCAL_LIB_DIRS has been altered to skip addition of directories to top level SUBDIR variable when their parent directory is included in LOCAL_DIRS. Users with build systems with such hierarchies and without SUBDIR entries in the parent directory Makefiles should add them or add the directories to LOCAL_DIRS. 20141109: faith(4) and faithd(8) have been removed from the base system. Faith has been obsolete for a very long time. 20141104: vt(4), the new console driver, is enabled by default. It brings support for Unicode and double-width characters, as well as support for UEFI and integration with the KMS kernel video drivers. You may need to update your console settings in /etc/rc.conf, most probably the keymap. During boot, /etc/rc.d/syscons will indicate what you need to do. vt(4) still has issues and lacks some features compared to syscons(4). See the wiki for up-to-date information: https://wiki.freebsd.org/Newcons If you want to keep using syscons(4), you can do so by adding the following line to /boot/loader.conf: kern.vty=sc 20141102: pjdfstest has been integrated into kyua as an opt-in test suite. Please see share/doc/pjdfstest/README for more details on how to execute it. 20141009: gperf has been removed from the base system for architectures that use clang. Ports that require gperf will obtain it from the devel/gperf port. 20140923: pjdfstest has been moved from tools/regression/pjdfstest to contrib/pjdfstest . 20140922: At svn r271982, The default linux compat kernel ABI has been adjusted to 2.6.18 in support of the linux-c6 compat ports infrastructure update. If you wish to continue using the linux-f10 compat ports, add compat.linux.osrelease=2.6.16 to your local sysctl.conf. Users are encouraged to update their linux-compat packages to linux-c6 during their next update cycle. 20140729: The ofwfb driver, used to provide a graphics console on PowerPC when using vt(4), no longer allows mmap() of all physical memory. This will prevent Xorg on PowerPC with some ATI graphics cards from initializing properly unless x11-servers/xorg-server is updated to 1.12.4_8 or newer. 20140723: The xdev targets have been converted to using TARGET and TARGET_ARCH instead of XDEV and XDEV_ARCH. 20140719: The default unbound configuration has been modified to address issues with reverse lookups on networks that use private address ranges. If you use the local_unbound service, run "service local_unbound setup" as root to regenerate your configuration, then "service local_unbound reload" to load the new configuration. 20140709: The GNU texinfo and GNU info pages are not built and installed anymore, WITH_INFO knob has been added to allow to built and install them again. UPDATE: see 20150102 entry on texinfo's removal 20140708: The GNU readline library is now an INTERNALLIB - that is, it is statically linked into consumers (GDB and variants) in the base system, and the shared library is no longer installed. The devel/readline port is available for third party software that requires readline. 20140702: The Itanium architecture (ia64) has been removed from the list of known architectures. This is the first step in the removal of the architecture. 20140701: Commit r268115 has added NFSv4.1 server support, merged from projects/nfsv4.1-server. Since this includes changes to the internal interfaces between the NFS related modules, a full build of the kernel and modules will be necessary. __FreeBSD_version has been bumped. 20140629: The WITHOUT_VT_SUPPORT kernel config knob has been renamed WITHOUT_VT. (The other _SUPPORT knobs have a consistent meaning which differs from the behaviour controlled by this knob.) 20140619: Maximal length of the serial number in CTL was increased from 16 to 64 chars, that breaks ABI. All CTL-related tools, such as ctladm and ctld, need to be rebuilt to work with a new kernel. 20140606: The libatf-c and libatf-c++ major versions were downgraded to 0 and 1 respectively to match the upstream numbers. They were out of sync because, when they were originally added to FreeBSD, the upstream versions were not respected. These libraries are private and not yet built by default, so renumbering them should be a non-issue. However, unclean source trees will yield broken test programs once the operator executes "make delete-old-libs" after a "make installworld". Additionally, the atf-sh binary was made private by moving it into /usr/libexec/. Already-built shell test programs will keep the path to the old binary so they will break after "make delete-old" is run. If you are using WITH_TESTS=yes (not the default), wipe the object tree and rebuild from scratch to prevent spurious test failures. This is only needed once: the misnumbered libraries and misplaced binaries have been added to OptionalObsoleteFiles.inc so they will be removed during a clean upgrade. 20140512: Clang and llvm have been upgraded to 3.4.1 release. 20140508: We bogusly installed src.opts.mk in /usr/share/mk. This file should be removed to avoid issues in the future (and has been added to ObsoleteFiles.inc). 20140505: /etc/src.conf now affects only builds of the FreeBSD src tree. In the past, it affected all builds that used the bsd.*.mk files. The old behavior was a bug, but people may have relied upon it. To get this behavior back, you can .include /etc/src.conf from /etc/make.conf (which is still global and isn't changed). This also changes the behavior of incremental builds inside the tree of individual directories. Set MAKESYSPATH to ".../share/mk" to do that. Although this has survived make universe and some upgrade scenarios, other upgrade scenarios may have broken. At least one form of temporary breakage was fixed with MAKESYSPATH settings for buildworld as well... In cases where MAKESYSPATH isn't working with this setting, you'll need to set it to the full path to your tree. One side effect of all this cleaning up is that bsd.compiler.mk is no longer implicitly included by bsd.own.mk. If you wish to use COMPILER_TYPE, you must now explicitly include bsd.compiler.mk as well. 20140430: The lindev device has been removed since /dev/full has been made a standard device. __FreeBSD_version has been bumped. 20140424: The knob WITHOUT_VI was added to the base system, which controls building ex(1), vi(1), etc. Older releases of FreeBSD required ex(1) in order to reorder files share/termcap and didn't build ex(1) as a build tool, so building/installing with WITH_VI is highly advised for build hosts for older releases. This issue has been fixed in stable/9 and stable/10 in r277022 and r276991, respectively. 20140418: The YES_HESIOD knob has been removed. It has been obsolete for a decade. Please move to using WITH_HESIOD instead or your builds will silently lack HESIOD. 20140405: The uart(4) driver has been changed with respect to its handling of the low-level console. Previously the uart(4) driver prevented any process from changing the baudrate or the CLOCAL and HUPCL control flags. By removing the restrictions, operators can make changes to the serial console port without having to reboot. However, when getty(8) is started on the serial device that is associated with the low-level console, a misconfigured terminal line in /etc/ttys will now have a real impact. Before upgrading the kernel, make sure that /etc/ttys has the serial console device configured as 3wire without baudrate to preserve the previous behaviour. E.g: ttyu0 "/usr/libexec/getty 3wire" vt100 on secure 20140306: Support for libwrap (TCP wrappers) in rpcbind was disabled by default to improve performance. To re-enable it, if needed, run rpcbind with command line option -W. 20140226: Switched back to the GPL dtc compiler due to updates in the upstream dts files not being supported by the BSDL dtc compiler. You will need to rebuild your kernel toolchain to pick up the new compiler. Core dumps may result while building dtb files during a kernel build if you fail to do so. Set WITHOUT_GPL_DTC if you require the BSDL compiler. 20140216: Clang and llvm have been upgraded to 3.4 release. 20140216: The nve(4) driver has been removed. Please use the nfe(4) driver for NVIDIA nForce MCP Ethernet adapters instead. 20140212: An ABI incompatibility crept into the libc++ 3.4 import in r261283. This could cause certain C++ applications using shared libraries built against the previous version of libc++ to crash. The incompatibility has now been fixed, but any C++ applications or shared libraries built between r261283 and r261801 should be recompiled. 20140204: OpenSSH will now ignore errors caused by kernel lacking of Capsicum capability mode support. Please note that enabling the feature in kernel is still highly recommended. 20140131: OpenSSH is now built with sandbox support, and will use sandbox as the default privilege separation method. This requires Capsicum capability mode support in kernel. 20140128: The libelf and libdwarf libraries have been updated to newer versions from upstream. Shared library version numbers for these two libraries were bumped. Any ports or binaries requiring these two libraries should be recompiled. __FreeBSD_version is bumped to 1100006. 20140110: If a Makefile in a tests/ directory was auto-generating a Kyuafile instead of providing an explicit one, this would prevent such Makefile from providing its own Kyuafile in the future during NO_CLEAN builds. This has been fixed in the Makefiles but manual intervention is needed to clean an objdir if you use NO_CLEAN: # find /usr/obj -name Kyuafile | xargs rm -f 20131213: The behavior of gss_pseudo_random() for the krb5 mechanism has changed, for applications requesting a longer random string than produced by the underlying enctype's pseudo-random() function. In particular, the random string produced from a session key of enctype aes256-cts-hmac-sha1-96 or aes256-cts-hmac-sha1-96 will be different at the 17th octet and later, after this change. The counter used in the PRF+ construction is now encoded as a big-endian integer in accordance with RFC 4402. __FreeBSD_version is bumped to 1100004. 20131108: The WITHOUT_ATF build knob has been removed and its functionality has been subsumed into the more generic WITHOUT_TESTS. If you were using the former to disable the build of the ATF libraries, you should change your settings to use the latter. 20131025: The default version of mtree is nmtree which is obtained from NetBSD. The output is generally the same, but may vary slightly. If you found you need identical output adding "-F freebsd9" to the command line should do the trick. For the time being, the old mtree is available as fmtree. 20131014: libbsdyml has been renamed to libyaml and moved to /usr/lib/private. This will break ports-mgmt/pkg. Rebuild the port, or upgrade to pkg 1.1.4_8 and verify bsdyml not linked in, before running "make delete-old-libs": # make -C /usr/ports/ports-mgmt/pkg build deinstall install clean or # pkg install pkg; ldd /usr/local/sbin/pkg | grep bsdyml 20131010: The stable/10 branch has been created in subversion from head revision r256279. 20131010: The rc.d/jail script has been updated to support jail(8) configuration file. The "jail__*" rc.conf(5) variables for per-jail configuration are automatically converted to /var/run/jail..conf before the jail(8) utility is invoked. This is transparently backward compatible. See below about some incompatibilities and rc.conf(5) manual page for more details. These variables are now deprecated in favor of jail(8) configuration file. One can use "rc.d/jail config " command to generate a jail(8) configuration file in /var/run/jail..conf without running the jail(8) utility. The default pathname of the configuration file is /etc/jail.conf and can be specified by using $jail_conf or $jail__conf variables. Please note that jail_devfs_ruleset accepts an integer at this moment. Please consider to rewrite the ruleset name with an integer. 20130930: BIND has been removed from the base system. If all you need is a local resolver, simply enable and start the local_unbound service instead. Otherwise, several versions of BIND are available in the ports tree. The dns/bind99 port is one example. With this change, nslookup(1) and dig(1) are no longer in the base system. Users should instead use host(1) and drill(1) which are in the base system. Alternatively, nslookup and dig can be obtained by installing the dns/bind-tools port. 20130916: With the addition of unbound(8), a new unbound user is now required during installworld. "mergemaster -p" can be used to add the user prior to installworld, as documented in the handbook. 20130911: OpenSSH is now built with DNSSEC support, and will by default silently trust signed SSHFP records. This can be controlled with the VerifyHostKeyDNS client configuration setting. DNSSEC support can be disabled entirely with the WITHOUT_LDNS option in src.conf. 20130906: The GNU Compiler Collection and C++ standard library (libstdc++) are no longer built by default on platforms where clang is the system compiler. You can enable them with the WITH_GCC and WITH_GNUCXX options in src.conf. 20130905: The PROCDESC kernel option is now part of the GENERIC kernel configuration and is required for the rwhod(8) to work. If you are using custom kernel configuration, you should include 'options PROCDESC'. 20130905: The API and ABI related to the Capsicum framework was modified in backward incompatible way. The userland libraries and programs have to be recompiled to work with the new kernel. This includes the following libraries and programs, but the whole buildworld is advised: libc, libprocstat, dhclient, tcpdump, hastd, hastctl, kdump, procstat, rwho, rwhod, uniq. 20130903: AES-NI intrinsic support has been added to gcc. The AES-NI module has been updated to use this support. A new gcc is required to build the aesni module on both i386 and amd64. 20130821: The PADLOCK_RNG and RDRAND_RNG kernel options are now devices. Thus "device padlock_rng" and "device rdrand_rng" should be used instead of "options PADLOCK_RNG" & "options RDRAND_RNG". 20130813: WITH_ICONV has been split into two feature sets. WITH_ICONV now enables just the iconv* functionality and is now on by default. WITH_LIBICONV_COMPAT enables the libiconv api and link time compatability. Set WITHOUT_ICONV to build the old way. If you have been using WITH_ICONV before, you will very likely need to turn on WITH_LIBICONV_COMPAT. 20130806: INVARIANTS option now enables DEBUG for code with OpenSolaris and Illumos origin, including ZFS. If you have INVARIANTS in your kernel configuration, then there is no need to set DEBUG or ZFS_DEBUG explicitly. DEBUG used to enable witness(9) tracking of OpenSolaris (mostly ZFS) locks if WITNESS option was set. Because that generated a lot of witness(9) reports and all of them were believed to be false positives, this is no longer done. New option OPENSOLARIS_WITNESS can be used to achieve the previous behavior. 20130806: Timer values in IPv6 data structures now use time_uptime instead of time_second. Although this is not a user-visible functional change, userland utilities which directly use them---ndp(8), rtadvd(8), and rtsold(8) in the base system---need to be updated to r253970 or later. 20130802: find -delete can now delete the pathnames given as arguments, instead of only files found below them or if the pathname did not contain any slashes. Formerly, the following error message would result: find: -delete: : relative path potentially not safe Deleting the pathnames given as arguments can be prevented without error messages using -mindepth 1 or by changing directory and passing "." as argument to find. This works in the old as well as the new version of find. 20130726: Behavior of devfs rules path matching has been changed. Pattern is now always matched against fully qualified devfs path and slash characters must be explicitly matched by slashes in pattern (FNM_PATHNAME). Rulesets involving devfs subdirectories must be reviewed. 20130716: The default ARM ABI has changed to the ARM EABI. The old ABI is incompatible with the ARM EABI and all programs and modules will need to be rebuilt to work with a new kernel. To keep using the old ABI ensure the WITHOUT_ARM_EABI knob is set. NOTE: Support for the old ABI will be removed in the future and users are advised to upgrade. 20130709: pkg_install has been disconnected from the build if you really need it you should add WITH_PKGTOOLS in your src.conf(5). 20130709: Most of network statistics structures were changed to be able keep 64-bits counters. Thus all tools, that work with networking statistics, must be rebuilt (netstat(1), bsnmpd(1), etc.) 20130618: Fix a bug that allowed a tracing process (e.g. gdb) to write to a memory-mapped file in the traced process's address space even if neither the traced process nor the tracing process had write access to that file. 20130615: CVS has been removed from the base system. An exact copy of the code is available from the devel/cvs port. 20130613: Some people report the following error after the switch to bmake: make: illegal option -- J usage: make [-BPSXeiknpqrstv] [-C directory] [-D variable] ... *** [buildworld] Error code 2 this likely due to an old instance of make in ${MAKEPATH} (${MAKEOBJDIRPREFIX}${.CURDIR}/make.${MACHINE}) which src/Makefile will use that blindly, if it exists, so if you see the above error: rm -rf `make -V MAKEPATH` should resolve it. 20130516: Use bmake by default. Whereas before one could choose to build with bmake via -DWITH_BMAKE one must now use -DWITHOUT_BMAKE to use the old make. The goal is to remove these knobs for 10-RELEASE. It is worth noting that bmake (like gmake) treats the command line as the unit of failure, rather than statements within the command line. Thus '(cd some/where && dosomething)' is safer than 'cd some/where; dosomething'. The '()' allows consistent behavior in parallel build. 20130429: Fix a bug that allows NFS clients to issue READDIR on files. 20130426: The WITHOUT_IDEA option has been removed because the IDEA patent expired. 20130426: The sysctl which controls TRIM support under ZFS has been renamed from vfs.zfs.trim_disable -> vfs.zfs.trim.enabled and has been enabled by default. 20130425: The mergemaster command now uses the default MAKEOBJDIRPREFIX rather than creating it's own in the temporary directory in order allow access to bootstrapped versions of tools such as install and mtree. When upgrading from version of FreeBSD where the install command does not support -l, you will need to install a new mergemaster command if mergemaster -p is required. This can be accomplished with the command (cd src/usr.sbin/mergemaster && make install). 20130404: Legacy ATA stack, disabled and replaced by new CAM-based one since FreeBSD 9.0, completely removed from the sources. Kernel modules atadisk and atapi*, user-level tools atacontrol and burncd are removed. Kernel option `options ATA_CAM` is now permanently enabled and removed. 20130319: SOCK_CLOEXEC and SOCK_NONBLOCK flags have been added to socket(2) and socketpair(2). Software, in particular Kerberos, may automatically detect and use these during building. The resulting binaries will not work on older kernels. 20130308: CTL_DISABLE has also been added to the sparc64 GENERIC (for further information, see the respective 20130304 entry). 20130304: Recent commits to callout(9) changed the size of struct callout, so the KBI is probably heavily disturbed. Also, some functions in callout(9)/sleep(9)/sleepqueue(9)/condvar(9) KPIs were replaced by macros. Every kernel module using it won't load, so rebuild is requested. The ctl device has been re-enabled in GENERIC for i386 and amd64, but does not initialize by default (because of the new CTL_DISABLE option) to save memory. To re-enable it, remove the CTL_DISABLE option from the kernel config file or set kern.cam.ctl.disable=0 in /boot/loader.conf. 20130301: The ctl device has been disabled in GENERIC for i386 and amd64. This was done due to the extra memory being allocated at system initialisation time by the ctl driver which was only used if a CAM target device was created. This makes a FreeBSD system unusable on 128MB or less of RAM. 20130208: A new compression method (lz4) has been merged to -HEAD. Please refer to zpool-features(7) for more information. Please refer to the "ZFS notes" section of this file for information on upgrading boot ZFS pools. 20130129: A BSD-licensed patch(1) variant has been added and is installed as bsdpatch, being the GNU version the default patch. To inverse the logic and use the BSD-licensed one as default, while having the GNU version installed as gnupatch, rebuild and install world with the WITH_BSD_PATCH knob set. 20130121: Due to the use of the new -l option to install(1) during build and install, you must take care not to directly set the INSTALL make variable in your /etc/make.conf, /etc/src.conf, or on the command line. If you wish to use the -C flag for all installs you may be able to add INSTALL+=-C to /etc/make.conf or /etc/src.conf. 20130118: The install(1) option -M has changed meaning and now takes an argument that is a file or path to append logs to. In the unlikely event that -M was the last option on the command line and the command line contained at least two files and a target directory the first file will have logs appended to it. The -M option served little practical purpose in the last decade so its use is expected to be extremely rare. 20121223: After switching to Clang as the default compiler some users of ZFS on i386 systems started to experience stack overflow kernel panics. Please consider using 'options KSTACK_PAGES=4' in such configurations. 20121222: GEOM_LABEL now mangles label names read from file system metadata. Mangling affect labels containing spaces, non-printable characters, '%' or '"'. Device names in /etc/fstab and other places may need to be updated. 20121217: By default, only the 10 most recent kernel dumps will be saved. To restore the previous behaviour (no limit on the number of kernel dumps stored in the dump directory) add the following line to /etc/rc.conf: savecore_flags="" 20121201: With the addition of auditdistd(8), a new auditdistd user is now required during installworld. "mergemaster -p" can be used to add the user prior to installworld, as documented in the handbook. 20121117: The sin6_scope_id member variable in struct sockaddr_in6 is now filled by the kernel before passing the structure to the userland via sysctl or routing socket. This means the KAME-specific embedded scope id in sin6_addr.s6_addr[2] is always cleared in userland application. This behavior can be controlled by net.inet6.ip6.deembed_scopeid. __FreeBSD_version is bumped to 1000025. 20121105: On i386 and amd64 systems WITH_CLANG_IS_CC is now the default. This means that the world and kernel will be compiled with clang and that clang will be installed as /usr/bin/cc, /usr/bin/c++, and /usr/bin/cpp. To disable this behavior and revert to building with gcc, compile with WITHOUT_CLANG_IS_CC. Really old versions of current may need to bootstrap WITHOUT_CLANG first if the clang build fails (its compatibility window doesn't extend to the 9 stable branch point). 20121102: The IPFIREWALL_FORWARD kernel option has been removed. Its functionality now turned on by default. 20121023: The ZERO_COPY_SOCKET kernel option has been removed and split into SOCKET_SEND_COW and SOCKET_RECV_PFLIP. NB: SOCKET_SEND_COW uses the VM page based copy-on-write mechanism which is not safe and may result in kernel crashes. NB: The SOCKET_RECV_PFLIP mechanism is useless as no current driver supports disposeable external page sized mbuf storage. Proper replacements for both zero-copy mechanisms are under consideration and will eventually lead to complete removal of the two kernel options. 20121023: The IPv4 network stack has been converted to network byte order. The following modules need to be recompiled together with kernel: carp(4), divert(4), gif(4), siftr(4), gre(4), pf(4), ipfw(4), ng_ipfw(4), stf(4). 20121022: Support for non-MPSAFE filesystems was removed from VFS. The VFS_VERSION was bumped, all filesystem modules shall be recompiled. 20121018: All the non-MPSAFE filesystems have been disconnected from the build. The full list includes: codafs, hpfs, ntfs, nwfs, portalfs, smbfs, xfs. 20121016: The interface cloning API and ABI has changed. The following modules need to be recompiled together with kernel: ipfw(4), pfsync(4), pflog(4), usb(4), wlan(4), stf(4), vlan(4), disc(4), edsc(4), if_bridge(4), gif(4), tap(4), faith(4), epair(4), enc(4), tun(4), if_lagg(4), gre(4). 20121015: The sdhci driver was split in two parts: sdhci (generic SD Host Controller logic) and sdhci_pci (actual hardware driver). No kernel config modifications are required, but if you load sdhc as a module you must switch to sdhci_pci instead. 20121014: Import the FUSE kernel and userland support into base system. 20121013: The GNU sort(1) program has been removed since the BSD-licensed sort(1) has been the default for quite some time and no serious problems have been reported. The corresponding WITH_GNU_SORT knob has also gone. 20121006: The pfil(9) API/ABI for AF_INET family has been changed. Packet filtering modules: pf(4), ipfw(4), ipfilter(4) need to be recompiled with new kernel. 20121001: The net80211(4) ABI has been changed to allow for improved driver PS-POLL and power-save support. All wireless drivers need to be recompiled to work with the new kernel. 20120913: The random(4) support for the VIA hardware random number generator (`PADLOCK') is no longer enabled unconditionally. Add the padlock_rng device in the custom kernel config if needed. The GENERIC kernels on i386 and amd64 do include the device, so the change only affects the custom kernel configurations. 20120908: The pf(4) packet filter ABI has been changed. pfctl(8) and snmp_pf module need to be recompiled to work with new kernel. 20120828: A new ZFS feature flag "com.delphix:empty_bpobj" has been merged to -HEAD. Pools that have empty_bpobj in active state can not be imported read-write with ZFS implementations that do not support this feature. For more information read the zpool-features(5) manual page. 20120727: The sparc64 ZFS loader has been changed to no longer try to auto- detect ZFS providers based on diskN aliases but now requires these to be explicitly listed in the OFW boot-device environment variable. 20120712: The OpenSSL has been upgraded to 1.0.1c. Any binaries requiring libcrypto.so.6 or libssl.so.6 must be recompiled. Also, there are configuration changes. Make sure to merge /etc/ssl/openssl.cnf. 20120712: The following sysctls and tunables have been renamed for consistency with other variables: kern.cam.da.da_send_ordered -> kern.cam.da.send_ordered kern.cam.ada.ada_send_ordered -> kern.cam.ada.send_ordered 20120628: The sort utility has been replaced with BSD sort. For now, GNU sort is also available as "gnusort" or the default can be set back to GNU sort by setting WITH_GNU_SORT. In this case, BSD sort will be installed as "bsdsort". 20120611: A new version of ZFS (pool version 5000) has been merged to -HEAD. Starting with this version the old system of ZFS pool versioning is superseded by "feature flags". This concept enables forward compatibility against certain future changes in functionality of ZFS pools. The first read-only compatible "feature flag" for ZFS pools is named "com.delphix:async_destroy". For more information read the new zpool-features(5) manual page. Please refer to the "ZFS notes" section of this file for information on upgrading boot ZFS pools. 20120417: The malloc(3) implementation embedded in libc now uses sources imported as contrib/jemalloc. The most disruptive API change is to /etc/malloc.conf. If your system has an old-style /etc/malloc.conf, delete it prior to installworld, and optionally re-create it using the new format after rebooting. See malloc.conf(5) for details (specifically the TUNING section and the "opt.*" entries in the MALLCTL NAMESPACE section). 20120328: Big-endian MIPS TARGET_ARCH values no longer end in "eb". mips64eb is now spelled mips64. mipsn32eb is now spelled mipsn32. mipseb is now spelled mips. This is to aid compatibility with third-party software that expects this naming scheme in uname(3). Little-endian settings are unchanged. If you are updating a big-endian mips64 machine from before this change, you may need to set MACHINE_ARCH=mips64 in your environment before the new build system will recognize your machine. 20120306: Disable by default the option VFS_ALLOW_NONMPSAFE for all supported platforms. 20120229: Now unix domain sockets behave "as expected" on nullfs(5). Previously nullfs(5) did not pass through all behaviours to the underlying layer, as a result if we bound to a socket on the lower layer we could connect only to the lower path; if we bound to the upper layer we could connect only to the upper path. The new behavior is one can connect to both the lower and the upper paths regardless what layer path one binds to. 20120211: The getifaddrs upgrade path broken with 20111215 has been restored. If you have upgraded in between 20111215 and 20120209 you need to recompile libc again with your kernel. You still need to recompile world to be able to configure CARP but this restriction already comes from 20111215. 20120114: The set_rcvar() function has been removed from /etc/rc.subr. All base and ports rc.d scripts have been updated, so if you have a port installed with a script in /usr/local/etc/rc.d you can either hand-edit the rcvar= line, or reinstall the port. An easy way to handle the mass-update of /etc/rc.d: rm /etc/rc.d/* && mergemaster -i 20120109: panic(9) now stops other CPUs in the SMP systems, disables interrupts on the current CPU and prevents other threads from running. This behavior can be reverted using the kern.stop_scheduler_on_panic tunable/sysctl. The new behavior can be incompatible with kern.sync_on_panic. 20111215: The carp(4) facility has been changed significantly. Configuration of the CARP protocol via ifconfig(8) has changed, as well as format of CARP events submitted to devd(8) has changed. See manual pages for more information. The arpbalance feature of carp(4) is currently not supported anymore. Size of struct in_aliasreq, struct in6_aliasreq has changed. User utilities using SIOCAIFADDR, SIOCAIFADDR_IN6, e.g. ifconfig(8), need to be recompiled. 20111122: The acpi_wmi(4) status device /dev/wmistat has been renamed to /dev/wmistat0. 20111108: The option VFS_ALLOW_NONMPSAFE option has been added in order to explicitely support non-MPSAFE filesystems. It is on by default for all supported platform at this present time. 20111101: The broken amd(4) driver has been replaced with esp(4) in the amd64, i386 and pc98 GENERIC kernel configuration files. 20110930: sysinstall has been removed 20110923: The stable/9 branch created in subversion. This corresponds to the RELENG_9 branch in CVS. COMMON ITEMS: General Notes ------------- Avoid using make -j when upgrading. While generally safe, there are sometimes problems using -j to upgrade. If your upgrade fails with -j, please try again without -j. From time to time in the past there have been problems using -j with buildworld and/or installworld. This is especially true when upgrading between "distant" versions (eg one that cross a major release boundary or several minor releases, or when several months have passed on the -current branch). Sometimes, obscure build problems are the result of environment poisoning. This can happen because the make utility reads its environment when searching for values for global variables. To run your build attempts in an "environmental clean room", prefix all make commands with 'env -i '. See the env(1) manual page for more details. When upgrading from one major version to another it is generally best to upgrade to the latest code in the currently installed branch first, then do an upgrade to the new branch. This is the best-tested upgrade path, and has the highest probability of being successful. Please try this approach before reporting problems with a major version upgrade. When upgrading a live system, having a root shell around before installing anything can help undo problems. Not having a root shell around can lead to problems if pam has changed too much from your starting point to allow continued authentication after the upgrade. This file should be read as a log of events. When a later event changes information of a prior event, the prior event should not be deleted. Instead, a pointer to the entry with the new information should be placed in the old entry. Readers of this file should also sanity check older entries before relying on them blindly. Authors of new entries should write them with this in mind. ZFS notes --------- When upgrading the boot ZFS pool to a new version, always follow these two steps: 1.) recompile and reinstall the ZFS boot loader and boot block (this is part of "make buildworld" and "make installworld") 2.) update the ZFS boot block on your boot drive The following example updates the ZFS boot block on the first partition (freebsd-boot) of a GPT partitioned drive ada0: "gpart bootcode -p /boot/gptzfsboot -i 1 ada0" Non-boot pools do not need these updates. To build a kernel ----------------- If you are updating from a prior version of FreeBSD (even one just a few days old), you should follow this procedure. It is the most failsafe as it uses a /usr/obj tree with a fresh mini-buildworld, make kernel-toolchain make -DALWAYS_CHECK_MAKE buildkernel KERNCONF=YOUR_KERNEL_HERE make -DALWAYS_CHECK_MAKE installkernel KERNCONF=YOUR_KERNEL_HERE To test a kernel once --------------------- If you just want to boot a kernel once (because you are not sure if it works, or if you want to boot a known bad kernel to provide debugging information) run make installkernel KERNCONF=YOUR_KERNEL_HERE KODIR=/boot/testkernel nextboot -k testkernel To just build a kernel when you know that it won't mess you up -------------------------------------------------------------- This assumes you are already running a CURRENT system. Replace ${arch} with the architecture of your machine (e.g. "i386", "arm", "amd64", "ia64", "pc98", "sparc64", "powerpc", "mips", etc). cd src/sys/${arch}/conf config KERNEL_NAME_HERE cd ../compile/KERNEL_NAME_HERE make depend make make install If this fails, go to the "To build a kernel" section. To rebuild everything and install it on the current system. ----------------------------------------------------------- # Note: sometimes if you are running current you gotta do more than # is listed here if you are upgrading from a really old current. make buildworld make kernel KERNCONF=YOUR_KERNEL_HERE [1] [3] mergemaster -Fp [5] make installworld mergemaster -Fi [4] make delete-old [6] To cross-install current onto a separate partition -------------------------------------------------- # In this approach we use a separate partition to hold # current's root, 'usr', and 'var' directories. A partition # holding "/", "/usr" and "/var" should be about 2GB in # size. make buildworld make buildkernel KERNCONF=YOUR_KERNEL_HERE make installworld DESTDIR=${CURRENT_ROOT} -DDB_FROM_SRC make distribution DESTDIR=${CURRENT_ROOT} # if newfs'd make installkernel KERNCONF=YOUR_KERNEL_HERE DESTDIR=${CURRENT_ROOT} cp /etc/fstab ${CURRENT_ROOT}/etc/fstab # if newfs'd To upgrade in-place from stable to current ---------------------------------------------- make buildworld [9] make kernel KERNCONF=YOUR_KERNEL_HERE [8] [1] [3] mergemaster -Fp [5] make installworld mergemaster -Fi [4] make delete-old [6] Make sure that you've read the UPDATING file to understand the tweaks to various things you need. At this point in the life cycle of current, things change often and you are on your own to cope. The defaults can also change, so please read ALL of the UPDATING entries. Also, if you are tracking -current, you must be subscribed to freebsd-current@freebsd.org. Make sure that before you update your sources that you have read and understood all the recent messages there. If in doubt, please track -stable which has much fewer pitfalls. [1] If you have third party modules, such as vmware, you should disable them at this point so they don't crash your system on reboot. [3] From the bootblocks, boot -s, and then do fsck -p mount -u / mount -a cd src adjkerntz -i # if CMOS is wall time Also, when doing a major release upgrade, it is required that you boot into single user mode to do the installworld. [4] Note: This step is non-optional. Failure to do this step can result in a significant reduction in the functionality of the system. Attempting to do it by hand is not recommended and those that pursue this avenue should read this file carefully, as well as the archives of freebsd-current and freebsd-hackers mailing lists for potential gotchas. The -U option is also useful to consider. See mergemaster(8) for more information. [5] Usually this step is a noop. However, from time to time you may need to do this if you get unknown user in the following step. It never hurts to do it all the time. You may need to install a new mergemaster (cd src/usr.sbin/mergemaster && make install) after the buildworld before this step if you last updated from current before 20130425 or from -stable before 20130430. [6] This only deletes old files and directories. Old libraries can be deleted by "make delete-old-libs", but you have to make sure that no program is using those libraries anymore. [8] In order to have a kernel that can run the 4.x binaries needed to do an installworld, you must include the COMPAT_FREEBSD4 option in your kernel. Failure to do so may leave you with a system that is hard to boot to recover. A similar kernel option COMPAT_FREEBSD5 is required to run the 5.x binaries on more recent kernels. And so on for COMPAT_FREEBSD6 and COMPAT_FREEBSD7. Make sure that you merge any new devices from GENERIC since the last time you updated your kernel config file. [9] When checking out sources, you must include the -P flag to have cvs prune empty directories. If CPUTYPE is defined in your /etc/make.conf, make sure to use the "?=" instead of the "=" assignment operator, so that buildworld can override the CPUTYPE if it needs to. MAKEOBJDIRPREFIX must be defined in an environment variable, and not on the command line, or in /etc/make.conf. buildworld will warn if it is improperly defined. FORMAT: This file contains a list, in reverse chronological order, of major breakages in tracking -current. It is not guaranteed to be a complete list of such breakages, and only contains entries since September 23, 2011. If you need to see UPDATING entries from before that date, you will need to fetch an UPDATING file from an older FreeBSD release. Copyright information: Copyright 1998-2009 M. Warner Losh. All Rights Reserved. Redistribution, publication, translation and use, with or without modification, in full or in part, in any form or format of this document are permitted without further permission from the author. THIS DOCUMENT IS PROVIDED BY WARNER LOSH ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WARNER LOSH BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Contact Warner Losh if you have any questions about your use of this document. $FreeBSD$ Index: user/ngie/make_check/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_config.c =================================================================== --- user/ngie/make_check/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_config.c (revision 295077) +++ user/ngie/make_check/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_config.c (revision 295078) @@ -1,453 +1,455 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright 2009 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ /* * Copyright (c) 2012 by Delphix. All rights reserved. + * Copyright (c) 2015 by Syneto S.R.L. All rights reserved. */ /* * The pool configuration repository is stored in /etc/zfs/zpool.cache as a * single packed nvlist. While it would be nice to just read in this * file from userland, this wouldn't work from a local zone. So we have to have * a zpool ioctl to return the complete configuration for all pools. In the * global zone, this will be identical to reading the file and unpacking it in * userland. */ #include #include #include #include #include #include #include #include #include "libzfs_impl.h" typedef struct config_node { char *cn_name; nvlist_t *cn_config; uu_avl_node_t cn_avl; } config_node_t; /* ARGSUSED */ static int config_node_compare(const void *a, const void *b, void *unused) { int ret; const config_node_t *ca = (config_node_t *)a; const config_node_t *cb = (config_node_t *)b; ret = strcmp(ca->cn_name, cb->cn_name); if (ret < 0) return (-1); else if (ret > 0) return (1); else return (0); } void namespace_clear(libzfs_handle_t *hdl) { if (hdl->libzfs_ns_avl) { config_node_t *cn; void *cookie = NULL; while ((cn = uu_avl_teardown(hdl->libzfs_ns_avl, &cookie)) != NULL) { nvlist_free(cn->cn_config); free(cn->cn_name); free(cn); } uu_avl_destroy(hdl->libzfs_ns_avl); hdl->libzfs_ns_avl = NULL; } if (hdl->libzfs_ns_avlpool) { uu_avl_pool_destroy(hdl->libzfs_ns_avlpool); hdl->libzfs_ns_avlpool = NULL; } } /* * Loads the pool namespace, or re-loads it if the cache has changed. */ static int namespace_reload(libzfs_handle_t *hdl) { nvlist_t *config; config_node_t *cn; nvpair_t *elem; zfs_cmd_t zc = { 0 }; void *cookie; if (hdl->libzfs_ns_gen == 0) { /* * This is the first time we've accessed the configuration * cache. Initialize the AVL tree and then fall through to the * common code. */ if ((hdl->libzfs_ns_avlpool = uu_avl_pool_create("config_pool", sizeof (config_node_t), offsetof(config_node_t, cn_avl), config_node_compare, UU_DEFAULT)) == NULL) return (no_memory(hdl)); if ((hdl->libzfs_ns_avl = uu_avl_create(hdl->libzfs_ns_avlpool, NULL, UU_DEFAULT)) == NULL) return (no_memory(hdl)); } if (zcmd_alloc_dst_nvlist(hdl, &zc, 0) != 0) return (-1); for (;;) { zc.zc_cookie = hdl->libzfs_ns_gen; if (ioctl(hdl->libzfs_fd, ZFS_IOC_POOL_CONFIGS, &zc) != 0) { switch (errno) { case EEXIST: /* * The namespace hasn't changed. */ zcmd_free_nvlists(&zc); return (0); case ENOMEM: if (zcmd_expand_dst_nvlist(hdl, &zc) != 0) { zcmd_free_nvlists(&zc); return (-1); } break; default: zcmd_free_nvlists(&zc); return (zfs_standard_error(hdl, errno, dgettext(TEXT_DOMAIN, "failed to read " "pool configuration"))); } } else { hdl->libzfs_ns_gen = zc.zc_cookie; break; } } if (zcmd_read_dst_nvlist(hdl, &zc, &config) != 0) { zcmd_free_nvlists(&zc); return (-1); } zcmd_free_nvlists(&zc); /* * Clear out any existing configuration information. */ cookie = NULL; while ((cn = uu_avl_teardown(hdl->libzfs_ns_avl, &cookie)) != NULL) { nvlist_free(cn->cn_config); free(cn->cn_name); free(cn); } elem = NULL; while ((elem = nvlist_next_nvpair(config, elem)) != NULL) { nvlist_t *child; uu_avl_index_t where; if ((cn = zfs_alloc(hdl, sizeof (config_node_t))) == NULL) { nvlist_free(config); return (-1); } if ((cn->cn_name = zfs_strdup(hdl, nvpair_name(elem))) == NULL) { free(cn); nvlist_free(config); return (-1); } verify(nvpair_value_nvlist(elem, &child) == 0); if (nvlist_dup(child, &cn->cn_config, 0) != 0) { free(cn->cn_name); free(cn); nvlist_free(config); return (no_memory(hdl)); } verify(uu_avl_find(hdl->libzfs_ns_avl, cn, NULL, &where) == NULL); uu_avl_insert(hdl->libzfs_ns_avl, cn, where); } nvlist_free(config); return (0); } /* * Retrieve the configuration for the given pool. The configuration is a nvlist * describing the vdevs, as well as the statistics associated with each one. */ nvlist_t * zpool_get_config(zpool_handle_t *zhp, nvlist_t **oldconfig) { if (oldconfig) *oldconfig = zhp->zpool_old_config; return (zhp->zpool_config); } /* * Retrieves a list of enabled features and their refcounts and caches it in * the pool handle. */ nvlist_t * zpool_get_features(zpool_handle_t *zhp) { nvlist_t *config, *features; config = zpool_get_config(zhp, NULL); if (config == NULL || !nvlist_exists(config, ZPOOL_CONFIG_FEATURE_STATS)) { int error; boolean_t missing = B_FALSE; error = zpool_refresh_stats(zhp, &missing); if (error != 0 || missing) return (NULL); config = zpool_get_config(zhp, NULL); } - verify(nvlist_lookup_nvlist(config, ZPOOL_CONFIG_FEATURE_STATS, - &features) == 0); + if (nvlist_lookup_nvlist(config, ZPOOL_CONFIG_FEATURE_STATS, + &features) != 0) + return (NULL); return (features); } /* * Refresh the vdev statistics associated with the given pool. This is used in * iostat to show configuration changes and determine the delta from the last * time the function was called. This function can fail, in case the pool has * been destroyed. */ int zpool_refresh_stats(zpool_handle_t *zhp, boolean_t *missing) { zfs_cmd_t zc = { 0 }; int error; nvlist_t *config; libzfs_handle_t *hdl = zhp->zpool_hdl; *missing = B_FALSE; (void) strcpy(zc.zc_name, zhp->zpool_name); if (zhp->zpool_config_size == 0) zhp->zpool_config_size = 1 << 16; if (zcmd_alloc_dst_nvlist(hdl, &zc, zhp->zpool_config_size) != 0) return (-1); for (;;) { if (ioctl(zhp->zpool_hdl->libzfs_fd, ZFS_IOC_POOL_STATS, &zc) == 0) { /* * The real error is returned in the zc_cookie field. */ error = zc.zc_cookie; break; } if (errno == ENOMEM) { if (zcmd_expand_dst_nvlist(hdl, &zc) != 0) { zcmd_free_nvlists(&zc); return (-1); } } else { zcmd_free_nvlists(&zc); if (errno == ENOENT || errno == EINVAL) *missing = B_TRUE; zhp->zpool_state = POOL_STATE_UNAVAIL; return (0); } } if (zcmd_read_dst_nvlist(hdl, &zc, &config) != 0) { zcmd_free_nvlists(&zc); return (-1); } zcmd_free_nvlists(&zc); zhp->zpool_config_size = zc.zc_nvlist_dst_size; if (zhp->zpool_config != NULL) { uint64_t oldtxg, newtxg; verify(nvlist_lookup_uint64(zhp->zpool_config, ZPOOL_CONFIG_POOL_TXG, &oldtxg) == 0); verify(nvlist_lookup_uint64(config, ZPOOL_CONFIG_POOL_TXG, &newtxg) == 0); if (zhp->zpool_old_config != NULL) nvlist_free(zhp->zpool_old_config); if (oldtxg != newtxg) { nvlist_free(zhp->zpool_config); zhp->zpool_old_config = NULL; } else { zhp->zpool_old_config = zhp->zpool_config; } } zhp->zpool_config = config; if (error) zhp->zpool_state = POOL_STATE_UNAVAIL; else zhp->zpool_state = POOL_STATE_ACTIVE; return (0); } /* * If the __ZFS_POOL_RESTRICT environment variable is set we only iterate over * pools it lists. * * This is an undocumented feature for use during testing only. * * This function returns B_TRUE if the pool should be skipped * during iteration. */ static boolean_t check_restricted(const char *poolname) { static boolean_t initialized = B_FALSE; static char *restricted = NULL; const char *cur, *end; int len, namelen; if (!initialized) { initialized = B_TRUE; restricted = getenv("__ZFS_POOL_RESTRICT"); } if (NULL == restricted) return (B_FALSE); cur = restricted; namelen = strlen(poolname); do { end = strchr(cur, ' '); len = (NULL == end) ? strlen(cur) : (end - cur); if (len == namelen && 0 == strncmp(cur, poolname, len)) { return (B_FALSE); } cur += (len + 1); } while (NULL != end); return (B_TRUE); } /* * Iterate over all pools in the system. */ int zpool_iter(libzfs_handle_t *hdl, zpool_iter_f func, void *data) { config_node_t *cn; zpool_handle_t *zhp; int ret; /* * If someone makes a recursive call to zpool_iter(), we want to avoid * refreshing the namespace because that will invalidate the parent * context. We allow recursive calls, but simply re-use the same * namespace AVL tree. */ if (!hdl->libzfs_pool_iter && namespace_reload(hdl) != 0) return (-1); hdl->libzfs_pool_iter++; for (cn = uu_avl_first(hdl->libzfs_ns_avl); cn != NULL; cn = uu_avl_next(hdl->libzfs_ns_avl, cn)) { if (check_restricted(cn->cn_name)) continue; if (zpool_open_silent(hdl, cn->cn_name, &zhp) != 0) { hdl->libzfs_pool_iter--; return (-1); } if (zhp == NULL) continue; if ((ret = func(zhp, data)) != 0) { hdl->libzfs_pool_iter--; return (ret); } } hdl->libzfs_pool_iter--; return (0); } /* * Iterate over root datasets, calling the given function for each. The zfs * handle passed each time must be explicitly closed by the callback. */ int zfs_iter_root(libzfs_handle_t *hdl, zfs_iter_f func, void *data) { config_node_t *cn; zfs_handle_t *zhp; int ret; if (namespace_reload(hdl) != 0) return (-1); for (cn = uu_avl_first(hdl->libzfs_ns_avl); cn != NULL; cn = uu_avl_next(hdl->libzfs_ns_avl, cn)) { if (check_restricted(cn->cn_name)) continue; if ((zhp = make_dataset_handle(hdl, cn->cn_name)) == NULL) continue; if ((ret = func(zhp, data)) != 0) return (ret); } return (0); } Index: user/ngie/make_check/cddl/contrib/opensolaris/lib/libzfs =================================================================== --- user/ngie/make_check/cddl/contrib/opensolaris/lib/libzfs (revision 295077) +++ user/ngie/make_check/cddl/contrib/opensolaris/lib/libzfs (revision 295078) Property changes on: user/ngie/make_check/cddl/contrib/opensolaris/lib/libzfs ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,2 ## Merged /vendor/illumos/dist/lib/libzfs:r295046 Merged /head/cddl/contrib/opensolaris/lib/libzfs:r295034-295077 Index: user/ngie/make_check/cddl/contrib/opensolaris =================================================================== --- user/ngie/make_check/cddl/contrib/opensolaris (revision 295077) +++ user/ngie/make_check/cddl/contrib/opensolaris (revision 295078) Property changes on: user/ngie/make_check/cddl/contrib/opensolaris ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,2 ## Merged /vendor/illumos/dist:r295046 Merged /head/cddl/contrib/opensolaris:r295034-295077 Index: user/ngie/make_check/cddl =================================================================== --- user/ngie/make_check/cddl (revision 295077) +++ user/ngie/make_check/cddl (revision 295078) Property changes on: user/ngie/make_check/cddl ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/cddl:r295034-295077 Index: user/ngie/make_check/lib/libc/gen/Makefile.inc =================================================================== --- user/ngie/make_check/lib/libc/gen/Makefile.inc (revision 295077) +++ user/ngie/make_check/lib/libc/gen/Makefile.inc (revision 295078) @@ -1,515 +1,519 @@ # @(#)Makefile.inc 8.6 (Berkeley) 5/4/95 # $FreeBSD$ # machine-independent gen sources .PATH: ${LIBC_SRCTOP}/${LIBC_ARCH}/gen ${LIBC_SRCTOP}/gen SRCS+= __getosreldate.c \ __pthread_mutex_init_calloc_cb_stub.c \ __xuname.c \ _once_stub.c \ _pthread_stubs.c \ _rand48.c \ _spinlock_stub.c \ _thread_init.c \ alarm.c \ arc4random.c \ assert.c \ auxv.c \ basename.c \ cap_sandboxed.c \ check_utility_compat.c \ clock.c \ clock_getcpuclockid.c \ closedir.c \ confstr.c \ crypt.c \ ctermid.c \ daemon.c \ devname.c \ dirfd.c \ dirname.c \ disklabel.c \ dlfcn.c \ drand48.c \ dup3.c \ elf_utils.c \ erand48.c \ err.c \ errlst.c \ errno.c \ exec.c \ fdevname.c \ feature_present.c \ fmtcheck.c \ fmtmsg.c \ fnmatch.c \ fpclassify.c \ frexp.c \ fstab.c \ ftok.c \ fts.c \ fts-compat.c \ ftw.c \ getbootfile.c \ getbsize.c \ getcap.c \ getcwd.c \ getdomainname.c \ getgrent.c \ getgrouplist.c \ gethostname.c \ getloadavg.c \ getlogin.c \ getmntinfo.c \ getnetgrent.c \ getosreldate.c \ getpagesize.c \ getpagesizes.c \ getpeereid.c \ getprogname.c \ getpwent.c \ getttyent.c \ getusershell.c \ getutxent.c \ getvfsbyname.c \ glob.c \ initgroups.c \ isatty.c \ isinf.c \ isnan.c \ jrand48.c \ lcong48.c \ libc_dlopen.c \ lockf.c \ lrand48.c \ mrand48.c \ nftw.c \ nice.c \ nlist.c \ nrand48.c \ opendir.c \ pause.c \ pmadvise.c \ popen.c \ posix_spawn.c \ psignal.c \ pututxline.c \ pw_scan.c \ raise.c \ readdir.c \ readpassphrase.c \ + recvmmsg.c \ rewinddir.c \ scandir.c \ seed48.c \ seekdir.c \ semctl.c \ + sendmmsg.c \ setdomainname.c \ sethostname.c \ setjmperr.c \ setmode.c \ setproctitle.c \ setprogname.c \ siginterrupt.c \ siglist.c \ signal.c \ sigsetops.c \ sleep.c \ srand48.c \ statvfs.c \ stringlist.c \ strtofflags.c \ sysconf.c \ sysctl.c \ sysctlbyname.c \ sysctlnametomib.c \ syslog.c \ telldir.c \ termios.c \ time.c \ times.c \ timezone.c \ tls.c \ ttyname.c \ ttyslot.c \ ualarm.c \ ulimit.c \ uname.c \ unvis-compat.c \ usleep.c \ utime.c \ utxdb.c \ valloc.c \ wait.c \ wait3.c \ waitpid.c \ waitid.c \ wordexp.c .PATH: ${LIBC_SRCTOP}/../../contrib/libc-pwcache SRCS+= pwcache.c pwcache.h .PATH: ${LIBC_SRCTOP}/../../contrib/libc-vis CFLAGS+= -I${LIBC_SRCTOP}/../../contrib/libc-vis SRCS+= unvis.c vis.c MISRCS+=modf.c CANCELPOINTS_SRCS=sem.c sem_new.c .for src in ${CANCELPOINTS_SRCS} SRCS+=cancelpoints_${src} CLEANFILES+=cancelpoints_${src} cancelpoints_${src}: ${LIBC_SRCTOP}/gen/${src} .NOMETA ln -sf ${.ALLSRC} ${.TARGET} .endfor SYM_MAPS+=${LIBC_SRCTOP}/gen/Symbol.map # machine-dependent gen sources .sinclude "${LIBC_SRCTOP}/${LIBC_ARCH}/gen/Makefile.inc" MAN+= alarm.3 \ arc4random.3 \ basename.3 \ cap_rights_get.3 \ cap_sandboxed.3 \ check_utility_compat.3 \ clock.3 \ clock_getcpuclockid.3 \ confstr.3 \ ctermid.3 \ daemon.3 \ devname.3 \ directory.3 \ dirname.3 \ dl_iterate_phdr.3 \ dladdr.3 \ dlinfo.3 \ dllockinit.3 \ dlopen.3 \ dup3.3 \ err.3 \ exec.3 \ feature_present.3 \ fmtcheck.3 \ fmtmsg.3 \ fnmatch.3 \ fpclassify.3 \ frexp.3 \ ftok.3 \ fts.3 \ ftw.3 \ getbootfile.3 \ getbsize.3 \ getcap.3 \ getcontext.3 \ getcwd.3 \ getdiskbyname.3 \ getdomainname.3 \ getfsent.3 \ getgrent.3 \ getgrouplist.3 \ gethostname.3 \ getloadavg.3 \ getmntinfo.3 \ getnetgrent.3 \ getosreldate.3 \ getpagesize.3 \ getpagesizes.3 \ getpass.3 \ getpeereid.3 \ getprogname.3 \ getpwent.3 \ getttyent.3 \ getusershell.3 \ getutxent.3 \ getvfsbyname.3 \ glob.3 \ initgroups.3 \ isgreater.3 \ ldexp.3 \ lockf.3 \ makecontext.3 \ modf.3 \ nice.3 \ nlist.3 \ pause.3 \ popen.3 \ posix_spawn.3 \ posix_spawn_file_actions_addopen.3 \ posix_spawn_file_actions_init.3 \ posix_spawnattr_getflags.3 \ posix_spawnattr_getpgroup.3 \ posix_spawnattr_getschedparam.3 \ posix_spawnattr_getschedpolicy.3 \ posix_spawnattr_init.3 \ posix_spawnattr_getsigdefault.3 \ posix_spawnattr_getsigmask.3 \ psignal.3 \ pwcache.3 \ raise.3 \ rand48.3 \ readpassphrase.3 \ rfork_thread.3 \ scandir.3 \ sem_destroy.3 \ sem_getvalue.3 \ sem_init.3 \ sem_open.3 \ sem_post.3 \ sem_timedwait.3 \ sem_wait.3 \ setjmp.3 \ setmode.3 \ setproctitle.3 \ siginterrupt.3 \ signal.3 \ sigsetops.3 \ sleep.3 \ statvfs.3 \ stringlist.3 \ strtofflags.3 \ sysconf.3 \ sysctl.3 \ syslog.3 \ tcgetpgrp.3 \ tcgetsid.3 \ tcsendbreak.3 \ tcsetattr.3 \ tcsetpgrp.3 \ tcsetsid.3 \ time.3 \ times.3 \ timezone.3 \ ttyname.3 \ tzset.3 \ ualarm.3 \ ucontext.3 \ ulimit.3 \ uname.3 \ unvis.3 \ usleep.3 \ utime.3 \ valloc.3 \ vis.3 \ wordexp.3 MLINKS+=arc4random.3 arc4random_addrandom.3 \ arc4random.3 arc4random_stir.3 \ arc4random.3 arc4random_buf.3 \ arc4random.3 arc4random_uniform.3 MLINKS+=basename.3 basename_r.3 MLINKS+=ctermid.3 ctermid_r.3 MLINKS+=devname.3 devname_r.3 MLINKS+=devname.3 fdevname.3 MLINKS+=devname.3 fdevname_r.3 MLINKS+=directory.3 closedir.3 \ directory.3 dirfd.3 \ directory.3 fdclosedir.3 \ directory.3 fdopendir.3 \ directory.3 opendir.3 \ directory.3 readdir.3 \ directory.3 readdir_r.3 \ directory.3 rewinddir.3 \ directory.3 seekdir.3 \ directory.3 telldir.3 MLINKS+=dlopen.3 fdlopen.3 \ dlopen.3 dlclose.3 \ dlopen.3 dlerror.3 \ dlopen.3 dlfunc.3 \ dlopen.3 dlsym.3 MLINKS+=err.3 err_set_exit.3 \ err.3 err_set_file.3 \ err.3 errc.3 \ err.3 errx.3 \ err.3 verr.3 \ err.3 verrc.3 \ err.3 verrx.3 \ err.3 vwarn.3 \ err.3 vwarnc.3 \ err.3 vwarnx.3 \ err.3 warnc.3 \ err.3 warn.3 \ err.3 warnx.3 MLINKS+=exec.3 execl.3 \ exec.3 execle.3 \ exec.3 execlp.3 \ exec.3 exect.3 \ exec.3 execv.3 \ exec.3 execvP.3 \ exec.3 execvp.3 MLINKS+=fpclassify.3 finite.3 \ fpclassify.3 finitef.3 \ fpclassify.3 isfinite.3 \ fpclassify.3 isinf.3 \ fpclassify.3 isnan.3 \ fpclassify.3 isnormal.3 MLINKS+=frexp.3 frexpf.3 \ frexp.3 frexpl.3 MLINKS+=fts.3 fts_children.3 \ fts.3 fts_close.3 \ fts.3 fts_open.3 \ fts.3 fts_read.3 \ fts.3 fts_set.3 \ fts.3 fts_set_clientptr.3 \ fts.3 fts_get_clientptr.3 \ fts.3 fts_get_stream.3 MLINKS+=ftw.3 nftw.3 MLINKS+=getcap.3 cgetcap.3 \ getcap.3 cgetclose.3 \ getcap.3 cgetent.3 \ getcap.3 cgetfirst.3 \ getcap.3 cgetmatch.3 \ getcap.3 cgetnext.3 \ getcap.3 cgetnum.3 \ getcap.3 cgetset.3 \ getcap.3 cgetstr.3 \ getcap.3 cgetustr.3 MLINKS+=getcwd.3 getwd.3 MLINKS+=getcontext.3 getcontextx.3 MLINKS+=getcontext.3 setcontext.3 MLINKS+=getdomainname.3 setdomainname.3 MLINKS+=getfsent.3 endfsent.3 \ getfsent.3 getfsfile.3 \ getfsent.3 getfsspec.3 \ getfsent.3 getfstype.3 \ getfsent.3 setfsent.3 \ getfsent.3 setfstab.3 \ getfsent.3 getfstab.3 MLINKS+=getgrent.3 endgrent.3 \ getgrent.3 getgrgid.3 \ getgrent.3 getgrnam.3 \ getgrent.3 setgrent.3 \ getgrent.3 setgroupent.3 \ getgrent.3 getgrent_r.3 \ getgrent.3 getgrnam_r.3 \ getgrent.3 getgrgid_r.3 MLINKS+=gethostname.3 sethostname.3 MLINKS+=getnetgrent.3 endnetgrent.3 \ getnetgrent.3 innetgr.3 \ getnetgrent.3 setnetgrent.3 MLINKS+=getprogname.3 setprogname.3 MLINKS+=getpwent.3 endpwent.3 \ getpwent.3 getpwnam.3 \ getpwent.3 getpwuid.3 \ getpwent.3 setpassent.3 \ getpwent.3 setpwent.3 \ getpwent.3 setpwfile.3 \ getpwent.3 getpwent_r.3 \ getpwent.3 getpwnam_r.3 \ getpwent.3 getpwuid_r.3 MLINKS+=getttyent.3 endttyent.3 \ getttyent.3 getttynam.3 \ getttyent.3 isdialuptty.3 \ getttyent.3 isnettty.3 \ getttyent.3 setttyent.3 MLINKS+=getusershell.3 endusershell.3 \ getusershell.3 setusershell.3 MLINKS+=getutxent.3 endutxent.3 \ getutxent.3 getutxid.3 \ getutxent.3 getutxline.3 \ getutxent.3 getutxuser.3 \ getutxent.3 pututxline.3 \ getutxent.3 setutxdb.3 \ getutxent.3 setutxent.3 \ getutxent.3 utmpx.3 MLINKS+=glob.3 globfree.3 MLINKS+=isgreater.3 isgreaterequal.3 \ isgreater.3 isless.3 \ isgreater.3 islessequal.3 \ isgreater.3 islessgreater.3 \ isgreater.3 isunordered.3 MLINKS+=ldexp.3 ldexpf.3 \ ldexp.3 ldexpl.3 MLINKS+=makecontext.3 swapcontext.3 MLINKS+=modf.3 modff.3 \ modf.3 modfl.3 MLINKS+=popen.3 pclose.3 MLINKS+=posix_spawn.3 posix_spawnp.3 \ posix_spawn_file_actions_addopen.3 posix_spawn_file_actions_addclose.3 \ posix_spawn_file_actions_addopen.3 posix_spawn_file_actions_adddup2.3 \ posix_spawn_file_actions_init.3 posix_spawn_file_actions_destroy.3 \ posix_spawnattr_getflags.3 posix_spawnattr_setflags.3 \ posix_spawnattr_getpgroup.3 posix_spawnattr_setpgroup.3 \ posix_spawnattr_getschedparam.3 posix_spawnattr_setschedparam.3 \ posix_spawnattr_getschedpolicy.3 posix_spawnattr_setschedpolicy.3 \ posix_spawnattr_getsigdefault.3 posix_spawnattr_setsigdefault.3 \ posix_spawnattr_getsigmask.3 posix_spawnattr_setsigmask.3 \ posix_spawnattr_init.3 posix_spawnattr_destroy.3 MLINKS+=psignal.3 strsignal.3 \ psignal.3 sys_siglist.3 \ psignal.3 sys_signame.3 MLINKS+=pwcache.3 group_from_gid.3 \ pwcache.3 user_from_uid.3 MLINKS+=rand48.3 _rand48.3 \ rand48.3 drand48.3 \ rand48.3 erand48.3 \ rand48.3 jrand48.3 \ rand48.3 lcong48.3 \ rand48.3 lrand48.3 \ rand48.3 mrand48.3 \ rand48.3 nrand48.3 \ rand48.3 seed48.3 \ rand48.3 srand48.3 +MLINKS+=recv.2 recvmmsg.2 MLINKS+=scandir.3 alphasort.3 MLINKS+=sem_open.3 sem_close.3 \ sem_open.3 sem_unlink.3 MLINKS+=sem_wait.3 sem_trywait.3 +MLINKS+=send.2 sendmmsg.2 MLINKS+=setjmp.3 _longjmp.3 \ setjmp.3 _setjmp.3 \ setjmp.3 longjmp.3 \ setjmp.3 longjmperr.3 \ setjmp.3 longjmperror.3 \ setjmp.3 siglongjmp.3 \ setjmp.3 sigsetjmp.3 MLINKS+=setmode.3 getmode.3 MLINKS+=sigsetops.3 sigaddset.3 \ sigsetops.3 sigdelset.3 \ sigsetops.3 sigemptyset.3 \ sigsetops.3 sigfillset.3 \ sigsetops.3 sigismember.3 MLINKS+=statvfs.3 fstatvfs.3 MLINKS+=stringlist.3 sl_add.3 \ stringlist.3 sl_find.3 \ stringlist.3 sl_free.3 \ stringlist.3 sl_init.3 MLINKS+=strtofflags.3 fflagstostr.3 MLINKS+=sysctl.3 sysctlbyname.3 \ sysctl.3 sysctlnametomib.3 MLINKS+=syslog.3 closelog.3 \ syslog.3 openlog.3 \ syslog.3 setlogmask.3 \ syslog.3 vsyslog.3 MLINKS+=tcsendbreak.3 tcdrain.3 \ tcsendbreak.3 tcflow.3 \ tcsendbreak.3 tcflush.3 MLINKS+=tcsetattr.3 cfgetispeed.3 \ tcsetattr.3 cfgetospeed.3 \ tcsetattr.3 cfmakeraw.3 \ tcsetattr.3 cfmakesane.3 \ tcsetattr.3 cfsetispeed.3 \ tcsetattr.3 cfsetospeed.3 \ tcsetattr.3 cfsetspeed.3 \ tcsetattr.3 tcgetattr.3 MLINKS+=ttyname.3 isatty.3 \ ttyname.3 ttyname_r.3 MLINKS+=tzset.3 tzsetwall.3 MLINKS+=unvis.3 strunvis.3 \ unvis.3 strunvisx.3 MLINKS+=vis.3 nvis.3 \ vis.3 snvis.3 \ vis.3 strenvisx.3 \ vis.3 strnunvis.3 \ vis.3 strnunvisx.3 \ vis.3 strnvis.3 \ vis.3 strnvisx.3 \ vis.3 strsenvisx.3 \ vis.3 strsnvis.3 \ vis.3 strsnvisx.3 \ vis.3 strsvis.3 \ vis.3 strsvisx.3 \ vis.3 strvis.3 \ vis.3 strvisx.3 \ vis.3 svis.3 MLINKS+=wordexp.3 wordfree.3 Index: user/ngie/make_check/lib/libc/gen/recvmmsg.c =================================================================== --- user/ngie/make_check/lib/libc/gen/recvmmsg.c (nonexistent) +++ user/ngie/make_check/lib/libc/gen/recvmmsg.c (revision 295078) @@ -0,0 +1,96 @@ +/* + * Copyright (c) 2016 Boris Astardzhiev, Smartcom-Bulgaria AD + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice(s), this list of conditions and the following disclaimer as + * the first lines of this file unmodified other than the possible + * addition of one or more copyright notices. + * 2. Redistributions in binary form must reproduce the above copyright + * notice(s), this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, + * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE + * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, + * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include "libc_private.h" + +ssize_t +recvmmsg(int s, struct mmsghdr *__restrict msgvec, size_t vlen, int flags, + const struct timespec *__restrict timeout) +{ + struct pollfd pfd[1]; + size_t i, rcvd; + ssize_t ret; + int res; + short ev; + + if (timeout != NULL) { + pfd[0].fd = s; + pfd[0].revents = 0; + pfd[0].events = ev = POLLIN | POLLRDNORM | POLLRDBAND | + POLLPRI; + res = ppoll(&pfd[0], 1, timeout, NULL); + if (res == -1 || res == 0) + return (res); + if (pfd[0].revents & POLLNVAL) { + errno = EBADF; + return (-1); + } + if ((pfd[0].revents & ev) == 0) { + errno = ETIMEDOUT; + return (-1); + } + } + + ret = __sys_recvmsg(s, &msgvec[0].msg_hdr, flags); + if (ret == -1) + return (ret); + + /* + * Do non-blocking receive for second and later messages if + * WAITFORONE is set. + */ + if (flags & MSG_WAITFORONE) + flags |= MSG_DONTWAIT; + + rcvd = 1; + for (i = rcvd; i < vlen; i++, rcvd++) { + ret = __sys_recvmsg(s, &msgvec[i].msg_hdr, flags); + if (ret == -1) { + /* We have received messages. Let caller know + * about the data received, socket + * error is returned on next + * invocation. + */ + return (rcvd); + } + + /* Save received bytes. */ + msgvec[i].msg_len = ret; + } + + return (rcvd); +} Property changes on: user/ngie/make_check/lib/libc/gen/recvmmsg.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/lib/libc/gen/sendmmsg.c =================================================================== --- user/ngie/make_check/lib/libc/gen/sendmmsg.c (nonexistent) +++ user/ngie/make_check/lib/libc/gen/sendmmsg.c (revision 295078) @@ -0,0 +1,64 @@ +/* + * Copyright (c) 2016 Boris Astardzhiev, Smartcom-Bulgaria AD + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice(s), this list of conditions and the following disclaimer as + * the first lines of this file unmodified other than the possible + * addition of one or more copyright notices. + * 2. Redistributions in binary form must reproduce the above copyright + * notice(s), this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDER(S) ``AS IS'' AND ANY + * EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER(S) BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR + * BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, + * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE + * OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, + * EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include "libc_private.h" + +ssize_t +sendmmsg(int s, struct mmsghdr *__restrict msgvec, size_t vlen, int flags) +{ + size_t i, sent; + ssize_t ret; + + sent = 0; + for (i = 0; i < vlen; i++, sent++) { + ret = __sys_sendmsg(s, &msgvec[i].msg_hdr, flags); + if (ret == -1) { + if (sent != 0) { + /* + * We have sent messages. Let caller + * know about the data sent, socket + * error is returned on next + * invocation. + */ + return (sent); + } + return (ret); + } + + /* Save sent bytes. */ + msgvec[i].msg_len = ret; + } + + return (sent); +} Property changes on: user/ngie/make_check/lib/libc/gen/sendmmsg.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/lib/libc/include/namespace.h =================================================================== --- user/ngie/make_check/lib/libc/include/namespace.h (revision 295077) +++ user/ngie/make_check/lib/libc/include/namespace.h (revision 295078) @@ -1,266 +1,268 @@ /* * Copyright (c) 2001 Daniel Eischen . * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _NAMESPACE_H_ #define _NAMESPACE_H_ /* * Adjust names so that headers declare "hidden" names. * * README: When modifying this file don't forget to make the appropriate * changes in un-namespace.h!!! */ /* * ISO C (C90) section. Most names in libc aren't in ISO C, so they * should be here. Most aren't here... */ #define err _err #define warn _warn #define nsdispatch _nsdispatch /* * Prototypes for syscalls/functions that need to be overridden * in libc_r/libpthread. */ #define accept _accept #define __acl_aclcheck_fd ___acl_aclcheck_fd #define __acl_delete_fd ___acl_delete_fd #define __acl_get_fd ___acl_get_fd #define __acl_set_fd ___acl_set_fd #define bind _bind #define __cap_get_fd ___cap_get_fd #define __cap_set_fd ___cap_set_fd #define close _close #define connect _connect #define dup _dup #define dup2 _dup2 #define execve _execve #define fcntl _fcntl /*#define flock _flock */ #define flockfile _flockfile #define fpathconf _fpathconf #define fstat _fstat #define fstatfs _fstatfs #define fsync _fsync #define funlockfile _funlockfile #define getdirentries _getdirentries #define getlogin _getlogin #define getpeername _getpeername #define getprogname _getprogname #define getsockname _getsockname #define getsockopt _getsockopt #define ioctl _ioctl /* #define kevent _kevent */ #define listen _listen #define nanosleep _nanosleep #define open _open #define openat _openat #define poll _poll #define pthread_atfork _pthread_atfork #define pthread_attr_destroy _pthread_attr_destroy #define pthread_attr_get_np _pthread_attr_get_np #define pthread_attr_getaffinity_np _pthread_attr_getaffinity_np #define pthread_attr_getdetachstate _pthread_attr_getdetachstate #define pthread_attr_getguardsize _pthread_attr_getguardsize #define pthread_attr_getinheritsched _pthread_attr_getinheritsched #define pthread_attr_getschedparam _pthread_attr_getschedparam #define pthread_attr_getschedpolicy _pthread_attr_getschedpolicy #define pthread_attr_getscope _pthread_attr_getscope #define pthread_attr_getstack _pthread_attr_getstack #define pthread_attr_getstackaddr _pthread_attr_getstackaddr #define pthread_attr_getstacksize _pthread_attr_getstacksize #define pthread_attr_init _pthread_attr_init #define pthread_attr_setaffinity_np _pthread_attr_setaffinity_np #define pthread_attr_setcreatesuspend_np _pthread_attr_setcreatesuspend_np #define pthread_attr_setdetachstate _pthread_attr_setdetachstate #define pthread_attr_setguardsize _pthread_attr_setguardsize #define pthread_attr_setinheritsched _pthread_attr_setinheritsched #define pthread_attr_setschedparam _pthread_attr_setschedparam #define pthread_attr_setschedpolicy _pthread_attr_setschedpolicy #define pthread_attr_setscope _pthread_attr_setscope #define pthread_attr_setstack _pthread_attr_setstack #define pthread_attr_setstackaddr _pthread_attr_setstackaddr #define pthread_attr_setstacksize _pthread_attr_setstacksize #define pthread_barrier_destroy _pthread_barrier_destroy #define pthread_barrier_init _pthread_barrier_init #define pthread_barrier_wait _pthread_barrier_wait #define pthread_barrierattr_destroy _pthread_barrierattr_destroy #define pthread_barrierattr_getpshared _pthread_barrierattr_getpshared #define pthread_barrierattr_init _pthread_barrierattr_init #define pthread_barrierattr_setpshared _pthread_barrierattr_setpshared #define pthread_cancel _pthread_cancel #define pthread_cond_broadcast _pthread_cond_broadcast #define pthread_cond_destroy _pthread_cond_destroy #define pthread_cond_init _pthread_cond_init #define pthread_cond_signal _pthread_cond_signal #define pthread_cond_timedwait _pthread_cond_timedwait #define pthread_cond_wait _pthread_cond_wait #define pthread_condattr_destroy _pthread_condattr_destroy #define pthread_condattr_getclock _pthread_condattr_getclock #define pthread_condattr_getpshared _pthread_condattr_getpshared #define pthread_condattr_init _pthread_condattr_init #define pthread_condattr_setclock _pthread_condattr_setclock #define pthread_condattr_setpshared _pthread_condattr_setpshared #define pthread_create _pthread_create #define pthread_detach _pthread_detach #define pthread_equal _pthread_equal #define pthread_exit _pthread_exit #define pthread_getaffinity_np _pthread_getaffinity_np #define pthread_getconcurrency _pthread_getconcurrency #define pthread_getcpuclockid _pthread_getcpuclockid #define pthread_getprio _pthread_getprio #define pthread_getschedparam _pthread_getschedparam #define pthread_getspecific _pthread_getspecific #define pthread_getthreadid_np _pthread_getthreadid_np #define pthread_join _pthread_join #define pthread_key_create _pthread_key_create #define pthread_key_delete _pthread_key_delete #define pthread_kill _pthread_kill #define pthread_main_np _pthread_main_np #define pthread_multi_np _pthread_multi_np #define pthread_mutex_destroy _pthread_mutex_destroy #define pthread_mutex_getprioceiling _pthread_mutex_getprioceiling #define pthread_mutex_init _pthread_mutex_init #define pthread_mutex_isowned_np _pthread_mutex_isowned_np #define pthread_mutex_lock _pthread_mutex_lock #define pthread_mutex_setprioceiling _pthread_mutex_setprioceiling #define pthread_mutex_timedlock _pthread_mutex_timedlock #define pthread_mutex_trylock _pthread_mutex_trylock #define pthread_mutex_unlock _pthread_mutex_unlock #define pthread_mutexattr_destroy _pthread_mutexattr_destroy #define pthread_mutexattr_getkind_np _pthread_mutexattr_getkind_np #define pthread_mutexattr_getprioceiling _pthread_mutexattr_getprioceiling #define pthread_mutexattr_getprotocol _pthread_mutexattr_getprotocol #define pthread_mutexattr_getpshared _pthread_mutexattr_getpshared #define pthread_mutexattr_gettype _pthread_mutexattr_gettype #define pthread_mutexattr_init _pthread_mutexattr_init #define pthread_mutexattr_setkind_np _pthread_mutexattr_setkind_np #define pthread_mutexattr_setprioceiling _pthread_mutexattr_setprioceiling #define pthread_mutexattr_setprotocol _pthread_mutexattr_setprotocol #define pthread_mutexattr_setpshared _pthread_mutexattr_setpshared #define pthread_mutexattr_settype _pthread_mutexattr_settype #define pthread_once _pthread_once #define pthread_resume_all_np _pthread_resume_all_np #define pthread_resume_np _pthread_resume_np #define pthread_rwlock_destroy _pthread_rwlock_destroy #define pthread_rwlock_init _pthread_rwlock_init #define pthread_rwlock_rdlock _pthread_rwlock_rdlock #define pthread_rwlock_timedrdlock _pthread_rwlock_timedrdlock #define pthread_rwlock_timedwrlock _pthread_rwlock_timedwrlock #define pthread_rwlock_tryrdlock _pthread_rwlock_tryrdlock #define pthread_rwlock_trywrlock _pthread_rwlock_trywrlock #define pthread_rwlock_unlock _pthread_rwlock_unlock #define pthread_rwlock_wrlock _pthread_rwlock_wrlock #define pthread_rwlockattr_destroy _pthread_rwlockattr_destroy #define pthread_rwlockattr_getpshared _pthread_rwlockattr_getpshared #define pthread_rwlockattr_init _pthread_rwlockattr_init #define pthread_rwlockattr_setpshared _pthread_rwlockattr_setpshared #define pthread_self _pthread_self #define pthread_set_name_np _pthread_set_name_np #define pthread_setaffinity_np _pthread_setaffinity_np #define pthread_setcancelstate _pthread_setcancelstate #define pthread_setcanceltype _pthread_setcanceltype #define pthread_setconcurrency _pthread_setconcurrency #define pthread_setprio _pthread_setprio #define pthread_setschedparam _pthread_setschedparam #define pthread_setspecific _pthread_setspecific #define pthread_sigmask _pthread_sigmask #define pthread_single_np _pthread_single_np #define pthread_spin_destroy _pthread_spin_destroy #define pthread_spin_init _pthread_spin_init #define pthread_spin_lock _pthread_spin_lock #define pthread_spin_trylock _pthread_spin_trylock #define pthread_spin_unlock _pthread_spin_unlock #define pthread_suspend_all_np _pthread_suspend_all_np #define pthread_suspend_np _pthread_suspend_np #define pthread_switch_add_np _pthread_switch_add_np #define pthread_switch_delete_np _pthread_switch_delete_np #define pthread_testcancel _pthread_testcancel #define pthread_timedjoin_np _pthread_timedjoin_np #define pthread_yield _pthread_yield #define read _read #define readv _readv #define recvfrom _recvfrom #define recvmsg _recvmsg +#define recvmmsg _recvmmsg #define select _select #define sem_close _sem_close #define sem_destroy _sem_destroy #define sem_getvalue _sem_getvalue #define sem_init _sem_init #define sem_open _sem_open #define sem_post _sem_post #define sem_timedwait _sem_timedwait #define sem_trywait _sem_trywait #define sem_unlink _sem_unlink #define sem_wait _sem_wait #define sendmsg _sendmsg +#define sendmmsg _sendmmsg #define sendto _sendto #define setsockopt _setsockopt /*#define sigaction _sigaction*/ #define sigprocmask _sigprocmask #define sigsuspend _sigsuspend #define socket _socket #define socketpair _socketpair #define usleep _usleep #define wait4 _wait4 #define wait6 _wait6 #define waitpid _waitpid #define write _write #define writev _writev /* * Other hidden syscalls/functions that libc_r needs to override * but are not used internally by libc. * * XXX - When modifying libc to use one of the following, remove * the prototype from below and place it in the list above. */ #if 0 #define creat _creat #define fchflags _fchflags #define fchmod _fchmod #define ftrylockfile _ftrylockfile #define msync _msync #define nfssvc _nfssvc #define pause _pause #define sched_yield _sched_yield #define sendfile _sendfile #define shutdown _shutdown #define sigaltstack _sigaltstack #define sigpending _sigpending #define sigreturn _sigreturn #define sigsetmask _sigsetmask #define sleep _sleep #define system _system #define tcdrain _tcdrain #define wait _wait #endif #endif /* _NAMESPACE_H_ */ Index: user/ngie/make_check/lib/libc/include/un-namespace.h =================================================================== --- user/ngie/make_check/lib/libc/include/un-namespace.h (revision 295077) +++ user/ngie/make_check/lib/libc/include/un-namespace.h (revision 295078) @@ -1,256 +1,258 @@ /* * Copyright (c) 2001 Daniel Eischen . * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _UN_NAMESPACE_H_ #define _UN_NAMESPACE_H_ #undef accept #undef __acl_aclcheck_fd #undef __acl_delete_fd #undef __acl_get_fd #undef __acl_set_fd #undef bind #undef __cap_get_fd #undef __cap_set_fd #undef close #undef connect #undef dup #undef dup2 #undef execve #undef fcntl #undef flock #undef flockfile #undef fpathconf #undef fstat #undef fstatfs #undef fsync #undef funlockfile #undef getdirentries #undef getlogin #undef getpeername #undef getprogname #undef getsockname #undef getsockopt #undef ioctl #undef kevent #undef listen #undef nanosleep #undef open #undef openat #undef poll #undef pthread_atfork #undef pthread_attr_destroy #undef pthread_attr_get_np #undef pthread_attr_getaffinity_np #undef pthread_attr_getdetachstate #undef pthread_attr_getguardsize #undef pthread_attr_getinheritsched #undef pthread_attr_getschedparam #undef pthread_attr_getschedpolicy #undef pthread_attr_getscope #undef pthread_attr_getstack #undef pthread_attr_getstackaddr #undef pthread_attr_getstacksize #undef pthread_attr_init #undef pthread_attr_setaffinity_np #undef pthread_attr_setcreatesuspend_np #undef pthread_attr_setdetachstate #undef pthread_attr_setguardsize #undef pthread_attr_setinheritsched #undef pthread_attr_setschedparam #undef pthread_attr_setschedpolicy #undef pthread_attr_setscope #undef pthread_attr_setstack #undef pthread_attr_setstackaddr #undef pthread_attr_setstacksize #undef pthread_barrier_destroy #undef pthread_barrier_init #undef pthread_barrier_wait #undef pthread_barrierattr_destroy #undef pthread_barrierattr_getpshared #undef pthread_barrierattr_init #undef pthread_barrierattr_setpshared #undef pthread_cancel #undef pthread_cond_broadcast #undef pthread_cond_destroy #undef pthread_cond_init #undef pthread_cond_signal #undef pthread_cond_timedwait #undef pthread_cond_wait #undef pthread_condattr_destroy #undef pthread_condattr_getclock #undef pthread_condattr_getpshared #undef pthread_condattr_init #undef pthread_condattr_setclock #undef pthread_condattr_setpshared #undef pthread_create #undef pthread_detach #undef pthread_equal #undef pthread_exit #undef pthread_getaffinity_np #undef pthread_getconcurrency #undef pthread_getcpuclockid #undef pthread_getprio #undef pthread_getschedparam #undef pthread_getspecific #undef pthread_getthreadid_np #undef pthread_join #undef pthread_key_create #undef pthread_key_delete #undef pthread_kill #undef pthread_main_np #undef pthread_multi_np #undef pthread_mutex_destroy #undef pthread_mutex_getprioceiling #undef pthread_mutex_init #undef pthread_mutex_isowned_np #undef pthread_mutex_lock #undef pthread_mutex_setprioceiling #undef pthread_mutex_timedlock #undef pthread_mutex_trylock #undef pthread_mutex_unlock #undef pthread_mutexattr_destroy #undef pthread_mutexattr_getkind_np #undef pthread_mutexattr_getprioceiling #undef pthread_mutexattr_getprotocol #undef pthread_mutexattr_getpshared #undef pthread_mutexattr_gettype #undef pthread_mutexattr_init #undef pthread_mutexattr_setkind_np #undef pthread_mutexattr_setprioceiling #undef pthread_mutexattr_setprotocol #undef pthread_mutexattr_setpshared #undef pthread_mutexattr_settype #undef pthread_once #undef pthread_resume_all_np #undef pthread_resume_np #undef pthread_rwlock_destroy #undef pthread_rwlock_init #undef pthread_rwlock_rdlock #undef pthread_rwlock_timedrdlock #undef pthread_rwlock_timedwrlock #undef pthread_rwlock_tryrdlock #undef pthread_rwlock_trywrlock #undef pthread_rwlock_unlock #undef pthread_rwlock_wrlock #undef pthread_rwlockattr_destroy #undef pthread_rwlockattr_getpshared #undef pthread_rwlockattr_init #undef pthread_rwlockattr_setpshared #undef pthread_self #undef pthread_set_name_np #undef pthread_setaffinity_np #undef pthread_setcancelstate #undef pthread_setcanceltype #undef pthread_setconcurrency #undef pthread_setprio #undef pthread_setschedparam #undef pthread_setspecific #undef pthread_sigmask #undef pthread_single_np #undef pthread_spin_destroy #undef pthread_spin_init #undef pthread_spin_lock #undef pthread_spin_trylock #undef pthread_spin_unlock #undef pthread_suspend_all_np #undef pthread_suspend_np #undef pthread_switch_add_np #undef pthread_switch_delete_np #undef pthread_testcancel #undef pthread_timedjoin_np #undef pthread_yield #undef read #undef readv #undef recvfrom #undef recvmsg +#undef recvmmsg #undef select #undef sem_close #undef sem_destroy #undef sem_getvalue #undef sem_init #undef sem_open #undef sem_post #undef sem_timedwait #undef sem_trywait #undef sem_unlink #undef sem_wait #undef sendmsg +#undef sendmmsg #undef sendto #undef setsockopt #undef sigaction #undef sigprocmask #undef sigsuspend #undef socket #undef socketpair #undef usleep #undef wait4 #undef wait6 #undef waitpid #undef write #undef writev #if 0 #undef creat #undef fchflags #undef fchmod #undef ftrylockfile #undef msync #undef nfssvc #undef pause #undef sched_yield #undef sendfile #undef shutdown #undef sigaltstack #undef sigpending #undef sigreturn #undef sigsetmask #undef sleep #undef system #undef tcdrain #undef wait #endif /* 0 */ #ifdef _SIGNAL_H_ int _sigaction(int, const struct sigaction *, struct sigaction *); #endif #ifdef _SYS_EVENT_H_ int _kevent(int, const struct kevent *, int, struct kevent *, int, const struct timespec *); #endif #ifdef _SYS_FCNTL_H_ int _flock(int, int); #endif #undef err #undef warn #undef nsdispatch #endif /* _UN_NAMESPACE_H_ */ Index: user/ngie/make_check/lib/libc/sys/Symbol.map =================================================================== --- user/ngie/make_check/lib/libc/sys/Symbol.map (revision 295077) +++ user/ngie/make_check/lib/libc/sys/Symbol.map (revision 295078) @@ -1,1054 +1,1056 @@ /* * $FreeBSD$ */ /* * It'd be nice to automatically generate the syscall symbols, but we * don't know to what version they will eventually belong to, so for now * it has to be manual. */ FBSD_1.0 { __acl_aclcheck_fd; __acl_aclcheck_file; __acl_aclcheck_link; __acl_delete_fd; __acl_delete_file; __acl_delete_link; __acl_get_fd; __acl_get_file; __acl_get_link; __acl_set_fd; __acl_set_file; __acl_set_link; __getcwd; __mac_execve; __mac_get_fd; __mac_get_file; __mac_get_link; __mac_get_pid; __mac_get_proc; __mac_set_fd; __mac_set_file; __mac_set_link; __mac_set_proc; __setugid; __syscall; __sysctl; _umtx_lock; _umtx_op; _umtx_unlock; abort2; accept; access; acct; adjtime; aio_cancel; aio_error; aio_fsync; aio_read; aio_return; aio_suspend; aio_waitcomplete; aio_write; audit; auditctl; auditon; bind; chdir; chflags; chmod; chown; chroot; clock_getres; clock_gettime; clock_settime; close; connect; dup; dup2; eaccess; execve; extattr_delete_fd; extattr_delete_file; extattr_delete_link; extattr_get_fd; extattr_get_file; extattr_get_link; extattr_list_fd; extattr_list_file; extattr_list_link; extattr_set_fd; extattr_set_file; extattr_set_link; extattrctl; fchdir; fchflags; fchmod; fchown; fcntl; fhopen; fhstat; fhstatfs; flock; fork; fpathconf; fstat; fstatfs; fsync; futimes; getaudit; getaudit_addr; getauid; getcontext; getdents; getdirentries; getdtablesize; getegid; geteuid; getfh; getfsstat; getgid; getgroups; getitimer; getpeername; getpgid; getpgrp; getpid; getppid; getpriority; getresgid; getresuid; getrlimit; getrusage; getsid; getsockname; getsockopt; gettimeofday; getuid; ioctl; issetugid; jail; jail_attach; kenv; kevent; kill; kldfind; kldfirstmod; kldload; kldnext; kldstat; kldsym; kldunload; kldunloadf; kqueue; kmq_notify; /* Do we want these to be public interfaces? */ kmq_open; /* librt uses them to provide mq_xxx. */ kmq_setattr; kmq_timedreceive; kmq_timedsend; kmq_unlink; ksem_close; ksem_destroy; ksem_getvalue; ksem_init; ksem_open; ksem_post; ksem_timedwait; ksem_trywait; ksem_unlink; ksem_wait; ktrace; lchflags; lchmod; lchown; lgetfh; link; lio_listio; listen; lstat; lutimes; mac_syscall; madvise; mincore; minherit; mkdir; mkfifo; mknod; mlock; mlockall; modfind; modfnext; modnext; modstat; mount; mprotect; msgget; msgrcv; msgsnd; msgsys; msync; munlock; munlockall; munmap; nanosleep; netbsd_lchown; netbsd_msync; nfssvc; nfstat; nlstat; nmount; nstat; ntp_adjtime; ntp_gettime; oaio_read; oaio_write; olio_listio; open; pathconf; pipe; poll; posix_openpt; preadv; profil; pselect; ptrace; pwritev; quotactl; read; readlink; readv; reboot; recvfrom; recvmsg; rename; revoke; rfork; rmdir; rtprio; rtprio_thread; sched_get_priority_max; sched_get_priority_min; sched_getparam; sched_getscheduler; sched_rr_get_interval; sched_setparam; sched_setscheduler; sched_yield; select; semget; semop; semsys; sendfile; sendmsg; sendto; setaudit; setaudit_addr; setauid; setegid; seteuid; setgid; setgroups; setitimer; setlogin; setpgid; setpriority; setregid; setresgid; setresuid; setreuid; setrlimit; setsid; setsockopt; settimeofday; setuid; shm_open; shm_unlink; shmat; shmdt; shmget; shmsys; shutdown; sigaction; sigaltstack; sigpending; sigprocmask; sigqueue; sigreturn; sigsuspend; sigtimedwait; sigwait; sigwaitinfo; socket; socketpair; stat; statfs; swapoff; swapon; symlink; sync; sysarch; syscall; thr_create; thr_exit; thr_kill; thr_kill2; thr_new; thr_self; thr_set_name; thr_suspend; thr_wake; ktimer_create; /* Do we want these to be public interfaces? */ ktimer_delete; /* librt uses them to provide timer_xxx. */ ktimer_getoverrun; ktimer_gettime; ktimer_settime; umask; undelete; unlink; unmount; utimes; utrace; uuidgen; vadvise; wait4; write; writev; __error; ftruncate; lseek; mmap; pread; pwrite; truncate; }; FBSD_1.1 { __semctl; closefrom; cpuset; cpuset_getid; cpuset_setid; cpuset_getaffinity; cpuset_setaffinity; faccessat; fchmodat; fchownat; fexecve; fstatat; futimesat; jail_get; jail_set; jail_remove; linkat; lpathconf; mkdirat; mkfifoat; mknodat; msgctl; readlinkat; renameat; setfib; shmctl; symlinkat; unlinkat; }; FBSD_1.2 { cap_enter; cap_getmode; getloginclass; pdfork; pdgetpid; pdkill; posix_fallocate; rctl_get_racct; rctl_get_rules; rctl_get_limits; rctl_add_rule; rctl_remove_rule; setloginclass; }; FBSD_1.3 { accept4; aio_mlock; bindat; cap_fcntls_get; cap_fcntls_limit; cap_ioctls_get; cap_ioctls_limit; __cap_rights_get; cap_rights_limit; cap_sandboxed; chflagsat; clock_getcpuclockid2; connectat; ffclock_getcounter; ffclock_getestimate; ffclock_setestimate; pipe2; posix_fadvise; procctl; wait6; }; FBSD_1.4 { futimens; ppoll; utimensat; numa_setaffinity; numa_getaffinity; + sendmmsg; + recvmmsg; }; FBSDprivate_1.0 { ___acl_aclcheck_fd; __sys___acl_aclcheck_fd; ___acl_aclcheck_file; __sys___acl_aclcheck_file; ___acl_aclcheck_link; __sys___acl_aclcheck_link; ___acl_delete_fd; __sys___acl_delete_fd; ___acl_delete_file; __sys___acl_delete_file; ___acl_delete_link; __sys___acl_delete_link; ___acl_get_fd; __sys___acl_get_fd; ___acl_get_file; __sys___acl_get_file; ___acl_get_link; __sys___acl_get_link; ___acl_set_fd; __sys___acl_set_fd; ___acl_set_file; __sys___acl_set_file; ___acl_set_link; __sys___acl_set_link; ___getcwd; __sys___getcwd; ___mac_execve; __sys___mac_execve; ___mac_get_fd; __sys___mac_get_fd; ___mac_get_file; __sys___mac_get_file; ___mac_get_link; __sys___mac_get_link; ___mac_get_pid; __sys___mac_get_pid; ___mac_get_proc; __sys___mac_get_proc; ___mac_set_fd; __sys___mac_set_fd; ___mac_set_file; __sys___mac_set_file; ___mac_set_link; __sys___mac_set_link; ___mac_set_proc; __sys___mac_set_proc; ___semctl; __sys___semctl; ___setugid; __sys___setugid; ___syscall; __sys___syscall; ___sysctl; __sys___sysctl; __umtx_lock; __sys__umtx_lock; __umtx_op; __sys__umtx_op; __umtx_unlock; __sys__umtx_unlock; _abort2; __sys_abort2; _accept; __sys_accept; _accept4; __sys_accept4; _access; __sys_access; _acct; __sys_acct; _adjtime; __sys_adjtime; _aio_cancel; __sys_aio_cancel; _aio_error; __sys_aio_error; _aio_fsync; __sys_aio_fsync; _aio_read; __sys_aio_read; _aio_return; __sys_aio_return; _aio_suspend; __sys_aio_suspend; _aio_waitcomplete; __sys_aio_waitcomplete; _aio_write; __sys_aio_write; _audit; __sys_audit; _auditctl; __sys_auditctl; _auditon; __sys_auditon; _bind; __sys_bind; _chdir; __sys_chdir; _chflags; __sys_chflags; _chmod; __sys_chmod; _chown; __sys_chown; _chroot; __sys_chroot; _clock_getcpuclockid2; __sys_clock_getcpuclockid2; _clock_getres; __sys_clock_getres; _clock_gettime; __sys_clock_gettime; _clock_settime; __sys_clock_settime; _close; __sys_close; _closefrom; __sys_closefrom; _connect; __sys_connect; _cpuset; __sys_cpuset; _cpuset_getid; __sys_cpuset_getid; _cpuset_setid; __sys_cpuset_setid; _cpuset_getaffinity; __sys_cpuset_getaffinity; _cpuset_setaffinity; __sys_cpuset_setaffinity; _dup; __sys_dup; _dup2; __sys_dup2; _eaccess; __sys_eaccess; _execve; __sys_execve; _extattr_delete_fd; __sys_extattr_delete_fd; _extattr_delete_file; __sys_extattr_delete_file; _extattr_delete_link; __sys_extattr_delete_link; _extattr_get_fd; __sys_extattr_get_fd; _extattr_get_file; __sys_extattr_get_file; _extattr_get_link; __sys_extattr_get_link; _extattr_list_fd; __sys_extattr_list_fd; _extattr_list_file; __sys_extattr_list_file; _extattr_list_link; __sys_extattr_list_link; _extattr_set_fd; __sys_extattr_set_fd; _extattr_set_file; __sys_extattr_set_file; _extattr_set_link; __sys_extattr_set_link; _extattrctl; __sys_extattrctl; _fchdir; __sys_fchdir; _fchflags; __sys_fchflags; _fchmod; __sys_fchmod; _fchown; __sys_fchown; _fcntl; __sys_fcntl; __fcntl_compat; _fhopen; __sys_fhopen; _fhstat; __sys_fhstat; _fhstatfs; __sys_fhstatfs; _flock; __sys_flock; _fork; __sys_fork; _fpathconf; __sys_fpathconf; _fstat; __sys_fstat; _fstatfs; __sys_fstatfs; _fsync; __sys_fsync; _futimes; __sys_futimes; _getaudit; __sys_getaudit; _getaudit_addr; __sys_getaudit_addr; _getauid; __sys_getauid; _getcontext; __sys_getcontext; _getdents; __sys_getdents; _getdirentries; __sys_getdirentries; _getdtablesize; __sys_getdtablesize; _getegid; __sys_getegid; _geteuid; __sys_geteuid; _getfh; __sys_getfh; _getfsstat; __sys_getfsstat; _getgid; __sys_getgid; _getgroups; __sys_getgroups; _getitimer; __sys_getitimer; _getpeername; __sys_getpeername; _getpgid; __sys_getpgid; _getpgrp; __sys_getpgrp; _getpid; __sys_getpid; _getppid; __sys_getppid; _getpriority; __sys_getpriority; _getresgid; __sys_getresgid; _getresuid; __sys_getresuid; _getrlimit; __sys_getrlimit; _getrusage; __sys_getrusage; _getsid; __sys_getsid; _getsockname; __sys_getsockname; _getsockopt; __sys_getsockopt; _gettimeofday; __sys_gettimeofday; _getuid; __sys_getuid; _ioctl; __sys_ioctl; _issetugid; __sys_issetugid; _jail; __sys_jail; _jail_attach; __sys_jail_attach; _kenv; __sys_kenv; _kevent; __sys_kevent; _kill; __sys_kill; _kldfind; __sys_kldfind; _kldfirstmod; __sys_kldfirstmod; _kldload; __sys_kldload; _kldnext; __sys_kldnext; _kldstat; __sys_kldstat; _kldsym; __sys_kldsym; _kldunload; __sys_kldunload; _kldunloadf; __sys_kldunloadf; _kmq_notify; __sys_kmq_notify; _kmq_open; __sys_kmq_open; _kmq_setattr; __sys_kmq_setattr; _kmq_timedreceive; __sys_kmq_timedreceive; _kmq_timedsend; __sys_kmq_timedsend; _kmq_unlink; __sys_kmq_unlink; _kqueue; __sys_kqueue; _ksem_close; __sys_ksem_close; _ksem_destroy; __sys_ksem_destroy; _ksem_getvalue; __sys_ksem_getvalue; _ksem_init; __sys_ksem_init; _ksem_open; __sys_ksem_open; _ksem_post; __sys_ksem_post; _ksem_timedwait; __sys_ksem_timedwait; _ksem_trywait; __sys_ksem_trywait; _ksem_unlink; __sys_ksem_unlink; _ksem_wait; __sys_ksem_wait; _ktrace; __sys_ktrace; _lchflags; __sys_lchflags; _lchmod; __sys_lchmod; _lchown; __sys_lchown; _lgetfh; __sys_lgetfh; _link; __sys_link; _lio_listio; __sys_lio_listio; _listen; __sys_listen; _lstat; __sys_lstat; _lutimes; __sys_lutimes; _mac_syscall; __sys_mac_syscall; _madvise; __sys_madvise; _mincore; __sys_mincore; _minherit; __sys_minherit; _mkdir; __sys_mkdir; _mkfifo; __sys_mkfifo; _mknod; __sys_mknod; _mlock; __sys_mlock; _mlockall; __sys_mlockall; _modfind; __sys_modfind; _modfnext; __sys_modfnext; _modnext; __sys_modnext; _modstat; __sys_modstat; _mount; __sys_mount; _mprotect; __sys_mprotect; _msgctl; __sys_msgctl; _msgget; __sys_msgget; _msgrcv; __sys_msgrcv; _msgsnd; __sys_msgsnd; _msgsys; __sys_msgsys; _msync; __sys_msync; _munlock; __sys_munlock; _munlockall; __sys_munlockall; _munmap; __sys_munmap; _nanosleep; __sys_nanosleep; _netbsd_lchown; __sys_netbsd_lchown; _netbsd_msync; __sys_netbsd_msync; _nfssvc; __sys_nfssvc; _nfstat; __sys_nfstat; _nlstat; __sys_nlstat; _nmount; __sys_nmount; _nstat; __sys_nstat; _ntp_adjtime; __sys_ntp_adjtime; _ntp_gettime; __sys_ntp_gettime; _oaio_read; __sys_oaio_read; _oaio_write; __sys_oaio_write; _olio_listio; __sys_olio_listio; _open; __sys_open; _openat; __sys_openat; _pathconf; __sys_pathconf; _pipe; __sys_pipe; _poll; __sys_poll; _ppoll; __sys_ppoll; _preadv; __sys_preadv; _procctl; __sys_procctl; _profil; __sys_profil; _pselect; __sys_pselect; _ptrace; __sys_ptrace; _pwritev; __sys_pwritev; _quotactl; __sys_quotactl; _read; __sys_read; _readlink; __sys_readlink; _readv; __sys_readv; _reboot; __sys_reboot; _recvfrom; __sys_recvfrom; _recvmsg; __sys_recvmsg; _rename; __sys_rename; _revoke; __sys_revoke; _rfork; __sys_rfork; _rmdir; __sys_rmdir; _rtprio; __sys_rtprio; _rtprio_thread; __sys_rtprio_thread; _sched_get_priority_max; __sys_sched_get_priority_max; _sched_get_priority_min; __sys_sched_get_priority_min; _sched_getparam; __sys_sched_getparam; _sched_getscheduler; __sys_sched_getscheduler; _sched_rr_get_interval; __sys_sched_rr_get_interval; _sched_setparam; __sys_sched_setparam; _sched_setscheduler; __sys_sched_setscheduler; _sched_yield; __sys_sched_yield; _select; __sys_select; _semget; __sys_semget; _semop; __sys_semop; _semsys; __sys_semsys; _sendfile; __sys_sendfile; _sendmsg; __sys_sendmsg; _sendto; __sys_sendto; _setaudit; __sys_setaudit; _setaudit_addr; __sys_setaudit_addr; _setauid; __sys_setauid; _setcontext; __sys_setcontext; _setegid; __sys_setegid; _seteuid; __sys_seteuid; _setgid; __sys_setgid; _setgroups; __sys_setgroups; _setitimer; __sys_setitimer; _setlogin; __sys_setlogin; _setpgid; __sys_setpgid; _setpriority; __sys_setpriority; _setregid; __sys_setregid; _setresgid; __sys_setresgid; _setresuid; __sys_setresuid; _setreuid; __sys_setreuid; _setrlimit; __sys_setrlimit; _setsid; __sys_setsid; _setsockopt; __sys_setsockopt; _settimeofday; __sys_settimeofday; _setuid; __sys_setuid; _shm_open; __sys_shm_open; _shm_unlink; __sys_shm_unlink; _shmat; __sys_shmat; _shmctl; __sys_shmctl; _shmdt; __sys_shmdt; _shmget; __sys_shmget; _shmsys; __sys_shmsys; _shutdown; __sys_shutdown; _sigaction; __sys_sigaction; _sigaltstack; __sys_sigaltstack; _sigpending; __sys_sigpending; _sigprocmask; __sys_sigprocmask; _sigqueue; __sys_sigqueue; _sigreturn; __sys_sigreturn; _sigsuspend; __sys_sigsuspend; _sigtimedwait; __sys_sigtimedwait; _sigwait; __sigwait; __sys_sigwait; _sigwaitinfo; __sys_sigwaitinfo; _socket; __sys_socket; _socketpair; __sys_socketpair; _stat; __sys_stat; _statfs; __sys_statfs; _swapcontext; __sys_swapcontext; _swapoff; __sys_swapoff; _swapon; __sys_swapon; _symlink; __sys_symlink; _sync; __sys_sync; _sysarch; __sys_sysarch; _syscall; __sys_syscall; _thr_create; __sys_thr_create; _thr_exit; __sys_thr_exit; _thr_kill; __sys_thr_kill; _thr_kill2; __sys_thr_kill2; _thr_new; __sys_thr_new; _thr_self; __sys_thr_self; _thr_set_name; __sys_thr_set_name; _thr_suspend; __sys_thr_suspend; _thr_wake; __sys_thr_wake; _ktimer_create; __sys_ktimer_create; _ktimer_delete; __sys_ktimer_delete; _ktimer_getoverrun; __sys_ktimer_getoverrun; _ktimer_gettime; __sys_ktimer_gettime; _ktimer_settime; __sys_ktimer_settime; _umask; __sys_umask; _undelete; __sys_undelete; _unlink; __sys_unlink; _unmount; __sys_unmount; _utimes; __sys_utimes; _utrace; __sys_utrace; _uuidgen; __sys_uuidgen; _vadvise; __sys_vadvise; _wait4; __sys_wait4; _wait6; __sys_wait6; _write; __sys_write; _writev; __sys_writev; __set_error_selector; nlm_syscall; gssd_syscall; __libc_interposing_slot; __libc_sigwait; }; Index: user/ngie/make_check/lib/libc/sys/recv.2 =================================================================== --- user/ngie/make_check/lib/libc/sys/recv.2 (revision 295077) +++ user/ngie/make_check/lib/libc/sys/recv.2 (revision 295078) @@ -1,349 +1,424 @@ .\" Copyright (c) 1983, 1990, 1991, 1993 .\" The Regents of the University of California. All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" 4. Neither the name of the University nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" @(#)recv.2 8.3 (Berkeley) 2/21/94 .\" $FreeBSD$ .\" -.Dd October 15, 2014 +.Dd January 29, 2016 .Dt RECV 2 .Os .Sh NAME .Nm recv , .Nm recvfrom , -.Nm recvmsg -.Nd receive a message from a socket +.Nm recvmsg , +.Nm recvmmsg +.Nd receive message(s) from a socket .Sh LIBRARY .Lb libc .Sh SYNOPSIS .In sys/types.h .In sys/socket.h .Ft ssize_t .Fn recv "int s" "void *buf" "size_t len" "int flags" .Ft ssize_t .Fn recvfrom "int s" "void *buf" "size_t len" "int flags" "struct sockaddr * restrict from" "socklen_t * restrict fromlen" .Ft ssize_t .Fn recvmsg "int s" "struct msghdr *msg" "int flags" +.Ft ssize_t +.Fn recvmmsg "int s" "struct mmsghdr * restrict msgvec" "size_t vlen" "int flags" "const struct timespec * restrict timeout" .Sh DESCRIPTION The -.Fn recvfrom +.Fn recvfrom , +.Fn recvmsg , and -.Fn recvmsg +.Fn recvmmsg system calls are used to receive messages from a socket, and may be used to receive data on a socket whether or not it is connection-oriented. .Pp If .Fa from is not a null pointer and the socket is not connection-oriented, the source address of the message is filled in. The .Fa fromlen argument is a value-result argument, initialized to the size of the buffer associated with .Fa from , and modified on return to indicate the actual size of the address stored there. .Pp The .Fn recv function is normally used only on a .Em connected socket (see .Xr connect 2 ) and is identical to .Fn recvfrom with a null pointer passed as its .Fa from argument. .Pp -All three routines return the length of the message on successful -completion. +The +.Fn recvmmsg +function is used to receive multiple +messages at a call. +Their number is supplied by +.Fa vlen . +The messages are placed in the buffers described by +.Fa msgvec +vector, after reception. +The size of each received message is placed in the +.Fa msg_len +field of each element of the vector. +If +.Fa timeout +is NULL the call blocks until the data is available for each +supplied message buffer. +Otherwise it waits for data for the specified amount of time. +If the timeout expired and there is no data received, +a value 0 is returned. +The +.Xr ppoll 2 +system call is used to implement the timeout mechanism, +before first receive is performed. +.Pp +The +.Fn recv , +.Fn recvfrom +and +.Fn recvmsg +return the length of the message on successful +completion, whereas +.Fn recvmmsg +returns the number of received messages. If a message is too long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from (see .Xr socket 2 ) . .Pp If no messages are available at the socket, the receive call waits for a message to arrive, unless the socket is non-blocking (see .Xr fcntl 2 ) in which case the value \-1 is returned and the global variable .Va errno is set to .Er EAGAIN . -The receive calls normally return any data available, +The receive calls except +.Fn recvmmsg +normally return any data available, up to the requested amount, rather than waiting for receipt of the full amount requested; this behavior is affected by the socket-level options .Dv SO_RCVLOWAT and .Dv SO_RCVTIMEO described in .Xr getsockopt 2 . +The +.Fn recvmmsg +function implements this behaviour for each message in the vector. .Pp The .Xr select 2 system call may be used to determine when more data arrives. .Pp The .Fa flags argument to a .Fn recv function is formed by .Em or Ap ing one or more of the values: .Bl -column ".Dv MSG_CMSG_CLOEXEC" -offset indent .It Dv MSG_OOB Ta process out-of-band data .It Dv MSG_PEEK Ta peek at incoming message .It Dv MSG_WAITALL Ta wait for full request or error .It Dv MSG_DONTWAIT Ta do not block .It Dv MSG_CMSG_CLOEXEC Ta set received fds close-on-exec +.It Dv MSG_WAITFORONE Ta do not block after receiving the first message +(only for +.Fn recvmmsg +) .El .Pp The .Dv MSG_OOB flag requests receipt of out-of-band data that would not be received in the normal data stream. Some protocols place expedited data at the head of the normal data queue, and thus this flag cannot be used with such protocols. The .Dv MSG_PEEK flag causes the receive operation to return data from the beginning of the receive queue without removing that data from the queue. Thus, a subsequent receive call will return the same data. The .Dv MSG_WAITALL flag requests that the operation block until the full request is satisfied. However, the call may still return less data than requested if a signal is caught, an error or disconnect occurs, or the next data to be received is of a different type than that returned. The .Dv MSG_DONTWAIT flag requests the call to return when it would block otherwise. If no data is available, .Va errno is set to .Er EAGAIN . This flag is not available in strict .Tn ANSI or C99 compilation mode. +The +.Dv MSG_WAITFORONE +flag sets MSG_DONTWAIT after the first message has been received. +This flag is only relevant for +.Fn recvmmsg . .Pp The .Fn recvmsg system call uses a .Fa msghdr structure to minimize the number of directly supplied arguments. This structure has the following form, as defined in .In sys/socket.h : .Bd -literal struct msghdr { void *msg_name; /* optional address */ socklen_t msg_namelen; /* size of address */ struct iovec *msg_iov; /* scatter/gather array */ int msg_iovlen; /* # elements in msg_iov */ void *msg_control; /* ancillary data, see below */ socklen_t msg_controllen;/* ancillary data buffer len */ int msg_flags; /* flags on received message */ }; .Ed .Pp Here .Fa msg_name and .Fa msg_namelen specify the destination address if the socket is unconnected; .Fa msg_name may be given as a null pointer if no names are desired or required. The .Fa msg_iov and .Fa msg_iovlen arguments describe scatter gather locations, as discussed in .Xr read 2 . The .Fa msg_control argument, which has length .Fa msg_controllen , points to a buffer for other protocol control related messages or other miscellaneous ancillary data. The messages are of the form: .Bd -literal struct cmsghdr { socklen_t cmsg_len; /* data byte count, including hdr */ int cmsg_level; /* originating protocol */ int cmsg_type; /* protocol-specific type */ /* followed by u_char cmsg_data[]; */ }; .Ed .Pp As an example, one could use this to learn of changes in the data-stream in XNS/SPP, or in ISO, to obtain user-connection-request data by requesting a .Fn recvmsg with no data buffer provided immediately after an .Fn accept system call. .Pp Open file descriptors are now passed as ancillary data for .Dv AF_UNIX domain sockets, with .Fa cmsg_level set to .Dv SOL_SOCKET and .Fa cmsg_type set to .Dv SCM_RIGHTS . The close-on-exec flag on received descriptors is set according to the .Dv MSG_CMSG_CLOEXEC flag passed to .Fn recvmsg . .Pp Process credentials can also be passed as ancillary data for .Dv AF_UNIX domain sockets using a .Fa cmsg_type of .Dv SCM_CREDS . In this case, .Fa cmsg_data should be a structure of type .Fa cmsgcred , which is defined in .In sys/socket.h as follows: .Bd -literal struct cmsgcred { pid_t cmcred_pid; /* PID of sending process */ uid_t cmcred_uid; /* real UID of sending process */ uid_t cmcred_euid; /* effective UID of sending process */ gid_t cmcred_gid; /* real GID of sending process */ short cmcred_ngroups; /* number or groups */ gid_t cmcred_groups[CMGROUP_MAX]; /* groups */ }; .Ed .Pp If a sender supplies ancillary data with enough space for the above struct tagged as .Dv SCM_CREDS control message type to the .Fn sendmsg system call, then kernel will fill in the credential information of the sending process and deliver it to the receiver. Since receiver usually has no control over a sender, this method of retrieving credential information isn't reliable. For reliable retrieval of remote side credentials it is advised to use the .Dv LOCAL_CREDS socket option on the receiving socket. See .Xr unix 4 for details. .Pp The .Fa msg_flags field is set on return according to the message received. .Dv MSG_EOR indicates end-of-record; the data returned completed a record (generally used with sockets of type .Dv SOCK_SEQPACKET ) . .Dv MSG_TRUNC indicates that the trailing portion of a datagram was discarded because the datagram was larger than the buffer supplied. .Dv MSG_CTRUNC indicates that some control data were discarded due to lack of space in the buffer for ancillary data. .Dv MSG_OOB is returned to indicate that expedited or out-of-band data were received. +.Pp +The +.Fn recvmmsg +system call uses the +.Fa mmsghdr +structure, defined as follows in the +.In sys/socket.h +header : +.Bd -literal +struct mmsghdr { + struct msghdr msg_hdr; /* message header */ + ssize_t msg_len; /* message length */ +}; +.Ed +.Pp +On data reception the +.Fa msg_len +field is updated to the length of the received message. .Sh RETURN VALUES -These calls return the number of bytes received, or -1 -if an error occurred. +These calls except +.Fn recvmmsg +return the number of bytes received. +.Fn recvmmsg +returns the number of messages received. +A value of -1 is returned if an error occurred. .Sh ERRORS The calls fail if: .Bl -tag -width Er .It Bq Er EBADF The argument .Fa s is an invalid descriptor. .It Bq Er ECONNRESET The remote socket end is forcibly closed. .It Bq Er ENOTCONN The socket is associated with a connection-oriented protocol and has not been connected (see .Xr connect 2 and .Xr accept 2 ) . .It Bq Er ENOTSOCK The argument .Fa s does not refer to a socket. .It Bq Er EMSGSIZE The .Fn recvmsg system call was used to receive rights (file descriptors) that were in flight on the connection. However, the receiving program did not have enough free file descriptor slots to accept them. In this case the descriptors are closed, any pending data can be returned by another call to .Fn recvmsg . .It Bq Er EAGAIN The socket is marked non-blocking and the receive operation would block, or a receive timeout had been set and the timeout expired before data were received. .It Bq Er EINTR The receive was interrupted by delivery of a signal before any data were available. .It Bq Er EFAULT The receive buffer pointer(s) point outside the process's address space. .El .Sh SEE ALSO .Xr fcntl 2 , .Xr getsockopt 2 , .Xr read 2 , .Xr select 2 , .Xr socket 2 , .Xr unix 4 .Sh HISTORY The .Fn recv function appeared in .Bx 4.2 . +The +.Fn recvmmsg +function appeared in +.Fx 11.0 . Index: user/ngie/make_check/lib/libc/sys/send.2 =================================================================== --- user/ngie/make_check/lib/libc/sys/send.2 (revision 295077) +++ user/ngie/make_check/lib/libc/sys/send.2 (revision 295078) @@ -1,232 +1,270 @@ .\" Copyright (c) 1983, 1991, 1993 .\" The Regents of the University of California. All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" 4. Neither the name of the University nor the names of its contributors .\" may be used to endorse or promote products derived from this software .\" without specific prior written permission. .\" .\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" From: @(#)send.2 8.2 (Berkeley) 2/21/94 .\" $FreeBSD$ .\" -.Dd February 5, 2009 +.Dd January 29, 2016 .Dt SEND 2 .Os .Sh NAME .Nm send , .Nm sendto , -.Nm sendmsg -.Nd send a message from a socket +.Nm sendmsg , +.Nm sendmmsg +.Nd send message(s) from a socket .Sh LIBRARY .Lb libc .Sh SYNOPSIS .In sys/types.h .In sys/socket.h .Ft ssize_t .Fn send "int s" "const void *msg" "size_t len" "int flags" .Ft ssize_t .Fn sendto "int s" "const void *msg" "size_t len" "int flags" "const struct sockaddr *to" "socklen_t tolen" .Ft ssize_t .Fn sendmsg "int s" "const struct msghdr *msg" "int flags" +.Ft ssize_t +.Fn sendmmsg "int s" "struct mmsghdr * restrict msgvec" "size_t vlen" "int flags" .Sh DESCRIPTION The .Fn send -function, and +.Fn sendmmsg +functions, +and .Fn sendto and .Fn sendmsg system calls -are used to transmit a message to another socket. +are used to transmit one or more messages (with the +.Fn sendmmsg +call) to +another socket. The .Fn send function may be used only when the socket is in a .Em connected state, while -.Fn sendto -and +.Fn sendto , .Fn sendmsg +and +.Fn sendmmsg may be used at any time. .Pp The address of the target is given by .Fa to with .Fa tolen specifying its size. The length of the message is given by .Fa len . If the message is too long to pass atomically through the underlying protocol, the error .Er EMSGSIZE is returned, and the message is not transmitted. .Pp +The +.Fn sendmmsg +function sends multiple messages at a call. +They are given by the +.Fa msgvec +vector along with +.Fa vlen +specifying the vector size. +The number of octets sent per each message is placed in the +.Fa msg_len +field of each processed element of the vector after transmission. +.Pp No indication of failure to deliver is implicit in a .Fn send . Locally detected errors are indicated by a return value of -1. .Pp If no messages space is available at the socket to hold the message to be transmitted, then .Fn send normally blocks, unless the socket has been placed in non-blocking I/O mode. The .Xr select 2 system call may be used to determine when it is possible to send more data. .Pp The .Fa flags argument may include one or more of the following: .Bd -literal #define MSG_OOB 0x00001 /* process out-of-band data */ #define MSG_DONTROUTE 0x00004 /* bypass routing, use direct interface */ #define MSG_EOR 0x00008 /* data completes record */ #define MSG_EOF 0x00100 /* data completes transaction */ #define MSG_NOSIGNAL 0x20000 /* do not generate SIGPIPE on EOF */ .Ed .Pp The flag .Dv MSG_OOB is used to send .Dq out-of-band data on sockets that support this notion (e.g.\& .Dv SOCK_STREAM ) ; the underlying protocol must also support .Dq out-of-band data. .Dv MSG_EOR is used to indicate a record mark for protocols which support the concept. .Dv MSG_EOF requests that the sender side of a socket be shut down, and that an appropriate indication be sent at the end of the specified data; this flag is only implemented for .Dv SOCK_STREAM sockets in the .Dv PF_INET protocol family. .Dv MSG_DONTROUTE is usually used only by diagnostic or routing programs. .Dv MSG_NOSIGNAL is used to prevent .Dv SIGPIPE generation when writing a socket that may be closed. .Pp See .Xr recv 2 for a description of the .Fa msghdr +structure and the +.Fa mmsghdr structure. .Sh RETURN VALUES -The call returns the number of characters sent, or -1 -if an error occurred. +The +.Fn send , +.Fn sendto +and +.Fn sendmsg +calls +return the number of octets sent. +The +.Fn sendmmsg +call returns the number of messages sent. +If an error occurred a value of -1 is returned. .Sh ERRORS The .Fn send -function and +and +.Fn sendmmsg +functions and .Fn sendto and .Fn sendmsg system calls fail if: .Bl -tag -width Er .It Bq Er EBADF An invalid descriptor was specified. .It Bq Er EACCES The destination address is a broadcast address, and .Dv SO_BROADCAST has not been set on the socket. .It Bq Er ENOTSOCK The argument .Fa s is not a socket. .It Bq Er EFAULT An invalid user space address was specified for an argument. .It Bq Er EMSGSIZE The socket requires that message be sent atomically, and the size of the message to be sent made this impossible. .It Bq Er EAGAIN The socket is marked non-blocking and the requested operation would block. .It Bq Er ENOBUFS The system was unable to allocate an internal buffer. The operation may succeed when buffers become available. .It Bq Er ENOBUFS The output queue for a network interface was full. This generally indicates that the interface has stopped sending, but may be caused by transient congestion. .It Bq Er EHOSTUNREACH The remote host was unreachable. .It Bq Er EISCONN A destination address was specified and the socket is already connected. .It Bq Er ECONNREFUSED The socket received an ICMP destination unreachable message from the last message sent. This typically means that the receiver is not listening on the remote port. .It Bq Er EHOSTDOWN The remote host was down. .It Bq Er ENETDOWN The remote network was down. .It Bq Er EADDRNOTAVAIL The process using a .Dv SOCK_RAW socket was jailed and the source address specified in the IP header did not match the IP address bound to the prison. .It Bq Er EPIPE The socket is unable to send anymore data .Dv ( SBS_CANTSENDMORE has been set on the socket). This typically means that the socket is not connected. .El .Sh SEE ALSO .Xr fcntl 2 , .Xr getsockopt 2 , .Xr recv 2 , .Xr select 2 , .Xr socket 2 , .Xr write 2 .Sh HISTORY The .Fn send function appeared in .Bx 4.2 . +The +.Fn sendmmsg +function appeared in +.Fx 11.0 . .Sh BUGS Because .Fn sendmsg does not necessarily block until the data has been transferred, it is possible to transfer an open file descriptor across an .Dv AF_UNIX domain socket (see .Xr recv 2 ) , then .Fn close it before it has actually been sent, the result being that the receiver gets a closed file descriptor. It is left to the application to implement an acknowledgment mechanism to prevent this from happening. Index: user/ngie/make_check/lib/libc/tests/gen/Makefile =================================================================== --- user/ngie/make_check/lib/libc/tests/gen/Makefile (revision 295077) +++ user/ngie/make_check/lib/libc/tests/gen/Makefile (revision 295078) @@ -1,78 +1,79 @@ # $FreeBSD$ .include ATF_TESTS_C+= arc4random_test ATF_TESTS_C+= fmtcheck2_test ATF_TESTS_C+= fmtmsg_test ATF_TESTS_C+= fnmatch2_test ATF_TESTS_C+= fpclassify2_test ATF_TESTS_C+= ftw_test ATF_TESTS_C+= popen_test ATF_TESTS_C+= posix_spawn_test ATF_TESTS_C+= wordexp_test +ATF_TESTS_C+= dlopen_empty_test # TODO: t_closefrom, t_cpuset, t_fmtcheck, t_randomid, t_sleep # TODO: t_siginfo (fixes require further inspection) # TODO: t_sethostname_test (consistently screws up the hostname) NETBSD_ATF_TESTS_C= alarm_test NETBSD_ATF_TESTS_C+= assert_test NETBSD_ATF_TESTS_C+= basedirname_test NETBSD_ATF_TESTS_C+= dir_test NETBSD_ATF_TESTS_C+= floatunditf_test NETBSD_ATF_TESTS_C+= fnmatch_test NETBSD_ATF_TESTS_C+= fpclassify_test NETBSD_ATF_TESTS_C+= fpsetmask_test NETBSD_ATF_TESTS_C+= fpsetround_test NETBSD_ATF_TESTS_C+= ftok_test NETBSD_ATF_TESTS_C+= getcwd_test NETBSD_ATF_TESTS_C+= getgrent_test NETBSD_ATF_TESTS_C+= glob_test NETBSD_ATF_TESTS_C+= humanize_number_test NETBSD_ATF_TESTS_C+= isnan_test NETBSD_ATF_TESTS_C+= nice_test NETBSD_ATF_TESTS_C+= pause_test NETBSD_ATF_TESTS_C+= raise_test NETBSD_ATF_TESTS_C+= realpath_test NETBSD_ATF_TESTS_C+= setdomainname_test NETBSD_ATF_TESTS_C+= sethostname_test NETBSD_ATF_TESTS_C+= sleep_test NETBSD_ATF_TESTS_C+= syslog_test NETBSD_ATF_TESTS_C+= time_test NETBSD_ATF_TESTS_C+= ttyname_test NETBSD_ATF_TESTS_C+= vis_test .include "../Makefile.netbsd-tests" LIBADD.humanize_number_test+= util LIBADD.fpclassify_test+=m LIBADD.fpsetround_test+=m LIBADD.siginfo_test+= m LIBADD.nice_test+= pthread LIBADD.syslog_test+= pthread CFLAGS+= -I${.CURDIR} SRCS.fmtcheck2_test= fmtcheck_test.c SRCS.fnmatch2_test= fnmatch_test.c TESTS_SUBDIRS= execve TESTS_SUBDIRS+= posix_spawn # The old testcase name TEST_FNMATCH= test-fnmatch CLEANFILES+= ${GEN_SH_CASE_TESTCASES} sh-tests: .PHONY .for target in clean obj depend all @cd ${.CURDIR} && ${MAKE} PROG=${TEST_FNMATCH} \ -DNO_SUBDIR ${target} .endfor @cd ${.OBJDIR} && ./${TEST_FNMATCH} -s 1 > \ ${SRCTOP}/bin/sh/tests/builtins/case2.0 @cd ${.OBJDIR} && ./${TEST_FNMATCH} -s 2 > \ ${SRCTOP}/bin/sh/tests/builtins/case3.0 .include Index: user/ngie/make_check/lib/libc/tests/gen/dlopen_empty_test.c =================================================================== --- user/ngie/make_check/lib/libc/tests/gen/dlopen_empty_test.c (nonexistent) +++ user/ngie/make_check/lib/libc/tests/gen/dlopen_empty_test.c (revision 295078) @@ -0,0 +1,97 @@ +/*- + * Copyright (c) 2016 Maksym Sobolyev + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +static const char *funname; +static char *soname; + +static void +sigsegv_handler(int sig __unused) +{ + unlink(soname); + free(soname); + atf_tc_fail("got SIGSEGV in the %s(3)", funname); +} + +ATF_TC(dlopen_empty_test); +ATF_TC_HEAD(dlopen_empty_test, tc) +{ + atf_tc_set_md_var(tc, "descr", "Tests the dlopen() of an empty file " + "returns an error"); +} +ATF_TC_BODY(dlopen_empty_test, tc) +{ + char tempname[] = "/tmp/temp.XXXXXX"; + char *fname; + int fd; + void *dlh; + struct sigaction act, oact; + + fname = mktemp(tempname); + ATF_REQUIRE_MSG(fname != NULL, "mktemp failed; errno=%d", errno); + asprintf(&soname, "%s.so", fname); + ATF_REQUIRE_MSG(soname != NULL, "asprintf failed; errno=%d", ENOMEM); + fd = open(soname, O_WRONLY | O_CREAT | O_TRUNC, DEFFILEMODE); + ATF_REQUIRE_MSG(fd != -1, "open(\"%s\") failed; errno=%d", soname, errno); + close(fd); + + act.sa_handler = sigsegv_handler; + act.sa_flags = 0; + sigemptyset(&act.sa_mask); + ATF_CHECK_MSG(sigaction(SIGSEGV, &act, &oact) != -1, + "sigaction() failed"); + + funname = "dlopen"; + dlh = dlopen(soname, RTLD_LAZY); + if (dlh != NULL) { + funname = "dlclose"; + dlclose(dlh); + } + ATF_REQUIRE_MSG(dlh == NULL, "dlopen(\"%s\") did not fail", soname); + unlink(soname); + free(soname); +} + +ATF_TP_ADD_TCS(tp) +{ + + ATF_TP_ADD_TC(tp, dlopen_empty_test); + + return (atf_no_error()); +} Property changes on: user/ngie/make_check/lib/libc/tests/gen/dlopen_empty_test.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/lib/libc =================================================================== --- user/ngie/make_check/lib/libc (revision 295077) +++ user/ngie/make_check/lib/libc (revision 295078) Property changes on: user/ngie/make_check/lib/libc ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/lib/libc:r295034-295077 Index: user/ngie/make_check/lib/libsysdecode/syscallnames.c =================================================================== --- user/ngie/make_check/lib/libsysdecode/syscallnames.c (revision 295077) +++ user/ngie/make_check/lib/libsysdecode/syscallnames.c (revision 295078) @@ -1,105 +1,105 @@ /*- * Copyright (c) 2015 John H. Baldwin * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* * Map system call codes to names for the supported ABIs on each * platform. Rather than regnerating system call name tables locally * during the build, use the generated tables in the kernel source * tree. */ #include #include #include static #include #if defined(__amd64__) || defined(__powerpc64__) static #include #endif #if defined(__amd64__) || defined(__i386__) static #ifdef __amd64__ #include #else #include #endif #endif #ifdef __amd64__ static #include #endif #if defined(__amd64__) || defined(__aarch64__) static #include #endif const char * sysdecode_syscallname(enum sysdecode_abi abi, unsigned int code) { switch (abi) { - case FREEBSD: + case SYSDECODE_ABI_FREEBSD: if (code < nitems(syscallnames)) return (syscallnames[code]); break; #if defined(__amd64__) || defined(__powerpc64__) - case FREEBSD32: + case SYSDECODE_ABI_FREEBSD32: if (code < nitems(freebsd32_syscallnames)) return (freebsd32_syscallnames[code]); break; #endif #if defined(__amd64__) || defined(__i386__) - case LINUX: + case SYSDECODE_ABI_LINUX: if (code < nitems(linux_syscallnames)) return (linux_syscallnames[code]); break; #endif #ifdef __amd64__ - case LINUX32: + case SYSDECODE_ABI_LINUX32: if (code < nitems(linux32_syscallnames)) return (linux32_syscallnames[code]); break; #endif #if defined(__amd64__) || defined(__aarch64__) - case CLOUDABI64: + case SYSDECODE_ABI_CLOUDABI64: if (code < nitems(cloudabi64_syscallnames)) return (cloudabi64_syscallnames[code]); break; #endif default: break; } return (NULL); } Index: user/ngie/make_check/lib/libsysdecode/sysdecode.3 =================================================================== --- user/ngie/make_check/lib/libsysdecode/sysdecode.3 (revision 295077) +++ user/ngie/make_check/lib/libsysdecode/sysdecode.3 (revision 295078) @@ -1,74 +1,74 @@ .\" .\" Copyright (c) 2015 John Baldwin .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" -.Dd January 24, 2016 +.Dd January 29, 2016 .Dt SYSDECODE 3 .Os .Sh NAME .Nm sysdecode .Nd system argument decoding library .Sh LIBRARY .Lb libsysdecode .Sh DESCRIPTION The .Nm library includes several functions that provide descriptive names of values associated with system calls. .Ss Supported ABIs Some functions in this library provide ABI-specific descriptions. The supported ABIs are named by the .Vt enum sysdecode_abi enumeration. .Pp -.Bl -tag -width "Li UNKNOWN_ABI" -compact -.It Li FREEBSD +.Bl -tag -width "Li SYSDECODE_ABI_CLOUDABI64" -compact +.It Li SYSDECODE_ABI_FREEBSD Native FreeBSD binaries. Supported on all platforms. -.It Li FREEBSD32 +.It Li SYSDECODE_ABI_FREEBSD32 32-bit FreeBSD binaries. Supported on amd64 and powerpc64. -.It Li LINUX +.It Li SYSDECODE_ABI_LINUX Linux binaries of the same platform. Supported on amd64 and i386. -.It Li LINUX32 +.It Li SYSDECODE_ABI_LINUX32 32-bit Linux binaries. Supported on amd64. -.It Li CLOUDABI64 +.It Li SYSDECODE_ABI_CLOUDABI64 64-bit CloudABI binaries. Supported on aarch64 and amd64. -.It Li UNKNOWN_ABI +.It Li SYSDECODE_ABI_UNKNOWN A placeholder for use when the ABI is not known. .El .Sh SEE ALSO .Xr sysdecode_ioctlname 3 , .Xr sysdecode_syscallnames 3 , .Xr sysdecode_utrace 3 .Sh HISTORY The .Nm library first appeared in .Fx 11.0 . Index: user/ngie/make_check/lib/libsysdecode/sysdecode.h =================================================================== --- user/ngie/make_check/lib/libsysdecode/sysdecode.h (revision 295077) +++ user/ngie/make_check/lib/libsysdecode/sysdecode.h (revision 295078) @@ -1,45 +1,45 @@ /*- * Copyright (c) 2015 John H. Baldwin * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef __SYSDECODE_H__ #define __SYSDECODE_H__ enum sysdecode_abi { - UNKNOWN_ABI = 0, - FREEBSD, - FREEBSD32, - LINUX, - LINUX32, - CLOUDABI64 + SYSDECODE_ABI_UNKNOWN = 0, + SYSDECODE_ABI_FREEBSD, + SYSDECODE_ABI_FREEBSD32, + SYSDECODE_ABI_LINUX, + SYSDECODE_ABI_LINUX32, + SYSDECODE_ABI_CLOUDABI64 }; const char *sysdecode_ioctlname(unsigned long _val); const char *sysdecode_syscallname(enum sysdecode_abi _abi, unsigned int _code); int sysdecode_utrace(FILE *_fp, void *_buf, size_t _len); #endif /* !__SYSDECODE_H__ */ Index: user/ngie/make_check/libexec/rtld-elf/map_object.c =================================================================== --- user/ngie/make_check/libexec/rtld-elf/map_object.c (revision 295077) +++ user/ngie/make_check/libexec/rtld-elf/map_object.c (revision 295078) @@ -1,467 +1,473 @@ /*- * Copyright 1996-1998 John D. Polstra. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * $FreeBSD$ */ #include #include #include #include #include #include #include #include #include "debug.h" #include "rtld.h" -static Elf_Ehdr *get_elf_header(int, const char *); +static Elf_Ehdr *get_elf_header(int, const char *, const struct stat *); static int convert_prot(int); /* Elf flags -> mmap protection */ static int convert_flags(int); /* Elf flags -> mmap flags */ /* * Map a shared object into memory. The "fd" argument is a file descriptor, * which must be open on the object and positioned at its beginning. * The "path" argument is a pathname that is used only for error messages. * * The return value is a pointer to a newly-allocated Obj_Entry structure * for the shared object. Returns NULL on failure. */ Obj_Entry * map_object(int fd, const char *path, const struct stat *sb) { Obj_Entry *obj; Elf_Ehdr *hdr; int i; Elf_Phdr *phdr; Elf_Phdr *phlimit; Elf_Phdr **segs; int nsegs; Elf_Phdr *phdyn; Elf_Phdr *phinterp; Elf_Phdr *phtls; caddr_t mapbase; size_t mapsize; Elf_Addr base_vaddr; Elf_Addr base_vlimit; caddr_t base_addr; int base_flags; Elf_Off data_offset; Elf_Addr data_vaddr; Elf_Addr data_vlimit; caddr_t data_addr; int data_prot; int data_flags; Elf_Addr clear_vaddr; caddr_t clear_addr; caddr_t clear_page; Elf_Addr phdr_vaddr; size_t nclear, phsize; Elf_Addr bss_vaddr; Elf_Addr bss_vlimit; caddr_t bss_addr; Elf_Word stack_flags; Elf_Addr relro_page; size_t relro_size; Elf_Addr note_start; Elf_Addr note_end; char *note_map; size_t note_map_len; - hdr = get_elf_header(fd, path); + hdr = get_elf_header(fd, path, sb); if (hdr == NULL) return (NULL); /* * Scan the program header entries, and save key information. * * We expect that the loadable segments are ordered by load address. */ phdr = (Elf_Phdr *) ((char *)hdr + hdr->e_phoff); phsize = hdr->e_phnum * sizeof (phdr[0]); phlimit = phdr + hdr->e_phnum; nsegs = -1; phdyn = phinterp = phtls = NULL; phdr_vaddr = 0; relro_page = 0; relro_size = 0; note_start = 0; note_end = 0; note_map = NULL; segs = alloca(sizeof(segs[0]) * hdr->e_phnum); stack_flags = RTLD_DEFAULT_STACK_PF_EXEC | PF_R | PF_W; while (phdr < phlimit) { switch (phdr->p_type) { case PT_INTERP: phinterp = phdr; break; case PT_LOAD: segs[++nsegs] = phdr; if ((segs[nsegs]->p_align & (PAGE_SIZE - 1)) != 0) { _rtld_error("%s: PT_LOAD segment %d not page-aligned", path, nsegs); goto error; } break; case PT_PHDR: phdr_vaddr = phdr->p_vaddr; phsize = phdr->p_memsz; break; case PT_DYNAMIC: phdyn = phdr; break; case PT_TLS: phtls = phdr; break; case PT_GNU_STACK: stack_flags = phdr->p_flags; break; case PT_GNU_RELRO: relro_page = phdr->p_vaddr; relro_size = phdr->p_memsz; break; case PT_NOTE: if (phdr->p_offset > PAGE_SIZE || phdr->p_offset + phdr->p_filesz > PAGE_SIZE) { note_map_len = round_page(phdr->p_offset + phdr->p_filesz) - trunc_page(phdr->p_offset); note_map = mmap(NULL, note_map_len, PROT_READ, MAP_PRIVATE, fd, trunc_page(phdr->p_offset)); if (note_map == MAP_FAILED) { _rtld_error("%s: error mapping PT_NOTE (%d)", path, errno); goto error; } note_start = (Elf_Addr)(note_map + phdr->p_offset - trunc_page(phdr->p_offset)); } else { note_start = (Elf_Addr)(char *)hdr + phdr->p_offset; } note_end = note_start + phdr->p_filesz; break; } ++phdr; } if (phdyn == NULL) { _rtld_error("%s: object is not dynamically-linked", path); goto error; } if (nsegs < 0) { _rtld_error("%s: too few PT_LOAD segments", path); goto error; } /* * Map the entire address space of the object, to stake out our * contiguous region, and to establish the base address for relocation. */ base_vaddr = trunc_page(segs[0]->p_vaddr); base_vlimit = round_page(segs[nsegs]->p_vaddr + segs[nsegs]->p_memsz); mapsize = base_vlimit - base_vaddr; base_addr = (caddr_t) base_vaddr; base_flags = MAP_PRIVATE | MAP_ANON | MAP_NOCORE; if (npagesizes > 1 && round_page(segs[0]->p_filesz) >= pagesizes[1]) base_flags |= MAP_ALIGNED_SUPER; mapbase = mmap(base_addr, mapsize, PROT_NONE, base_flags, -1, 0); if (mapbase == (caddr_t) -1) { _rtld_error("%s: mmap of entire address space failed: %s", path, rtld_strerror(errno)); goto error; } if (base_addr != NULL && mapbase != base_addr) { _rtld_error("%s: mmap returned wrong address: wanted %p, got %p", path, base_addr, mapbase); goto error1; } for (i = 0; i <= nsegs; i++) { /* Overlay the segment onto the proper region. */ data_offset = trunc_page(segs[i]->p_offset); data_vaddr = trunc_page(segs[i]->p_vaddr); data_vlimit = round_page(segs[i]->p_vaddr + segs[i]->p_filesz); data_addr = mapbase + (data_vaddr - base_vaddr); data_prot = convert_prot(segs[i]->p_flags); data_flags = convert_flags(segs[i]->p_flags) | MAP_FIXED; if (mmap(data_addr, data_vlimit - data_vaddr, data_prot, data_flags | MAP_PREFAULT_READ, fd, data_offset) == (caddr_t) -1) { _rtld_error("%s: mmap of data failed: %s", path, rtld_strerror(errno)); goto error1; } /* Do BSS setup */ if (segs[i]->p_filesz != segs[i]->p_memsz) { /* Clear any BSS in the last page of the segment. */ clear_vaddr = segs[i]->p_vaddr + segs[i]->p_filesz; clear_addr = mapbase + (clear_vaddr - base_vaddr); clear_page = mapbase + (trunc_page(clear_vaddr) - base_vaddr); if ((nclear = data_vlimit - clear_vaddr) > 0) { /* Make sure the end of the segment is writable */ if ((data_prot & PROT_WRITE) == 0 && -1 == mprotect(clear_page, PAGE_SIZE, data_prot|PROT_WRITE)) { _rtld_error("%s: mprotect failed: %s", path, rtld_strerror(errno)); goto error1; } memset(clear_addr, 0, nclear); /* Reset the data protection back */ if ((data_prot & PROT_WRITE) == 0) mprotect(clear_page, PAGE_SIZE, data_prot); } /* Overlay the BSS segment onto the proper region. */ bss_vaddr = data_vlimit; bss_vlimit = round_page(segs[i]->p_vaddr + segs[i]->p_memsz); bss_addr = mapbase + (bss_vaddr - base_vaddr); if (bss_vlimit > bss_vaddr) { /* There is something to do */ if (mmap(bss_addr, bss_vlimit - bss_vaddr, data_prot, data_flags | MAP_ANON, -1, 0) == (caddr_t)-1) { _rtld_error("%s: mmap of bss failed: %s", path, rtld_strerror(errno)); goto error1; } } } if (phdr_vaddr == 0 && data_offset <= hdr->e_phoff && (data_vlimit - data_vaddr + data_offset) >= (hdr->e_phoff + hdr->e_phnum * sizeof (Elf_Phdr))) { phdr_vaddr = data_vaddr + hdr->e_phoff - data_offset; } } obj = obj_new(); if (sb != NULL) { obj->dev = sb->st_dev; obj->ino = sb->st_ino; } obj->mapbase = mapbase; obj->mapsize = mapsize; obj->textsize = round_page(segs[0]->p_vaddr + segs[0]->p_memsz) - base_vaddr; obj->vaddrbase = base_vaddr; obj->relocbase = mapbase - base_vaddr; obj->dynamic = (const Elf_Dyn *) (obj->relocbase + phdyn->p_vaddr); if (hdr->e_entry != 0) obj->entry = (caddr_t) (obj->relocbase + hdr->e_entry); if (phdr_vaddr != 0) { obj->phdr = (const Elf_Phdr *) (obj->relocbase + phdr_vaddr); } else { obj->phdr = malloc(phsize); if (obj->phdr == NULL) { obj_free(obj); _rtld_error("%s: cannot allocate program header", path); goto error1; } memcpy((char *)obj->phdr, (char *)hdr + hdr->e_phoff, phsize); obj->phdr_alloc = true; } obj->phsize = phsize; if (phinterp != NULL) obj->interp = (const char *) (obj->relocbase + phinterp->p_vaddr); if (phtls != NULL) { tls_dtv_generation++; obj->tlsindex = ++tls_max_index; obj->tlssize = phtls->p_memsz; obj->tlsalign = phtls->p_align; obj->tlsinitsize = phtls->p_filesz; obj->tlsinit = mapbase + phtls->p_vaddr; } obj->stack_flags = stack_flags; obj->relro_page = obj->relocbase + trunc_page(relro_page); obj->relro_size = round_page(relro_size); if (note_start < note_end) digest_notes(obj, note_start, note_end); if (note_map != NULL) munmap(note_map, note_map_len); munmap(hdr, PAGE_SIZE); return (obj); error1: munmap(mapbase, mapsize); error: if (note_map != NULL && note_map != MAP_FAILED) munmap(note_map, note_map_len); munmap(hdr, PAGE_SIZE); return (NULL); } static Elf_Ehdr * -get_elf_header(int fd, const char *path) +get_elf_header(int fd, const char *path, const struct stat *sbp) { Elf_Ehdr *hdr; + + /* Make sure file has enough data for the ELF header */ + if (sbp != NULL && sbp->st_size < sizeof(Elf_Ehdr)) { + _rtld_error("%s: invalid file format", path); + return (NULL); + } hdr = mmap(NULL, PAGE_SIZE, PROT_READ, MAP_PRIVATE | MAP_PREFAULT_READ, fd, 0); if (hdr == (Elf_Ehdr *)MAP_FAILED) { _rtld_error("%s: read error: %s", path, rtld_strerror(errno)); return (NULL); } /* Make sure the file is valid */ if (!IS_ELF(*hdr)) { _rtld_error("%s: invalid file format", path); goto error; } if (hdr->e_ident[EI_CLASS] != ELF_TARG_CLASS || hdr->e_ident[EI_DATA] != ELF_TARG_DATA) { _rtld_error("%s: unsupported file layout", path); goto error; } if (hdr->e_ident[EI_VERSION] != EV_CURRENT || hdr->e_version != EV_CURRENT) { _rtld_error("%s: unsupported file version", path); goto error; } if (hdr->e_type != ET_EXEC && hdr->e_type != ET_DYN) { _rtld_error("%s: unsupported file type", path); goto error; } if (hdr->e_machine != ELF_TARG_MACH) { _rtld_error("%s: unsupported machine", path); goto error; } /* * We rely on the program header being in the first page. This is * not strictly required by the ABI specification, but it seems to * always true in practice. And, it simplifies things considerably. */ if (hdr->e_phentsize != sizeof(Elf_Phdr)) { _rtld_error( "%s: invalid shared object: e_phentsize != sizeof(Elf_Phdr)", path); goto error; } if (hdr->e_phoff + hdr->e_phnum * sizeof(Elf_Phdr) > (size_t)PAGE_SIZE) { _rtld_error("%s: program header too large", path); goto error; } return (hdr); error: munmap(hdr, PAGE_SIZE); return (NULL); } void obj_free(Obj_Entry *obj) { Objlist_Entry *elm; if (obj->tls_done) free_tls_offset(obj); while (obj->needed != NULL) { Needed_Entry *needed = obj->needed; obj->needed = needed->next; free(needed); } while (!STAILQ_EMPTY(&obj->names)) { Name_Entry *entry = STAILQ_FIRST(&obj->names); STAILQ_REMOVE_HEAD(&obj->names, link); free(entry); } while (!STAILQ_EMPTY(&obj->dldags)) { elm = STAILQ_FIRST(&obj->dldags); STAILQ_REMOVE_HEAD(&obj->dldags, link); free(elm); } while (!STAILQ_EMPTY(&obj->dagmembers)) { elm = STAILQ_FIRST(&obj->dagmembers); STAILQ_REMOVE_HEAD(&obj->dagmembers, link); free(elm); } if (obj->vertab) free(obj->vertab); if (obj->origin_path) free(obj->origin_path); if (obj->z_origin) free(obj->rpath); if (obj->priv) free(obj->priv); if (obj->path) free(obj->path); if (obj->phdr_alloc) free((void *)obj->phdr); free(obj); } Obj_Entry * obj_new(void) { Obj_Entry *obj; obj = CNEW(Obj_Entry); STAILQ_INIT(&obj->dldags); STAILQ_INIT(&obj->dagmembers); STAILQ_INIT(&obj->names); return obj; } /* * Given a set of ELF protection flags, return the corresponding protection * flags for MMAP. */ static int convert_prot(int elfflags) { int prot = 0; if (elfflags & PF_R) prot |= PROT_READ; if (elfflags & PF_W) prot |= PROT_WRITE; if (elfflags & PF_X) prot |= PROT_EXEC; return prot; } static int convert_flags(int elfflags) { int flags = MAP_PRIVATE; /* All mappings are private */ /* * Readonly mappings are marked "MAP_NOCORE", because they can be * reconstructed by a debugger. */ if (!(elfflags & PF_W)) flags |= MAP_NOCORE; return flags; } Index: user/ngie/make_check/sys/arm/arm/pmap-v6-new.c =================================================================== --- user/ngie/make_check/sys/arm/arm/pmap-v6-new.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/pmap-v6-new.c (nonexistent) @@ -1,6634 +0,0 @@ -/*- - * Copyright (c) 1991 Regents of the University of California. - * Copyright (c) 1994 John S. Dyson - * Copyright (c) 1994 David Greenman - * Copyright (c) 2005-2010 Alan L. Cox - * Copyright (c) 2014 Svatopluk Kraus - * Copyright (c) 2014 Michal Meloun - * All rights reserved. - * - * This code is derived from software contributed to Berkeley by - * the Systems Programming Group of the University of Utah Computer - * Science Department and William Jolitz of UUNET Technologies Inc. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * 3. Neither the name of the University nor the names of its contributors - * may be used to endorse or promote products derived from this software - * without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - * - * from: @(#)pmap.c 7.7 (Berkeley) 5/12/91 - */ -/*- - * Copyright (c) 2003 Networks Associates Technology, Inc. - * All rights reserved. - * - * This software was developed for the FreeBSD Project by Jake Burkholder, - * Safeport Network Services, and Network Associates Laboratories, the - * Security Research Division of Network Associates, Inc. under - * DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA - * CHATS research program. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -#include -__FBSDID("$FreeBSD$"); - -/* - * Manages physical address maps. - * - * Since the information managed by this module is - * also stored by the logical address mapping module, - * this module may throw away valid virtual-to-physical - * mappings at almost any time. However, invalidations - * of virtual-to-physical mappings must be done as - * requested. - * - * In order to cope with hardware architectures which - * make virtual-to-physical map invalidates expensive, - * this module may delay invalidate or reduced protection - * operations until such time as they are actually - * necessary. This module is given full information as - * to which processors are currently using which maps, - * and to when physical maps must be made correct. - */ - -#include "opt_vm.h" -#include "opt_pmap.h" -#include "opt_ddb.h" - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#ifdef SMP -#include -#else -#include -#endif - -#ifdef DDB -#include -#endif - -#include -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include -#include -#include -#ifdef SMP -#include -#endif - -#ifndef PMAP_SHPGPERPROC -#define PMAP_SHPGPERPROC 200 -#endif - -#ifndef DIAGNOSTIC -#define PMAP_INLINE __inline -#else -#define PMAP_INLINE -#endif - -#ifdef PMAP_DEBUG -static void pmap_zero_page_check(vm_page_t m); -void pmap_debug(int level); -int pmap_pid_dump(int pid); - -#define PDEBUG(_lev_,_stat_) \ - if (pmap_debug_level >= (_lev_)) \ - ((_stat_)) -#define dprintf printf -int pmap_debug_level = 1; -#else /* PMAP_DEBUG */ -#define PDEBUG(_lev_,_stat_) /* Nothing */ -#define dprintf(x, arg...) -#endif /* PMAP_DEBUG */ - -/* - * Level 2 page tables map definion ('max' is excluded). - */ - -#define PT2V_MIN_ADDRESS ((vm_offset_t)PT2MAP) -#define PT2V_MAX_ADDRESS ((vm_offset_t)PT2MAP + PT2MAP_SIZE) - -#define UPT2V_MIN_ADDRESS ((vm_offset_t)PT2MAP) -#define UPT2V_MAX_ADDRESS \ - ((vm_offset_t)(PT2MAP + (KERNBASE >> PT2MAP_SHIFT))) - -/* - * Promotion to a 1MB (PTE1) page mapping requires that the corresponding - * 4KB (PTE2) page mappings have identical settings for the following fields: - */ -#define PTE2_PROMOTE (PTE2_V | PTE2_A | PTE2_NM | PTE2_S | PTE2_NG | \ - PTE2_NX | PTE2_RO | PTE2_U | PTE2_W | \ - PTE2_ATTR_MASK) - -#define PTE1_PROMOTE (PTE1_V | PTE1_A | PTE1_NM | PTE1_S | PTE1_NG | \ - PTE1_NX | PTE1_RO | PTE1_U | PTE1_W | \ - PTE1_ATTR_MASK) - -#define ATTR_TO_L1(l2_attr) ((((l2_attr) & L2_TEX0) ? L1_S_TEX0 : 0) | \ - (((l2_attr) & L2_C) ? L1_S_C : 0) | \ - (((l2_attr) & L2_B) ? L1_S_B : 0) | \ - (((l2_attr) & PTE2_A) ? PTE1_A : 0) | \ - (((l2_attr) & PTE2_NM) ? PTE1_NM : 0) | \ - (((l2_attr) & PTE2_S) ? PTE1_S : 0) | \ - (((l2_attr) & PTE2_NG) ? PTE1_NG : 0) | \ - (((l2_attr) & PTE2_NX) ? PTE1_NX : 0) | \ - (((l2_attr) & PTE2_RO) ? PTE1_RO : 0) | \ - (((l2_attr) & PTE2_U) ? PTE1_U : 0) | \ - (((l2_attr) & PTE2_W) ? PTE1_W : 0)) - -#define ATTR_TO_L2(l1_attr) ((((l1_attr) & L1_S_TEX0) ? L2_TEX0 : 0) | \ - (((l1_attr) & L1_S_C) ? L2_C : 0) | \ - (((l1_attr) & L1_S_B) ? L2_B : 0) | \ - (((l1_attr) & PTE1_A) ? PTE2_A : 0) | \ - (((l1_attr) & PTE1_NM) ? PTE2_NM : 0) | \ - (((l1_attr) & PTE1_S) ? PTE2_S : 0) | \ - (((l1_attr) & PTE1_NG) ? PTE2_NG : 0) | \ - (((l1_attr) & PTE1_NX) ? PTE2_NX : 0) | \ - (((l1_attr) & PTE1_RO) ? PTE2_RO : 0) | \ - (((l1_attr) & PTE1_U) ? PTE2_U : 0) | \ - (((l1_attr) & PTE1_W) ? PTE2_W : 0)) - -/* - * PTE2 descriptors creation macros. - */ -#define PTE2_KPT(pa) PTE2_KERN(pa, PTE2_AP_KRW, pt_memattr) -#define PTE2_KPT_NG(pa) PTE2_KERN_NG(pa, PTE2_AP_KRW, pt_memattr) - -#define PTE2_KRW(pa) PTE2_KERN(pa, PTE2_AP_KRW, PTE2_ATTR_NORMAL) -#define PTE2_KRO(pa) PTE2_KERN(pa, PTE2_AP_KR, PTE2_ATTR_NORMAL) - -#define PV_STATS -#ifdef PV_STATS -#define PV_STAT(x) do { x ; } while (0) -#else -#define PV_STAT(x) do { } while (0) -#endif - -/* - * The boot_pt1 is used temporary in very early boot stage as L1 page table. - * We can init many things with no memory allocation thanks to its static - * allocation and this brings two main advantages: - * (1) other cores can be started very simply, - * (2) various boot loaders can be supported as its arguments can be processed - * in virtual address space and can be moved to safe location before - * first allocation happened. - * Only disadvantage is that boot_pt1 is used only in very early boot stage. - * However, the table is uninitialized and so lays in bss. Therefore kernel - * image size is not influenced. - * - * QQQ: In the future, maybe, boot_pt1 can be used for soft reset and - * CPU suspend/resume game. - */ -extern pt1_entry_t boot_pt1[]; - -vm_paddr_t base_pt1; -pt1_entry_t *kern_pt1; -pt2_entry_t *kern_pt2tab; -pt2_entry_t *PT2MAP; - -static uint32_t ttb_flags; -static vm_memattr_t pt_memattr; -ttb_entry_t pmap_kern_ttb; - -/* XXX use converion function*/ -#define PTE2_ATTR_NORMAL VM_MEMATTR_DEFAULT -#define PTE1_ATTR_NORMAL ATTR_TO_L1(PTE2_ATTR_NORMAL) - -struct pmap kernel_pmap_store; -LIST_HEAD(pmaplist, pmap); -static struct pmaplist allpmaps; -static struct mtx allpmaps_lock; - -vm_offset_t virtual_avail; /* VA of first avail page (after kernel bss) */ -vm_offset_t virtual_end; /* VA of last avail page (end of kernel AS) */ - -static vm_offset_t kernel_vm_end_new; -vm_offset_t kernel_vm_end = KERNBASE + NKPT2PG * NPT2_IN_PG * PTE1_SIZE; -vm_offset_t vm_max_kernel_address; -vm_paddr_t kernel_l1pa; - -static struct rwlock __aligned(CACHE_LINE_SIZE) pvh_global_lock; - -/* - * Data for the pv entry allocation mechanism - */ -static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); -static int pv_entry_count = 0, pv_entry_max = 0, pv_entry_high_water = 0; -static struct md_page *pv_table; /* XXX: Is it used only the list in md_page? */ -static int shpgperproc = PMAP_SHPGPERPROC; - -struct pv_chunk *pv_chunkbase; /* KVA block for pv_chunks */ -int pv_maxchunks; /* How many chunks we have KVA for */ -vm_offset_t pv_vafree; /* freelist stored in the PTE */ - -vm_paddr_t first_managed_pa; -#define pa_to_pvh(pa) (&pv_table[pte1_index(pa - first_managed_pa)]) - -/* - * All those kernel PT submaps that BSD is so fond of - */ -struct sysmaps { - struct mtx lock; - pt2_entry_t *CMAP1; - pt2_entry_t *CMAP2; - pt2_entry_t *CMAP3; - caddr_t CADDR1; - caddr_t CADDR2; - caddr_t CADDR3; -}; -static struct sysmaps sysmaps_pcpu[MAXCPU]; -static pt2_entry_t *CMAP3; -static caddr_t CADDR3; -caddr_t _tmppt = 0; - -struct msgbuf *msgbufp = 0; /* XXX move it to machdep.c */ - -/* - * Crashdump maps. - */ -static caddr_t crashdumpmap; - -static pt2_entry_t *PMAP1 = 0, *PMAP2; -static pt2_entry_t *PADDR1 = 0, *PADDR2; -#ifdef DDB -static pt2_entry_t *PMAP3; -static pt2_entry_t *PADDR3; -static int PMAP3cpu __unused; /* for SMP only */ -#endif -#ifdef SMP -static int PMAP1cpu; -static int PMAP1changedcpu; -SYSCTL_INT(_debug, OID_AUTO, PMAP1changedcpu, CTLFLAG_RD, - &PMAP1changedcpu, 0, - "Number of times pmap_pte2_quick changed CPU with same PMAP1"); -#endif -static int PMAP1changed; -SYSCTL_INT(_debug, OID_AUTO, PMAP1changed, CTLFLAG_RD, - &PMAP1changed, 0, - "Number of times pmap_pte2_quick changed PMAP1"); -static int PMAP1unchanged; -SYSCTL_INT(_debug, OID_AUTO, PMAP1unchanged, CTLFLAG_RD, - &PMAP1unchanged, 0, - "Number of times pmap_pte2_quick didn't change PMAP1"); -static struct mtx PMAP2mutex; - -static __inline void pt2_wirecount_init(vm_page_t m); -static boolean_t pmap_demote_pte1(pmap_t pmap, pt1_entry_t *pte1p, - vm_offset_t va); -void cache_icache_sync_fresh(vm_offset_t va, vm_paddr_t pa, vm_size_t size); - -/* - * Function to set the debug level of the pmap code. - */ -#ifdef PMAP_DEBUG -void -pmap_debug(int level) -{ - - pmap_debug_level = level; - dprintf("pmap_debug: level=%d\n", pmap_debug_level); -} -#endif /* PMAP_DEBUG */ - -/* - * This table must corespond with memory attribute configuration in vm.h. - * First entry is used for normal system mapping. - * - * Device memory is always marked as shared. - * Normal memory is shared only in SMP . - * Not outer shareable bits are not used yet. - * Class 6 cannot be used on ARM11. - */ -#define TEXDEF_TYPE_SHIFT 0 -#define TEXDEF_TYPE_MASK 0x3 -#define TEXDEF_INNER_SHIFT 2 -#define TEXDEF_INNER_MASK 0x3 -#define TEXDEF_OUTER_SHIFT 4 -#define TEXDEF_OUTER_MASK 0x3 -#define TEXDEF_NOS_SHIFT 6 -#define TEXDEF_NOS_MASK 0x1 - -#define TEX(t, i, o, s) \ - ((t) << TEXDEF_TYPE_SHIFT) | \ - ((i) << TEXDEF_INNER_SHIFT) | \ - ((o) << TEXDEF_OUTER_SHIFT | \ - ((s) << TEXDEF_NOS_SHIFT)) - -static uint32_t tex_class[8] = { -/* type inner cache outer cache */ - TEX(PRRR_MEM, NMRR_WB_WA, NMRR_WB_WA, 0), /* 0 - ATTR_WB_WA */ - TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 1 - ATTR_NOCACHE */ - TEX(PRRR_DEV, NMRR_NC, NMRR_NC, 0), /* 2 - ATTR_DEVICE */ - TEX(PRRR_SO, NMRR_NC, NMRR_NC, 0), /* 3 - ATTR_SO */ - TEX(PRRR_MEM, NMRR_WT, NMRR_WT, 0), /* 4 - ATTR_WT */ - TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 5 - NOT USED YET */ - TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 6 - NOT USED YET */ - TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 7 - NOT USED YET */ -}; -#undef TEX - -/* - * Convert TEX definition entry to TTB flags. - */ -static uint32_t -encode_ttb_flags(int idx) -{ - uint32_t inner, outer, nos, reg; - - inner = (tex_class[idx] >> TEXDEF_INNER_SHIFT) & - TEXDEF_INNER_MASK; - outer = (tex_class[idx] >> TEXDEF_OUTER_SHIFT) & - TEXDEF_OUTER_MASK; - nos = (tex_class[idx] >> TEXDEF_NOS_SHIFT) & - TEXDEF_NOS_MASK; - - reg = nos << 5; - reg |= outer << 3; - if (cpuinfo.coherent_walk) - reg |= (inner & 0x1) << 6; - reg |= (inner & 0x2) >> 1; -#ifdef SMP - reg |= 1 << 1; -#endif - return reg; -} - -/* - * Set TEX remapping registers in current CPU. - */ -void -pmap_set_tex(void) -{ - uint32_t prrr, nmrr; - uint32_t type, inner, outer, nos; - int i; - -#ifdef PMAP_PTE_NOCACHE - /* XXX fixme */ - if (cpuinfo.coherent_walk) { - pt_memattr = VM_MEMATTR_WB_WA; - ttb_flags = encode_ttb_flags(0); - } - else { - pt_memattr = VM_MEMATTR_NOCACHE; - ttb_flags = encode_ttb_flags(1); - } -#else - pt_memattr = VM_MEMATTR_WB_WA; - ttb_flags = encode_ttb_flags(0); -#endif - - prrr = 0; - nmrr = 0; - - /* Build remapping register from TEX classes. */ - for (i = 0; i < 8; i++) { - type = (tex_class[i] >> TEXDEF_TYPE_SHIFT) & - TEXDEF_TYPE_MASK; - inner = (tex_class[i] >> TEXDEF_INNER_SHIFT) & - TEXDEF_INNER_MASK; - outer = (tex_class[i] >> TEXDEF_OUTER_SHIFT) & - TEXDEF_OUTER_MASK; - nos = (tex_class[i] >> TEXDEF_NOS_SHIFT) & - TEXDEF_NOS_MASK; - - prrr |= type << (i * 2); - prrr |= nos << (i + 24); - nmrr |= inner << (i * 2); - nmrr |= outer << (i * 2 + 16); - } - /* Add shareable bits for device memory. */ - prrr |= PRRR_DS0 | PRRR_DS1; - - /* Add shareable bits for normal memory in SMP case. */ -#ifdef SMP - prrr |= PRRR_NS1; -#endif - cp15_prrr_set(prrr); - cp15_nmrr_set(nmrr); - - /* Caches are disabled, so full TLB flush should be enough. */ - tlb_flush_all_local(); -} - -/* - * KERNBASE must be multiple of NPT2_IN_PG * PTE1_SIZE. In other words, - * KERNBASE is mapped by first L2 page table in L2 page table page. It - * meets same constrain due to PT2MAP being placed just under KERNBASE. - */ -CTASSERT((KERNBASE & (NPT2_IN_PG * PTE1_SIZE - 1)) == 0); -CTASSERT((KERNBASE - VM_MAXUSER_ADDRESS) >= PT2MAP_SIZE); - -/* - * In crazy dreams, PAGE_SIZE could be a multiple of PTE2_SIZE in general. - * For now, anyhow, the following check must be fulfilled. - */ -CTASSERT(PAGE_SIZE == PTE2_SIZE); -/* - * We don't want to mess up MI code with all MMU and PMAP definitions, - * so some things, which depend on other ones, are defined independently. - * Now, it is time to check that we don't screw up something. - */ -CTASSERT(PDRSHIFT == PTE1_SHIFT); -/* - * Check L1 and L2 page table entries definitions consistency. - */ -CTASSERT(NB_IN_PT1 == (sizeof(pt1_entry_t) * NPTE1_IN_PT1)); -CTASSERT(NB_IN_PT2 == (sizeof(pt2_entry_t) * NPTE2_IN_PT2)); -/* - * Check L2 page tables page consistency. - */ -CTASSERT(PAGE_SIZE == (NPT2_IN_PG * NB_IN_PT2)); -CTASSERT((1 << PT2PG_SHIFT) == NPT2_IN_PG); -/* - * Check PT2TAB consistency. - * PT2TAB_ENTRIES is defined as a division of NPTE1_IN_PT1 by NPT2_IN_PG. - * This should be done without remainder. - */ -CTASSERT(NPTE1_IN_PT1 == (PT2TAB_ENTRIES * NPT2_IN_PG)); - -/* - * A PT2MAP magic. - * - * All level 2 page tables (PT2s) are mapped continuously and accordingly - * into PT2MAP address space. As PT2 size is less than PAGE_SIZE, this can - * be done only if PAGE_SIZE is a multiple of PT2 size. All PT2s in one page - * must be used together, but not necessary at once. The first PT2 in a page - * must map things on correctly aligned address and the others must follow - * in right order. - */ -#define NB_IN_PT2TAB (PT2TAB_ENTRIES * sizeof(pt2_entry_t)) -#define NPT2_IN_PT2TAB (NB_IN_PT2TAB / NB_IN_PT2) -#define NPG_IN_PT2TAB (NB_IN_PT2TAB / PAGE_SIZE) - -/* - * Check PT2TAB consistency. - * NPT2_IN_PT2TAB is defined as a division of NB_IN_PT2TAB by NB_IN_PT2. - * NPG_IN_PT2TAB is defined as a division of NB_IN_PT2TAB by PAGE_SIZE. - * The both should be done without remainder. - */ -CTASSERT(NB_IN_PT2TAB == (NPT2_IN_PT2TAB * NB_IN_PT2)); -CTASSERT(NB_IN_PT2TAB == (NPG_IN_PT2TAB * PAGE_SIZE)); -/* - * The implementation was made general, however, with the assumption - * bellow in mind. In case of another value of NPG_IN_PT2TAB, - * the code should be once more rechecked. - */ -CTASSERT(NPG_IN_PT2TAB == 1); - -/* - * Get offset of PT2 in a page - * associated with given PT1 index. - */ -static __inline u_int -page_pt2off(u_int pt1_idx) -{ - - return ((pt1_idx & PT2PG_MASK) * NB_IN_PT2); -} - -/* - * Get physical address of PT2 - * associated with given PT2s page and PT1 index. - */ -static __inline vm_paddr_t -page_pt2pa(vm_paddr_t pgpa, u_int pt1_idx) -{ - - return (pgpa + page_pt2off(pt1_idx)); -} - -/* - * Get first entry of PT2 - * associated with given PT2s page and PT1 index. - */ -static __inline pt2_entry_t * -page_pt2(vm_offset_t pgva, u_int pt1_idx) -{ - - return ((pt2_entry_t *)(pgva + page_pt2off(pt1_idx))); -} - -/* - * Get virtual address of PT2s page (mapped in PT2MAP) - * which holds PT2 which holds entry which maps given virtual address. - */ -static __inline vm_offset_t -pt2map_pt2pg(vm_offset_t va) -{ - - va &= ~(NPT2_IN_PG * PTE1_SIZE - 1); - return ((vm_offset_t)pt2map_entry(va)); -} - -/***************************************************************************** - * - * THREE pmap initialization milestones exist: - * - * locore.S - * -> fundamental init (including MMU) in ASM - * - * initarm() - * -> fundamental init continues in C - * -> first available physical address is known - * - * pmap_bootstrap_prepare() -> FIRST PMAP MILESTONE (first epoch begins) - * -> basic (safe) interface for physical address allocation is made - * -> basic (safe) interface for virtual mapping is made - * -> limited not SMP coherent work is possible - * - * -> more fundamental init continues in C - * -> locks and some more things are available - * -> all fundamental allocations and mappings are done - * - * pmap_bootstrap() -> SECOND PMAP MILESTONE (second epoch begins) - * -> phys_avail[] and virtual_avail is set - * -> control is passed to vm subsystem - * -> physical and virtual address allocation are off limit - * -> low level mapping functions, some SMP coherent, - * are available, which cannot be used before vm subsystem - * is being inited - * - * mi_startup() - * -> vm subsystem is being inited - * - * pmap_init() -> THIRD PMAP MILESTONE (third epoch begins) - * -> pmap is fully inited - * - *****************************************************************************/ - -/***************************************************************************** - * - * PMAP first stage initialization and utility functions - * for pre-bootstrap epoch. - * - * After pmap_bootstrap_prepare() is called, the following functions - * can be used: - * - * (1) strictly only for this stage functions for physical page allocations, - * virtual space allocations, and mappings: - * - * vm_paddr_t pmap_preboot_get_pages(u_int num); - * void pmap_preboot_map_pages(vm_paddr_t pa, vm_offset_t va, u_int num); - * vm_offset_t pmap_preboot_reserve_pages(u_int num); - * vm_offset_t pmap_preboot_get_vpages(u_int num); - * void pmap_preboot_map_attr(vm_paddr_t pa, vm_offset_t va, vm_size_t size, - * int prot, int attr); - * - * (2) for all stages: - * - * vm_paddr_t pmap_kextract(vm_offset_t va); - * - * NOTE: This is not SMP coherent stage. - * - *****************************************************************************/ - -#define KERNEL_P2V(pa) \ - ((vm_offset_t)((pa) - arm_physmem_kernaddr + KERNVIRTADDR)) -#define KERNEL_V2P(va) \ - ((vm_paddr_t)((va) - KERNVIRTADDR + arm_physmem_kernaddr)) - -static vm_paddr_t last_paddr; - -/* - * Pre-bootstrap epoch page allocator. - */ -vm_paddr_t -pmap_preboot_get_pages(u_int num) -{ - vm_paddr_t ret; - - ret = last_paddr; - last_paddr += num * PAGE_SIZE; - - return (ret); -} - -/* - * The fundamental initalization of PMAP stuff. - * - * Some things already happened in locore.S and some things could happen - * before pmap_bootstrap_prepare() is called, so let's recall what is done: - * 1. Caches are disabled. - * 2. We are running on virtual addresses already with 'boot_pt1' - * as L1 page table. - * 3. So far, all virtual addresses can be converted to physical ones and - * vice versa by the following macros: - * KERNEL_P2V(pa) .... physical to virtual ones, - * KERNEL_V2P(va) .... virtual to physical ones. - * - * What is done herein: - * 1. The 'boot_pt1' is replaced by real kernel L1 page table 'kern_pt1'. - * 2. PT2MAP magic is brought to live. - * 3. Basic preboot functions for page allocations and mappings can be used. - * 4. Everything is prepared for L1 cache enabling. - * - * Variations: - * 1. To use second TTB register, so kernel and users page tables will be - * separated. This way process forking - pmap_pinit() - could be faster, - * it saves physical pages and KVA per a process, and it's simple change. - * However, it will lead, due to hardware matter, to the following: - * (a) 2G space for kernel and 2G space for users. - * (b) 1G space for kernel in low addresses and 3G for users above it. - * A question is: Is the case (b) really an option? Note that case (b) - * does save neither physical memory and KVA. - */ -void -pmap_bootstrap_prepare(vm_paddr_t last) -{ - vm_paddr_t pt2pg_pa, pt2tab_pa, pa, size; - vm_offset_t pt2pg_va; - pt1_entry_t *pte1p; - pt2_entry_t *pte2p; - u_int i; - uint32_t actlr_mask, actlr_set; - - /* - * Now, we are going to make real kernel mapping. Note that we are - * already running on some mapping made in locore.S and we expect - * that it's large enough to ensure nofault access to physical memory - * allocated herein before switch. - * - * As kernel image and everything needed before are and will be mapped - * by section mappings, we align last physical address to PTE1_SIZE. - */ - last_paddr = pte1_roundup(last); - - /* - * Allocate and zero page(s) for kernel L1 page table. - * - * Note that it's first allocation on space which was PTE1_SIZE - * aligned and as such base_pt1 is aligned to NB_IN_PT1 too. - */ - base_pt1 = pmap_preboot_get_pages(NPG_IN_PT1); - kern_pt1 = (pt1_entry_t *)KERNEL_P2V(base_pt1); - bzero((void*)kern_pt1, NB_IN_PT1); - pte1_sync_range(kern_pt1, NB_IN_PT1); - - /* Allocate and zero page(s) for kernel PT2TAB. */ - pt2tab_pa = pmap_preboot_get_pages(NPG_IN_PT2TAB); - kern_pt2tab = (pt2_entry_t *)KERNEL_P2V(pt2tab_pa); - bzero(kern_pt2tab, NB_IN_PT2TAB); - pte2_sync_range(kern_pt2tab, NB_IN_PT2TAB); - - /* Allocate and zero page(s) for kernel L2 page tables. */ - pt2pg_pa = pmap_preboot_get_pages(NKPT2PG); - pt2pg_va = KERNEL_P2V(pt2pg_pa); - size = NKPT2PG * PAGE_SIZE; - bzero((void*)pt2pg_va, size); - pte2_sync_range((pt2_entry_t *)pt2pg_va, size); - - /* - * Add a physical memory segment (vm_phys_seg) corresponding to the - * preallocated pages for kernel L2 page tables so that vm_page - * structures representing these pages will be created. The vm_page - * structures are required for promotion of the corresponding kernel - * virtual addresses to section mappings. - */ - vm_phys_add_seg(pt2tab_pa, pmap_preboot_get_pages(0)); - - /* - * Insert allocated L2 page table pages to PT2TAB and make - * link to all PT2s in L1 page table. See how kernel_vm_end - * is initialized. - * - * We play simple and safe. So every KVA will have underlaying - * L2 page table, even kernel image mapped by sections. - */ - pte2p = kern_pt2tab_entry(KERNBASE); - for (pa = pt2pg_pa; pa < pt2pg_pa + size; pa += PTE2_SIZE) - pt2tab_store(pte2p++, PTE2_KPT(pa)); - - pte1p = kern_pte1(KERNBASE); - for (pa = pt2pg_pa; pa < pt2pg_pa + size; pa += NB_IN_PT2) - pte1_store(pte1p++, PTE1_LINK(pa)); - - /* Make section mappings for kernel. */ - pte1p = kern_pte1(KERNBASE); - for (pa = KERNEL_V2P(KERNBASE); pa < last; pa += PTE1_SIZE) - pte1_store(pte1p++, PTE1_KERN(pa, PTE1_AP_KRW, - ATTR_TO_L1(PTE2_ATTR_WB_WA))); - - /* - * Get free and aligned space for PT2MAP and make L1 page table links - * to L2 page tables held in PT2TAB. - * - * Note that pages holding PT2s are stored in PT2TAB as pt2_entry_t - * descriptors and PT2TAB page(s) itself is(are) used as PT2s. Thus - * each entry in PT2TAB maps all PT2s in a page. This implies that - * virtual address of PT2MAP must be aligned to NPT2_IN_PG * PTE1_SIZE. - */ - PT2MAP = (pt2_entry_t *)(KERNBASE - PT2MAP_SIZE); - pte1p = kern_pte1((vm_offset_t)PT2MAP); - for (pa = pt2tab_pa, i = 0; i < NPT2_IN_PT2TAB; i++, pa += NB_IN_PT2) { - pte1_store(pte1p++, PTE1_LINK(pa)); - } - - /* - * Store PT2TAB in PT2TAB itself, i.e. self reference mapping. - * Each pmap will hold own PT2TAB, so the mapping should be not global. - */ - pte2p = kern_pt2tab_entry((vm_offset_t)PT2MAP); - for (pa = pt2tab_pa, i = 0; i < NPG_IN_PT2TAB; i++, pa += PTE2_SIZE) { - pt2tab_store(pte2p++, PTE2_KPT_NG(pa)); - } - - /* - * Choose correct L2 page table and make mappings for allocations - * made herein which replaces temporary locore.S mappings after a while. - * Note that PT2MAP cannot be used until we switch to kern_pt1. - * - * Note, that these allocations started aligned on 1M section and - * kernel PT1 was allocated first. Making of mappings must follow - * order of physical allocations as we've used KERNEL_P2V() macro - * for virtual addresses resolution. - */ - pte2p = kern_pt2tab_entry((vm_offset_t)kern_pt1); - pt2pg_va = KERNEL_P2V(pte2_pa(pte2_load(pte2p))); - - pte2p = page_pt2(pt2pg_va, pte1_index((vm_offset_t)kern_pt1)); - - /* Make mapping for kernel L1 page table. */ - for (pa = base_pt1, i = 0; i < NPG_IN_PT1; i++, pa += PTE2_SIZE) - pte2_store(pte2p++, PTE2_KPT(pa)); - - /* Make mapping for kernel PT2TAB. */ - for (pa = pt2tab_pa, i = 0; i < NPG_IN_PT2TAB; i++, pa += PTE2_SIZE) - pte2_store(pte2p++, PTE2_KPT(pa)); - - /* Finally, switch from 'boot_pt1' to 'kern_pt1'. */ - pmap_kern_ttb = base_pt1 | ttb_flags; - cpuinfo_get_actlr_modifier(&actlr_mask, &actlr_set); - reinit_mmu(pmap_kern_ttb, actlr_mask, actlr_set); - /* - * Initialize the first available KVA. As kernel image is mapped by - * sections, we are leaving some gap behind. - */ - virtual_avail = (vm_offset_t)kern_pt2tab + NPG_IN_PT2TAB * PAGE_SIZE; -} - -/* - * Setup L2 page table page for given KVA. - * Used in pre-bootstrap epoch. - * - * Note that we have allocated NKPT2PG pages for L2 page tables in advance - * and used them for mapping KVA starting from KERNBASE. However, this is not - * enough. Vectors and devices need L2 page tables too. Note that they are - * even above VM_MAX_KERNEL_ADDRESS. - */ -static __inline vm_paddr_t -pmap_preboot_pt2pg_setup(vm_offset_t va) -{ - pt2_entry_t *pte2p, pte2; - vm_paddr_t pt2pg_pa; - - /* Get associated entry in PT2TAB. */ - pte2p = kern_pt2tab_entry(va); - - /* Just return, if PT2s page exists already. */ - pte2 = pt2tab_load(pte2p); - if (pte2_is_valid(pte2)) - return (pte2_pa(pte2)); - - KASSERT(va >= VM_MAX_KERNEL_ADDRESS, - ("%s: NKPT2PG too small", __func__)); - - /* - * Allocate page for PT2s and insert it to PT2TAB. - * In other words, map it into PT2MAP space. - */ - pt2pg_pa = pmap_preboot_get_pages(1); - pt2tab_store(pte2p, PTE2_KPT(pt2pg_pa)); - - /* Zero all PT2s in allocated page. */ - bzero((void*)pt2map_pt2pg(va), PAGE_SIZE); - pte2_sync_range((pt2_entry_t *)pt2map_pt2pg(va), PAGE_SIZE); - - return (pt2pg_pa); -} - -/* - * Setup L2 page table for given KVA. - * Used in pre-bootstrap epoch. - */ -static void -pmap_preboot_pt2_setup(vm_offset_t va) -{ - pt1_entry_t *pte1p; - vm_paddr_t pt2pg_pa, pt2_pa; - - /* Setup PT2's page. */ - pt2pg_pa = pmap_preboot_pt2pg_setup(va); - pt2_pa = page_pt2pa(pt2pg_pa, pte1_index(va)); - - /* Insert PT2 to PT1. */ - pte1p = kern_pte1(va); - pte1_store(pte1p, PTE1_LINK(pt2_pa)); -} - -/* - * Get L2 page entry associated with given KVA. - * Used in pre-bootstrap epoch. - */ -static __inline pt2_entry_t* -pmap_preboot_vtopte2(vm_offset_t va) -{ - pt1_entry_t *pte1p; - - /* Setup PT2 if needed. */ - pte1p = kern_pte1(va); - if (!pte1_is_valid(pte1_load(pte1p))) /* XXX - sections ?! */ - pmap_preboot_pt2_setup(va); - - return (pt2map_entry(va)); -} - -/* - * Pre-bootstrap epoch page(s) mapping(s). - */ -void -pmap_preboot_map_pages(vm_paddr_t pa, vm_offset_t va, u_int num) -{ - u_int i; - pt2_entry_t *pte2p; - - /* Map all the pages. */ - for (i = 0; i < num; i++) { - pte2p = pmap_preboot_vtopte2(va); - pte2_store(pte2p, PTE2_KRW(pa)); - va += PAGE_SIZE; - pa += PAGE_SIZE; - } -} - -/* - * Pre-bootstrap epoch virtual space alocator. - */ -vm_offset_t -pmap_preboot_reserve_pages(u_int num) -{ - u_int i; - vm_offset_t start, va; - pt2_entry_t *pte2p; - - /* Allocate virtual space. */ - start = va = virtual_avail; - virtual_avail += num * PAGE_SIZE; - - /* Zero the mapping. */ - for (i = 0; i < num; i++) { - pte2p = pmap_preboot_vtopte2(va); - pte2_store(pte2p, 0); - va += PAGE_SIZE; - } - - return (start); -} - -/* - * Pre-bootstrap epoch page(s) allocation and mapping(s). - */ -vm_offset_t -pmap_preboot_get_vpages(u_int num) -{ - vm_paddr_t pa; - vm_offset_t va; - - /* Allocate physical page(s). */ - pa = pmap_preboot_get_pages(num); - - /* Allocate virtual space. */ - va = virtual_avail; - virtual_avail += num * PAGE_SIZE; - - /* Map and zero all. */ - pmap_preboot_map_pages(pa, va, num); - bzero((void *)va, num * PAGE_SIZE); - - return (va); -} - -/* - * Pre-bootstrap epoch page mapping(s) with attributes. - */ -void -pmap_preboot_map_attr(vm_paddr_t pa, vm_offset_t va, vm_size_t size, int prot, - int attr) -{ - u_int num; - u_int l1_attr, l1_prot; - pt1_entry_t *pte1p; - pt2_entry_t *pte2p; - - l1_prot = ATTR_TO_L1(prot); - l1_attr = ATTR_TO_L1(attr); - - /* Map all the pages. */ - num = round_page(size); - while (num > 0) { - if ((((va | pa) & PTE1_OFFSET) == 0) && (num >= PTE1_SIZE)) { - pte1p = kern_pte1(va); - pte1_store(pte1p, PTE1_KERN(pa, l1_prot, l1_attr)); - va += PTE1_SIZE; - pa += PTE1_SIZE; - num -= PTE1_SIZE; - } else { - pte2p = pmap_preboot_vtopte2(va); - pte2_store(pte2p, PTE2_KERN(pa, prot, attr)); - va += PAGE_SIZE; - pa += PAGE_SIZE; - num -= PAGE_SIZE; - } - } - -} - -/* - * Extract from the kernel page table the physical address - * that is mapped by the given virtual address "va". - */ -vm_paddr_t -pmap_kextract(vm_offset_t va) -{ - vm_paddr_t pa; - pt1_entry_t pte1; - pt2_entry_t pte2; - - pte1 = pte1_load(kern_pte1(va)); - if (pte1_is_section(pte1)) { - pa = pte1_pa(pte1) | (va & PTE1_OFFSET); - } else if (pte1_is_link(pte1)) { - /* - * We should beware of concurrent promotion that changes - * pte1 at this point. However, it's not a problem as PT2 - * page is preserved by promotion in PT2TAB. So even if - * it happens, using of PT2MAP is still safe. - * - * QQQ: However, concurrent removing is a problem which - * ends in abort on PT2MAP space. Locking must be used - * to deal with this. - */ - pte2 = pte2_load(pt2map_entry(va)); - pa = pte2_pa(pte2) | (va & PTE2_OFFSET); - } - else { - panic("%s: va %#x pte1 %#x", __func__, va, pte1); - } - return (pa); -} - -/* - * Extract from the kernel page table the physical address - * that is mapped by the given virtual address "va". Also - * return L2 page table entry which maps the address. - * - * This is only intended to be used for panic dumps. - */ -vm_paddr_t -pmap_dump_kextract(vm_offset_t va, pt2_entry_t *pte2p) -{ - vm_paddr_t pa; - pt1_entry_t pte1; - pt2_entry_t pte2; - - pte1 = pte1_load(kern_pte1(va)); - if (pte1_is_section(pte1)) { - pa = pte1_pa(pte1) | (va & PTE1_OFFSET); - pte2 = pa | ATTR_TO_L2(pte1) | PTE2_V; - } else if (pte1_is_link(pte1)) { - pte2 = pte2_load(pt2map_entry(va)); - pa = pte2_pa(pte2); - } else { - pte2 = 0; - pa = 0; - } - if (pte2p != NULL) - *pte2p = pte2; - return (pa); -} - -/***************************************************************************** - * - * PMAP second stage initialization and utility functions - * for bootstrap epoch. - * - * After pmap_bootstrap() is called, the following functions for - * mappings can be used: - * - * void pmap_kenter(vm_offset_t va, vm_paddr_t pa); - * void pmap_kremove(vm_offset_t va); - * vm_offset_t pmap_map(vm_offset_t *virt, vm_paddr_t start, vm_paddr_t end, - * int prot); - * - * NOTE: This is not SMP coherent stage. And physical page allocation is not - * allowed during this stage. - * - *****************************************************************************/ - -/* - * Initialize kernel PMAP locks and lists, kernel_pmap itself, and - * reserve various virtual spaces for temporary mappings. - */ -void -pmap_bootstrap(vm_offset_t firstaddr) -{ - pt2_entry_t *unused __unused; - struct sysmaps *sysmaps; - u_int i; - - /* - * Initialize the kernel pmap (which is statically allocated). - */ - PMAP_LOCK_INIT(kernel_pmap); - kernel_l1pa = (vm_paddr_t)kern_pt1; /* for libkvm */ - kernel_pmap->pm_pt1 = kern_pt1; - kernel_pmap->pm_pt2tab = kern_pt2tab; - CPU_FILL(&kernel_pmap->pm_active); /* don't allow deactivation */ - TAILQ_INIT(&kernel_pmap->pm_pvchunk); - - /* - * Initialize the global pv list lock. - */ - rw_init(&pvh_global_lock, "pmap pv global"); - - LIST_INIT(&allpmaps); - - /* - * Request a spin mutex so that changes to allpmaps cannot be - * preempted by smp_rendezvous_cpus(). - */ - mtx_init(&allpmaps_lock, "allpmaps", NULL, MTX_SPIN); - mtx_lock_spin(&allpmaps_lock); - LIST_INSERT_HEAD(&allpmaps, kernel_pmap, pm_list); - mtx_unlock_spin(&allpmaps_lock); - - /* - * Reserve some special page table entries/VA space for temporary - * mapping of pages. - */ -#define SYSMAP(c, p, v, n) do { \ - v = (c)pmap_preboot_reserve_pages(n); \ - p = pt2map_entry((vm_offset_t)v); \ - } while (0) - - /* - * Local CMAP1/CMAP2 are used for zeroing and copying pages. - * Local CMAP3 is used for data cache cleaning. - * Global CMAP3 is used for the idle process page zeroing. - */ - for (i = 0; i < MAXCPU; i++) { - sysmaps = &sysmaps_pcpu[i]; - mtx_init(&sysmaps->lock, "SYSMAPS", NULL, MTX_DEF); - SYSMAP(caddr_t, sysmaps->CMAP1, sysmaps->CADDR1, 1); - SYSMAP(caddr_t, sysmaps->CMAP2, sysmaps->CADDR2, 1); - SYSMAP(caddr_t, sysmaps->CMAP3, sysmaps->CADDR3, 1); - } - SYSMAP(caddr_t, CMAP3, CADDR3, 1); - - /* - * Crashdump maps. - */ - SYSMAP(caddr_t, unused, crashdumpmap, MAXDUMPPGS); - - /* - * _tmppt is used for reading arbitrary physical pages via /dev/mem. - */ - SYSMAP(caddr_t, unused, _tmppt, 1); - - /* - * PADDR1 and PADDR2 are used by pmap_pte2_quick() and pmap_pte2(), - * respectively. PADDR3 is used by pmap_pte2_ddb(). - */ - SYSMAP(pt2_entry_t *, PMAP1, PADDR1, 1); - SYSMAP(pt2_entry_t *, PMAP2, PADDR2, 1); -#ifdef DDB - SYSMAP(pt2_entry_t *, PMAP3, PADDR3, 1); -#endif - mtx_init(&PMAP2mutex, "PMAP2", NULL, MTX_DEF); - - /* - * Note that in very short time in initarm(), we are going to - * initialize phys_avail[] array and no futher page allocation - * can happen after that until vm subsystem will be initialized. - */ - kernel_vm_end_new = kernel_vm_end; - virtual_end = vm_max_kernel_address; -} - -static void -pmap_init_qpages(void) -{ - struct pcpu *pc; - int i; - - CPU_FOREACH(i) { - pc = pcpu_find(i); - pc->pc_qmap_addr = kva_alloc(PAGE_SIZE); - if (pc->pc_qmap_addr == 0) - panic("%s: unable to allocate KVA", __func__); - } -} -SYSINIT(qpages_init, SI_SUB_CPU, SI_ORDER_ANY, pmap_init_qpages, NULL); - -/* - * The function can already be use in second initialization stage. - * As such, the function DOES NOT call pmap_growkernel() where PT2 - * allocation can happen. So if used, be sure that PT2 for given - * virtual address is allocated already! - * - * Add a wired page to the kva. - * Note: not SMP coherent. - */ -static __inline void -pmap_kenter_prot_attr(vm_offset_t va, vm_paddr_t pa, uint32_t prot, - uint32_t attr) -{ - pt1_entry_t *pte1p; - pt2_entry_t *pte2p; - - pte1p = kern_pte1(va); - if (!pte1_is_valid(pte1_load(pte1p))) { /* XXX - sections ?! */ - /* - * This is a very low level function, so PT2 and particularly - * PT2PG associated with given virtual address must be already - * allocated. It's a pain mainly during pmap initialization - * stage. However, called after pmap initialization with - * virtual address not under kernel_vm_end will lead to - * the same misery. - */ - if (!pte2_is_valid(pte2_load(kern_pt2tab_entry(va)))) - panic("%s: kernel PT2 not allocated!", __func__); - } - - pte2p = pt2map_entry(va); - pte2_store(pte2p, PTE2_KERN(pa, prot, attr)); -} - -static __inline void -pmap_kenter_attr(vm_offset_t va, vm_paddr_t pa, int attr) -{ - - pmap_kenter_prot_attr(va, pa, PTE2_AP_KRW, attr); -} - -PMAP_INLINE void -pmap_kenter(vm_offset_t va, vm_paddr_t pa) -{ - - pmap_kenter_prot_attr(va, pa, PTE2_AP_KRW, PTE2_ATTR_NORMAL); -} - -/* - * Remove a page from the kernel pagetables. - * Note: not SMP coherent. - */ -PMAP_INLINE void -pmap_kremove(vm_offset_t va) -{ - pt2_entry_t *pte2p; - - pte2p = pt2map_entry(va); - pte2_clear(pte2p); -} - -/* - * Share new kernel PT2PG with all pmaps. - * The caller is responsible for maintaining TLB consistency. - */ -static void -pmap_kenter_pt2tab(vm_offset_t va, pt2_entry_t npte2) -{ - pmap_t pmap; - pt2_entry_t *pte2p; - - mtx_lock_spin(&allpmaps_lock); - LIST_FOREACH(pmap, &allpmaps, pm_list) { - pte2p = pmap_pt2tab_entry(pmap, va); - pt2tab_store(pte2p, npte2); - } - mtx_unlock_spin(&allpmaps_lock); -} - -/* - * Share new kernel PTE1 with all pmaps. - * The caller is responsible for maintaining TLB consistency. - */ -static void -pmap_kenter_pte1(vm_offset_t va, pt1_entry_t npte1) -{ - pmap_t pmap; - pt1_entry_t *pte1p; - - mtx_lock_spin(&allpmaps_lock); - LIST_FOREACH(pmap, &allpmaps, pm_list) { - pte1p = pmap_pte1(pmap, va); - pte1_store(pte1p, npte1); - } - mtx_unlock_spin(&allpmaps_lock); -} - -/* - * Used to map a range of physical addresses into kernel - * virtual address space. - * - * The value passed in '*virt' is a suggested virtual address for - * the mapping. Architectures which can support a direct-mapped - * physical to virtual region can return the appropriate address - * within that region, leaving '*virt' unchanged. Other - * architectures should map the pages starting at '*virt' and - * update '*virt' with the first usable address after the mapped - * region. - * - * NOTE: Read the comments above pmap_kenter_prot_attr() as - * the function is used herein! - */ -vm_offset_t -pmap_map(vm_offset_t *virt, vm_paddr_t start, vm_paddr_t end, int prot) -{ - vm_offset_t va, sva; - vm_paddr_t pte1_offset; - pt1_entry_t npte1; - u_int l1prot,l2prot; - - PDEBUG(1, printf("%s: virt = %#x, start = %#x, end = %#x (size = %#x)," - " prot = %d\n", __func__, *virt, start, end, end - start, prot)); - - l2prot = (prot & VM_PROT_WRITE) ? PTE2_AP_KRW : PTE1_AP_KR; - l2prot |= (prot & VM_PROT_EXECUTE) ? PTE2_X : PTE2_NX; - l1prot = ATTR_TO_L1(l2prot); - - va = *virt; - /* - * Does the physical address range's size and alignment permit at - * least one section mapping to be created? - */ - pte1_offset = start & PTE1_OFFSET; - if ((end - start) - ((PTE1_SIZE - pte1_offset) & PTE1_OFFSET) >= - PTE1_SIZE) { - /* - * Increase the starting virtual address so that its alignment - * does not preclude the use of section mappings. - */ - if ((va & PTE1_OFFSET) < pte1_offset) - va = pte1_trunc(va) + pte1_offset; - else if ((va & PTE1_OFFSET) > pte1_offset) - va = pte1_roundup(va) + pte1_offset; - } - sva = va; - while (start < end) { - if ((start & PTE1_OFFSET) == 0 && end - start >= PTE1_SIZE) { - KASSERT((va & PTE1_OFFSET) == 0, - ("%s: misaligned va %#x", __func__, va)); - npte1 = PTE1_KERN(start, l1prot, PTE1_ATTR_NORMAL); - pmap_kenter_pte1(va, npte1); - va += PTE1_SIZE; - start += PTE1_SIZE; - } else { - pmap_kenter_prot_attr(va, start, l2prot, - PTE2_ATTR_NORMAL); - va += PAGE_SIZE; - start += PAGE_SIZE; - } - } - tlb_flush_range(sva, va - sva); - *virt = va; - return (sva); -} - -/* - * Make a temporary mapping for a physical address. - * This is only intended to be used for panic dumps. - */ -void * -pmap_kenter_temporary(vm_paddr_t pa, int i) -{ - vm_offset_t va; - - /* QQQ: 'i' should be less or equal to MAXDUMPPGS. */ - - va = (vm_offset_t)crashdumpmap + (i * PAGE_SIZE); - pmap_kenter(va, pa); - tlb_flush_local(va); - return ((void *)crashdumpmap); -} - - -/************************************* - * - * TLB & cache maintenance routines. - * - *************************************/ - -/* - * We inline these within pmap.c for speed. - */ -PMAP_INLINE void -pmap_tlb_flush(pmap_t pmap, vm_offset_t va) -{ - - if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) - tlb_flush(va); -} - -PMAP_INLINE void -pmap_tlb_flush_range(pmap_t pmap, vm_offset_t sva, vm_size_t size) -{ - - if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) - tlb_flush_range(sva, size); -} - -/* - * Abuse the pte2 nodes for unmapped kva to thread a kva freelist through. - * Requirements: - * - Must deal with pages in order to ensure that none of the PTE2_* bits - * are ever set, PTE2_V in particular. - * - Assumes we can write to pte2s without pte2_store() atomic ops. - * - Assumes nothing will ever test these addresses for 0 to indicate - * no mapping instead of correctly checking PTE2_V. - * - Assumes a vm_offset_t will fit in a pte2 (true for arm). - * Because PTE2_V is never set, there can be no mappings to invalidate. - */ -static vm_offset_t -pmap_pte2list_alloc(vm_offset_t *head) -{ - pt2_entry_t *pte2p; - vm_offset_t va; - - va = *head; - if (va == 0) - panic("pmap_ptelist_alloc: exhausted ptelist KVA"); - pte2p = pt2map_entry(va); - *head = *pte2p; - if (*head & PTE2_V) - panic("%s: va with PTE2_V set!", __func__); - *pte2p = 0; - return (va); -} - -static void -pmap_pte2list_free(vm_offset_t *head, vm_offset_t va) -{ - pt2_entry_t *pte2p; - - if (va & PTE2_V) - panic("%s: freeing va with PTE2_V set!", __func__); - pte2p = pt2map_entry(va); - *pte2p = *head; /* virtual! PTE2_V is 0 though */ - *head = va; -} - -static void -pmap_pte2list_init(vm_offset_t *head, void *base, int npages) -{ - int i; - vm_offset_t va; - - *head = 0; - for (i = npages - 1; i >= 0; i--) { - va = (vm_offset_t)base + i * PAGE_SIZE; - pmap_pte2list_free(head, va); - } -} - -/***************************************************************************** - * - * PMAP third and final stage initialization. - * - * After pmap_init() is called, PMAP subsystem is fully initialized. - * - *****************************************************************************/ - -SYSCTL_NODE(_vm, OID_AUTO, pmap, CTLFLAG_RD, 0, "VM/pmap parameters"); - -SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_max, CTLFLAG_RD, &pv_entry_max, 0, - "Max number of PV entries"); -SYSCTL_INT(_vm_pmap, OID_AUTO, shpgperproc, CTLFLAG_RD, &shpgperproc, 0, - "Page share factor per proc"); - -static u_long nkpt2pg = NKPT2PG; -SYSCTL_ULONG(_vm_pmap, OID_AUTO, nkpt2pg, CTLFLAG_RD, - &nkpt2pg, 0, "Pre-allocated pages for kernel PT2s"); - -static int sp_enabled = 1; -SYSCTL_INT(_vm_pmap, OID_AUTO, sp_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, - &sp_enabled, 0, "Are large page mappings enabled?"); - -static SYSCTL_NODE(_vm_pmap, OID_AUTO, pte1, CTLFLAG_RD, 0, - "1MB page mapping counters"); - -static u_long pmap_pte1_demotions; -SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, demotions, CTLFLAG_RD, - &pmap_pte1_demotions, 0, "1MB page demotions"); - -static u_long pmap_pte1_mappings; -SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, mappings, CTLFLAG_RD, - &pmap_pte1_mappings, 0, "1MB page mappings"); - -static u_long pmap_pte1_p_failures; -SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, p_failures, CTLFLAG_RD, - &pmap_pte1_p_failures, 0, "1MB page promotion failures"); - -static u_long pmap_pte1_promotions; -SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, promotions, CTLFLAG_RD, - &pmap_pte1_promotions, 0, "1MB page promotions"); - -static __inline ttb_entry_t -pmap_ttb_get(pmap_t pmap) -{ - - return (vtophys(pmap->pm_pt1) | ttb_flags); -} - -/* - * Initialize a vm_page's machine-dependent fields. - * - * Variations: - * 1. Pages for L2 page tables are always not managed. So, pv_list and - * pt2_wirecount can share same physical space. However, proper - * initialization on a page alloc for page tables and reinitialization - * on the page free must be ensured. - */ -void -pmap_page_init(vm_page_t m) -{ - - TAILQ_INIT(&m->md.pv_list); - pt2_wirecount_init(m); - m->md.pat_mode = PTE2_ATTR_NORMAL; -} - -/* - * Virtualization for faster way how to zero whole page. - */ -static __inline void -pagezero(void *page) -{ - - bzero(page, PAGE_SIZE); -} - -/* - * Zero L2 page table page. - * Use same KVA as in pmap_zero_page(). - */ -static __inline vm_paddr_t -pmap_pt2pg_zero(vm_page_t m) -{ - vm_paddr_t pa; - struct sysmaps *sysmaps; - - pa = VM_PAGE_TO_PHYS(m); - - /* - * XXX: For now, we map whole page even if it's already zero, - * to sync it even if the sync is only DSB. - */ - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (pte2_load(sysmaps->CMAP2) != 0) - panic("%s: CMAP2 busy", __func__); - pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(pa, PTE2_AP_KRW, - m->md.pat_mode)); - /* Even VM_ALLOC_ZERO request is only advisory. */ - if ((m->flags & PG_ZERO) == 0) - pagezero(sysmaps->CADDR2); - pte2_sync_range((pt2_entry_t *)sysmaps->CADDR2, PAGE_SIZE); - pte2_clear(sysmaps->CMAP2); - tlb_flush((vm_offset_t)sysmaps->CADDR2); - sched_unpin(); - mtx_unlock(&sysmaps->lock); - - return (pa); -} - -/* - * Init just allocated page as L2 page table(s) holder - * and return its physical address. - */ -static __inline vm_paddr_t -pmap_pt2pg_init(pmap_t pmap, vm_offset_t va, vm_page_t m) -{ - vm_paddr_t pa; - pt2_entry_t *pte2p; - - /* Check page attributes. */ - if (pmap_page_get_memattr(m) != pt_memattr) - pmap_page_set_memattr(m, pt_memattr); - - /* Zero page and init wire counts. */ - pa = pmap_pt2pg_zero(m); - pt2_wirecount_init(m); - - /* - * Map page to PT2MAP address space for given pmap. - * Note that PT2MAP space is shared with all pmaps. - */ - if (pmap == kernel_pmap) - pmap_kenter_pt2tab(va, PTE2_KPT(pa)); - else { - pte2p = pmap_pt2tab_entry(pmap, va); - pt2tab_store(pte2p, PTE2_KPT_NG(pa)); - } - - return (pa); -} - -/* - * Initialize the pmap module. - * Called by vm_init, to initialize any structures that the pmap - * system needs to map virtual memory. - */ -void -pmap_init(void) -{ - vm_size_t s; - pt2_entry_t *pte2p, pte2; - u_int i, pte1_idx, pv_npg; - - PDEBUG(1, printf("%s: phys_start = %#x\n", __func__, PHYSADDR)); - - /* - * Initialize the vm page array entries for kernel pmap's - * L2 page table pages allocated in advance. - */ - pte1_idx = pte1_index(KERNBASE - PT2MAP_SIZE); - pte2p = kern_pt2tab_entry(KERNBASE - PT2MAP_SIZE); - for (i = 0; i < nkpt2pg + NPG_IN_PT2TAB; i++, pte2p++) { - vm_paddr_t pa; - vm_page_t m; - - pte2 = pte2_load(pte2p); - KASSERT(pte2_is_valid(pte2), ("%s: no valid entry", __func__)); - - pa = pte2_pa(pte2); - m = PHYS_TO_VM_PAGE(pa); - KASSERT(m >= vm_page_array && - m < &vm_page_array[vm_page_array_size], - ("%s: L2 page table page is out of range", __func__)); - - m->pindex = pte1_idx; - m->phys_addr = pa; - pte1_idx += NPT2_IN_PG; - } - - /* - * Initialize the address space (zone) for the pv entries. Set a - * high water mark so that the system can recover from excessive - * numbers of pv entries. - */ - TUNABLE_INT_FETCH("vm.pmap.shpgperproc", &shpgperproc); - pv_entry_max = shpgperproc * maxproc + vm_cnt.v_page_count; - TUNABLE_INT_FETCH("vm.pmap.pv_entries", &pv_entry_max); - pv_entry_max = roundup(pv_entry_max, _NPCPV); - pv_entry_high_water = 9 * (pv_entry_max / 10); - - /* - * Are large page mappings enabled? - */ - TUNABLE_INT_FETCH("vm.pmap.sp_enabled", &sp_enabled); - if (sp_enabled) { - KASSERT(MAXPAGESIZES > 1 && pagesizes[1] == 0, - ("%s: can't assign to pagesizes[1]", __func__)); - pagesizes[1] = PTE1_SIZE; - } - - /* - * Calculate the size of the pv head table for sections. - * Handle the possibility that "vm_phys_segs[...].end" is zero. - * Note that the table is only for sections which could be promoted. - */ - first_managed_pa = pte1_trunc(vm_phys_segs[0].start); - pv_npg = (pte1_trunc(vm_phys_segs[vm_phys_nsegs - 1].end - PAGE_SIZE) - - first_managed_pa) / PTE1_SIZE + 1; - - /* - * Allocate memory for the pv head table for sections. - */ - s = (vm_size_t)(pv_npg * sizeof(struct md_page)); - s = round_page(s); - pv_table = (struct md_page *)kmem_malloc(kernel_arena, s, - M_WAITOK | M_ZERO); - for (i = 0; i < pv_npg; i++) - TAILQ_INIT(&pv_table[i].pv_list); - - pv_maxchunks = MAX(pv_entry_max / _NPCPV, maxproc); - pv_chunkbase = (struct pv_chunk *)kva_alloc(PAGE_SIZE * pv_maxchunks); - if (pv_chunkbase == NULL) - panic("%s: not enough kvm for pv chunks", __func__); - pmap_pte2list_init(&pv_vafree, pv_chunkbase, pv_maxchunks); -} - -/* - * Add a list of wired pages to the kva - * this routine is only used for temporary - * kernel mappings that do not need to have - * page modification or references recorded. - * Note that old mappings are simply written - * over. The page *must* be wired. - * Note: SMP coherent. Uses a ranged shootdown IPI. - */ -void -pmap_qenter(vm_offset_t sva, vm_page_t *ma, int count) -{ - u_int anychanged; - pt2_entry_t *epte2p, *pte2p, pte2; - vm_page_t m; - vm_paddr_t pa; - - anychanged = 0; - pte2p = pt2map_entry(sva); - epte2p = pte2p + count; - while (pte2p < epte2p) { - m = *ma++; - pa = VM_PAGE_TO_PHYS(m); - pte2 = pte2_load(pte2p); - if ((pte2_pa(pte2) != pa) || - (pte2_attr(pte2) != m->md.pat_mode)) { - anychanged++; - pte2_store(pte2p, PTE2_KERN(pa, PTE2_AP_KRW, - m->md.pat_mode)); - } - pte2p++; - } - if (__predict_false(anychanged)) - tlb_flush_range(sva, count * PAGE_SIZE); -} - -/* - * This routine tears out page mappings from the - * kernel -- it is meant only for temporary mappings. - * Note: SMP coherent. Uses a ranged shootdown IPI. - */ -void -pmap_qremove(vm_offset_t sva, int count) -{ - vm_offset_t va; - - va = sva; - while (count-- > 0) { - pmap_kremove(va); - va += PAGE_SIZE; - } - tlb_flush_range(sva, va - sva); -} - -/* - * Are we current address space or kernel? - */ -static __inline int -pmap_is_current(pmap_t pmap) -{ - - return (pmap == kernel_pmap || - (pmap == vmspace_pmap(curthread->td_proc->p_vmspace))); -} - -/* - * If the given pmap is not the current or kernel pmap, the returned - * pte2 must be released by passing it to pmap_pte2_release(). - */ -static pt2_entry_t * -pmap_pte2(pmap_t pmap, vm_offset_t va) -{ - pt1_entry_t pte1; - vm_paddr_t pt2pg_pa; - - pte1 = pte1_load(pmap_pte1(pmap, va)); - if (pte1_is_section(pte1)) - panic("%s: attempt to map PTE1", __func__); - if (pte1_is_link(pte1)) { - /* Are we current address space or kernel? */ - if (pmap_is_current(pmap)) - return (pt2map_entry(va)); - /* Note that L2 page table size is not equal to PAGE_SIZE. */ - pt2pg_pa = trunc_page(pte1_link_pa(pte1)); - mtx_lock(&PMAP2mutex); - if (pte2_pa(pte2_load(PMAP2)) != pt2pg_pa) { - pte2_store(PMAP2, PTE2_KPT(pt2pg_pa)); - tlb_flush((vm_offset_t)PADDR2); - } - return (PADDR2 + (arm32_btop(va) & (NPTE2_IN_PG - 1))); - } - return (NULL); -} - -/* - * Releases a pte2 that was obtained from pmap_pte2(). - * Be prepared for the pte2p being NULL. - */ -static __inline void -pmap_pte2_release(pt2_entry_t *pte2p) -{ - - if ((pt2_entry_t *)(trunc_page((vm_offset_t)pte2p)) == PADDR2) { - mtx_unlock(&PMAP2mutex); - } -} - -/* - * Super fast pmap_pte2 routine best used when scanning - * the pv lists. This eliminates many coarse-grained - * invltlb calls. Note that many of the pv list - * scans are across different pmaps. It is very wasteful - * to do an entire tlb flush for checking a single mapping. - * - * If the given pmap is not the current pmap, pvh_global_lock - * must be held and curthread pinned to a CPU. - */ -static pt2_entry_t * -pmap_pte2_quick(pmap_t pmap, vm_offset_t va) -{ - pt1_entry_t pte1; - vm_paddr_t pt2pg_pa; - - pte1 = pte1_load(pmap_pte1(pmap, va)); - if (pte1_is_section(pte1)) - panic("%s: attempt to map PTE1", __func__); - if (pte1_is_link(pte1)) { - /* Are we current address space or kernel? */ - if (pmap_is_current(pmap)) - return (pt2map_entry(va)); - rw_assert(&pvh_global_lock, RA_WLOCKED); - KASSERT(curthread->td_pinned > 0, - ("%s: curthread not pinned", __func__)); - /* Note that L2 page table size is not equal to PAGE_SIZE. */ - pt2pg_pa = trunc_page(pte1_link_pa(pte1)); - if (pte2_pa(pte2_load(PMAP1)) != pt2pg_pa) { - pte2_store(PMAP1, PTE2_KPT(pt2pg_pa)); -#ifdef SMP - PMAP1cpu = PCPU_GET(cpuid); -#endif - tlb_flush_local((vm_offset_t)PADDR1); - PMAP1changed++; - } else -#ifdef SMP - if (PMAP1cpu != PCPU_GET(cpuid)) { - PMAP1cpu = PCPU_GET(cpuid); - tlb_flush_local((vm_offset_t)PADDR1); - PMAP1changedcpu++; - } else -#endif - PMAP1unchanged++; - return (PADDR1 + (arm32_btop(va) & (NPTE2_IN_PG - 1))); - } - return (NULL); -} - -/* - * Routine: pmap_extract - * Function: - * Extract the physical page address associated - * with the given map/virtual_address pair. - */ -vm_paddr_t -pmap_extract(pmap_t pmap, vm_offset_t va) -{ - vm_paddr_t pa; - pt1_entry_t pte1; - pt2_entry_t *pte2p; - - PMAP_LOCK(pmap); - pte1 = pte1_load(pmap_pte1(pmap, va)); - if (pte1_is_section(pte1)) - pa = pte1_pa(pte1) | (va & PTE1_OFFSET); - else if (pte1_is_link(pte1)) { - pte2p = pmap_pte2(pmap, va); - pa = pte2_pa(pte2_load(pte2p)) | (va & PTE2_OFFSET); - pmap_pte2_release(pte2p); - } else - pa = 0; - PMAP_UNLOCK(pmap); - return (pa); -} - -/* - * Routine: pmap_extract_and_hold - * Function: - * Atomically extract and hold the physical page - * with the given pmap and virtual address pair - * if that mapping permits the given protection. - */ -vm_page_t -pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) -{ - vm_paddr_t pa, lockpa; - pt1_entry_t pte1; - pt2_entry_t pte2, *pte2p; - vm_page_t m; - - lockpa = 0; - m = NULL; - PMAP_LOCK(pmap); -retry: - pte1 = pte1_load(pmap_pte1(pmap, va)); - if (pte1_is_section(pte1)) { - if (!(pte1 & PTE1_RO) || !(prot & VM_PROT_WRITE)) { - pa = pte1_pa(pte1) | (va & PTE1_OFFSET); - if (vm_page_pa_tryrelock(pmap, pa, &lockpa)) - goto retry; - m = PHYS_TO_VM_PAGE(pa); - vm_page_hold(m); - } - } else if (pte1_is_link(pte1)) { - pte2p = pmap_pte2(pmap, va); - pte2 = pte2_load(pte2p); - pmap_pte2_release(pte2p); - if (pte2_is_valid(pte2) && - (!(pte2 & PTE2_RO) || !(prot & VM_PROT_WRITE))) { - pa = pte2_pa(pte2); - if (vm_page_pa_tryrelock(pmap, pa, &lockpa)) - goto retry; - m = PHYS_TO_VM_PAGE(pa); - vm_page_hold(m); - } - } - PA_UNLOCK_COND(lockpa); - PMAP_UNLOCK(pmap); - return (m); -} - -/* - * Grow the number of kernel L2 page table entries, if needed. - */ -void -pmap_growkernel(vm_offset_t addr) -{ - vm_page_t m; - vm_paddr_t pt2pg_pa, pt2_pa; - pt1_entry_t pte1; - pt2_entry_t pte2; - - PDEBUG(1, printf("%s: addr = %#x\n", __func__, addr)); - /* - * All the time kernel_vm_end is first KVA for which underlying - * L2 page table is either not allocated or linked from L1 page table - * (not considering sections). Except for two possible cases: - * - * (1) in the very beginning as long as pmap_growkernel() was - * not called, it could be first unused KVA (which is not - * rounded up to PTE1_SIZE), - * - * (2) when all KVA space is mapped and kernel_map->max_offset - * address is not rounded up to PTE1_SIZE. (For example, - * it could be 0xFFFFFFFF.) - */ - kernel_vm_end = pte1_roundup(kernel_vm_end); - mtx_assert(&kernel_map->system_mtx, MA_OWNED); - addr = roundup2(addr, PTE1_SIZE); - if (addr - 1 >= kernel_map->max_offset) - addr = kernel_map->max_offset; - while (kernel_vm_end < addr) { - pte1 = pte1_load(kern_pte1(kernel_vm_end)); - if (pte1_is_valid(pte1)) { - kernel_vm_end += PTE1_SIZE; - if (kernel_vm_end - 1 >= kernel_map->max_offset) { - kernel_vm_end = kernel_map->max_offset; - break; - } - continue; - } - - /* - * kernel_vm_end_new is used in pmap_pinit() when kernel - * mappings are entered to new pmap all at once to avoid race - * between pmap_kenter_pte1() and kernel_vm_end increase. - * The same aplies to pmap_kenter_pt2tab(). - */ - kernel_vm_end_new = kernel_vm_end + PTE1_SIZE; - - pte2 = pt2tab_load(kern_pt2tab_entry(kernel_vm_end)); - if (!pte2_is_valid(pte2)) { - /* - * Install new PT2s page into kernel PT2TAB. - */ - m = vm_page_alloc(NULL, - pte1_index(kernel_vm_end) & ~PT2PG_MASK, - VM_ALLOC_INTERRUPT | VM_ALLOC_NOOBJ | - VM_ALLOC_WIRED | VM_ALLOC_ZERO); - if (m == NULL) - panic("%s: no memory to grow kernel", __func__); - /* - * QQQ: To link all new L2 page tables from L1 page - * table now and so pmap_kenter_pte1() them - * at once together with pmap_kenter_pt2tab() - * could be nice speed up. However, - * pmap_growkernel() does not happen so often... - * QQQ: The other TTBR is another option. - */ - pt2pg_pa = pmap_pt2pg_init(kernel_pmap, kernel_vm_end, - m); - } else - pt2pg_pa = pte2_pa(pte2); - - pt2_pa = page_pt2pa(pt2pg_pa, pte1_index(kernel_vm_end)); - pmap_kenter_pte1(kernel_vm_end, PTE1_LINK(pt2_pa)); - - kernel_vm_end = kernel_vm_end_new; - if (kernel_vm_end - 1 >= kernel_map->max_offset) { - kernel_vm_end = kernel_map->max_offset; - break; - } - } -} - -static int -kvm_size(SYSCTL_HANDLER_ARGS) -{ - unsigned long ksize = vm_max_kernel_address - KERNBASE; - - return (sysctl_handle_long(oidp, &ksize, 0, req)); -} -SYSCTL_PROC(_vm, OID_AUTO, kvm_size, CTLTYPE_LONG|CTLFLAG_RD, - 0, 0, kvm_size, "IU", "Size of KVM"); - -static int -kvm_free(SYSCTL_HANDLER_ARGS) -{ - unsigned long kfree = vm_max_kernel_address - kernel_vm_end; - - return (sysctl_handle_long(oidp, &kfree, 0, req)); -} -SYSCTL_PROC(_vm, OID_AUTO, kvm_free, CTLTYPE_LONG|CTLFLAG_RD, - 0, 0, kvm_free, "IU", "Amount of KVM free"); - -/*********************************************** - * - * Pmap allocation/deallocation routines. - * - ***********************************************/ - -/* - * Initialize the pmap for the swapper process. - */ -void -pmap_pinit0(pmap_t pmap) -{ - PDEBUG(1, printf("%s: pmap = %p\n", __func__, pmap)); - - PMAP_LOCK_INIT(pmap); - - /* - * Kernel page table directory and pmap stuff around is already - * initialized, we are using it right now and here. So, finish - * only PMAP structures initialization for process0 ... - * - * Since the L1 page table and PT2TAB is shared with the kernel pmap, - * which is already included in the list "allpmaps", this pmap does - * not need to be inserted into that list. - */ - pmap->pm_pt1 = kern_pt1; - pmap->pm_pt2tab = kern_pt2tab; - CPU_ZERO(&pmap->pm_active); - PCPU_SET(curpmap, pmap); - TAILQ_INIT(&pmap->pm_pvchunk); - bzero(&pmap->pm_stats, sizeof pmap->pm_stats); - CPU_SET(0, &pmap->pm_active); -} - -static __inline void -pte1_copy_nosync(pt1_entry_t *spte1p, pt1_entry_t *dpte1p, vm_offset_t sva, - vm_offset_t eva) -{ - u_int idx, count; - - idx = pte1_index(sva); - count = (pte1_index(eva) - idx + 1) * sizeof(pt1_entry_t); - bcopy(spte1p + idx, dpte1p + idx, count); -} - -static __inline void -pt2tab_copy_nosync(pt2_entry_t *spte2p, pt2_entry_t *dpte2p, vm_offset_t sva, - vm_offset_t eva) -{ - u_int idx, count; - - idx = pt2tab_index(sva); - count = (pt2tab_index(eva) - idx + 1) * sizeof(pt2_entry_t); - bcopy(spte2p + idx, dpte2p + idx, count); -} - -/* - * Initialize a preallocated and zeroed pmap structure, - * such as one in a vmspace structure. - */ -int -pmap_pinit(pmap_t pmap) -{ - pt1_entry_t *pte1p; - pt2_entry_t *pte2p; - vm_paddr_t pa, pt2tab_pa; - u_int i; - - PDEBUG(6, printf("%s: pmap = %p, pm_pt1 = %p\n", __func__, pmap, - pmap->pm_pt1)); - - /* - * No need to allocate L2 page table space yet but we do need - * a valid L1 page table and PT2TAB table. - * - * Install shared kernel mappings to these tables. It's a little - * tricky as some parts of KVA are reserved for vectors, devices, - * and whatever else. These parts are supposed to be above - * vm_max_kernel_address. Thus two regions should be installed: - * - * (1) . - * - * QQQ: The second region should be stable enough to be installed - * only once in time when the tables are allocated. - * QQQ: Maybe copy of both regions at once could be faster ... - * QQQ: Maybe the other TTBR is an option. - * - * Finally, install own PT2TAB table to these tables. - */ - - if (pmap->pm_pt1 == NULL) { - pmap->pm_pt1 = (pt1_entry_t *)kmem_alloc_contig(kernel_arena, - NB_IN_PT1, M_NOWAIT | M_ZERO, 0, -1UL, NB_IN_PT1, 0, - pt_memattr); - if (pmap->pm_pt1 == NULL) - return (0); - } - if (pmap->pm_pt2tab == NULL) { - /* - * QQQ: (1) PT2TAB must be contiguous. If PT2TAB is one page - * only, what should be the only size for 32 bit systems, - * then we could allocate it with vm_page_alloc() and all - * the stuff needed as other L2 page table pages. - * (2) Note that a process PT2TAB is special L2 page table - * page. Its mapping in kernel_arena is permanent and can - * be used no matter which process is current. Its mapping - * in PT2MAP can be used only for current process. - */ - pmap->pm_pt2tab = (pt2_entry_t *)kmem_alloc_attr(kernel_arena, - NB_IN_PT2TAB, M_NOWAIT | M_ZERO, 0, -1UL, pt_memattr); - if (pmap->pm_pt2tab == NULL) { - /* - * QQQ: As struct pmap is allocated from UMA with - * UMA_ZONE_NOFREE flag, it's important to leave - * no allocation in pmap if initialization failed. - */ - kmem_free(kernel_arena, (vm_offset_t)pmap->pm_pt1, - NB_IN_PT1); - pmap->pm_pt1 = NULL; - return (0); - } - /* - * QQQ: Each L2 page table page vm_page_t has pindex set to - * pte1 index of virtual address mapped by this page. - * It's not valid for non kernel PT2TABs themselves. - * The pindex of these pages can not be altered because - * of the way how they are allocated now. However, it - * should not be a problem. - */ - } - - mtx_lock_spin(&allpmaps_lock); - /* - * To avoid race with pmap_kenter_pte1() and pmap_kenter_pt2tab(), - * kernel_vm_end_new is used here instead of kernel_vm_end. - */ - pte1_copy_nosync(kern_pt1, pmap->pm_pt1, KERNBASE, - kernel_vm_end_new - 1); - pte1_copy_nosync(kern_pt1, pmap->pm_pt1, vm_max_kernel_address, - 0xFFFFFFFF); - pt2tab_copy_nosync(kern_pt2tab, pmap->pm_pt2tab, KERNBASE, - kernel_vm_end_new - 1); - pt2tab_copy_nosync(kern_pt2tab, pmap->pm_pt2tab, vm_max_kernel_address, - 0xFFFFFFFF); - LIST_INSERT_HEAD(&allpmaps, pmap, pm_list); - mtx_unlock_spin(&allpmaps_lock); - - /* - * Store PT2MAP PT2 pages (a.k.a. PT2TAB) in PT2TAB itself. - * I.e. self reference mapping. The PT2TAB is private, however mapped - * into shared PT2MAP space, so the mapping should be not global. - */ - pt2tab_pa = vtophys(pmap->pm_pt2tab); - pte2p = pmap_pt2tab_entry(pmap, (vm_offset_t)PT2MAP); - for (pa = pt2tab_pa, i = 0; i < NPG_IN_PT2TAB; i++, pa += PTE2_SIZE) { - pt2tab_store(pte2p++, PTE2_KPT_NG(pa)); - } - - /* Insert PT2MAP PT2s into pmap PT1. */ - pte1p = pmap_pte1(pmap, (vm_offset_t)PT2MAP); - for (pa = pt2tab_pa, i = 0; i < NPT2_IN_PT2TAB; i++, pa += NB_IN_PT2) { - pte1_store(pte1p++, PTE1_LINK(pa)); - } - - /* - * Now synchronize new mapping which was made above. - */ - pte1_sync_range(pmap->pm_pt1, NB_IN_PT1); - pte2_sync_range(pmap->pm_pt2tab, NB_IN_PT2TAB); - - CPU_ZERO(&pmap->pm_active); - TAILQ_INIT(&pmap->pm_pvchunk); - bzero(&pmap->pm_stats, sizeof pmap->pm_stats); - - return (1); -} - -#ifdef INVARIANTS -static boolean_t -pt2tab_user_is_empty(pt2_entry_t *tab) -{ - u_int i, end; - - end = pt2tab_index(VM_MAXUSER_ADDRESS); - for (i = 0; i < end; i++) - if (tab[i] != 0) return (FALSE); - return (TRUE); -} -#endif -/* - * Release any resources held by the given physical map. - * Called when a pmap initialized by pmap_pinit is being released. - * Should only be called if the map contains no valid mappings. - */ -void -pmap_release(pmap_t pmap) -{ -#ifdef INVARIANTS - vm_offset_t start, end; -#endif - KASSERT(pmap->pm_stats.resident_count == 0, - ("%s: pmap resident count %ld != 0", __func__, - pmap->pm_stats.resident_count)); - KASSERT(pt2tab_user_is_empty(pmap->pm_pt2tab), - ("%s: has allocated user PT2(s)", __func__)); - KASSERT(CPU_EMPTY(&pmap->pm_active), - ("%s: pmap %p is active on some CPU(s)", __func__, pmap)); - - mtx_lock_spin(&allpmaps_lock); - LIST_REMOVE(pmap, pm_list); - mtx_unlock_spin(&allpmaps_lock); - -#ifdef INVARIANTS - start = pte1_index(KERNBASE) * sizeof(pt1_entry_t); - end = (pte1_index(0xFFFFFFFF) + 1) * sizeof(pt1_entry_t); - bzero((char *)pmap->pm_pt1 + start, end - start); - - start = pt2tab_index(KERNBASE) * sizeof(pt2_entry_t); - end = (pt2tab_index(0xFFFFFFFF) + 1) * sizeof(pt2_entry_t); - bzero((char *)pmap->pm_pt2tab + start, end - start); -#endif - /* - * We are leaving PT1 and PT2TAB allocated on released pmap, - * so hopefully UMA vmspace_zone will always be inited with - * UMA_ZONE_NOFREE flag. - */ -} - -/********************************************************* - * - * L2 table pages and their pages management routines. - * - *********************************************************/ - -/* - * Virtual interface for L2 page table wire counting. - * - * Each L2 page table in a page has own counter which counts a number of - * valid mappings in a table. Global page counter counts mappings in all - * tables in a page plus a single itself mapping in PT2TAB. - * - * During a promotion we leave the associated L2 page table counter - * untouched, so the table (strictly speaking a page which holds it) - * is never freed if promoted. - * - * If a page m->wire_count == 1 then no valid mappings exist in any L2 page - * table in the page and the page itself is only mapped in PT2TAB. - */ - -static __inline void -pt2_wirecount_init(vm_page_t m) -{ - u_int i; - - /* - * Note: A page m is allocated with VM_ALLOC_WIRED flag and - * m->wire_count should be already set correctly. - * So, there is no need to set it again herein. - */ - for (i = 0; i < NPT2_IN_PG; i++) - m->md.pt2_wirecount[i] = 0; -} - -static __inline void -pt2_wirecount_inc(vm_page_t m, uint32_t pte1_idx) -{ - - /* - * Note: A just modificated pte2 (i.e. already allocated) - * is acquiring one extra reference which must be - * explicitly cleared. It influences the KASSERTs herein. - * All L2 page tables in a page always belong to the same - * pmap, so we allow only one extra reference for the page. - */ - KASSERT(m->md.pt2_wirecount[pte1_idx & PT2PG_MASK] < (NPTE2_IN_PT2 + 1), - ("%s: PT2 is overflowing ...", __func__)); - KASSERT(m->wire_count <= (NPTE2_IN_PG + 1), - ("%s: PT2PG is overflowing ...", __func__)); - - m->wire_count++; - m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]++; -} - -static __inline void -pt2_wirecount_dec(vm_page_t m, uint32_t pte1_idx) -{ - - KASSERT(m->md.pt2_wirecount[pte1_idx & PT2PG_MASK] != 0, - ("%s: PT2 is underflowing ...", __func__)); - KASSERT(m->wire_count > 1, - ("%s: PT2PG is underflowing ...", __func__)); - - m->wire_count--; - m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]--; -} - -static __inline void -pt2_wirecount_set(vm_page_t m, uint32_t pte1_idx, uint16_t count) -{ - - KASSERT(count <= NPTE2_IN_PT2, - ("%s: invalid count %u", __func__, count)); - KASSERT(m->wire_count > m->md.pt2_wirecount[pte1_idx & PT2PG_MASK], - ("%s: PT2PG corrupting (%u, %u) ...", __func__, m->wire_count, - m->md.pt2_wirecount[pte1_idx & PT2PG_MASK])); - - m->wire_count -= m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]; - m->wire_count += count; - m->md.pt2_wirecount[pte1_idx & PT2PG_MASK] = count; - - KASSERT(m->wire_count <= (NPTE2_IN_PG + 1), - ("%s: PT2PG is overflowed (%u) ...", __func__, m->wire_count)); -} - -static __inline uint32_t -pt2_wirecount_get(vm_page_t m, uint32_t pte1_idx) -{ - - return (m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]); -} - -static __inline boolean_t -pt2_is_empty(vm_page_t m, vm_offset_t va) -{ - - return (m->md.pt2_wirecount[pte1_index(va) & PT2PG_MASK] == 0); -} - -static __inline boolean_t -pt2_is_full(vm_page_t m, vm_offset_t va) -{ - - return (m->md.pt2_wirecount[pte1_index(va) & PT2PG_MASK] == - NPTE2_IN_PT2); -} - -static __inline boolean_t -pt2pg_is_empty(vm_page_t m) -{ - - return (m->wire_count == 1); -} - -/* - * This routine is called if the L2 page table - * is not mapped correctly. - */ -static vm_page_t -_pmap_allocpte2(pmap_t pmap, vm_offset_t va, u_int flags) -{ - uint32_t pte1_idx; - pt1_entry_t *pte1p; - pt2_entry_t pte2; - vm_page_t m; - vm_paddr_t pt2pg_pa, pt2_pa; - - pte1_idx = pte1_index(va); - pte1p = pmap->pm_pt1 + pte1_idx; - - KASSERT(pte1_load(pte1p) == 0, - ("%s: pm_pt1[%#x] is not zero: %#x", __func__, pte1_idx, - pte1_load(pte1p))); - - pte2 = pt2tab_load(pmap_pt2tab_entry(pmap, va)); - if (!pte2_is_valid(pte2)) { - /* - * Install new PT2s page into pmap PT2TAB. - */ - m = vm_page_alloc(NULL, pte1_idx & ~PT2PG_MASK, - VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO); - if (m == NULL) { - if ((flags & PMAP_ENTER_NOSLEEP) == 0) { - PMAP_UNLOCK(pmap); - rw_wunlock(&pvh_global_lock); - VM_WAIT; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - } - - /* - * Indicate the need to retry. While waiting, - * the L2 page table page may have been allocated. - */ - return (NULL); - } - pmap->pm_stats.resident_count++; - pt2pg_pa = pmap_pt2pg_init(pmap, va, m); - } else { - pt2pg_pa = pte2_pa(pte2); - m = PHYS_TO_VM_PAGE(pt2pg_pa); - } - - pt2_wirecount_inc(m, pte1_idx); - pt2_pa = page_pt2pa(pt2pg_pa, pte1_idx); - pte1_store(pte1p, PTE1_LINK(pt2_pa)); - - return (m); -} - -static vm_page_t -pmap_allocpte2(pmap_t pmap, vm_offset_t va, u_int flags) -{ - u_int pte1_idx; - pt1_entry_t *pte1p, pte1; - vm_page_t m; - - pte1_idx = pte1_index(va); -retry: - pte1p = pmap->pm_pt1 + pte1_idx; - pte1 = pte1_load(pte1p); - - /* - * This supports switching from a 1MB page to a - * normal 4K page. - */ - if (pte1_is_section(pte1)) { - (void)pmap_demote_pte1(pmap, pte1p, va); - /* - * Reload pte1 after demotion. - * - * Note: Demotion can even fail as either PT2 is not find for - * the virtual address or PT2PG can not be allocated. - */ - pte1 = pte1_load(pte1p); - } - - /* - * If the L2 page table page is mapped, we just increment the - * hold count, and activate it. - */ - if (pte1_is_link(pte1)) { - m = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); - pt2_wirecount_inc(m, pte1_idx); - } else { - /* - * Here if the PT2 isn't mapped, or if it has - * been deallocated. - */ - m = _pmap_allocpte2(pmap, va, flags); - if (m == NULL && (flags & PMAP_ENTER_NOSLEEP) == 0) - goto retry; - } - - return (m); -} - -static __inline void -pmap_free_zero_pages(struct spglist *free) -{ - vm_page_t m; - - while ((m = SLIST_FIRST(free)) != NULL) { - SLIST_REMOVE_HEAD(free, plinks.s.ss); - /* Preserve the page's PG_ZERO setting. */ - vm_page_free_toq(m); - } -} - -/* - * Schedule the specified unused L2 page table page to be freed. Specifically, - * add the page to the specified list of pages that will be released to the - * physical memory manager after the TLB has been updated. - */ -static __inline void -pmap_add_delayed_free_list(vm_page_t m, struct spglist *free) -{ - - /* - * Put page on a list so that it is released after - * *ALL* TLB shootdown is done - */ -#ifdef PMAP_DEBUG - pmap_zero_page_check(m); -#endif - m->flags |= PG_ZERO; - SLIST_INSERT_HEAD(free, m, plinks.s.ss); -} - -/* - * Unwire L2 page tables page. - */ -static void -pmap_unwire_pt2pg(pmap_t pmap, vm_offset_t va, vm_page_t m) -{ - pt1_entry_t *pte1p, opte1 __unused; - pt2_entry_t *pte2p; - uint32_t i; - - KASSERT(pt2pg_is_empty(m), - ("%s: pmap %p PT2PG %p wired", __func__, pmap, m)); - - /* - * Unmap all L2 page tables in the page from L1 page table. - * - * QQQ: Individual L2 page tables (except the last one) can be unmapped - * earlier. However, we are doing that this way. - */ - KASSERT(m->pindex == (pte1_index(va) & ~PT2PG_MASK), - ("%s: pmap %p va %#x PT2PG %p bad index", __func__, pmap, va, m)); - pte1p = pmap->pm_pt1 + m->pindex; - for (i = 0; i < NPT2_IN_PG; i++, pte1p++) { - KASSERT(m->md.pt2_wirecount[i] == 0, - ("%s: pmap %p PT2 %u (PG %p) wired", __func__, pmap, i, m)); - opte1 = pte1_load(pte1p); - if (pte1_is_link(opte1)) { - pte1_clear(pte1p); - /* - * Flush intermediate TLB cache. - */ - pmap_tlb_flush(pmap, (m->pindex + i) << PTE1_SHIFT); - } -#ifdef INVARIANTS - else - KASSERT((opte1 == 0) || pte1_is_section(opte1), - ("%s: pmap %p va %#x bad pte1 %x at %u", __func__, - pmap, va, opte1, i)); -#endif - } - - /* - * Unmap the page from PT2TAB. - */ - pte2p = pmap_pt2tab_entry(pmap, va); - (void)pt2tab_load_clear(pte2p); - pmap_tlb_flush(pmap, pt2map_pt2pg(va)); - - m->wire_count = 0; - pmap->pm_stats.resident_count--; - - /* - * This is a release store so that the ordinary store unmapping - * the L2 page table page is globally performed before TLB shoot- - * down is begun. - */ - atomic_subtract_rel_int(&vm_cnt.v_wire_count, 1); -} - -/* - * Decrements a L2 page table page's wire count, which is used to record the - * number of valid page table entries within the page. If the wire count - * drops to zero, then the page table page is unmapped. Returns TRUE if the - * page table page was unmapped and FALSE otherwise. - */ -static __inline boolean_t -pmap_unwire_pt2(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free) -{ - pt2_wirecount_dec(m, pte1_index(va)); - if (pt2pg_is_empty(m)) { - /* - * QQQ: Wire count is zero, so whole page should be zero and - * we can set PG_ZERO flag to it. - * Note that when promotion is enabled, it takes some - * more efforts. See pmap_unwire_pt2_all() below. - */ - pmap_unwire_pt2pg(pmap, va, m); - pmap_add_delayed_free_list(m, free); - return (TRUE); - } else - return (FALSE); -} - -/* - * Drop a L2 page table page's wire count at once, which is used to record - * the number of valid L2 page table entries within the page. If the wire - * count drops to zero, then the L2 page table page is unmapped. - */ -static __inline void -pmap_unwire_pt2_all(pmap_t pmap, vm_offset_t va, vm_page_t m, - struct spglist *free) -{ - u_int pte1_idx = pte1_index(va); - - KASSERT(m->pindex == (pte1_idx & ~PT2PG_MASK), - ("%s: PT2 page's pindex is wrong", __func__)); - KASSERT(m->wire_count > pt2_wirecount_get(m, pte1_idx), - ("%s: bad pt2 wire count %u > %u", __func__, m->wire_count, - pt2_wirecount_get(m, pte1_idx))); - - /* - * It's possible that the L2 page table was never used. - * It happened in case that a section was created without promotion. - */ - if (pt2_is_full(m, va)) { - pt2_wirecount_set(m, pte1_idx, 0); - - /* - * QQQ: We clear L2 page table now, so when L2 page table page - * is going to be freed, we can set it PG_ZERO flag ... - * This function is called only on section mappings, so - * hopefully it's not to big overload. - * - * XXX: If pmap is current, existing PT2MAP mapping could be - * used for zeroing. - */ - pmap_zero_page_area(m, page_pt2off(pte1_idx), NB_IN_PT2); - } -#ifdef INVARIANTS - else - KASSERT(pt2_is_empty(m, va), ("%s: PT2 is not empty (%u)", - __func__, pt2_wirecount_get(m, pte1_idx))); -#endif - if (pt2pg_is_empty(m)) { - pmap_unwire_pt2pg(pmap, va, m); - pmap_add_delayed_free_list(m, free); - } -} - -/* - * After removing a L2 page table entry, this routine is used to - * conditionally free the page, and manage the hold/wire counts. - */ -static boolean_t -pmap_unuse_pt2(pmap_t pmap, vm_offset_t va, struct spglist *free) -{ - pt1_entry_t pte1; - vm_page_t mpte; - - if (va >= VM_MAXUSER_ADDRESS) - return (FALSE); - pte1 = pte1_load(pmap_pte1(pmap, va)); - mpte = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); - return (pmap_unwire_pt2(pmap, va, mpte, free)); -} - -/************************************* - * - * Page management routines. - * - *************************************/ - -CTASSERT(sizeof(struct pv_chunk) == PAGE_SIZE); -CTASSERT(_NPCM == 11); -CTASSERT(_NPCPV == 336); - -static __inline struct pv_chunk * -pv_to_chunk(pv_entry_t pv) -{ - - return ((struct pv_chunk *)((uintptr_t)pv & ~(uintptr_t)PAGE_MASK)); -} - -#define PV_PMAP(pv) (pv_to_chunk(pv)->pc_pmap) - -#define PC_FREE0_9 0xfffffffful /* Free values for index 0 through 9 */ -#define PC_FREE10 0x0000fffful /* Free values for index 10 */ - -static const uint32_t pc_freemask[_NPCM] = { - PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, - PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, - PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, - PC_FREE0_9, PC_FREE10 -}; - -SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_count, CTLFLAG_RD, &pv_entry_count, 0, - "Current number of pv entries"); - -#ifdef PV_STATS -static int pc_chunk_count, pc_chunk_allocs, pc_chunk_frees, pc_chunk_tryfail; - -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_count, CTLFLAG_RD, &pc_chunk_count, 0, - "Current number of pv entry chunks"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_allocs, CTLFLAG_RD, &pc_chunk_allocs, 0, - "Current number of pv entry chunks allocated"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_frees, CTLFLAG_RD, &pc_chunk_frees, 0, - "Current number of pv entry chunks frees"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_tryfail, CTLFLAG_RD, &pc_chunk_tryfail, - 0, "Number of times tried to get a chunk page but failed."); - -static long pv_entry_frees, pv_entry_allocs; -static int pv_entry_spare; - -SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_frees, CTLFLAG_RD, &pv_entry_frees, 0, - "Current number of pv entry frees"); -SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_allocs, CTLFLAG_RD, &pv_entry_allocs, - 0, "Current number of pv entry allocs"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_spare, CTLFLAG_RD, &pv_entry_spare, 0, - "Current number of spare pv entries"); -#endif - -/* - * Is given page managed? - */ -static __inline boolean_t -is_managed(vm_paddr_t pa) -{ - vm_offset_t pgnum; - vm_page_t m; - - pgnum = atop(pa); - if (pgnum >= first_page) { - m = PHYS_TO_VM_PAGE(pa); - if (m == NULL) - return (FALSE); - if ((m->oflags & VPO_UNMANAGED) == 0) - return (TRUE); - } - return (FALSE); -} - -static __inline boolean_t -pte1_is_managed(pt1_entry_t pte1) -{ - - return (is_managed(pte1_pa(pte1))); -} - -static __inline boolean_t -pte2_is_managed(pt2_entry_t pte2) -{ - - return (is_managed(pte2_pa(pte2))); -} - -/* - * We are in a serious low memory condition. Resort to - * drastic measures to free some pages so we can allocate - * another pv entry chunk. - */ -static vm_page_t -pmap_pv_reclaim(pmap_t locked_pmap) -{ - struct pch newtail; - struct pv_chunk *pc; - struct md_page *pvh; - pt1_entry_t *pte1p; - pmap_t pmap; - pt2_entry_t *pte2p, tpte2; - pv_entry_t pv; - vm_offset_t va; - vm_page_t m, m_pc; - struct spglist free; - uint32_t inuse; - int bit, field, freed; - - PMAP_LOCK_ASSERT(locked_pmap, MA_OWNED); - pmap = NULL; - m_pc = NULL; - SLIST_INIT(&free); - TAILQ_INIT(&newtail); - while ((pc = TAILQ_FIRST(&pv_chunks)) != NULL && (pv_vafree == 0 || - SLIST_EMPTY(&free))) { - TAILQ_REMOVE(&pv_chunks, pc, pc_lru); - if (pmap != pc->pc_pmap) { - if (pmap != NULL) { - if (pmap != locked_pmap) - PMAP_UNLOCK(pmap); - } - pmap = pc->pc_pmap; - /* Avoid deadlock and lock recursion. */ - if (pmap > locked_pmap) - PMAP_LOCK(pmap); - else if (pmap != locked_pmap && !PMAP_TRYLOCK(pmap)) { - pmap = NULL; - TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); - continue; - } - } - - /* - * Destroy every non-wired, 4 KB page mapping in the chunk. - */ - freed = 0; - for (field = 0; field < _NPCM; field++) { - for (inuse = ~pc->pc_map[field] & pc_freemask[field]; - inuse != 0; inuse &= ~(1UL << bit)) { - bit = ffs(inuse) - 1; - pv = &pc->pc_pventry[field * 32 + bit]; - va = pv->pv_va; - pte1p = pmap_pte1(pmap, va); - if (pte1_is_section(pte1_load(pte1p))) - continue; - pte2p = pmap_pte2(pmap, va); - tpte2 = pte2_load(pte2p); - if ((tpte2 & PTE2_W) == 0) - tpte2 = pte2_load_clear(pte2p); - pmap_pte2_release(pte2p); - if ((tpte2 & PTE2_W) != 0) - continue; - KASSERT(tpte2 != 0, - ("pmap_pv_reclaim: pmap %p va %#x zero pte", - pmap, va)); - pmap_tlb_flush(pmap, va); - m = PHYS_TO_VM_PAGE(pte2_pa(tpte2)); - if (pte2_is_dirty(tpte2)) - vm_page_dirty(m); - if ((tpte2 & PTE2_A) != 0) - vm_page_aflag_set(m, PGA_REFERENCED); - TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); - if (TAILQ_EMPTY(&m->md.pv_list) && - (m->flags & PG_FICTITIOUS) == 0) { - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - if (TAILQ_EMPTY(&pvh->pv_list)) { - vm_page_aflag_clear(m, - PGA_WRITEABLE); - } - } - pc->pc_map[field] |= 1UL << bit; - pmap_unuse_pt2(pmap, va, &free); - freed++; - } - } - if (freed == 0) { - TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); - continue; - } - /* Every freed mapping is for a 4 KB page. */ - pmap->pm_stats.resident_count -= freed; - PV_STAT(pv_entry_frees += freed); - PV_STAT(pv_entry_spare += freed); - pv_entry_count -= freed; - TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - for (field = 0; field < _NPCM; field++) - if (pc->pc_map[field] != pc_freemask[field]) { - TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, - pc_list); - TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); - - /* - * One freed pv entry in locked_pmap is - * sufficient. - */ - if (pmap == locked_pmap) - goto out; - break; - } - if (field == _NPCM) { - PV_STAT(pv_entry_spare -= _NPCPV); - PV_STAT(pc_chunk_count--); - PV_STAT(pc_chunk_frees++); - /* Entire chunk is free; return it. */ - m_pc = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)pc)); - pmap_qremove((vm_offset_t)pc, 1); - pmap_pte2list_free(&pv_vafree, (vm_offset_t)pc); - break; - } - } -out: - TAILQ_CONCAT(&pv_chunks, &newtail, pc_lru); - if (pmap != NULL) { - if (pmap != locked_pmap) - PMAP_UNLOCK(pmap); - } - if (m_pc == NULL && pv_vafree != 0 && SLIST_EMPTY(&free)) { - m_pc = SLIST_FIRST(&free); - SLIST_REMOVE_HEAD(&free, plinks.s.ss); - /* Recycle a freed page table page. */ - m_pc->wire_count = 1; - atomic_add_int(&vm_cnt.v_wire_count, 1); - } - pmap_free_zero_pages(&free); - return (m_pc); -} - -static void -free_pv_chunk(struct pv_chunk *pc) -{ - vm_page_t m; - - TAILQ_REMOVE(&pv_chunks, pc, pc_lru); - PV_STAT(pv_entry_spare -= _NPCPV); - PV_STAT(pc_chunk_count--); - PV_STAT(pc_chunk_frees++); - /* entire chunk is free, return it */ - m = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)pc)); - pmap_qremove((vm_offset_t)pc, 1); - vm_page_unwire(m, PQ_NONE); - vm_page_free(m); - pmap_pte2list_free(&pv_vafree, (vm_offset_t)pc); -} - -/* - * Free the pv_entry back to the free list. - */ -static void -free_pv_entry(pmap_t pmap, pv_entry_t pv) -{ - struct pv_chunk *pc; - int idx, field, bit; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - PV_STAT(pv_entry_frees++); - PV_STAT(pv_entry_spare++); - pv_entry_count--; - pc = pv_to_chunk(pv); - idx = pv - &pc->pc_pventry[0]; - field = idx / 32; - bit = idx % 32; - pc->pc_map[field] |= 1ul << bit; - for (idx = 0; idx < _NPCM; idx++) - if (pc->pc_map[idx] != pc_freemask[idx]) { - /* - * 98% of the time, pc is already at the head of the - * list. If it isn't already, move it to the head. - */ - if (__predict_false(TAILQ_FIRST(&pmap->pm_pvchunk) != - pc)) { - TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, - pc_list); - } - return; - } - TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - free_pv_chunk(pc); -} - -/* - * Get a new pv_entry, allocating a block from the system - * when needed. - */ -static pv_entry_t -get_pv_entry(pmap_t pmap, boolean_t try) -{ - static const struct timeval printinterval = { 60, 0 }; - static struct timeval lastprint; - int bit, field; - pv_entry_t pv; - struct pv_chunk *pc; - vm_page_t m; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - PV_STAT(pv_entry_allocs++); - pv_entry_count++; - if (pv_entry_count > pv_entry_high_water) - if (ratecheck(&lastprint, &printinterval)) - printf("Approaching the limit on PV entries, consider " - "increasing either the vm.pmap.shpgperproc or the " - "vm.pmap.pv_entry_max tunable.\n"); -retry: - pc = TAILQ_FIRST(&pmap->pm_pvchunk); - if (pc != NULL) { - for (field = 0; field < _NPCM; field++) { - if (pc->pc_map[field]) { - bit = ffs(pc->pc_map[field]) - 1; - break; - } - } - if (field < _NPCM) { - pv = &pc->pc_pventry[field * 32 + bit]; - pc->pc_map[field] &= ~(1ul << bit); - /* If this was the last item, move it to tail */ - for (field = 0; field < _NPCM; field++) - if (pc->pc_map[field] != 0) { - PV_STAT(pv_entry_spare--); - return (pv); /* not full, return */ - } - TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - TAILQ_INSERT_TAIL(&pmap->pm_pvchunk, pc, pc_list); - PV_STAT(pv_entry_spare--); - return (pv); - } - } - /* - * Access to the pte2list "pv_vafree" is synchronized by the pvh - * global lock. If "pv_vafree" is currently non-empty, it will - * remain non-empty until pmap_pte2list_alloc() completes. - */ - if (pv_vafree == 0 || (m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | - VM_ALLOC_NOOBJ | VM_ALLOC_WIRED)) == NULL) { - if (try) { - pv_entry_count--; - PV_STAT(pc_chunk_tryfail++); - return (NULL); - } - m = pmap_pv_reclaim(pmap); - if (m == NULL) - goto retry; - } - PV_STAT(pc_chunk_count++); - PV_STAT(pc_chunk_allocs++); - pc = (struct pv_chunk *)pmap_pte2list_alloc(&pv_vafree); - pmap_qenter((vm_offset_t)pc, &m, 1); - pc->pc_pmap = pmap; - pc->pc_map[0] = pc_freemask[0] & ~1ul; /* preallocated bit 0 */ - for (field = 1; field < _NPCM; field++) - pc->pc_map[field] = pc_freemask[field]; - TAILQ_INSERT_TAIL(&pv_chunks, pc, pc_lru); - pv = &pc->pc_pventry[0]; - TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); - PV_STAT(pv_entry_spare += _NPCPV - 1); - return (pv); -} - -/* - * Create a pv entry for page at pa for - * (pmap, va). - */ -static void -pmap_insert_entry(pmap_t pmap, vm_offset_t va, vm_page_t m) -{ - pv_entry_t pv; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - pv = get_pv_entry(pmap, FALSE); - pv->pv_va = va; - TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); -} - -static __inline pv_entry_t -pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, vm_offset_t va) -{ - pv_entry_t pv; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { - if (pmap == PV_PMAP(pv) && va == pv->pv_va) { - TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); - break; - } - } - return (pv); -} - -static void -pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va) -{ - pv_entry_t pv; - - pv = pmap_pvh_remove(pvh, pmap, va); - KASSERT(pv != NULL, ("pmap_pvh_free: pv not found")); - free_pv_entry(pmap, pv); -} - -static void -pmap_remove_entry(pmap_t pmap, vm_page_t m, vm_offset_t va) -{ - struct md_page *pvh; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - pmap_pvh_free(&m->md, pmap, va); - if (TAILQ_EMPTY(&m->md.pv_list) && (m->flags & PG_FICTITIOUS) == 0) { - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - if (TAILQ_EMPTY(&pvh->pv_list)) - vm_page_aflag_clear(m, PGA_WRITEABLE); - } -} - -static void -pmap_pv_demote_pte1(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) -{ - struct md_page *pvh; - pv_entry_t pv; - vm_offset_t va_last; - vm_page_t m; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - KASSERT((pa & PTE1_OFFSET) == 0, - ("pmap_pv_demote_pte1: pa is not 1mpage aligned")); - - /* - * Transfer the 1mpage's pv entry for this mapping to the first - * page's pv list. - */ - pvh = pa_to_pvh(pa); - va = pte1_trunc(va); - pv = pmap_pvh_remove(pvh, pmap, va); - KASSERT(pv != NULL, ("pmap_pv_demote_pte1: pv not found")); - m = PHYS_TO_VM_PAGE(pa); - TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); - /* Instantiate the remaining NPTE2_IN_PT2 - 1 pv entries. */ - va_last = va + PTE1_SIZE - PAGE_SIZE; - do { - m++; - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_pv_demote_pte1: page %p is not managed", m)); - va += PAGE_SIZE; - pmap_insert_entry(pmap, va, m); - } while (va < va_last); -} - -static void -pmap_pv_promote_pte1(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) -{ - struct md_page *pvh; - pv_entry_t pv; - vm_offset_t va_last; - vm_page_t m; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - KASSERT((pa & PTE1_OFFSET) == 0, - ("pmap_pv_promote_pte1: pa is not 1mpage aligned")); - - /* - * Transfer the first page's pv entry for this mapping to the - * 1mpage's pv list. Aside from avoiding the cost of a call - * to get_pv_entry(), a transfer avoids the possibility that - * get_pv_entry() calls pmap_pv_reclaim() and that pmap_pv_reclaim() - * removes one of the mappings that is being promoted. - */ - m = PHYS_TO_VM_PAGE(pa); - va = pte1_trunc(va); - pv = pmap_pvh_remove(&m->md, pmap, va); - KASSERT(pv != NULL, ("pmap_pv_promote_pte1: pv not found")); - pvh = pa_to_pvh(pa); - TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); - /* Free the remaining NPTE2_IN_PT2 - 1 pv entries. */ - va_last = va + PTE1_SIZE - PAGE_SIZE; - do { - m++; - va += PAGE_SIZE; - pmap_pvh_free(&m->md, pmap, va); - } while (va < va_last); -} - -/* - * Conditionally create a pv entry. - */ -static boolean_t -pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, vm_page_t m) -{ - pv_entry_t pv; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - if (pv_entry_count < pv_entry_high_water && - (pv = get_pv_entry(pmap, TRUE)) != NULL) { - pv->pv_va = va; - TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); - return (TRUE); - } else - return (FALSE); -} - -/* - * Create the pv entries for each of the pages within a section. - */ -static boolean_t -pmap_pv_insert_pte1(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) -{ - struct md_page *pvh; - pv_entry_t pv; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - if (pv_entry_count < pv_entry_high_water && - (pv = get_pv_entry(pmap, TRUE)) != NULL) { - pv->pv_va = va; - pvh = pa_to_pvh(pa); - TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); - return (TRUE); - } else - return (FALSE); -} - -/* - * Tries to promote the NPTE2_IN_PT2, contiguous 4KB page mappings that are - * within a single page table page (PT2) to a single 1MB page mapping. - * For promotion to occur, two conditions must be met: (1) the 4KB page - * mappings must map aligned, contiguous physical memory and (2) the 4KB page - * mappings must have identical characteristics. - * - * Managed (PG_MANAGED) mappings within the kernel address space are not - * promoted. The reason is that kernel PTE1s are replicated in each pmap but - * pmap_remove_write(), pmap_clear_modify(), and pmap_clear_reference() only - * read the PTE1 from the kernel pmap. - */ -static void -pmap_promote_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t va) -{ - pt1_entry_t npte1; - pt2_entry_t *fpte2p, fpte2, fpte2_fav; - pt2_entry_t *pte2p, pte2; - vm_offset_t pteva __unused; - vm_page_t m __unused; - - PDEBUG(6, printf("%s(%p): try for va %#x pte1 %#x at %p\n", __func__, - pmap, va, pte1_load(pte1p), pte1p)); - - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - - /* - * Examine the first PTE2 in the specified PT2. Abort if this PTE2 is - * either invalid, unused, or does not map the first 4KB physical page - * within a 1MB page. - */ - fpte2p = pmap_pte2_quick(pmap, pte1_trunc(va)); -setpte1: - fpte2 = pte2_load(fpte2p); - if ((fpte2 & ((PTE2_FRAME & PTE1_OFFSET) | PTE2_A | PTE2_V)) != - (PTE2_A | PTE2_V)) { - pmap_pte1_p_failures++; - CTR3(KTR_PMAP, "%s: failure(1) for va %#x in pmap %p", - __func__, va, pmap); - return; - } - if (pte2_is_managed(fpte2) && pmap == kernel_pmap) { - pmap_pte1_p_failures++; - CTR3(KTR_PMAP, "%s: failure(2) for va %#x in pmap %p", - __func__, va, pmap); - return; - } - if ((fpte2 & (PTE2_NM | PTE2_RO)) == PTE2_NM) { - /* - * When page is not modified, PTE2_RO can be set without - * a TLB invalidation. - * - * Note: When modified bit is being set, then in hardware case, - * the TLB entry is re-read (updated) from PT2, and in - * software case (abort), the PTE2 is read from PT2 and - * TLB flushed if changed. The following cmpset() solves - * any race with setting this bit in both cases. - */ - if (!pte2_cmpset(fpte2p, fpte2, fpte2 | PTE2_RO)) - goto setpte1; - fpte2 |= PTE2_RO; - } - - /* - * Examine each of the other PTE2s in the specified PT2. Abort if this - * PTE2 maps an unexpected 4KB physical page or does not have identical - * characteristics to the first PTE2. - */ - fpte2_fav = (fpte2 & (PTE2_FRAME | PTE2_A | PTE2_V)); - fpte2_fav += PTE1_SIZE - PTE2_SIZE; /* examine from the end */ - for (pte2p = fpte2p + NPTE2_IN_PT2 - 1; pte2p > fpte2p; pte2p--) { -setpte2: - pte2 = pte2_load(pte2p); - if ((pte2 & (PTE2_FRAME | PTE2_A | PTE2_V)) != fpte2_fav) { - pmap_pte1_p_failures++; - CTR3(KTR_PMAP, "%s: failure(3) for va %#x in pmap %p", - __func__, va, pmap); - return; - } - if ((pte2 & (PTE2_NM | PTE2_RO)) == PTE2_NM) { - /* - * When page is not modified, PTE2_RO can be set - * without a TLB invalidation. See note above. - */ - if (!pte2_cmpset(pte2p, pte2, pte2 | PTE2_RO)) - goto setpte2; - pte2 |= PTE2_RO; - pteva = pte1_trunc(va) | (pte2 & PTE1_OFFSET & - PTE2_FRAME); - CTR3(KTR_PMAP, "%s: protect for va %#x in pmap %p", - __func__, pteva, pmap); - } - if ((pte2 & PTE2_PROMOTE) != (fpte2 & PTE2_PROMOTE)) { - pmap_pte1_p_failures++; - CTR3(KTR_PMAP, "%s: failure(4) for va %#x in pmap %p", - __func__, va, pmap); - return; - } - - fpte2_fav -= PTE2_SIZE; - } - /* - * The page table page in its current state will stay in PT2TAB - * until the PTE1 mapping the section is demoted by pmap_demote_pte1() - * or destroyed by pmap_remove_pte1(). - * - * Note that L2 page table size is not equal to PAGE_SIZE. - */ - m = PHYS_TO_VM_PAGE(trunc_page(pte1_link_pa(pte1_load(pte1p)))); - KASSERT(m >= vm_page_array && m < &vm_page_array[vm_page_array_size], - ("%s: PT2 page is out of range", __func__)); - KASSERT(m->pindex == (pte1_index(va) & ~PT2PG_MASK), - ("%s: PT2 page's pindex is wrong", __func__)); - - /* - * Get pte1 from pte2 format. - */ - npte1 = (fpte2 & PTE1_FRAME) | ATTR_TO_L1(fpte2) | PTE1_V; - - /* - * Promote the pv entries. - */ - if (pte2_is_managed(fpte2)) - pmap_pv_promote_pte1(pmap, va, pte1_pa(npte1)); - - /* - * Map the section. - */ - if (pmap == kernel_pmap) - pmap_kenter_pte1(va, npte1); - else - pte1_store(pte1p, npte1); - /* - * Flush old small mappings. We call single pmap_tlb_flush() in - * pmap_demote_pte1() and pmap_remove_pte1(), so we must be sure that - * no small mappings survive. We assume that given pmap is current and - * don't play game with PTE2_NG. - */ - pmap_tlb_flush_range(pmap, pte1_trunc(va), PTE1_SIZE); - - pmap_pte1_promotions++; - CTR3(KTR_PMAP, "%s: success for va %#x in pmap %p", - __func__, va, pmap); - - PDEBUG(6, printf("%s(%p): success for va %#x pte1 %#x(%#x) at %p\n", - __func__, pmap, va, npte1, pte1_load(pte1p), pte1p)); -} - -/* - * Zero L2 page table page. - */ -static __inline void -pmap_clear_pt2(pt2_entry_t *fpte2p) -{ - pt2_entry_t *pte2p; - - for (pte2p = fpte2p; pte2p < fpte2p + NPTE2_IN_PT2; pte2p++) - pte2_clear(pte2p); - -} - -/* - * Removes a 1MB page mapping from the kernel pmap. - */ -static void -pmap_remove_kernel_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t va) -{ - vm_page_t m; - uint32_t pte1_idx; - pt2_entry_t *fpte2p; - vm_paddr_t pt2_pa; - - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - m = pmap_pt2_page(pmap, va); - if (m == NULL) - /* - * QQQ: Is this function called only on promoted pte1? - * We certainly do section mappings directly - * (without promotion) in kernel !!! - */ - panic("%s: missing pt2 page", __func__); - - pte1_idx = pte1_index(va); - - /* - * Initialize the L2 page table. - */ - fpte2p = page_pt2(pt2map_pt2pg(va), pte1_idx); - pmap_clear_pt2(fpte2p); - - /* - * Remove the mapping. - */ - pt2_pa = page_pt2pa(VM_PAGE_TO_PHYS(m), pte1_idx); - pmap_kenter_pte1(va, PTE1_LINK(pt2_pa)); - - /* - * QQQ: We do not need to invalidate PT2MAP mapping - * as we did not change it. I.e. the L2 page table page - * was and still is mapped the same way. - */ -} - -/* - * Do the things to unmap a section in a process - */ -static void -pmap_remove_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t sva, - struct spglist *free) -{ - pt1_entry_t opte1; - struct md_page *pvh; - vm_offset_t eva, va; - vm_page_t m; - - PDEBUG(6, printf("%s(%p): va %#x pte1 %#x at %p\n", __func__, pmap, sva, - pte1_load(pte1p), pte1p)); - - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - KASSERT((sva & PTE1_OFFSET) == 0, - ("%s: sva is not 1mpage aligned", __func__)); - - /* - * Clear and invalidate the mapping. It should occupy one and only TLB - * entry. So, pmap_tlb_flush() called with aligned address should be - * sufficient. - */ - opte1 = pte1_load_clear(pte1p); - pmap_tlb_flush(pmap, sva); - - if (pte1_is_wired(opte1)) - pmap->pm_stats.wired_count -= PTE1_SIZE / PAGE_SIZE; - pmap->pm_stats.resident_count -= PTE1_SIZE / PAGE_SIZE; - if (pte1_is_managed(opte1)) { - pvh = pa_to_pvh(pte1_pa(opte1)); - pmap_pvh_free(pvh, pmap, sva); - eva = sva + PTE1_SIZE; - for (va = sva, m = PHYS_TO_VM_PAGE(pte1_pa(opte1)); - va < eva; va += PAGE_SIZE, m++) { - if (pte1_is_dirty(opte1)) - vm_page_dirty(m); - if (opte1 & PTE1_A) - vm_page_aflag_set(m, PGA_REFERENCED); - if (TAILQ_EMPTY(&m->md.pv_list) && - TAILQ_EMPTY(&pvh->pv_list)) - vm_page_aflag_clear(m, PGA_WRITEABLE); - } - } - if (pmap == kernel_pmap) { - /* - * L2 page table(s) can't be removed from kernel map as - * kernel counts on it (stuff around pmap_growkernel()). - */ - pmap_remove_kernel_pte1(pmap, pte1p, sva); - } else { - /* - * Get associated L2 page table page. - * It's possible that the page was never allocated. - */ - m = pmap_pt2_page(pmap, sva); - if (m != NULL) - pmap_unwire_pt2_all(pmap, sva, m, free); - } -} - -/* - * Fills L2 page table page with mappings to consecutive physical pages. - */ -static __inline void -pmap_fill_pt2(pt2_entry_t *fpte2p, pt2_entry_t npte2) -{ - pt2_entry_t *pte2p; - - for (pte2p = fpte2p; pte2p < fpte2p + NPTE2_IN_PT2; pte2p++) { - pte2_store(pte2p, npte2); - npte2 += PTE2_SIZE; - } -} - -/* - * Tries to demote a 1MB page mapping. If demotion fails, the - * 1MB page mapping is invalidated. - */ -static boolean_t -pmap_demote_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t va) -{ - pt1_entry_t opte1, npte1; - pt2_entry_t *fpte2p, npte2; - vm_paddr_t pt2pg_pa, pt2_pa; - vm_page_t m; - struct spglist free; - uint32_t pte1_idx, isnew = 0; - - PDEBUG(6, printf("%s(%p): try for va %#x pte1 %#x at %p\n", __func__, - pmap, va, pte1_load(pte1p), pte1p)); - - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - - opte1 = pte1_load(pte1p); - KASSERT(pte1_is_section(opte1), ("%s: opte1 not a section", __func__)); - - if ((opte1 & PTE1_A) == 0 || (m = pmap_pt2_page(pmap, va)) == NULL) { - KASSERT(!pte1_is_wired(opte1), - ("%s: PT2 page for a wired mapping is missing", __func__)); - - /* - * Invalidate the 1MB page mapping and return - * "failure" if the mapping was never accessed or the - * allocation of the new page table page fails. - */ - if ((opte1 & PTE1_A) == 0 || (m = vm_page_alloc(NULL, - pte1_index(va) & ~PT2PG_MASK, VM_ALLOC_NOOBJ | - VM_ALLOC_NORMAL | VM_ALLOC_WIRED)) == NULL) { - SLIST_INIT(&free); - pmap_remove_pte1(pmap, pte1p, pte1_trunc(va), &free); - pmap_free_zero_pages(&free); - CTR3(KTR_PMAP, "%s: failure for va %#x in pmap %p", - __func__, va, pmap); - return (FALSE); - } - if (va < VM_MAXUSER_ADDRESS) - pmap->pm_stats.resident_count++; - - isnew = 1; - - /* - * We init all L2 page tables in the page even if - * we are going to change everything for one L2 page - * table in a while. - */ - pt2pg_pa = pmap_pt2pg_init(pmap, va, m); - } else { - if (va < VM_MAXUSER_ADDRESS) { - if (pt2_is_empty(m, va)) - isnew = 1; /* Demoting section w/o promotion. */ -#ifdef INVARIANTS - else - KASSERT(pt2_is_full(m, va), ("%s: bad PT2 wire" - " count %u", __func__, - pt2_wirecount_get(m, pte1_index(va)))); -#endif - } - } - - pt2pg_pa = VM_PAGE_TO_PHYS(m); - pte1_idx = pte1_index(va); - /* - * If the pmap is current, then the PT2MAP can provide access to - * the page table page (promoted L2 page tables are not unmapped). - * Otherwise, temporarily map the L2 page table page (m) into - * the kernel's address space at either PADDR1 or PADDR2. - * - * Note that L2 page table size is not equal to PAGE_SIZE. - */ - if (pmap_is_current(pmap)) - fpte2p = page_pt2(pt2map_pt2pg(va), pte1_idx); - else if (curthread->td_pinned > 0 && rw_wowned(&pvh_global_lock)) { - if (pte2_pa(pte2_load(PMAP1)) != pt2pg_pa) { - pte2_store(PMAP1, PTE2_KPT(pt2pg_pa)); -#ifdef SMP - PMAP1cpu = PCPU_GET(cpuid); -#endif - tlb_flush_local((vm_offset_t)PADDR1); - PMAP1changed++; - } else -#ifdef SMP - if (PMAP1cpu != PCPU_GET(cpuid)) { - PMAP1cpu = PCPU_GET(cpuid); - tlb_flush_local((vm_offset_t)PADDR1); - PMAP1changedcpu++; - } else -#endif - PMAP1unchanged++; - fpte2p = page_pt2((vm_offset_t)PADDR1, pte1_idx); - } else { - mtx_lock(&PMAP2mutex); - if (pte2_pa(pte2_load(PMAP2)) != pt2pg_pa) { - pte2_store(PMAP2, PTE2_KPT(pt2pg_pa)); - tlb_flush((vm_offset_t)PADDR2); - } - fpte2p = page_pt2((vm_offset_t)PADDR2, pte1_idx); - } - pt2_pa = page_pt2pa(pt2pg_pa, pte1_idx); - npte1 = PTE1_LINK(pt2_pa); - - KASSERT((opte1 & PTE1_A) != 0, - ("%s: opte1 is missing PTE1_A", __func__)); - KASSERT((opte1 & (PTE1_NM | PTE1_RO)) != PTE1_NM, - ("%s: opte1 has PTE1_NM", __func__)); - - /* - * Get pte2 from pte1 format. - */ - npte2 = pte1_pa(opte1) | ATTR_TO_L2(opte1) | PTE2_V; - - /* - * If the L2 page table page is new, initialize it. If the mapping - * has changed attributes, update the page table entries. - */ - if (isnew != 0) { - pt2_wirecount_set(m, pte1_idx, NPTE2_IN_PT2); - pmap_fill_pt2(fpte2p, npte2); - } else if ((pte2_load(fpte2p) & PTE2_PROMOTE) != - (npte2 & PTE2_PROMOTE)) - pmap_fill_pt2(fpte2p, npte2); - - KASSERT(pte2_pa(pte2_load(fpte2p)) == pte2_pa(npte2), - ("%s: fpte2p and npte2 map different physical addresses", - __func__)); - - if (fpte2p == PADDR2) - mtx_unlock(&PMAP2mutex); - - /* - * Demote the mapping. This pmap is locked. The old PTE1 has - * PTE1_A set. If the old PTE1 has not PTE1_RO set, it also - * has not PTE1_NM set. Thus, there is no danger of a race with - * another processor changing the setting of PTE1_A and/or PTE1_NM - * between the read above and the store below. - */ - if (pmap == kernel_pmap) - pmap_kenter_pte1(va, npte1); - else - pte1_store(pte1p, npte1); - - /* - * Flush old big mapping. The mapping should occupy one and only - * TLB entry. So, pmap_tlb_flush() called with aligned address - * should be sufficient. - */ - pmap_tlb_flush(pmap, pte1_trunc(va)); - - /* - * Demote the pv entry. This depends on the earlier demotion - * of the mapping. Specifically, the (re)creation of a per- - * page pv entry might trigger the execution of pmap_pv_reclaim(), - * which might reclaim a newly (re)created per-page pv entry - * and destroy the associated mapping. In order to destroy - * the mapping, the PTE1 must have already changed from mapping - * the 1mpage to referencing the page table page. - */ - if (pte1_is_managed(opte1)) - pmap_pv_demote_pte1(pmap, va, pte1_pa(opte1)); - - pmap_pte1_demotions++; - CTR3(KTR_PMAP, "%s: success for va %#x in pmap %p", - __func__, va, pmap); - - PDEBUG(6, printf("%s(%p): success for va %#x pte1 %#x(%#x) at %p\n", - __func__, pmap, va, npte1, pte1_load(pte1p), pte1p)); - return (TRUE); -} - -/* - * Insert the given physical page (p) at - * the specified virtual address (v) in the - * target physical map with the protection requested. - * - * If specified, the page will be wired down, meaning - * that the related pte can not be reclaimed. - * - * NB: This is the only routine which MAY NOT lazy-evaluate - * or lose information. That is, this routine must actually - * insert this page into the given map NOW. - */ -int -pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, - u_int flags, int8_t psind) -{ - pt1_entry_t *pte1p; - pt2_entry_t *pte2p; - pt2_entry_t npte2, opte2; - pv_entry_t pv; - vm_paddr_t opa, pa; - vm_page_t mpte2, om; - boolean_t wired; - - va = trunc_page(va); - mpte2 = NULL; - wired = (flags & PMAP_ENTER_WIRED) != 0; - - KASSERT(va <= vm_max_kernel_address, ("%s: toobig", __func__)); - KASSERT(va < UPT2V_MIN_ADDRESS || va >= UPT2V_MAX_ADDRESS, - ("%s: invalid to pmap_enter page table pages (va: 0x%x)", __func__, - va)); - if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m)) - VM_OBJECT_ASSERT_LOCKED(m->object); - - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - sched_pin(); - - /* - * In the case that a page table page is not - * resident, we are creating it here. - */ - if (va < VM_MAXUSER_ADDRESS) { - mpte2 = pmap_allocpte2(pmap, va, flags); - if (mpte2 == NULL) { - KASSERT((flags & PMAP_ENTER_NOSLEEP) != 0, - ("pmap_allocpte2 failed with sleep allowed")); - sched_unpin(); - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); - return (KERN_RESOURCE_SHORTAGE); - } - } - pte1p = pmap_pte1(pmap, va); - if (pte1_is_section(pte1_load(pte1p))) - panic("%s: attempted on 1MB page", __func__); - pte2p = pmap_pte2_quick(pmap, va); - if (pte2p == NULL) - panic("%s: invalid L1 page table entry va=%#x", __func__, va); - - om = NULL; - pa = VM_PAGE_TO_PHYS(m); - opte2 = pte2_load(pte2p); - opa = pte2_pa(opte2); - /* - * Mapping has not changed, must be protection or wiring change. - */ - if (pte2_is_valid(opte2) && (opa == pa)) { - /* - * Wiring change, just update stats. We don't worry about - * wiring PT2 pages as they remain resident as long as there - * are valid mappings in them. Hence, if a user page is wired, - * the PT2 page will be also. - */ - if (wired && !pte2_is_wired(opte2)) - pmap->pm_stats.wired_count++; - else if (!wired && pte2_is_wired(opte2)) - pmap->pm_stats.wired_count--; - - /* - * Remove extra pte2 reference - */ - if (mpte2) - pt2_wirecount_dec(mpte2, pte1_index(va)); - if (pte2_is_managed(opte2)) - om = m; - goto validate; - } - - /* - * QQQ: We think that changing physical address on writeable mapping - * is not safe. Well, maybe on kernel address space with correct - * locking, it can make a sense. However, we have no idea why - * anyone should do that on user address space. Are we wrong? - */ - KASSERT((opa == 0) || (opa == pa) || - !pte2_is_valid(opte2) || ((opte2 & PTE2_RO) != 0), - ("%s: pmap %p va %#x(%#x) opa %#x pa %#x - gotcha %#x %#x!", - __func__, pmap, va, opte2, opa, pa, flags, prot)); - - pv = NULL; - - /* - * Mapping has changed, invalidate old range and fall through to - * handle validating new mapping. - */ - if (opa) { - if (pte2_is_wired(opte2)) - pmap->pm_stats.wired_count--; - if (pte2_is_managed(opte2)) { - om = PHYS_TO_VM_PAGE(opa); - pv = pmap_pvh_remove(&om->md, pmap, va); - } - /* - * Remove extra pte2 reference - */ - if (mpte2 != NULL) - pt2_wirecount_dec(mpte2, va >> PTE1_SHIFT); - } else - pmap->pm_stats.resident_count++; - - /* - * Enter on the PV list if part of our managed memory. - */ - if ((m->oflags & VPO_UNMANAGED) == 0) { - KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva, - ("%s: managed mapping within the clean submap", __func__)); - if (pv == NULL) - pv = get_pv_entry(pmap, FALSE); - pv->pv_va = va; - TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); - } else if (pv != NULL) - free_pv_entry(pmap, pv); - - /* - * Increment counters - */ - if (wired) - pmap->pm_stats.wired_count++; - -validate: - /* - * Now validate mapping with desired protection/wiring. - */ - npte2 = PTE2(pa, PTE2_NM, m->md.pat_mode); - if (prot & VM_PROT_WRITE) { - if (pte2_is_managed(npte2)) - vm_page_aflag_set(m, PGA_WRITEABLE); - } - else - npte2 |= PTE2_RO; - if ((prot & VM_PROT_EXECUTE) == 0) - npte2 |= PTE2_NX; - if (wired) - npte2 |= PTE2_W; - if (va < VM_MAXUSER_ADDRESS) - npte2 |= PTE2_U; - if (pmap != kernel_pmap) - npte2 |= PTE2_NG; - - /* - * If the mapping or permission bits are different, we need - * to update the pte2. - * - * QQQ: Think again and again what to do - * if the mapping is going to be changed! - */ - if ((opte2 & ~(PTE2_NM | PTE2_A)) != (npte2 & ~(PTE2_NM | PTE2_A))) { - /* - * Sync icache if exec permission and attribute PTE2_ATTR_WB_WA - * is set. Do it now, before the mapping is stored and made - * valid for hardware table walk. If done later, there is a race - * for other threads of current process in lazy loading case. - * Don't do it for kernel memory which is mapped with exec - * permission even if the memory isn't going to hold executable - * code. The only time when icache sync is needed is after - * kernel module is loaded and the relocation info is processed. - * And it's done in elf_cpu_load_file(). - * - * QQQ: (1) Does it exist any better way where - * or how to sync icache? - * (2) Now, we do it on a page basis. - */ - if ((prot & VM_PROT_EXECUTE) && pmap != kernel_pmap && - m->md.pat_mode == PTE2_ATTR_WB_WA && - (opa != pa || (opte2 & PTE2_NX))) - cache_icache_sync_fresh(va, pa, PAGE_SIZE); - - npte2 |= PTE2_A; - if (flags & VM_PROT_WRITE) - npte2 &= ~PTE2_NM; - if (opte2 & PTE2_V) { - /* Change mapping with break-before-make approach. */ - opte2 = pte2_load_clear(pte2p); - pmap_tlb_flush(pmap, va); - pte2_store(pte2p, npte2); - if (opte2 & PTE2_A) { - if (pte2_is_managed(opte2)) - vm_page_aflag_set(om, PGA_REFERENCED); - } - if (pte2_is_dirty(opte2)) { - if (pte2_is_managed(opte2)) - vm_page_dirty(om); - } - if (pte2_is_managed(opte2) && - TAILQ_EMPTY(&om->md.pv_list) && - ((om->flags & PG_FICTITIOUS) != 0 || - TAILQ_EMPTY(&pa_to_pvh(opa)->pv_list))) - vm_page_aflag_clear(om, PGA_WRITEABLE); - } else - pte2_store(pte2p, npte2); - } -#if 0 - else { - /* - * QQQ: In time when both access and not mofified bits are - * emulated by software, this should not happen. Some - * analysis is need, if this really happen. Missing - * tlb flush somewhere could be the reason. - */ - panic("%s: pmap %p va %#x opte2 %x npte2 %x !!", __func__, pmap, - va, opte2, npte2); - } -#endif - /* - * If both the L2 page table page and the reservation are fully - * populated, then attempt promotion. - */ - if ((mpte2 == NULL || pt2_is_full(mpte2, va)) && - sp_enabled && (m->flags & PG_FICTITIOUS) == 0 && - vm_reserv_level_iffullpop(m) == 0) - pmap_promote_pte1(pmap, pte1p, va); - sched_unpin(); - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); - return (KERN_SUCCESS); -} - -/* - * Do the things to unmap a page in a process. - */ -static int -pmap_remove_pte2(pmap_t pmap, pt2_entry_t *pte2p, vm_offset_t va, - struct spglist *free) -{ - pt2_entry_t opte2; - vm_page_t m; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - - /* Clear and invalidate the mapping. */ - opte2 = pte2_load_clear(pte2p); - pmap_tlb_flush(pmap, va); - - KASSERT(pte2_is_valid(opte2), ("%s: pmap %p va %#x not link pte2 %#x", - __func__, pmap, va, opte2)); - - if (opte2 & PTE2_W) - pmap->pm_stats.wired_count -= 1; - pmap->pm_stats.resident_count -= 1; - if (pte2_is_managed(opte2)) { - m = PHYS_TO_VM_PAGE(pte2_pa(opte2)); - if (pte2_is_dirty(opte2)) - vm_page_dirty(m); - if (opte2 & PTE2_A) - vm_page_aflag_set(m, PGA_REFERENCED); - pmap_remove_entry(pmap, m, va); - } - return (pmap_unuse_pt2(pmap, va, free)); -} - -/* - * Remove a single page from a process address space. - */ -static void -pmap_remove_page(pmap_t pmap, vm_offset_t va, struct spglist *free) -{ - pt2_entry_t *pte2p; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - KASSERT(curthread->td_pinned > 0, - ("%s: curthread not pinned", __func__)); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - if ((pte2p = pmap_pte2_quick(pmap, va)) == NULL || - !pte2_is_valid(pte2_load(pte2p))) - return; - pmap_remove_pte2(pmap, pte2p, va, free); -} - -/* - * Remove the given range of addresses from the specified map. - * - * It is assumed that the start and end are properly - * rounded to the page size. - */ -void -pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) -{ - vm_offset_t nextva; - pt1_entry_t *pte1p, pte1; - pt2_entry_t *pte2p, pte2; - struct spglist free; - - /* - * Perform an unsynchronized read. This is, however, safe. - */ - if (pmap->pm_stats.resident_count == 0) - return; - - SLIST_INIT(&free); - - rw_wlock(&pvh_global_lock); - sched_pin(); - PMAP_LOCK(pmap); - - /* - * Special handling of removing one page. A very common - * operation and easy to short circuit some code. - */ - if (sva + PAGE_SIZE == eva) { - pte1 = pte1_load(pmap_pte1(pmap, sva)); - if (pte1_is_link(pte1)) { - pmap_remove_page(pmap, sva, &free); - goto out; - } - } - - for (; sva < eva; sva = nextva) { - /* - * Calculate address for next L2 page table. - */ - nextva = pte1_trunc(sva + PTE1_SIZE); - if (nextva < sva) - nextva = eva; - if (pmap->pm_stats.resident_count == 0) - break; - - pte1p = pmap_pte1(pmap, sva); - pte1 = pte1_load(pte1p); - - /* - * Weed out invalid mappings. Note: we assume that the L1 page - * table is always allocated, and in kernel virtual. - */ - if (pte1 == 0) - continue; - - if (pte1_is_section(pte1)) { - /* - * Are we removing the entire large page? If not, - * demote the mapping and fall through. - */ - if (sva + PTE1_SIZE == nextva && eva >= nextva) { - pmap_remove_pte1(pmap, pte1p, sva, &free); - continue; - } else if (!pmap_demote_pte1(pmap, pte1p, sva)) { - /* The large page mapping was destroyed. */ - continue; - } -#ifdef INVARIANTS - else { - /* Update pte1 after demotion. */ - pte1 = pte1_load(pte1p); - } -#endif - } - - KASSERT(pte1_is_link(pte1), ("%s: pmap %p va %#x pte1 %#x at %p" - " is not link", __func__, pmap, sva, pte1, pte1p)); - - /* - * Limit our scan to either the end of the va represented - * by the current L2 page table page, or to the end of the - * range being removed. - */ - if (nextva > eva) - nextva = eva; - - for (pte2p = pmap_pte2_quick(pmap, sva); sva != nextva; - pte2p++, sva += PAGE_SIZE) { - pte2 = pte2_load(pte2p); - if (!pte2_is_valid(pte2)) - continue; - if (pmap_remove_pte2(pmap, pte2p, sva, &free)) - break; - } - } -out: - sched_unpin(); - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); - pmap_free_zero_pages(&free); -} - -/* - * Routine: pmap_remove_all - * Function: - * Removes this physical page from - * all physical maps in which it resides. - * Reflects back modify bits to the pager. - * - * Notes: - * Original versions of this routine were very - * inefficient because they iteratively called - * pmap_remove (slow...) - */ - -void -pmap_remove_all(vm_page_t m) -{ - struct md_page *pvh; - pv_entry_t pv; - pmap_t pmap; - pt2_entry_t *pte2p, opte2; - pt1_entry_t *pte1p; - vm_offset_t va; - struct spglist free; - - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("%s: page %p is not managed", __func__, m)); - SLIST_INIT(&free); - rw_wlock(&pvh_global_lock); - sched_pin(); - if ((m->flags & PG_FICTITIOUS) != 0) - goto small_mappings; - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - while ((pv = TAILQ_FIRST(&pvh->pv_list)) != NULL) { - va = pv->pv_va; - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1p = pmap_pte1(pmap, va); - (void)pmap_demote_pte1(pmap, pte1p, va); - PMAP_UNLOCK(pmap); - } -small_mappings: - while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pmap->pm_stats.resident_count--; - pte1p = pmap_pte1(pmap, pv->pv_va); - KASSERT(!pte1_is_section(pte1_load(pte1p)), ("%s: found " - "a 1mpage in page %p's pv list", __func__, m)); - pte2p = pmap_pte2_quick(pmap, pv->pv_va); - opte2 = pte2_load_clear(pte2p); - pmap_tlb_flush(pmap, pv->pv_va); - KASSERT(pte2_is_valid(opte2), ("%s: pmap %p va %x zero pte2", - __func__, pmap, pv->pv_va)); - if (pte2_is_wired(opte2)) - pmap->pm_stats.wired_count--; - if (opte2 & PTE2_A) - vm_page_aflag_set(m, PGA_REFERENCED); - - /* - * Update the vm_page_t clean and reference bits. - */ - if (pte2_is_dirty(opte2)) - vm_page_dirty(m); - pmap_unuse_pt2(pmap, pv->pv_va, &free); - TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); - free_pv_entry(pmap, pv); - PMAP_UNLOCK(pmap); - } - vm_page_aflag_clear(m, PGA_WRITEABLE); - sched_unpin(); - rw_wunlock(&pvh_global_lock); - pmap_free_zero_pages(&free); -} - -/* - * Just subroutine for pmap_remove_pages() to reasonably satisfy - * good coding style, a.k.a. 80 character line width limit hell. - */ -static __inline void -pmap_remove_pte1_quick(pmap_t pmap, pt1_entry_t pte1, pv_entry_t pv, - struct spglist *free) -{ - vm_paddr_t pa; - vm_page_t m, mt, mpt2pg; - struct md_page *pvh; - - pa = pte1_pa(pte1); - m = PHYS_TO_VM_PAGE(pa); - - KASSERT(m->phys_addr == pa, ("%s: vm_page_t %p addr mismatch %#x %#x", - __func__, m, m->phys_addr, pa)); - KASSERT((m->flags & PG_FICTITIOUS) != 0 || - m < &vm_page_array[vm_page_array_size], - ("%s: bad pte1 %#x", __func__, pte1)); - - if (pte1_is_dirty(pte1)) { - for (mt = m; mt < &m[PTE1_SIZE / PAGE_SIZE]; mt++) - vm_page_dirty(mt); - } - - pmap->pm_stats.resident_count -= PTE1_SIZE / PAGE_SIZE; - pvh = pa_to_pvh(pa); - TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); - if (TAILQ_EMPTY(&pvh->pv_list)) { - for (mt = m; mt < &m[PTE1_SIZE / PAGE_SIZE]; mt++) - if (TAILQ_EMPTY(&mt->md.pv_list)) - vm_page_aflag_clear(mt, PGA_WRITEABLE); - } - mpt2pg = pmap_pt2_page(pmap, pv->pv_va); - if (mpt2pg != NULL) - pmap_unwire_pt2_all(pmap, pv->pv_va, mpt2pg, free); -} - -/* - * Just subroutine for pmap_remove_pages() to reasonably satisfy - * good coding style, a.k.a. 80 character line width limit hell. - */ -static __inline void -pmap_remove_pte2_quick(pmap_t pmap, pt2_entry_t pte2, pv_entry_t pv, - struct spglist *free) -{ - vm_paddr_t pa; - vm_page_t m; - struct md_page *pvh; - - pa = pte2_pa(pte2); - m = PHYS_TO_VM_PAGE(pa); - - KASSERT(m->phys_addr == pa, ("%s: vm_page_t %p addr mismatch %#x %#x", - __func__, m, m->phys_addr, pa)); - KASSERT((m->flags & PG_FICTITIOUS) != 0 || - m < &vm_page_array[vm_page_array_size], - ("%s: bad pte2 %#x", __func__, pte2)); - - if (pte2_is_dirty(pte2)) - vm_page_dirty(m); - - pmap->pm_stats.resident_count--; - TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); - if (TAILQ_EMPTY(&m->md.pv_list) && (m->flags & PG_FICTITIOUS) == 0) { - pvh = pa_to_pvh(pa); - if (TAILQ_EMPTY(&pvh->pv_list)) - vm_page_aflag_clear(m, PGA_WRITEABLE); - } - pmap_unuse_pt2(pmap, pv->pv_va, free); -} - -/* - * Remove all pages from specified address space this aids process - * exit speeds. Also, this code is special cased for current process - * only, but can have the more generic (and slightly slower) mode enabled. - * This is much faster than pmap_remove in the case of running down - * an entire address space. - */ -void -pmap_remove_pages(pmap_t pmap) -{ - pt1_entry_t *pte1p, pte1; - pt2_entry_t *pte2p, pte2; - pv_entry_t pv; - struct pv_chunk *pc, *npc; - struct spglist free; - int field, idx; - int32_t bit; - uint32_t inuse, bitmask; - boolean_t allfree; - - /* - * Assert that the given pmap is only active on the current - * CPU. Unfortunately, we cannot block another CPU from - * activating the pmap while this function is executing. - */ - KASSERT(pmap == vmspace_pmap(curthread->td_proc->p_vmspace), - ("%s: non-current pmap %p", __func__, pmap)); -#if defined(SMP) && defined(INVARIANTS) - { - cpuset_t other_cpus; - - sched_pin(); - other_cpus = pmap->pm_active; - CPU_CLR(PCPU_GET(cpuid), &other_cpus); - sched_unpin(); - KASSERT(CPU_EMPTY(&other_cpus), - ("%s: pmap %p active on other cpus", __func__, pmap)); - } -#endif - SLIST_INIT(&free); - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - sched_pin(); - TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { - KASSERT(pc->pc_pmap == pmap, ("%s: wrong pmap %p %p", - __func__, pmap, pc->pc_pmap)); - allfree = TRUE; - for (field = 0; field < _NPCM; field++) { - inuse = (~(pc->pc_map[field])) & pc_freemask[field]; - while (inuse != 0) { - bit = ffs(inuse) - 1; - bitmask = 1UL << bit; - idx = field * 32 + bit; - pv = &pc->pc_pventry[idx]; - inuse &= ~bitmask; - - /* - * Note that we cannot remove wired pages - * from a process' mapping at this time - */ - pte1p = pmap_pte1(pmap, pv->pv_va); - pte1 = pte1_load(pte1p); - if (pte1_is_section(pte1)) { - if (pte1_is_wired(pte1)) { - allfree = FALSE; - continue; - } - pte1_clear(pte1p); - pmap_remove_pte1_quick(pmap, pte1, pv, - &free); - } - else if (pte1_is_link(pte1)) { - pte2p = pt2map_entry(pv->pv_va); - pte2 = pte2_load(pte2p); - - if (!pte2_is_valid(pte2)) { - printf("%s: pmap %p va %#x " - "pte2 %#x\n", __func__, - pmap, pv->pv_va, pte2); - panic("bad pte2"); - } - - if (pte2_is_wired(pte2)) { - allfree = FALSE; - continue; - } - pte2_clear(pte2p); - pmap_remove_pte2_quick(pmap, pte2, pv, - &free); - } else { - printf("%s: pmap %p va %#x pte1 %#x\n", - __func__, pmap, pv->pv_va, pte1); - panic("bad pte1"); - } - - /* Mark free */ - PV_STAT(pv_entry_frees++); - PV_STAT(pv_entry_spare++); - pv_entry_count--; - pc->pc_map[field] |= bitmask; - } - } - if (allfree) { - TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - free_pv_chunk(pc); - } - } - tlb_flush_all_ng_local(); - sched_unpin(); - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); - pmap_free_zero_pages(&free); -} - -/* - * This code makes some *MAJOR* assumptions: - * 1. Current pmap & pmap exists. - * 2. Not wired. - * 3. Read access. - * 4. No L2 page table pages. - * but is *MUCH* faster than pmap_enter... - */ -static vm_page_t -pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, - vm_prot_t prot, vm_page_t mpt2pg) -{ - pt2_entry_t *pte2p, pte2; - vm_paddr_t pa; - struct spglist free; - uint32_t l2prot; - - KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva || - (m->oflags & VPO_UNMANAGED) != 0, - ("%s: managed mapping within the clean submap", __func__)); - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - - /* - * In the case that a L2 page table page is not - * resident, we are creating it here. - */ - if (va < VM_MAXUSER_ADDRESS) { - u_int pte1_idx; - pt1_entry_t pte1, *pte1p; - vm_paddr_t pt2_pa; - - /* - * Get L1 page table things. - */ - pte1_idx = pte1_index(va); - pte1p = pmap_pte1(pmap, va); - pte1 = pte1_load(pte1p); - - if (mpt2pg && (mpt2pg->pindex == (pte1_idx & ~PT2PG_MASK))) { - /* - * Each of NPT2_IN_PG L2 page tables on the page can - * come here. Make sure that associated L1 page table - * link is established. - * - * QQQ: It comes that we don't establish all links to - * L2 page tables for newly allocated L2 page - * tables page. - */ - KASSERT(!pte1_is_section(pte1), - ("%s: pte1 %#x is section", __func__, pte1)); - if (!pte1_is_link(pte1)) { - pt2_pa = page_pt2pa(VM_PAGE_TO_PHYS(mpt2pg), - pte1_idx); - pte1_store(pte1p, PTE1_LINK(pt2_pa)); - } - pt2_wirecount_inc(mpt2pg, pte1_idx); - } else { - /* - * If the L2 page table page is mapped, we just - * increment the hold count, and activate it. - */ - if (pte1_is_section(pte1)) { - return (NULL); - } else if (pte1_is_link(pte1)) { - mpt2pg = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); - pt2_wirecount_inc(mpt2pg, pte1_idx); - } else { - mpt2pg = _pmap_allocpte2(pmap, va, - PMAP_ENTER_NOSLEEP); - if (mpt2pg == NULL) - return (NULL); - } - } - } else { - mpt2pg = NULL; - } - - /* - * This call to pt2map_entry() makes the assumption that we are - * entering the page into the current pmap. In order to support - * quick entry into any pmap, one would likely use pmap_pte2_quick(). - * But that isn't as quick as pt2map_entry(). - */ - pte2p = pt2map_entry(va); - pte2 = pte2_load(pte2p); - if (pte2_is_valid(pte2)) { - if (mpt2pg != NULL) { - /* - * Remove extra pte2 reference - */ - pt2_wirecount_dec(mpt2pg, pte1_index(va)); - mpt2pg = NULL; - } - return (NULL); - } - - /* - * Enter on the PV list if part of our managed memory. - */ - if ((m->oflags & VPO_UNMANAGED) == 0 && - !pmap_try_insert_pv_entry(pmap, va, m)) { - if (mpt2pg != NULL) { - SLIST_INIT(&free); - if (pmap_unwire_pt2(pmap, va, mpt2pg, &free)) { - pmap_tlb_flush(pmap, va); - pmap_free_zero_pages(&free); - } - - mpt2pg = NULL; - } - return (NULL); - } - - /* - * Increment counters - */ - pmap->pm_stats.resident_count++; - - /* - * Now validate mapping with RO protection - */ - pa = VM_PAGE_TO_PHYS(m); - l2prot = PTE2_RO | PTE2_NM; - if (va < VM_MAXUSER_ADDRESS) - l2prot |= PTE2_U | PTE2_NG; - if ((prot & VM_PROT_EXECUTE) == 0) - l2prot |= PTE2_NX; - else if (m->md.pat_mode == PTE2_ATTR_WB_WA && pmap != kernel_pmap) { - /* - * Sync icache if exec permission and attribute PTE2_ATTR_WB_WA - * is set. QQQ: For more info, see comments in pmap_enter(). - */ - cache_icache_sync_fresh(va, pa, PAGE_SIZE); - } - pte2_store(pte2p, PTE2(pa, l2prot, m->md.pat_mode)); - - return (mpt2pg); -} - -void -pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) -{ - - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - (void)pmap_enter_quick_locked(pmap, va, m, prot, NULL); - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); -} - -/* - * Tries to create 1MB page mapping. Returns TRUE if successful and - * FALSE otherwise. Fails if (1) a page table page cannot be allocated without - * blocking, (2) a mapping already exists at the specified virtual address, or - * (3) a pv entry cannot be allocated without reclaiming another pv entry. - */ -static boolean_t -pmap_enter_pte1(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) -{ - pt1_entry_t *pte1p; - vm_paddr_t pa; - uint32_t l1prot; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - pte1p = pmap_pte1(pmap, va); - if (pte1_is_valid(pte1_load(pte1p))) { - CTR3(KTR_PMAP, "%s: failure for va %#lx in pmap %p", __func__, - va, pmap); - return (FALSE); - } - if ((m->oflags & VPO_UNMANAGED) == 0) { - /* - * Abort this mapping if its PV entry could not be created. - */ - if (!pmap_pv_insert_pte1(pmap, va, VM_PAGE_TO_PHYS(m))) { - CTR3(KTR_PMAP, "%s: failure for va %#lx in pmap %p", - __func__, va, pmap); - return (FALSE); - } - } - /* - * Increment counters. - */ - pmap->pm_stats.resident_count += PTE1_SIZE / PAGE_SIZE; - - /* - * Map the section. - * - * QQQ: Why VM_PROT_WRITE is not evaluated and the mapping is - * made readonly? - */ - pa = VM_PAGE_TO_PHYS(m); - l1prot = PTE1_RO | PTE1_NM; - if (va < VM_MAXUSER_ADDRESS) - l1prot |= PTE1_U | PTE1_NG; - if ((prot & VM_PROT_EXECUTE) == 0) - l1prot |= PTE1_NX; - else if (m->md.pat_mode == PTE2_ATTR_WB_WA && pmap != kernel_pmap) { - /* - * Sync icache if exec permission and attribute PTE2_ATTR_WB_WA - * is set. QQQ: For more info, see comments in pmap_enter(). - */ - cache_icache_sync_fresh(va, pa, PTE1_SIZE); - } - pte1_store(pte1p, PTE1(pa, l1prot, ATTR_TO_L1(m->md.pat_mode))); - - pmap_pte1_mappings++; - CTR3(KTR_PMAP, "%s: success for va %#lx in pmap %p", __func__, va, - pmap); - return (TRUE); -} - -/* - * Maps a sequence of resident pages belonging to the same object. - * The sequence begins with the given page m_start. This page is - * mapped at the given virtual address start. Each subsequent page is - * mapped at a virtual address that is offset from start by the same - * amount as the page is offset from m_start within the object. The - * last page in the sequence is the page with the largest offset from - * m_start that can be mapped at a virtual address less than the given - * virtual address end. Not every virtual page between start and end - * is mapped; only those for which a resident page exists with the - * corresponding offset from m_start are mapped. - */ -void -pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, - vm_page_t m_start, vm_prot_t prot) -{ - vm_offset_t va; - vm_page_t m, mpt2pg; - vm_pindex_t diff, psize; - - PDEBUG(6, printf("%s: pmap %p start %#x end %#x m %p prot %#x\n", - __func__, pmap, start, end, m_start, prot)); - - VM_OBJECT_ASSERT_LOCKED(m_start->object); - psize = atop(end - start); - mpt2pg = NULL; - m = m_start; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { - va = start + ptoa(diff); - if ((va & PTE1_OFFSET) == 0 && va + PTE1_SIZE <= end && - m->psind == 1 && sp_enabled && - pmap_enter_pte1(pmap, va, m, prot)) - m = &m[PTE1_SIZE / PAGE_SIZE - 1]; - else - mpt2pg = pmap_enter_quick_locked(pmap, va, m, prot, - mpt2pg); - m = TAILQ_NEXT(m, listq); - } - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); -} - -/* - * This code maps large physical mmap regions into the - * processor address space. Note that some shortcuts - * are taken, but the code works. - */ -void -pmap_object_init_pt(pmap_t pmap, vm_offset_t addr, vm_object_t object, - vm_pindex_t pindex, vm_size_t size) -{ - pt1_entry_t *pte1p; - vm_paddr_t pa, pte2_pa; - vm_page_t p; - int pat_mode; - u_int l1attr, l1prot; - - VM_OBJECT_ASSERT_WLOCKED(object); - KASSERT(object->type == OBJT_DEVICE || object->type == OBJT_SG, - ("%s: non-device object", __func__)); - if ((addr & PTE1_OFFSET) == 0 && (size & PTE1_OFFSET) == 0) { - if (!vm_object_populate(object, pindex, pindex + atop(size))) - return; - p = vm_page_lookup(object, pindex); - KASSERT(p->valid == VM_PAGE_BITS_ALL, - ("%s: invalid page %p", __func__, p)); - pat_mode = p->md.pat_mode; - - /* - * Abort the mapping if the first page is not physically - * aligned to a 1MB page boundary. - */ - pte2_pa = VM_PAGE_TO_PHYS(p); - if (pte2_pa & PTE1_OFFSET) - return; - - /* - * Skip the first page. Abort the mapping if the rest of - * the pages are not physically contiguous or have differing - * memory attributes. - */ - p = TAILQ_NEXT(p, listq); - for (pa = pte2_pa + PAGE_SIZE; pa < pte2_pa + size; - pa += PAGE_SIZE) { - KASSERT(p->valid == VM_PAGE_BITS_ALL, - ("%s: invalid page %p", __func__, p)); - if (pa != VM_PAGE_TO_PHYS(p) || - pat_mode != p->md.pat_mode) - return; - p = TAILQ_NEXT(p, listq); - } - - /* - * Map using 1MB pages. - * - * QQQ: Well, we are mapping a section, so same condition must - * be hold like during promotion. It looks that only RW mapping - * is done here, so readonly mapping must be done elsewhere. - */ - l1prot = PTE1_U | PTE1_NG | PTE1_RW | PTE1_M | PTE1_A; - l1attr = ATTR_TO_L1(pat_mode); - PMAP_LOCK(pmap); - for (pa = pte2_pa; pa < pte2_pa + size; pa += PTE1_SIZE) { - pte1p = pmap_pte1(pmap, addr); - if (!pte1_is_valid(pte1_load(pte1p))) { - pte1_store(pte1p, PTE1(pa, l1prot, l1attr)); - pmap->pm_stats.resident_count += PTE1_SIZE / - PAGE_SIZE; - pmap_pte1_mappings++; - } - /* Else continue on if the PTE1 is already valid. */ - addr += PTE1_SIZE; - } - PMAP_UNLOCK(pmap); - } -} - -/* - * Do the things to protect a 1mpage in a process. - */ -static void -pmap_protect_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t sva, - vm_prot_t prot) -{ - pt1_entry_t npte1, opte1; - vm_offset_t eva, va; - vm_page_t m; - - PMAP_LOCK_ASSERT(pmap, MA_OWNED); - KASSERT((sva & PTE1_OFFSET) == 0, - ("%s: sva is not 1mpage aligned", __func__)); -retry: - opte1 = npte1 = pte1_load(pte1p); - if (pte1_is_managed(opte1)) { - eva = sva + PTE1_SIZE; - for (va = sva, m = PHYS_TO_VM_PAGE(pte1_pa(opte1)); - va < eva; va += PAGE_SIZE, m++) - if (pte1_is_dirty(opte1)) - vm_page_dirty(m); - } - if ((prot & VM_PROT_WRITE) == 0) - npte1 |= PTE1_RO | PTE1_NM; - if ((prot & VM_PROT_EXECUTE) == 0) - npte1 |= PTE1_NX; - - /* - * QQQ: Herein, execute permission is never set. - * It only can be cleared. So, no icache - * syncing is needed. - */ - - if (npte1 != opte1) { - if (!pte1_cmpset(pte1p, opte1, npte1)) - goto retry; - pmap_tlb_flush(pmap, sva); - } -} - -/* - * Set the physical protection on the - * specified range of this map as requested. - */ -void -pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) -{ - boolean_t pv_lists_locked; - vm_offset_t nextva; - pt1_entry_t *pte1p, pte1; - pt2_entry_t *pte2p, opte2, npte2; - - KASSERT((prot & ~VM_PROT_ALL) == 0, ("invalid prot %x", prot)); - if (prot == VM_PROT_NONE) { - pmap_remove(pmap, sva, eva); - return; - } - - if ((prot & (VM_PROT_WRITE | VM_PROT_EXECUTE)) == - (VM_PROT_WRITE | VM_PROT_EXECUTE)) - return; - - if (pmap_is_current(pmap)) - pv_lists_locked = FALSE; - else { - pv_lists_locked = TRUE; -resume: - rw_wlock(&pvh_global_lock); - sched_pin(); - } - - PMAP_LOCK(pmap); - for (; sva < eva; sva = nextva) { - /* - * Calculate address for next L2 page table. - */ - nextva = pte1_trunc(sva + PTE1_SIZE); - if (nextva < sva) - nextva = eva; - - pte1p = pmap_pte1(pmap, sva); - pte1 = pte1_load(pte1p); - - /* - * Weed out invalid mappings. Note: we assume that L1 page - * page table is always allocated, and in kernel virtual. - */ - if (pte1 == 0) - continue; - - if (pte1_is_section(pte1)) { - /* - * Are we protecting the entire large page? If not, - * demote the mapping and fall through. - */ - if (sva + PTE1_SIZE == nextva && eva >= nextva) { - pmap_protect_pte1(pmap, pte1p, sva, prot); - continue; - } else { - if (!pv_lists_locked) { - pv_lists_locked = TRUE; - if (!rw_try_wlock(&pvh_global_lock)) { - PMAP_UNLOCK(pmap); - goto resume; - } - sched_pin(); - } - if (!pmap_demote_pte1(pmap, pte1p, sva)) { - /* - * The large page mapping - * was destroyed. - */ - continue; - } -#ifdef INVARIANTS - else { - /* Update pte1 after demotion */ - pte1 = pte1_load(pte1p); - } -#endif - } - } - - KASSERT(pte1_is_link(pte1), ("%s: pmap %p va %#x pte1 %#x at %p" - " is not link", __func__, pmap, sva, pte1, pte1p)); - - /* - * Limit our scan to either the end of the va represented - * by the current L2 page table page, or to the end of the - * range being protected. - */ - if (nextva > eva) - nextva = eva; - - for (pte2p = pmap_pte2_quick(pmap, sva); sva != nextva; pte2p++, - sva += PAGE_SIZE) { - vm_page_t m; -retry: - opte2 = npte2 = pte2_load(pte2p); - if (!pte2_is_valid(opte2)) - continue; - - if ((prot & VM_PROT_WRITE) == 0) { - if (pte2_is_managed(opte2) && - pte2_is_dirty(opte2)) { - m = PHYS_TO_VM_PAGE(pte2_pa(opte2)); - vm_page_dirty(m); - } - npte2 |= PTE2_RO | PTE2_NM; - } - - if ((prot & VM_PROT_EXECUTE) == 0) - npte2 |= PTE2_NX; - - /* - * QQQ: Herein, execute permission is never set. - * It only can be cleared. So, no icache - * syncing is needed. - */ - - if (npte2 != opte2) { - - if (!pte2_cmpset(pte2p, opte2, npte2)) - goto retry; - pmap_tlb_flush(pmap, sva); - } - } - } - if (pv_lists_locked) { - sched_unpin(); - rw_wunlock(&pvh_global_lock); - } - PMAP_UNLOCK(pmap); -} - -/* - * pmap_pvh_wired_mappings: - * - * Return the updated number "count" of managed mappings that are wired. - */ -static int -pmap_pvh_wired_mappings(struct md_page *pvh, int count) -{ - pmap_t pmap; - pt1_entry_t pte1; - pt2_entry_t pte2; - pv_entry_t pv; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - sched_pin(); - TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1 = pte1_load(pmap_pte1(pmap, pv->pv_va)); - if (pte1_is_section(pte1)) { - if (pte1_is_wired(pte1)) - count++; - } else { - KASSERT(pte1_is_link(pte1), - ("%s: pte1 %#x is not link", __func__, pte1)); - pte2 = pte2_load(pmap_pte2_quick(pmap, pv->pv_va)); - if (pte2_is_wired(pte2)) - count++; - } - PMAP_UNLOCK(pmap); - } - sched_unpin(); - return (count); -} - -/* - * pmap_page_wired_mappings: - * - * Return the number of managed mappings to the given physical page - * that are wired. - */ -int -pmap_page_wired_mappings(vm_page_t m) -{ - int count; - - count = 0; - if ((m->oflags & VPO_UNMANAGED) != 0) - return (count); - rw_wlock(&pvh_global_lock); - count = pmap_pvh_wired_mappings(&m->md, count); - if ((m->flags & PG_FICTITIOUS) == 0) { - count = pmap_pvh_wired_mappings(pa_to_pvh(VM_PAGE_TO_PHYS(m)), - count); - } - rw_wunlock(&pvh_global_lock); - return (count); -} - -/* - * Returns TRUE if any of the given mappings were used to modify - * physical memory. Otherwise, returns FALSE. Both page and 1mpage - * mappings are supported. - */ -static boolean_t -pmap_is_modified_pvh(struct md_page *pvh) -{ - pv_entry_t pv; - pt1_entry_t pte1; - pt2_entry_t pte2; - pmap_t pmap; - boolean_t rv; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - rv = FALSE; - sched_pin(); - TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1 = pte1_load(pmap_pte1(pmap, pv->pv_va)); - if (pte1_is_section(pte1)) { - rv = pte1_is_dirty(pte1); - } else { - KASSERT(pte1_is_link(pte1), - ("%s: pte1 %#x is not link", __func__, pte1)); - pte2 = pte2_load(pmap_pte2_quick(pmap, pv->pv_va)); - rv = pte2_is_dirty(pte2); - } - PMAP_UNLOCK(pmap); - if (rv) - break; - } - sched_unpin(); - return (rv); -} - -/* - * pmap_is_modified: - * - * Return whether or not the specified physical page was modified - * in any physical maps. - */ -boolean_t -pmap_is_modified(vm_page_t m) -{ - boolean_t rv; - - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("%s: page %p is not managed", __func__, m)); - - /* - * If the page is not exclusive busied, then PGA_WRITEABLE cannot be - * concurrently set while the object is locked. Thus, if PGA_WRITEABLE - * is clear, no PTE2s can have PG_M set. - */ - VM_OBJECT_ASSERT_WLOCKED(m->object); - if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) - return (FALSE); - rw_wlock(&pvh_global_lock); - rv = pmap_is_modified_pvh(&m->md) || - ((m->flags & PG_FICTITIOUS) == 0 && - pmap_is_modified_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m)))); - rw_wunlock(&pvh_global_lock); - return (rv); -} - -/* - * pmap_is_prefaultable: - * - * Return whether or not the specified virtual address is eligible - * for prefault. - */ -boolean_t -pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr) -{ - pt1_entry_t pte1; - pt2_entry_t pte2; - boolean_t rv; - - rv = FALSE; - PMAP_LOCK(pmap); - pte1 = pte1_load(pmap_pte1(pmap, addr)); - if (pte1_is_link(pte1)) { - pte2 = pte2_load(pt2map_entry(addr)); - rv = !pte2_is_valid(pte2) ; - } - PMAP_UNLOCK(pmap); - return (rv); -} - -/* - * Returns TRUE if any of the given mappings were referenced and FALSE - * otherwise. Both page and 1mpage mappings are supported. - */ -static boolean_t -pmap_is_referenced_pvh(struct md_page *pvh) -{ - - pv_entry_t pv; - pt1_entry_t pte1; - pt2_entry_t pte2; - pmap_t pmap; - boolean_t rv; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - rv = FALSE; - sched_pin(); - TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1 = pte1_load(pmap_pte1(pmap, pv->pv_va)); - if (pte1_is_section(pte1)) { - rv = (pte1 & (PTE1_A | PTE1_V)) == (PTE1_A | PTE1_V); - } else { - pte2 = pte2_load(pmap_pte2_quick(pmap, pv->pv_va)); - rv = (pte2 & (PTE2_A | PTE2_V)) == (PTE2_A | PTE2_V); - } - PMAP_UNLOCK(pmap); - if (rv) - break; - } - sched_unpin(); - return (rv); -} - -/* - * pmap_is_referenced: - * - * Return whether or not the specified physical page was referenced - * in any physical maps. - */ -boolean_t -pmap_is_referenced(vm_page_t m) -{ - boolean_t rv; - - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("%s: page %p is not managed", __func__, m)); - rw_wlock(&pvh_global_lock); - rv = pmap_is_referenced_pvh(&m->md) || - ((m->flags & PG_FICTITIOUS) == 0 && - pmap_is_referenced_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m)))); - rw_wunlock(&pvh_global_lock); - return (rv); -} - -#define PMAP_TS_REFERENCED_MAX 5 - -/* - * pmap_ts_referenced: - * - * Return a count of reference bits for a page, clearing those bits. - * It is not necessary for every reference bit to be cleared, but it - * is necessary that 0 only be returned when there are truly no - * reference bits set. - * - * XXX: The exact number of bits to check and clear is a matter that - * should be tested and standardized at some point in the future for - * optimal aging of shared pages. - */ -int -pmap_ts_referenced(vm_page_t m) -{ - struct md_page *pvh; - pv_entry_t pv, pvf; - pmap_t pmap; - pt1_entry_t *pte1p, opte1; - pt2_entry_t *pte2p; - vm_paddr_t pa; - int rtval = 0; - - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("%s: page %p is not managed", __func__, m)); - pa = VM_PAGE_TO_PHYS(m); - pvh = pa_to_pvh(pa); - rw_wlock(&pvh_global_lock); - sched_pin(); - if ((m->flags & PG_FICTITIOUS) != 0 || - (pvf = TAILQ_FIRST(&pvh->pv_list)) == NULL) - goto small_mappings; - pv = pvf; - do { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1p = pmap_pte1(pmap, pv->pv_va); - opte1 = pte1_load(pte1p); - if ((opte1 & PTE1_A) != 0) { - /* - * Since this reference bit is shared by 256 4KB pages, - * it should not be cleared every time it is tested. - * Apply a simple "hash" function on the physical page - * number, the virtual section number, and the pmap - * address to select one 4KB page out of the 256 - * on which testing the reference bit will result - * in clearing that bit. This function is designed - * to avoid the selection of the same 4KB page - * for every 1MB page mapping. - * - * On demotion, a mapping that hasn't been referenced - * is simply destroyed. To avoid the possibility of a - * subsequent page fault on a demoted wired mapping, - * always leave its reference bit set. Moreover, - * since the section is wired, the current state of - * its reference bit won't affect page replacement. - */ - if ((((pa >> PAGE_SHIFT) ^ (pv->pv_va >> PTE1_SHIFT) ^ - (uintptr_t)pmap) & (NPTE2_IN_PG - 1)) == 0 && - !pte1_is_wired(opte1)) { - pte1_clear_bit(pte1p, PTE1_A); - pmap_tlb_flush(pmap, pv->pv_va); - } - rtval++; - } - PMAP_UNLOCK(pmap); - /* Rotate the PV list if it has more than one entry. */ - if (TAILQ_NEXT(pv, pv_next) != NULL) { - TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); - TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); - } - if (rtval >= PMAP_TS_REFERENCED_MAX) - goto out; - } while ((pv = TAILQ_FIRST(&pvh->pv_list)) != pvf); -small_mappings: - if ((pvf = TAILQ_FIRST(&m->md.pv_list)) == NULL) - goto out; - pv = pvf; - do { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1p = pmap_pte1(pmap, pv->pv_va); - KASSERT(pte1_is_link(pte1_load(pte1p)), - ("%s: not found a link in page %p's pv list", __func__, m)); - - pte2p = pmap_pte2_quick(pmap, pv->pv_va); - if ((pte2_load(pte2p) & PTE2_A) != 0) { - pte2_clear_bit(pte2p, PTE2_A); - pmap_tlb_flush(pmap, pv->pv_va); - rtval++; - } - PMAP_UNLOCK(pmap); - /* Rotate the PV list if it has more than one entry. */ - if (TAILQ_NEXT(pv, pv_next) != NULL) { - TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); - TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); - } - } while ((pv = TAILQ_FIRST(&m->md.pv_list)) != pvf && rtval < - PMAP_TS_REFERENCED_MAX); -out: - sched_unpin(); - rw_wunlock(&pvh_global_lock); - return (rtval); -} - -/* - * Clear the wired attribute from the mappings for the specified range of - * addresses in the given pmap. Every valid mapping within that range - * must have the wired attribute set. In contrast, invalid mappings - * cannot have the wired attribute set, so they are ignored. - * - * The wired attribute of the page table entry is not a hardware feature, - * so there is no need to invalidate any TLB entries. - */ -void -pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) -{ - vm_offset_t nextva; - pt1_entry_t *pte1p, pte1; - pt2_entry_t *pte2p, pte2; - boolean_t pv_lists_locked; - - if (pmap_is_current(pmap)) - pv_lists_locked = FALSE; - else { - pv_lists_locked = TRUE; -resume: - rw_wlock(&pvh_global_lock); - sched_pin(); - } - PMAP_LOCK(pmap); - for (; sva < eva; sva = nextva) { - nextva = pte1_trunc(sva + PTE1_SIZE); - if (nextva < sva) - nextva = eva; - - pte1p = pmap_pte1(pmap, sva); - pte1 = pte1_load(pte1p); - - /* - * Weed out invalid mappings. Note: we assume that L1 page - * page table is always allocated, and in kernel virtual. - */ - if (pte1 == 0) - continue; - - if (pte1_is_section(pte1)) { - if (!pte1_is_wired(pte1)) - panic("%s: pte1 %#x not wired", __func__, pte1); - - /* - * Are we unwiring the entire large page? If not, - * demote the mapping and fall through. - */ - if (sva + PTE1_SIZE == nextva && eva >= nextva) { - pte1_clear_bit(pte1p, PTE1_W); - pmap->pm_stats.wired_count -= PTE1_SIZE / - PAGE_SIZE; - continue; - } else { - if (!pv_lists_locked) { - pv_lists_locked = TRUE; - if (!rw_try_wlock(&pvh_global_lock)) { - PMAP_UNLOCK(pmap); - /* Repeat sva. */ - goto resume; - } - sched_pin(); - } - if (!pmap_demote_pte1(pmap, pte1p, sva)) - panic("%s: demotion failed", __func__); -#ifdef INVARIANTS - else { - /* Update pte1 after demotion */ - pte1 = pte1_load(pte1p); - } -#endif - } - } - - KASSERT(pte1_is_link(pte1), ("%s: pmap %p va %#x pte1 %#x at %p" - " is not link", __func__, pmap, sva, pte1, pte1p)); - - /* - * Limit our scan to either the end of the va represented - * by the current L2 page table page, or to the end of the - * range being protected. - */ - if (nextva > eva) - nextva = eva; - - for (pte2p = pmap_pte2_quick(pmap, sva); sva != nextva; pte2p++, - sva += PAGE_SIZE) { - pte2 = pte2_load(pte2p); - if (!pte2_is_valid(pte2)) - continue; - if (!pte2_is_wired(pte2)) - panic("%s: pte2 %#x is missing PTE2_W", - __func__, pte2); - - /* - * PTE2_W must be cleared atomically. Although the pmap - * lock synchronizes access to PTE2_W, another processor - * could be changing PTE2_NM and/or PTE2_A concurrently. - */ - pte2_clear_bit(pte2p, PTE2_W); - pmap->pm_stats.wired_count--; - } - } - if (pv_lists_locked) { - sched_unpin(); - rw_wunlock(&pvh_global_lock); - } - PMAP_UNLOCK(pmap); -} - -/* - * Clear the write and modified bits in each of the given page's mappings. - */ -void -pmap_remove_write(vm_page_t m) -{ - struct md_page *pvh; - pv_entry_t next_pv, pv; - pmap_t pmap; - pt1_entry_t *pte1p; - pt2_entry_t *pte2p, opte2; - vm_offset_t va; - - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("%s: page %p is not managed", __func__, m)); - - /* - * If the page is not exclusive busied, then PGA_WRITEABLE cannot be - * set by another thread while the object is locked. Thus, - * if PGA_WRITEABLE is clear, no page table entries need updating. - */ - VM_OBJECT_ASSERT_WLOCKED(m->object); - if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) - return; - rw_wlock(&pvh_global_lock); - sched_pin(); - if ((m->flags & PG_FICTITIOUS) != 0) - goto small_mappings; - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_next, next_pv) { - va = pv->pv_va; - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1p = pmap_pte1(pmap, va); - if (!(pte1_load(pte1p) & PTE1_RO)) - (void)pmap_demote_pte1(pmap, pte1p, va); - PMAP_UNLOCK(pmap); - } -small_mappings: - TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1p = pmap_pte1(pmap, pv->pv_va); - KASSERT(!pte1_is_section(pte1_load(pte1p)), ("%s: found" - " a section in page %p's pv list", __func__, m)); - pte2p = pmap_pte2_quick(pmap, pv->pv_va); -retry: - opte2 = pte2_load(pte2p); - if (!(opte2 & PTE2_RO)) { - if (!pte2_cmpset(pte2p, opte2, - opte2 | (PTE2_RO | PTE2_NM))) - goto retry; - if (pte2_is_dirty(opte2)) - vm_page_dirty(m); - pmap_tlb_flush(pmap, pv->pv_va); - } - PMAP_UNLOCK(pmap); - } - vm_page_aflag_clear(m, PGA_WRITEABLE); - sched_unpin(); - rw_wunlock(&pvh_global_lock); -} - -/* - * Apply the given advice to the specified range of addresses within the - * given pmap. Depending on the advice, clear the referenced and/or - * modified flags in each mapping and set the mapped page's dirty field. - */ -void -pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice) -{ - pt1_entry_t *pte1p, opte1; - pt2_entry_t *pte2p, pte2; - vm_offset_t pdnxt; - vm_page_t m; - boolean_t pv_lists_locked; - - if (advice != MADV_DONTNEED && advice != MADV_FREE) - return; - if (pmap_is_current(pmap)) - pv_lists_locked = FALSE; - else { - pv_lists_locked = TRUE; -resume: - rw_wlock(&pvh_global_lock); - sched_pin(); - } - PMAP_LOCK(pmap); - for (; sva < eva; sva = pdnxt) { - pdnxt = pte1_trunc(sva + PTE1_SIZE); - if (pdnxt < sva) - pdnxt = eva; - pte1p = pmap_pte1(pmap, sva); - opte1 = pte1_load(pte1p); - if (!pte1_is_valid(opte1)) /* XXX */ - continue; - else if (pte1_is_section(opte1)) { - if (!pte1_is_managed(opte1)) - continue; - if (!pv_lists_locked) { - pv_lists_locked = TRUE; - if (!rw_try_wlock(&pvh_global_lock)) { - PMAP_UNLOCK(pmap); - goto resume; - } - sched_pin(); - } - if (!pmap_demote_pte1(pmap, pte1p, sva)) { - /* - * The large page mapping was destroyed. - */ - continue; - } - - /* - * Unless the page mappings are wired, remove the - * mapping to a single page so that a subsequent - * access may repromote. Since the underlying L2 page - * table is fully populated, this removal never - * frees a L2 page table page. - */ - if (!pte1_is_wired(opte1)) { - pte2p = pmap_pte2_quick(pmap, sva); - KASSERT(pte2_is_valid(pte2_load(pte2p)), - ("%s: invalid PTE2", __func__)); - pmap_remove_pte2(pmap, pte2p, sva, NULL); - } - } - if (pdnxt > eva) - pdnxt = eva; - for (pte2p = pmap_pte2_quick(pmap, sva); sva != pdnxt; pte2p++, - sva += PAGE_SIZE) { - pte2 = pte2_load(pte2p); - if (!pte2_is_valid(pte2) || !pte2_is_managed(pte2)) - continue; - else if (pte2_is_dirty(pte2)) { - if (advice == MADV_DONTNEED) { - /* - * Future calls to pmap_is_modified() - * can be avoided by making the page - * dirty now. - */ - m = PHYS_TO_VM_PAGE(pte2_pa(pte2)); - vm_page_dirty(m); - } - pte2_set_bit(pte2p, PTE2_NM); - pte2_clear_bit(pte2p, PTE2_A); - } else if ((pte2 & PTE2_A) != 0) - pte2_clear_bit(pte2p, PTE2_A); - else - continue; - pmap_tlb_flush(pmap, sva); - } - } - if (pv_lists_locked) { - sched_unpin(); - rw_wunlock(&pvh_global_lock); - } - PMAP_UNLOCK(pmap); -} - -/* - * Clear the modify bits on the specified physical page. - */ -void -pmap_clear_modify(vm_page_t m) -{ - struct md_page *pvh; - pv_entry_t next_pv, pv; - pmap_t pmap; - pt1_entry_t *pte1p, opte1; - pt2_entry_t *pte2p, opte2; - vm_offset_t va; - - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("%s: page %p is not managed", __func__, m)); - VM_OBJECT_ASSERT_WLOCKED(m->object); - KASSERT(!vm_page_xbusied(m), - ("%s: page %p is exclusive busy", __func__, m)); - - /* - * If the page is not PGA_WRITEABLE, then no PTE2s can have PTE2_NM - * cleared. If the object containing the page is locked and the page - * is not exclusive busied, then PGA_WRITEABLE cannot be concurrently - * set. - */ - if ((m->flags & PGA_WRITEABLE) == 0) - return; - rw_wlock(&pvh_global_lock); - sched_pin(); - if ((m->flags & PG_FICTITIOUS) != 0) - goto small_mappings; - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_next, next_pv) { - va = pv->pv_va; - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1p = pmap_pte1(pmap, va); - opte1 = pte1_load(pte1p); - if (!(opte1 & PTE1_RO)) { - if (pmap_demote_pte1(pmap, pte1p, va) && - !pte1_is_wired(opte1)) { - /* - * Write protect the mapping to a - * single page so that a subsequent - * write access may repromote. - */ - va += VM_PAGE_TO_PHYS(m) - pte1_pa(opte1); - pte2p = pmap_pte2_quick(pmap, va); - opte2 = pte2_load(pte2p); - if ((opte2 & PTE2_V)) { - pte2_set_bit(pte2p, PTE2_NM | PTE2_RO); - vm_page_dirty(m); - pmap_tlb_flush(pmap, va); - } - } - } - PMAP_UNLOCK(pmap); - } -small_mappings: - TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pte1p = pmap_pte1(pmap, pv->pv_va); - KASSERT(!pte1_is_section(pte1_load(pte1p)), ("%s: found" - " a section in page %p's pv list", __func__, m)); - pte2p = pmap_pte2_quick(pmap, pv->pv_va); - if (pte2_is_dirty(pte2_load(pte2p))) { - pte2_set_bit(pte2p, PTE2_NM); - pmap_tlb_flush(pmap, pv->pv_va); - } - PMAP_UNLOCK(pmap); - } - sched_unpin(); - rw_wunlock(&pvh_global_lock); -} - - -/* - * Sets the memory attribute for the specified page. - */ -void -pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma) -{ - struct sysmaps *sysmaps; - vm_memattr_t oma; - vm_paddr_t pa; - - oma = m->md.pat_mode; - m->md.pat_mode = ma; - - CTR5(KTR_PMAP, "%s: page %p - 0x%08X oma: %d, ma: %d", __func__, m, - VM_PAGE_TO_PHYS(m), oma, ma); - if ((m->flags & PG_FICTITIOUS) != 0) - return; -#if 0 - /* - * If "m" is a normal page, flush it from the cache. - * - * First, try to find an existing mapping of the page by sf - * buffer. sf_buf_invalidate_cache() modifies mapping and - * flushes the cache. - */ - if (sf_buf_invalidate_cache(m, oma)) - return; -#endif - /* - * If page is not mapped by sf buffer, map the page - * transient and do invalidation. - */ - if (ma != oma) { - pa = VM_PAGE_TO_PHYS(m); - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (*sysmaps->CMAP2) - panic("%s: CMAP2 busy", __func__); - pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(pa, PTE2_AP_KRW, ma)); - dcache_wbinv_poc((vm_offset_t)sysmaps->CADDR2, pa, PAGE_SIZE); - pte2_clear(sysmaps->CMAP2); - tlb_flush((vm_offset_t)sysmaps->CADDR2); - sched_unpin(); - mtx_unlock(&sysmaps->lock); - } -} - -/* - * Miscellaneous support routines follow - */ - -/* - * Returns TRUE if the given page is mapped individually or as part of - * a 1mpage. Otherwise, returns FALSE. - */ -boolean_t -pmap_page_is_mapped(vm_page_t m) -{ - boolean_t rv; - - if ((m->oflags & VPO_UNMANAGED) != 0) - return (FALSE); - rw_wlock(&pvh_global_lock); - rv = !TAILQ_EMPTY(&m->md.pv_list) || - ((m->flags & PG_FICTITIOUS) == 0 && - !TAILQ_EMPTY(&pa_to_pvh(VM_PAGE_TO_PHYS(m))->pv_list)); - rw_wunlock(&pvh_global_lock); - return (rv); -} - -/* - * Returns true if the pmap's pv is one of the first - * 16 pvs linked to from this page. This count may - * be changed upwards or downwards in the future; it - * is only necessary that true be returned for a small - * subset of pmaps for proper page aging. - */ -boolean_t -pmap_page_exists_quick(pmap_t pmap, vm_page_t m) -{ - struct md_page *pvh; - pv_entry_t pv; - int loops = 0; - boolean_t rv; - - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("%s: page %p is not managed", __func__, m)); - rv = FALSE; - rw_wlock(&pvh_global_lock); - TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { - if (PV_PMAP(pv) == pmap) { - rv = TRUE; - break; - } - loops++; - if (loops >= 16) - break; - } - if (!rv && loops < 16 && (m->flags & PG_FICTITIOUS) == 0) { - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { - if (PV_PMAP(pv) == pmap) { - rv = TRUE; - break; - } - loops++; - if (loops >= 16) - break; - } - } - rw_wunlock(&pvh_global_lock); - return (rv); -} - -/* - * pmap_zero_page zeros the specified hardware page by mapping - * the page into KVM and using bzero to clear its contents. - */ -void -pmap_zero_page(vm_page_t m) -{ - struct sysmaps *sysmaps; - - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (pte2_load(sysmaps->CMAP2) != 0) - panic("%s: CMAP2 busy", __func__); - pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, - m->md.pat_mode)); - pagezero(sysmaps->CADDR2); - pte2_clear(sysmaps->CMAP2); - tlb_flush((vm_offset_t)sysmaps->CADDR2); - sched_unpin(); - mtx_unlock(&sysmaps->lock); -} - -/* - * pmap_zero_page_area zeros the specified hardware page by mapping - * the page into KVM and using bzero to clear its contents. - * - * off and size may not cover an area beyond a single hardware page. - */ -void -pmap_zero_page_area(vm_page_t m, int off, int size) -{ - struct sysmaps *sysmaps; - - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (pte2_load(sysmaps->CMAP2) != 0) - panic("%s: CMAP2 busy", __func__); - pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, - m->md.pat_mode)); - if (off == 0 && size == PAGE_SIZE) - pagezero(sysmaps->CADDR2); - else - bzero(sysmaps->CADDR2 + off, size); - pte2_clear(sysmaps->CMAP2); - tlb_flush((vm_offset_t)sysmaps->CADDR2); - sched_unpin(); - mtx_unlock(&sysmaps->lock); -} - -/* - * pmap_zero_page_idle zeros the specified hardware page by mapping - * the page into KVM and using bzero to clear its contents. This - * is intended to be called from the vm_pagezero process only and - * outside of Giant. - */ -void -pmap_zero_page_idle(vm_page_t m) -{ - - if (pte2_load(CMAP3) != 0) - panic("%s: CMAP3 busy", __func__); - sched_pin(); - pte2_store(CMAP3, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, - m->md.pat_mode)); - pagezero(CADDR3); - pte2_clear(CMAP3); - tlb_flush((vm_offset_t)CADDR3); - sched_unpin(); -} - -/* - * pmap_copy_page copies the specified (machine independent) - * page by mapping the page into virtual memory and using - * bcopy to copy the page, one machine dependent page at a - * time. - */ -void -pmap_copy_page(vm_page_t src, vm_page_t dst) -{ - struct sysmaps *sysmaps; - - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (pte2_load(sysmaps->CMAP1) != 0) - panic("%s: CMAP1 busy", __func__); - if (pte2_load(sysmaps->CMAP2) != 0) - panic("%s: CMAP2 busy", __func__); - pte2_store(sysmaps->CMAP1, PTE2_KERN_NG(VM_PAGE_TO_PHYS(src), - PTE2_AP_KR | PTE2_NM, src->md.pat_mode)); - pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(dst), - PTE2_AP_KRW, dst->md.pat_mode)); - bcopy(sysmaps->CADDR1, sysmaps->CADDR2, PAGE_SIZE); - pte2_clear(sysmaps->CMAP1); - tlb_flush((vm_offset_t)sysmaps->CADDR1); - pte2_clear(sysmaps->CMAP2); - tlb_flush((vm_offset_t)sysmaps->CADDR2); - sched_unpin(); - mtx_unlock(&sysmaps->lock); -} - -int unmapped_buf_allowed = 1; - -void -pmap_copy_pages(vm_page_t ma[], vm_offset_t a_offset, vm_page_t mb[], - vm_offset_t b_offset, int xfersize) -{ - struct sysmaps *sysmaps; - vm_page_t a_pg, b_pg; - char *a_cp, *b_cp; - vm_offset_t a_pg_offset, b_pg_offset; - int cnt; - - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (*sysmaps->CMAP1 != 0) - panic("pmap_copy_pages: CMAP1 busy"); - if (*sysmaps->CMAP2 != 0) - panic("pmap_copy_pages: CMAP2 busy"); - while (xfersize > 0) { - a_pg = ma[a_offset >> PAGE_SHIFT]; - a_pg_offset = a_offset & PAGE_MASK; - cnt = min(xfersize, PAGE_SIZE - a_pg_offset); - b_pg = mb[b_offset >> PAGE_SHIFT]; - b_pg_offset = b_offset & PAGE_MASK; - cnt = min(cnt, PAGE_SIZE - b_pg_offset); - pte2_store(sysmaps->CMAP1, PTE2_KERN_NG(VM_PAGE_TO_PHYS(a_pg), - PTE2_AP_KR | PTE2_NM, a_pg->md.pat_mode)); - tlb_flush_local((vm_offset_t)sysmaps->CADDR1); - pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(b_pg), - PTE2_AP_KRW, b_pg->md.pat_mode)); - tlb_flush_local((vm_offset_t)sysmaps->CADDR2); - a_cp = sysmaps->CADDR1 + a_pg_offset; - b_cp = sysmaps->CADDR2 + b_pg_offset; - bcopy(a_cp, b_cp, cnt); - a_offset += cnt; - b_offset += cnt; - xfersize -= cnt; - } - pte2_clear(sysmaps->CMAP1); - tlb_flush((vm_offset_t)sysmaps->CADDR1); - pte2_clear(sysmaps->CMAP2); - tlb_flush((vm_offset_t)sysmaps->CADDR2); - sched_unpin(); - mtx_unlock(&sysmaps->lock); -} - -vm_offset_t -pmap_quick_enter_page(vm_page_t m) -{ - pt2_entry_t *pte2p; - vm_offset_t qmap_addr; - - critical_enter(); - qmap_addr = PCPU_GET(qmap_addr); - pte2p = pt2map_entry(qmap_addr); - - KASSERT(pte2_load(pte2p) == 0, ("%s: PTE2 busy", __func__)); - - pte2_store(pte2p, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, - pmap_page_get_memattr(m))); - return (qmap_addr); -} - -void -pmap_quick_remove_page(vm_offset_t addr) -{ - pt2_entry_t *pte2p; - vm_offset_t qmap_addr; - - qmap_addr = PCPU_GET(qmap_addr); - pte2p = pt2map_entry(qmap_addr); - - KASSERT(addr == qmap_addr, ("%s: invalid address", __func__)); - KASSERT(pte2_load(pte2p) != 0, ("%s: PTE2 not in use", __func__)); - - pte2_clear(pte2p); - tlb_flush(qmap_addr); - critical_exit(); -} - -/* - * Copy the range specified by src_addr/len - * from the source map to the range dst_addr/len - * in the destination map. - * - * This routine is only advisory and need not do anything. - */ -void -pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t len, - vm_offset_t src_addr) -{ - struct spglist free; - vm_offset_t addr; - vm_offset_t end_addr = src_addr + len; - vm_offset_t nextva; - - if (dst_addr != src_addr) - return; - - if (!pmap_is_current(src_pmap)) - return; - - rw_wlock(&pvh_global_lock); - if (dst_pmap < src_pmap) { - PMAP_LOCK(dst_pmap); - PMAP_LOCK(src_pmap); - } else { - PMAP_LOCK(src_pmap); - PMAP_LOCK(dst_pmap); - } - sched_pin(); - for (addr = src_addr; addr < end_addr; addr = nextva) { - pt2_entry_t *src_pte2p, *dst_pte2p; - vm_page_t dst_mpt2pg, src_mpt2pg; - pt1_entry_t src_pte1; - u_int pte1_idx; - - KASSERT(addr < VM_MAXUSER_ADDRESS, - ("%s: invalid to pmap_copy page tables", __func__)); - - nextva = pte1_trunc(addr + PTE1_SIZE); - if (nextva < addr) - nextva = end_addr; - - pte1_idx = pte1_index(addr); - src_pte1 = src_pmap->pm_pt1[pte1_idx]; - if (pte1_is_section(src_pte1)) { - if ((addr & PTE1_OFFSET) != 0 || - (addr + PTE1_SIZE) > end_addr) - continue; - if (dst_pmap->pm_pt1[pte1_idx] == 0 && - (!pte1_is_managed(src_pte1) || - pmap_pv_insert_pte1(dst_pmap, addr, - pte1_pa(src_pte1)))) { - dst_pmap->pm_pt1[pte1_idx] = src_pte1 & - ~PTE1_W; - dst_pmap->pm_stats.resident_count += - PTE1_SIZE / PAGE_SIZE; - pmap_pte1_mappings++; - } - continue; - } else if (!pte1_is_link(src_pte1)) - continue; - - src_mpt2pg = PHYS_TO_VM_PAGE(pte1_link_pa(src_pte1)); - - /* - * We leave PT2s to be linked from PT1 even if they are not - * referenced until all PT2s in a page are without reference. - * - * QQQ: It could be changed ... - */ -#if 0 /* single_pt2_link_is_cleared */ - KASSERT(pt2_wirecount_get(src_mpt2pg, pte1_idx) > 0, - ("%s: source page table page is unused", __func__)); -#else - if (pt2_wirecount_get(src_mpt2pg, pte1_idx) == 0) - continue; -#endif - if (nextva > end_addr) - nextva = end_addr; - - src_pte2p = pt2map_entry(addr); - while (addr < nextva) { - pt2_entry_t temp_pte2; - temp_pte2 = pte2_load(src_pte2p); - /* - * we only virtual copy managed pages - */ - if (pte2_is_managed(temp_pte2)) { - dst_mpt2pg = pmap_allocpte2(dst_pmap, addr, - PMAP_ENTER_NOSLEEP); - if (dst_mpt2pg == NULL) - goto out; - dst_pte2p = pmap_pte2_quick(dst_pmap, addr); - if (!pte2_is_valid(pte2_load(dst_pte2p)) && - pmap_try_insert_pv_entry(dst_pmap, addr, - PHYS_TO_VM_PAGE(pte2_pa(temp_pte2)))) { - /* - * Clear the wired, modified, and - * accessed (referenced) bits - * during the copy. - */ - temp_pte2 &= ~(PTE2_W | PTE2_A); - temp_pte2 |= PTE2_NM; - pte2_store(dst_pte2p, temp_pte2); - dst_pmap->pm_stats.resident_count++; - } else { - SLIST_INIT(&free); - if (pmap_unwire_pt2(dst_pmap, addr, - dst_mpt2pg, &free)) { - pmap_tlb_flush(dst_pmap, addr); - pmap_free_zero_pages(&free); - } - goto out; - } - if (pt2_wirecount_get(dst_mpt2pg, pte1_idx) >= - pt2_wirecount_get(src_mpt2pg, pte1_idx)) - break; - } - addr += PAGE_SIZE; - src_pte2p++; - } - } -out: - sched_unpin(); - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(src_pmap); - PMAP_UNLOCK(dst_pmap); -} - -/* - * Increase the starting virtual address of the given mapping if a - * different alignment might result in more section mappings. - */ -void -pmap_align_superpage(vm_object_t object, vm_ooffset_t offset, - vm_offset_t *addr, vm_size_t size) -{ - vm_offset_t pte1_offset; - - if (size < PTE1_SIZE) - return; - if (object != NULL && (object->flags & OBJ_COLORED) != 0) - offset += ptoa(object->pg_color); - pte1_offset = offset & PTE1_OFFSET; - if (size - ((PTE1_SIZE - pte1_offset) & PTE1_OFFSET) < PTE1_SIZE || - (*addr & PTE1_OFFSET) == pte1_offset) - return; - if ((*addr & PTE1_OFFSET) < pte1_offset) - *addr = pte1_trunc(*addr) + pte1_offset; - else - *addr = pte1_roundup(*addr) + pte1_offset; -} - -void -pmap_activate(struct thread *td) -{ - pmap_t pmap, oldpmap; - u_int cpuid, ttb; - - PDEBUG(9, printf("%s: td = %08x\n", __func__, (uint32_t)td)); - - critical_enter(); - pmap = vmspace_pmap(td->td_proc->p_vmspace); - oldpmap = PCPU_GET(curpmap); - cpuid = PCPU_GET(cpuid); - -#if defined(SMP) - CPU_CLR_ATOMIC(cpuid, &oldpmap->pm_active); - CPU_SET_ATOMIC(cpuid, &pmap->pm_active); -#else - CPU_CLR(cpuid, &oldpmap->pm_active); - CPU_SET(cpuid, &pmap->pm_active); -#endif - - ttb = pmap_ttb_get(pmap); - - /* - * pmap_activate is for the current thread on the current cpu - */ - td->td_pcb->pcb_pagedir = ttb; - cp15_ttbr_set(ttb); - PCPU_SET(curpmap, pmap); - critical_exit(); -} - -/* - * Perform the pmap work for mincore. - */ -int -pmap_mincore(pmap_t pmap, vm_offset_t addr, vm_paddr_t *locked_pa) -{ - pt1_entry_t *pte1p, pte1; - pt2_entry_t *pte2p, pte2; - vm_paddr_t pa; - boolean_t managed; - int val; - - PMAP_LOCK(pmap); -retry: - pte1p = pmap_pte1(pmap, addr); - pte1 = pte1_load(pte1p); - if (pte1_is_section(pte1)) { - pa = trunc_page(pte1_pa(pte1) | (addr & PTE1_OFFSET)); - managed = pte1_is_managed(pte1); - val = MINCORE_SUPER | MINCORE_INCORE; - if (pte1_is_dirty(pte1)) - val |= MINCORE_MODIFIED | MINCORE_MODIFIED_OTHER; - if (pte1 & PTE1_A) - val |= MINCORE_REFERENCED | MINCORE_REFERENCED_OTHER; - } else if (pte1_is_link(pte1)) { - pte2p = pmap_pte2(pmap, addr); - pte2 = pte2_load(pte2p); - pmap_pte2_release(pte2p); - pa = pte2_pa(pte2); - managed = pte2_is_managed(pte2); - val = MINCORE_INCORE; - if (pte2_is_dirty(pte2)) - val |= MINCORE_MODIFIED | MINCORE_MODIFIED_OTHER; - if (pte2 & PTE2_A) - val |= MINCORE_REFERENCED | MINCORE_REFERENCED_OTHER; - } else { - managed = FALSE; - val = 0; - } - if ((val & (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER)) != - (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER) && managed) { - /* Ensure that "PHYS_TO_VM_PAGE(pa)->object" doesn't change. */ - if (vm_page_pa_tryrelock(pmap, pa, locked_pa)) - goto retry; - } else - PA_UNLOCK_COND(*locked_pa); - PMAP_UNLOCK(pmap); - return (val); -} - -void -pmap_kenter_device(vm_offset_t va, vm_size_t size, vm_paddr_t pa) -{ - vm_offset_t sva; - - KASSERT((size & PAGE_MASK) == 0, - ("%s: device mapping not page-sized", __func__)); - - sva = va; - while (size != 0) { - pmap_kenter_prot_attr(va, pa, PTE2_AP_KRW, PTE2_ATTR_DEVICE); - va += PAGE_SIZE; - pa += PAGE_SIZE; - size -= PAGE_SIZE; - } - tlb_flush_range(sva, va - sva); -} - -void -pmap_kremove_device(vm_offset_t va, vm_size_t size) -{ - vm_offset_t sva; - - KASSERT((size & PAGE_MASK) == 0, - ("%s: device mapping not page-sized", __func__)); - - sva = va; - while (size != 0) { - pmap_kremove(va); - va += PAGE_SIZE; - size -= PAGE_SIZE; - } - tlb_flush_range(sva, va - sva); -} - -void -pmap_set_pcb_pagedir(pmap_t pmap, struct pcb *pcb) -{ - - pcb->pcb_pagedir = pmap_ttb_get(pmap); -} - - -/* - * Clean L1 data cache range by physical address. - * The range must be within a single page. - */ -static void -pmap_dcache_wb_pou(vm_paddr_t pa, vm_size_t size, vm_memattr_t ma) -{ - struct sysmaps *sysmaps; - - KASSERT(((pa & PAGE_MASK) + size) <= PAGE_SIZE, - ("%s: not on single page", __func__)); - - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (*sysmaps->CMAP3) - panic("%s: CMAP3 busy", __func__); - pte2_store(sysmaps->CMAP3, PTE2_KERN_NG(pa, PTE2_AP_KRW, ma)); - dcache_wb_pou((vm_offset_t)sysmaps->CADDR3 + (pa & PAGE_MASK), size); - pte2_clear(sysmaps->CMAP3); - tlb_flush((vm_offset_t)sysmaps->CADDR3); - sched_unpin(); - mtx_unlock(&sysmaps->lock); -} - -/* - * Sync instruction cache range which is not mapped yet. - */ -void -cache_icache_sync_fresh(vm_offset_t va, vm_paddr_t pa, vm_size_t size) -{ - uint32_t len, offset; - vm_page_t m; - - /* Write back d-cache on given address range. */ - offset = pa & PAGE_MASK; - for ( ; size != 0; size -= len, pa += len, offset = 0) { - len = min(PAGE_SIZE - offset, size); - m = PHYS_TO_VM_PAGE(pa); - KASSERT(m != NULL, ("%s: vm_page_t is null for %#x", - __func__, pa)); - pmap_dcache_wb_pou(pa, len, m->md.pat_mode); - } - /* - * I-cache is VIPT. Only way how to flush all virtual mappings - * on given physical address is to invalidate all i-cache. - */ - icache_inv_all(); -} - -void -pmap_sync_icache(pmap_t pmap, vm_offset_t va, vm_size_t size) -{ - - /* Write back d-cache on given address range. */ - if (va >= VM_MIN_KERNEL_ADDRESS) { - dcache_wb_pou(va, size); - } else { - uint32_t len, offset; - vm_paddr_t pa; - vm_page_t m; - - offset = va & PAGE_MASK; - for ( ; size != 0; size -= len, va += len, offset = 0) { - pa = pmap_extract(pmap, va); /* offset is preserved */ - len = min(PAGE_SIZE - offset, size); - m = PHYS_TO_VM_PAGE(pa); - KASSERT(m != NULL, ("%s: vm_page_t is null for %#x", - __func__, pa)); - pmap_dcache_wb_pou(pa, len, m->md.pat_mode); - } - } - /* - * I-cache is VIPT. Only way how to flush all virtual mappings - * on given physical address is to invalidate all i-cache. - */ - icache_inv_all(); -} - -/* - * The implementation of pmap_fault() uses IN_RANGE2() macro which - * depends on the fact that given range size is a power of 2. - */ -CTASSERT(powerof2(NB_IN_PT1)); -CTASSERT(powerof2(PT2MAP_SIZE)); - -#define IN_RANGE2(addr, start, size) \ - ((vm_offset_t)(start) == ((vm_offset_t)(addr) & ~((size) - 1))) - -/* - * Handle access and R/W emulation faults. - */ -int -pmap_fault(pmap_t pmap, vm_offset_t far, uint32_t fsr, int idx, bool usermode) -{ - pt1_entry_t *pte1p, pte1; - pt2_entry_t *pte2p, pte2; - - if (pmap == NULL) - pmap = kernel_pmap; - - /* - * In kernel, we should never get abort with FAR which is in range of - * pmap->pm_pt1 or PT2MAP address spaces. If it happens, stop here - * and print out a useful abort message and even get to the debugger - * otherwise it likely ends with never ending loop of aborts. - */ - if (__predict_false(IN_RANGE2(far, pmap->pm_pt1, NB_IN_PT1))) { - /* - * All L1 tables should always be mapped and present. - * However, we check only current one herein. For user mode, - * only permission abort from malicious user is not fatal. - * And alignment abort as it may have higher priority. - */ - if (!usermode || (idx != FAULT_ALIGN && idx != FAULT_PERM_L2)) { - CTR4(KTR_PMAP, "%s: pmap %#x pm_pt1 %#x far %#x", - __func__, pmap, pmap->pm_pt1, far); - panic("%s: pm_pt1 abort", __func__); - } - return (KERN_INVALID_ADDRESS); - } - if (__predict_false(IN_RANGE2(far, PT2MAP, PT2MAP_SIZE))) { - /* - * PT2MAP should be always mapped and present in current - * L1 table. However, only existing L2 tables are mapped - * in PT2MAP. For user mode, only L2 translation abort and - * permission abort from malicious user is not fatal. - * And alignment abort as it may have higher priority. - */ - if (!usermode || (idx != FAULT_ALIGN && - idx != FAULT_TRAN_L2 && idx != FAULT_PERM_L2)) { - CTR4(KTR_PMAP, "%s: pmap %#x PT2MAP %#x far %#x", - __func__, pmap, PT2MAP, far); - panic("%s: PT2MAP abort", __func__); - } - return (KERN_INVALID_ADDRESS); - } - - /* - * Accesss bits for page and section. Note that the entry - * is not in TLB yet, so TLB flush is not necessary. - * - * QQQ: This is hardware emulation, we do not call userret() - * for aborts from user mode. - * We do not lock PMAP, so cmpset() is a need. Hopefully, - * no one removes the mapping when we are here. - */ - if (idx == FAULT_ACCESS_L2) { - pte2p = pt2map_entry(far); -pte2_seta: - pte2 = pte2_load(pte2p); - if (pte2_is_valid(pte2)) { - if (!pte2_cmpset(pte2p, pte2, pte2 | PTE2_A)) { - goto pte2_seta; - } - return (KERN_SUCCESS); - } - } - if (idx == FAULT_ACCESS_L1) { - pte1p = pmap_pte1(pmap, far); -pte1_seta: - pte1 = pte1_load(pte1p); - if (pte1_is_section(pte1)) { - if (!pte1_cmpset(pte1p, pte1, pte1 | PTE1_A)) { - goto pte1_seta; - } - return (KERN_SUCCESS); - } - } - - /* - * Handle modify bits for page and section. Note that the modify - * bit is emulated by software. So PTEx_RO is software read only - * bit and PTEx_NM flag is real hardware read only bit. - * - * QQQ: This is hardware emulation, we do not call userret() - * for aborts from user mode. - * We do not lock PMAP, so cmpset() is a need. Hopefully, - * no one removes the mapping when we are here. - */ - if ((fsr & FSR_WNR) && (idx == FAULT_PERM_L2)) { - pte2p = pt2map_entry(far); -pte2_setrw: - pte2 = pte2_load(pte2p); - if (pte2_is_valid(pte2) && !(pte2 & PTE2_RO) && - (pte2 & PTE2_NM)) { - if (!pte2_cmpset(pte2p, pte2, pte2 & ~PTE2_NM)) { - goto pte2_setrw; - } - tlb_flush(trunc_page(far)); - return (KERN_SUCCESS); - } - } - if ((fsr & FSR_WNR) && (idx == FAULT_PERM_L1)) { - pte1p = pmap_pte1(pmap, far); -pte1_setrw: - pte1 = pte1_load(pte1p); - if (pte1_is_section(pte1) && !(pte1 & PTE1_RO) && - (pte1 & PTE1_NM)) { - if (!pte1_cmpset(pte1p, pte1, pte1 & ~PTE1_NM)) { - goto pte1_setrw; - } - tlb_flush(pte1_trunc(far)); - return (KERN_SUCCESS); - } - } - - /* - * QQQ: The previous code, mainly fast handling of access and - * modify bits aborts, could be moved to ASM. Now we are - * starting to deal with not fast aborts. - */ - -#ifdef INVARIANTS - /* - * Read an entry in PT2TAB associated with both pmap and far. - * It's safe because PT2TAB is always mapped. - * - * QQQ: We do not lock PMAP, so false positives could happen if - * the mapping is removed concurrently. - */ - pte2 = pt2tab_load(pmap_pt2tab_entry(pmap, far)); - if (pte2_is_valid(pte2)) { - /* - * Now, when we know that L2 page table is allocated, - * we can use PT2MAP to get L2 page table entry. - */ - pte2 = pte2_load(pt2map_entry(far)); - if (pte2_is_valid(pte2)) { - /* - * If L2 page table entry is valid, make sure that - * L1 page table entry is valid too. Note that we - * leave L2 page entries untouched when promoted. - */ - pte1 = pte1_load(pmap_pte1(pmap, far)); - if (!pte1_is_valid(pte1)) { - panic("%s: missing L1 page entry (%p, %#x)", - __func__, pmap, far); - } - } - } -#endif - return (KERN_FAILURE); -} - -/* !!!! REMOVE !!!! */ -void -pmap_pte_init_mmu_v6(void) -{ -} - -void vector_page_setprot(int p) -{ -} - -#if defined(PMAP_DEBUG) -/* - * Reusing of KVA used in pmap_zero_page function !!! - */ -static void -pmap_zero_page_check(vm_page_t m) -{ - uint32_t *p, *end; - struct sysmaps *sysmaps; - - sched_pin(); - sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; - mtx_lock(&sysmaps->lock); - if (pte2_load(sysmaps->CMAP2) != 0) - panic("%s: CMAP2 busy", __func__); - pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, - m->md.pat_mode)); - end = (uint32_t*)(sysmaps->CADDR2 + PAGE_SIZE); - for (p = (uint32_t*)sysmaps->CADDR2; p < end; p++) - if (*p != 0) - panic("%s: page %p not zero, va: %p", __func__, m, - sysmaps->CADDR2); - pte2_clear(sysmaps->CMAP2); - tlb_flush((vm_offset_t)sysmaps->CADDR2); - sched_unpin(); - mtx_unlock(&sysmaps->lock); -} - -int -pmap_pid_dump(int pid) -{ - pmap_t pmap; - struct proc *p; - int npte2 = 0; - int i, j, index; - - sx_slock(&allproc_lock); - FOREACH_PROC_IN_SYSTEM(p) { - if (p->p_pid != pid || p->p_vmspace == NULL) - continue; - index = 0; - pmap = vmspace_pmap(p->p_vmspace); - for (i = 0; i < NPTE1_IN_PT1; i++) { - pt1_entry_t pte1; - pt2_entry_t *pte2p, pte2; - vm_offset_t base, va; - vm_paddr_t pa; - vm_page_t m; - - base = i << PTE1_SHIFT; - pte1 = pte1_load(&pmap->pm_pt1[i]); - - if (pte1_is_section(pte1)) { - /* - * QQQ: Do something here! - */ - } else if (pte1_is_link(pte1)) { - for (j = 0; j < NPTE2_IN_PT2; j++) { - va = base + (j << PAGE_SHIFT); - if (va >= VM_MIN_KERNEL_ADDRESS) { - if (index) { - index = 0; - printf("\n"); - } - sx_sunlock(&allproc_lock); - return (npte2); - } - pte2p = pmap_pte2(pmap, va); - pte2 = pte2_load(pte2p); - pmap_pte2_release(pte2p); - if (!pte2_is_valid(pte2)) - continue; - - pa = pte2_pa(pte2); - m = PHYS_TO_VM_PAGE(pa); - printf("va: 0x%x, pa: 0x%x, h: %d, w:" - " %d, f: 0x%x", va, pa, - m->hold_count, m->wire_count, - m->flags); - npte2++; - index++; - if (index >= 2) { - index = 0; - printf("\n"); - } else { - printf(" "); - } - } - } - } - } - sx_sunlock(&allproc_lock); - return (npte2); -} - -#endif - -#ifdef DDB -static pt2_entry_t * -pmap_pte2_ddb(pmap_t pmap, vm_offset_t va) -{ - pt1_entry_t pte1; - vm_paddr_t pt2pg_pa; - - pte1 = pte1_load(pmap_pte1(pmap, va)); - if (!pte1_is_link(pte1)) - return (NULL); - - if (pmap_is_current(pmap)) - return (pt2map_entry(va)); - - /* Note that L2 page table size is not equal to PAGE_SIZE. */ - pt2pg_pa = trunc_page(pte1_link_pa(pte1)); - if (pte2_pa(pte2_load(PMAP3)) != pt2pg_pa) { - pte2_store(PMAP3, PTE2_KPT(pt2pg_pa)); -#ifdef SMP - PMAP3cpu = PCPU_GET(cpuid); -#endif - tlb_flush_local((vm_offset_t)PADDR3); - } -#ifdef SMP - else if (PMAP3cpu != PCPU_GET(cpuid)) { - PMAP3cpu = PCPU_GET(cpuid); - tlb_flush_local((vm_offset_t)PADDR3); - } -#endif - return (PADDR3 + (arm32_btop(va) & (NPTE2_IN_PG - 1))); -} - -static void -dump_pmap(pmap_t pmap) -{ - - printf("pmap %p\n", pmap); - printf(" pm_pt1: %p\n", pmap->pm_pt1); - printf(" pm_pt2tab: %p\n", pmap->pm_pt2tab); - printf(" pm_active: 0x%08lX\n", pmap->pm_active.__bits[0]); -} - -DB_SHOW_COMMAND(pmaps, pmap_list_pmaps) -{ - - pmap_t pmap; - LIST_FOREACH(pmap, &allpmaps, pm_list) { - dump_pmap(pmap); - } -} - -static int -pte2_class(pt2_entry_t pte2) -{ - int cls; - - cls = (pte2 >> 2) & 0x03; - cls |= (pte2 >> 4) & 0x04; - return (cls); -} - -static void -dump_section(pmap_t pmap, uint32_t pte1_idx) -{ -} - -static void -dump_link(pmap_t pmap, uint32_t pte1_idx, boolean_t invalid_ok) -{ - uint32_t i; - vm_offset_t va; - pt2_entry_t *pte2p, pte2; - vm_page_t m; - - va = pte1_idx << PTE1_SHIFT; - pte2p = pmap_pte2_ddb(pmap, va); - for (i = 0; i < NPTE2_IN_PT2; i++, pte2p++, va += PAGE_SIZE) { - pte2 = pte2_load(pte2p); - if (pte2 == 0) - continue; - if (!pte2_is_valid(pte2)) { - printf(" 0x%08X: 0x%08X", va, pte2); - if (!invalid_ok) - printf(" - not valid !!!"); - printf("\n"); - continue; - } - m = PHYS_TO_VM_PAGE(pte2_pa(pte2)); - printf(" 0x%08X: 0x%08X, TEX%d, s:%d, g:%d, m:%p", va , pte2, - pte2_class(pte2), !!(pte2 & PTE2_S), !(pte2 & PTE2_NG), m); - if (m != NULL) { - printf(" v:%d h:%d w:%d f:0x%04X\n", m->valid, - m->hold_count, m->wire_count, m->flags); - } else { - printf("\n"); - } - } -} - -static __inline boolean_t -is_pv_chunk_space(vm_offset_t va) -{ - - if ((((vm_offset_t)pv_chunkbase) <= va) && - (va < ((vm_offset_t)pv_chunkbase + PAGE_SIZE * pv_maxchunks))) - return (TRUE); - return (FALSE); -} - -DB_SHOW_COMMAND(pmap, pmap_pmap_print) -{ - /* XXX convert args. */ - pmap_t pmap = (pmap_t)addr; - pt1_entry_t pte1; - pt2_entry_t pte2; - vm_offset_t va, eva; - vm_page_t m; - uint32_t i; - boolean_t invalid_ok, dump_link_ok, dump_pv_chunk; - - if (have_addr) { - pmap_t pm; - - LIST_FOREACH(pm, &allpmaps, pm_list) - if (pm == pmap) break; - if (pm == NULL) { - printf("given pmap %p is not in allpmaps list\n", pmap); - return; - } - } else - pmap = PCPU_GET(curpmap); - - eva = (modif[0] == 'u') ? VM_MAXUSER_ADDRESS : 0xFFFFFFFF; - dump_pv_chunk = FALSE; /* XXX evaluate from modif[] */ - - printf("pmap: 0x%08X\n", (uint32_t)pmap); - printf("PT2MAP: 0x%08X\n", (uint32_t)PT2MAP); - printf("pt2tab: 0x%08X\n", (uint32_t)pmap->pm_pt2tab); - - for(i = 0; i < NPTE1_IN_PT1; i++) { - pte1 = pte1_load(&pmap->pm_pt1[i]); - if (pte1 == 0) - continue; - va = i << PTE1_SHIFT; - if (va >= eva) - break; - - if (pte1_is_section(pte1)) { - printf("0x%08X: Section 0x%08X, s:%d g:%d\n", va, pte1, - !!(pte1 & PTE1_S), !(pte1 & PTE1_NG)); - dump_section(pmap, i); - } else if (pte1_is_link(pte1)) { - dump_link_ok = TRUE; - invalid_ok = FALSE; - pte2 = pte2_load(pmap_pt2tab_entry(pmap, va)); - m = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); - printf("0x%08X: Link 0x%08X, pt2tab: 0x%08X m: %p", - va, pte1, pte2, m); - if (is_pv_chunk_space(va)) { - printf(" - pv_chunk space"); - if (dump_pv_chunk) - invalid_ok = TRUE; - else - dump_link_ok = FALSE; - } - else if (m != NULL) - printf(" w:%d w2:%u", m->wire_count, - pt2_wirecount_get(m, pte1_index(va))); - if (pte2 == 0) - printf(" !!! pt2tab entry is ZERO"); - else if (pte2_pa(pte1) != pte2_pa(pte2)) - printf(" !!! pt2tab entry is DIFFERENT - m: %p", - PHYS_TO_VM_PAGE(pte2_pa(pte2))); - printf("\n"); - if (dump_link_ok) - dump_link(pmap, i, invalid_ok); - } else - printf("0x%08X: Invalid entry 0x%08X\n", va, pte1); - } -} - -static void -dump_pt2tab(pmap_t pmap) -{ - uint32_t i; - pt2_entry_t pte2; - vm_offset_t va; - vm_paddr_t pa; - vm_page_t m; - - printf("PT2TAB:\n"); - for (i = 0; i < PT2TAB_ENTRIES; i++) { - pte2 = pte2_load(&pmap->pm_pt2tab[i]); - if (!pte2_is_valid(pte2)) - continue; - va = i << PT2TAB_SHIFT; - pa = pte2_pa(pte2); - m = PHYS_TO_VM_PAGE(pa); - printf(" 0x%08X: 0x%08X, TEX%d, s:%d, m:%p", va, pte2, - pte2_class(pte2), !!(pte2 & PTE2_S), m); - if (m != NULL) - printf(" , h: %d, w: %d, f: 0x%04X pidx: %lld", - m->hold_count, m->wire_count, m->flags, m->pindex); - printf("\n"); - } -} - -DB_SHOW_COMMAND(pmap_pt2tab, pmap_pt2tab_print) -{ - /* XXX convert args. */ - pmap_t pmap = (pmap_t)addr; - pt1_entry_t pte1; - pt2_entry_t pte2; - vm_offset_t va; - uint32_t i, start; - - if (have_addr) { - printf("supported only on current pmap\n"); - return; - } - - pmap = PCPU_GET(curpmap); - printf("curpmap: 0x%08X\n", (uint32_t)pmap); - printf("PT2MAP: 0x%08X\n", (uint32_t)PT2MAP); - printf("pt2tab: 0x%08X\n", (uint32_t)pmap->pm_pt2tab); - - start = pte1_index((vm_offset_t)PT2MAP); - for (i = start; i < (start + NPT2_IN_PT2TAB); i++) { - pte1 = pte1_load(&pmap->pm_pt1[i]); - if (pte1 == 0) - continue; - va = i << PTE1_SHIFT; - if (pte1_is_section(pte1)) { - printf("0x%08X: Section 0x%08X, s:%d\n", va, pte1, - !!(pte1 & PTE1_S)); - dump_section(pmap, i); - } else if (pte1_is_link(pte1)) { - pte2 = pte2_load(pmap_pt2tab_entry(pmap, va)); - printf("0x%08X: Link 0x%08X, pt2tab: 0x%08X\n", va, - pte1, pte2); - if (pte2 == 0) - printf(" !!! pt2tab entry is ZERO\n"); - } else - printf("0x%08X: Invalid entry 0x%08X\n", va, pte1); - } - dump_pt2tab(pmap); -} -#endif Property changes on: user/ngie/make_check/sys/arm/arm/pmap-v6-new.c ___________________________________________________________________ Deleted: svn:eol-style ## -1 +0,0 ## -native \ No newline at end of property Deleted: svn:keywords ## -1 +0,0 ## -FreeBSD=%H \ No newline at end of property Deleted: svn:mime-type ## -1 +0,0 ## -text/plain \ No newline at end of property Index: user/ngie/make_check/sys/arm/arm/cpufunc.c =================================================================== --- user/ngie/make_check/sys/arm/arm/cpufunc.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/cpufunc.c (revision 295078) @@ -1,1415 +1,1409 @@ /* $NetBSD: cpufunc.c,v 1.65 2003/11/05 12:53:15 scw Exp $ */ /*- * arm9 support code Copyright (C) 2001 ARM Ltd * Copyright (c) 1997 Mark Brinicombe. * Copyright (c) 1997 Causality Limited * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Causality Limited. * 4. The name of Causality Limited may not be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY CAUSALITY LIMITED ``AS IS'' AND ANY EXPRESS * OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE * DISCLAIMED. IN NO EVENT SHALL CAUSALITY LIMITED BE LIABLE FOR ANY DIRECT, * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * RiscBSD kernel project * * cpufuncs.c * * C functions for supporting CPU / MMU / TLB specific operations. * * Created : 30/01/97 */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #if defined(CPU_XSCALE_80321) || defined(CPU_XSCALE_80219) #include #include #endif /* * Some definitions in i81342reg.h clash with i80321reg.h. * This only happens for the LINT kernel. As it happens, * we don't need anything from i81342reg.h that we already * got from somewhere else during a LINT compile. */ #if defined(CPU_XSCALE_81342) && !defined(COMPILING_LINT) #include #endif #ifdef CPU_XSCALE_IXP425 #include #include #endif /* PRIMARY CACHE VARIABLES */ int arm_picache_size; int arm_picache_line_size; int arm_picache_ways; int arm_pdcache_size; /* and unified */ int arm_pdcache_line_size; int arm_pdcache_ways; int arm_pcache_type; int arm_pcache_unified; int arm_dcache_align; int arm_dcache_align_mask; u_int arm_cache_level; u_int arm_cache_type[14]; u_int arm_cache_loc; int ctrl; #ifdef CPU_ARM9 struct cpu_functions arm9_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ cpufunc_nullop, /* cpwait */ /* MMU functions */ cpufunc_control, /* control */ cpufunc_domains, /* Domain */ arm9_setttb, /* Setttb */ cpufunc_faultstatus, /* Faultstatus */ cpufunc_faultaddress, /* Faultaddress */ /* TLB functions */ armv4_tlb_flushID, /* tlb_flushID */ arm9_tlb_flushID_SE, /* tlb_flushID_SE */ armv4_tlb_flushI, /* tlb_flushI */ (void *)armv4_tlb_flushI, /* tlb_flushI_SE */ armv4_tlb_flushD, /* tlb_flushD */ armv4_tlb_flushD_SE, /* tlb_flushD_SE */ /* Cache operations */ arm9_icache_sync_all, /* icache_sync_all */ arm9_icache_sync_range, /* icache_sync_range */ arm9_dcache_wbinv_all, /* dcache_wbinv_all */ arm9_dcache_wbinv_range, /* dcache_wbinv_range */ arm9_dcache_inv_range, /* dcache_inv_range */ arm9_dcache_wb_range, /* dcache_wb_range */ armv4_idcache_inv_all, /* idcache_inv_all */ arm9_idcache_wbinv_all, /* idcache_wbinv_all */ arm9_idcache_wbinv_range, /* idcache_wbinv_range */ cpufunc_nullop, /* l2cache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_inv_range */ (void *)cpufunc_nullop, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ cpufunc_nullop, /* flush_prefetchbuf */ armv4_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ (void *)cpufunc_nullop, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ arm9_context_switch, /* context_switch */ arm9_setup /* cpu setup */ }; #endif /* CPU_ARM9 */ #if defined(CPU_ARM9E) struct cpu_functions armv5_ec_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ cpufunc_nullop, /* cpwait */ /* MMU functions */ cpufunc_control, /* control */ cpufunc_domains, /* Domain */ armv5_ec_setttb, /* Setttb */ cpufunc_faultstatus, /* Faultstatus */ cpufunc_faultaddress, /* Faultaddress */ /* TLB functions */ armv4_tlb_flushID, /* tlb_flushID */ arm10_tlb_flushID_SE, /* tlb_flushID_SE */ armv4_tlb_flushI, /* tlb_flushI */ arm10_tlb_flushI_SE, /* tlb_flushI_SE */ armv4_tlb_flushD, /* tlb_flushD */ armv4_tlb_flushD_SE, /* tlb_flushD_SE */ /* Cache operations */ armv5_ec_icache_sync_all, /* icache_sync_all */ armv5_ec_icache_sync_range, /* icache_sync_range */ armv5_ec_dcache_wbinv_all, /* dcache_wbinv_all */ armv5_ec_dcache_wbinv_range, /* dcache_wbinv_range */ armv5_ec_dcache_inv_range, /* dcache_inv_range */ armv5_ec_dcache_wb_range, /* dcache_wb_range */ armv4_idcache_inv_all, /* idcache_inv_all */ armv5_ec_idcache_wbinv_all, /* idcache_wbinv_all */ armv5_ec_idcache_wbinv_range, /* idcache_wbinv_range */ cpufunc_nullop, /* l2cache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_inv_range */ (void *)cpufunc_nullop, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ cpufunc_nullop, /* flush_prefetchbuf */ armv4_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ (void *)cpufunc_nullop, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ arm10_context_switch, /* context_switch */ arm10_setup /* cpu setup */ }; struct cpu_functions sheeva_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ cpufunc_nullop, /* cpwait */ /* MMU functions */ cpufunc_control, /* control */ cpufunc_domains, /* Domain */ sheeva_setttb, /* Setttb */ cpufunc_faultstatus, /* Faultstatus */ cpufunc_faultaddress, /* Faultaddress */ /* TLB functions */ armv4_tlb_flushID, /* tlb_flushID */ arm10_tlb_flushID_SE, /* tlb_flushID_SE */ armv4_tlb_flushI, /* tlb_flushI */ arm10_tlb_flushI_SE, /* tlb_flushI_SE */ armv4_tlb_flushD, /* tlb_flushD */ armv4_tlb_flushD_SE, /* tlb_flushD_SE */ /* Cache operations */ armv5_ec_icache_sync_all, /* icache_sync_all */ armv5_ec_icache_sync_range, /* icache_sync_range */ armv5_ec_dcache_wbinv_all, /* dcache_wbinv_all */ sheeva_dcache_wbinv_range, /* dcache_wbinv_range */ sheeva_dcache_inv_range, /* dcache_inv_range */ sheeva_dcache_wb_range, /* dcache_wb_range */ armv4_idcache_inv_all, /* idcache_inv_all */ armv5_ec_idcache_wbinv_all, /* idcache_wbinv_all */ sheeva_idcache_wbinv_range, /* idcache_wbinv_all */ sheeva_l2cache_wbinv_all, /* l2cache_wbinv_all */ sheeva_l2cache_wbinv_range, /* l2cache_wbinv_range */ sheeva_l2cache_inv_range, /* l2cache_inv_range */ sheeva_l2cache_wb_range, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ cpufunc_nullop, /* flush_prefetchbuf */ armv4_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ sheeva_cpu_sleep, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ arm10_context_switch, /* context_switch */ arm10_setup /* cpu setup */ }; #endif /* CPU_ARM9E */ #ifdef CPU_MV_PJ4B struct cpu_functions pj4bv7_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ armv7_drain_writebuf, /* cpwait */ /* MMU functions */ cpufunc_control, /* control */ cpufunc_domains, /* Domain */ armv7_setttb, /* Setttb */ cpufunc_faultstatus, /* Faultstatus */ cpufunc_faultaddress, /* Faultaddress */ /* TLB functions */ armv7_tlb_flushID, /* tlb_flushID */ armv7_tlb_flushID_SE, /* tlb_flushID_SE */ armv7_tlb_flushID, /* tlb_flushI */ armv7_tlb_flushID_SE, /* tlb_flushI_SE */ armv7_tlb_flushID, /* tlb_flushD */ armv7_tlb_flushID_SE, /* tlb_flushD_SE */ /* Cache operations */ armv7_idcache_wbinv_all, /* icache_sync_all */ armv7_icache_sync_range, /* icache_sync_range */ armv7_dcache_wbinv_all, /* dcache_wbinv_all */ armv7_dcache_wbinv_range, /* dcache_wbinv_range */ armv7_dcache_inv_range, /* dcache_inv_range */ armv7_dcache_wb_range, /* dcache_wb_range */ armv7_idcache_inv_all, /* idcache_inv_all */ armv7_idcache_wbinv_all, /* idcache_wbinv_all */ armv7_idcache_wbinv_range, /* idcache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_inv_range */ (void *)cpufunc_nullop, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ cpufunc_nullop, /* flush_prefetchbuf */ armv7_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ (void *)cpufunc_nullop, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ armv7_context_switch, /* context_switch */ pj4bv7_setup /* cpu setup */ }; #endif /* CPU_MV_PJ4B */ #if defined(CPU_XSCALE_80321) || \ defined(CPU_XSCALE_PXA2X0) || defined(CPU_XSCALE_IXP425) || \ defined(CPU_XSCALE_80219) struct cpu_functions xscale_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ xscale_cpwait, /* cpwait */ /* MMU functions */ xscale_control, /* control */ cpufunc_domains, /* domain */ xscale_setttb, /* setttb */ cpufunc_faultstatus, /* faultstatus */ cpufunc_faultaddress, /* faultaddress */ /* TLB functions */ armv4_tlb_flushID, /* tlb_flushID */ xscale_tlb_flushID_SE, /* tlb_flushID_SE */ armv4_tlb_flushI, /* tlb_flushI */ (void *)armv4_tlb_flushI, /* tlb_flushI_SE */ armv4_tlb_flushD, /* tlb_flushD */ armv4_tlb_flushD_SE, /* tlb_flushD_SE */ /* Cache operations */ xscale_cache_syncI, /* icache_sync_all */ xscale_cache_syncI_rng, /* icache_sync_range */ xscale_cache_purgeD, /* dcache_wbinv_all */ xscale_cache_purgeD_rng, /* dcache_wbinv_range */ xscale_cache_flushD_rng, /* dcache_inv_range */ xscale_cache_cleanD_rng, /* dcache_wb_range */ xscale_cache_flushID, /* idcache_inv_all */ xscale_cache_purgeID, /* idcache_wbinv_all */ xscale_cache_purgeID_rng, /* idcache_wbinv_range */ cpufunc_nullop, /* l2cache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_inv_range */ (void *)cpufunc_nullop, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ cpufunc_nullop, /* flush_prefetchbuf */ armv4_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ xscale_cpu_sleep, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ xscale_context_switch, /* context_switch */ xscale_setup /* cpu setup */ }; #endif /* CPU_XSCALE_80321 || CPU_XSCALE_PXA2X0 || CPU_XSCALE_IXP425 CPU_XSCALE_80219 */ #ifdef CPU_XSCALE_81342 struct cpu_functions xscalec3_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ xscale_cpwait, /* cpwait */ /* MMU functions */ xscale_control, /* control */ cpufunc_domains, /* domain */ xscalec3_setttb, /* setttb */ cpufunc_faultstatus, /* faultstatus */ cpufunc_faultaddress, /* faultaddress */ /* TLB functions */ armv4_tlb_flushID, /* tlb_flushID */ xscale_tlb_flushID_SE, /* tlb_flushID_SE */ armv4_tlb_flushI, /* tlb_flushI */ (void *)armv4_tlb_flushI, /* tlb_flushI_SE */ armv4_tlb_flushD, /* tlb_flushD */ armv4_tlb_flushD_SE, /* tlb_flushD_SE */ /* Cache operations */ xscalec3_cache_syncI, /* icache_sync_all */ xscalec3_cache_syncI_rng, /* icache_sync_range */ xscalec3_cache_purgeD, /* dcache_wbinv_all */ xscalec3_cache_purgeD_rng, /* dcache_wbinv_range */ xscale_cache_flushD_rng, /* dcache_inv_range */ xscalec3_cache_cleanD_rng, /* dcache_wb_range */ xscale_cache_flushID, /* idcache_inv_all */ xscalec3_cache_purgeID, /* idcache_wbinv_all */ xscalec3_cache_purgeID_rng, /* idcache_wbinv_range */ xscalec3_l2cache_purge, /* l2cache_wbinv_all */ xscalec3_l2cache_purge_rng, /* l2cache_wbinv_range */ xscalec3_l2cache_flush_rng, /* l2cache_inv_range */ xscalec3_l2cache_clean_rng, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ cpufunc_nullop, /* flush_prefetchbuf */ armv4_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ xscale_cpu_sleep, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ xscalec3_context_switch, /* context_switch */ xscale_setup /* cpu setup */ }; #endif /* CPU_XSCALE_81342 */ #if defined(CPU_FA526) struct cpu_functions fa526_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ cpufunc_nullop, /* cpwait */ /* MMU functions */ cpufunc_control, /* control */ cpufunc_domains, /* domain */ fa526_setttb, /* setttb */ cpufunc_faultstatus, /* faultstatus */ cpufunc_faultaddress, /* faultaddress */ /* TLB functions */ armv4_tlb_flushID, /* tlb_flushID */ fa526_tlb_flushID_SE, /* tlb_flushID_SE */ armv4_tlb_flushI, /* tlb_flushI */ fa526_tlb_flushI_SE, /* tlb_flushI_SE */ armv4_tlb_flushD, /* tlb_flushD */ armv4_tlb_flushD_SE, /* tlb_flushD_SE */ /* Cache operations */ fa526_icache_sync_all, /* icache_sync_all */ fa526_icache_sync_range, /* icache_sync_range */ fa526_dcache_wbinv_all, /* dcache_wbinv_all */ fa526_dcache_wbinv_range, /* dcache_wbinv_range */ fa526_dcache_inv_range, /* dcache_inv_range */ fa526_dcache_wb_range, /* dcache_wb_range */ armv4_idcache_inv_all, /* idcache_inv_all */ fa526_idcache_wbinv_all, /* idcache_wbinv_all */ fa526_idcache_wbinv_range, /* idcache_wbinv_range */ cpufunc_nullop, /* l2cache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_inv_range */ (void *)cpufunc_nullop, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ fa526_flush_prefetchbuf, /* flush_prefetchbuf */ armv4_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ fa526_flush_brnchtgt_E, /* flush_brnchtgt_E */ fa526_cpu_sleep, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ fa526_context_switch, /* context_switch */ fa526_setup /* cpu setup */ }; #endif /* CPU_FA526 */ #if defined(CPU_ARM1176) struct cpu_functions arm1176_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ cpufunc_nullop, /* cpwait */ /* MMU functions */ cpufunc_control, /* control */ cpufunc_domains, /* Domain */ arm11x6_setttb, /* Setttb */ cpufunc_faultstatus, /* Faultstatus */ cpufunc_faultaddress, /* Faultaddress */ /* TLB functions */ arm11_tlb_flushID, /* tlb_flushID */ arm11_tlb_flushID_SE, /* tlb_flushID_SE */ arm11_tlb_flushI, /* tlb_flushI */ arm11_tlb_flushI_SE, /* tlb_flushI_SE */ arm11_tlb_flushD, /* tlb_flushD */ arm11_tlb_flushD_SE, /* tlb_flushD_SE */ /* Cache operations */ arm11x6_icache_sync_all, /* icache_sync_all */ arm11x6_icache_sync_range, /* icache_sync_range */ arm11x6_dcache_wbinv_all, /* dcache_wbinv_all */ armv6_dcache_wbinv_range, /* dcache_wbinv_range */ armv6_dcache_inv_range, /* dcache_inv_range */ armv6_dcache_wb_range, /* dcache_wb_range */ armv6_idcache_inv_all, /* idcache_inv_all */ arm11x6_idcache_wbinv_all, /* idcache_wbinv_all */ arm11x6_idcache_wbinv_range, /* idcache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_inv_range */ (void *)cpufunc_nullop, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ arm11x6_flush_prefetchbuf, /* flush_prefetchbuf */ arm11_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ arm11x6_sleep, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ arm11_context_switch, /* context_switch */ arm11x6_setup /* cpu setup */ }; #endif /*CPU_ARM1176 */ #if defined(CPU_CORTEXA) || defined(CPU_KRAIT) struct cpu_functions cortexa_cpufuncs = { /* CPU functions */ cpufunc_id, /* id */ cpufunc_nullop, /* cpwait */ /* MMU functions */ cpufunc_control, /* control */ cpufunc_domains, /* Domain */ armv7_setttb, /* Setttb */ cpufunc_faultstatus, /* Faultstatus */ cpufunc_faultaddress, /* Faultaddress */ /* * TLB functions. ARMv7 does all TLB ops based on a unified TLB model * whether the hardware implements separate I+D or not, so we use the * same 'ID' functions for all 3 variations. */ armv7_tlb_flushID, /* tlb_flushID */ armv7_tlb_flushID_SE, /* tlb_flushID_SE */ armv7_tlb_flushID, /* tlb_flushI */ armv7_tlb_flushID_SE, /* tlb_flushI_SE */ armv7_tlb_flushID, /* tlb_flushD */ armv7_tlb_flushID_SE, /* tlb_flushD_SE */ /* Cache operations */ armv7_icache_sync_all, /* icache_sync_all */ armv7_icache_sync_range, /* icache_sync_range */ armv7_dcache_wbinv_all, /* dcache_wbinv_all */ armv7_dcache_wbinv_range, /* dcache_wbinv_range */ armv7_dcache_inv_range, /* dcache_inv_range */ armv7_dcache_wb_range, /* dcache_wb_range */ armv7_idcache_inv_all, /* idcache_inv_all */ armv7_idcache_wbinv_all, /* idcache_wbinv_all */ armv7_idcache_wbinv_range, /* idcache_wbinv_range */ /* * Note: For CPUs using the PL310 the L2 ops are filled in when the * L2 cache controller is actually enabled. */ cpufunc_nullop, /* l2cache_wbinv_all */ (void *)cpufunc_nullop, /* l2cache_wbinv_range */ (void *)cpufunc_nullop, /* l2cache_inv_range */ (void *)cpufunc_nullop, /* l2cache_wb_range */ (void *)cpufunc_nullop, /* l2cache_drain_writebuf */ /* Other functions */ cpufunc_nullop, /* flush_prefetchbuf */ armv7_drain_writebuf, /* drain_writebuf */ cpufunc_nullop, /* flush_brnchtgt_C */ (void *)cpufunc_nullop, /* flush_brnchtgt_E */ armv7_cpu_sleep, /* sleep */ /* Soft functions */ cpufunc_null_fixup, /* dataabt_fixup */ cpufunc_null_fixup, /* prefetchabt_fixup */ armv7_context_switch, /* context_switch */ cortexa_setup /* cpu setup */ }; #endif /* CPU_CORTEXA */ /* * Global constants also used by locore.s */ struct cpu_functions cpufuncs; u_int cputype; u_int cpu_reset_needs_v4_MMU_disable; /* flag used in locore.s */ #if defined(CPU_ARM9) || \ defined (CPU_ARM9E) || \ defined(CPU_ARM1176) || defined(CPU_XSCALE_80321) || \ defined(CPU_XSCALE_PXA2X0) || defined(CPU_XSCALE_IXP425) || \ defined(CPU_FA526) || defined(CPU_MV_PJ4B) || \ defined(CPU_XSCALE_80219) || defined(CPU_XSCALE_81342) || \ defined(CPU_CORTEXA) || defined(CPU_KRAIT) /* Global cache line sizes, use 32 as default */ int arm_dcache_min_line_size = 32; int arm_icache_min_line_size = 32; int arm_idcache_min_line_size = 32; static void get_cachetype_cp15(void); /* Additional cache information local to this file. Log2 of some of the above numbers. */ static int arm_dcache_l2_nsets; static int arm_dcache_l2_assoc; static int arm_dcache_l2_linesize; static void get_cachetype_cp15() { u_int ctype, isize, dsize, cpuid; u_int clevel, csize, i, sel; u_int multiplier; u_char type; __asm __volatile("mrc p15, 0, %0, c0, c0, 1" : "=r" (ctype)); cpuid = cpufunc_id(); /* * ...and thus spake the ARM ARM: * * If an value corresponding to an unimplemented or * reserved ID register is encountered, the System Control * processor returns the value of the main ID register. */ if (ctype == cpuid) goto out; if (CPU_CT_FORMAT(ctype) == CPU_CT_ARMV7) { /* Resolve minimal cache line sizes */ arm_dcache_min_line_size = 1 << (CPU_CT_DMINLINE(ctype) + 2); arm_icache_min_line_size = 1 << (CPU_CT_IMINLINE(ctype) + 2); arm_idcache_min_line_size = min(arm_icache_min_line_size, arm_dcache_min_line_size); __asm __volatile("mrc p15, 1, %0, c0, c0, 1" : "=r" (clevel)); arm_cache_level = clevel; arm_cache_loc = CPU_CLIDR_LOC(arm_cache_level); i = 0; while ((type = (clevel & 0x7)) && i < 7) { if (type == CACHE_DCACHE || type == CACHE_UNI_CACHE || type == CACHE_SEP_CACHE) { sel = i << 1; __asm __volatile("mcr p15, 2, %0, c0, c0, 0" : : "r" (sel)); __asm __volatile("mrc p15, 1, %0, c0, c0, 0" : "=r" (csize)); arm_cache_type[sel] = csize; arm_dcache_align = 1 << (CPUV7_CT_xSIZE_LEN(csize) + 4); arm_dcache_align_mask = arm_dcache_align - 1; } if (type == CACHE_ICACHE || type == CACHE_SEP_CACHE) { sel = (i << 1) | 1; __asm __volatile("mcr p15, 2, %0, c0, c0, 0" : : "r" (sel)); __asm __volatile("mrc p15, 1, %0, c0, c0, 0" : "=r" (csize)); arm_cache_type[sel] = csize; } i++; clevel >>= 3; } } else { if ((ctype & CPU_CT_S) == 0) arm_pcache_unified = 1; /* * If you want to know how this code works, go read the ARM ARM. */ arm_pcache_type = CPU_CT_CTYPE(ctype); if (arm_pcache_unified == 0) { isize = CPU_CT_ISIZE(ctype); multiplier = (isize & CPU_CT_xSIZE_M) ? 3 : 2; arm_picache_line_size = 1U << (CPU_CT_xSIZE_LEN(isize) + 3); if (CPU_CT_xSIZE_ASSOC(isize) == 0) { if (isize & CPU_CT_xSIZE_M) arm_picache_line_size = 0; /* not present */ else arm_picache_ways = 1; } else { arm_picache_ways = multiplier << (CPU_CT_xSIZE_ASSOC(isize) - 1); } arm_picache_size = multiplier << (CPU_CT_xSIZE_SIZE(isize) + 8); } dsize = CPU_CT_DSIZE(ctype); multiplier = (dsize & CPU_CT_xSIZE_M) ? 3 : 2; arm_pdcache_line_size = 1U << (CPU_CT_xSIZE_LEN(dsize) + 3); if (CPU_CT_xSIZE_ASSOC(dsize) == 0) { if (dsize & CPU_CT_xSIZE_M) arm_pdcache_line_size = 0; /* not present */ else arm_pdcache_ways = 1; } else { arm_pdcache_ways = multiplier << (CPU_CT_xSIZE_ASSOC(dsize) - 1); } arm_pdcache_size = multiplier << (CPU_CT_xSIZE_SIZE(dsize) + 8); arm_dcache_align = arm_pdcache_line_size; arm_dcache_l2_assoc = CPU_CT_xSIZE_ASSOC(dsize) + multiplier - 2; arm_dcache_l2_linesize = CPU_CT_xSIZE_LEN(dsize) + 3; arm_dcache_l2_nsets = 6 + CPU_CT_xSIZE_SIZE(dsize) - CPU_CT_xSIZE_ASSOC(dsize) - CPU_CT_xSIZE_LEN(dsize); out: arm_dcache_align_mask = arm_dcache_align - 1; } } #endif /* ARM9 || XSCALE */ /* * Cannot panic here as we may not have a console yet ... */ int set_cpufuncs() { cputype = cpufunc_id(); cputype &= CPU_ID_CPU_MASK; #ifdef CPU_ARM9 if (((cputype & CPU_ID_IMPLEMENTOR_MASK) == CPU_ID_ARM_LTD || (cputype & CPU_ID_IMPLEMENTOR_MASK) == CPU_ID_TI) && (cputype & 0x0000f000) == 0x00009000) { cpufuncs = arm9_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* V4 or higher */ get_cachetype_cp15(); arm9_dcache_sets_inc = 1U << arm_dcache_l2_linesize; arm9_dcache_sets_max = (1U << (arm_dcache_l2_linesize + arm_dcache_l2_nsets)) - arm9_dcache_sets_inc; arm9_dcache_index_inc = 1U << (32 - arm_dcache_l2_assoc); arm9_dcache_index_max = 0U - arm9_dcache_index_inc; pmap_pte_init_generic(); goto out; } #endif /* CPU_ARM9 */ #if defined(CPU_ARM9E) if (cputype == CPU_ID_MV88FR131 || cputype == CPU_ID_MV88FR571_VD || cputype == CPU_ID_MV88FR571_41) { uint32_t sheeva_ctrl; sheeva_ctrl = (MV_DC_STREAM_ENABLE | MV_BTB_DISABLE | MV_L2_ENABLE); /* * Workaround for Marvell MV78100 CPU: Cache prefetch * mechanism may affect the cache coherency validity, * so it needs to be disabled. * * Refer to errata document MV-S501058-00C.pdf (p. 3.1 * L2 Prefetching Mechanism) for details. */ if (cputype == CPU_ID_MV88FR571_VD || cputype == CPU_ID_MV88FR571_41) sheeva_ctrl |= MV_L2_PREFETCH_DISABLE; sheeva_control_ext(0xffffffff & ~MV_WA_ENABLE, sheeva_ctrl); cpufuncs = sheeva_cpufuncs; get_cachetype_cp15(); pmap_pte_init_generic(); goto out; } else if (cputype == CPU_ID_ARM926EJS) { cpufuncs = armv5_ec_cpufuncs; get_cachetype_cp15(); pmap_pte_init_generic(); goto out; } #endif /* CPU_ARM9E */ #if defined(CPU_ARM1176) if (cputype == CPU_ID_ARM1176JZS) { cpufuncs = arm1176_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* V4 or higher */ get_cachetype_cp15(); - - pmap_pte_init_mmu_v6(); - goto out; } #endif /* CPU_ARM1176 */ #if defined(CPU_CORTEXA) || defined(CPU_KRAIT) if (cputype == CPU_ID_CORTEXA5 || cputype == CPU_ID_CORTEXA7 || cputype == CPU_ID_CORTEXA8R1 || cputype == CPU_ID_CORTEXA8R2 || cputype == CPU_ID_CORTEXA8R3 || cputype == CPU_ID_CORTEXA9R1 || cputype == CPU_ID_CORTEXA9R2 || cputype == CPU_ID_CORTEXA9R3 || cputype == CPU_ID_CORTEXA9R4 || cputype == CPU_ID_CORTEXA12R0 || cputype == CPU_ID_CORTEXA15R0 || cputype == CPU_ID_CORTEXA15R1 || cputype == CPU_ID_CORTEXA15R2 || cputype == CPU_ID_CORTEXA15R3 || cputype == CPU_ID_KRAIT300R0 || cputype == CPU_ID_KRAIT300R1 ) { cpufuncs = cortexa_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* V4 or higher */ get_cachetype_cp15(); - - pmap_pte_init_mmu_v6(); goto out; } #endif /* CPU_CORTEXA */ #if defined(CPU_MV_PJ4B) if (cputype == CPU_ID_MV88SV581X_V7 || cputype == CPU_ID_MV88SV584X_V7 || cputype == CPU_ID_ARM_88SV581X_V7) { cpufuncs = pj4bv7_cpufuncs; get_cachetype_cp15(); - pmap_pte_init_mmu_v6(); goto out; } #endif /* CPU_MV_PJ4B */ #if defined(CPU_FA526) if (cputype == CPU_ID_FA526 || cputype == CPU_ID_FA626TE) { cpufuncs = fa526_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* SA needs it */ get_cachetype_cp15(); pmap_pte_init_generic(); goto out; } #endif /* CPU_FA526 */ #if defined(CPU_XSCALE_80321) || defined(CPU_XSCALE_80219) if (cputype == CPU_ID_80321_400 || cputype == CPU_ID_80321_600 || cputype == CPU_ID_80321_400_B0 || cputype == CPU_ID_80321_600_B0 || cputype == CPU_ID_80219_400 || cputype == CPU_ID_80219_600) { cpufuncs = xscale_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* XScale needs it */ get_cachetype_cp15(); pmap_pte_init_xscale(); goto out; } #endif /* CPU_XSCALE_80321 */ #if defined(CPU_XSCALE_81342) if (cputype == CPU_ID_81342) { cpufuncs = xscalec3_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* XScale needs it */ get_cachetype_cp15(); pmap_pte_init_xscale(); goto out; } #endif /* CPU_XSCALE_81342 */ #ifdef CPU_XSCALE_PXA2X0 /* ignore core revision to test PXA2xx CPUs */ if ((cputype & ~CPU_ID_XSCALE_COREREV_MASK) == CPU_ID_PXA250 || (cputype & ~CPU_ID_XSCALE_COREREV_MASK) == CPU_ID_PXA27X || (cputype & ~CPU_ID_XSCALE_COREREV_MASK) == CPU_ID_PXA210) { cpufuncs = xscale_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* XScale needs it */ get_cachetype_cp15(); pmap_pte_init_xscale(); goto out; } #endif /* CPU_XSCALE_PXA2X0 */ #ifdef CPU_XSCALE_IXP425 if (cputype == CPU_ID_IXP425_533 || cputype == CPU_ID_IXP425_400 || cputype == CPU_ID_IXP425_266 || cputype == CPU_ID_IXP435) { cpufuncs = xscale_cpufuncs; cpu_reset_needs_v4_MMU_disable = 1; /* XScale needs it */ get_cachetype_cp15(); pmap_pte_init_xscale(); goto out; } #endif /* CPU_XSCALE_IXP425 */ /* * Bzzzz. And the answer was ... */ panic("No support for this CPU type (%08x) in kernel", cputype); return(ARCHITECTURE_NOT_PRESENT); out: uma_set_align(arm_dcache_align_mask); return (0); } /* * Fixup routines for data and prefetch aborts. * * Several compile time symbols are used * * DEBUG_FAULT_CORRECTION - Print debugging information during the * correction of registers after a fault. */ /* * Null abort fixup routine. * For use when no fixup is required. */ int cpufunc_null_fixup(arg) void *arg; { return(ABORT_FIXUP_OK); } /* * CPU Setup code */ #ifdef CPU_ARM9 void arm9_setup(void) { int cpuctrl, cpuctrlmask; cpuctrl = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_32BP_ENABLE | CPU_CONTROL_32BD_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_LABT_ENABLE | CPU_CONTROL_ROUNDROBIN; cpuctrlmask = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_32BP_ENABLE | CPU_CONTROL_32BD_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_ROM_ENABLE | CPU_CONTROL_BEND_ENABLE | CPU_CONTROL_AFLT_ENABLE | CPU_CONTROL_LABT_ENABLE | CPU_CONTROL_VECRELOC | CPU_CONTROL_ROUNDROBIN; #ifndef ARM32_DISABLE_ALIGNMENT_FAULTS cpuctrl |= CPU_CONTROL_AFLT_ENABLE; #endif #ifdef __ARMEB__ cpuctrl |= CPU_CONTROL_BEND_ENABLE; #endif if (vector_page == ARM_VECTORS_HIGH) cpuctrl |= CPU_CONTROL_VECRELOC; /* Clear out the cache */ cpu_idcache_wbinv_all(); /* Set the control register */ cpu_control(cpuctrlmask, cpuctrl); ctrl = cpuctrl; } #endif /* CPU_ARM9 */ #if defined(CPU_ARM9E) void arm10_setup(void) { int cpuctrl, cpuctrlmask; cpuctrl = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_BPRD_ENABLE; cpuctrlmask = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_ROM_ENABLE | CPU_CONTROL_BEND_ENABLE | CPU_CONTROL_AFLT_ENABLE | CPU_CONTROL_BPRD_ENABLE | CPU_CONTROL_ROUNDROBIN | CPU_CONTROL_CPCLK; #ifndef ARM32_DISABLE_ALIGNMENT_FAULTS cpuctrl |= CPU_CONTROL_AFLT_ENABLE; #endif #ifdef __ARMEB__ cpuctrl |= CPU_CONTROL_BEND_ENABLE; #endif /* Clear out the cache */ cpu_idcache_wbinv_all(); /* Now really make sure they are clean. */ __asm __volatile ("mcr\tp15, 0, r0, c7, c7, 0" : : ); if (vector_page == ARM_VECTORS_HIGH) cpuctrl |= CPU_CONTROL_VECRELOC; /* Set the control register */ ctrl = cpuctrl; cpu_control(0xffffffff, cpuctrl); /* And again. */ cpu_idcache_wbinv_all(); } #endif /* CPU_ARM9E || CPU_ARM10 */ #if defined(CPU_ARM1176) \ || defined(CPU_MV_PJ4B) \ || defined(CPU_CORTEXA) || defined(CPU_KRAIT) static __inline void cpu_scc_setup_ccnt(void) { /* This is how you give userland access to the CCNT and PMCn * registers. * BEWARE! This gives write access also, which may not be what * you want! */ #ifdef _PMC_USER_READ_WRITE_ /* Set PMUSERENR[0] to allow userland access */ cp15_pmuserenr_set(1); #endif #if defined(CPU_ARM1176) /* Set PMCR[2,0] to enable counters and reset CCNT */ cp15_pmcr_set(5); #else /* Set up the PMCCNTR register as a cyclecounter: * Set PMINTENCLR to 0xFFFFFFFF to block interrupts * Set PMCR[2,0] to enable counters and reset CCNT * Set PMCNTENSET to 0x80000000 to enable CCNT */ cp15_pminten_clr(0xFFFFFFFF); cp15_pmcr_set(5); cp15_pmcnten_set(0x80000000); #endif } #endif #if defined(CPU_ARM1176) void arm11x6_setup(void) { int cpuctrl, cpuctrl_wax; uint32_t auxctrl, auxctrl_wax; uint32_t tmp, tmp2; uint32_t sbz=0; uint32_t cpuid; cpuid = cpufunc_id(); cpuctrl = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_32BP_ENABLE | CPU_CONTROL_32BD_ENABLE | CPU_CONTROL_LABT_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_UNAL_ENABLE; /* * "write as existing" bits * inverse of this is mask */ cpuctrl_wax = (3 << 30) | /* SBZ */ (1 << 29) | /* FA */ (1 << 28) | /* TR */ (3 << 26) | /* SBZ */ (3 << 19) | /* SBZ */ (1 << 17); /* SBZ */ cpuctrl |= CPU_CONTROL_BPRD_ENABLE; cpuctrl |= CPU_CONTROL_V6_EXTPAGE; #ifdef __ARMEB__ cpuctrl |= CPU_CONTROL_BEND_ENABLE; #endif if (vector_page == ARM_VECTORS_HIGH) cpuctrl |= CPU_CONTROL_VECRELOC; auxctrl = 0; auxctrl_wax = ~0; /* * Enable an errata workaround */ if ((cpuid & CPU_ID_CPU_MASK) == CPU_ID_ARM1176JZS) { /* ARM1176JZSr0 */ auxctrl = ARM1176_AUXCTL_PHD; auxctrl_wax = ~ARM1176_AUXCTL_PHD; } /* Clear out the cache */ cpu_idcache_wbinv_all(); /* Now really make sure they are clean. */ __asm volatile ("mcr\tp15, 0, %0, c7, c7, 0" : : "r"(sbz)); /* Allow detection code to find the VFP if it's fitted. */ cp15_cpacr_set(0x0fffffff); /* Set the control register */ ctrl = cpuctrl; cpu_control(~cpuctrl_wax, cpuctrl); tmp = cp15_actlr_get(); tmp2 = tmp; tmp &= auxctrl_wax; tmp |= auxctrl; if (tmp != tmp2) cp15_actlr_set(tmp); /* And again. */ cpu_idcache_wbinv_all(); cpu_scc_setup_ccnt(); } #endif /* CPU_ARM1176 */ #ifdef CPU_MV_PJ4B void pj4bv7_setup(void) { int cpuctrl; pj4b_config(); cpuctrl = CPU_CONTROL_MMU_ENABLE; #ifndef ARM32_DISABLE_ALIGNMENT_FAULTS cpuctrl |= CPU_CONTROL_AFLT_ENABLE; #endif cpuctrl |= CPU_CONTROL_DC_ENABLE; cpuctrl |= (0xf << 3); cpuctrl |= CPU_CONTROL_BPRD_ENABLE; cpuctrl |= CPU_CONTROL_IC_ENABLE; if (vector_page == ARM_VECTORS_HIGH) cpuctrl |= CPU_CONTROL_VECRELOC; cpuctrl |= (0x5 << 16) | (1 < 22); cpuctrl |= CPU_CONTROL_V6_EXTPAGE; /* Clear out the cache */ cpu_idcache_wbinv_all(); /* Set the control register */ ctrl = cpuctrl; cpu_control(0xFFFFFFFF, cpuctrl); /* And again. */ cpu_idcache_wbinv_all(); cpu_scc_setup_ccnt(); } #endif /* CPU_MV_PJ4B */ #if defined(CPU_CORTEXA) || defined(CPU_KRAIT) void cortexa_setup(void) { int cpuctrl, cpuctrlmask; cpuctrlmask = CPU_CONTROL_MMU_ENABLE | /* MMU enable [0] */ CPU_CONTROL_AFLT_ENABLE | /* Alignment fault [1] */ CPU_CONTROL_DC_ENABLE | /* DCache enable [2] */ CPU_CONTROL_BPRD_ENABLE | /* Branch prediction [11] */ CPU_CONTROL_IC_ENABLE | /* ICache enable [12] */ CPU_CONTROL_VECRELOC; /* Vector relocation [13] */ cpuctrl = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_BPRD_ENABLE; #ifndef ARM32_DISABLE_ALIGNMENT_FAULTS cpuctrl |= CPU_CONTROL_AFLT_ENABLE; #endif /* Switch to big endian */ #ifdef __ARMEB__ cpuctrl |= CPU_CONTROL_BEND_ENABLE; #endif /* Check if the vector page is at the high address (0xffff0000) */ if (vector_page == ARM_VECTORS_HIGH) cpuctrl |= CPU_CONTROL_VECRELOC; /* Clear out the cache */ cpu_idcache_wbinv_all(); /* Set the control register */ ctrl = cpuctrl; cpu_control(cpuctrlmask, cpuctrl); /* And again. */ cpu_idcache_wbinv_all(); #if defined(SMP) && !defined(ARM_NEW_PMAP) armv7_auxctrl((1 << 6) | (1 << 0), (1 << 6) | (1 << 0)); /* Enable SMP + TLB broadcasting */ #endif cpu_scc_setup_ccnt(); } #endif /* CPU_CORTEXA */ #if defined(CPU_FA526) void fa526_setup(void) { int cpuctrl, cpuctrlmask; cpuctrl = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_32BP_ENABLE | CPU_CONTROL_32BD_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_LABT_ENABLE | CPU_CONTROL_BPRD_ENABLE; cpuctrlmask = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_32BP_ENABLE | CPU_CONTROL_32BD_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_ROM_ENABLE | CPU_CONTROL_BEND_ENABLE | CPU_CONTROL_AFLT_ENABLE | CPU_CONTROL_LABT_ENABLE | CPU_CONTROL_BPRD_ENABLE | CPU_CONTROL_CPCLK | CPU_CONTROL_VECRELOC; #ifndef ARM32_DISABLE_ALIGNMENT_FAULTS cpuctrl |= CPU_CONTROL_AFLT_ENABLE; #endif #ifdef __ARMEB__ cpuctrl |= CPU_CONTROL_BEND_ENABLE; #endif if (vector_page == ARM_VECTORS_HIGH) cpuctrl |= CPU_CONTROL_VECRELOC; /* Clear out the cache */ cpu_idcache_wbinv_all(); /* Set the control register */ ctrl = cpuctrl; cpu_control(0xffffffff, cpuctrl); } #endif /* CPU_FA526 */ #if defined(CPU_XSCALE_80321) || \ defined(CPU_XSCALE_PXA2X0) || defined(CPU_XSCALE_IXP425) || \ defined(CPU_XSCALE_80219) || defined(CPU_XSCALE_81342) void xscale_setup(void) { uint32_t auxctl; int cpuctrl, cpuctrlmask; /* * The XScale Write Buffer is always enabled. Our option * is to enable/disable coalescing. Note that bits 6:3 * must always be enabled. */ cpuctrl = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_32BP_ENABLE | CPU_CONTROL_32BD_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_LABT_ENABLE | CPU_CONTROL_BPRD_ENABLE; cpuctrlmask = CPU_CONTROL_MMU_ENABLE | CPU_CONTROL_32BP_ENABLE | CPU_CONTROL_32BD_ENABLE | CPU_CONTROL_SYST_ENABLE | CPU_CONTROL_IC_ENABLE | CPU_CONTROL_DC_ENABLE | CPU_CONTROL_WBUF_ENABLE | CPU_CONTROL_ROM_ENABLE | CPU_CONTROL_BEND_ENABLE | CPU_CONTROL_AFLT_ENABLE | CPU_CONTROL_LABT_ENABLE | CPU_CONTROL_BPRD_ENABLE | CPU_CONTROL_CPCLK | CPU_CONTROL_VECRELOC | \ CPU_CONTROL_L2_ENABLE; #ifndef ARM32_DISABLE_ALIGNMENT_FAULTS cpuctrl |= CPU_CONTROL_AFLT_ENABLE; #endif #ifdef __ARMEB__ cpuctrl |= CPU_CONTROL_BEND_ENABLE; #endif if (vector_page == ARM_VECTORS_HIGH) cpuctrl |= CPU_CONTROL_VECRELOC; #ifdef CPU_XSCALE_CORE3 cpuctrl |= CPU_CONTROL_L2_ENABLE; #endif /* Clear out the cache */ cpu_idcache_wbinv_all(); /* * Set the control register. Note that bits 6:3 must always * be set to 1. */ ctrl = cpuctrl; /* cpu_control(cpuctrlmask, cpuctrl);*/ cpu_control(0xffffffff, cpuctrl); /* Make sure write coalescing is turned on */ __asm __volatile("mrc p15, 0, %0, c1, c0, 1" : "=r" (auxctl)); #ifdef XSCALE_NO_COALESCE_WRITES auxctl |= XSCALE_AUXCTL_K; #else auxctl &= ~XSCALE_AUXCTL_K; #endif #ifdef CPU_XSCALE_CORE3 auxctl |= XSCALE_AUXCTL_LLR; auxctl |= XSCALE_AUXCTL_MD_MASK; #endif __asm __volatile("mcr p15, 0, %0, c1, c0, 1" : : "r" (auxctl)); } #endif /* CPU_XSCALE_80321 || CPU_XSCALE_PXA2X0 || CPU_XSCALE_IXP425 CPU_XSCALE_80219 */ Index: user/ngie/make_check/sys/arm/arm/db_interface.c =================================================================== --- user/ngie/make_check/sys/arm/arm/db_interface.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/db_interface.c (revision 295078) @@ -1,330 +1,330 @@ /* $NetBSD: db_interface.c,v 1.33 2003/08/25 04:51:10 mrg Exp $ */ /*- * Copyright (c) 1996 Scott K. Stevens * * Mach Operating System * Copyright (c) 1991,1990 Carnegie Mellon University * All Rights Reserved. * * Permission to use, copy, modify and distribute this software and its * documentation is hereby granted, provided that both the copyright * notice and this permission notice appear in all copies of the * software, derivative works or modified versions, and any portions * thereof, and that both notices appear in supporting documentation. * * CARNEGIE MELLON ALLOWS FREE USE OF THIS SOFTWARE IN ITS "AS IS" * CONDITION. CARNEGIE MELLON DISCLAIMS ANY LIABILITY OF ANY KIND FOR * ANY DAMAGES WHATSOEVER RESULTING FROM THE USE OF THIS SOFTWARE. * * Carnegie Mellon requests users of this software to return to * * Software Distribution Coordinator or Software.Distribution@CS.CMU.EDU * School of Computer Science * Carnegie Mellon University * Pittsburgh PA 15213-3890 * * any improvements or extensions that they make and grant Carnegie the * rights to redistribute these changes. * * From: db_interface.c,v 2.4 1991/02/05 17:11:13 mrt (CMU) */ /* * Interface to new debugger. */ #include __FBSDID("$FreeBSD$"); #include "opt_ddb.h" #include #include #include #include /* just for boothowto */ #include #ifdef KDB #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include static int nil = 0; int db_access_und_sp (struct db_variable *, db_expr_t *, int); int db_access_abt_sp (struct db_variable *, db_expr_t *, int); int db_access_irq_sp (struct db_variable *, db_expr_t *, int); static db_varfcn_t db_frame; #define DB_OFFSET(x) (db_expr_t *)offsetof(struct trapframe, x) struct db_variable db_regs[] = { { "spsr", DB_OFFSET(tf_spsr), db_frame }, { "r0", DB_OFFSET(tf_r0), db_frame }, { "r1", DB_OFFSET(tf_r1), db_frame }, { "r2", DB_OFFSET(tf_r2), db_frame }, { "r3", DB_OFFSET(tf_r3), db_frame }, { "r4", DB_OFFSET(tf_r4), db_frame }, { "r5", DB_OFFSET(tf_r5), db_frame }, { "r6", DB_OFFSET(tf_r6), db_frame }, { "r7", DB_OFFSET(tf_r7), db_frame }, { "r8", DB_OFFSET(tf_r8), db_frame }, { "r9", DB_OFFSET(tf_r9), db_frame }, { "r10", DB_OFFSET(tf_r10), db_frame }, { "r11", DB_OFFSET(tf_r11), db_frame }, { "r12", DB_OFFSET(tf_r12), db_frame }, { "usr_sp", DB_OFFSET(tf_usr_sp), db_frame }, { "usr_lr", DB_OFFSET(tf_usr_lr), db_frame }, { "svc_sp", DB_OFFSET(tf_svc_sp), db_frame }, { "svc_lr", DB_OFFSET(tf_svc_lr), db_frame }, { "pc", DB_OFFSET(tf_pc), db_frame }, { "und_sp", &nil, db_access_und_sp, }, { "abt_sp", &nil, db_access_abt_sp, }, { "irq_sp", &nil, db_access_irq_sp, }, }; struct db_variable *db_eregs = db_regs + sizeof(db_regs)/sizeof(db_regs[0]); int db_access_und_sp(struct db_variable *vp, db_expr_t *valp, int rw) { if (rw == DB_VAR_GET) { *valp = get_stackptr(PSR_UND32_MODE); return (1); } return (0); } int db_access_abt_sp(struct db_variable *vp, db_expr_t *valp, int rw) { if (rw == DB_VAR_GET) { *valp = get_stackptr(PSR_ABT32_MODE); return (1); } return (0); } int db_access_irq_sp(struct db_variable *vp, db_expr_t *valp, int rw) { if (rw == DB_VAR_GET) { *valp = get_stackptr(PSR_IRQ32_MODE); return (1); } return (0); } int db_frame(struct db_variable *vp, db_expr_t *valp, int rw) { int *reg; if (kdb_frame == NULL) return (0); reg = (int *)((uintptr_t)kdb_frame + (db_expr_t)vp->valuep); if (rw == DB_VAR_GET) *valp = *reg; else *reg = *valp; return (1); } void db_show_mdpcpu(struct pcpu *pc) { #if __ARM_ARCH >= 6 db_printf("curpmap = %p\n", pc->pc_curpmap); #endif } int db_validate_address(vm_offset_t addr) { struct proc *p = curproc; struct pmap *pmap; if (!p || !p->p_vmspace || !p->p_vmspace->vm_map.pmap || #ifndef ARM32_NEW_VM_LAYOUT addr >= VM_MAXUSER_ADDRESS #else addr >= VM_MIN_KERNEL_ADDRESS #endif ) - pmap = pmap_kernel(); + pmap = kernel_pmap; else pmap = p->p_vmspace->vm_map.pmap; return (pmap_extract(pmap, addr) == FALSE); } /* * Read bytes from kernel address space for debugger. */ int db_read_bytes(addr, size, data) vm_offset_t addr; size_t size; char *data; { char *src = (char *)addr; if (db_validate_address((u_int)src)) { db_printf("address %p is invalid\n", src); return (-1); } if (size == 4 && (addr & 3) == 0 && ((uintptr_t)data & 3) == 0) { *((int*)data) = *((int*)src); return (0); } if (size == 2 && (addr & 1) == 0 && ((uintptr_t)data & 1) == 0) { *((short*)data) = *((short*)src); return (0); } while (size-- > 0) { if (db_validate_address((u_int)src)) { db_printf("address %p is invalid\n", src); return (-1); } *data++ = *src++; } return (0); } /* * Write bytes to kernel address space for debugger. */ int db_write_bytes(vm_offset_t addr, size_t size, char *data) { char *dst; size_t loop; dst = (char *)addr; if (db_validate_address((u_int)dst)) { db_printf("address %p is invalid\n", dst); return (0); } if (size == 4 && (addr & 3) == 0 && ((uintptr_t)data & 3) == 0) *((int*)dst) = *((int*)data); else if (size == 2 && (addr & 1) == 0 && ((uintptr_t)data & 1) == 0) *((short*)dst) = *((short*)data); else { loop = size; while (loop-- > 0) { if (db_validate_address((u_int)dst)) { db_printf("address %p is invalid\n", dst); return (-1); } *dst++ = *data++; } } /* make sure the caches and memory are in sync */ cpu_icache_sync_range(addr, size); /* In case the current page tables have been modified ... */ cpu_tlb_flushID(); cpu_cpwait(); return (0); } static u_int db_fetch_reg(int reg) { switch (reg) { case 0: return (kdb_frame->tf_r0); case 1: return (kdb_frame->tf_r1); case 2: return (kdb_frame->tf_r2); case 3: return (kdb_frame->tf_r3); case 4: return (kdb_frame->tf_r4); case 5: return (kdb_frame->tf_r5); case 6: return (kdb_frame->tf_r6); case 7: return (kdb_frame->tf_r7); case 8: return (kdb_frame->tf_r8); case 9: return (kdb_frame->tf_r9); case 10: return (kdb_frame->tf_r10); case 11: return (kdb_frame->tf_r11); case 12: return (kdb_frame->tf_r12); case 13: return (kdb_frame->tf_svc_sp); case 14: return (kdb_frame->tf_svc_lr); case 15: return (kdb_frame->tf_pc); default: panic("db_fetch_reg: botch"); } } static u_int db_branch_taken_read_int(void *cookie __unused, vm_offset_t offset, u_int *val) { u_int ret; db_read_bytes(offset, 4, (char *)&ret); *val = ret; return (0); } static u_int db_branch_taken_fetch_reg(void *cookie __unused, int reg) { return (db_fetch_reg(reg)); } u_int branch_taken(u_int insn, db_addr_t pc) { register_t new_pc; int ret; ret = arm_predict_branch(NULL, insn, (register_t)pc, &new_pc, db_branch_taken_fetch_reg, db_branch_taken_read_int); if (ret != 0) kdb_reenter(); return (new_pc); } Index: user/ngie/make_check/sys/arm/arm/genassym.c =================================================================== --- user/ngie/make_check/sys/arm/arm/genassym.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/genassym.c (revision 295078) @@ -1,170 +1,170 @@ /*- * Copyright (c) 2004 Olivier Houchard * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include ASSYM(KERNBASE, KERNBASE); ASSYM(PCB_NOALIGNFLT, PCB_NOALIGNFLT); -#ifdef ARM_NEW_PMAP +#if __ARM_ARCH >= 6 ASSYM(CPU_ASID_KERNEL,CPU_ASID_KERNEL); #endif ASSYM(PCB_ONFAULT, offsetof(struct pcb, pcb_onfault)); -#ifndef ARM_NEW_PMAP +#if __ARM_ARCH < 6 ASSYM(PCB_DACR, offsetof(struct pcb, pcb_dacr)); #endif ASSYM(PCB_FLAGS, offsetof(struct pcb, pcb_flags)); ASSYM(PCB_PAGEDIR, offsetof(struct pcb, pcb_pagedir)); -#ifndef ARM_NEW_PMAP +#if __ARM_ARCH < 6 ASSYM(PCB_L1VEC, offsetof(struct pcb, pcb_l1vec)); ASSYM(PCB_PL1VEC, offsetof(struct pcb, pcb_pl1vec)); #endif ASSYM(PCB_R4, offsetof(struct pcb, pcb_regs.sf_r4)); ASSYM(PCB_R5, offsetof(struct pcb, pcb_regs.sf_r5)); ASSYM(PCB_R6, offsetof(struct pcb, pcb_regs.sf_r6)); ASSYM(PCB_R7, offsetof(struct pcb, pcb_regs.sf_r7)); ASSYM(PCB_R8, offsetof(struct pcb, pcb_regs.sf_r8)); ASSYM(PCB_R9, offsetof(struct pcb, pcb_regs.sf_r9)); ASSYM(PCB_R10, offsetof(struct pcb, pcb_regs.sf_r10)); ASSYM(PCB_R11, offsetof(struct pcb, pcb_regs.sf_r11)); ASSYM(PCB_R12, offsetof(struct pcb, pcb_regs.sf_r12)); ASSYM(PCB_SP, offsetof(struct pcb, pcb_regs.sf_sp)); ASSYM(PCB_LR, offsetof(struct pcb, pcb_regs.sf_lr)); ASSYM(PCB_PC, offsetof(struct pcb, pcb_regs.sf_pc)); ASSYM(PC_CURPCB, offsetof(struct pcpu, pc_curpcb)); ASSYM(PC_CURTHREAD, offsetof(struct pcpu, pc_curthread)); ASSYM(M_LEN, offsetof(struct mbuf, m_len)); ASSYM(M_DATA, offsetof(struct mbuf, m_data)); ASSYM(M_NEXT, offsetof(struct mbuf, m_next)); ASSYM(IP_SRC, offsetof(struct ip, ip_src)); ASSYM(IP_DST, offsetof(struct ip, ip_dst)); ASSYM(CF_SETTTB, offsetof(struct cpu_functions, cf_setttb)); ASSYM(CF_CONTROL, offsetof(struct cpu_functions, cf_control)); ASSYM(CF_CONTEXT_SWITCH, offsetof(struct cpu_functions, cf_context_switch)); ASSYM(CF_DCACHE_WB_RANGE, offsetof(struct cpu_functions, cf_dcache_wb_range)); ASSYM(CF_L2CACHE_WB_RANGE, offsetof(struct cpu_functions, cf_l2cache_wb_range)); ASSYM(CF_IDCACHE_WBINV_ALL, offsetof(struct cpu_functions, cf_idcache_wbinv_all)); ASSYM(CF_L2CACHE_WBINV_ALL, offsetof(struct cpu_functions, cf_l2cache_wbinv_all)); ASSYM(CF_TLB_FLUSHID_SE, offsetof(struct cpu_functions, cf_tlb_flushID_SE)); ASSYM(CF_ICACHE_SYNC, offsetof(struct cpu_functions, cf_icache_sync_all)); ASSYM(V_TRAP, offsetof(struct vmmeter, v_trap)); ASSYM(V_SOFT, offsetof(struct vmmeter, v_soft)); ASSYM(V_INTR, offsetof(struct vmmeter, v_intr)); ASSYM(TD_PCB, offsetof(struct thread, td_pcb)); ASSYM(TD_FLAGS, offsetof(struct thread, td_flags)); ASSYM(TD_PROC, offsetof(struct thread, td_proc)); ASSYM(TD_FRAME, offsetof(struct thread, td_frame)); ASSYM(TD_MD, offsetof(struct thread, td_md)); ASSYM(TD_LOCK, offsetof(struct thread, td_lock)); ASSYM(MD_TP, offsetof(struct mdthread, md_tp)); ASSYM(MD_RAS_START, offsetof(struct mdthread, md_ras_start)); ASSYM(MD_RAS_END, offsetof(struct mdthread, md_ras_end)); ASSYM(TF_SPSR, offsetof(struct trapframe, tf_spsr)); ASSYM(TF_R0, offsetof(struct trapframe, tf_r0)); ASSYM(TF_R1, offsetof(struct trapframe, tf_r1)); ASSYM(TF_PC, offsetof(struct trapframe, tf_pc)); ASSYM(P_PID, offsetof(struct proc, p_pid)); ASSYM(P_FLAG, offsetof(struct proc, p_flag)); ASSYM(SIGF_UC, offsetof(struct sigframe, sf_uc)); #if __ARM_ARCH < 6 ASSYM(ARM_TP_ADDRESS, ARM_TP_ADDRESS); ASSYM(ARM_RAS_START, ARM_RAS_START); ASSYM(ARM_RAS_END, ARM_RAS_END); #endif #ifdef VFP ASSYM(PCB_VFPSTATE, offsetof(struct pcb, pcb_vfpstate)); #endif #if __ARM_ARCH >= 6 ASSYM(PC_CURPMAP, offsetof(struct pcpu, pc_curpmap)); #endif ASSYM(PAGE_SIZE, PAGE_SIZE); ASSYM(PMAP_DOMAIN_KERNEL, PMAP_DOMAIN_KERNEL); #ifdef PMAP_INCLUDE_PTE_SYNC ASSYM(PMAP_INCLUDE_PTE_SYNC, 1); #endif ASSYM(TDF_ASTPENDING, TDF_ASTPENDING); ASSYM(TDF_NEEDRESCHED, TDF_NEEDRESCHED); ASSYM(P_TRACED, P_TRACED); ASSYM(P_SIGEVENT, P_SIGEVENT); ASSYM(P_PROFIL, P_PROFIL); ASSYM(TRAPFRAMESIZE, sizeof(struct trapframe)); ASSYM(MAXCOMLEN, MAXCOMLEN); ASSYM(MAXCPU, MAXCPU); ASSYM(_NCPUWORDS, _NCPUWORDS); ASSYM(NIRQ, NIRQ); ASSYM(PCPU_SIZE, sizeof(struct pcpu)); ASSYM(P_VMSPACE, offsetof(struct proc, p_vmspace)); ASSYM(VM_PMAP, offsetof(struct vmspace, vm_pmap)); ASSYM(PM_ACTIVE, offsetof(struct pmap, pm_active)); ASSYM(PC_CPUID, offsetof(struct pcpu, pc_cpuid)); ASSYM(VM_MAXUSER_ADDRESS, VM_MAXUSER_ADDRESS); ASSYM(DCACHE_LINE_SIZE, offsetof(struct cpuinfo, dcache_line_size)); ASSYM(DCACHE_LINE_MASK, offsetof(struct cpuinfo, dcache_line_mask)); ASSYM(ICACHE_LINE_SIZE, offsetof(struct cpuinfo, icache_line_size)); ASSYM(ICACHE_LINE_MASK, offsetof(struct cpuinfo, icache_line_mask)); Index: user/ngie/make_check/sys/arm/arm/locore-v6.S =================================================================== --- user/ngie/make_check/sys/arm/arm/locore-v6.S (revision 295077) +++ user/ngie/make_check/sys/arm/arm/locore-v6.S (revision 295078) @@ -1,607 +1,587 @@ /*- * Copyright 2004-2014 Olivier Houchard * Copyright 2012-2014 Ian Lepore * Copyright 2013-2014 Andrew Turner * Copyright 2014 Svatopluk Kraus * Copyright 2014 Michal Meloun * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include "assym.s" #include +#include #include #include #include #include #include #include __FBSDID("$FreeBSD$"); -#ifndef ARM_NEW_PMAP -#define PTE1_OFFSET L1_S_OFFSET -#define PTE1_SHIFT L1_S_SHIFT -#define PTE1_SIZE L1_S_SIZE -#endif #if __ARM_ARCH >= 7 #if defined(__ARM_ARCH_7VE__) || defined(__clang__) /* * HYP support is in bintuils >= 2.21 and gcc >= 4.9 defines __ARM_ARCH_7VE__ * when enabled. llvm >= 3.6 supports it too. */ .arch_extension virt #define MSR_ELR_HYP(regnum) msr elr_hyp, lr #define ERET eret #else #define MSR_ELR_HYP(regnum) .word (0xe12ef300 | regnum) #define ERET .word 0xe160006e #endif #endif /* __ARM_ARCH >= 7 */ /* A small statically-allocated stack used only during initarm() and AP startup. */ #define INIT_ARM_STACK_SIZE 2048 .text .align 2 #if __ARM_ARCH >= 7 #define LEAVE_HYP \ /* Leave HYP mode */ ;\ mrs r0, cpsr ;\ and r0, r0, #(PSR_MODE) /* Mode is in the low 5 bits of CPSR */ ;\ teq r0, #(PSR_HYP32_MODE) /* Hyp Mode? */ ;\ bne 1f ;\ /* Ensure that IRQ, FIQ and Aborts will be disabled after eret */ ;\ mrs r0, cpsr ;\ bic r0, r0, #(PSR_MODE) ;\ orr r0, r0, #(PSR_SVC32_MODE) ;\ orr r0, r0, #(PSR_I | PSR_F | PSR_A) ;\ msr spsr_cxsf, r0 ;\ /* Exit hypervisor mode */ ;\ adr lr, 1f ;\ MSR_ELR_HYP(14) ;\ ERET ;\ 1: #else #define LEAVE_HYP #endif /* __ARM_ARCH >= 7 */ /* * On entry for FreeBSD boot ABI: * r0 - metadata pointer or 0 (boothowto on AT91's boot2) * r1 - if (r0 == 0) then metadata pointer * On entry for Linux boot ABI: * r0 - 0 * r1 - machine type (passed as arg2 to initarm) * r2 - Pointer to a tagged list or dtb image (phys addr) (passed as arg1 initarm) * * For both types of boot we gather up the args, put them in a struct arm_boot_params * structure and pass that to initarm. */ .globl btext btext: ASENTRY_NP(_start) STOP_UNWINDING /* Can't unwind into the bootloader! */ /* Make sure interrupts are disabled. */ cpsid ifa mov r8, r0 /* 0 or boot mode from boot2 */ mov r9, r1 /* Save Machine type */ mov r10, r2 /* Save meta data */ mov r11, r3 /* Future expansion */ LEAVE_HYP /* * Check whether data cache is enabled. If it is, then we know * current tags are valid (not power-on garbage values) and there * might be dirty lines that need cleaning. Disable cache to prevent * new lines being allocated, then call wbinv_poc_all to clean it. */ mrc CP15_SCTLR(r7) tst r7, #CPU_CONTROL_DC_ENABLE blne dcache_wbinv_poc_all /* ! Do not write to memory between wbinv and disabling cache ! */ /* * Now there are no dirty lines, but there may still be lines marked * valid. Disable all caches and the MMU, and invalidate everything * before setting up new page tables and re-enabling the mmu. */ 1: bic r7, #CPU_CONTROL_DC_ENABLE bic r7, #CPU_CONTROL_MMU_ENABLE bic r7, #CPU_CONTROL_IC_ENABLE bic r7, #CPU_CONTROL_UNAL_ENABLE bic r7, #CPU_CONTROL_BPRD_ENABLE bic r7, #CPU_CONTROL_SW_ENABLE orr r7, #CPU_CONTROL_AFLT_ENABLE orr r7, #CPU_CONTROL_VECRELOC mcr CP15_SCTLR(r7) DSB ISB bl dcache_inv_poc_all mcr CP15_ICIALLU DSB ISB /* * Build page table from scratch. */ /* * Figure out the physical address we're loaded at by assuming this * entry point code is in the first L1 section and so if we clear the * offset bits of the pc that will give us the section-aligned load * address, which remains in r5 throughout all the following code. */ ldr r2, =(L1_S_OFFSET) bic r5, pc, r2 /* Find the delta between VA and PA, result stays in r0 throughout. */ adr r0, Lpagetable bl translate_va_to_pa /* * First map the entire 4GB address space as VA=PA. It's mapped as * normal (cached) memory because it's for things like accessing the * parameters passed in from the bootloader, which might be at any * physical address, different for every platform. */ mov r1, #0 mov r2, #0 mov r3, #4096 bl build_pagetables /* * Next we do 64MiB starting at the physical load address, mapped to * the VA the kernel is linked for. */ mov r1, r5 ldr r2, =(KERNVIRTADDR) mov r3, #64 bl build_pagetables /* Create a device mapping for early_printf if specified. */ #if defined(SOCDEV_PA) && defined(SOCDEV_VA) ldr r1, =SOCDEV_PA ldr r2, =SOCDEV_VA mov r3, #1 bl build_device_pagetables #endif bl init_mmu /* Transition the PC from physical to virtual addressing. */ ldr pc, =1f 1: /* Setup stack, clear BSS */ ldr r1, =.Lstart ldmia r1, {r1, r2, sp} /* Set initial stack and */ add sp, sp, #INIT_ARM_STACK_SIZE sub r2, r2, r1 /* get zero init data */ mov r3, #0 2: str r3, [r1], #0x0004 /* get zero init data */ subs r2, r2, #4 bgt 2b mov r1, #28 /* loader info size is 28 bytes also second arg */ subs sp, sp, r1 /* allocate arm_boot_params struct on stack */ mov r0, sp /* loader info pointer is first arg */ bic sp, sp, #7 /* align stack to 8 bytes */ str r1, [r0] /* Store length of loader info */ str r8, [r0, #4] /* Store r0 from boot loader */ str r9, [r0, #8] /* Store r1 from boot loader */ str r10, [r0, #12] /* store r2 from boot loader */ str r11, [r0, #16] /* store r3 from boot loader */ str r5, [r0, #20] /* store the physical address */ adr r4, Lpagetable /* load the pagetable address */ ldr r5, [r4, #4] str r5, [r0, #24] /* store the pagetable address */ mov fp, #0 /* trace back starts here */ bl _C_LABEL(initarm) /* Off we go */ /* init arm will return the new stack pointer. */ mov sp, r0 bl _C_LABEL(mi_startup) /* call mi_startup()! */ ldr r0, =.Lmainreturned b _C_LABEL(panic) /* NOTREACHED */ END(_start) #define VA_TO_PA_POINTER(name, table) \ name: ;\ .word . ;\ .word table /* * Returns the physical address of a magic va to pa pointer. * r0 - The pagetable data pointer. This must be built using the * VA_TO_PA_POINTER macro. * e.g. * VA_TO_PA_POINTER(Lpagetable, pagetable) * ... * adr r0, Lpagetable * bl translate_va_to_pa * r0 will now contain the physical address of pagetable * r1, r2 - Trashed */ translate_va_to_pa: ldr r1, [r0] sub r2, r1, r0 /* At this point: r2 = VA - PA */ /* * Find the physical address of the table. After these two * instructions: * r1 = va(pagetable) * * r0 = va(pagetable) - (VA - PA) * = va(pagetable) - VA + PA * = pa(pagetable) */ ldr r1, [r0, #4] sub r0, r1, r2 mov pc, lr /* * Init MMU * r0 - the table base address */ ASENTRY_NP(init_mmu) /* Setup TLB and MMU registers */ mcr CP15_TTBR0(r0) /* Set TTB */ mov r0, #0 mcr CP15_CONTEXTIDR(r0) /* Set ASID to 0 */ /* Set the Domain Access register */ mov r0, #((DOMAIN_CLIENT << (PMAP_DOMAIN_KERNEL*2)) | DOMAIN_CLIENT) mcr CP15_DACR(r0) -#ifdef ARM_NEW_PMAP /* * Set TEX remap registers * - All is set to uncacheable memory */ ldr r0, =0xAAAAA mcr CP15_PRRR(r0) mov r0, #0 mcr CP15_NMRR(r0) -#endif mcr CP15_TLBIALL /* Flush TLB */ DSB ISB /* Enable MMU */ mrc CP15_SCTLR(r0) orr r0, r0, #CPU_CONTROL_MMU_ENABLE orr r0, r0, #CPU_CONTROL_V6_EXTPAGE -#ifdef ARM_NEW_PMAP orr r0, r0, #CPU_CONTROL_TR_ENABLE -#endif orr r0, r0, #CPU_CONTROL_AF_ENABLE mcr CP15_SCTLR(r0) DSB ISB mcr CP15_TLBIALL /* Flush TLB */ mcr CP15_BPIALL /* Flush Branch predictor */ DSB ISB mov pc, lr END(init_mmu) /* * Init SMP coherent mode, enable caching and switch to final MMU table. * Called with disabled caches * r0 - The table base address * r1 - clear bits for aux register * r2 - set bits for aux register */ ASENTRY_NP(reinit_mmu) push {r4-r11, lr} mov r4, r0 mov r5, r1 mov r6, r2 /* !! Be very paranoid here !! */ /* !! We cannot write single bit here !! */ #if 0 /* XXX writeback shouldn't be necessary */ /* Write back and invalidate all integrated caches */ bl dcache_wbinv_poc_all #else bl dcache_inv_pou_all #endif mcr CP15_ICIALLU DSB ISB /* Set auxiliary register */ mrc CP15_ACTLR(r7) bic r8, r7, r5 /* Mask bits */ eor r8, r8, r6 /* Set bits */ teq r7, r8 mcrne CP15_ACTLR(r8) DSB ISB /* Enable caches. */ mrc CP15_SCTLR(r7) orr r7, #CPU_CONTROL_DC_ENABLE orr r7, #CPU_CONTROL_IC_ENABLE orr r7, #CPU_CONTROL_BPRD_ENABLE mcr CP15_SCTLR(r7) DSB mcr CP15_TTBR0(r4) /* Set new TTB */ DSB ISB mcr CP15_TLBIALL /* Flush TLB */ mcr CP15_BPIALL /* Flush Branch predictor */ DSB ISB #if 0 /* XXX writeback shouldn't be necessary */ /* Write back and invalidate all integrated caches */ bl dcache_wbinv_poc_all #else bl dcache_inv_pou_all #endif mcr CP15_ICIALLU DSB ISB pop {r4-r11, pc} END(reinit_mmu) /* * Builds the page table * r0 - The table base address * r1 - The physical address (trashed) * r2 - The virtual address (trashed) * r3 - The number of 1MiB sections * r4 - Trashed * * Addresses must be 1MiB aligned */ build_device_pagetables: -#if defined(ARM_NEW_PMAP) ldr r4, =PTE1_V|PTE1_A|PTE1_AP_KRW|TEX1_CLASS_0 -#elif defined(SMP) - ldr r4, =(L1_TYPE_S|L1_S_AP(AP_KRW)|L1_SHARED) -#else - ldr r4, =(L1_TYPE_S|L1_S_AP(AP_KRW)) -#endif b 1f build_pagetables: /* Set the required page attributed */ -#if defined(ARM_NEW_PMAP) ldr r4, =PTE1_V|PTE1_A|PTE1_AP_KRW|TEX1_CLASS_0 -#elif defined(SMP) - ldr r4, =(L1_TYPE_S|L1_S_C|L1_S_AP(AP_KRW)|L1_SHARED) -#else - ldr r4, =(L1_TYPE_S|L1_S_C|L1_S_AP(AP_KRW)) -#endif 1: orr r1, r4 /* Move the virtual address to the correct bit location */ lsr r2, #(PTE1_SHIFT - 2) mov r4, r3 2: str r1, [r0, r2] add r2, r2, #4 add r1, r1, #(PTE1_SIZE) adds r4, r4, #-1 bhi 2b mov pc, lr VA_TO_PA_POINTER(Lpagetable, boot_pt1) .Lstart: .word _edata /* Note that these three items are */ .word _ebss /* loaded with a single ldmia and */ .word svcstk /* must remain in order together. */ .Lmainreturned: .asciz "main() returned" .align 2 .bss svcstk: .space INIT_ARM_STACK_SIZE * MAXCPU /* * Memory for the initial pagetable. We are unable to place this in * the bss as this will be cleared after the table is loaded. */ .section ".init_pagetable" .align 14 /* 16KiB aligned */ .globl boot_pt1 boot_pt1: .space L1_TABLE_SIZE .text .align 2 .Lcpufuncs: .word _C_LABEL(cpufuncs) #if defined(SMP) ASENTRY_NP(mpentry) /* Make sure interrupts are disabled. */ cpsid ifa LEAVE_HYP /* Setup core, disable all caches. */ mrc CP15_SCTLR(r0) bic r0, #CPU_CONTROL_MMU_ENABLE bic r0, #CPU_CONTROL_DC_ENABLE bic r0, #CPU_CONTROL_IC_ENABLE bic r0, #CPU_CONTROL_UNAL_ENABLE bic r0, #CPU_CONTROL_BPRD_ENABLE bic r0, #CPU_CONTROL_SW_ENABLE orr r0, #CPU_CONTROL_AFLT_ENABLE orr r0, #CPU_CONTROL_VECRELOC mcr CP15_SCTLR(r0) DSB ISB /* Invalidate L1 cache I+D cache */ bl dcache_inv_pou_all mcr CP15_ICIALLU DSB ISB /* Find the delta between VA and PA */ adr r0, Lpagetable bl translate_va_to_pa bl init_mmu adr r1, .Lstart+8 /* Get initstack pointer from */ ldr sp, [r1] /* startup data. */ mrc CP15_MPIDR(r0) /* Get processor id number. */ and r0, r0, #0x0f mov r1, #INIT_ARM_STACK_SIZE mul r2, r1, r0 /* Point sp to initstack */ add sp, sp, r2 /* area for this processor. */ /* Switch to virtual addresses. */ ldr pc, =1f 1: mov fp, #0 /* trace back starts here */ bl _C_LABEL(init_secondary)/* Off we go, cpu id in r0. */ adr r0, .Lmpreturned b _C_LABEL(panic) /* NOTREACHED */ END(mpentry) .Lmpreturned: .asciz "init_secondary() returned" .align 2 #endif ENTRY_NP(cpu_halt) /* XXX re-implement !!! */ cpsid ifa bl dcache_wbinv_poc_all ldr r4, .Lcpu_reset_address ldr r4, [r4] teq r4, #0 movne pc, r4 1: WFI b 1b /* * _cpu_reset_address contains the address to branch to, to complete * the cpu reset after turning the MMU off * This variable is provided by the hardware specific code */ .Lcpu_reset_address: .word _C_LABEL(cpu_reset_address) END(cpu_halt) /* * setjump + longjmp */ ENTRY(setjmp) stmia r0, {r4-r14} mov r0, #0x00000000 RET END(setjmp) ENTRY(longjmp) ldmia r0, {r4-r14} mov r0, #0x00000001 RET END(longjmp) .data .global _C_LABEL(esym) _C_LABEL(esym): .word _C_LABEL(end) ENTRY_NP(abort) b _C_LABEL(abort) END(abort) ENTRY_NP(sigcode) mov r0, sp add r0, r0, #SIGF_UC /* * Call the sigreturn system call. * * We have to load r7 manually rather than using * "ldr r7, =SYS_sigreturn" to ensure the value of szsigcode is * correct. Using the alternative places esigcode at the address * of the data rather than the address one past the data. */ ldr r7, [pc, #12] /* Load SYS_sigreturn */ swi SYS_sigreturn /* Well if that failed we better exit quick ! */ ldr r7, [pc, #8] /* Load SYS_exit */ swi SYS_exit /* Branch back to retry SYS_sigreturn */ b . - 16 END(sigcode) .word SYS_sigreturn .word SYS_exit .align 2 .global _C_LABEL(esigcode) _C_LABEL(esigcode): .data .global szsigcode szsigcode: .long esigcode-sigcode /* End of locore.S */ Index: user/ngie/make_check/sys/arm/arm/machdep.c =================================================================== --- user/ngie/make_check/sys/arm/arm/machdep.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/machdep.c (revision 295078) @@ -1,1920 +1,1916 @@ /* $NetBSD: arm32_machdep.c,v 1.44 2004/03/24 15:34:47 atatat Exp $ */ /*- * Copyright (c) 2004 Olivier Houchard * Copyright (c) 1994-1998 Mark Brinicombe. * Copyright (c) 1994 Brini. * All rights reserved. * * This code is derived from software written for Brini by Mark Brinicombe * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Mark Brinicombe * for the NetBSD Project. * 4. The name of the company nor the name of the author may be used to * endorse or promote products derived from this software without specific * prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * Machine dependant functions for kernel setup * * Created : 17/09/94 * Updated : 18/04/01 updated for new wscons */ #include "opt_compat.h" #include "opt_ddb.h" #include "opt_kstack_pages.h" #include "opt_platform.h" #include "opt_sched.h" #include "opt_timer.h" #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef FDT #include #include #endif #ifdef DDB #include #if __ARM_ARCH >= 6 #include DB_SHOW_COMMAND(cp15, db_show_cp15) { u_int reg; reg = cp15_midr_get(); db_printf("Cpu ID: 0x%08x\n", reg); reg = cp15_ctr_get(); db_printf("Current Cache Lvl ID: 0x%08x\n",reg); reg = cp15_sctlr_get(); db_printf("Ctrl: 0x%08x\n",reg); reg = cp15_actlr_get(); db_printf("Aux Ctrl: 0x%08x\n",reg); reg = cp15_id_pfr0_get(); db_printf("Processor Feat 0: 0x%08x\n", reg); reg = cp15_id_pfr1_get(); db_printf("Processor Feat 1: 0x%08x\n", reg); reg = cp15_id_dfr0_get(); db_printf("Debug Feat 0: 0x%08x\n", reg); reg = cp15_id_afr0_get(); db_printf("Auxiliary Feat 0: 0x%08x\n", reg); reg = cp15_id_mmfr0_get(); db_printf("Memory Model Feat 0: 0x%08x\n", reg); reg = cp15_id_mmfr1_get(); db_printf("Memory Model Feat 1: 0x%08x\n", reg); reg = cp15_id_mmfr2_get(); db_printf("Memory Model Feat 2: 0x%08x\n", reg); reg = cp15_id_mmfr3_get(); db_printf("Memory Model Feat 3: 0x%08x\n", reg); reg = cp15_ttbr_get(); db_printf("TTB0: 0x%08x\n", reg); } DB_SHOW_COMMAND(vtop, db_show_vtop) { u_int reg; if (have_addr) { cp15_ats1cpr_set(addr); reg = cp15_par_get(); db_printf("Physical address reg: 0x%08x\n",reg); } else db_printf("show vtop \n"); } #endif /* __ARM_ARCH >= 6 */ #endif /* DDB */ #ifdef DEBUG #define debugf(fmt, args...) printf(fmt, ##args) #else #define debugf(fmt, args...) #endif struct pcpu __pcpu[MAXCPU]; struct pcpu *pcpup = &__pcpu[0]; static struct trapframe proc0_tf; uint32_t cpu_reset_address = 0; int cold = 1; vm_offset_t vector_page; int (*_arm_memcpy)(void *, void *, int, int) = NULL; int (*_arm_bzero)(void *, int, int) = NULL; int _min_memcpy_size = 0; int _min_bzero_size = 0; extern int *end; #ifdef FDT static char *loader_envp; vm_paddr_t pmap_pa; -#ifdef ARM_NEW_PMAP +#if __ARM_ARCH >= 6 vm_offset_t systempage; vm_offset_t irqstack; vm_offset_t undstack; vm_offset_t abtstack; #else /* * This is the number of L2 page tables required for covering max * (hypothetical) memsize of 4GB and all kernel mappings (vectors, msgbuf, * stacks etc.), uprounded to be divisible by 4. */ #define KERNEL_PT_MAX 78 static struct pv_addr kernel_pt_table[KERNEL_PT_MAX]; struct pv_addr systempage; static struct pv_addr msgbufpv; struct pv_addr irqstack; struct pv_addr undstack; struct pv_addr abtstack; static struct pv_addr kernelstack; #endif #endif #if defined(LINUX_BOOT_ABI) #define LBABI_MAX_BANKS 10 uint32_t board_id; struct arm_lbabi_tag *atag_list; char linux_command_line[LBABI_MAX_COMMAND_LINE + 1]; char atags[LBABI_MAX_COMMAND_LINE * 2]; uint32_t memstart[LBABI_MAX_BANKS]; uint32_t memsize[LBABI_MAX_BANKS]; uint32_t membanks; #endif static uint32_t board_revision; /* hex representation of uint64_t */ static char board_serial[32]; SYSCTL_NODE(_hw, OID_AUTO, board, CTLFLAG_RD, 0, "Board attributes"); SYSCTL_UINT(_hw_board, OID_AUTO, revision, CTLFLAG_RD, &board_revision, 0, "Board revision"); SYSCTL_STRING(_hw_board, OID_AUTO, serial, CTLFLAG_RD, board_serial, 0, "Board serial"); int vfp_exists; SYSCTL_INT(_hw, HW_FLOATINGPT, floatingpoint, CTLFLAG_RD, &vfp_exists, 0, "Floating point support enabled"); void board_set_serial(uint64_t serial) { snprintf(board_serial, sizeof(board_serial)-1, "%016jx", serial); } void board_set_revision(uint32_t revision) { board_revision = revision; } void sendsig(catcher, ksi, mask) sig_t catcher; ksiginfo_t *ksi; sigset_t *mask; { struct thread *td; struct proc *p; struct trapframe *tf; struct sigframe *fp, frame; struct sigacts *psp; struct sysentvec *sysent; int onstack; int sig; int code; td = curthread; p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); sig = ksi->ksi_signo; code = ksi->ksi_code; psp = p->p_sigacts; mtx_assert(&psp->ps_mtx, MA_OWNED); tf = td->td_frame; onstack = sigonstack(tf->tf_usr_sp); CTR4(KTR_SIG, "sendsig: td=%p (%s) catcher=%p sig=%d", td, p->p_comm, catcher, sig); /* Allocate and validate space for the signal handler context. */ if ((td->td_pflags & TDP_ALTSTACK) != 0 && !(onstack) && SIGISMEMBER(psp->ps_sigonstack, sig)) { fp = (struct sigframe *)((uintptr_t)td->td_sigstk.ss_sp + td->td_sigstk.ss_size); #if defined(COMPAT_43) td->td_sigstk.ss_flags |= SS_ONSTACK; #endif } else fp = (struct sigframe *)td->td_frame->tf_usr_sp; /* make room on the stack */ fp--; /* make the stack aligned */ fp = (struct sigframe *)STACKALIGN(fp); /* Populate the siginfo frame. */ get_mcontext(td, &frame.sf_uc.uc_mcontext, 0); frame.sf_si = ksi->ksi_info; frame.sf_uc.uc_sigmask = *mask; frame.sf_uc.uc_stack.ss_flags = (td->td_pflags & TDP_ALTSTACK ) ? ((onstack) ? SS_ONSTACK : 0) : SS_DISABLE; frame.sf_uc.uc_stack = td->td_sigstk; mtx_unlock(&psp->ps_mtx); PROC_UNLOCK(td->td_proc); /* Copy the sigframe out to the user's stack. */ if (copyout(&frame, fp, sizeof(*fp)) != 0) { /* Process has trashed its stack. Kill it. */ CTR2(KTR_SIG, "sendsig: sigexit td=%p fp=%p", td, fp); PROC_LOCK(p); sigexit(td, SIGILL); } /* * Build context to run handler in. We invoke the handler * directly, only returning via the trampoline. Note the * trampoline version numbers are coordinated with machine- * dependent code in libc. */ tf->tf_r0 = sig; tf->tf_r1 = (register_t)&fp->sf_si; tf->tf_r2 = (register_t)&fp->sf_uc; /* the trampoline uses r5 as the uc address */ tf->tf_r5 = (register_t)&fp->sf_uc; tf->tf_pc = (register_t)catcher; tf->tf_usr_sp = (register_t)fp; sysent = p->p_sysent; if (sysent->sv_sigcode_base != 0) tf->tf_usr_lr = (register_t)sysent->sv_sigcode_base; else tf->tf_usr_lr = (register_t)(sysent->sv_psstrings - *(sysent->sv_szsigcode)); /* Set the mode to enter in the signal handler */ #if __ARM_ARCH >= 7 if ((register_t)catcher & 1) tf->tf_spsr |= PSR_T; else tf->tf_spsr &= ~PSR_T; #endif CTR3(KTR_SIG, "sendsig: return td=%p pc=%#x sp=%#x", td, tf->tf_usr_lr, tf->tf_usr_sp); PROC_LOCK(p); mtx_lock(&psp->ps_mtx); } struct kva_md_info kmi; /* * arm32_vector_init: * * Initialize the vector page, and select whether or not to * relocate the vectors. * * NOTE: We expect the vector page to be mapped at its expected * destination. */ extern unsigned int page0[], page0_data[]; void arm_vector_init(vm_offset_t va, int which) { unsigned int *vectors = (int *) va; unsigned int *vectors_data = vectors + (page0_data - page0); int vec; /* * Loop through the vectors we're taking over, and copy the * vector's insn and data word. */ for (vec = 0; vec < ARM_NVEC; vec++) { if ((which & (1 << vec)) == 0) { /* Don't want to take over this vector. */ continue; } vectors[vec] = page0[vec]; vectors_data[vec] = page0_data[vec]; } /* Now sync the vectors. */ cpu_icache_sync_range(va, (ARM_NVEC * 2) * sizeof(u_int)); vector_page = va; if (va == ARM_VECTORS_HIGH) { /* * Assume the MD caller knows what it's doing here, and * really does want the vector page relocated. * * Note: This has to be done here (and not just in * cpu_setup()) because the vector page needs to be * accessible *before* cpu_startup() is called. * Think ddb(9) ... * * NOTE: If the CPU control register is not readable, * this will totally fail! We'll just assume that * any system that has high vector support has a * readable CPU control register, for now. If we * ever encounter one that does not, we'll have to * rethink this. */ cpu_control(CPU_CONTROL_VECRELOC, CPU_CONTROL_VECRELOC); } } static void cpu_startup(void *dummy) { struct pcb *pcb = thread0.td_pcb; const unsigned int mbyte = 1024 * 1024; -#ifdef ARM_TP_ADDRESS -#ifndef ARM_CACHE_LOCK_ENABLE +#if __ARM_ARCH < 6 && !defined(ARM_CACHE_LOCK_ENABLE) vm_page_t m; #endif -#endif identify_arm_cpu(); vm_ksubmap_init(&kmi); /* * Display the RAM layout. */ printf("real memory = %ju (%ju MB)\n", (uintmax_t)arm32_ptob(realmem), (uintmax_t)arm32_ptob(realmem) / mbyte); printf("avail memory = %ju (%ju MB)\n", (uintmax_t)arm32_ptob(vm_cnt.v_free_count), (uintmax_t)arm32_ptob(vm_cnt.v_free_count) / mbyte); if (bootverbose) { arm_physmem_print_tables(); arm_devmap_print_table(); } bufinit(); vm_pager_bufferinit(); pcb->pcb_regs.sf_sp = (u_int)thread0.td_kstack + USPACE_SVC_STACK_TOP; - pmap_set_pcb_pagedir(pmap_kernel(), pcb); -#ifndef ARM_NEW_PMAP + pmap_set_pcb_pagedir(kernel_pmap, pcb); +#if __ARM_ARCH < 6 vector_page_setprot(VM_PROT_READ); pmap_postinit(); -#endif -#ifdef ARM_TP_ADDRESS #ifdef ARM_CACHE_LOCK_ENABLE pmap_kenter_user(ARM_TP_ADDRESS, ARM_TP_ADDRESS); arm_lock_cache_line(ARM_TP_ADDRESS); #else m = vm_page_alloc(NULL, 0, VM_ALLOC_NOOBJ | VM_ALLOC_ZERO); pmap_kenter_user(ARM_TP_ADDRESS, VM_PAGE_TO_PHYS(m)); #endif *(uint32_t *)ARM_RAS_START = 0; *(uint32_t *)ARM_RAS_END = 0xffffffff; #endif } SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); /* * Flush the D-cache for non-DMA I/O so that the I-cache can * be made coherent later. */ void cpu_flush_dcache(void *ptr, size_t len) { cpu_dcache_wb_range((uintptr_t)ptr, len); #ifdef ARM_L2_PIPT cpu_l2cache_wb_range((uintptr_t)vtophys(ptr), len); #else cpu_l2cache_wb_range((uintptr_t)ptr, len); #endif } /* Get current clock frequency for the given cpu id. */ int cpu_est_clockrate(int cpu_id, uint64_t *rate) { return (ENXIO); } void cpu_idle(int busy) { CTR2(KTR_SPARE2, "cpu_idle(%d) at %d", busy, curcpu); spinlock_enter(); #ifndef NO_EVENTTIMERS if (!busy) cpu_idleclock(); #endif if (!sched_runnable()) cpu_sleep(0); #ifndef NO_EVENTTIMERS if (!busy) cpu_activeclock(); #endif spinlock_exit(); CTR2(KTR_SPARE2, "cpu_idle(%d) at %d done", busy, curcpu); } int cpu_idle_wakeup(int cpu) { return (0); } /* * Most ARM platforms don't need to do anything special to init their clocks * (they get intialized during normal device attachment), and by not defining a * cpu_initclocks() function they get this generic one. Any platform that needs * to do something special can just provide their own implementation, which will * override this one due to the weak linkage. */ void arm_generic_initclocks(void) { #ifndef NO_EVENTTIMERS #ifdef SMP if (PCPU_GET(cpuid) == 0) cpu_initclocks_bsp(); else cpu_initclocks_ap(); #else cpu_initclocks_bsp(); #endif #endif } __weak_reference(arm_generic_initclocks, cpu_initclocks); int fill_regs(struct thread *td, struct reg *regs) { struct trapframe *tf = td->td_frame; bcopy(&tf->tf_r0, regs->r, sizeof(regs->r)); regs->r_sp = tf->tf_usr_sp; regs->r_lr = tf->tf_usr_lr; regs->r_pc = tf->tf_pc; regs->r_cpsr = tf->tf_spsr; return (0); } int fill_fpregs(struct thread *td, struct fpreg *regs) { bzero(regs, sizeof(*regs)); return (0); } int set_regs(struct thread *td, struct reg *regs) { struct trapframe *tf = td->td_frame; bcopy(regs->r, &tf->tf_r0, sizeof(regs->r)); tf->tf_usr_sp = regs->r_sp; tf->tf_usr_lr = regs->r_lr; tf->tf_pc = regs->r_pc; tf->tf_spsr &= ~PSR_FLAGS; tf->tf_spsr |= regs->r_cpsr & PSR_FLAGS; return (0); } int set_fpregs(struct thread *td, struct fpreg *regs) { return (0); } int fill_dbregs(struct thread *td, struct dbreg *regs) { return (0); } int set_dbregs(struct thread *td, struct dbreg *regs) { return (0); } static int ptrace_read_int(struct thread *td, vm_offset_t addr, uint32_t *v) { if (proc_readmem(td, td->td_proc, addr, v, sizeof(*v)) != sizeof(*v)) return (ENOMEM); return (0); } static int ptrace_write_int(struct thread *td, vm_offset_t addr, uint32_t v) { if (proc_writemem(td, td->td_proc, addr, &v, sizeof(v)) != sizeof(v)) return (ENOMEM); return (0); } static u_int ptrace_get_usr_reg(void *cookie, int reg) { int ret; struct thread *td = cookie; KASSERT(((reg >= 0) && (reg <= ARM_REG_NUM_PC)), ("reg is outside range")); switch(reg) { case ARM_REG_NUM_PC: ret = td->td_frame->tf_pc; break; case ARM_REG_NUM_LR: ret = td->td_frame->tf_usr_lr; break; case ARM_REG_NUM_SP: ret = td->td_frame->tf_usr_sp; break; default: ret = *((register_t*)&td->td_frame->tf_r0 + reg); break; } return (ret); } static u_int ptrace_get_usr_int(void* cookie, vm_offset_t offset, u_int* val) { struct thread *td = cookie; u_int error; error = ptrace_read_int(td, offset, val); return (error); } /** * This function parses current instruction opcode and decodes * any possible jump (change in PC) which might occur after * the instruction is executed. * * @param td Thread structure of analysed task * @param cur_instr Currently executed instruction * @param alt_next_address Pointer to the variable where * the destination address of the * jump instruction shall be stored. * * @return <0> when jump is possible * otherwise */ static int ptrace_get_alternative_next(struct thread *td, uint32_t cur_instr, uint32_t *alt_next_address) { int error; if (inst_branch(cur_instr) || inst_call(cur_instr) || inst_return(cur_instr)) { error = arm_predict_branch(td, cur_instr, td->td_frame->tf_pc, alt_next_address, ptrace_get_usr_reg, ptrace_get_usr_int); return (error); } return (EINVAL); } int ptrace_single_step(struct thread *td) { struct proc *p; int error, error_alt; uint32_t cur_instr, alt_next = 0; /* TODO: This needs to be updated for Thumb-2 */ if ((td->td_frame->tf_spsr & PSR_T) != 0) return (EINVAL); KASSERT(td->td_md.md_ptrace_instr == 0, ("Didn't clear single step")); KASSERT(td->td_md.md_ptrace_instr_alt == 0, ("Didn't clear alternative single step")); p = td->td_proc; PROC_UNLOCK(p); error = ptrace_read_int(td, td->td_frame->tf_pc, &cur_instr); if (error) goto out; error = ptrace_read_int(td, td->td_frame->tf_pc + INSN_SIZE, &td->td_md.md_ptrace_instr); if (error == 0) { error = ptrace_write_int(td, td->td_frame->tf_pc + INSN_SIZE, PTRACE_BREAKPOINT); if (error) { td->td_md.md_ptrace_instr = 0; } else { td->td_md.md_ptrace_addr = td->td_frame->tf_pc + INSN_SIZE; } } error_alt = ptrace_get_alternative_next(td, cur_instr, &alt_next); if (error_alt == 0) { error_alt = ptrace_read_int(td, alt_next, &td->td_md.md_ptrace_instr_alt); if (error_alt) { td->td_md.md_ptrace_instr_alt = 0; } else { error_alt = ptrace_write_int(td, alt_next, PTRACE_BREAKPOINT); if (error_alt) td->td_md.md_ptrace_instr_alt = 0; else td->td_md.md_ptrace_addr_alt = alt_next; } } out: PROC_LOCK(p); return ((error != 0) && (error_alt != 0)); } int ptrace_clear_single_step(struct thread *td) { struct proc *p; /* TODO: This needs to be updated for Thumb-2 */ if ((td->td_frame->tf_spsr & PSR_T) != 0) return (EINVAL); if (td->td_md.md_ptrace_instr != 0) { p = td->td_proc; PROC_UNLOCK(p); ptrace_write_int(td, td->td_md.md_ptrace_addr, td->td_md.md_ptrace_instr); PROC_LOCK(p); td->td_md.md_ptrace_instr = 0; } if (td->td_md.md_ptrace_instr_alt != 0) { p = td->td_proc; PROC_UNLOCK(p); ptrace_write_int(td, td->td_md.md_ptrace_addr_alt, td->td_md.md_ptrace_instr_alt); PROC_LOCK(p); td->td_md.md_ptrace_instr_alt = 0; } return (0); } int ptrace_set_pc(struct thread *td, unsigned long addr) { td->td_frame->tf_pc = addr; return (0); } void cpu_pcpu_init(struct pcpu *pcpu, int cpuid, size_t size) { } void spinlock_enter(void) { struct thread *td; register_t cspr; td = curthread; if (td->td_md.md_spinlock_count == 0) { cspr = disable_interrupts(PSR_I | PSR_F); td->td_md.md_spinlock_count = 1; td->td_md.md_saved_cspr = cspr; } else td->td_md.md_spinlock_count++; critical_enter(); } void spinlock_exit(void) { struct thread *td; register_t cspr; td = curthread; critical_exit(); cspr = td->td_md.md_saved_cspr; td->td_md.md_spinlock_count--; if (td->td_md.md_spinlock_count == 0) restore_interrupts(cspr); } /* * Clear registers on exec */ void exec_setregs(struct thread *td, struct image_params *imgp, u_long stack) { struct trapframe *tf = td->td_frame; memset(tf, 0, sizeof(*tf)); tf->tf_usr_sp = stack; tf->tf_usr_lr = imgp->entry_addr; tf->tf_svc_lr = 0x77777777; tf->tf_pc = imgp->entry_addr; tf->tf_spsr = PSR_USR32_MODE; } /* * Get machine context. */ int get_mcontext(struct thread *td, mcontext_t *mcp, int clear_ret) { struct trapframe *tf = td->td_frame; __greg_t *gr = mcp->__gregs; if (clear_ret & GET_MC_CLEAR_RET) { gr[_REG_R0] = 0; gr[_REG_CPSR] = tf->tf_spsr & ~PSR_C; } else { gr[_REG_R0] = tf->tf_r0; gr[_REG_CPSR] = tf->tf_spsr; } gr[_REG_R1] = tf->tf_r1; gr[_REG_R2] = tf->tf_r2; gr[_REG_R3] = tf->tf_r3; gr[_REG_R4] = tf->tf_r4; gr[_REG_R5] = tf->tf_r5; gr[_REG_R6] = tf->tf_r6; gr[_REG_R7] = tf->tf_r7; gr[_REG_R8] = tf->tf_r8; gr[_REG_R9] = tf->tf_r9; gr[_REG_R10] = tf->tf_r10; gr[_REG_R11] = tf->tf_r11; gr[_REG_R12] = tf->tf_r12; gr[_REG_SP] = tf->tf_usr_sp; gr[_REG_LR] = tf->tf_usr_lr; gr[_REG_PC] = tf->tf_pc; return (0); } /* * Set machine context. * * However, we don't set any but the user modifiable flags, and we won't * touch the cs selector. */ int set_mcontext(struct thread *td, mcontext_t *mcp) { struct trapframe *tf = td->td_frame; const __greg_t *gr = mcp->__gregs; tf->tf_r0 = gr[_REG_R0]; tf->tf_r1 = gr[_REG_R1]; tf->tf_r2 = gr[_REG_R2]; tf->tf_r3 = gr[_REG_R3]; tf->tf_r4 = gr[_REG_R4]; tf->tf_r5 = gr[_REG_R5]; tf->tf_r6 = gr[_REG_R6]; tf->tf_r7 = gr[_REG_R7]; tf->tf_r8 = gr[_REG_R8]; tf->tf_r9 = gr[_REG_R9]; tf->tf_r10 = gr[_REG_R10]; tf->tf_r11 = gr[_REG_R11]; tf->tf_r12 = gr[_REG_R12]; tf->tf_usr_sp = gr[_REG_SP]; tf->tf_usr_lr = gr[_REG_LR]; tf->tf_pc = gr[_REG_PC]; tf->tf_spsr = gr[_REG_CPSR]; return (0); } /* * MPSAFE */ int sys_sigreturn(td, uap) struct thread *td; struct sigreturn_args /* { const struct __ucontext *sigcntxp; } */ *uap; { ucontext_t uc; int spsr; if (uap == NULL) return (EFAULT); if (copyin(uap->sigcntxp, &uc, sizeof(uc))) return (EFAULT); /* * Make sure the processor mode has not been tampered with and * interrupts have not been disabled. */ spsr = uc.uc_mcontext.__gregs[_REG_CPSR]; if ((spsr & PSR_MODE) != PSR_USR32_MODE || (spsr & (PSR_I | PSR_F)) != 0) return (EINVAL); /* Restore register context. */ set_mcontext(td, &uc.uc_mcontext); /* Restore signal mask. */ kern_sigprocmask(td, SIG_SETMASK, &uc.uc_sigmask, NULL, 0); return (EJUSTRETURN); } /* * Construct a PCB from a trapframe. This is called from kdb_trap() where * we want to start a backtrace from the function that caused us to enter * the debugger. We have the context in the trapframe, but base the trace * on the PCB. The PCB doesn't have to be perfect, as long as it contains * enough for a backtrace. */ void makectx(struct trapframe *tf, struct pcb *pcb) { pcb->pcb_regs.sf_r4 = tf->tf_r4; pcb->pcb_regs.sf_r5 = tf->tf_r5; pcb->pcb_regs.sf_r6 = tf->tf_r6; pcb->pcb_regs.sf_r7 = tf->tf_r7; pcb->pcb_regs.sf_r8 = tf->tf_r8; pcb->pcb_regs.sf_r9 = tf->tf_r9; pcb->pcb_regs.sf_r10 = tf->tf_r10; pcb->pcb_regs.sf_r11 = tf->tf_r11; pcb->pcb_regs.sf_r12 = tf->tf_r12; pcb->pcb_regs.sf_pc = tf->tf_pc; pcb->pcb_regs.sf_lr = tf->tf_usr_lr; pcb->pcb_regs.sf_sp = tf->tf_usr_sp; } /* * Fake up a boot descriptor table */ vm_offset_t fake_preload_metadata(struct arm_boot_params *abp __unused) { #ifdef DDB vm_offset_t zstart = 0, zend = 0; #endif vm_offset_t lastaddr; int i = 0; static uint32_t fake_preload[35]; fake_preload[i++] = MODINFO_NAME; fake_preload[i++] = strlen("kernel") + 1; strcpy((char*)&fake_preload[i++], "kernel"); i += 1; fake_preload[i++] = MODINFO_TYPE; fake_preload[i++] = strlen("elf kernel") + 1; strcpy((char*)&fake_preload[i++], "elf kernel"); i += 2; fake_preload[i++] = MODINFO_ADDR; fake_preload[i++] = sizeof(vm_offset_t); fake_preload[i++] = KERNVIRTADDR; fake_preload[i++] = MODINFO_SIZE; fake_preload[i++] = sizeof(uint32_t); fake_preload[i++] = (uint32_t)&end - KERNVIRTADDR; #ifdef DDB if (*(uint32_t *)KERNVIRTADDR == MAGIC_TRAMP_NUMBER) { fake_preload[i++] = MODINFO_METADATA|MODINFOMD_SSYM; fake_preload[i++] = sizeof(vm_offset_t); fake_preload[i++] = *(uint32_t *)(KERNVIRTADDR + 4); fake_preload[i++] = MODINFO_METADATA|MODINFOMD_ESYM; fake_preload[i++] = sizeof(vm_offset_t); fake_preload[i++] = *(uint32_t *)(KERNVIRTADDR + 8); lastaddr = *(uint32_t *)(KERNVIRTADDR + 8); zend = lastaddr; zstart = *(uint32_t *)(KERNVIRTADDR + 4); db_fetch_ksymtab(zstart, zend); } else #endif lastaddr = (vm_offset_t)&end; fake_preload[i++] = 0; fake_preload[i] = 0; preload_metadata = (void *)fake_preload; init_static_kenv(NULL, 0); return (lastaddr); } void pcpu0_init(void) { #if __ARM_ARCH >= 6 set_curthread(&thread0); #endif pcpu_init(pcpup, 0, sizeof(struct pcpu)); PCPU_SET(curthread, &thread0); } #if defined(LINUX_BOOT_ABI) vm_offset_t linux_parse_boot_param(struct arm_boot_params *abp) { struct arm_lbabi_tag *walker; uint32_t revision; uint64_t serial; /* * Linux boot ABI: r0 = 0, r1 is the board type (!= 0) and r2 * is atags or dtb pointer. If all of these aren't satisfied, * then punt. */ if (!(abp->abp_r0 == 0 && abp->abp_r1 != 0 && abp->abp_r2 != 0)) return 0; board_id = abp->abp_r1; walker = (struct arm_lbabi_tag *) (abp->abp_r2 + KERNVIRTADDR - abp->abp_physaddr); /* xxx - Need to also look for binary device tree */ if (ATAG_TAG(walker) != ATAG_CORE) return 0; atag_list = walker; while (ATAG_TAG(walker) != ATAG_NONE) { switch (ATAG_TAG(walker)) { case ATAG_CORE: break; case ATAG_MEM: arm_physmem_hardware_region(walker->u.tag_mem.start, walker->u.tag_mem.size); break; case ATAG_INITRD2: break; case ATAG_SERIAL: serial = walker->u.tag_sn.low | ((uint64_t)walker->u.tag_sn.high << 32); board_set_serial(serial); break; case ATAG_REVISION: revision = walker->u.tag_rev.rev; board_set_revision(revision); break; case ATAG_CMDLINE: /* XXX open question: Parse this for boothowto? */ bcopy(walker->u.tag_cmd.command, linux_command_line, ATAG_SIZE(walker)); break; default: break; } walker = ATAG_NEXT(walker); } /* Save a copy for later */ bcopy(atag_list, atags, (char *)walker - (char *)atag_list + ATAG_SIZE(walker)); init_static_kenv(NULL, 0); return fake_preload_metadata(abp); } #endif #if defined(FREEBSD_BOOT_LOADER) vm_offset_t freebsd_parse_boot_param(struct arm_boot_params *abp) { vm_offset_t lastaddr = 0; void *mdp; void *kmdp; #ifdef DDB vm_offset_t ksym_start; vm_offset_t ksym_end; #endif /* * Mask metadata pointer: it is supposed to be on page boundary. If * the first argument (mdp) doesn't point to a valid address the * bootloader must have passed us something else than the metadata * ptr, so we give up. Also give up if we cannot find metadta section * the loader creates that we get all this data out of. */ if ((mdp = (void *)(abp->abp_r0 & ~PAGE_MASK)) == NULL) return 0; preload_metadata = mdp; kmdp = preload_search_by_type("elf kernel"); if (kmdp == NULL) return 0; boothowto = MD_FETCH(kmdp, MODINFOMD_HOWTO, int); loader_envp = MD_FETCH(kmdp, MODINFOMD_ENVP, char *); init_static_kenv(loader_envp, 0); lastaddr = MD_FETCH(kmdp, MODINFOMD_KERNEND, vm_offset_t); #ifdef DDB ksym_start = MD_FETCH(kmdp, MODINFOMD_SSYM, uintptr_t); ksym_end = MD_FETCH(kmdp, MODINFOMD_ESYM, uintptr_t); db_fetch_ksymtab(ksym_start, ksym_end); #endif return lastaddr; } #endif vm_offset_t default_parse_boot_param(struct arm_boot_params *abp) { vm_offset_t lastaddr; #if defined(LINUX_BOOT_ABI) if ((lastaddr = linux_parse_boot_param(abp)) != 0) return lastaddr; #endif #if defined(FREEBSD_BOOT_LOADER) if ((lastaddr = freebsd_parse_boot_param(abp)) != 0) return lastaddr; #endif /* Fall back to hardcoded metadata. */ lastaddr = fake_preload_metadata(abp); return lastaddr; } /* * Stub version of the boot parameter parsing routine. We are * called early in initarm, before even VM has been initialized. * This routine needs to preserve any data that the boot loader * has passed in before the kernel starts to grow past the end * of the BSS, traditionally the place boot-loaders put this data. * * Since this is called so early, things that depend on the vm system * being setup (including access to some SoC's serial ports), about * all that can be done in this routine is to copy the arguments. * * This is the default boot parameter parsing routine. Individual * kernels/boards can override this weak function with one of their * own. We just fake metadata... */ __weak_reference(default_parse_boot_param, parse_boot_param); /* * Initialize proc0 */ void init_proc0(vm_offset_t kstack) { proc_linkup0(&proc0, &thread0); thread0.td_kstack = kstack; thread0.td_pcb = (struct pcb *) (thread0.td_kstack + kstack_pages * PAGE_SIZE) - 1; thread0.td_pcb->pcb_flags = 0; thread0.td_pcb->pcb_vfpcpu = -1; thread0.td_pcb->pcb_vfpstate.fpscr = VFPSCR_DN; thread0.td_frame = &proc0_tf; pcpup->pc_curpcb = thread0.td_pcb; } int arm_predict_branch(void *cookie, u_int insn, register_t pc, register_t *new_pc, u_int (*fetch_reg)(void*, int), u_int (*read_int)(void*, vm_offset_t, u_int*)) { u_int addr, nregs, offset = 0; int error = 0; switch ((insn >> 24) & 0xf) { case 0x2: /* add pc, reg1, #value */ case 0x0: /* add pc, reg1, reg2, lsl #offset */ addr = fetch_reg(cookie, (insn >> 16) & 0xf); if (((insn >> 16) & 0xf) == 15) addr += 8; if (insn & 0x0200000) { offset = (insn >> 7) & 0x1e; offset = (insn & 0xff) << (32 - offset) | (insn & 0xff) >> offset; } else { offset = fetch_reg(cookie, insn & 0x0f); if ((insn & 0x0000ff0) != 0x00000000) { if (insn & 0x10) nregs = fetch_reg(cookie, (insn >> 8) & 0xf); else nregs = (insn >> 7) & 0x1f; switch ((insn >> 5) & 3) { case 0: /* lsl */ offset = offset << nregs; break; case 1: /* lsr */ offset = offset >> nregs; break; default: break; /* XXX */ } } *new_pc = addr + offset; return (0); } case 0xa: /* b ... */ case 0xb: /* bl ... */ addr = ((insn << 2) & 0x03ffffff); if (addr & 0x02000000) addr |= 0xfc000000; *new_pc = (pc + 8 + addr); return (0); case 0x7: /* ldr pc, [pc, reg, lsl #2] */ addr = fetch_reg(cookie, insn & 0xf); addr = pc + 8 + (addr << 2); error = read_int(cookie, addr, &addr); *new_pc = addr; return (error); case 0x1: /* mov pc, reg */ *new_pc = fetch_reg(cookie, insn & 0xf); return (0); case 0x4: case 0x5: /* ldr pc, [reg] */ addr = fetch_reg(cookie, (insn >> 16) & 0xf); /* ldr pc, [reg, #offset] */ if (insn & (1 << 24)) offset = insn & 0xfff; if (insn & 0x00800000) addr += offset; else addr -= offset; error = read_int(cookie, addr, &addr); *new_pc = addr; return (error); case 0x8: /* ldmxx reg, {..., pc} */ case 0x9: addr = fetch_reg(cookie, (insn >> 16) & 0xf); nregs = (insn & 0x5555) + ((insn >> 1) & 0x5555); nregs = (nregs & 0x3333) + ((nregs >> 2) & 0x3333); nregs = (nregs + (nregs >> 4)) & 0x0f0f; nregs = (nregs + (nregs >> 8)) & 0x001f; switch ((insn >> 23) & 0x3) { case 0x0: /* ldmda */ addr = addr - 0; break; case 0x1: /* ldmia */ addr = addr + 0 + ((nregs - 1) << 2); break; case 0x2: /* ldmdb */ addr = addr - 4; break; case 0x3: /* ldmib */ addr = addr + 4 + ((nregs - 1) << 2); break; } error = read_int(cookie, addr, &addr); *new_pc = addr; return (error); default: return (EINVAL); } } -#ifdef ARM_NEW_PMAP +#if __ARM_ARCH >= 6 void set_stackptrs(int cpu) { set_stackptr(PSR_IRQ32_MODE, irqstack + ((IRQ_STACK_SIZE * PAGE_SIZE) * (cpu + 1))); set_stackptr(PSR_ABT32_MODE, abtstack + ((ABT_STACK_SIZE * PAGE_SIZE) * (cpu + 1))); set_stackptr(PSR_UND32_MODE, undstack + ((UND_STACK_SIZE * PAGE_SIZE) * (cpu + 1))); } #else void set_stackptrs(int cpu) { set_stackptr(PSR_IRQ32_MODE, irqstack.pv_va + ((IRQ_STACK_SIZE * PAGE_SIZE) * (cpu + 1))); set_stackptr(PSR_ABT32_MODE, abtstack.pv_va + ((ABT_STACK_SIZE * PAGE_SIZE) * (cpu + 1))); set_stackptr(PSR_UND32_MODE, undstack.pv_va + ((UND_STACK_SIZE * PAGE_SIZE) * (cpu + 1))); } #endif #ifdef EFI #define efi_next_descriptor(ptr, size) \ ((struct efi_md *)(((uint8_t *) ptr) + size)) static void add_efi_map_entries(struct efi_map_header *efihdr, struct mem_region *mr, int *mrcnt, uint32_t *memsize) { struct efi_md *map, *p; const char *type; size_t efisz, memory_size; int ndesc, i, j; static const char *types[] = { "Reserved", "LoaderCode", "LoaderData", "BootServicesCode", "BootServicesData", "RuntimeServicesCode", "RuntimeServicesData", "ConventionalMemory", "UnusableMemory", "ACPIReclaimMemory", "ACPIMemoryNVS", "MemoryMappedIO", "MemoryMappedIOPortSpace", "PalCode" }; *mrcnt = 0; *memsize = 0; /* * Memory map data provided by UEFI via the GetMemoryMap * Boot Services API. */ efisz = roundup2(sizeof(struct efi_map_header), 0x10); map = (struct efi_md *)((uint8_t *)efihdr + efisz); if (efihdr->descriptor_size == 0) return; ndesc = efihdr->memory_size / efihdr->descriptor_size; if (boothowto & RB_VERBOSE) printf("%23s %12s %12s %8s %4s\n", "Type", "Physical", "Virtual", "#Pages", "Attr"); memory_size = 0; for (i = 0, j = 0, p = map; i < ndesc; i++, p = efi_next_descriptor(p, efihdr->descriptor_size)) { if (boothowto & RB_VERBOSE) { if (p->md_type <= EFI_MD_TYPE_PALCODE) type = types[p->md_type]; else type = ""; printf("%23s %012llx %12p %08llx ", type, p->md_phys, p->md_virt, p->md_pages); if (p->md_attr & EFI_MD_ATTR_UC) printf("UC "); if (p->md_attr & EFI_MD_ATTR_WC) printf("WC "); if (p->md_attr & EFI_MD_ATTR_WT) printf("WT "); if (p->md_attr & EFI_MD_ATTR_WB) printf("WB "); if (p->md_attr & EFI_MD_ATTR_UCE) printf("UCE "); if (p->md_attr & EFI_MD_ATTR_WP) printf("WP "); if (p->md_attr & EFI_MD_ATTR_RP) printf("RP "); if (p->md_attr & EFI_MD_ATTR_XP) printf("XP "); if (p->md_attr & EFI_MD_ATTR_RT) printf("RUNTIME"); printf("\n"); } switch (p->md_type) { case EFI_MD_TYPE_CODE: case EFI_MD_TYPE_DATA: case EFI_MD_TYPE_BS_CODE: case EFI_MD_TYPE_BS_DATA: case EFI_MD_TYPE_FREE: /* * We're allowed to use any entry with these types. */ break; default: continue; } j++; if (j >= FDT_MEM_REGIONS) break; mr[j].mr_start = p->md_phys; mr[j].mr_size = p->md_pages * PAGE_SIZE; memory_size += mr[j].mr_size; } *mrcnt = j; *memsize = memory_size; } #endif /* EFI */ #ifdef FDT static char * kenv_next(char *cp) { if (cp != NULL) { while (*cp != 0) cp++; cp++; if (*cp == 0) cp = NULL; } return (cp); } static void print_kenv(void) { char *cp; debugf("loader passed (static) kenv:\n"); if (loader_envp == NULL) { debugf(" no env, null ptr\n"); return; } debugf(" loader_envp = 0x%08x\n", (uint32_t)loader_envp); for (cp = loader_envp; cp != NULL; cp = kenv_next(cp)) debugf(" %x %s\n", (uint32_t)cp, cp); } -#ifndef ARM_NEW_PMAP +#if __ARM_ARCH < 6 void * initarm(struct arm_boot_params *abp) { struct mem_region mem_regions[FDT_MEM_REGIONS]; struct pv_addr kernel_l1pt; struct pv_addr dpcpu; vm_offset_t dtbp, freemempos, l2_start, lastaddr; uint32_t memsize, l2size; char *env; void *kmdp; u_int l1pagetable; int i, j, err_devmap, mem_regions_sz; lastaddr = parse_boot_param(abp); arm_physmem_kernaddr = abp->abp_physaddr; memsize = 0; cpuinfo_init(); set_cpufuncs(); /* * Find the dtb passed in by the boot loader. */ kmdp = preload_search_by_type("elf kernel"); if (kmdp != NULL) dtbp = MD_FETCH(kmdp, MODINFOMD_DTBP, vm_offset_t); else dtbp = (vm_offset_t)NULL; #if defined(FDT_DTB_STATIC) /* * In case the device tree blob was not retrieved (from metadata) try * to use the statically embedded one. */ if (dtbp == (vm_offset_t)NULL) dtbp = (vm_offset_t)&fdt_static_dtb; #endif if (OF_install(OFW_FDT, 0) == FALSE) panic("Cannot install FDT"); if (OF_init((void *)dtbp) != 0) panic("OF_init failed with the found device tree"); /* Grab physical memory regions information from device tree. */ if (fdt_get_mem_regions(mem_regions, &mem_regions_sz, &memsize) != 0) panic("Cannot get physical memory regions"); arm_physmem_hardware_regions(mem_regions, mem_regions_sz); /* Grab reserved memory regions information from device tree. */ if (fdt_get_reserved_regions(mem_regions, &mem_regions_sz) == 0) arm_physmem_exclude_regions(mem_regions, mem_regions_sz, EXFLAG_NODUMP | EXFLAG_NOALLOC); /* Platform-specific initialisation */ platform_probe_and_attach(); pcpu0_init(); /* Do basic tuning, hz etc */ init_param1(); /* Calculate number of L2 tables needed for mapping vm_page_array */ l2size = (memsize / PAGE_SIZE) * sizeof(struct vm_page); l2size = (l2size >> L1_S_SHIFT) + 1; /* * Add one table for end of kernel map, one for stacks, msgbuf and * L1 and L2 tables map and one for vectors map. */ l2size += 3; /* Make it divisible by 4 */ l2size = (l2size + 3) & ~3; freemempos = (lastaddr + PAGE_MASK) & ~PAGE_MASK; /* Define a macro to simplify memory allocation */ #define valloc_pages(var, np) \ alloc_pages((var).pv_va, (np)); \ (var).pv_pa = (var).pv_va + (abp->abp_physaddr - KERNVIRTADDR); #define alloc_pages(var, np) \ (var) = freemempos; \ freemempos += (np * PAGE_SIZE); \ memset((char *)(var), 0, ((np) * PAGE_SIZE)); while (((freemempos - L1_TABLE_SIZE) & (L1_TABLE_SIZE - 1)) != 0) freemempos += PAGE_SIZE; valloc_pages(kernel_l1pt, L1_TABLE_SIZE / PAGE_SIZE); for (i = 0, j = 0; i < l2size; ++i) { if (!(i % (PAGE_SIZE / L2_TABLE_SIZE_REAL))) { valloc_pages(kernel_pt_table[i], L2_TABLE_SIZE / PAGE_SIZE); j = i; } else { kernel_pt_table[i].pv_va = kernel_pt_table[j].pv_va + L2_TABLE_SIZE_REAL * (i - j); kernel_pt_table[i].pv_pa = kernel_pt_table[i].pv_va - KERNVIRTADDR + abp->abp_physaddr; } } /* * Allocate a page for the system page mapped to 0x00000000 * or 0xffff0000. This page will just contain the system vectors * and can be shared by all processes. */ valloc_pages(systempage, 1); /* Allocate dynamic per-cpu area. */ valloc_pages(dpcpu, DPCPU_SIZE / PAGE_SIZE); dpcpu_init((void *)dpcpu.pv_va, 0); /* Allocate stacks for all modes */ valloc_pages(irqstack, IRQ_STACK_SIZE * MAXCPU); valloc_pages(abtstack, ABT_STACK_SIZE * MAXCPU); valloc_pages(undstack, UND_STACK_SIZE * MAXCPU); valloc_pages(kernelstack, kstack_pages * MAXCPU); valloc_pages(msgbufpv, round_page(msgbufsize) / PAGE_SIZE); /* * Now we start construction of the L1 page table * We start by mapping the L2 page tables into the L1. * This means that we can replace L1 mappings later on if necessary */ l1pagetable = kernel_l1pt.pv_va; /* * Try to map as much as possible of kernel text and data using * 1MB section mapping and for the rest of initial kernel address * space use L2 coarse tables. * * Link L2 tables for mapping remainder of kernel (modulo 1MB) * and kernel structures */ l2_start = lastaddr & ~(L1_S_OFFSET); for (i = 0 ; i < l2size - 1; i++) pmap_link_l2pt(l1pagetable, l2_start + i * L1_S_SIZE, &kernel_pt_table[i]); pmap_curmaxkvaddr = l2_start + (l2size - 1) * L1_S_SIZE; /* Map kernel code and data */ pmap_map_chunk(l1pagetable, KERNVIRTADDR, abp->abp_physaddr, (((uint32_t)(lastaddr) - KERNVIRTADDR) + PAGE_MASK) & ~PAGE_MASK, VM_PROT_READ|VM_PROT_WRITE, PTE_CACHE); /* Map L1 directory and allocated L2 page tables */ pmap_map_chunk(l1pagetable, kernel_l1pt.pv_va, kernel_l1pt.pv_pa, L1_TABLE_SIZE, VM_PROT_READ|VM_PROT_WRITE, PTE_PAGETABLE); pmap_map_chunk(l1pagetable, kernel_pt_table[0].pv_va, kernel_pt_table[0].pv_pa, L2_TABLE_SIZE_REAL * l2size, VM_PROT_READ|VM_PROT_WRITE, PTE_PAGETABLE); /* Map allocated DPCPU, stacks and msgbuf */ pmap_map_chunk(l1pagetable, dpcpu.pv_va, dpcpu.pv_pa, freemempos - dpcpu.pv_va, VM_PROT_READ|VM_PROT_WRITE, PTE_CACHE); /* Link and map the vector page */ pmap_link_l2pt(l1pagetable, ARM_VECTORS_HIGH, &kernel_pt_table[l2size - 1]); pmap_map_entry(l1pagetable, ARM_VECTORS_HIGH, systempage.pv_pa, VM_PROT_READ|VM_PROT_WRITE|VM_PROT_EXECUTE, PTE_CACHE); /* Establish static device mappings. */ err_devmap = platform_devmap_init(); arm_devmap_bootstrap(l1pagetable, NULL); vm_max_kernel_address = platform_lastaddr(); cpu_domains((DOMAIN_CLIENT << (PMAP_DOMAIN_KERNEL * 2)) | DOMAIN_CLIENT); pmap_pa = kernel_l1pt.pv_pa; setttb(kernel_l1pt.pv_pa); cpu_tlb_flushID(); cpu_domains(DOMAIN_CLIENT << (PMAP_DOMAIN_KERNEL * 2)); /* * Now that proper page tables are installed, call cpu_setup() to enable * instruction and data caches and other chip-specific features. */ cpu_setup(); /* * Only after the SOC registers block is mapped we can perform device * tree fixups, as they may attempt to read parameters from hardware. */ OF_interpret("perform-fixup", 0); platform_gpio_init(); cninit(); debugf("initarm: console initialized\n"); debugf(" arg1 kmdp = 0x%08x\n", (uint32_t)kmdp); debugf(" boothowto = 0x%08x\n", boothowto); debugf(" dtbp = 0x%08x\n", (uint32_t)dtbp); print_kenv(); env = kern_getenv("kernelname"); if (env != NULL) { strlcpy(kernelname, env, sizeof(kernelname)); freeenv(env); } if (err_devmap != 0) printf("WARNING: could not fully configure devmap, error=%d\n", err_devmap); platform_late_init(); /* * Pages were allocated during the secondary bootstrap for the * stacks for different CPU modes. * We must now set the r13 registers in the different CPU modes to * point to these stacks. * Since the ARM stacks use STMFD etc. we must set r13 to the top end * of the stack memory. */ cpu_control(CPU_CONTROL_MMU_ENABLE, CPU_CONTROL_MMU_ENABLE); set_stackptrs(0); /* * We must now clean the cache again.... * Cleaning may be done by reading new data to displace any * dirty data in the cache. This will have happened in setttb() * but since we are boot strapping the addresses used for the read * may have just been remapped and thus the cache could be out * of sync. A re-clean after the switch will cure this. * After booting there are no gross relocations of the kernel thus * this problem will not occur after initarm(). */ cpu_idcache_wbinv_all(); undefined_init(); init_proc0(kernelstack.pv_va); arm_vector_init(ARM_VECTORS_HIGH, ARM_VEC_ALL); pmap_bootstrap(freemempos, &kernel_l1pt); msgbufp = (void *)msgbufpv.pv_va; msgbufinit(msgbufp, msgbufsize); mutex_init(); /* * Exclude the kernel (and all the things we allocated which immediately * follow the kernel) from the VM allocation pool but not from crash * dumps. virtual_avail is a global variable which tracks the kva we've * "allocated" while setting up pmaps. * * Prepare the list of physical memory available to the vm subsystem. */ arm_physmem_exclude_region(abp->abp_physaddr, (virtual_avail - KERNVIRTADDR), EXFLAG_NOALLOC); arm_physmem_init_kernel_globals(); init_param2(physmem); dbg_monitor_init(); kdb_init(); return ((void *)(kernelstack.pv_va + USPACE_SVC_STACK_TOP - sizeof(struct pcb))); } -#else /* !ARM_NEW_PMAP */ +#else /* __ARM_ARCH < 6 */ void * initarm(struct arm_boot_params *abp) { struct mem_region mem_regions[FDT_MEM_REGIONS]; vm_paddr_t lastaddr; vm_offset_t dtbp, kernelstack, dpcpu; uint32_t memsize; char *env; void *kmdp; int err_devmap, mem_regions_sz; #ifdef EFI struct efi_map_header *efihdr; #endif /* get last allocated physical address */ arm_physmem_kernaddr = abp->abp_physaddr; lastaddr = parse_boot_param(abp) - KERNVIRTADDR + arm_physmem_kernaddr; memsize = 0; set_cpufuncs(); cpuinfo_init(); /* * Find the dtb passed in by the boot loader. */ kmdp = preload_search_by_type("elf kernel"); dtbp = MD_FETCH(kmdp, MODINFOMD_DTBP, vm_offset_t); #if defined(FDT_DTB_STATIC) /* * In case the device tree blob was not retrieved (from metadata) try * to use the statically embedded one. */ if (dtbp == (vm_offset_t)NULL) dtbp = (vm_offset_t)&fdt_static_dtb; #endif if (OF_install(OFW_FDT, 0) == FALSE) panic("Cannot install FDT"); if (OF_init((void *)dtbp) != 0) panic("OF_init failed with the found device tree"); #ifdef EFI efihdr = (struct efi_map_header *)preload_search_info(kmdp, MODINFO_METADATA | MODINFOMD_EFI_MAP); if (efihdr != NULL) { add_efi_map_entries(efihdr, mem_regions, &mem_regions_sz, &memsize); } else #endif { /* Grab physical memory regions information from device tree. */ if (fdt_get_mem_regions(mem_regions, &mem_regions_sz, &memsize) != 0) panic("Cannot get physical memory regions"); } arm_physmem_hardware_regions(mem_regions, mem_regions_sz); /* Grab reserved memory regions information from device tree. */ if (fdt_get_reserved_regions(mem_regions, &mem_regions_sz) == 0) arm_physmem_exclude_regions(mem_regions, mem_regions_sz, EXFLAG_NODUMP | EXFLAG_NOALLOC); /* * Set TEX remapping registers. * Setup kernel page tables and switch to kernel L1 page table. */ pmap_set_tex(); pmap_bootstrap_prepare(lastaddr); /* * Now that proper page tables are installed, call cpu_setup() to enable * instruction and data caches and other chip-specific features. */ cpu_setup(); /* Platform-specific initialisation */ platform_probe_and_attach(); pcpu0_init(); /* Do basic tuning, hz etc */ init_param1(); /* * Allocate a page for the system page mapped to 0xffff0000 * This page will just contain the system vectors and can be * shared by all processes. */ systempage = pmap_preboot_get_pages(1); /* Map the vector page. */ pmap_preboot_map_pages(systempage, ARM_VECTORS_HIGH, 1); if (virtual_end >= ARM_VECTORS_HIGH) virtual_end = ARM_VECTORS_HIGH - 1; /* Allocate dynamic per-cpu area. */ dpcpu = pmap_preboot_get_vpages(DPCPU_SIZE / PAGE_SIZE); dpcpu_init((void *)dpcpu, 0); /* Allocate stacks for all modes */ irqstack = pmap_preboot_get_vpages(IRQ_STACK_SIZE * MAXCPU); abtstack = pmap_preboot_get_vpages(ABT_STACK_SIZE * MAXCPU); undstack = pmap_preboot_get_vpages(UND_STACK_SIZE * MAXCPU ); kernelstack = pmap_preboot_get_vpages(kstack_pages * MAXCPU); /* Allocate message buffer. */ msgbufp = (void *)pmap_preboot_get_vpages( round_page(msgbufsize) / PAGE_SIZE); /* * Pages were allocated during the secondary bootstrap for the * stacks for different CPU modes. * We must now set the r13 registers in the different CPU modes to * point to these stacks. * Since the ARM stacks use STMFD etc. we must set r13 to the top end * of the stack memory. */ set_stackptrs(0); mutex_init(); /* Establish static device mappings. */ err_devmap = platform_devmap_init(); arm_devmap_bootstrap(0, NULL); vm_max_kernel_address = platform_lastaddr(); /* * Only after the SOC registers block is mapped we can perform device * tree fixups, as they may attempt to read parameters from hardware. */ OF_interpret("perform-fixup", 0); platform_gpio_init(); cninit(); debugf("initarm: console initialized\n"); debugf(" arg1 kmdp = 0x%08x\n", (uint32_t)kmdp); debugf(" boothowto = 0x%08x\n", boothowto); debugf(" dtbp = 0x%08x\n", (uint32_t)dtbp); debugf(" lastaddr1: 0x%08x\n", lastaddr); print_kenv(); env = kern_getenv("kernelname"); if (env != NULL) strlcpy(kernelname, env, sizeof(kernelname)); if (err_devmap != 0) printf("WARNING: could not fully configure devmap, error=%d\n", err_devmap); platform_late_init(); /* * We must now clean the cache again.... * Cleaning may be done by reading new data to displace any * dirty data in the cache. This will have happened in setttb() * but since we are boot strapping the addresses used for the read * may have just been remapped and thus the cache could be out * of sync. A re-clean after the switch will cure this. * After booting there are no gross relocations of the kernel thus * this problem will not occur after initarm(). */ /* Set stack for exception handlers */ undefined_init(); init_proc0(kernelstack); arm_vector_init(ARM_VECTORS_HIGH, ARM_VEC_ALL); enable_interrupts(PSR_A); pmap_bootstrap(0); /* Exclude the kernel (and all the things we allocated which immediately * follow the kernel) from the VM allocation pool but not from crash * dumps. virtual_avail is a global variable which tracks the kva we've * "allocated" while setting up pmaps. * * Prepare the list of physical memory available to the vm subsystem. */ arm_physmem_exclude_region(abp->abp_physaddr, pmap_preboot_get_pages(0) - abp->abp_physaddr, EXFLAG_NOALLOC); arm_physmem_init_kernel_globals(); init_param2(physmem); /* Init message buffer. */ msgbufinit(msgbufp, msgbufsize); dbg_monitor_init(); kdb_init(); return ((void *)STACKALIGN(thread0.td_pcb)); } -#endif /* !ARM_NEW_PMAP */ +#endif /* __ARM_ARCH < 6 */ #endif /* FDT */ uint32_t (*arm_cpu_fill_vdso_timehands)(struct vdso_timehands *, struct timecounter *); uint32_t cpu_fill_vdso_timehands(struct vdso_timehands *vdso_th, struct timecounter *tc) { return (arm_cpu_fill_vdso_timehands != NULL ? arm_cpu_fill_vdso_timehands(vdso_th, tc) : 0); } Index: user/ngie/make_check/sys/arm/arm/mem.c =================================================================== --- user/ngie/make_check/sys/arm/arm/mem.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/mem.c (revision 295078) @@ -1,170 +1,171 @@ /*- * Copyright (c) 1988 University of Utah. * Copyright (c) 1982, 1986, 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department, and code derived from software contributed to * Berkeley by William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: Utah $Hdr: mem.c 1.13 89/10/08$ * from: @(#)mem.c 7.2 (Berkeley) 5/9/91 */ #include __FBSDID("$FreeBSD$"); /* * Memory special file */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include +#include #include #include /* * Used in /dev/mem drivers and elsewhere */ MALLOC_DEFINE(M_MEMDESC, "memdesc", "memory range descriptors"); struct mem_range_softc mem_range_softc; static struct sx tmppt_lock; SX_SYSINIT(tmppt, &tmppt_lock, "mem4map"); /* ARGSUSED */ int memrw(struct cdev *dev, struct uio *uio, int flags) { int o; u_int c = 0, v; struct iovec *iov; int error = 0; vm_offset_t addr, eaddr; while (uio->uio_resid > 0 && error == 0) { iov = uio->uio_iov; if (iov->iov_len == 0) { uio->uio_iov++; uio->uio_iovcnt--; if (uio->uio_iovcnt < 0) panic("memrw"); continue; } if (dev2unit(dev) == CDEV_MINOR_MEM) { int i; int address_valid = 0; v = uio->uio_offset; v &= ~PAGE_MASK; for (i = 0; dump_avail[i] || dump_avail[i + 1]; i += 2) { if (v >= dump_avail[i] && v < dump_avail[i + 1]) { address_valid = 1; break; } } if (!address_valid) return (EINVAL); sx_xlock(&tmppt_lock); pmap_kenter((vm_offset_t)_tmppt, v); -#ifdef ARM_NEW_PMAP +#if __ARM_ARCH >= 6 pmap_tlb_flush(kernel_pmap, (vm_offset_t)_tmppt); #endif o = (int)uio->uio_offset & PAGE_MASK; c = (u_int)(PAGE_SIZE - ((int)iov->iov_base & PAGE_MASK)); c = min(c, (u_int)(PAGE_SIZE - o)); c = min(c, (u_int)iov->iov_len); error = uiomove((caddr_t)&_tmppt[o], (int)c, uio); pmap_qremove((vm_offset_t)_tmppt, 1); sx_xunlock(&tmppt_lock); continue; } else if (dev2unit(dev) == CDEV_MINOR_KMEM) { c = iov->iov_len; /* * Make sure that all of the pages are currently * resident so that we don't create any zero-fill * pages. */ addr = trunc_page(uio->uio_offset); eaddr = round_page(uio->uio_offset + c); for (; addr < eaddr; addr += PAGE_SIZE) if (pmap_extract(kernel_pmap, addr) == 0) return (EFAULT); if (!kernacc((caddr_t)(int)uio->uio_offset, c, uio->uio_rw == UIO_READ ? VM_PROT_READ : VM_PROT_WRITE)) return (EFAULT); error = uiomove((caddr_t)(int)uio->uio_offset, (int)c, uio); continue; } /* else panic! */ } return (error); } /* * allow user processes to MMAP some memory sections * instead of going through read/write */ /* ARGSUSED */ int memmmap(struct cdev *dev, vm_ooffset_t offset, vm_paddr_t *paddr, int prot __unused, vm_memattr_t *memattr __unused) { if (dev2unit(dev) == CDEV_MINOR_MEM) *paddr = offset; else if (dev2unit(dev) == CDEV_MINOR_KMEM) *paddr = vtophys(offset); /* else panic! */ return (0); } Index: user/ngie/make_check/sys/arm/arm/mp_machdep.c =================================================================== --- user/ngie/make_check/sys/arm/arm/mp_machdep.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/mp_machdep.c (revision 295078) @@ -1,556 +1,528 @@ /*- * Copyright (c) 2011 Semihalf. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include +#include #include #include #include #include #include #include #include #include #include #include #include #ifdef VFP #include #endif #ifdef CPU_MV_PJ4B #include #include #endif #include "opt_smp.h" extern struct pcpu __pcpu[]; /* used to hold the AP's until we are ready to release them */ struct mtx ap_boot_mtx; struct pcb stoppcbs[MAXCPU]; /* # of Applications processors */ volatile int mp_naps; /* Set to 1 once we're ready to let the APs out of the pen. */ volatile int aps_ready = 0; #ifndef ARM_INTRNG static int ipi_handler(void *arg); #endif void set_stackptrs(int cpu); /* Temporary variables for init_secondary() */ void *dpcpu[MAXCPU - 1]; /* Determine if we running MP machine */ int cpu_mp_probe(void) { CPU_SETOF(0, &all_cpus); return (platform_mp_probe()); } /* Start Application Processor via platform specific function */ static int check_ap(void) { uint32_t ms; for (ms = 0; ms < 2000; ++ms) { if ((mp_naps + 1) == mp_ncpus) return (0); /* success */ else DELAY(1000); } return (-2); } extern unsigned char _end[]; /* Initialize and fire up non-boot processors */ void cpu_mp_start(void) { int error, i; mtx_init(&ap_boot_mtx, "ap boot", NULL, MTX_SPIN); /* Reserve memory for application processors */ for(i = 0; i < (mp_ncpus - 1); i++) dpcpu[i] = (void *)kmem_malloc(kernel_arena, DPCPU_SIZE, M_WAITOK | M_ZERO); cpu_idcache_wbinv_all(); cpu_l2cache_wbinv_all(); cpu_idcache_wbinv_all(); /* Initialize boot code and start up processors */ platform_mp_start_ap(); /* Check if ap's started properly */ error = check_ap(); if (error) printf("WARNING: Some AP's failed to start\n"); else for (i = 1; i < mp_ncpus; i++) CPU_SET(i, &all_cpus); } /* Introduce rest of cores to the world */ void cpu_mp_announce(void) { } extern vm_paddr_t pmap_pa; void init_secondary(int cpu) { struct pcpu *pc; uint32_t loop_counter; #ifndef ARM_INTRNG int start = 0, end = 0; #endif -#ifdef ARM_NEW_PMAP uint32_t actlr_mask, actlr_set; pmap_set_tex(); cpuinfo_get_actlr_modifier(&actlr_mask, &actlr_set); reinit_mmu(pmap_kern_ttb, actlr_mask, actlr_set); cpu_setup(); /* Provide stack pointers for other processor modes. */ set_stackptrs(cpu); enable_interrupts(PSR_A); -#else /* ARM_NEW_PMAP */ - cpu_setup(); - setttb(pmap_pa); - cpu_tlb_flushID(); -#endif /* ARM_NEW_PMAP */ pc = &__pcpu[cpu]; /* * pcpu_init() updates queue, so it should not be executed in parallel * on several cores */ while(mp_naps < (cpu - 1)) ; pcpu_init(pc, cpu, sizeof(struct pcpu)); dpcpu_init(dpcpu[cpu - 1], cpu); -#ifndef ARM_NEW_PMAP - /* Provide stack pointers for other processor modes. */ - set_stackptrs(cpu); -#endif /* Signal our startup to BSP */ atomic_add_rel_32(&mp_naps, 1); /* Spin until the BSP releases the APs */ while (!atomic_load_acq_int(&aps_ready)) { #if __ARM_ARCH >= 7 __asm __volatile("wfe"); #endif } /* Initialize curthread */ KASSERT(PCPU_GET(idlethread) != NULL, ("no idle thread")); pc->pc_curthread = pc->pc_idlethread; pc->pc_curpcb = pc->pc_idlethread->td_pcb; set_curthread(pc->pc_idlethread); #ifdef VFP vfp_init(); #endif mtx_lock_spin(&ap_boot_mtx); atomic_add_rel_32(&smp_cpus, 1); if (smp_cpus == mp_ncpus) { /* enable IPI's, tlb shootdown, freezes etc */ atomic_store_rel_int(&smp_started, 1); } mtx_unlock_spin(&ap_boot_mtx); #ifndef ARM_INTRNG /* Enable ipi */ #ifdef IPI_IRQ_START start = IPI_IRQ_START; #ifdef IPI_IRQ_END end = IPI_IRQ_END; #else end = IPI_IRQ_START; #endif #endif for (int i = start; i <= end; i++) arm_unmask_irq(i); #endif /* INTRNG */ enable_interrupts(PSR_I); loop_counter = 0; while (smp_started == 0) { DELAY(100); loop_counter++; if (loop_counter == 1000) CTR0(KTR_SMP, "AP still wait for smp_started"); } /* Start per-CPU event timers. */ cpu_initclocks_ap(); CTR0(KTR_SMP, "go into scheduler"); platform_mp_init_secondary(); /* Enter the scheduler */ sched_throw(NULL); panic("scheduler returned us to %s", __func__); /* NOTREACHED */ } #ifdef ARM_INTRNG static void ipi_rendezvous(void *dummy __unused) { CTR0(KTR_SMP, "IPI_RENDEZVOUS"); smp_rendezvous_action(); } static void ipi_ast(void *dummy __unused) { CTR0(KTR_SMP, "IPI_AST"); } static void ipi_stop(void *dummy __unused) { u_int cpu; /* * IPI_STOP_HARD is mapped to IPI_STOP. */ CTR0(KTR_SMP, "IPI_STOP or IPI_STOP_HARD"); cpu = PCPU_GET(cpuid); savectx(&stoppcbs[cpu]); /* * CPUs are stopped when entering the debugger and at * system shutdown, both events which can precede a * panic dump. For the dump to be correct, all caches * must be flushed and invalidated, but on ARM there's * no way to broadcast a wbinv_all to other cores. * Instead, we have each core do the local wbinv_all as * part of stopping the core. The core requesting the * stop will do the l2 cache flush after all other cores * have done their l1 flushes and stopped. */ cpu_idcache_wbinv_all(); /* Indicate we are stopped */ CPU_SET_ATOMIC(cpu, &stopped_cpus); /* Wait for restart */ while (!CPU_ISSET(cpu, &started_cpus)) cpu_spinwait(); CPU_CLR_ATOMIC(cpu, &started_cpus); CPU_CLR_ATOMIC(cpu, &stopped_cpus); #ifdef DDB dbg_resume_dbreg(); #endif CTR0(KTR_SMP, "IPI_STOP (restart)"); } static void ipi_preempt(void *arg) { struct trapframe *oldframe; struct thread *td; critical_enter(); td = curthread; td->td_intr_nesting_level++; oldframe = td->td_intr_frame; td->td_intr_frame = (struct trapframe *)arg; CTR1(KTR_SMP, "%s: IPI_PREEMPT", __func__); sched_preempt(td); td->td_intr_frame = oldframe; td->td_intr_nesting_level--; critical_exit(); } static void ipi_hardclock(void *arg) { struct trapframe *oldframe; struct thread *td; critical_enter(); td = curthread; td->td_intr_nesting_level++; oldframe = td->td_intr_frame; td->td_intr_frame = (struct trapframe *)arg; CTR1(KTR_SMP, "%s: IPI_HARDCLOCK", __func__); hardclockintr(); td->td_intr_frame = oldframe; td->td_intr_nesting_level--; critical_exit(); } -static void -ipi_tlb(void *dummy __unused) -{ - - CTR1(KTR_SMP, "%s: IPI_TLB", __func__); - cpufuncs.cf_tlb_flushID(); -} #else static int ipi_handler(void *arg) { u_int cpu, ipi; cpu = PCPU_GET(cpuid); ipi = pic_ipi_read((int)arg); while ((ipi != 0x3ff)) { switch (ipi) { case IPI_RENDEZVOUS: CTR0(KTR_SMP, "IPI_RENDEZVOUS"); smp_rendezvous_action(); break; case IPI_AST: CTR0(KTR_SMP, "IPI_AST"); break; case IPI_STOP: /* * IPI_STOP_HARD is mapped to IPI_STOP so it is not * necessary to add it in the switch. */ CTR0(KTR_SMP, "IPI_STOP or IPI_STOP_HARD"); savectx(&stoppcbs[cpu]); /* * CPUs are stopped when entering the debugger and at * system shutdown, both events which can precede a * panic dump. For the dump to be correct, all caches * must be flushed and invalidated, but on ARM there's * no way to broadcast a wbinv_all to other cores. * Instead, we have each core do the local wbinv_all as * part of stopping the core. The core requesting the * stop will do the l2 cache flush after all other cores * have done their l1 flushes and stopped. */ cpu_idcache_wbinv_all(); /* Indicate we are stopped */ CPU_SET_ATOMIC(cpu, &stopped_cpus); /* Wait for restart */ while (!CPU_ISSET(cpu, &started_cpus)) cpu_spinwait(); CPU_CLR_ATOMIC(cpu, &started_cpus); CPU_CLR_ATOMIC(cpu, &stopped_cpus); #ifdef DDB dbg_resume_dbreg(); #endif CTR0(KTR_SMP, "IPI_STOP (restart)"); break; case IPI_PREEMPT: CTR1(KTR_SMP, "%s: IPI_PREEMPT", __func__); sched_preempt(curthread); break; case IPI_HARDCLOCK: CTR1(KTR_SMP, "%s: IPI_HARDCLOCK", __func__); hardclockintr(); break; - case IPI_TLB: - CTR1(KTR_SMP, "%s: IPI_TLB", __func__); - cpufuncs.cf_tlb_flushID(); - break; default: panic("Unknown IPI 0x%0x on cpu %d", ipi, curcpu); } pic_ipi_clear(ipi); ipi = pic_ipi_read(-1); } return (FILTER_HANDLED); } #endif static void release_aps(void *dummy __unused) { uint32_t loop_counter; #ifndef ARM_INTRNG int start = 0, end = 0; #endif if (mp_ncpus == 1) return; #ifdef ARM_INTRNG intr_ipi_set_handler(IPI_RENDEZVOUS, "rendezvous", ipi_rendezvous, NULL, 0); intr_ipi_set_handler(IPI_AST, "ast", ipi_ast, NULL, 0); intr_ipi_set_handler(IPI_STOP, "stop", ipi_stop, NULL, 0); intr_ipi_set_handler(IPI_PREEMPT, "preempt", ipi_preempt, NULL, 0); intr_ipi_set_handler(IPI_HARDCLOCK, "hardclock", ipi_hardclock, NULL, 0); - intr_ipi_set_handler(IPI_TLB, "tlb", ipi_tlb, NULL, 0); #else #ifdef IPI_IRQ_START start = IPI_IRQ_START; #ifdef IPI_IRQ_END end = IPI_IRQ_END; #else end = IPI_IRQ_START; #endif #endif for (int i = start; i <= end; i++) { /* * IPI handler */ /* * Use 0xdeadbeef as the argument value for irq 0, * if we used 0, the intr code will give the trap frame * pointer instead. */ arm_setup_irqhandler("ipi", ipi_handler, NULL, (void *)i, i, INTR_TYPE_MISC | INTR_EXCL, NULL); /* Enable ipi */ arm_unmask_irq(i); } #endif atomic_store_rel_int(&aps_ready, 1); /* Wake the other threads up */ #if __ARM_ARCH >= 7 armv7_sev(); #endif printf("Release APs\n"); for (loop_counter = 0; loop_counter < 2000; loop_counter++) { if (smp_started) return; DELAY(1000); } printf("AP's not started\n"); } SYSINIT(start_aps, SI_SUB_SMP, SI_ORDER_FIRST, release_aps, NULL); struct cpu_group * cpu_topo(void) { return (smp_topo_1level(CG_SHARE_L2, mp_ncpus, 0)); } void cpu_mp_setmaxid(void) { platform_mp_setmaxid(); } /* Sending IPI */ void ipi_all_but_self(u_int ipi) { cpuset_t other_cpus; other_cpus = all_cpus; CPU_CLR(PCPU_GET(cpuid), &other_cpus); CTR2(KTR_SMP, "%s: ipi: %x", __func__, ipi); platform_ipi_send(other_cpus, ipi); } void ipi_cpu(int cpu, u_int ipi) { cpuset_t cpus; CPU_ZERO(&cpus); CPU_SET(cpu, &cpus); CTR3(KTR_SMP, "%s: cpu: %d, ipi: %x", __func__, cpu, ipi); platform_ipi_send(cpus, ipi); } void ipi_selected(cpuset_t cpus, u_int ipi) { CTR2(KTR_SMP, "%s: ipi: %x", __func__, ipi); platform_ipi_send(cpus, ipi); } -void -tlb_broadcast(int ipi) -{ - - if (smp_started) - ipi_all_but_self(ipi); -} Index: user/ngie/make_check/sys/arm/arm/pmap-v6.c =================================================================== --- user/ngie/make_check/sys/arm/arm/pmap-v6.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/pmap-v6.c (revision 295078) @@ -1,5452 +1,6629 @@ -/* From: $NetBSD: pmap.c,v 1.148 2004/04/03 04:35:48 bsh Exp $ */ /*- - * Copyright 2011 Semihalf - * Copyright 2004 Olivier Houchard. - * Copyright 2003 Wasabi Systems, Inc. + * Copyright (c) 1991 Regents of the University of California. + * Copyright (c) 1994 John S. Dyson + * Copyright (c) 1994 David Greenman + * Copyright (c) 2005-2010 Alan L. Cox + * Copyright (c) 2014 Svatopluk Kraus + * Copyright (c) 2014 Michal Meloun * All rights reserved. * - * Written by Steve C. Woodford for Wasabi Systems, Inc. + * This code is derived from software contributed to Berkeley by + * the Systems Programming Group of the University of Utah Computer + * Science Department and William Jolitz of UUNET Technologies Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. - * 3. All advertising materials mentioning features or use of this software - * must display the following acknowledgement: - * This product includes software developed for the NetBSD Project by - * Wasabi Systems, Inc. - * 4. The name of Wasabi Systems, Inc. may not be used to endorse - * or promote products derived from this software without specific prior - * written permission. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. * - * THIS SOFTWARE IS PROVIDED BY WASABI SYSTEMS, INC. ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASABI SYSTEMS, INC - * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. - * - * From: FreeBSD: src/sys/arm/arm/pmap.c,v 1.113 2009/07/24 13:50:29 - */ - -/*- - * Copyright (c) 2002-2003 Wasabi Systems, Inc. - * Copyright (c) 2001 Richard Earnshaw - * Copyright (c) 2001-2002 Christopher Gilbert - * All rights reserved. - * - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * 3. The name of the company nor the name of the author may be used to - * endorse or promote products derived from this software without specific - * prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED - * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF - * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. - * IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, - * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES - * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR - * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. + * + * from: @(#)pmap.c 7.7 (Berkeley) 5/12/91 */ /*- - * Copyright (c) 1999 The NetBSD Foundation, Inc. + * Copyright (c) 2003 Networks Associates Technology, Inc. * All rights reserved. * - * This code is derived from software contributed to The NetBSD Foundation - * by Charles M. Hannum. + * This software was developed for the FreeBSD Project by Jake Burkholder, + * Safeport Network Services, and Network Associates Laboratories, the + * Security Research Division of Network Associates, Inc. under + * DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA + * CHATS research program. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * - * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS - * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED - * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR - * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS - * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. */ -/*- - * Copyright (c) 1994-1998 Mark Brinicombe. - * Copyright (c) 1994 Brini. - * All rights reserved. +#include +__FBSDID("$FreeBSD$"); + +/* + * Manages physical address maps. * - * This code is derived from software written for Brini by Mark Brinicombe + * Since the information managed by this module is + * also stored by the logical address mapping module, + * this module may throw away valid virtual-to-physical + * mappings at almost any time. However, invalidations + * of virtual-to-physical mappings must be done as + * requested. * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * 3. All advertising materials mentioning features or use of this software - * must display the following acknowledgement: - * This product includes software developed by Mark Brinicombe. - * 4. The name of the author may not be used to endorse or promote products - * derived from this software without specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR - * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES - * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. - * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT - * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, - * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY - * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT - * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF - * - * RiscBSD kernel project - * - * pmap.c - * - * Machine dependant vm stuff - * - * Created : 20/09/94 + * In order to cope with hardware architectures which + * make virtual-to-physical map invalidates expensive, + * this module may delay invalidate or reduced protection + * operations until such time as they are actually + * necessary. This module is given full information as + * to which processors are currently using which maps, + * and to when physical maps must be made correct. */ -/* - * Special compilation symbols - * PMAP_DEBUG - Build in pmap_debug_level code - * - * Note that pmap_mapdev() and pmap_unmapdev() are implemented in arm/devmap.c -*/ -/* Include header files */ - #include "opt_vm.h" #include "opt_pmap.h" +#include "opt_ddb.h" -#include -__FBSDID("$FreeBSD$"); #include #include #include #include #include #include +#include #include -#include -#include #include +#include #include -#include +#include #include #include #include +#ifdef SMP +#include +#else +#include +#endif +#ifdef DDB +#include +#endif + +#include +#include + #include -#include #include #include +#include #include #include #include #include #include #include #include #include +#include +#include #include +#include #include -#include +#include #include +#include +#ifdef SMP +#include +#endif -#ifdef DEBUG -extern int last_fault_code; +#ifndef PMAP_SHPGPERPROC +#define PMAP_SHPGPERPROC 200 #endif +#ifndef DIAGNOSTIC +#define PMAP_INLINE __inline +#else +#define PMAP_INLINE +#endif + #ifdef PMAP_DEBUG +static void pmap_zero_page_check(vm_page_t m); +void pmap_debug(int level); +int pmap_pid_dump(int pid); + #define PDEBUG(_lev_,_stat_) \ - if (pmap_debug_level >= (_lev_)) \ - ((_stat_)) + if (pmap_debug_level >= (_lev_)) \ + ((_stat_)) #define dprintf printf - -int pmap_debug_level = 0; -#define PMAP_INLINE +int pmap_debug_level = 1; #else /* PMAP_DEBUG */ #define PDEBUG(_lev_,_stat_) /* Nothing */ #define dprintf(x, arg...) -#define PMAP_INLINE __inline #endif /* PMAP_DEBUG */ -#ifdef PV_STATS -#define PV_STAT(x) do { x ; } while (0) -#else -#define PV_STAT(x) do { } while (0) -#endif +/* + * Level 2 page tables map definion ('max' is excluded). + */ -#define pa_to_pvh(pa) (&pv_table[pa_index(pa)]) +#define PT2V_MIN_ADDRESS ((vm_offset_t)PT2MAP) +#define PT2V_MAX_ADDRESS ((vm_offset_t)PT2MAP + PT2MAP_SIZE) -#ifdef ARM_L2_PIPT -#define pmap_l2cache_wbinv_range(va, pa, size) cpu_l2cache_wbinv_range((pa), (size)) -#define pmap_l2cache_inv_range(va, pa, size) cpu_l2cache_inv_range((pa), (size)) -#else -#define pmap_l2cache_wbinv_range(va, pa, size) cpu_l2cache_wbinv_range((va), (size)) -#define pmap_l2cache_inv_range(va, pa, size) cpu_l2cache_inv_range((va), (size)) -#endif +#define UPT2V_MIN_ADDRESS ((vm_offset_t)PT2MAP) +#define UPT2V_MAX_ADDRESS \ + ((vm_offset_t)(PT2MAP + (KERNBASE >> PT2MAP_SHIFT))) -extern struct pv_addr systempage; +/* + * Promotion to a 1MB (PTE1) page mapping requires that the corresponding + * 4KB (PTE2) page mappings have identical settings for the following fields: + */ +#define PTE2_PROMOTE (PTE2_V | PTE2_A | PTE2_NM | PTE2_S | PTE2_NG | \ + PTE2_NX | PTE2_RO | PTE2_U | PTE2_W | \ + PTE2_ATTR_MASK) +#define PTE1_PROMOTE (PTE1_V | PTE1_A | PTE1_NM | PTE1_S | PTE1_NG | \ + PTE1_NX | PTE1_RO | PTE1_U | PTE1_W | \ + PTE1_ATTR_MASK) + +#define ATTR_TO_L1(l2_attr) ((((l2_attr) & L2_TEX0) ? L1_S_TEX0 : 0) | \ + (((l2_attr) & L2_C) ? L1_S_C : 0) | \ + (((l2_attr) & L2_B) ? L1_S_B : 0) | \ + (((l2_attr) & PTE2_A) ? PTE1_A : 0) | \ + (((l2_attr) & PTE2_NM) ? PTE1_NM : 0) | \ + (((l2_attr) & PTE2_S) ? PTE1_S : 0) | \ + (((l2_attr) & PTE2_NG) ? PTE1_NG : 0) | \ + (((l2_attr) & PTE2_NX) ? PTE1_NX : 0) | \ + (((l2_attr) & PTE2_RO) ? PTE1_RO : 0) | \ + (((l2_attr) & PTE2_U) ? PTE1_U : 0) | \ + (((l2_attr) & PTE2_W) ? PTE1_W : 0)) + +#define ATTR_TO_L2(l1_attr) ((((l1_attr) & L1_S_TEX0) ? L2_TEX0 : 0) | \ + (((l1_attr) & L1_S_C) ? L2_C : 0) | \ + (((l1_attr) & L1_S_B) ? L2_B : 0) | \ + (((l1_attr) & PTE1_A) ? PTE2_A : 0) | \ + (((l1_attr) & PTE1_NM) ? PTE2_NM : 0) | \ + (((l1_attr) & PTE1_S) ? PTE2_S : 0) | \ + (((l1_attr) & PTE1_NG) ? PTE2_NG : 0) | \ + (((l1_attr) & PTE1_NX) ? PTE2_NX : 0) | \ + (((l1_attr) & PTE1_RO) ? PTE2_RO : 0) | \ + (((l1_attr) & PTE1_U) ? PTE2_U : 0) | \ + (((l1_attr) & PTE1_W) ? PTE2_W : 0)) + /* - * Internal function prototypes + * PTE2 descriptors creation macros. */ +#define PTE2_KPT(pa) PTE2_KERN(pa, PTE2_AP_KRW, pt_memattr) +#define PTE2_KPT_NG(pa) PTE2_KERN_NG(pa, PTE2_AP_KRW, pt_memattr) -static PMAP_INLINE -struct pv_entry *pmap_find_pv(struct md_page *, pmap_t, vm_offset_t); -static void pmap_free_pv_chunk(struct pv_chunk *pc); -static void pmap_free_pv_entry(pmap_t pmap, pv_entry_t pv); -static pv_entry_t pmap_get_pv_entry(pmap_t pmap, boolean_t try); -static vm_page_t pmap_pv_reclaim(pmap_t locked_pmap); -static boolean_t pmap_pv_insert_section(pmap_t, vm_offset_t, - vm_paddr_t); -static struct pv_entry *pmap_remove_pv(struct vm_page *, pmap_t, vm_offset_t); -static int pmap_pvh_wired_mappings(struct md_page *, int); +#define PTE2_KRW(pa) PTE2_KERN(pa, PTE2_AP_KRW, PTE2_ATTR_NORMAL) +#define PTE2_KRO(pa) PTE2_KERN(pa, PTE2_AP_KR, PTE2_ATTR_NORMAL) -static int pmap_enter_locked(pmap_t, vm_offset_t, vm_page_t, - vm_prot_t, u_int); -static vm_paddr_t pmap_extract_locked(pmap_t pmap, vm_offset_t va); -static void pmap_alloc_l1(pmap_t); -static void pmap_free_l1(pmap_t); +#define PV_STATS +#ifdef PV_STATS +#define PV_STAT(x) do { x ; } while (0) +#else +#define PV_STAT(x) do { } while (0) +#endif -static void pmap_map_section(pmap_t, vm_offset_t, vm_offset_t, - vm_prot_t, boolean_t); -static void pmap_promote_section(pmap_t, vm_offset_t); -static boolean_t pmap_demote_section(pmap_t, vm_offset_t); -static boolean_t pmap_enter_section(pmap_t, vm_offset_t, vm_page_t, - vm_prot_t); -static void pmap_remove_section(pmap_t, vm_offset_t); +/* + * The boot_pt1 is used temporary in very early boot stage as L1 page table. + * We can init many things with no memory allocation thanks to its static + * allocation and this brings two main advantages: + * (1) other cores can be started very simply, + * (2) various boot loaders can be supported as its arguments can be processed + * in virtual address space and can be moved to safe location before + * first allocation happened. + * Only disadvantage is that boot_pt1 is used only in very early boot stage. + * However, the table is uninitialized and so lays in bss. Therefore kernel + * image size is not influenced. + * + * QQQ: In the future, maybe, boot_pt1 can be used for soft reset and + * CPU suspend/resume game. + */ +extern pt1_entry_t boot_pt1[]; -static int pmap_clearbit(struct vm_page *, u_int); +vm_paddr_t base_pt1; +pt1_entry_t *kern_pt1; +pt2_entry_t *kern_pt2tab; +pt2_entry_t *PT2MAP; -static struct l2_bucket *pmap_get_l2_bucket(pmap_t, vm_offset_t); -static struct l2_bucket *pmap_alloc_l2_bucket(pmap_t, vm_offset_t); -static void pmap_free_l2_bucket(pmap_t, struct l2_bucket *, u_int); -static vm_offset_t kernel_pt_lookup(vm_paddr_t); +static uint32_t ttb_flags; +static vm_memattr_t pt_memattr; +ttb_entry_t pmap_kern_ttb; -static MALLOC_DEFINE(M_VMPMAP, "pmap", "PMAP L1"); +/* XXX use converion function*/ +#define PTE2_ATTR_NORMAL VM_MEMATTR_DEFAULT +#define PTE1_ATTR_NORMAL ATTR_TO_L1(PTE2_ATTR_NORMAL) +struct pmap kernel_pmap_store; +LIST_HEAD(pmaplist, pmap); +static struct pmaplist allpmaps; +static struct mtx allpmaps_lock; + vm_offset_t virtual_avail; /* VA of first avail page (after kernel bss) */ vm_offset_t virtual_end; /* VA of last avail page (end of kernel AS) */ -vm_offset_t pmap_curmaxkvaddr; + +static vm_offset_t kernel_vm_end_new; +vm_offset_t kernel_vm_end = KERNBASE + NKPT2PG * NPT2_IN_PG * PTE1_SIZE; +vm_offset_t vm_max_kernel_address; vm_paddr_t kernel_l1pa; -vm_offset_t kernel_vm_end = 0; +static struct rwlock __aligned(CACHE_LINE_SIZE) pvh_global_lock; -vm_offset_t vm_max_kernel_address; +/* + * Data for the pv entry allocation mechanism + */ +static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); +static int pv_entry_count = 0, pv_entry_max = 0, pv_entry_high_water = 0; +static struct md_page *pv_table; /* XXX: Is it used only the list in md_page? */ +static int shpgperproc = PMAP_SHPGPERPROC; -struct pmap kernel_pmap_store; +struct pv_chunk *pv_chunkbase; /* KVA block for pv_chunks */ +int pv_maxchunks; /* How many chunks we have KVA for */ +vm_offset_t pv_vafree; /* freelist stored in the PTE */ +vm_paddr_t first_managed_pa; +#define pa_to_pvh(pa) (&pv_table[pte1_index(pa - first_managed_pa)]) + /* - * Resources for quickly copying and zeroing pages using virtual address space - * and page table entries that are pre-allocated per-CPU by pmap_init(). + * All those kernel PT submaps that BSD is so fond of */ -struct czpages { - struct mtx lock; - pt_entry_t *srcptep; - pt_entry_t *dstptep; - vm_offset_t srcva; - vm_offset_t dstva; +struct sysmaps { + struct mtx lock; + pt2_entry_t *CMAP1; + pt2_entry_t *CMAP2; + pt2_entry_t *CMAP3; + caddr_t CADDR1; + caddr_t CADDR2; + caddr_t CADDR3; }; -static struct czpages cpu_czpages[MAXCPU]; +static struct sysmaps sysmaps_pcpu[MAXCPU]; +static pt2_entry_t *CMAP3; +static caddr_t CADDR3; +caddr_t _tmppt = 0; -static void pmap_init_l1(struct l1_ttable *, pd_entry_t *); +struct msgbuf *msgbufp = 0; /* XXX move it to machdep.c */ + /* - * These routines are called when the CPU type is identified to set up - * the PTE prototypes, cache modes, etc. - * - * The variables are always here, just in case LKMs need to reference - * them (though, they shouldn't). + * Crashdump maps. */ -static void pmap_set_prot(pt_entry_t *pte, vm_prot_t prot, uint8_t user); -pt_entry_t pte_l1_s_cache_mode; -pt_entry_t pte_l1_s_cache_mode_pt; +static caddr_t crashdumpmap; -pt_entry_t pte_l2_l_cache_mode; -pt_entry_t pte_l2_l_cache_mode_pt; +static pt2_entry_t *PMAP1 = 0, *PMAP2; +static pt2_entry_t *PADDR1 = 0, *PADDR2; +#ifdef DDB +static pt2_entry_t *PMAP3; +static pt2_entry_t *PADDR3; +static int PMAP3cpu __unused; /* for SMP only */ +#endif +#ifdef SMP +static int PMAP1cpu; +static int PMAP1changedcpu; +SYSCTL_INT(_debug, OID_AUTO, PMAP1changedcpu, CTLFLAG_RD, + &PMAP1changedcpu, 0, + "Number of times pmap_pte2_quick changed CPU with same PMAP1"); +#endif +static int PMAP1changed; +SYSCTL_INT(_debug, OID_AUTO, PMAP1changed, CTLFLAG_RD, + &PMAP1changed, 0, + "Number of times pmap_pte2_quick changed PMAP1"); +static int PMAP1unchanged; +SYSCTL_INT(_debug, OID_AUTO, PMAP1unchanged, CTLFLAG_RD, + &PMAP1unchanged, 0, + "Number of times pmap_pte2_quick didn't change PMAP1"); +static struct mtx PMAP2mutex; -pt_entry_t pte_l2_s_cache_mode; -pt_entry_t pte_l2_s_cache_mode_pt; +static __inline void pt2_wirecount_init(vm_page_t m); +static boolean_t pmap_demote_pte1(pmap_t pmap, pt1_entry_t *pte1p, + vm_offset_t va); +void cache_icache_sync_fresh(vm_offset_t va, vm_paddr_t pa, vm_size_t size); -struct msgbuf *msgbufp = 0; +/* + * Function to set the debug level of the pmap code. + */ +#ifdef PMAP_DEBUG +void +pmap_debug(int level) +{ + pmap_debug_level = level; + dprintf("pmap_debug: level=%d\n", pmap_debug_level); +} +#endif /* PMAP_DEBUG */ + /* - * Crashdump maps. + * This table must corespond with memory attribute configuration in vm.h. + * First entry is used for normal system mapping. + * + * Device memory is always marked as shared. + * Normal memory is shared only in SMP . + * Not outer shareable bits are not used yet. + * Class 6 cannot be used on ARM11. */ -static caddr_t crashdumpmap; +#define TEXDEF_TYPE_SHIFT 0 +#define TEXDEF_TYPE_MASK 0x3 +#define TEXDEF_INNER_SHIFT 2 +#define TEXDEF_INNER_MASK 0x3 +#define TEXDEF_OUTER_SHIFT 4 +#define TEXDEF_OUTER_MASK 0x3 +#define TEXDEF_NOS_SHIFT 6 +#define TEXDEF_NOS_MASK 0x1 -extern void bcopy_page(vm_offset_t, vm_offset_t); -extern void bzero_page(vm_offset_t); +#define TEX(t, i, o, s) \ + ((t) << TEXDEF_TYPE_SHIFT) | \ + ((i) << TEXDEF_INNER_SHIFT) | \ + ((o) << TEXDEF_OUTER_SHIFT | \ + ((s) << TEXDEF_NOS_SHIFT)) -char *_tmppt; +static uint32_t tex_class[8] = { +/* type inner cache outer cache */ + TEX(PRRR_MEM, NMRR_WB_WA, NMRR_WB_WA, 0), /* 0 - ATTR_WB_WA */ + TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 1 - ATTR_NOCACHE */ + TEX(PRRR_DEV, NMRR_NC, NMRR_NC, 0), /* 2 - ATTR_DEVICE */ + TEX(PRRR_SO, NMRR_NC, NMRR_NC, 0), /* 3 - ATTR_SO */ + TEX(PRRR_MEM, NMRR_WT, NMRR_WT, 0), /* 4 - ATTR_WT */ + TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 5 - NOT USED YET */ + TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 6 - NOT USED YET */ + TEX(PRRR_MEM, NMRR_NC, NMRR_NC, 0), /* 7 - NOT USED YET */ +}; +#undef TEX /* - * Metadata for L1 translation tables. + * Convert TEX definition entry to TTB flags. */ -struct l1_ttable { - /* Entry on the L1 Table list */ - SLIST_ENTRY(l1_ttable) l1_link; +static uint32_t +encode_ttb_flags(int idx) +{ + uint32_t inner, outer, nos, reg; - /* Entry on the L1 Least Recently Used list */ - TAILQ_ENTRY(l1_ttable) l1_lru; + inner = (tex_class[idx] >> TEXDEF_INNER_SHIFT) & + TEXDEF_INNER_MASK; + outer = (tex_class[idx] >> TEXDEF_OUTER_SHIFT) & + TEXDEF_OUTER_MASK; + nos = (tex_class[idx] >> TEXDEF_NOS_SHIFT) & + TEXDEF_NOS_MASK; - /* Track how many domains are allocated from this L1 */ - volatile u_int l1_domain_use_count; + reg = nos << 5; + reg |= outer << 3; + if (cpuinfo.coherent_walk) + reg |= (inner & 0x1) << 6; + reg |= (inner & 0x2) >> 1; +#ifdef SMP + reg |= 1 << 1; +#endif + return reg; +} - /* - * A free-list of domain numbers for this L1. - * We avoid using ffs() and a bitmap to track domains since ffs() - * is slow on ARM. - */ - u_int8_t l1_domain_first; - u_int8_t l1_domain_free[PMAP_DOMAINS]; +/* + * Set TEX remapping registers in current CPU. + */ +void +pmap_set_tex(void) +{ + uint32_t prrr, nmrr; + uint32_t type, inner, outer, nos; + int i; - /* Physical address of this L1 page table */ - vm_paddr_t l1_physaddr; +#ifdef PMAP_PTE_NOCACHE + /* XXX fixme */ + if (cpuinfo.coherent_walk) { + pt_memattr = VM_MEMATTR_WB_WA; + ttb_flags = encode_ttb_flags(0); + } + else { + pt_memattr = VM_MEMATTR_NOCACHE; + ttb_flags = encode_ttb_flags(1); + } +#else + pt_memattr = VM_MEMATTR_WB_WA; + ttb_flags = encode_ttb_flags(0); +#endif - /* KVA of this L1 page table */ - pd_entry_t *l1_kva; -}; + prrr = 0; + nmrr = 0; + /* Build remapping register from TEX classes. */ + for (i = 0; i < 8; i++) { + type = (tex_class[i] >> TEXDEF_TYPE_SHIFT) & + TEXDEF_TYPE_MASK; + inner = (tex_class[i] >> TEXDEF_INNER_SHIFT) & + TEXDEF_INNER_MASK; + outer = (tex_class[i] >> TEXDEF_OUTER_SHIFT) & + TEXDEF_OUTER_MASK; + nos = (tex_class[i] >> TEXDEF_NOS_SHIFT) & + TEXDEF_NOS_MASK; + + prrr |= type << (i * 2); + prrr |= nos << (i + 24); + nmrr |= inner << (i * 2); + nmrr |= outer << (i * 2 + 16); + } + /* Add shareable bits for device memory. */ + prrr |= PRRR_DS0 | PRRR_DS1; + + /* Add shareable bits for normal memory in SMP case. */ +#ifdef SMP + prrr |= PRRR_NS1; +#endif + cp15_prrr_set(prrr); + cp15_nmrr_set(nmrr); + + /* Caches are disabled, so full TLB flush should be enough. */ + tlb_flush_all_local(); +} + /* - * Convert a virtual address into its L1 table index. That is, the - * index used to locate the L2 descriptor table pointer in an L1 table. - * This is basically used to index l1->l1_kva[]. - * - * Each L2 descriptor table represents 1MB of VA space. + * KERNBASE must be multiple of NPT2_IN_PG * PTE1_SIZE. In other words, + * KERNBASE is mapped by first L2 page table in L2 page table page. It + * meets same constrain due to PT2MAP being placed just under KERNBASE. */ -#define L1_IDX(va) (((vm_offset_t)(va)) >> L1_S_SHIFT) +CTASSERT((KERNBASE & (NPT2_IN_PG * PTE1_SIZE - 1)) == 0); +CTASSERT((KERNBASE - VM_MAXUSER_ADDRESS) >= PT2MAP_SIZE); /* - * L1 Page Tables are tracked using a Least Recently Used list. - * - New L1s are allocated from the HEAD. - * - Freed L1s are added to the TAIl. - * - Recently accessed L1s (where an 'access' is some change to one of - * the userland pmaps which owns this L1) are moved to the TAIL. + * In crazy dreams, PAGE_SIZE could be a multiple of PTE2_SIZE in general. + * For now, anyhow, the following check must be fulfilled. */ -static TAILQ_HEAD(, l1_ttable) l1_lru_list; +CTASSERT(PAGE_SIZE == PTE2_SIZE); /* - * A list of all L1 tables + * We don't want to mess up MI code with all MMU and PMAP definitions, + * so some things, which depend on other ones, are defined independently. + * Now, it is time to check that we don't screw up something. */ -static SLIST_HEAD(, l1_ttable) l1_list; -static struct mtx l1_lru_lock; - +CTASSERT(PDRSHIFT == PTE1_SHIFT); /* - * The l2_dtable tracks L2_BUCKET_SIZE worth of L1 slots. - * - * This is normally 16MB worth L2 page descriptors for any given pmap. - * Reference counts are maintained for L2 descriptors so they can be - * freed when empty. + * Check L1 and L2 page table entries definitions consistency. */ -struct l2_dtable { - /* The number of L2 page descriptors allocated to this l2_dtable */ - u_int l2_occupancy; - - /* List of L2 page descriptors */ - struct l2_bucket { - pt_entry_t *l2b_kva; /* KVA of L2 Descriptor Table */ - vm_paddr_t l2b_phys; /* Physical address of same */ - u_short l2b_l1idx; /* This L2 table's L1 index */ - u_short l2b_occupancy; /* How many active descriptors */ - } l2_bucket[L2_BUCKET_SIZE]; -}; - -/* pmap_kenter_internal flags */ -#define KENTER_CACHE 0x1 -#define KENTER_DEVICE 0x2 -#define KENTER_USER 0x4 - +CTASSERT(NB_IN_PT1 == (sizeof(pt1_entry_t) * NPTE1_IN_PT1)); +CTASSERT(NB_IN_PT2 == (sizeof(pt2_entry_t) * NPTE2_IN_PT2)); /* - * Given an L1 table index, calculate the corresponding l2_dtable index - * and bucket index within the l2_dtable. + * Check L2 page tables page consistency. */ -#define L2_IDX(l1idx) (((l1idx) >> L2_BUCKET_LOG2) & \ - (L2_SIZE - 1)) -#define L2_BUCKET(l1idx) ((l1idx) & (L2_BUCKET_SIZE - 1)) - +CTASSERT(PAGE_SIZE == (NPT2_IN_PG * NB_IN_PT2)); +CTASSERT((1 << PT2PG_SHIFT) == NPT2_IN_PG); /* - * Given a virtual address, this macro returns the - * virtual address required to drop into the next L2 bucket. + * Check PT2TAB consistency. + * PT2TAB_ENTRIES is defined as a division of NPTE1_IN_PT1 by NPT2_IN_PG. + * This should be done without remainder. */ -#define L2_NEXT_BUCKET(va) (((va) & L1_S_FRAME) + L1_S_SIZE) +CTASSERT(NPTE1_IN_PT1 == (PT2TAB_ENTRIES * NPT2_IN_PG)); /* - * We try to map the page tables write-through, if possible. However, not - * all CPUs have a write-through cache mode, so on those we have to sync - * the cache when we frob page tables. + * A PT2MAP magic. * - * We try to evaluate this at compile time, if possible. However, it's - * not always possible to do that, hence this run-time var. + * All level 2 page tables (PT2s) are mapped continuously and accordingly + * into PT2MAP address space. As PT2 size is less than PAGE_SIZE, this can + * be done only if PAGE_SIZE is a multiple of PT2 size. All PT2s in one page + * must be used together, but not necessary at once. The first PT2 in a page + * must map things on correctly aligned address and the others must follow + * in right order. */ -int pmap_needs_pte_sync; +#define NB_IN_PT2TAB (PT2TAB_ENTRIES * sizeof(pt2_entry_t)) +#define NPT2_IN_PT2TAB (NB_IN_PT2TAB / NB_IN_PT2) +#define NPG_IN_PT2TAB (NB_IN_PT2TAB / PAGE_SIZE) /* - * Macro to determine if a mapping might be resident in the - * instruction cache and/or TLB + * Check PT2TAB consistency. + * NPT2_IN_PT2TAB is defined as a division of NB_IN_PT2TAB by NB_IN_PT2. + * NPG_IN_PT2TAB is defined as a division of NB_IN_PT2TAB by PAGE_SIZE. + * The both should be done without remainder. */ -#define PTE_BEEN_EXECD(pte) (L2_S_EXECUTABLE(pte) && L2_S_REFERENCED(pte)) - +CTASSERT(NB_IN_PT2TAB == (NPT2_IN_PT2TAB * NB_IN_PT2)); +CTASSERT(NB_IN_PT2TAB == (NPG_IN_PT2TAB * PAGE_SIZE)); /* - * Macro to determine if a mapping might be resident in the - * data cache and/or TLB + * The implementation was made general, however, with the assumption + * bellow in mind. In case of another value of NPG_IN_PT2TAB, + * the code should be once more rechecked. */ -#define PTE_BEEN_REFD(pte) (L2_S_REFERENCED(pte)) +CTASSERT(NPG_IN_PT2TAB == 1); -#ifndef PMAP_SHPGPERPROC -#define PMAP_SHPGPERPROC 200 -#endif - -#define pmap_is_current(pm) ((pm) == pmap_kernel() || \ - curproc->p_vmspace->vm_map.pmap == (pm)) - /* - * Data for the pv entry allocation mechanism + * Get offset of PT2 in a page + * associated with given PT1 index. */ -static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); -static int pv_entry_count, pv_entry_max, pv_entry_high_water; -static struct md_page *pv_table; -static int shpgperproc = PMAP_SHPGPERPROC; - -struct pv_chunk *pv_chunkbase; /* KVA block for pv_chunks */ -int pv_maxchunks; /* How many chunks we have KVA for */ -vm_offset_t pv_vafree; /* Freelist stored in the PTE */ - -static __inline struct pv_chunk * -pv_to_chunk(pv_entry_t pv) +static __inline u_int +page_pt2off(u_int pt1_idx) { - return ((struct pv_chunk *)((uintptr_t)pv & ~(uintptr_t)PAGE_MASK)); + return ((pt1_idx & PT2PG_MASK) * NB_IN_PT2); } -#define PV_PMAP(pv) (pv_to_chunk(pv)->pc_pmap) - -CTASSERT(sizeof(struct pv_chunk) == PAGE_SIZE); -CTASSERT(_NPCM == 8); -CTASSERT(_NPCPV == 252); - -#define PC_FREE0_6 0xfffffffful /* Free values for index 0 through 6 */ -#define PC_FREE7 0x0ffffffful /* Free values for index 7 */ - -static const uint32_t pc_freemask[_NPCM] = { - PC_FREE0_6, PC_FREE0_6, PC_FREE0_6, - PC_FREE0_6, PC_FREE0_6, PC_FREE0_6, - PC_FREE0_6, PC_FREE7 -}; - -static SYSCTL_NODE(_vm, OID_AUTO, pmap, CTLFLAG_RD, 0, "VM/pmap parameters"); - -/* Superpages utilization enabled = 1 / disabled = 0 */ -static int sp_enabled = 1; -SYSCTL_INT(_vm_pmap, OID_AUTO, sp_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &sp_enabled, 0, - "Are large page mappings enabled?"); - -SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_count, CTLFLAG_RD, &pv_entry_count, 0, - "Current number of pv entries"); - -#ifdef PV_STATS -static int pc_chunk_count, pc_chunk_allocs, pc_chunk_frees, pc_chunk_tryfail; - -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_count, CTLFLAG_RD, &pc_chunk_count, 0, - "Current number of pv entry chunks"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_allocs, CTLFLAG_RD, &pc_chunk_allocs, 0, - "Current number of pv entry chunks allocated"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_frees, CTLFLAG_RD, &pc_chunk_frees, 0, - "Current number of pv entry chunks frees"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_tryfail, CTLFLAG_RD, &pc_chunk_tryfail, 0, - "Number of times tried to get a chunk page but failed."); - -static long pv_entry_frees, pv_entry_allocs; -static int pv_entry_spare; - -SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_frees, CTLFLAG_RD, &pv_entry_frees, 0, - "Current number of pv entry frees"); -SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_allocs, CTLFLAG_RD, &pv_entry_allocs, 0, - "Current number of pv entry allocs"); -SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_spare, CTLFLAG_RD, &pv_entry_spare, 0, - "Current number of spare pv entries"); -#endif - -uma_zone_t l2zone; -static uma_zone_t l2table_zone; -static vm_offset_t pmap_kernel_l2dtable_kva; -static vm_offset_t pmap_kernel_l2ptp_kva; -static vm_paddr_t pmap_kernel_l2ptp_phys; -static struct rwlock pvh_global_lock; - -int l1_mem_types[] = { - ARM_L1S_STRONG_ORD, - ARM_L1S_DEVICE_NOSHARE, - ARM_L1S_DEVICE_SHARE, - ARM_L1S_NRML_NOCACHE, - ARM_L1S_NRML_IWT_OWT, - ARM_L1S_NRML_IWB_OWB, - ARM_L1S_NRML_IWBA_OWBA -}; - -int l2l_mem_types[] = { - ARM_L2L_STRONG_ORD, - ARM_L2L_DEVICE_NOSHARE, - ARM_L2L_DEVICE_SHARE, - ARM_L2L_NRML_NOCACHE, - ARM_L2L_NRML_IWT_OWT, - ARM_L2L_NRML_IWB_OWB, - ARM_L2L_NRML_IWBA_OWBA -}; - -int l2s_mem_types[] = { - ARM_L2S_STRONG_ORD, - ARM_L2S_DEVICE_NOSHARE, - ARM_L2S_DEVICE_SHARE, - ARM_L2S_NRML_NOCACHE, - ARM_L2S_NRML_IWT_OWT, - ARM_L2S_NRML_IWB_OWB, - ARM_L2S_NRML_IWBA_OWBA -}; - /* - * This list exists for the benefit of pmap_map_chunk(). It keeps track - * of the kernel L2 tables during bootstrap, so that pmap_map_chunk() can - * find them as necessary. - * - * Note that the data on this list MUST remain valid after initarm() returns, - * as pmap_bootstrap() uses it to contruct L2 table metadata. + * Get physical address of PT2 + * associated with given PT2s page and PT1 index. */ -SLIST_HEAD(, pv_addr) kernel_pt_list = SLIST_HEAD_INITIALIZER(kernel_pt_list); - -static void -pmap_init_l1(struct l1_ttable *l1, pd_entry_t *l1pt) +static __inline vm_paddr_t +page_pt2pa(vm_paddr_t pgpa, u_int pt1_idx) { - int i; - l1->l1_kva = l1pt; - l1->l1_domain_use_count = 0; - l1->l1_domain_first = 0; - - for (i = 0; i < PMAP_DOMAINS; i++) - l1->l1_domain_free[i] = i + 1; - - /* - * Copy the kernel's L1 entries to each new L1. - */ - if (l1pt != pmap_kernel()->pm_l1->l1_kva) - memcpy(l1pt, pmap_kernel()->pm_l1->l1_kva, L1_TABLE_SIZE); - - if ((l1->l1_physaddr = pmap_extract(pmap_kernel(), (vm_offset_t)l1pt)) == 0) - panic("pmap_init_l1: can't get PA of L1 at %p", l1pt); - SLIST_INSERT_HEAD(&l1_list, l1, l1_link); - TAILQ_INSERT_TAIL(&l1_lru_list, l1, l1_lru); + return (pgpa + page_pt2off(pt1_idx)); } -static vm_offset_t -kernel_pt_lookup(vm_paddr_t pa) -{ - struct pv_addr *pv; - - SLIST_FOREACH(pv, &kernel_pt_list, pv_list) { - if (pv->pv_pa == pa) - return (pv->pv_va); - } - return (0); -} - -void -pmap_pte_init_mmu_v6(void) -{ - - if (PTE_PAGETABLE >= 3) - pmap_needs_pte_sync = 1; - pte_l1_s_cache_mode = l1_mem_types[PTE_CACHE]; - pte_l2_l_cache_mode = l2l_mem_types[PTE_CACHE]; - pte_l2_s_cache_mode = l2s_mem_types[PTE_CACHE]; - - pte_l1_s_cache_mode_pt = l1_mem_types[PTE_PAGETABLE]; - pte_l2_l_cache_mode_pt = l2l_mem_types[PTE_PAGETABLE]; - pte_l2_s_cache_mode_pt = l2s_mem_types[PTE_PAGETABLE]; - -} - /* - * Allocate an L1 translation table for the specified pmap. - * This is called at pmap creation time. + * Get first entry of PT2 + * associated with given PT2s page and PT1 index. */ -static void -pmap_alloc_l1(pmap_t pmap) +static __inline pt2_entry_t * +page_pt2(vm_offset_t pgva, u_int pt1_idx) { - struct l1_ttable *l1; - u_int8_t domain; - /* - * Remove the L1 at the head of the LRU list - */ - mtx_lock(&l1_lru_lock); - l1 = TAILQ_FIRST(&l1_lru_list); - TAILQ_REMOVE(&l1_lru_list, l1, l1_lru); - - /* - * Pick the first available domain number, and update - * the link to the next number. - */ - domain = l1->l1_domain_first; - l1->l1_domain_first = l1->l1_domain_free[domain]; - - /* - * If there are still free domain numbers in this L1, - * put it back on the TAIL of the LRU list. - */ - if (++l1->l1_domain_use_count < PMAP_DOMAINS) - TAILQ_INSERT_TAIL(&l1_lru_list, l1, l1_lru); - - mtx_unlock(&l1_lru_lock); - - /* - * Fix up the relevant bits in the pmap structure - */ - pmap->pm_l1 = l1; - pmap->pm_domain = domain + 1; + return ((pt2_entry_t *)(pgva + page_pt2off(pt1_idx))); } /* - * Free an L1 translation table. - * This is called at pmap destruction time. + * Get virtual address of PT2s page (mapped in PT2MAP) + * which holds PT2 which holds entry which maps given virtual address. */ -static void -pmap_free_l1(pmap_t pmap) +static __inline vm_offset_t +pt2map_pt2pg(vm_offset_t va) { - struct l1_ttable *l1 = pmap->pm_l1; - mtx_lock(&l1_lru_lock); + va &= ~(NPT2_IN_PG * PTE1_SIZE - 1); + return ((vm_offset_t)pt2map_entry(va)); +} - /* - * If this L1 is currently on the LRU list, remove it. - */ - if (l1->l1_domain_use_count < PMAP_DOMAINS) - TAILQ_REMOVE(&l1_lru_list, l1, l1_lru); +/***************************************************************************** + * + * THREE pmap initialization milestones exist: + * + * locore.S + * -> fundamental init (including MMU) in ASM + * + * initarm() + * -> fundamental init continues in C + * -> first available physical address is known + * + * pmap_bootstrap_prepare() -> FIRST PMAP MILESTONE (first epoch begins) + * -> basic (safe) interface for physical address allocation is made + * -> basic (safe) interface for virtual mapping is made + * -> limited not SMP coherent work is possible + * + * -> more fundamental init continues in C + * -> locks and some more things are available + * -> all fundamental allocations and mappings are done + * + * pmap_bootstrap() -> SECOND PMAP MILESTONE (second epoch begins) + * -> phys_avail[] and virtual_avail is set + * -> control is passed to vm subsystem + * -> physical and virtual address allocation are off limit + * -> low level mapping functions, some SMP coherent, + * are available, which cannot be used before vm subsystem + * is being inited + * + * mi_startup() + * -> vm subsystem is being inited + * + * pmap_init() -> THIRD PMAP MILESTONE (third epoch begins) + * -> pmap is fully inited + * + *****************************************************************************/ - /* - * Free up the domain number which was allocated to the pmap - */ - l1->l1_domain_free[pmap->pm_domain - 1] = l1->l1_domain_first; - l1->l1_domain_first = pmap->pm_domain - 1; - l1->l1_domain_use_count--; +/***************************************************************************** + * + * PMAP first stage initialization and utility functions + * for pre-bootstrap epoch. + * + * After pmap_bootstrap_prepare() is called, the following functions + * can be used: + * + * (1) strictly only for this stage functions for physical page allocations, + * virtual space allocations, and mappings: + * + * vm_paddr_t pmap_preboot_get_pages(u_int num); + * void pmap_preboot_map_pages(vm_paddr_t pa, vm_offset_t va, u_int num); + * vm_offset_t pmap_preboot_reserve_pages(u_int num); + * vm_offset_t pmap_preboot_get_vpages(u_int num); + * void pmap_preboot_map_attr(vm_paddr_t pa, vm_offset_t va, vm_size_t size, + * int prot, int attr); + * + * (2) for all stages: + * + * vm_paddr_t pmap_kextract(vm_offset_t va); + * + * NOTE: This is not SMP coherent stage. + * + *****************************************************************************/ - /* - * The L1 now must have at least 1 free domain, so add - * it back to the LRU list. If the use count is zero, - * put it at the head of the list, otherwise it goes - * to the tail. - */ - if (l1->l1_domain_use_count == 0) { - TAILQ_INSERT_HEAD(&l1_lru_list, l1, l1_lru); - } else - TAILQ_INSERT_TAIL(&l1_lru_list, l1, l1_lru); +#define KERNEL_P2V(pa) \ + ((vm_offset_t)((pa) - arm_physmem_kernaddr + KERNVIRTADDR)) +#define KERNEL_V2P(va) \ + ((vm_paddr_t)((va) - KERNVIRTADDR + arm_physmem_kernaddr)) - mtx_unlock(&l1_lru_lock); -} +static vm_paddr_t last_paddr; /* - * Returns a pointer to the L2 bucket associated with the specified pmap - * and VA, or NULL if no L2 bucket exists for the address. + * Pre-bootstrap epoch page allocator. */ -static PMAP_INLINE struct l2_bucket * -pmap_get_l2_bucket(pmap_t pmap, vm_offset_t va) +vm_paddr_t +pmap_preboot_get_pages(u_int num) { - struct l2_dtable *l2; - struct l2_bucket *l2b; - u_short l1idx; + vm_paddr_t ret; - l1idx = L1_IDX(va); + ret = last_paddr; + last_paddr += num * PAGE_SIZE; - if ((l2 = pmap->pm_l2[L2_IDX(l1idx)]) == NULL || - (l2b = &l2->l2_bucket[L2_BUCKET(l1idx)])->l2b_kva == NULL) - return (NULL); - - return (l2b); + return (ret); } /* - * Returns a pointer to the L2 bucket associated with the specified pmap - * and VA. + * The fundamental initalization of PMAP stuff. * - * If no L2 bucket exists, perform the necessary allocations to put an L2 - * bucket/page table in place. + * Some things already happened in locore.S and some things could happen + * before pmap_bootstrap_prepare() is called, so let's recall what is done: + * 1. Caches are disabled. + * 2. We are running on virtual addresses already with 'boot_pt1' + * as L1 page table. + * 3. So far, all virtual addresses can be converted to physical ones and + * vice versa by the following macros: + * KERNEL_P2V(pa) .... physical to virtual ones, + * KERNEL_V2P(va) .... virtual to physical ones. * - * Note that if a new L2 bucket/page was allocated, the caller *must* - * increment the bucket occupancy counter appropriately *before* - * releasing the pmap's lock to ensure no other thread or cpu deallocates - * the bucket/page in the meantime. + * What is done herein: + * 1. The 'boot_pt1' is replaced by real kernel L1 page table 'kern_pt1'. + * 2. PT2MAP magic is brought to live. + * 3. Basic preboot functions for page allocations and mappings can be used. + * 4. Everything is prepared for L1 cache enabling. + * + * Variations: + * 1. To use second TTB register, so kernel and users page tables will be + * separated. This way process forking - pmap_pinit() - could be faster, + * it saves physical pages and KVA per a process, and it's simple change. + * However, it will lead, due to hardware matter, to the following: + * (a) 2G space for kernel and 2G space for users. + * (b) 1G space for kernel in low addresses and 3G for users above it. + * A question is: Is the case (b) really an option? Note that case (b) + * does save neither physical memory and KVA. */ -static struct l2_bucket * -pmap_alloc_l2_bucket(pmap_t pmap, vm_offset_t va) +void +pmap_bootstrap_prepare(vm_paddr_t last) { - struct l2_dtable *l2; - struct l2_bucket *l2b; - u_short l1idx; + vm_paddr_t pt2pg_pa, pt2tab_pa, pa, size; + vm_offset_t pt2pg_va; + pt1_entry_t *pte1p; + pt2_entry_t *pte2p; + u_int i; + uint32_t actlr_mask, actlr_set; - l1idx = L1_IDX(va); + /* + * Now, we are going to make real kernel mapping. Note that we are + * already running on some mapping made in locore.S and we expect + * that it's large enough to ensure nofault access to physical memory + * allocated herein before switch. + * + * As kernel image and everything needed before are and will be mapped + * by section mappings, we align last physical address to PTE1_SIZE. + */ + last_paddr = pte1_roundup(last); - PMAP_ASSERT_LOCKED(pmap); - rw_assert(&pvh_global_lock, RA_WLOCKED); - if ((l2 = pmap->pm_l2[L2_IDX(l1idx)]) == NULL) { - /* - * No mapping at this address, as there is - * no entry in the L1 table. - * Need to allocate a new l2_dtable. - */ - PMAP_UNLOCK(pmap); - rw_wunlock(&pvh_global_lock); - if ((l2 = uma_zalloc(l2table_zone, M_NOWAIT)) == NULL) { - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - return (NULL); - } - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - if (pmap->pm_l2[L2_IDX(l1idx)] != NULL) { - /* - * Someone already allocated the l2_dtable while - * we were doing the same. - */ - uma_zfree(l2table_zone, l2); - l2 = pmap->pm_l2[L2_IDX(l1idx)]; - } else { - bzero(l2, sizeof(*l2)); - /* - * Link it into the parent pmap - */ - pmap->pm_l2[L2_IDX(l1idx)] = l2; - } - } - - l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; - /* - * Fetch pointer to the L2 page table associated with the address. + * Allocate and zero page(s) for kernel L1 page table. + * + * Note that it's first allocation on space which was PTE1_SIZE + * aligned and as such base_pt1 is aligned to NB_IN_PT1 too. */ - if (l2b->l2b_kva == NULL) { - pt_entry_t *ptep; + base_pt1 = pmap_preboot_get_pages(NPG_IN_PT1); + kern_pt1 = (pt1_entry_t *)KERNEL_P2V(base_pt1); + bzero((void*)kern_pt1, NB_IN_PT1); + pte1_sync_range(kern_pt1, NB_IN_PT1); - /* - * No L2 page table has been allocated. Chances are, this - * is because we just allocated the l2_dtable, above. - */ - l2->l2_occupancy++; - PMAP_UNLOCK(pmap); - rw_wunlock(&pvh_global_lock); - ptep = uma_zalloc(l2zone, M_NOWAIT); - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - if (l2b->l2b_kva != 0) { - /* We lost the race. */ - l2->l2_occupancy--; - uma_zfree(l2zone, ptep); - return (l2b); - } - l2b->l2b_phys = vtophys(ptep); - if (ptep == NULL) { - /* - * Oops, no more L2 page tables available at this - * time. We may need to deallocate the l2_dtable - * if we allocated a new one above. - */ - l2->l2_occupancy--; - if (l2->l2_occupancy == 0) { - pmap->pm_l2[L2_IDX(l1idx)] = NULL; - uma_zfree(l2table_zone, l2); - } - return (NULL); - } + /* Allocate and zero page(s) for kernel PT2TAB. */ + pt2tab_pa = pmap_preboot_get_pages(NPG_IN_PT2TAB); + kern_pt2tab = (pt2_entry_t *)KERNEL_P2V(pt2tab_pa); + bzero(kern_pt2tab, NB_IN_PT2TAB); + pte2_sync_range(kern_pt2tab, NB_IN_PT2TAB); - l2b->l2b_kva = ptep; - l2b->l2b_l1idx = l1idx; - } + /* Allocate and zero page(s) for kernel L2 page tables. */ + pt2pg_pa = pmap_preboot_get_pages(NKPT2PG); + pt2pg_va = KERNEL_P2V(pt2pg_pa); + size = NKPT2PG * PAGE_SIZE; + bzero((void*)pt2pg_va, size); + pte2_sync_range((pt2_entry_t *)pt2pg_va, size); - return (l2b); -} - -static PMAP_INLINE void -pmap_free_l2_ptp(pt_entry_t *l2) -{ - uma_zfree(l2zone, l2); -} -/* - * One or more mappings in the specified L2 descriptor table have just been - * invalidated. - * - * Garbage collect the metadata and descriptor table itself if necessary. - * - * The pmap lock must be acquired when this is called (not necessary - * for the kernel pmap). - */ -static void -pmap_free_l2_bucket(pmap_t pmap, struct l2_bucket *l2b, u_int count) -{ - struct l2_dtable *l2; - pd_entry_t *pl1pd, l1pd; - pt_entry_t *ptep; - u_short l1idx; - - /* - * Update the bucket's reference count according to how many - * PTEs the caller has just invalidated. + * Add a physical memory segment (vm_phys_seg) corresponding to the + * preallocated pages for kernel L2 page tables so that vm_page + * structures representing these pages will be created. The vm_page + * structures are required for promotion of the corresponding kernel + * virtual addresses to section mappings. */ - l2b->l2b_occupancy -= count; + vm_phys_add_seg(pt2tab_pa, pmap_preboot_get_pages(0)); /* - * Note: + * Insert allocated L2 page table pages to PT2TAB and make + * link to all PT2s in L1 page table. See how kernel_vm_end + * is initialized. * - * Level 2 page tables allocated to the kernel pmap are never freed - * as that would require checking all Level 1 page tables and - * removing any references to the Level 2 page table. See also the - * comment elsewhere about never freeing bootstrap L2 descriptors. - * - * We make do with just invalidating the mapping in the L2 table. - * - * This isn't really a big deal in practice and, in fact, leads - * to a performance win over time as we don't need to continually - * alloc/free. + * We play simple and safe. So every KVA will have underlaying + * L2 page table, even kernel image mapped by sections. */ - if (l2b->l2b_occupancy > 0 || pmap == pmap_kernel()) - return; + pte2p = kern_pt2tab_entry(KERNBASE); + for (pa = pt2pg_pa; pa < pt2pg_pa + size; pa += PTE2_SIZE) + pt2tab_store(pte2p++, PTE2_KPT(pa)); - /* - * There are no more valid mappings in this level 2 page table. - * Go ahead and NULL-out the pointer in the bucket, then - * free the page table. - */ - l1idx = l2b->l2b_l1idx; - ptep = l2b->l2b_kva; - l2b->l2b_kva = NULL; + pte1p = kern_pte1(KERNBASE); + for (pa = pt2pg_pa; pa < pt2pg_pa + size; pa += NB_IN_PT2) + pte1_store(pte1p++, PTE1_LINK(pa)); - pl1pd = &pmap->pm_l1->l1_kva[l1idx]; + /* Make section mappings for kernel. */ + pte1p = kern_pte1(KERNBASE); + for (pa = KERNEL_V2P(KERNBASE); pa < last; pa += PTE1_SIZE) + pte1_store(pte1p++, PTE1_KERN(pa, PTE1_AP_KRW, + ATTR_TO_L1(PTE2_ATTR_WB_WA))); /* - * If the L1 slot matches the pmap's domain - * number, then invalidate it. + * Get free and aligned space for PT2MAP and make L1 page table links + * to L2 page tables held in PT2TAB. + * + * Note that pages holding PT2s are stored in PT2TAB as pt2_entry_t + * descriptors and PT2TAB page(s) itself is(are) used as PT2s. Thus + * each entry in PT2TAB maps all PT2s in a page. This implies that + * virtual address of PT2MAP must be aligned to NPT2_IN_PG * PTE1_SIZE. */ - l1pd = *pl1pd & (L1_TYPE_MASK | L1_C_DOM_MASK); - if (l1pd == (L1_C_DOM(pmap->pm_domain) | L1_TYPE_C)) { - *pl1pd = 0; - PTE_SYNC(pl1pd); - cpu_tlb_flushD_SE((vm_offset_t)ptep); - cpu_cpwait(); + PT2MAP = (pt2_entry_t *)(KERNBASE - PT2MAP_SIZE); + pte1p = kern_pte1((vm_offset_t)PT2MAP); + for (pa = pt2tab_pa, i = 0; i < NPT2_IN_PT2TAB; i++, pa += NB_IN_PT2) { + pte1_store(pte1p++, PTE1_LINK(pa)); } /* - * Release the L2 descriptor table back to the pool cache. + * Store PT2TAB in PT2TAB itself, i.e. self reference mapping. + * Each pmap will hold own PT2TAB, so the mapping should be not global. */ - pmap_free_l2_ptp(ptep); + pte2p = kern_pt2tab_entry((vm_offset_t)PT2MAP); + for (pa = pt2tab_pa, i = 0; i < NPG_IN_PT2TAB; i++, pa += PTE2_SIZE) { + pt2tab_store(pte2p++, PTE2_KPT_NG(pa)); + } /* - * Update the reference count in the associated l2_dtable + * Choose correct L2 page table and make mappings for allocations + * made herein which replaces temporary locore.S mappings after a while. + * Note that PT2MAP cannot be used until we switch to kern_pt1. + * + * Note, that these allocations started aligned on 1M section and + * kernel PT1 was allocated first. Making of mappings must follow + * order of physical allocations as we've used KERNEL_P2V() macro + * for virtual addresses resolution. */ - l2 = pmap->pm_l2[L2_IDX(l1idx)]; - if (--l2->l2_occupancy > 0) - return; + pte2p = kern_pt2tab_entry((vm_offset_t)kern_pt1); + pt2pg_va = KERNEL_P2V(pte2_pa(pte2_load(pte2p))); - /* - * There are no more valid mappings in any of the Level 1 - * slots managed by this l2_dtable. Go ahead and NULL-out - * the pointer in the parent pmap and free the l2_dtable. - */ - pmap->pm_l2[L2_IDX(l1idx)] = NULL; - uma_zfree(l2table_zone, l2); -} + pte2p = page_pt2(pt2pg_va, pte1_index((vm_offset_t)kern_pt1)); -/* - * Pool cache constructors for L2 descriptor tables, metadata and pmap - * structures. - */ -static int -pmap_l2ptp_ctor(void *mem, int size, void *arg, int flags) -{ - struct l2_bucket *l2b; - pt_entry_t *ptep, pte; - vm_offset_t va = (vm_offset_t)mem & ~PAGE_MASK; + /* Make mapping for kernel L1 page table. */ + for (pa = base_pt1, i = 0; i < NPG_IN_PT1; i++, pa += PTE2_SIZE) + pte2_store(pte2p++, PTE2_KPT(pa)); + /* Make mapping for kernel PT2TAB. */ + for (pa = pt2tab_pa, i = 0; i < NPG_IN_PT2TAB; i++, pa += PTE2_SIZE) + pte2_store(pte2p++, PTE2_KPT(pa)); + + /* Finally, switch from 'boot_pt1' to 'kern_pt1'. */ + pmap_kern_ttb = base_pt1 | ttb_flags; + cpuinfo_get_actlr_modifier(&actlr_mask, &actlr_set); + reinit_mmu(pmap_kern_ttb, actlr_mask, actlr_set); /* - * The mappings for these page tables were initially made using - * pmap_kenter() by the pool subsystem. Therefore, the cache- - * mode will not be right for page table mappings. To avoid - * polluting the pmap_kenter() code with a special case for - * page tables, we simply fix up the cache-mode here if it's not - * correct. + * Initialize the first available KVA. As kernel image is mapped by + * sections, we are leaving some gap behind. */ - l2b = pmap_get_l2_bucket(pmap_kernel(), va); - ptep = &l2b->l2b_kva[l2pte_index(va)]; - pte = *ptep; - - cpu_idcache_wbinv_range(va, PAGE_SIZE); - pmap_l2cache_wbinv_range(va, pte & L2_S_FRAME, PAGE_SIZE); - if ((pte & L2_S_CACHE_MASK) != pte_l2_s_cache_mode_pt) { - /* - * Page tables must have the cache-mode set to - * Write-Thru. - */ - *ptep = (pte & ~L2_S_CACHE_MASK) | pte_l2_s_cache_mode_pt; - PTE_SYNC(ptep); - cpu_tlb_flushD_SE(va); - cpu_cpwait(); - } - - memset(mem, 0, L2_TABLE_SIZE_REAL); - return (0); + virtual_avail = (vm_offset_t)kern_pt2tab + NPG_IN_PT2TAB * PAGE_SIZE; } /* - * Modify pte bits for all ptes corresponding to the given physical address. - * We use `maskbits' rather than `clearbits' because we're always passing - * constants and the latter would require an extra inversion at run-time. + * Setup L2 page table page for given KVA. + * Used in pre-bootstrap epoch. + * + * Note that we have allocated NKPT2PG pages for L2 page tables in advance + * and used them for mapping KVA starting from KERNBASE. However, this is not + * enough. Vectors and devices need L2 page tables too. Note that they are + * even above VM_MAX_KERNEL_ADDRESS. */ -static int -pmap_clearbit(struct vm_page *m, u_int maskbits) +static __inline vm_paddr_t +pmap_preboot_pt2pg_setup(vm_offset_t va) { - struct l2_bucket *l2b; - struct pv_entry *pv, *pve, *next_pv; - struct md_page *pvh; - pd_entry_t *pl1pd; - pt_entry_t *ptep, npte, opte; - pmap_t pmap; - vm_offset_t va; - u_int oflags; - int count = 0; + pt2_entry_t *pte2p, pte2; + vm_paddr_t pt2pg_pa; - rw_wlock(&pvh_global_lock); - if ((m->flags & PG_FICTITIOUS) != 0) - goto small_mappings; + /* Get associated entry in PT2TAB. */ + pte2p = kern_pt2tab_entry(va); - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_list, next_pv) { - va = pv->pv_va; - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - KASSERT((*pl1pd & L1_TYPE_MASK) == L1_S_PROTO, - ("pmap_clearbit: valid section mapping expected")); - if ((maskbits & PVF_WRITE) && (pv->pv_flags & PVF_WRITE)) - (void)pmap_demote_section(pmap, va); - else if ((maskbits & PVF_REF) && L1_S_REFERENCED(*pl1pd)) { - if (pmap_demote_section(pmap, va)) { - if ((pv->pv_flags & PVF_WIRED) == 0) { - /* - * Remove the mapping to a single page - * so that a subsequent access may - * repromote. Since the underlying - * l2_bucket is fully populated, this - * removal never frees an entire - * l2_bucket. - */ - va += (VM_PAGE_TO_PHYS(m) & - L1_S_OFFSET); - l2b = pmap_get_l2_bucket(pmap, va); - KASSERT(l2b != NULL, - ("pmap_clearbit: no l2 bucket for " - "va 0x%#x, pmap 0x%p", va, pmap)); - ptep = &l2b->l2b_kva[l2pte_index(va)]; - *ptep = 0; - PTE_SYNC(ptep); - pmap_free_l2_bucket(pmap, l2b, 1); - pve = pmap_remove_pv(m, pmap, va); - KASSERT(pve != NULL, ("pmap_clearbit: " - "no PV entry for managed mapping")); - pmap_free_pv_entry(pmap, pve); + /* Just return, if PT2s page exists already. */ + pte2 = pt2tab_load(pte2p); + if (pte2_is_valid(pte2)) + return (pte2_pa(pte2)); - } - } - } else if ((maskbits & PVF_MOD) && L1_S_WRITABLE(*pl1pd)) { - if (pmap_demote_section(pmap, va)) { - if ((pv->pv_flags & PVF_WIRED) == 0) { - /* - * Write protect the mapping to a - * single page so that a subsequent - * write access may repromote. - */ - va += (VM_PAGE_TO_PHYS(m) & - L1_S_OFFSET); - l2b = pmap_get_l2_bucket(pmap, va); - KASSERT(l2b != NULL, - ("pmap_clearbit: no l2 bucket for " - "va 0x%#x, pmap 0x%p", va, pmap)); - ptep = &l2b->l2b_kva[l2pte_index(va)]; - if ((*ptep & L2_S_PROTO) != 0) { - pve = pmap_find_pv(&m->md, - pmap, va); - KASSERT(pve != NULL, - ("pmap_clearbit: no PV " - "entry for managed mapping")); - pve->pv_flags &= ~PVF_WRITE; - *ptep |= L2_APX; - PTE_SYNC(ptep); - } - } - } - } - PMAP_UNLOCK(pmap); - } + KASSERT(va >= VM_MAX_KERNEL_ADDRESS, + ("%s: NKPT2PG too small", __func__)); -small_mappings: - if (TAILQ_EMPTY(&m->md.pv_list)) { - rw_wunlock(&pvh_global_lock); - return (0); - } - /* - * Loop over all current mappings setting/clearing as appropos + * Allocate page for PT2s and insert it to PT2TAB. + * In other words, map it into PT2MAP space. */ - TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) { - va = pv->pv_va; - pmap = PV_PMAP(pv); - oflags = pv->pv_flags; - pv->pv_flags &= ~maskbits; + pt2pg_pa = pmap_preboot_get_pages(1); + pt2tab_store(pte2p, PTE2_KPT(pt2pg_pa)); - PMAP_LOCK(pmap); + /* Zero all PT2s in allocated page. */ + bzero((void*)pt2map_pt2pg(va), PAGE_SIZE); + pte2_sync_range((pt2_entry_t *)pt2map_pt2pg(va), PAGE_SIZE); - l2b = pmap_get_l2_bucket(pmap, va); - KASSERT(l2b != NULL, ("pmap_clearbit: no l2 bucket for " - "va 0x%#x, pmap 0x%p", va, pmap)); - - ptep = &l2b->l2b_kva[l2pte_index(va)]; - npte = opte = *ptep; - - if (maskbits & (PVF_WRITE | PVF_MOD)) { - /* make the pte read only */ - npte |= L2_APX; - } - - if (maskbits & PVF_REF) { - /* - * Clear referenced flag in PTE so that we - * will take a flag fault the next time the mapping - * is referenced. - */ - npte &= ~L2_S_REF; - } - - CTR4(KTR_PMAP,"clearbit: pmap:%p bits:%x pte:%x->%x", - pmap, maskbits, opte, npte); - if (npte != opte) { - count++; - *ptep = npte; - PTE_SYNC(ptep); - /* Flush the TLB entry if a current pmap. */ - if (PTE_BEEN_EXECD(opte)) - cpu_tlb_flushID_SE(pv->pv_va); - else if (PTE_BEEN_REFD(opte)) - cpu_tlb_flushD_SE(pv->pv_va); - cpu_cpwait(); - } - - PMAP_UNLOCK(pmap); - - } - - if (maskbits & PVF_WRITE) - vm_page_aflag_clear(m, PGA_WRITEABLE); - rw_wunlock(&pvh_global_lock); - return (count); + return (pt2pg_pa); } /* - * main pv_entry manipulation functions: - * pmap_enter_pv: enter a mapping onto a vm_page list - * pmap_remove_pv: remove a mappiing from a vm_page list - * - * NOTE: pmap_enter_pv expects to lock the pvh itself - * pmap_remove_pv expects the caller to lock the pvh before calling + * Setup L2 page table for given KVA. + * Used in pre-bootstrap epoch. */ - -/* - * pmap_enter_pv: enter a mapping onto a vm_page's PV list - * - * => caller should hold the proper lock on pvh_global_lock - * => caller should have pmap locked - * => we will (someday) gain the lock on the vm_page's PV list - * => caller should adjust ptp's wire_count before calling - * => caller should not adjust pmap's wire_count - */ static void -pmap_enter_pv(struct vm_page *m, struct pv_entry *pve, pmap_t pmap, - vm_offset_t va, u_int flags) +pmap_preboot_pt2_setup(vm_offset_t va) { + pt1_entry_t *pte1p; + vm_paddr_t pt2pg_pa, pt2_pa; - rw_assert(&pvh_global_lock, RA_WLOCKED); + /* Setup PT2's page. */ + pt2pg_pa = pmap_preboot_pt2pg_setup(va); + pt2_pa = page_pt2pa(pt2pg_pa, pte1_index(va)); - PMAP_ASSERT_LOCKED(pmap); - pve->pv_va = va; - pve->pv_flags = flags; - - TAILQ_INSERT_HEAD(&m->md.pv_list, pve, pv_list); - if (pve->pv_flags & PVF_WIRED) - ++pmap->pm_stats.wired_count; + /* Insert PT2 to PT1. */ + pte1p = kern_pte1(va); + pte1_store(pte1p, PTE1_LINK(pt2_pa)); } /* - * - * pmap_find_pv: Find a pv entry - * - * => caller should hold lock on vm_page + * Get L2 page entry associated with given KVA. + * Used in pre-bootstrap epoch. */ -static PMAP_INLINE struct pv_entry * -pmap_find_pv(struct md_page *md, pmap_t pmap, vm_offset_t va) +static __inline pt2_entry_t* +pmap_preboot_vtopte2(vm_offset_t va) { - struct pv_entry *pv; + pt1_entry_t *pte1p; - rw_assert(&pvh_global_lock, RA_WLOCKED); - TAILQ_FOREACH(pv, &md->pv_list, pv_list) - if (pmap == PV_PMAP(pv) && va == pv->pv_va) - break; + /* Setup PT2 if needed. */ + pte1p = kern_pte1(va); + if (!pte1_is_valid(pte1_load(pte1p))) /* XXX - sections ?! */ + pmap_preboot_pt2_setup(va); - return (pv); + return (pt2map_entry(va)); } /* - * vector_page_setprot: - * - * Manipulate the protection of the vector page. + * Pre-bootstrap epoch page(s) mapping(s). */ void -vector_page_setprot(int prot) +pmap_preboot_map_pages(vm_paddr_t pa, vm_offset_t va, u_int num) { - struct l2_bucket *l2b; - pt_entry_t *ptep; + u_int i; + pt2_entry_t *pte2p; - l2b = pmap_get_l2_bucket(pmap_kernel(), vector_page); - - ptep = &l2b->l2b_kva[l2pte_index(vector_page)]; - /* - * Set referenced flag. - * Vectors' page is always desired - * to be allowed to reside in TLB. - */ - *ptep |= L2_S_REF; - - pmap_set_prot(ptep, prot|VM_PROT_EXECUTE, 0); - PTE_SYNC(ptep); - cpu_tlb_flushID_SE(vector_page); - cpu_cpwait(); + /* Map all the pages. */ + for (i = 0; i < num; i++) { + pte2p = pmap_preboot_vtopte2(va); + pte2_store(pte2p, PTE2_KRW(pa)); + va += PAGE_SIZE; + pa += PAGE_SIZE; + } } -static void -pmap_set_prot(pt_entry_t *ptep, vm_prot_t prot, uint8_t user) -{ - - *ptep &= ~(L2_S_PROT_MASK | L2_XN); - - if (!(prot & VM_PROT_EXECUTE)) - *ptep |= L2_XN; - - /* Set defaults first - kernel read access */ - *ptep |= L2_APX; - *ptep |= L2_S_PROT_R; - /* Now tune APs as desired */ - if (user) - *ptep |= L2_S_PROT_U; - - if (prot & VM_PROT_WRITE) - *ptep &= ~(L2_APX); -} - /* - * pmap_remove_pv: try to remove a mapping from a pv_list - * - * => caller should hold proper lock on pmap_main_lock - * => pmap should be locked - * => caller should hold lock on vm_page [so that attrs can be adjusted] - * => caller should adjust ptp's wire_count and free PTP if needed - * => caller should NOT adjust pmap's wire_count - * => we return the removed pve + * Pre-bootstrap epoch virtual space alocator. */ -static struct pv_entry * -pmap_remove_pv(struct vm_page *m, pmap_t pmap, vm_offset_t va) +vm_offset_t +pmap_preboot_reserve_pages(u_int num) { - struct pv_entry *pve; + u_int i; + vm_offset_t start, va; + pt2_entry_t *pte2p; - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_ASSERT_LOCKED(pmap); + /* Allocate virtual space. */ + start = va = virtual_avail; + virtual_avail += num * PAGE_SIZE; - pve = pmap_find_pv(&m->md, pmap, va); /* find corresponding pve */ - if (pve != NULL) { - TAILQ_REMOVE(&m->md.pv_list, pve, pv_list); - if (pve->pv_flags & PVF_WIRED) - --pmap->pm_stats.wired_count; + /* Zero the mapping. */ + for (i = 0; i < num; i++) { + pte2p = pmap_preboot_vtopte2(va); + pte2_store(pte2p, 0); + va += PAGE_SIZE; } - if (TAILQ_EMPTY(&m->md.pv_list)) - vm_page_aflag_clear(m, PGA_WRITEABLE); - return(pve); /* return removed pve */ + return (start); } /* - * - * pmap_modify_pv: Update pv flags - * - * => caller should hold lock on vm_page [so that attrs can be adjusted] - * => caller should NOT adjust pmap's wire_count - * => we return the old flags - * - * Modify a physical-virtual mapping in the pv table + * Pre-bootstrap epoch page(s) allocation and mapping(s). */ -static u_int -pmap_modify_pv(struct vm_page *m, pmap_t pmap, vm_offset_t va, - u_int clr_mask, u_int set_mask) +vm_offset_t +pmap_preboot_get_vpages(u_int num) { - struct pv_entry *npv; - u_int flags, oflags; + vm_paddr_t pa; + vm_offset_t va; - PMAP_ASSERT_LOCKED(pmap); - rw_assert(&pvh_global_lock, RA_WLOCKED); - if ((npv = pmap_find_pv(&m->md, pmap, va)) == NULL) - return (0); + /* Allocate physical page(s). */ + pa = pmap_preboot_get_pages(num); - /* - * There is at least one VA mapping this page. - */ - oflags = npv->pv_flags; - npv->pv_flags = flags = (oflags & ~clr_mask) | set_mask; + /* Allocate virtual space. */ + va = virtual_avail; + virtual_avail += num * PAGE_SIZE; - if ((flags ^ oflags) & PVF_WIRED) { - if (flags & PVF_WIRED) - ++pmap->pm_stats.wired_count; - else - --pmap->pm_stats.wired_count; - } + /* Map and zero all. */ + pmap_preboot_map_pages(pa, va, num); + bzero((void *)va, num * PAGE_SIZE); - return (oflags); + return (va); } -/* Function to set the debug level of the pmap code */ -#ifdef PMAP_DEBUG -void -pmap_debug(int level) -{ - pmap_debug_level = level; - dprintf("pmap_debug: level=%d\n", pmap_debug_level); -} -#endif /* PMAP_DEBUG */ - -void -pmap_pinit0(struct pmap *pmap) -{ - PDEBUG(1, printf("pmap_pinit0: pmap = %08x\n", (u_int32_t) pmap)); - - bcopy(kernel_pmap, pmap, sizeof(*pmap)); - bzero(&pmap->pm_mtx, sizeof(pmap->pm_mtx)); - PMAP_LOCK_INIT(pmap); - TAILQ_INIT(&pmap->pm_pvchunk); -} - /* - * Initialize a vm_page's machine-dependent fields. + * Pre-bootstrap epoch page mapping(s) with attributes. */ void -pmap_page_init(vm_page_t m) +pmap_preboot_map_attr(vm_paddr_t pa, vm_offset_t va, vm_size_t size, int prot, + int attr) { + u_int num; + u_int l1_attr, l1_prot; + pt1_entry_t *pte1p; + pt2_entry_t *pte2p; - TAILQ_INIT(&m->md.pv_list); - m->md.pv_memattr = VM_MEMATTR_DEFAULT; -} + l1_prot = ATTR_TO_L1(prot); + l1_attr = ATTR_TO_L1(attr); -static vm_offset_t -pmap_ptelist_alloc(vm_offset_t *head) -{ - pt_entry_t *pte; - vm_offset_t va; + /* Map all the pages. */ + num = round_page(size); + while (num > 0) { + if ((((va | pa) & PTE1_OFFSET) == 0) && (num >= PTE1_SIZE)) { + pte1p = kern_pte1(va); + pte1_store(pte1p, PTE1_KERN(pa, l1_prot, l1_attr)); + va += PTE1_SIZE; + pa += PTE1_SIZE; + num -= PTE1_SIZE; + } else { + pte2p = pmap_preboot_vtopte2(va); + pte2_store(pte2p, PTE2_KERN(pa, prot, attr)); + va += PAGE_SIZE; + pa += PAGE_SIZE; + num -= PAGE_SIZE; + } + } - va = *head; - if (va == 0) - return (va); /* Out of memory */ - pte = vtopte(va); - *head = *pte; - if ((*head & L2_TYPE_MASK) != L2_TYPE_INV) - panic("%s: va is not L2_TYPE_INV!", __func__); - *pte = 0; - return (va); } -static void -pmap_ptelist_free(vm_offset_t *head, vm_offset_t va) +/* + * Extract from the kernel page table the physical address + * that is mapped by the given virtual address "va". + */ +vm_paddr_t +pmap_kextract(vm_offset_t va) { - pt_entry_t *pte; + vm_paddr_t pa; + pt1_entry_t pte1; + pt2_entry_t pte2; - if ((va & L2_TYPE_MASK) != L2_TYPE_INV) - panic("%s: freeing va that is not L2_TYPE INV!", __func__); - pte = vtopte(va); - *pte = *head; /* virtual! L2_TYPE is L2_TYPE_INV though */ - *head = va; + pte1 = pte1_load(kern_pte1(va)); + if (pte1_is_section(pte1)) { + pa = pte1_pa(pte1) | (va & PTE1_OFFSET); + } else if (pte1_is_link(pte1)) { + /* + * We should beware of concurrent promotion that changes + * pte1 at this point. However, it's not a problem as PT2 + * page is preserved by promotion in PT2TAB. So even if + * it happens, using of PT2MAP is still safe. + * + * QQQ: However, concurrent removing is a problem which + * ends in abort on PT2MAP space. Locking must be used + * to deal with this. + */ + pte2 = pte2_load(pt2map_entry(va)); + pa = pte2_pa(pte2) | (va & PTE2_OFFSET); + } + else { + panic("%s: va %#x pte1 %#x", __func__, va, pte1); + } + return (pa); } -static void -pmap_ptelist_init(vm_offset_t *head, void *base, int npages) +/* + * Extract from the kernel page table the physical address + * that is mapped by the given virtual address "va". Also + * return L2 page table entry which maps the address. + * + * This is only intended to be used for panic dumps. + */ +vm_paddr_t +pmap_dump_kextract(vm_offset_t va, pt2_entry_t *pte2p) { - int i; - vm_offset_t va; + vm_paddr_t pa; + pt1_entry_t pte1; + pt2_entry_t pte2; - *head = 0; - for (i = npages - 1; i >= 0; i--) { - va = (vm_offset_t)base + i * PAGE_SIZE; - pmap_ptelist_free(head, va); + pte1 = pte1_load(kern_pte1(va)); + if (pte1_is_section(pte1)) { + pa = pte1_pa(pte1) | (va & PTE1_OFFSET); + pte2 = pa | ATTR_TO_L2(pte1) | PTE2_V; + } else if (pte1_is_link(pte1)) { + pte2 = pte2_load(pt2map_entry(va)); + pa = pte2_pa(pte2); + } else { + pte2 = 0; + pa = 0; } + if (pte2p != NULL) + *pte2p = pte2; + return (pa); } +/***************************************************************************** + * + * PMAP second stage initialization and utility functions + * for bootstrap epoch. + * + * After pmap_bootstrap() is called, the following functions for + * mappings can be used: + * + * void pmap_kenter(vm_offset_t va, vm_paddr_t pa); + * void pmap_kremove(vm_offset_t va); + * vm_offset_t pmap_map(vm_offset_t *virt, vm_paddr_t start, vm_paddr_t end, + * int prot); + * + * NOTE: This is not SMP coherent stage. And physical page allocation is not + * allowed during this stage. + * + *****************************************************************************/ + /* - * Initialize the pmap module. - * Called by vm_init, to initialize any structures that the pmap - * system needs to map virtual memory. + * Initialize kernel PMAP locks and lists, kernel_pmap itself, and + * reserve various virtual spaces for temporary mappings. */ void -pmap_init(void) +pmap_bootstrap(vm_offset_t firstaddr) { - vm_size_t s; - int i, pv_npg; + pt2_entry_t *unused __unused; + struct sysmaps *sysmaps; + u_int i; - l2zone = uma_zcreate("L2 Table", L2_TABLE_SIZE_REAL, pmap_l2ptp_ctor, - NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); - l2table_zone = uma_zcreate("L2 Table", sizeof(struct l2_dtable), NULL, - NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); - /* - * Are large page mappings supported and enabled? + * Initialize the kernel pmap (which is statically allocated). */ - TUNABLE_INT_FETCH("vm.pmap.sp_enabled", &sp_enabled); - if (sp_enabled) { - KASSERT(MAXPAGESIZES > 1 && pagesizes[1] == 0, - ("pmap_init: can't assign to pagesizes[1]")); - pagesizes[1] = NBPDR; - } + PMAP_LOCK_INIT(kernel_pmap); + kernel_l1pa = (vm_paddr_t)kern_pt1; /* for libkvm */ + kernel_pmap->pm_pt1 = kern_pt1; + kernel_pmap->pm_pt2tab = kern_pt2tab; + CPU_FILL(&kernel_pmap->pm_active); /* don't allow deactivation */ + TAILQ_INIT(&kernel_pmap->pm_pvchunk); /* - * Calculate the size of the pv head table for superpages. - * Handle the possibility that "vm_phys_segs[...].end" is zero. + * Initialize the global pv list lock. */ - pv_npg = trunc_1mpage(vm_phys_segs[vm_phys_nsegs - 1].end - - PAGE_SIZE) / NBPDR + 1; + rw_init(&pvh_global_lock, "pmap pv global"); - /* - * Allocate memory for the pv head table for superpages. - */ - s = (vm_size_t)(pv_npg * sizeof(struct md_page)); - s = round_page(s); - pv_table = (struct md_page *)kmem_malloc(kernel_arena, s, - M_WAITOK | M_ZERO); - for (i = 0; i < pv_npg; i++) - TAILQ_INIT(&pv_table[i].pv_list); + LIST_INIT(&allpmaps); /* - * Initialize the address space for the pv chunks. + * Request a spin mutex so that changes to allpmaps cannot be + * preempted by smp_rendezvous_cpus(). */ + mtx_init(&allpmaps_lock, "allpmaps", NULL, MTX_SPIN); + mtx_lock_spin(&allpmaps_lock); + LIST_INSERT_HEAD(&allpmaps, kernel_pmap, pm_list); + mtx_unlock_spin(&allpmaps_lock); - TUNABLE_INT_FETCH("vm.pmap.shpgperproc", &shpgperproc); - pv_entry_max = shpgperproc * maxproc + vm_cnt.v_page_count; - TUNABLE_INT_FETCH("vm.pmap.pv_entries", &pv_entry_max); - pv_entry_max = roundup(pv_entry_max, _NPCPV); - pv_entry_high_water = 9 * (pv_entry_max / 10); - - pv_maxchunks = MAX(pv_entry_max / _NPCPV, maxproc); - pv_chunkbase = (struct pv_chunk *)kva_alloc(PAGE_SIZE * pv_maxchunks); - - if (pv_chunkbase == NULL) - panic("pmap_init: not enough kvm for pv chunks"); - - pmap_ptelist_init(&pv_vafree, pv_chunkbase, pv_maxchunks); - /* - * Now it is safe to enable pv_table recording. + * Reserve some special page table entries/VA space for temporary + * mapping of pages. */ - PDEBUG(1, printf("pmap_init: done!\n")); -} +#define SYSMAP(c, p, v, n) do { \ + v = (c)pmap_preboot_reserve_pages(n); \ + p = pt2map_entry((vm_offset_t)v); \ + } while (0) -SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_max, CTLFLAG_RD, &pv_entry_max, 0, - "Max number of PV entries"); -SYSCTL_INT(_vm_pmap, OID_AUTO, shpgperproc, CTLFLAG_RD, &shpgperproc, 0, - "Page share factor per proc"); - -static SYSCTL_NODE(_vm_pmap, OID_AUTO, section, CTLFLAG_RD, 0, - "1MB page mapping counters"); - -static u_long pmap_section_demotions; -SYSCTL_ULONG(_vm_pmap_section, OID_AUTO, demotions, CTLFLAG_RD, - &pmap_section_demotions, 0, "1MB page demotions"); - -static u_long pmap_section_mappings; -SYSCTL_ULONG(_vm_pmap_section, OID_AUTO, mappings, CTLFLAG_RD, - &pmap_section_mappings, 0, "1MB page mappings"); - -static u_long pmap_section_p_failures; -SYSCTL_ULONG(_vm_pmap_section, OID_AUTO, p_failures, CTLFLAG_RD, - &pmap_section_p_failures, 0, "1MB page promotion failures"); - -static u_long pmap_section_promotions; -SYSCTL_ULONG(_vm_pmap_section, OID_AUTO, promotions, CTLFLAG_RD, - &pmap_section_promotions, 0, "1MB page promotions"); - -int -pmap_fault_fixup(pmap_t pmap, vm_offset_t va, vm_prot_t ftype, int user) -{ - struct l2_dtable *l2; - struct l2_bucket *l2b; - pd_entry_t *pl1pd, l1pd; - pt_entry_t *ptep, pte; - vm_paddr_t pa; - u_int l1idx; - int rv = 0; - - l1idx = L1_IDX(va); - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); /* - * Check and possibly fix-up L1 section mapping - * only when superpage mappings are enabled to speed up. + * Local CMAP1/CMAP2 are used for zeroing and copying pages. + * Local CMAP3 is used for data cache cleaning. + * Global CMAP3 is used for the idle process page zeroing. */ - if (sp_enabled) { - pl1pd = &pmap->pm_l1->l1_kva[l1idx]; - l1pd = *pl1pd; - if ((l1pd & L1_TYPE_MASK) == L1_S_PROTO) { - /* Catch an access to the vectors section */ - if (l1idx == L1_IDX(vector_page)) - goto out; - /* - * Stay away from the kernel mappings. - * None of them should fault from L1 entry. - */ - if (pmap == pmap_kernel()) - goto out; - /* - * Catch a forbidden userland access - */ - if (user && !(l1pd & L1_S_PROT_U)) - goto out; - /* - * Superpage is always either mapped read only - * or it is modified and permitted to be written - * by default. Therefore, process only reference - * flag fault and demote page in case of write fault. - */ - if ((ftype & VM_PROT_WRITE) && !L1_S_WRITABLE(l1pd) && - L1_S_REFERENCED(l1pd)) { - (void)pmap_demote_section(pmap, va); - goto out; - } else if (!L1_S_REFERENCED(l1pd)) { - /* Mark the page "referenced" */ - *pl1pd = l1pd | L1_S_REF; - PTE_SYNC(pl1pd); - goto l1_section_out; - } else - goto out; - } + for (i = 0; i < MAXCPU; i++) { + sysmaps = &sysmaps_pcpu[i]; + mtx_init(&sysmaps->lock, "SYSMAPS", NULL, MTX_DEF); + SYSMAP(caddr_t, sysmaps->CMAP1, sysmaps->CADDR1, 1); + SYSMAP(caddr_t, sysmaps->CMAP2, sysmaps->CADDR2, 1); + SYSMAP(caddr_t, sysmaps->CMAP3, sysmaps->CADDR3, 1); } + SYSMAP(caddr_t, CMAP3, CADDR3, 1); + /* - * If there is no l2_dtable for this address, then the process - * has no business accessing it. - * - * Note: This will catch userland processes trying to access - * kernel addresses. + * Crashdump maps. */ - l2 = pmap->pm_l2[L2_IDX(l1idx)]; - if (l2 == NULL) - goto out; + SYSMAP(caddr_t, unused, crashdumpmap, MAXDUMPPGS); /* - * Likewise if there is no L2 descriptor table + * _tmppt is used for reading arbitrary physical pages via /dev/mem. */ - l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; - if (l2b->l2b_kva == NULL) - goto out; + SYSMAP(caddr_t, unused, _tmppt, 1); /* - * Check the PTE itself. + * PADDR1 and PADDR2 are used by pmap_pte2_quick() and pmap_pte2(), + * respectively. PADDR3 is used by pmap_pte2_ddb(). */ - ptep = &l2b->l2b_kva[l2pte_index(va)]; - pte = *ptep; - if (pte == 0) - goto out; + SYSMAP(pt2_entry_t *, PMAP1, PADDR1, 1); + SYSMAP(pt2_entry_t *, PMAP2, PADDR2, 1); +#ifdef DDB + SYSMAP(pt2_entry_t *, PMAP3, PADDR3, 1); +#endif + mtx_init(&PMAP2mutex, "PMAP2", NULL, MTX_DEF); /* - * Catch a userland access to the vector page mapped at 0x0 + * Note that in very short time in initarm(), we are going to + * initialize phys_avail[] array and no futher page allocation + * can happen after that until vm subsystem will be initialized. */ - if (user && !(pte & L2_S_PROT_U)) - goto out; - if (va == vector_page) - goto out; + kernel_vm_end_new = kernel_vm_end; + virtual_end = vm_max_kernel_address; +} - pa = l2pte_pa(pte); - CTR5(KTR_PMAP, "pmap_fault_fix: pmap:%p va:%x pte:0x%x ftype:%x user:%x", - pmap, va, pte, ftype, user); - if ((ftype & VM_PROT_WRITE) && !(L2_S_WRITABLE(pte)) && - L2_S_REFERENCED(pte)) { - /* - * This looks like a good candidate for "page modified" - * emulation... - */ - struct pv_entry *pv; - struct vm_page *m; +static void +pmap_init_qpages(void) +{ + struct pcpu *pc; + int i; - /* Extract the physical address of the page */ - if ((m = PHYS_TO_VM_PAGE(pa)) == NULL) { - goto out; - } - /* Get the current flags for this page. */ + CPU_FOREACH(i) { + pc = pcpu_find(i); + pc->pc_qmap_addr = kva_alloc(PAGE_SIZE); + if (pc->pc_qmap_addr == 0) + panic("%s: unable to allocate KVA", __func__); + } +} +SYSINIT(qpages_init, SI_SUB_CPU, SI_ORDER_ANY, pmap_init_qpages, NULL); - pv = pmap_find_pv(&m->md, pmap, va); - if (pv == NULL) { - goto out; - } +/* + * The function can already be use in second initialization stage. + * As such, the function DOES NOT call pmap_growkernel() where PT2 + * allocation can happen. So if used, be sure that PT2 for given + * virtual address is allocated already! + * + * Add a wired page to the kva. + * Note: not SMP coherent. + */ +static __inline void +pmap_kenter_prot_attr(vm_offset_t va, vm_paddr_t pa, uint32_t prot, + uint32_t attr) +{ + pt1_entry_t *pte1p; + pt2_entry_t *pte2p; + pte1p = kern_pte1(va); + if (!pte1_is_valid(pte1_load(pte1p))) { /* XXX - sections ?! */ /* - * Do the flags say this page is writable? If not then it - * is a genuine write fault. If yes then the write fault is - * our fault as we did not reflect the write access in the - * PTE. Now we know a write has occurred we can correct this - * and also set the modified bit + * This is a very low level function, so PT2 and particularly + * PT2PG associated with given virtual address must be already + * allocated. It's a pain mainly during pmap initialization + * stage. However, called after pmap initialization with + * virtual address not under kernel_vm_end will lead to + * the same misery. */ - if ((pv->pv_flags & PVF_WRITE) == 0) { - goto out; - } - - vm_page_dirty(m); - - /* Re-enable write permissions for the page */ - *ptep = (pte & ~L2_APX); - PTE_SYNC(ptep); - rv = 1; - CTR1(KTR_PMAP, "pmap_fault_fix: new pte:0x%x", *ptep); - } else if (!L2_S_REFERENCED(pte)) { - /* - * This looks like a good candidate for "page referenced" - * emulation. - */ - struct pv_entry *pv; - struct vm_page *m; - - /* Extract the physical address of the page */ - if ((m = PHYS_TO_VM_PAGE(pa)) == NULL) - goto out; - /* Get the current flags for this page. */ - pv = pmap_find_pv(&m->md, pmap, va); - if (pv == NULL) - goto out; - - vm_page_aflag_set(m, PGA_REFERENCED); - - /* Mark the page "referenced" */ - *ptep = pte | L2_S_REF; - PTE_SYNC(ptep); - rv = 1; - CTR1(KTR_PMAP, "pmap_fault_fix: new pte:0x%x", *ptep); + if (!pte2_is_valid(pte2_load(kern_pt2tab_entry(va)))) + panic("%s: kernel PT2 not allocated!", __func__); } - /* - * We know there is a valid mapping here, so simply - * fix up the L1 if necessary. - */ - pl1pd = &pmap->pm_l1->l1_kva[l1idx]; - l1pd = l2b->l2b_phys | L1_C_DOM(pmap->pm_domain) | L1_C_PROTO; - if (*pl1pd != l1pd) { - *pl1pd = l1pd; - PTE_SYNC(pl1pd); - rv = 1; - } + pte2p = pt2map_entry(va); + pte2_store(pte2p, PTE2_KERN(pa, prot, attr)); +} -#ifdef DEBUG - /* - * If 'rv == 0' at this point, it generally indicates that there is a - * stale TLB entry for the faulting address. This happens when two or - * more processes are sharing an L1. Since we don't flush the TLB on - * a context switch between such processes, we can take domain faults - * for mappings which exist at the same VA in both processes. EVEN IF - * WE'VE RECENTLY FIXED UP THE CORRESPONDING L1 in pmap_enter(), for - * example. - * - * This is extremely likely to happen if pmap_enter() updated the L1 - * entry for a recently entered mapping. In this case, the TLB is - * flushed for the new mapping, but there may still be TLB entries for - * other mappings belonging to other processes in the 1MB range - * covered by the L1 entry. - * - * Since 'rv == 0', we know that the L1 already contains the correct - * value, so the fault must be due to a stale TLB entry. - * - * Since we always need to flush the TLB anyway in the case where we - * fixed up the L1, or frobbed the L2 PTE, we effectively deal with - * stale TLB entries dynamically. - * - * However, the above condition can ONLY happen if the current L1 is - * being shared. If it happens when the L1 is unshared, it indicates - * that other parts of the pmap are not doing their job WRT managing - * the TLB. - */ - if (rv == 0 && pmap->pm_l1->l1_domain_use_count == 1) { - printf("fixup: pmap %p, va 0x%08x, ftype %d - nothing to do!\n", - pmap, va, ftype); - printf("fixup: l2 %p, l2b %p, ptep %p, pl1pd %p\n", - l2, l2b, ptep, pl1pd); - printf("fixup: pte 0x%x, l1pd 0x%x, last code 0x%x\n", - pte, l1pd, last_fault_code); -#ifdef DDB - Debugger(); -#endif - } -#endif +static __inline void +pmap_kenter_attr(vm_offset_t va, vm_paddr_t pa, int attr) +{ -l1_section_out: - cpu_tlb_flushID_SE(va); - cpu_cpwait(); - - rv = 1; - -out: - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); - return (rv); + pmap_kenter_prot_attr(va, pa, PTE2_AP_KRW, attr); } -void -pmap_postinit(void) +PMAP_INLINE void +pmap_kenter(vm_offset_t va, vm_paddr_t pa) { - struct l2_bucket *l2b; - struct l1_ttable *l1; - pd_entry_t *pl1pt; - pt_entry_t *ptep, pte; - vm_offset_t va, eva; - u_int loop, needed; - needed = (maxproc / PMAP_DOMAINS) + ((maxproc % PMAP_DOMAINS) ? 1 : 0); - needed -= 1; - l1 = malloc(sizeof(*l1) * needed, M_VMPMAP, M_WAITOK); - - for (loop = 0; loop < needed; loop++, l1++) { - /* Allocate a L1 page table */ - va = (vm_offset_t)contigmalloc(L1_TABLE_SIZE, M_VMPMAP, 0, 0x0, - 0xffffffff, L1_TABLE_SIZE, 0); - - if (va == 0) - panic("Cannot allocate L1 KVM"); - - eva = va + L1_TABLE_SIZE; - pl1pt = (pd_entry_t *)va; - - while (va < eva) { - l2b = pmap_get_l2_bucket(pmap_kernel(), va); - ptep = &l2b->l2b_kva[l2pte_index(va)]; - pte = *ptep; - pte = (pte & ~L2_S_CACHE_MASK) | pte_l2_s_cache_mode_pt; - *ptep = pte; - PTE_SYNC(ptep); - cpu_tlb_flushID_SE(va); - cpu_cpwait(); - va += PAGE_SIZE; - } - pmap_init_l1(l1, pl1pt); - } -#ifdef DEBUG - printf("pmap_postinit: Allocated %d static L1 descriptor tables\n", - needed); -#endif + pmap_kenter_prot_attr(va, pa, PTE2_AP_KRW, PTE2_ATTR_NORMAL); } /* - * This is used to stuff certain critical values into the PCB where they - * can be accessed quickly from cpu_switch() et al. + * Remove a page from the kernel pagetables. + * Note: not SMP coherent. */ -void -pmap_set_pcb_pagedir(pmap_t pmap, struct pcb *pcb) +PMAP_INLINE void +pmap_kremove(vm_offset_t va) { - struct l2_bucket *l2b; + pt2_entry_t *pte2p; - pcb->pcb_pagedir = pmap->pm_l1->l1_physaddr; - pcb->pcb_dacr = (DOMAIN_CLIENT << (PMAP_DOMAIN_KERNEL * 2)) | - (DOMAIN_CLIENT << (pmap->pm_domain * 2)); - - if (vector_page < KERNBASE) { - pcb->pcb_pl1vec = &pmap->pm_l1->l1_kva[L1_IDX(vector_page)]; - l2b = pmap_get_l2_bucket(pmap, vector_page); - pcb->pcb_l1vec = l2b->l2b_phys | L1_C_PROTO | - L1_C_DOM(pmap->pm_domain) | L1_C_DOM(PMAP_DOMAIN_KERNEL); - } else - pcb->pcb_pl1vec = NULL; + pte2p = pt2map_entry(va); + pte2_clear(pte2p); } -void -pmap_activate(struct thread *td) +/* + * Share new kernel PT2PG with all pmaps. + * The caller is responsible for maintaining TLB consistency. + */ +static void +pmap_kenter_pt2tab(vm_offset_t va, pt2_entry_t npte2) { pmap_t pmap; - struct pcb *pcb; + pt2_entry_t *pte2p; - pmap = vmspace_pmap(td->td_proc->p_vmspace); - pcb = td->td_pcb; - - critical_enter(); - pmap_set_pcb_pagedir(pmap, pcb); - - if (td == curthread) { - u_int cur_dacr, cur_ttb; - - __asm __volatile("mrc p15, 0, %0, c2, c0, 0" : "=r"(cur_ttb)); - __asm __volatile("mrc p15, 0, %0, c3, c0, 0" : "=r"(cur_dacr)); - - cur_ttb &= ~(L1_TABLE_SIZE - 1); - - if (cur_ttb == (u_int)pcb->pcb_pagedir && - cur_dacr == pcb->pcb_dacr) { - /* - * No need to switch address spaces. - */ - critical_exit(); - return; - } - - - /* - * We MUST, I repeat, MUST fix up the L1 entry corresponding - * to 'vector_page' in the incoming L1 table before switching - * to it otherwise subsequent interrupts/exceptions (including - * domain faults!) will jump into hyperspace. - */ - if (pcb->pcb_pl1vec) { - *pcb->pcb_pl1vec = pcb->pcb_l1vec; - } - - cpu_domains(pcb->pcb_dacr); - cpu_setttb(pcb->pcb_pagedir); + mtx_lock_spin(&allpmaps_lock); + LIST_FOREACH(pmap, &allpmaps, pm_list) { + pte2p = pmap_pt2tab_entry(pmap, va); + pt2tab_store(pte2p, npte2); } - critical_exit(); + mtx_unlock_spin(&allpmaps_lock); } -static int -pmap_set_pt_cache_mode(pd_entry_t *kl1, vm_offset_t va) -{ - pd_entry_t *pdep, pde; - pt_entry_t *ptep, pte; - vm_offset_t pa; - int rv = 0; - - /* - * Make sure the descriptor itself has the correct cache mode - */ - pdep = &kl1[L1_IDX(va)]; - pde = *pdep; - - if (l1pte_section_p(pde)) { - if ((pde & L1_S_CACHE_MASK) != pte_l1_s_cache_mode_pt) { - *pdep = (pde & ~L1_S_CACHE_MASK) | - pte_l1_s_cache_mode_pt; - PTE_SYNC(pdep); - rv = 1; - } - } else { - pa = (vm_paddr_t)(pde & L1_C_ADDR_MASK); - ptep = (pt_entry_t *)kernel_pt_lookup(pa); - if (ptep == NULL) - panic("pmap_bootstrap: No L2 for L2 @ va %p\n", ptep); - - ptep = &ptep[l2pte_index(va)]; - pte = *ptep; - if ((pte & L2_S_CACHE_MASK) != pte_l2_s_cache_mode_pt) { - *ptep = (pte & ~L2_S_CACHE_MASK) | - pte_l2_s_cache_mode_pt; - PTE_SYNC(ptep); - rv = 1; - } - } - - return (rv); -} - +/* + * Share new kernel PTE1 with all pmaps. + * The caller is responsible for maintaining TLB consistency. + */ static void -pmap_alloc_specials(vm_offset_t *availp, int pages, vm_offset_t *vap, - pt_entry_t **ptep) +pmap_kenter_pte1(vm_offset_t va, pt1_entry_t npte1) { - vm_offset_t va = *availp; - struct l2_bucket *l2b; + pmap_t pmap; + pt1_entry_t *pte1p; - if (ptep) { - l2b = pmap_get_l2_bucket(pmap_kernel(), va); - if (l2b == NULL) - panic("pmap_alloc_specials: no l2b for 0x%x", va); - - *ptep = &l2b->l2b_kva[l2pte_index(va)]; + mtx_lock_spin(&allpmaps_lock); + LIST_FOREACH(pmap, &allpmaps, pm_list) { + pte1p = pmap_pte1(pmap, va); + pte1_store(pte1p, npte1); } - - *vap = va; - *availp = va + (PAGE_SIZE * pages); + mtx_unlock_spin(&allpmaps_lock); } /* - * Bootstrap the system enough to run with virtual memory. + * Used to map a range of physical addresses into kernel + * virtual address space. * - * On the arm this is called after mapping has already been enabled - * and just syncs the pmap module with what has already been done. - * [We can't call it easily with mapping off since the kernel is not - * mapped with PA == VA, hence we would have to relocate every address - * from the linked base (virtual) address "KERNBASE" to the actual - * (physical) address starting relative to 0] + * The value passed in '*virt' is a suggested virtual address for + * the mapping. Architectures which can support a direct-mapped + * physical to virtual region can return the appropriate address + * within that region, leaving '*virt' unchanged. Other + * architectures should map the pages starting at '*virt' and + * update '*virt' with the first usable address after the mapped + * region. + * + * NOTE: Read the comments above pmap_kenter_prot_attr() as + * the function is used herein! */ -#define PMAP_STATIC_L2_SIZE 16 - -void -pmap_bootstrap(vm_offset_t firstaddr, struct pv_addr *l1pt) +vm_offset_t +pmap_map(vm_offset_t *virt, vm_paddr_t start, vm_paddr_t end, int prot) { - static struct l1_ttable static_l1; - static struct l2_dtable static_l2[PMAP_STATIC_L2_SIZE]; - struct l1_ttable *l1 = &static_l1; - struct l2_dtable *l2; - struct l2_bucket *l2b; - struct czpages *czp; - pd_entry_t pde; - pd_entry_t *kernel_l1pt = (pd_entry_t *)l1pt->pv_va; - pt_entry_t *ptep; - vm_paddr_t pa; - vm_offset_t va; - vm_size_t size; - int i, l1idx, l2idx, l2next = 0; + vm_offset_t va, sva; + vm_paddr_t pte1_offset; + pt1_entry_t npte1; + u_int l1prot,l2prot; - PDEBUG(1, printf("firstaddr = %08x, lastaddr = %08x\n", - firstaddr, vm_max_kernel_address)); + PDEBUG(1, printf("%s: virt = %#x, start = %#x, end = %#x (size = %#x)," + " prot = %d\n", __func__, *virt, start, end, end - start, prot)); - virtual_avail = firstaddr; - kernel_pmap->pm_l1 = l1; - kernel_l1pa = l1pt->pv_pa; + l2prot = (prot & VM_PROT_WRITE) ? PTE2_AP_KRW : PTE1_AP_KR; + l2prot |= (prot & VM_PROT_EXECUTE) ? PTE2_X : PTE2_NX; + l1prot = ATTR_TO_L1(l2prot); + va = *virt; /* - * Scan the L1 translation table created by initarm() and create - * the required metadata for all valid mappings found in it. + * Does the physical address range's size and alignment permit at + * least one section mapping to be created? */ - for (l1idx = 0; l1idx < (L1_TABLE_SIZE / sizeof(pd_entry_t)); l1idx++) { - pde = kernel_l1pt[l1idx]; - + pte1_offset = start & PTE1_OFFSET; + if ((end - start) - ((PTE1_SIZE - pte1_offset) & PTE1_OFFSET) >= + PTE1_SIZE) { /* - * We're only interested in Coarse mappings. - * pmap_extract() can deal with section mappings without - * recourse to checking L2 metadata. + * Increase the starting virtual address so that its alignment + * does not preclude the use of section mappings. */ - if ((pde & L1_TYPE_MASK) != L1_TYPE_C) - continue; - - /* - * Lookup the KVA of this L2 descriptor table - */ - pa = (vm_paddr_t)(pde & L1_C_ADDR_MASK); - ptep = (pt_entry_t *)kernel_pt_lookup(pa); - - if (ptep == NULL) { - panic("pmap_bootstrap: No L2 for va 0x%x, pa 0x%lx", - (u_int)l1idx << L1_S_SHIFT, (long unsigned int)pa); + if ((va & PTE1_OFFSET) < pte1_offset) + va = pte1_trunc(va) + pte1_offset; + else if ((va & PTE1_OFFSET) > pte1_offset) + va = pte1_roundup(va) + pte1_offset; + } + sva = va; + while (start < end) { + if ((start & PTE1_OFFSET) == 0 && end - start >= PTE1_SIZE) { + KASSERT((va & PTE1_OFFSET) == 0, + ("%s: misaligned va %#x", __func__, va)); + npte1 = PTE1_KERN(start, l1prot, PTE1_ATTR_NORMAL); + pmap_kenter_pte1(va, npte1); + va += PTE1_SIZE; + start += PTE1_SIZE; + } else { + pmap_kenter_prot_attr(va, start, l2prot, + PTE2_ATTR_NORMAL); + va += PAGE_SIZE; + start += PAGE_SIZE; } - - /* - * Fetch the associated L2 metadata structure. - * Allocate a new one if necessary. - */ - if ((l2 = kernel_pmap->pm_l2[L2_IDX(l1idx)]) == NULL) { - if (l2next == PMAP_STATIC_L2_SIZE) - panic("pmap_bootstrap: out of static L2s"); - kernel_pmap->pm_l2[L2_IDX(l1idx)] = l2 = - &static_l2[l2next++]; - } - - /* - * One more L1 slot tracked... - */ - l2->l2_occupancy++; - - /* - * Fill in the details of the L2 descriptor in the - * appropriate bucket. - */ - l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; - l2b->l2b_kva = ptep; - l2b->l2b_phys = pa; - l2b->l2b_l1idx = l1idx; - - /* - * Establish an initial occupancy count for this descriptor - */ - for (l2idx = 0; - l2idx < (L2_TABLE_SIZE_REAL / sizeof(pt_entry_t)); - l2idx++) { - if ((ptep[l2idx] & L2_TYPE_MASK) != L2_TYPE_INV) { - l2b->l2b_occupancy++; - } - } - - /* - * Make sure the descriptor itself has the correct cache mode. - * If not, fix it, but whine about the problem. Port-meisters - * should consider this a clue to fix up their initarm() - * function. :) - */ - if (pmap_set_pt_cache_mode(kernel_l1pt, (vm_offset_t)ptep)) { - printf("pmap_bootstrap: WARNING! wrong cache mode for " - "L2 pte @ %p\n", ptep); - } } - - - /* - * Ensure the primary (kernel) L1 has the correct cache mode for - * a page table. Bitch if it is not correctly set. - */ - for (va = (vm_offset_t)kernel_l1pt; - va < ((vm_offset_t)kernel_l1pt + L1_TABLE_SIZE); va += PAGE_SIZE) { - if (pmap_set_pt_cache_mode(kernel_l1pt, va)) - printf("pmap_bootstrap: WARNING! wrong cache mode for " - "primary L1 @ 0x%x\n", va); - } - - cpu_dcache_wbinv_all(); - cpu_l2cache_wbinv_all(); - cpu_tlb_flushID(); - cpu_cpwait(); - - PMAP_LOCK_INIT(kernel_pmap); - CPU_FILL(&kernel_pmap->pm_active); - kernel_pmap->pm_domain = PMAP_DOMAIN_KERNEL; - TAILQ_INIT(&kernel_pmap->pm_pvchunk); - - /* - * Initialize the global pv list lock. - */ - rw_init(&pvh_global_lock, "pmap pv global"); - - /* - * Reserve some special page table entries/VA space for temporary - * mapping of pages that are being copied or zeroed. - */ - for (czp = cpu_czpages, i = 0; i < MAXCPU; ++i, ++czp) { - mtx_init(&czp->lock, "czpages", NULL, MTX_DEF); - pmap_alloc_specials(&virtual_avail, 1, &czp->srcva, &czp->srcptep); - pmap_set_pt_cache_mode(kernel_l1pt, (vm_offset_t)czp->srcptep); - pmap_alloc_specials(&virtual_avail, 1, &czp->dstva, &czp->dstptep); - pmap_set_pt_cache_mode(kernel_l1pt, (vm_offset_t)czp->dstptep); - } - - size = ((vm_max_kernel_address - pmap_curmaxkvaddr) + L1_S_OFFSET) / - L1_S_SIZE; - pmap_alloc_specials(&virtual_avail, - round_page(size * L2_TABLE_SIZE_REAL) / PAGE_SIZE, - &pmap_kernel_l2ptp_kva, NULL); - - size = (size + (L2_BUCKET_SIZE - 1)) / L2_BUCKET_SIZE; - pmap_alloc_specials(&virtual_avail, - round_page(size * sizeof(struct l2_dtable)) / PAGE_SIZE, - &pmap_kernel_l2dtable_kva, NULL); - - pmap_alloc_specials(&virtual_avail, - 1, (vm_offset_t*)&_tmppt, NULL); - pmap_alloc_specials(&virtual_avail, - MAXDUMPPGS, (vm_offset_t *)&crashdumpmap, NULL); - SLIST_INIT(&l1_list); - TAILQ_INIT(&l1_lru_list); - mtx_init(&l1_lru_lock, "l1 list lock", NULL, MTX_DEF); - pmap_init_l1(l1, kernel_l1pt); - cpu_dcache_wbinv_all(); - cpu_l2cache_wbinv_all(); - cpu_tlb_flushID(); - cpu_cpwait(); - - virtual_avail = round_page(virtual_avail); - virtual_end = vm_max_kernel_address; - kernel_vm_end = pmap_curmaxkvaddr; - - pmap_set_pcb_pagedir(kernel_pmap, thread0.td_pcb); + tlb_flush_range(sva, va - sva); + *virt = va; + return (sva); } - -/*************************************************** - * Pmap allocation/deallocation routines. - ***************************************************/ - /* - * Release any resources held by the given physical map. - * Called when a pmap initialized by pmap_pinit is being released. - * Should only be called if the map contains no valid mappings. + * Make a temporary mapping for a physical address. + * This is only intended to be used for panic dumps. */ -void -pmap_release(pmap_t pmap) +void * +pmap_kenter_temporary(vm_paddr_t pa, int i) { - struct pcb *pcb; + vm_offset_t va; - cpu_tlb_flushID(); - cpu_cpwait(); - if (vector_page < KERNBASE) { - struct pcb *curpcb = PCPU_GET(curpcb); - pcb = thread0.td_pcb; - if (pmap_is_current(pmap)) { - /* - * Frob the L1 entry corresponding to the vector - * page so that it contains the kernel pmap's domain - * number. This will ensure pmap_remove() does not - * pull the current vector page out from under us. - */ - critical_enter(); - *pcb->pcb_pl1vec = pcb->pcb_l1vec; - cpu_domains(pcb->pcb_dacr); - cpu_setttb(pcb->pcb_pagedir); - critical_exit(); - } - pmap_remove(pmap, vector_page, vector_page + PAGE_SIZE); - /* - * Make sure cpu_switch(), et al, DTRT. This is safe to do - * since this process has no remaining mappings of its own. - */ - curpcb->pcb_pl1vec = pcb->pcb_pl1vec; - curpcb->pcb_l1vec = pcb->pcb_l1vec; - curpcb->pcb_dacr = pcb->pcb_dacr; - curpcb->pcb_pagedir = pcb->pcb_pagedir; + /* QQQ: 'i' should be less or equal to MAXDUMPPGS. */ - } - pmap_free_l1(pmap); - - dprintf("pmap_release()\n"); + va = (vm_offset_t)crashdumpmap + (i * PAGE_SIZE); + pmap_kenter(va, pa); + tlb_flush_local(va); + return ((void *)crashdumpmap); } +/************************************* + * + * TLB & cache maintenance routines. + * + *************************************/ /* - * Helper function for pmap_grow_l2_bucket() + * We inline these within pmap.c for speed. */ -static __inline int -pmap_grow_map(vm_offset_t va, pt_entry_t cache_mode, vm_paddr_t *pap) +PMAP_INLINE void +pmap_tlb_flush(pmap_t pmap, vm_offset_t va) { - struct l2_bucket *l2b; - pt_entry_t *ptep; - vm_paddr_t pa; - struct vm_page *m; - m = vm_page_alloc(NULL, 0, VM_ALLOC_NOOBJ | VM_ALLOC_WIRED); - if (m == NULL) - return (1); - pa = VM_PAGE_TO_PHYS(m); - - if (pap) - *pap = pa; - - l2b = pmap_get_l2_bucket(pmap_kernel(), va); - - ptep = &l2b->l2b_kva[l2pte_index(va)]; - *ptep = L2_S_PROTO | pa | cache_mode | L2_S_REF; - pmap_set_prot(ptep, VM_PROT_READ | VM_PROT_WRITE, 0); - PTE_SYNC(ptep); - cpu_tlb_flushD_SE(va); - cpu_cpwait(); - - return (0); + if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) + tlb_flush(va); } -/* - * This is the same as pmap_alloc_l2_bucket(), except that it is only - * used by pmap_growkernel(). - */ -static __inline struct l2_bucket * -pmap_grow_l2_bucket(pmap_t pmap, vm_offset_t va) +PMAP_INLINE void +pmap_tlb_flush_range(pmap_t pmap, vm_offset_t sva, vm_size_t size) { - struct l2_dtable *l2; - struct l2_bucket *l2b; - struct l1_ttable *l1; - pd_entry_t *pl1pd; - u_short l1idx; - vm_offset_t nva; - l1idx = L1_IDX(va); - - if ((l2 = pmap->pm_l2[L2_IDX(l1idx)]) == NULL) { - /* - * No mapping at this address, as there is - * no entry in the L1 table. - * Need to allocate a new l2_dtable. - */ - nva = pmap_kernel_l2dtable_kva; - if ((nva & PAGE_MASK) == 0) { - /* - * Need to allocate a backing page - */ - if (pmap_grow_map(nva, pte_l2_s_cache_mode, NULL)) - return (NULL); - } - - l2 = (struct l2_dtable *)nva; - nva += sizeof(struct l2_dtable); - - if ((nva & PAGE_MASK) < (pmap_kernel_l2dtable_kva & - PAGE_MASK)) { - /* - * The new l2_dtable straddles a page boundary. - * Map in another page to cover it. - */ - if (pmap_grow_map(nva, pte_l2_s_cache_mode, NULL)) - return (NULL); - } - - pmap_kernel_l2dtable_kva = nva; - - /* - * Link it into the parent pmap - */ - pmap->pm_l2[L2_IDX(l1idx)] = l2; - memset(l2, 0, sizeof(*l2)); - } - - l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; - - /* - * Fetch pointer to the L2 page table associated with the address. - */ - if (l2b->l2b_kva == NULL) { - pt_entry_t *ptep; - - /* - * No L2 page table has been allocated. Chances are, this - * is because we just allocated the l2_dtable, above. - */ - nva = pmap_kernel_l2ptp_kva; - ptep = (pt_entry_t *)nva; - if ((nva & PAGE_MASK) == 0) { - /* - * Need to allocate a backing page - */ - if (pmap_grow_map(nva, pte_l2_s_cache_mode_pt, - &pmap_kernel_l2ptp_phys)) - return (NULL); - } - memset(ptep, 0, L2_TABLE_SIZE_REAL); - l2->l2_occupancy++; - l2b->l2b_kva = ptep; - l2b->l2b_l1idx = l1idx; - l2b->l2b_phys = pmap_kernel_l2ptp_phys; - - pmap_kernel_l2ptp_kva += L2_TABLE_SIZE_REAL; - pmap_kernel_l2ptp_phys += L2_TABLE_SIZE_REAL; - } - - /* Distribute new L1 entry to all other L1s */ - SLIST_FOREACH(l1, &l1_list, l1_link) { - pl1pd = &l1->l1_kva[L1_IDX(va)]; - *pl1pd = l2b->l2b_phys | L1_C_DOM(PMAP_DOMAIN_KERNEL) | - L1_C_PROTO; - PTE_SYNC(pl1pd); - } - cpu_tlb_flushID_SE(va); - cpu_cpwait(); - - return (l2b); + if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) + tlb_flush_range(sva, size); } - /* - * grow the number of kernel page table entries, if needed + * Abuse the pte2 nodes for unmapped kva to thread a kva freelist through. + * Requirements: + * - Must deal with pages in order to ensure that none of the PTE2_* bits + * are ever set, PTE2_V in particular. + * - Assumes we can write to pte2s without pte2_store() atomic ops. + * - Assumes nothing will ever test these addresses for 0 to indicate + * no mapping instead of correctly checking PTE2_V. + * - Assumes a vm_offset_t will fit in a pte2 (true for arm). + * Because PTE2_V is never set, there can be no mappings to invalidate. */ -void -pmap_growkernel(vm_offset_t addr) +static vm_offset_t +pmap_pte2list_alloc(vm_offset_t *head) { - pmap_t kpmap = pmap_kernel(); + pt2_entry_t *pte2p; + vm_offset_t va; - if (addr <= pmap_curmaxkvaddr) - return; /* we are OK */ - - /* - * whoops! we need to add kernel PTPs - */ - - /* Map 1MB at a time */ - for (; pmap_curmaxkvaddr < addr; pmap_curmaxkvaddr += L1_S_SIZE) - pmap_grow_l2_bucket(kpmap, pmap_curmaxkvaddr); - - kernel_vm_end = pmap_curmaxkvaddr; + va = *head; + if (va == 0) + panic("pmap_ptelist_alloc: exhausted ptelist KVA"); + pte2p = pt2map_entry(va); + *head = *pte2p; + if (*head & PTE2_V) + panic("%s: va with PTE2_V set!", __func__); + *pte2p = 0; + return (va); } -/* - * Returns TRUE if the given page is mapped individually or as part of - * a 1MB section. Otherwise, returns FALSE. - */ -boolean_t -pmap_page_is_mapped(vm_page_t m) +static void +pmap_pte2list_free(vm_offset_t *head, vm_offset_t va) { - boolean_t rv; + pt2_entry_t *pte2p; - if ((m->oflags & VPO_UNMANAGED) != 0) - return (FALSE); - rw_wlock(&pvh_global_lock); - rv = !TAILQ_EMPTY(&m->md.pv_list) || - ((m->flags & PG_FICTITIOUS) == 0 && - !TAILQ_EMPTY(&pa_to_pvh(VM_PAGE_TO_PHYS(m))->pv_list)); - rw_wunlock(&pvh_global_lock); - return (rv); + if (va & PTE2_V) + panic("%s: freeing va with PTE2_V set!", __func__); + pte2p = pt2map_entry(va); + *pte2p = *head; /* virtual! PTE2_V is 0 though */ + *head = va; } -/* - * Remove all pages from specified address space - * this aids process exit speeds. Also, this code - * is special cased for current process only, but - * can have the more generic (and slightly slower) - * mode enabled. This is much faster than pmap_remove - * in the case of running down an entire address space. - */ -void -pmap_remove_pages(pmap_t pmap) +static void +pmap_pte2list_init(vm_offset_t *head, void *base, int npages) { - struct pv_entry *pv; - struct l2_bucket *l2b = NULL; - struct pv_chunk *pc, *npc; - struct md_page *pvh; - pd_entry_t *pl1pd, l1pd; - pt_entry_t *ptep; - vm_page_t m, mt; + int i; vm_offset_t va; - uint32_t inuse, bitmask; - int allfree, bit, field, idx; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); + *head = 0; + for (i = npages - 1; i >= 0; i--) { + va = (vm_offset_t)base + i * PAGE_SIZE; + pmap_pte2list_free(head, va); + } +} - TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { - allfree = 1; - for (field = 0; field < _NPCM; field++) { - inuse = ~pc->pc_map[field] & pc_freemask[field]; - while (inuse != 0) { - bit = ffs(inuse) - 1; - bitmask = 1ul << bit; - idx = field * sizeof(inuse) * NBBY + bit; - pv = &pc->pc_pventry[idx]; - va = pv->pv_va; - inuse &= ~bitmask; - if (pv->pv_flags & PVF_WIRED) { - /* Cannot remove wired pages now. */ - allfree = 0; - continue; - } - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - l1pd = *pl1pd; - l2b = pmap_get_l2_bucket(pmap, va); - if ((l1pd & L1_TYPE_MASK) == L1_S_PROTO) { - pvh = pa_to_pvh(l1pd & L1_S_FRAME); - TAILQ_REMOVE(&pvh->pv_list, pv, pv_list); - if (TAILQ_EMPTY(&pvh->pv_list)) { - m = PHYS_TO_VM_PAGE(l1pd & L1_S_FRAME); - KASSERT((vm_offset_t)m >= KERNBASE, - ("Trying to access non-existent page " - "va %x l1pd %x", trunc_1mpage(va), l1pd)); - for (mt = m; mt < &m[L2_PTE_NUM_TOTAL]; mt++) { - if (TAILQ_EMPTY(&mt->md.pv_list)) - vm_page_aflag_clear(mt, PGA_WRITEABLE); - } - } - if (l2b != NULL) { - KASSERT(l2b->l2b_occupancy == L2_PTE_NUM_TOTAL, - ("pmap_remove_pages: l2_bucket occupancy error")); - pmap_free_l2_bucket(pmap, l2b, L2_PTE_NUM_TOTAL); - } - pmap->pm_stats.resident_count -= L2_PTE_NUM_TOTAL; - *pl1pd = 0; - PTE_SYNC(pl1pd); - } else { - KASSERT(l2b != NULL, - ("No L2 bucket in pmap_remove_pages")); - ptep = &l2b->l2b_kva[l2pte_index(va)]; - m = PHYS_TO_VM_PAGE(l2pte_pa(*ptep)); - KASSERT((vm_offset_t)m >= KERNBASE, - ("Trying to access non-existent page " - "va %x pte %x", va, *ptep)); - TAILQ_REMOVE(&m->md.pv_list, pv, pv_list); - if (TAILQ_EMPTY(&m->md.pv_list) && - (m->flags & PG_FICTITIOUS) == 0) { - pvh = pa_to_pvh(l2pte_pa(*ptep)); - if (TAILQ_EMPTY(&pvh->pv_list)) - vm_page_aflag_clear(m, PGA_WRITEABLE); - } - *ptep = 0; - PTE_SYNC(ptep); - pmap_free_l2_bucket(pmap, l2b, 1); - pmap->pm_stats.resident_count--; - } +/***************************************************************************** + * + * PMAP third and final stage initialization. + * + * After pmap_init() is called, PMAP subsystem is fully initialized. + * + *****************************************************************************/ - /* Mark free */ - PV_STAT(pv_entry_frees++); - PV_STAT(pv_entry_spare++); - pv_entry_count--; - pc->pc_map[field] |= bitmask; - } - } - if (allfree) { - TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - pmap_free_pv_chunk(pc); - } +SYSCTL_NODE(_vm, OID_AUTO, pmap, CTLFLAG_RD, 0, "VM/pmap parameters"); - } +SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_max, CTLFLAG_RD, &pv_entry_max, 0, + "Max number of PV entries"); +SYSCTL_INT(_vm_pmap, OID_AUTO, shpgperproc, CTLFLAG_RD, &shpgperproc, 0, + "Page share factor per proc"); - rw_wunlock(&pvh_global_lock); - cpu_tlb_flushID(); - cpu_cpwait(); - PMAP_UNLOCK(pmap); -} +static u_long nkpt2pg = NKPT2PG; +SYSCTL_ULONG(_vm_pmap, OID_AUTO, nkpt2pg, CTLFLAG_RD, + &nkpt2pg, 0, "Pre-allocated pages for kernel PT2s"); -static void -pmap_init_qpages(void) -{ - struct pcpu *pc; - struct l2_bucket *l2b; - int i; +static int sp_enabled = 1; +SYSCTL_INT(_vm_pmap, OID_AUTO, sp_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, + &sp_enabled, 0, "Are large page mappings enabled?"); - CPU_FOREACH(i) { - pc = pcpu_find(i); - pc->pc_qmap_addr = kva_alloc(PAGE_SIZE); - if (pc->pc_qmap_addr == 0) - panic("pmap_init_qpages: unable to allocate KVA"); +static SYSCTL_NODE(_vm_pmap, OID_AUTO, pte1, CTLFLAG_RD, 0, + "1MB page mapping counters"); - l2b = pmap_get_l2_bucket(pmap_kernel(), pc->pc_qmap_addr); - if (l2b == NULL) - l2b = pmap_grow_l2_bucket(pmap_kernel(), - pc->pc_qmap_addr); - if (l2b == NULL) - panic("pmap_alloc_specials: no l2b for 0x%x", - pc->pc_qmap_addr); - pc->pc_qmap_pte = &l2b->l2b_kva[l2pte_index(pc->pc_qmap_addr)]; - } -} +static u_long pmap_pte1_demotions; +SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, demotions, CTLFLAG_RD, + &pmap_pte1_demotions, 0, "1MB page demotions"); -SYSINIT(qpages_init, SI_SUB_CPU, SI_ORDER_ANY, pmap_init_qpages, NULL); +static u_long pmap_pte1_mappings; +SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, mappings, CTLFLAG_RD, + &pmap_pte1_mappings, 0, "1MB page mappings"); -/*************************************************** - * Low level mapping routines..... - ***************************************************/ +static u_long pmap_pte1_p_failures; +SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, p_failures, CTLFLAG_RD, + &pmap_pte1_p_failures, 0, "1MB page promotion failures"); -#ifdef ARM_HAVE_SUPERSECTIONS -/* Map a super section into the KVA. */ +static u_long pmap_pte1_promotions; +SYSCTL_ULONG(_vm_pmap_pte1, OID_AUTO, promotions, CTLFLAG_RD, + &pmap_pte1_promotions, 0, "1MB page promotions"); -void -pmap_kenter_supersection(vm_offset_t va, uint64_t pa, int flags) +static __inline ttb_entry_t +pmap_ttb_get(pmap_t pmap) { - pd_entry_t pd = L1_S_PROTO | L1_S_SUPERSEC | (pa & L1_SUP_FRAME) | - (((pa >> 32) & 0xf) << 20) | L1_S_PROT(PTE_KERNEL, - VM_PROT_READ|VM_PROT_WRITE|VM_PROT_EXECUTE) | - L1_S_DOM(PMAP_DOMAIN_KERNEL); - struct l1_ttable *l1; - vm_offset_t va0, va_end; - KASSERT(((va | pa) & L1_SUP_OFFSET) == 0, - ("Not a valid super section mapping")); - if (flags & SECTION_CACHE) - pd |= pte_l1_s_cache_mode; - else if (flags & SECTION_PT) - pd |= pte_l1_s_cache_mode_pt; - - va0 = va & L1_SUP_FRAME; - va_end = va + L1_SUP_SIZE; - SLIST_FOREACH(l1, &l1_list, l1_link) { - va = va0; - for (; va < va_end; va += L1_S_SIZE) { - l1->l1_kva[L1_IDX(va)] = pd; - PTE_SYNC(&l1->l1_kva[L1_IDX(va)]); - } - } + return (vtophys(pmap->pm_pt1) | ttb_flags); } -#endif -/* Map a section into the KVA. */ - +/* + * Initialize a vm_page's machine-dependent fields. + * + * Variations: + * 1. Pages for L2 page tables are always not managed. So, pv_list and + * pt2_wirecount can share same physical space. However, proper + * initialization on a page alloc for page tables and reinitialization + * on the page free must be ensured. + */ void -pmap_kenter_section(vm_offset_t va, vm_offset_t pa, int flags) +pmap_page_init(vm_page_t m) { - pd_entry_t pd = L1_S_PROTO | pa | L1_S_PROT(PTE_KERNEL, - VM_PROT_READ|VM_PROT_WRITE|VM_PROT_EXECUTE) | L1_S_REF | - L1_S_DOM(PMAP_DOMAIN_KERNEL); - struct l1_ttable *l1; - KASSERT(((va | pa) & L1_S_OFFSET) == 0, - ("Not a valid section mapping")); - if (flags & SECTION_CACHE) - pd |= pte_l1_s_cache_mode; - else if (flags & SECTION_PT) - pd |= pte_l1_s_cache_mode_pt; - - SLIST_FOREACH(l1, &l1_list, l1_link) { - l1->l1_kva[L1_IDX(va)] = pd; - PTE_SYNC(&l1->l1_kva[L1_IDX(va)]); - } - cpu_tlb_flushID_SE(va); - cpu_cpwait(); + TAILQ_INIT(&m->md.pv_list); + pt2_wirecount_init(m); + m->md.pat_mode = PTE2_ATTR_NORMAL; } /* - * Make a temporary mapping for a physical address. This is only intended - * to be used for panic dumps. + * Virtualization for faster way how to zero whole page. */ -void * -pmap_kenter_temporary(vm_paddr_t pa, int i) +static __inline void +pagezero(void *page) { - vm_offset_t va; - va = (vm_offset_t)crashdumpmap + (i * PAGE_SIZE); - pmap_kenter(va, pa); - return ((void *)crashdumpmap); + bzero(page, PAGE_SIZE); } /* - * add a wired page to the kva - * note that in order for the mapping to take effect -- you - * should do a invltlb after doing the pmap_kenter... + * Zero L2 page table page. + * Use same KVA as in pmap_zero_page(). */ -static PMAP_INLINE void -pmap_kenter_internal(vm_offset_t va, vm_offset_t pa, int flags) +static __inline vm_paddr_t +pmap_pt2pg_zero(vm_page_t m) { - struct l2_bucket *l2b; - pt_entry_t *ptep; - pt_entry_t opte; + vm_paddr_t pa; + struct sysmaps *sysmaps; - PDEBUG(1, printf("pmap_kenter: va = %08x, pa = %08x\n", - (uint32_t) va, (uint32_t) pa)); + pa = VM_PAGE_TO_PHYS(m); + /* + * XXX: For now, we map whole page even if it's already zero, + * to sync it even if the sync is only DSB. + */ + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (pte2_load(sysmaps->CMAP2) != 0) + panic("%s: CMAP2 busy", __func__); + pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(pa, PTE2_AP_KRW, + m->md.pat_mode)); + /* Even VM_ALLOC_ZERO request is only advisory. */ + if ((m->flags & PG_ZERO) == 0) + pagezero(sysmaps->CADDR2); + pte2_sync_range((pt2_entry_t *)sysmaps->CADDR2, PAGE_SIZE); + pte2_clear(sysmaps->CMAP2); + tlb_flush((vm_offset_t)sysmaps->CADDR2); + sched_unpin(); + mtx_unlock(&sysmaps->lock); - l2b = pmap_get_l2_bucket(pmap_kernel(), va); - if (l2b == NULL) - l2b = pmap_grow_l2_bucket(pmap_kernel(), va); - KASSERT(l2b != NULL, ("No L2 Bucket")); + return (pa); +} - ptep = &l2b->l2b_kva[l2pte_index(va)]; - opte = *ptep; +/* + * Init just allocated page as L2 page table(s) holder + * and return its physical address. + */ +static __inline vm_paddr_t +pmap_pt2pg_init(pmap_t pmap, vm_offset_t va, vm_page_t m) +{ + vm_paddr_t pa; + pt2_entry_t *pte2p; - if (flags & KENTER_CACHE) - *ptep = L2_S_PROTO | l2s_mem_types[PTE_CACHE] | pa | L2_S_REF; - else if (flags & KENTER_DEVICE) - *ptep = L2_S_PROTO | l2s_mem_types[PTE_DEVICE] | pa | L2_S_REF; - else - *ptep = L2_S_PROTO | l2s_mem_types[PTE_NOCACHE] | pa | L2_S_REF; + /* Check page attributes. */ + if (pmap_page_get_memattr(m) != pt_memattr) + pmap_page_set_memattr(m, pt_memattr); - if (flags & KENTER_CACHE) { - pmap_set_prot(ptep, VM_PROT_READ | VM_PROT_WRITE, - flags & KENTER_USER); - } else { - pmap_set_prot(ptep, VM_PROT_READ|VM_PROT_WRITE|VM_PROT_EXECUTE, - 0); - } + /* Zero page and init wire counts. */ + pa = pmap_pt2pg_zero(m); + pt2_wirecount_init(m); - PTE_SYNC(ptep); - if (l2pte_valid(opte)) { - if (L2_S_EXECUTABLE(opte) || L2_S_EXECUTABLE(*ptep)) - cpu_tlb_flushID_SE(va); - else - cpu_tlb_flushD_SE(va); - } else { - if (opte == 0) - l2b->l2b_occupancy++; + /* + * Map page to PT2MAP address space for given pmap. + * Note that PT2MAP space is shared with all pmaps. + */ + if (pmap == kernel_pmap) + pmap_kenter_pt2tab(va, PTE2_KPT(pa)); + else { + pte2p = pmap_pt2tab_entry(pmap, va); + pt2tab_store(pte2p, PTE2_KPT_NG(pa)); } - cpu_cpwait(); - PDEBUG(1, printf("pmap_kenter: pte = %08x, opte = %08x, npte = %08x\n", - (uint32_t) ptep, opte, *ptep)); + return (pa); } +/* + * Initialize the pmap module. + * Called by vm_init, to initialize any structures that the pmap + * system needs to map virtual memory. + */ void -pmap_kenter(vm_offset_t va, vm_paddr_t pa) +pmap_init(void) { - pmap_kenter_internal(va, pa, KENTER_CACHE); -} + vm_size_t s; + pt2_entry_t *pte2p, pte2; + u_int i, pte1_idx, pv_npg; -void -pmap_kenter_nocache(vm_offset_t va, vm_paddr_t pa) -{ + PDEBUG(1, printf("%s: phys_start = %#x\n", __func__, PHYSADDR)); - pmap_kenter_internal(va, pa, 0); -} + /* + * Initialize the vm page array entries for kernel pmap's + * L2 page table pages allocated in advance. + */ + pte1_idx = pte1_index(KERNBASE - PT2MAP_SIZE); + pte2p = kern_pt2tab_entry(KERNBASE - PT2MAP_SIZE); + for (i = 0; i < nkpt2pg + NPG_IN_PT2TAB; i++, pte2p++) { + vm_paddr_t pa; + vm_page_t m; -void -pmap_kenter_device(vm_offset_t va, vm_size_t size, vm_paddr_t pa) -{ - vm_offset_t sva; + pte2 = pte2_load(pte2p); + KASSERT(pte2_is_valid(pte2), ("%s: no valid entry", __func__)); - KASSERT((size & PAGE_MASK) == 0, - ("%s: device mapping not page-sized", __func__)); + pa = pte2_pa(pte2); + m = PHYS_TO_VM_PAGE(pa); + KASSERT(m >= vm_page_array && + m < &vm_page_array[vm_page_array_size], + ("%s: L2 page table page is out of range", __func__)); - sva = va; - while (size != 0) { - pmap_kenter_internal(va, pa, KENTER_DEVICE); - va += PAGE_SIZE; - pa += PAGE_SIZE; - size -= PAGE_SIZE; + m->pindex = pte1_idx; + m->phys_addr = pa; + pte1_idx += NPT2_IN_PG; } -} -void -pmap_kremove_device(vm_offset_t va, vm_size_t size) -{ - vm_offset_t sva; + /* + * Initialize the address space (zone) for the pv entries. Set a + * high water mark so that the system can recover from excessive + * numbers of pv entries. + */ + TUNABLE_INT_FETCH("vm.pmap.shpgperproc", &shpgperproc); + pv_entry_max = shpgperproc * maxproc + vm_cnt.v_page_count; + TUNABLE_INT_FETCH("vm.pmap.pv_entries", &pv_entry_max); + pv_entry_max = roundup(pv_entry_max, _NPCPV); + pv_entry_high_water = 9 * (pv_entry_max / 10); - KASSERT((size & PAGE_MASK) == 0, - ("%s: device mapping not page-sized", __func__)); - - sva = va; - while (size != 0) { - pmap_kremove(va); - va += PAGE_SIZE; - size -= PAGE_SIZE; + /* + * Are large page mappings enabled? + */ + TUNABLE_INT_FETCH("vm.pmap.sp_enabled", &sp_enabled); + if (sp_enabled) { + KASSERT(MAXPAGESIZES > 1 && pagesizes[1] == 0, + ("%s: can't assign to pagesizes[1]", __func__)); + pagesizes[1] = PTE1_SIZE; } -} -void -pmap_kenter_user(vm_offset_t va, vm_paddr_t pa) -{ + /* + * Calculate the size of the pv head table for sections. + * Handle the possibility that "vm_phys_segs[...].end" is zero. + * Note that the table is only for sections which could be promoted. + */ + first_managed_pa = pte1_trunc(vm_phys_segs[0].start); + pv_npg = (pte1_trunc(vm_phys_segs[vm_phys_nsegs - 1].end - PAGE_SIZE) + - first_managed_pa) / PTE1_SIZE + 1; - pmap_kenter_internal(va, pa, KENTER_CACHE|KENTER_USER); /* - * Call pmap_fault_fixup now, to make sure we'll have no exception - * at the first use of the new address, or bad things will happen, - * as we use one of these addresses in the exception handlers. + * Allocate memory for the pv head table for sections. */ - pmap_fault_fixup(pmap_kernel(), va, VM_PROT_READ|VM_PROT_WRITE, 1); -} + s = (vm_size_t)(pv_npg * sizeof(struct md_page)); + s = round_page(s); + pv_table = (struct md_page *)kmem_malloc(kernel_arena, s, + M_WAITOK | M_ZERO); + for (i = 0; i < pv_npg; i++) + TAILQ_INIT(&pv_table[i].pv_list); -vm_paddr_t -pmap_kextract(vm_offset_t va) -{ - - if (kernel_vm_end == 0) - return (0); - return (pmap_extract_locked(kernel_pmap, va)); + pv_maxchunks = MAX(pv_entry_max / _NPCPV, maxproc); + pv_chunkbase = (struct pv_chunk *)kva_alloc(PAGE_SIZE * pv_maxchunks); + if (pv_chunkbase == NULL) + panic("%s: not enough kvm for pv chunks", __func__); + pmap_pte2list_init(&pv_vafree, pv_chunkbase, pv_maxchunks); } /* - * remove a page from the kernel pagetables + * Add a list of wired pages to the kva + * this routine is only used for temporary + * kernel mappings that do not need to have + * page modification or references recorded. + * Note that old mappings are simply written + * over. The page *must* be wired. + * Note: SMP coherent. Uses a ranged shootdown IPI. */ void -pmap_kremove(vm_offset_t va) +pmap_qenter(vm_offset_t sva, vm_page_t *ma, int count) { - struct l2_bucket *l2b; - pt_entry_t *ptep, opte; + u_int anychanged; + pt2_entry_t *epte2p, *pte2p, pte2; + vm_page_t m; + vm_paddr_t pa; - l2b = pmap_get_l2_bucket(pmap_kernel(), va); - if (!l2b) - return; - KASSERT(l2b != NULL, ("No L2 Bucket")); - ptep = &l2b->l2b_kva[l2pte_index(va)]; - opte = *ptep; - if (l2pte_valid(opte)) { - va = va & ~PAGE_MASK; - *ptep = 0; - PTE_SYNC(ptep); - if (L2_S_EXECUTABLE(opte)) - cpu_tlb_flushID_SE(va); - else - cpu_tlb_flushD_SE(va); - cpu_cpwait(); + anychanged = 0; + pte2p = pt2map_entry(sva); + epte2p = pte2p + count; + while (pte2p < epte2p) { + m = *ma++; + pa = VM_PAGE_TO_PHYS(m); + pte2 = pte2_load(pte2p); + if ((pte2_pa(pte2) != pa) || + (pte2_attr(pte2) != m->md.pat_mode)) { + anychanged++; + pte2_store(pte2p, PTE2_KERN(pa, PTE2_AP_KRW, + m->md.pat_mode)); + } + pte2p++; } + if (__predict_false(anychanged)) + tlb_flush_range(sva, count * PAGE_SIZE); } - /* - * Used to map a range of physical addresses into kernel - * virtual address space. - * - * The value passed in '*virt' is a suggested virtual address for - * the mapping. Architectures which can support a direct-mapped - * physical to virtual region can return the appropriate address - * within that region, leaving '*virt' unchanged. Other - * architectures should map the pages starting at '*virt' and - * update '*virt' with the first usable address after the mapped - * region. + * This routine tears out page mappings from the + * kernel -- it is meant only for temporary mappings. + * Note: SMP coherent. Uses a ranged shootdown IPI. */ -vm_offset_t -pmap_map(vm_offset_t *virt, vm_offset_t start, vm_offset_t end, int prot) +void +pmap_qremove(vm_offset_t sva, int count) { - vm_offset_t sva = *virt; - vm_offset_t va = sva; + vm_offset_t va; - PDEBUG(1, printf("pmap_map: virt = %08x, start = %08x, end = %08x, " - "prot = %d\n", (uint32_t) *virt, (uint32_t) start, (uint32_t) end, - prot)); - - while (start < end) { - pmap_kenter(va, start); + va = sva; + while (count-- > 0) { + pmap_kremove(va); va += PAGE_SIZE; - start += PAGE_SIZE; } - *virt = va; - return (sva); + tlb_flush_range(sva, va - sva); } /* - * Add a list of wired pages to the kva - * this routine is only used for temporary - * kernel mappings that do not need to have - * page modification or references recorded. - * Note that old mappings are simply written - * over. The page *must* be wired. + * Are we current address space or kernel? */ -void -pmap_qenter(vm_offset_t va, vm_page_t *m, int count) +static __inline int +pmap_is_current(pmap_t pmap) { - int i; - for (i = 0; i < count; i++) { - pmap_kenter_internal(va, VM_PAGE_TO_PHYS(m[i]), - KENTER_CACHE); - va += PAGE_SIZE; - } + return (pmap == kernel_pmap || + (pmap == vmspace_pmap(curthread->td_proc->p_vmspace))); } - /* - * this routine jerks page mappings from the - * kernel -- it is meant only for temporary mappings. + * If the given pmap is not the current or kernel pmap, the returned + * pte2 must be released by passing it to pmap_pte2_release(). */ -void -pmap_qremove(vm_offset_t va, int count) +static pt2_entry_t * +pmap_pte2(pmap_t pmap, vm_offset_t va) { - int i; + pt1_entry_t pte1; + vm_paddr_t pt2pg_pa; - for (i = 0; i < count; i++) { - if (vtophys(va)) - pmap_kremove(va); - - va += PAGE_SIZE; + pte1 = pte1_load(pmap_pte1(pmap, va)); + if (pte1_is_section(pte1)) + panic("%s: attempt to map PTE1", __func__); + if (pte1_is_link(pte1)) { + /* Are we current address space or kernel? */ + if (pmap_is_current(pmap)) + return (pt2map_entry(va)); + /* Note that L2 page table size is not equal to PAGE_SIZE. */ + pt2pg_pa = trunc_page(pte1_link_pa(pte1)); + mtx_lock(&PMAP2mutex); + if (pte2_pa(pte2_load(PMAP2)) != pt2pg_pa) { + pte2_store(PMAP2, PTE2_KPT(pt2pg_pa)); + tlb_flush((vm_offset_t)PADDR2); + } + return (PADDR2 + (arm32_btop(va) & (NPTE2_IN_PG - 1))); } + return (NULL); } - /* - * pmap_object_init_pt preloads the ptes for a given object - * into the specified pmap. This eliminates the blast of soft - * faults on process startup and immediately after an mmap. + * Releases a pte2 that was obtained from pmap_pte2(). + * Be prepared for the pte2p being NULL. */ -void -pmap_object_init_pt(pmap_t pmap, vm_offset_t addr, vm_object_t object, - vm_pindex_t pindex, vm_size_t size) +static __inline void +pmap_pte2_release(pt2_entry_t *pte2p) { - VM_OBJECT_ASSERT_WLOCKED(object); - KASSERT(object->type == OBJT_DEVICE || object->type == OBJT_SG, - ("pmap_object_init_pt: non-device object")); + if ((pt2_entry_t *)(trunc_page((vm_offset_t)pte2p)) == PADDR2) { + mtx_unlock(&PMAP2mutex); + } } - /* - * pmap_is_prefaultable: + * Super fast pmap_pte2 routine best used when scanning + * the pv lists. This eliminates many coarse-grained + * invltlb calls. Note that many of the pv list + * scans are across different pmaps. It is very wasteful + * to do an entire tlb flush for checking a single mapping. * - * Return whether or not the specified virtual address is elgible - * for prefault. + * If the given pmap is not the current pmap, pvh_global_lock + * must be held and curthread pinned to a CPU. */ -boolean_t -pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr) +static pt2_entry_t * +pmap_pte2_quick(pmap_t pmap, vm_offset_t va) { - pd_entry_t *pdep; - pt_entry_t *ptep; + pt1_entry_t pte1; + vm_paddr_t pt2pg_pa; - if (!pmap_get_pde_pte(pmap, addr, &pdep, &ptep)) - return (FALSE); - KASSERT((pdep != NULL && (l1pte_section_p(*pdep) || ptep != NULL)), - ("Valid mapping but no pte ?")); - if (*pdep != 0 && !l1pte_section_p(*pdep)) - if (*ptep == 0) - return (TRUE); - return (FALSE); + pte1 = pte1_load(pmap_pte1(pmap, va)); + if (pte1_is_section(pte1)) + panic("%s: attempt to map PTE1", __func__); + if (pte1_is_link(pte1)) { + /* Are we current address space or kernel? */ + if (pmap_is_current(pmap)) + return (pt2map_entry(va)); + rw_assert(&pvh_global_lock, RA_WLOCKED); + KASSERT(curthread->td_pinned > 0, + ("%s: curthread not pinned", __func__)); + /* Note that L2 page table size is not equal to PAGE_SIZE. */ + pt2pg_pa = trunc_page(pte1_link_pa(pte1)); + if (pte2_pa(pte2_load(PMAP1)) != pt2pg_pa) { + pte2_store(PMAP1, PTE2_KPT(pt2pg_pa)); +#ifdef SMP + PMAP1cpu = PCPU_GET(cpuid); +#endif + tlb_flush_local((vm_offset_t)PADDR1); + PMAP1changed++; + } else +#ifdef SMP + if (PMAP1cpu != PCPU_GET(cpuid)) { + PMAP1cpu = PCPU_GET(cpuid); + tlb_flush_local((vm_offset_t)PADDR1); + PMAP1changedcpu++; + } else +#endif + PMAP1unchanged++; + return (PADDR1 + (arm32_btop(va) & (NPTE2_IN_PG - 1))); + } + return (NULL); } /* - * Fetch pointers to the PDE/PTE for the given pmap/VA pair. - * Returns TRUE if the mapping exists, else FALSE. - * - * NOTE: This function is only used by a couple of arm-specific modules. - * It is not safe to take any pmap locks here, since we could be right - * in the middle of debugging the pmap anyway... - * - * It is possible for this routine to return FALSE even though a valid - * mapping does exist. This is because we don't lock, so the metadata - * state may be inconsistent. - * - * NOTE: We can return a NULL *ptp in the case where the L1 pde is - * a "section" mapping. + * Routine: pmap_extract + * Function: + * Extract the physical page address associated + * with the given map/virtual_address pair. */ -boolean_t -pmap_get_pde_pte(pmap_t pmap, vm_offset_t va, pd_entry_t **pdp, - pt_entry_t **ptp) +vm_paddr_t +pmap_extract(pmap_t pmap, vm_offset_t va) { - struct l2_dtable *l2; - pd_entry_t *pl1pd, l1pd; - pt_entry_t *ptep; - u_short l1idx; + vm_paddr_t pa; + pt1_entry_t pte1; + pt2_entry_t *pte2p; - if (pmap->pm_l1 == NULL) - return (FALSE); - - l1idx = L1_IDX(va); - *pdp = pl1pd = &pmap->pm_l1->l1_kva[l1idx]; - l1pd = *pl1pd; - - if (l1pte_section_p(l1pd)) { - *ptp = NULL; - return (TRUE); - } - - if (pmap->pm_l2 == NULL) - return (FALSE); - - l2 = pmap->pm_l2[L2_IDX(l1idx)]; - - if (l2 == NULL || - (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) { - return (FALSE); - } - - *ptp = &ptep[l2pte_index(va)]; - return (TRUE); + PMAP_LOCK(pmap); + pte1 = pte1_load(pmap_pte1(pmap, va)); + if (pte1_is_section(pte1)) + pa = pte1_pa(pte1) | (va & PTE1_OFFSET); + else if (pte1_is_link(pte1)) { + pte2p = pmap_pte2(pmap, va); + pa = pte2_pa(pte2_load(pte2p)) | (va & PTE2_OFFSET); + pmap_pte2_release(pte2p); + } else + pa = 0; + PMAP_UNLOCK(pmap); + return (pa); } /* - * Routine: pmap_remove_all - * Function: - * Removes this physical page from - * all physical maps in which it resides. - * Reflects back modify bits to the pager. - * - * Notes: - * Original versions of this routine were very - * inefficient because they iteratively called - * pmap_remove (slow...) + * Routine: pmap_extract_and_hold + * Function: + * Atomically extract and hold the physical page + * with the given pmap and virtual address pair + * if that mapping permits the given protection. */ -void -pmap_remove_all(vm_page_t m) +vm_page_t +pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) { - struct md_page *pvh; - pv_entry_t pv; - pmap_t pmap; - pt_entry_t *ptep; - struct l2_bucket *l2b; - boolean_t flush = FALSE; - pmap_t curpmap; - u_int is_exec = 0; + vm_paddr_t pa, lockpa; + pt1_entry_t pte1; + pt2_entry_t pte2, *pte2p; + vm_page_t m; - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_remove_all: page %p is not managed", m)); - rw_wlock(&pvh_global_lock); - if ((m->flags & PG_FICTITIOUS) != 0) - goto small_mappings; - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - while ((pv = TAILQ_FIRST(&pvh->pv_list)) != NULL) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pd_entry_t *pl1pd; - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(pv->pv_va)]; - KASSERT((*pl1pd & L1_TYPE_MASK) == L1_S_PROTO, - ("pmap_remove_all: valid section mapping expected")); - (void)pmap_demote_section(pmap, pv->pv_va); - PMAP_UNLOCK(pmap); - } -small_mappings: - curpmap = vmspace_pmap(curproc->p_vmspace); - while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { - pmap = PV_PMAP(pv); - if (flush == FALSE && (pmap == curpmap || - pmap == pmap_kernel())) - flush = TRUE; - - PMAP_LOCK(pmap); - l2b = pmap_get_l2_bucket(pmap, pv->pv_va); - KASSERT(l2b != NULL, ("No l2 bucket")); - ptep = &l2b->l2b_kva[l2pte_index(pv->pv_va)]; - is_exec |= PTE_BEEN_EXECD(*ptep); - *ptep = 0; - if (pmap_is_current(pmap)) - PTE_SYNC(ptep); - pmap_free_l2_bucket(pmap, l2b, 1); - pmap->pm_stats.resident_count--; - TAILQ_REMOVE(&m->md.pv_list, pv, pv_list); - if (pv->pv_flags & PVF_WIRED) - pmap->pm_stats.wired_count--; - pmap_free_pv_entry(pmap, pv); - PMAP_UNLOCK(pmap); - } - - if (flush) { - if (is_exec) - cpu_tlb_flushID(); - else - cpu_tlb_flushD(); - cpu_cpwait(); - } - vm_page_aflag_clear(m, PGA_WRITEABLE); - rw_wunlock(&pvh_global_lock); -} - -int -pmap_change_attr(vm_offset_t sva, vm_size_t len, int mode) -{ - vm_offset_t base, offset, tmpva; - vm_size_t size; - struct l2_bucket *l2b; - pt_entry_t *ptep, pte; - vm_offset_t next_bucket; - - PMAP_LOCK(kernel_pmap); - - base = trunc_page(sva); - offset = sva & PAGE_MASK; - size = roundup(offset + len, PAGE_SIZE); - - for (tmpva = base; tmpva < base + size; ) { - next_bucket = L2_NEXT_BUCKET(tmpva); - if (next_bucket > base + size) - next_bucket = base + size; - - l2b = pmap_get_l2_bucket(kernel_pmap, tmpva); - if (l2b == NULL) { - tmpva = next_bucket; - continue; + lockpa = 0; + m = NULL; + PMAP_LOCK(pmap); +retry: + pte1 = pte1_load(pmap_pte1(pmap, va)); + if (pte1_is_section(pte1)) { + if (!(pte1 & PTE1_RO) || !(prot & VM_PROT_WRITE)) { + pa = pte1_pa(pte1) | (va & PTE1_OFFSET); + if (vm_page_pa_tryrelock(pmap, pa, &lockpa)) + goto retry; + m = PHYS_TO_VM_PAGE(pa); + vm_page_hold(m); } - - ptep = &l2b->l2b_kva[l2pte_index(tmpva)]; - - if (*ptep == 0) { - PMAP_UNLOCK(kernel_pmap); - return(EINVAL); + } else if (pte1_is_link(pte1)) { + pte2p = pmap_pte2(pmap, va); + pte2 = pte2_load(pte2p); + pmap_pte2_release(pte2p); + if (pte2_is_valid(pte2) && + (!(pte2 & PTE2_RO) || !(prot & VM_PROT_WRITE))) { + pa = pte2_pa(pte2); + if (vm_page_pa_tryrelock(pmap, pa, &lockpa)) + goto retry; + m = PHYS_TO_VM_PAGE(pa); + vm_page_hold(m); } - - pte = *ptep &~ L2_S_CACHE_MASK; - cpu_idcache_wbinv_range(tmpva, PAGE_SIZE); - pmap_l2cache_wbinv_range(tmpva, pte & L2_S_FRAME, PAGE_SIZE); - *ptep = pte; - cpu_tlb_flushID_SE(tmpva); - cpu_cpwait(); - - dprintf("%s: for va:%x ptep:%x pte:%x\n", - __func__, tmpva, (uint32_t)ptep, pte); - tmpva += PAGE_SIZE; } - - PMAP_UNLOCK(kernel_pmap); - - return (0); + PA_UNLOCK_COND(lockpa); + PMAP_UNLOCK(pmap); + return (m); } /* - * Set the physical protection on the - * specified range of this map as requested. + * Grow the number of kernel L2 page table entries, if needed. */ void -pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) +pmap_growkernel(vm_offset_t addr) { - struct l2_bucket *l2b; - struct md_page *pvh; - struct pv_entry *pve; - pd_entry_t *pl1pd, l1pd; - pt_entry_t *ptep, pte; - vm_offset_t next_bucket; - u_int is_exec, is_refd; - int flush; + vm_page_t m; + vm_paddr_t pt2pg_pa, pt2_pa; + pt1_entry_t pte1; + pt2_entry_t pte2; - if ((prot & VM_PROT_READ) == 0) { - pmap_remove(pmap, sva, eva); - return; - } - - if (prot & VM_PROT_WRITE) { - /* - * If this is a read->write transition, just ignore it and let - * vm_fault() take care of it later. - */ - return; - } - - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - + PDEBUG(1, printf("%s: addr = %#x\n", __func__, addr)); /* - * OK, at this point, we know we're doing write-protect operation. - * If the pmap is active, write-back the range. + * All the time kernel_vm_end is first KVA for which underlying + * L2 page table is either not allocated or linked from L1 page table + * (not considering sections). Except for two possible cases: + * + * (1) in the very beginning as long as pmap_growkernel() was + * not called, it could be first unused KVA (which is not + * rounded up to PTE1_SIZE), + * + * (2) when all KVA space is mapped and kernel_map->max_offset + * address is not rounded up to PTE1_SIZE. (For example, + * it could be 0xFFFFFFFF.) */ + kernel_vm_end = pte1_roundup(kernel_vm_end); + mtx_assert(&kernel_map->system_mtx, MA_OWNED); + addr = roundup2(addr, PTE1_SIZE); + if (addr - 1 >= kernel_map->max_offset) + addr = kernel_map->max_offset; + while (kernel_vm_end < addr) { + pte1 = pte1_load(kern_pte1(kernel_vm_end)); + if (pte1_is_valid(pte1)) { + kernel_vm_end += PTE1_SIZE; + if (kernel_vm_end - 1 >= kernel_map->max_offset) { + kernel_vm_end = kernel_map->max_offset; + break; + } + continue; + } - flush = ((eva - sva) >= (PAGE_SIZE * 4)) ? 0 : -1; - is_exec = is_refd = 0; - - while (sva < eva) { - next_bucket = L2_NEXT_BUCKET(sva); /* - * Check for large page. + * kernel_vm_end_new is used in pmap_pinit() when kernel + * mappings are entered to new pmap all at once to avoid race + * between pmap_kenter_pte1() and kernel_vm_end increase. + * The same aplies to pmap_kenter_pt2tab(). */ - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(sva)]; - l1pd = *pl1pd; - if ((l1pd & L1_TYPE_MASK) == L1_S_PROTO) { - KASSERT(pmap != pmap_kernel(), - ("pmap_protect: trying to modify " - "kernel section protections")); + kernel_vm_end_new = kernel_vm_end + PTE1_SIZE; + + pte2 = pt2tab_load(kern_pt2tab_entry(kernel_vm_end)); + if (!pte2_is_valid(pte2)) { /* - * Are we protecting the entire large page? If not, - * demote the mapping and fall through. + * Install new PT2s page into kernel PT2TAB. */ - if (sva + L1_S_SIZE == next_bucket && - eva >= next_bucket) { - l1pd &= ~(L1_S_PROT_MASK | L1_S_XN); - if (!(prot & VM_PROT_EXECUTE)) - l1pd |= L1_S_XN; - /* - * At this point we are always setting - * write-protect bit. - */ - l1pd |= L1_S_APX; - /* All managed superpages are user pages. */ - l1pd |= L1_S_PROT_U; - *pl1pd = l1pd; - PTE_SYNC(pl1pd); - pvh = pa_to_pvh(l1pd & L1_S_FRAME); - pve = pmap_find_pv(pvh, pmap, - trunc_1mpage(sva)); - pve->pv_flags &= ~PVF_WRITE; - sva = next_bucket; - continue; - } else if (!pmap_demote_section(pmap, sva)) { - /* The large page mapping was destroyed. */ - sva = next_bucket; - continue; - } - } - if (next_bucket > eva) - next_bucket = eva; - l2b = pmap_get_l2_bucket(pmap, sva); - if (l2b == NULL) { - sva = next_bucket; - continue; - } + m = vm_page_alloc(NULL, + pte1_index(kernel_vm_end) & ~PT2PG_MASK, + VM_ALLOC_INTERRUPT | VM_ALLOC_NOOBJ | + VM_ALLOC_WIRED | VM_ALLOC_ZERO); + if (m == NULL) + panic("%s: no memory to grow kernel", __func__); + /* + * QQQ: To link all new L2 page tables from L1 page + * table now and so pmap_kenter_pte1() them + * at once together with pmap_kenter_pt2tab() + * could be nice speed up. However, + * pmap_growkernel() does not happen so often... + * QQQ: The other TTBR is another option. + */ + pt2pg_pa = pmap_pt2pg_init(kernel_pmap, kernel_vm_end, + m); + } else + pt2pg_pa = pte2_pa(pte2); - ptep = &l2b->l2b_kva[l2pte_index(sva)]; + pt2_pa = page_pt2pa(pt2pg_pa, pte1_index(kernel_vm_end)); + pmap_kenter_pte1(kernel_vm_end, PTE1_LINK(pt2_pa)); - while (sva < next_bucket) { - if ((pte = *ptep) != 0 && L2_S_WRITABLE(pte)) { - struct vm_page *m; - - m = PHYS_TO_VM_PAGE(l2pte_pa(pte)); - pmap_set_prot(ptep, prot, - !(pmap == pmap_kernel())); - PTE_SYNC(ptep); - - pmap_modify_pv(m, pmap, sva, PVF_WRITE, 0); - - if (flush >= 0) { - flush++; - is_exec |= PTE_BEEN_EXECD(pte); - is_refd |= PTE_BEEN_REFD(pte); - } else { - if (PTE_BEEN_EXECD(pte)) - cpu_tlb_flushID_SE(sva); - else if (PTE_BEEN_REFD(pte)) - cpu_tlb_flushD_SE(sva); - } - } - - sva += PAGE_SIZE; - ptep++; + kernel_vm_end = kernel_vm_end_new; + if (kernel_vm_end - 1 >= kernel_map->max_offset) { + kernel_vm_end = kernel_map->max_offset; + break; } } +} +static int +kvm_size(SYSCTL_HANDLER_ARGS) +{ + unsigned long ksize = vm_max_kernel_address - KERNBASE; - if (flush) { - if (is_exec) - cpu_tlb_flushID(); - else - if (is_refd) - cpu_tlb_flushD(); - cpu_cpwait(); - } - rw_wunlock(&pvh_global_lock); - - PMAP_UNLOCK(pmap); + return (sysctl_handle_long(oidp, &ksize, 0, req)); } +SYSCTL_PROC(_vm, OID_AUTO, kvm_size, CTLTYPE_LONG|CTLFLAG_RD, + 0, 0, kvm_size, "IU", "Size of KVM"); - -/* - * Insert the given physical page (p) at - * the specified virtual address (v) in the - * target physical map with the protection requested. - * - * If specified, the page will be wired down, meaning - * that the related pte can not be reclaimed. - * - * NB: This is the only routine which MAY NOT lazy-evaluate - * or lose information. That is, this routine must actually - * insert this page into the given map NOW. - */ - -int -pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, - u_int flags, int8_t psind __unused) +static int +kvm_free(SYSCTL_HANDLER_ARGS) { - struct l2_bucket *l2b; - int rv; + unsigned long kfree = vm_max_kernel_address - kernel_vm_end; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - rv = pmap_enter_locked(pmap, va, m, prot, flags); - if (rv == KERN_SUCCESS) { - /* - * If both the l2b_occupancy and the reservation are fully - * populated, then attempt promotion. - */ - l2b = pmap_get_l2_bucket(pmap, va); - if (l2b != NULL && l2b->l2b_occupancy == L2_PTE_NUM_TOTAL && - sp_enabled && (m->flags & PG_FICTITIOUS) == 0 && - vm_reserv_level_iffullpop(m) == 0) - pmap_promote_section(pmap, va); - } - PMAP_UNLOCK(pmap); - rw_wunlock(&pvh_global_lock); - return (rv); + return (sysctl_handle_long(oidp, &kfree, 0, req)); } +SYSCTL_PROC(_vm, OID_AUTO, kvm_free, CTLTYPE_LONG|CTLFLAG_RD, + 0, 0, kvm_free, "IU", "Amount of KVM free"); +/*********************************************** + * + * Pmap allocation/deallocation routines. + * + ***********************************************/ + /* - * The pvh global and pmap locks must be held. + * Initialize the pmap for the swapper process. */ -static int -pmap_enter_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, - u_int flags) +void +pmap_pinit0(pmap_t pmap) { - struct l2_bucket *l2b = NULL; - struct vm_page *om; - struct pv_entry *pve = NULL; - pd_entry_t *pl1pd, l1pd; - pt_entry_t *ptep, npte, opte; - u_int nflags; - u_int is_exec, is_refd; - vm_paddr_t pa; - u_char user; + PDEBUG(1, printf("%s: pmap = %p\n", __func__, pmap)); - PMAP_ASSERT_LOCKED(pmap); - rw_assert(&pvh_global_lock, RA_WLOCKED); - if (va == vector_page) { - pa = systempage.pv_pa; - m = NULL; - } else { - if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m)) - VM_OBJECT_ASSERT_LOCKED(m->object); - pa = VM_PAGE_TO_PHYS(m); - } + PMAP_LOCK_INIT(pmap); - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - if ((va < VM_MAXUSER_ADDRESS) && - (*pl1pd & L1_TYPE_MASK) == L1_S_PROTO) { - (void)pmap_demote_section(pmap, va); - } - - user = 0; /* - * Make sure userland mappings get the right permissions + * Kernel page table directory and pmap stuff around is already + * initialized, we are using it right now and here. So, finish + * only PMAP structures initialization for process0 ... + * + * Since the L1 page table and PT2TAB is shared with the kernel pmap, + * which is already included in the list "allpmaps", this pmap does + * not need to be inserted into that list. */ - if (pmap != pmap_kernel() && va != vector_page) - user = 1; + pmap->pm_pt1 = kern_pt1; + pmap->pm_pt2tab = kern_pt2tab; + CPU_ZERO(&pmap->pm_active); + PCPU_SET(curpmap, pmap); + TAILQ_INIT(&pmap->pm_pvchunk); + bzero(&pmap->pm_stats, sizeof pmap->pm_stats); + CPU_SET(0, &pmap->pm_active); +} - nflags = 0; +static __inline void +pte1_copy_nosync(pt1_entry_t *spte1p, pt1_entry_t *dpte1p, vm_offset_t sva, + vm_offset_t eva) +{ + u_int idx, count; - if (prot & VM_PROT_WRITE) - nflags |= PVF_WRITE; - if ((flags & PMAP_ENTER_WIRED) != 0) - nflags |= PVF_WIRED; + idx = pte1_index(sva); + count = (pte1_index(eva) - idx + 1) * sizeof(pt1_entry_t); + bcopy(spte1p + idx, dpte1p + idx, count); +} - PDEBUG(1, printf("pmap_enter: pmap = %08x, va = %08x, m = %08x, " - "prot = %x, flags = %x\n", (uint32_t) pmap, va, (uint32_t) m, - prot, flags)); +static __inline void +pt2tab_copy_nosync(pt2_entry_t *spte2p, pt2_entry_t *dpte2p, vm_offset_t sva, + vm_offset_t eva) +{ + u_int idx, count; - if (pmap == pmap_kernel()) { - l2b = pmap_get_l2_bucket(pmap, va); - if (l2b == NULL) - l2b = pmap_grow_l2_bucket(pmap, va); - } else { -do_l2b_alloc: - l2b = pmap_alloc_l2_bucket(pmap, va); - if (l2b == NULL) { - if ((flags & PMAP_ENTER_NOSLEEP) == 0) { - PMAP_UNLOCK(pmap); - rw_wunlock(&pvh_global_lock); - VM_WAIT; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - goto do_l2b_alloc; - } - return (KERN_RESOURCE_SHORTAGE); - } - } + idx = pt2tab_index(sva); + count = (pt2tab_index(eva) - idx + 1) * sizeof(pt2_entry_t); + bcopy(spte2p + idx, dpte2p + idx, count); +} - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - if ((*pl1pd & L1_TYPE_MASK) == L1_S_PROTO) - panic("pmap_enter: attempt to enter on 1MB page, va: %#x", va); +/* + * Initialize a preallocated and zeroed pmap structure, + * such as one in a vmspace structure. + */ +int +pmap_pinit(pmap_t pmap) +{ + pt1_entry_t *pte1p; + pt2_entry_t *pte2p; + vm_paddr_t pa, pt2tab_pa; + u_int i; - ptep = &l2b->l2b_kva[l2pte_index(va)]; + PDEBUG(6, printf("%s: pmap = %p, pm_pt1 = %p\n", __func__, pmap, + pmap->pm_pt1)); - opte = *ptep; - npte = pa; - is_exec = is_refd = 0; + /* + * No need to allocate L2 page table space yet but we do need + * a valid L1 page table and PT2TAB table. + * + * Install shared kernel mappings to these tables. It's a little + * tricky as some parts of KVA are reserved for vectors, devices, + * and whatever else. These parts are supposed to be above + * vm_max_kernel_address. Thus two regions should be installed: + * + * (1) . + * + * QQQ: The second region should be stable enough to be installed + * only once in time when the tables are allocated. + * QQQ: Maybe copy of both regions at once could be faster ... + * QQQ: Maybe the other TTBR is an option. + * + * Finally, install own PT2TAB table to these tables. + */ - if (opte) { - if (l2pte_pa(opte) == pa) { + if (pmap->pm_pt1 == NULL) { + pmap->pm_pt1 = (pt1_entry_t *)kmem_alloc_contig(kernel_arena, + NB_IN_PT1, M_NOWAIT | M_ZERO, 0, -1UL, NB_IN_PT1, 0, + pt_memattr); + if (pmap->pm_pt1 == NULL) + return (0); + } + if (pmap->pm_pt2tab == NULL) { + /* + * QQQ: (1) PT2TAB must be contiguous. If PT2TAB is one page + * only, what should be the only size for 32 bit systems, + * then we could allocate it with vm_page_alloc() and all + * the stuff needed as other L2 page table pages. + * (2) Note that a process PT2TAB is special L2 page table + * page. Its mapping in kernel_arena is permanent and can + * be used no matter which process is current. Its mapping + * in PT2MAP can be used only for current process. + */ + pmap->pm_pt2tab = (pt2_entry_t *)kmem_alloc_attr(kernel_arena, + NB_IN_PT2TAB, M_NOWAIT | M_ZERO, 0, -1UL, pt_memattr); + if (pmap->pm_pt2tab == NULL) { /* - * We're changing the attrs of an existing mapping. + * QQQ: As struct pmap is allocated from UMA with + * UMA_ZONE_NOFREE flag, it's important to leave + * no allocation in pmap if initialization failed. */ - if (m != NULL) - pmap_modify_pv(m, pmap, va, - PVF_WRITE | PVF_WIRED, nflags); - is_exec |= PTE_BEEN_EXECD(opte); - is_refd |= PTE_BEEN_REFD(opte); - goto validate; + kmem_free(kernel_arena, (vm_offset_t)pmap->pm_pt1, + NB_IN_PT1); + pmap->pm_pt1 = NULL; + return (0); } - if ((om = PHYS_TO_VM_PAGE(l2pte_pa(opte)))) { - /* - * Replacing an existing mapping with a new one. - * It is part of our managed memory so we - * must remove it from the PV list - */ - if ((pve = pmap_remove_pv(om, pmap, va))) { - is_exec |= PTE_BEEN_EXECD(opte); - is_refd |= PTE_BEEN_REFD(opte); - - if (m && ((m->oflags & VPO_UNMANAGED))) - pmap_free_pv_entry(pmap, pve); - } - } - - } else { /* - * Keep the stats up to date + * QQQ: Each L2 page table page vm_page_t has pindex set to + * pte1 index of virtual address mapped by this page. + * It's not valid for non kernel PT2TABs themselves. + * The pindex of these pages can not be altered because + * of the way how they are allocated now. However, it + * should not be a problem. */ - l2b->l2b_occupancy++; - pmap->pm_stats.resident_count++; } + mtx_lock_spin(&allpmaps_lock); /* - * Enter on the PV list if part of our managed memory. + * To avoid race with pmap_kenter_pte1() and pmap_kenter_pt2tab(), + * kernel_vm_end_new is used here instead of kernel_vm_end. */ - if ((m && !(m->oflags & VPO_UNMANAGED))) { - if ((!pve) && (pve = pmap_get_pv_entry(pmap, FALSE)) == NULL) - panic("pmap_enter: no pv entries"); + pte1_copy_nosync(kern_pt1, pmap->pm_pt1, KERNBASE, + kernel_vm_end_new - 1); + pte1_copy_nosync(kern_pt1, pmap->pm_pt1, vm_max_kernel_address, + 0xFFFFFFFF); + pt2tab_copy_nosync(kern_pt2tab, pmap->pm_pt2tab, KERNBASE, + kernel_vm_end_new - 1); + pt2tab_copy_nosync(kern_pt2tab, pmap->pm_pt2tab, vm_max_kernel_address, + 0xFFFFFFFF); + LIST_INSERT_HEAD(&allpmaps, pmap, pm_list); + mtx_unlock_spin(&allpmaps_lock); - KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva, - ("pmap_enter: managed mapping within the clean submap")); - KASSERT(pve != NULL, ("No pv")); - pmap_enter_pv(m, pve, pmap, va, nflags); - } - -validate: - /* Make the new PTE valid */ - npte |= L2_S_PROTO; -#ifdef SMP - npte |= L2_SHARED; -#endif - /* Set defaults first - kernel read access */ - npte |= L2_APX; - npte |= L2_S_PROT_R; - /* Set "referenced" flag */ - npte |= L2_S_REF; - - /* Now tune APs as desired */ - if (user) - npte |= L2_S_PROT_U; /* - * If this is not a vector_page - * then continue setting mapping parameters + * Store PT2MAP PT2 pages (a.k.a. PT2TAB) in PT2TAB itself. + * I.e. self reference mapping. The PT2TAB is private, however mapped + * into shared PT2MAP space, so the mapping should be not global. */ - if (m != NULL) { - if ((m->oflags & VPO_UNMANAGED) == 0) { - if (prot & (VM_PROT_ALL)) { - vm_page_aflag_set(m, PGA_REFERENCED); - } else { - /* - * Need to do page referenced emulation. - */ - npte &= ~L2_S_REF; - } - } + pt2tab_pa = vtophys(pmap->pm_pt2tab); + pte2p = pmap_pt2tab_entry(pmap, (vm_offset_t)PT2MAP); + for (pa = pt2tab_pa, i = 0; i < NPG_IN_PT2TAB; i++, pa += PTE2_SIZE) { + pt2tab_store(pte2p++, PTE2_KPT_NG(pa)); + } - if (prot & VM_PROT_WRITE) { - if ((m->oflags & VPO_UNMANAGED) == 0) { - vm_page_aflag_set(m, PGA_WRITEABLE); - /* - * XXX: Skip modified bit emulation for now. - * The emulation reveals problems - * that result in random failures - * during memory allocation on some - * platforms. - * Therefore, the page is marked RW - * immediately. - */ - npte &= ~(L2_APX); - vm_page_dirty(m); - } else - npte &= ~(L2_APX); - } - if (!(prot & VM_PROT_EXECUTE)) - npte |= L2_XN; - - if (m->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) - npte |= pte_l2_s_cache_mode; + /* Insert PT2MAP PT2s into pmap PT1. */ + pte1p = pmap_pte1(pmap, (vm_offset_t)PT2MAP); + for (pa = pt2tab_pa, i = 0; i < NPT2_IN_PT2TAB; i++, pa += NB_IN_PT2) { + pte1_store(pte1p++, PTE1_LINK(pa)); } - CTR5(KTR_PMAP,"enter: pmap:%p va:%x prot:%x pte:%x->%x", - pmap, va, prot, opte, npte); /* - * If this is just a wiring change, the two PTEs will be - * identical, so there's no need to update the page table. + * Now synchronize new mapping which was made above. */ - if (npte != opte) { - boolean_t is_cached = pmap_is_current(pmap); + pte1_sync_range(pmap->pm_pt1, NB_IN_PT1); + pte2_sync_range(pmap->pm_pt2tab, NB_IN_PT2TAB); - *ptep = npte; - PTE_SYNC(ptep); - if (is_cached) { - /* - * We only need to frob the cache/tlb if this pmap - * is current - */ - if (L1_IDX(va) != L1_IDX(vector_page) && - l2pte_valid(npte)) { - /* - * This mapping is likely to be accessed as - * soon as we return to userland. Fix up the - * L1 entry to avoid taking another - * page/domain fault. - */ - l1pd = l2b->l2b_phys | - L1_C_DOM(pmap->pm_domain) | L1_C_PROTO; - if (*pl1pd != l1pd) { - *pl1pd = l1pd; - PTE_SYNC(pl1pd); - } - } - } + CPU_ZERO(&pmap->pm_active); + TAILQ_INIT(&pmap->pm_pvchunk); + bzero(&pmap->pm_stats, sizeof pmap->pm_stats); - if (is_exec) - cpu_tlb_flushID_SE(va); - else if (is_refd) - cpu_tlb_flushD_SE(va); - cpu_cpwait(); - } - - if ((pmap != pmap_kernel()) && (pmap == &curproc->p_vmspace->vm_pmap)) - cpu_icache_sync_range(va, PAGE_SIZE); - return (KERN_SUCCESS); + return (1); } +#ifdef INVARIANTS +static boolean_t +pt2tab_user_is_empty(pt2_entry_t *tab) +{ + u_int i, end; + + end = pt2tab_index(VM_MAXUSER_ADDRESS); + for (i = 0; i < end; i++) + if (tab[i] != 0) return (FALSE); + return (TRUE); +} +#endif /* - * Maps a sequence of resident pages belonging to the same object. - * The sequence begins with the given page m_start. This page is - * mapped at the given virtual address start. Each subsequent page is - * mapped at a virtual address that is offset from start by the same - * amount as the page is offset from m_start within the object. The - * last page in the sequence is the page with the largest offset from - * m_start that can be mapped at a virtual address less than the given - * virtual address end. Not every virtual page between start and end - * is mapped; only those for which a resident page exists with the - * corresponding offset from m_start are mapped. + * Release any resources held by the given physical map. + * Called when a pmap initialized by pmap_pinit is being released. + * Should only be called if the map contains no valid mappings. */ void -pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, - vm_page_t m_start, vm_prot_t prot) +pmap_release(pmap_t pmap) { - vm_offset_t va; - vm_page_t m; - vm_pindex_t diff, psize; +#ifdef INVARIANTS + vm_offset_t start, end; +#endif + KASSERT(pmap->pm_stats.resident_count == 0, + ("%s: pmap resident count %ld != 0", __func__, + pmap->pm_stats.resident_count)); + KASSERT(pt2tab_user_is_empty(pmap->pm_pt2tab), + ("%s: has allocated user PT2(s)", __func__)); + KASSERT(CPU_EMPTY(&pmap->pm_active), + ("%s: pmap %p is active on some CPU(s)", __func__, pmap)); - VM_OBJECT_ASSERT_LOCKED(m_start->object); + mtx_lock_spin(&allpmaps_lock); + LIST_REMOVE(pmap, pm_list); + mtx_unlock_spin(&allpmaps_lock); - psize = atop(end - start); - m = m_start; - prot &= VM_PROT_READ | VM_PROT_EXECUTE; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { - va = start + ptoa(diff); - if ((va & L1_S_OFFSET) == 0 && L2_NEXT_BUCKET(va) <= end && - m->psind == 1 && sp_enabled && - pmap_enter_section(pmap, va, m, prot)) - m = &m[L1_S_SIZE / PAGE_SIZE - 1]; - else - pmap_enter_locked(pmap, va, m, prot, - PMAP_ENTER_NOSLEEP); - m = TAILQ_NEXT(m, listq); - } - PMAP_UNLOCK(pmap); - rw_wunlock(&pvh_global_lock); +#ifdef INVARIANTS + start = pte1_index(KERNBASE) * sizeof(pt1_entry_t); + end = (pte1_index(0xFFFFFFFF) + 1) * sizeof(pt1_entry_t); + bzero((char *)pmap->pm_pt1 + start, end - start); + + start = pt2tab_index(KERNBASE) * sizeof(pt2_entry_t); + end = (pt2tab_index(0xFFFFFFFF) + 1) * sizeof(pt2_entry_t); + bzero((char *)pmap->pm_pt2tab + start, end - start); +#endif + /* + * We are leaving PT1 and PT2TAB allocated on released pmap, + * so hopefully UMA vmspace_zone will always be inited with + * UMA_ZONE_NOFREE flag. + */ } +/********************************************************* + * + * L2 table pages and their pages management routines. + * + *********************************************************/ + /* - * this code makes some *MAJOR* assumptions: - * 1. Current pmap & pmap exists. - * 2. Not wired. - * 3. Read access. - * 4. No page table pages. - * but is *MUCH* faster than pmap_enter... + * Virtual interface for L2 page table wire counting. + * + * Each L2 page table in a page has own counter which counts a number of + * valid mappings in a table. Global page counter counts mappings in all + * tables in a page plus a single itself mapping in PT2TAB. + * + * During a promotion we leave the associated L2 page table counter + * untouched, so the table (strictly speaking a page which holds it) + * is never freed if promoted. + * + * If a page m->wire_count == 1 then no valid mappings exist in any L2 page + * table in the page and the page itself is only mapped in PT2TAB. */ -void -pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) +static __inline void +pt2_wirecount_init(vm_page_t m) { + u_int i; - prot &= VM_PROT_READ | VM_PROT_EXECUTE; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - pmap_enter_locked(pmap, va, m, prot, PMAP_ENTER_NOSLEEP); - PMAP_UNLOCK(pmap); - rw_wunlock(&pvh_global_lock); + /* + * Note: A page m is allocated with VM_ALLOC_WIRED flag and + * m->wire_count should be already set correctly. + * So, there is no need to set it again herein. + */ + for (i = 0; i < NPT2_IN_PG; i++) + m->md.pt2_wirecount[i] = 0; } -/* - * Clear the wired attribute from the mappings for the specified range of - * addresses in the given pmap. Every valid mapping within that range - * must have the wired attribute set. In contrast, invalid mappings - * cannot have the wired attribute set, so they are ignored. - * - * XXX Wired mappings of unmanaged pages cannot be counted by this pmap - * implementation. - */ -void -pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +static __inline void +pt2_wirecount_inc(vm_page_t m, uint32_t pte1_idx) { - struct l2_bucket *l2b; - struct md_page *pvh; - pd_entry_t l1pd; - pt_entry_t *ptep, pte; - pv_entry_t pv; - vm_offset_t next_bucket; - vm_paddr_t pa; - vm_page_t m; - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - while (sva < eva) { - next_bucket = L2_NEXT_BUCKET(sva); - l1pd = pmap->pm_l1->l1_kva[L1_IDX(sva)]; - if ((l1pd & L1_TYPE_MASK) == L1_S_PROTO) { - pa = l1pd & L1_S_FRAME; - m = PHYS_TO_VM_PAGE(pa); - KASSERT(m != NULL && (m->oflags & VPO_UNMANAGED) == 0, - ("pmap_unwire: unmanaged 1mpage %p", m)); - pvh = pa_to_pvh(pa); - pv = pmap_find_pv(pvh, pmap, trunc_1mpage(sva)); - if ((pv->pv_flags & PVF_WIRED) == 0) - panic("pmap_unwire: pv %p isn't wired", pv); + /* + * Note: A just modificated pte2 (i.e. already allocated) + * is acquiring one extra reference which must be + * explicitly cleared. It influences the KASSERTs herein. + * All L2 page tables in a page always belong to the same + * pmap, so we allow only one extra reference for the page. + */ + KASSERT(m->md.pt2_wirecount[pte1_idx & PT2PG_MASK] < (NPTE2_IN_PT2 + 1), + ("%s: PT2 is overflowing ...", __func__)); + KASSERT(m->wire_count <= (NPTE2_IN_PG + 1), + ("%s: PT2PG is overflowing ...", __func__)); - /* - * Are we unwiring the entire large page? If not, - * demote the mapping and fall through. - */ - if (sva + L1_S_SIZE == next_bucket && - eva >= next_bucket) { - pv->pv_flags &= ~PVF_WIRED; - pmap->pm_stats.wired_count -= L2_PTE_NUM_TOTAL; - sva = next_bucket; - continue; - } else if (!pmap_demote_section(pmap, sva)) - panic("pmap_unwire: demotion failed"); - } - if (next_bucket > eva) - next_bucket = eva; - l2b = pmap_get_l2_bucket(pmap, sva); - if (l2b == NULL) { - sva = next_bucket; - continue; - } - for (ptep = &l2b->l2b_kva[l2pte_index(sva)]; sva < next_bucket; - sva += PAGE_SIZE, ptep++) { - if ((pte = *ptep) == 0 || - (m = PHYS_TO_VM_PAGE(l2pte_pa(pte))) == NULL || - (m->oflags & VPO_UNMANAGED) != 0) - continue; - pv = pmap_find_pv(&m->md, pmap, sva); - if ((pv->pv_flags & PVF_WIRED) == 0) - panic("pmap_unwire: pv %p isn't wired", pv); - pv->pv_flags &= ~PVF_WIRED; - pmap->pm_stats.wired_count--; - } - } - rw_wunlock(&pvh_global_lock); - PMAP_UNLOCK(pmap); + m->wire_count++; + m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]++; } - -/* - * Copy the range specified by src_addr/len - * from the source map to the range dst_addr/len - * in the destination map. - * - * This routine is only advisory and need not do anything. - */ -void -pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, - vm_size_t len, vm_offset_t src_addr) +static __inline void +pt2_wirecount_dec(vm_page_t m, uint32_t pte1_idx) { -} + KASSERT(m->md.pt2_wirecount[pte1_idx & PT2PG_MASK] != 0, + ("%s: PT2 is underflowing ...", __func__)); + KASSERT(m->wire_count > 1, + ("%s: PT2PG is underflowing ...", __func__)); -/* - * Routine: pmap_extract - * Function: - * Extract the physical page address associated - * with the given map/virtual_address pair. - */ -vm_paddr_t -pmap_extract(pmap_t pmap, vm_offset_t va) + m->wire_count--; + m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]--; +} + +static __inline void +pt2_wirecount_set(vm_page_t m, uint32_t pte1_idx, uint16_t count) { - vm_paddr_t pa; - PMAP_LOCK(pmap); - pa = pmap_extract_locked(pmap, va); - PMAP_UNLOCK(pmap); - return (pa); + KASSERT(count <= NPTE2_IN_PT2, + ("%s: invalid count %u", __func__, count)); + KASSERT(m->wire_count > m->md.pt2_wirecount[pte1_idx & PT2PG_MASK], + ("%s: PT2PG corrupting (%u, %u) ...", __func__, m->wire_count, + m->md.pt2_wirecount[pte1_idx & PT2PG_MASK])); + + m->wire_count -= m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]; + m->wire_count += count; + m->md.pt2_wirecount[pte1_idx & PT2PG_MASK] = count; + + KASSERT(m->wire_count <= (NPTE2_IN_PG + 1), + ("%s: PT2PG is overflowed (%u) ...", __func__, m->wire_count)); } -static vm_paddr_t -pmap_extract_locked(pmap_t pmap, vm_offset_t va) +static __inline uint32_t +pt2_wirecount_get(vm_page_t m, uint32_t pte1_idx) { - struct l2_dtable *l2; - pd_entry_t l1pd; - pt_entry_t *ptep, pte; - vm_paddr_t pa; - u_int l1idx; - if (kernel_vm_end != 0 && pmap != kernel_pmap) - PMAP_ASSERT_LOCKED(pmap); - l1idx = L1_IDX(va); - l1pd = pmap->pm_l1->l1_kva[l1idx]; - if (l1pte_section_p(l1pd)) { - /* XXX: what to do about the bits > 32 ? */ - if (l1pd & L1_S_SUPERSEC) - pa = (l1pd & L1_SUP_FRAME) | (va & L1_SUP_OFFSET); - else - pa = (l1pd & L1_S_FRAME) | (va & L1_S_OFFSET); - } else { - /* - * Note that we can't rely on the validity of the L1 - * descriptor as an indication that a mapping exists. - * We have to look it up in the L2 dtable. - */ - l2 = pmap->pm_l2[L2_IDX(l1idx)]; - if (l2 == NULL || - (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) - return (0); - pte = ptep[l2pte_index(va)]; - if (pte == 0) - return (0); - if ((pte & L2_TYPE_MASK) == L2_TYPE_L) - pa = (pte & L2_L_FRAME) | (va & L2_L_OFFSET); - else - pa = (pte & L2_S_FRAME) | (va & L2_S_OFFSET); - } - return (pa); + return (m->md.pt2_wirecount[pte1_idx & PT2PG_MASK]); } -/* - * Atomically extract and hold the physical page with the given - * pmap and virtual address pair if that mapping permits the given - * protection. - * - */ -vm_page_t -pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) +static __inline boolean_t +pt2_is_empty(vm_page_t m, vm_offset_t va) { - struct l2_dtable *l2; - pd_entry_t l1pd; - pt_entry_t *ptep, pte; - vm_paddr_t pa, paddr; - vm_page_t m = NULL; - u_int l1idx; - l1idx = L1_IDX(va); - paddr = 0; - PMAP_LOCK(pmap); -retry: - l1pd = pmap->pm_l1->l1_kva[l1idx]; - if (l1pte_section_p(l1pd)) { - /* XXX: what to do about the bits > 32 ? */ - if (l1pd & L1_S_SUPERSEC) - pa = (l1pd & L1_SUP_FRAME) | (va & L1_SUP_OFFSET); - else - pa = (l1pd & L1_S_FRAME) | (va & L1_S_OFFSET); - if (vm_page_pa_tryrelock(pmap, pa & PG_FRAME, &paddr)) - goto retry; - if (L1_S_WRITABLE(l1pd) || (prot & VM_PROT_WRITE) == 0) { - m = PHYS_TO_VM_PAGE(pa); - vm_page_hold(m); - } - } else { - /* - * Note that we can't rely on the validity of the L1 - * descriptor as an indication that a mapping exists. - * We have to look it up in the L2 dtable. - */ - l2 = pmap->pm_l2[L2_IDX(l1idx)]; + return (m->md.pt2_wirecount[pte1_index(va) & PT2PG_MASK] == 0); +} - if (l2 == NULL || - (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) { - PMAP_UNLOCK(pmap); - return (NULL); - } +static __inline boolean_t +pt2_is_full(vm_page_t m, vm_offset_t va) +{ - ptep = &ptep[l2pte_index(va)]; - pte = *ptep; - - if (pte == 0) { - PMAP_UNLOCK(pmap); - return (NULL); - } else if ((prot & VM_PROT_WRITE) && (pte & L2_APX)) { - PMAP_UNLOCK(pmap); - return (NULL); - } else { - if ((pte & L2_TYPE_MASK) == L2_TYPE_L) - panic("extract and hold section mapping"); - else - pa = (pte & L2_S_FRAME) | (va & L2_S_OFFSET); - if (vm_page_pa_tryrelock(pmap, pa & PG_FRAME, &paddr)) - goto retry; - m = PHYS_TO_VM_PAGE(pa); - vm_page_hold(m); - } - } - - PMAP_UNLOCK(pmap); - PA_UNLOCK_COND(paddr); - return (m); + return (m->md.pt2_wirecount[pte1_index(va) & PT2PG_MASK] == + NPTE2_IN_PT2); } -vm_paddr_t -pmap_dump_kextract(vm_offset_t va, pt2_entry_t *pte2p) +static __inline boolean_t +pt2pg_is_empty(vm_page_t m) { - struct l2_dtable *l2; - pd_entry_t l1pd; - pt_entry_t *ptep, pte; - vm_paddr_t pa; - u_int l1idx; - l1idx = L1_IDX(va); - l1pd = kernel_pmap->pm_l1->l1_kva[l1idx]; - if (l1pte_section_p(l1pd)) { - if (l1pd & L1_S_SUPERSEC) - pa = (l1pd & L1_SUP_FRAME) | (va & L1_SUP_OFFSET); - else - pa = (l1pd & L1_S_FRAME) | (va & L1_S_OFFSET); - pte = L2_S_PROTO | pa | - L2_S_PROT(PTE_KERNEL, VM_PROT_READ | VM_PROT_WRITE); - } else { - l2 = kernel_pmap->pm_l2[L2_IDX(l1idx)]; - if (l2 == NULL || - (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) { - pte = 0; - pa = 0; - goto out; - } - pte = ptep[l2pte_index(va)]; - if (pte == 0) { - pa = 0; - goto out; - } - if ((pte & L2_TYPE_MASK) == L2_TYPE_L) - pa = (pte & L2_L_FRAME) | (va & L2_L_OFFSET); - else - pa = (pte & L2_S_FRAME) | (va & L2_S_OFFSET); - } -out: - if (pte2p != NULL) - *pte2p = pte; - return (pa); + return (m->wire_count == 1); } /* - * Initialize a preallocated and zeroed pmap structure, - * such as one in a vmspace structure. + * This routine is called if the L2 page table + * is not mapped correctly. */ - -int -pmap_pinit(pmap_t pmap) +static vm_page_t +_pmap_allocpte2(pmap_t pmap, vm_offset_t va, u_int flags) { - PDEBUG(1, printf("pmap_pinit: pmap = %08x\n", (uint32_t) pmap)); + uint32_t pte1_idx; + pt1_entry_t *pte1p; + pt2_entry_t pte2; + vm_page_t m; + vm_paddr_t pt2pg_pa, pt2_pa; - pmap_alloc_l1(pmap); - bzero(pmap->pm_l2, sizeof(pmap->pm_l2)); + pte1_idx = pte1_index(va); + pte1p = pmap->pm_pt1 + pte1_idx; - CPU_ZERO(&pmap->pm_active); + KASSERT(pte1_load(pte1p) == 0, + ("%s: pm_pt1[%#x] is not zero: %#x", __func__, pte1_idx, + pte1_load(pte1p))); - TAILQ_INIT(&pmap->pm_pvchunk); - bzero(&pmap->pm_stats, sizeof pmap->pm_stats); - pmap->pm_stats.resident_count = 1; - if (vector_page < KERNBASE) { - pmap_enter(pmap, vector_page, - PHYS_TO_VM_PAGE(systempage.pv_pa), VM_PROT_READ, - PMAP_ENTER_WIRED, 0); + pte2 = pt2tab_load(pmap_pt2tab_entry(pmap, va)); + if (!pte2_is_valid(pte2)) { + /* + * Install new PT2s page into pmap PT2TAB. + */ + m = vm_page_alloc(NULL, pte1_idx & ~PT2PG_MASK, + VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO); + if (m == NULL) { + if ((flags & PMAP_ENTER_NOSLEEP) == 0) { + PMAP_UNLOCK(pmap); + rw_wunlock(&pvh_global_lock); + VM_WAIT; + rw_wlock(&pvh_global_lock); + PMAP_LOCK(pmap); + } + + /* + * Indicate the need to retry. While waiting, + * the L2 page table page may have been allocated. + */ + return (NULL); + } + pmap->pm_stats.resident_count++; + pt2pg_pa = pmap_pt2pg_init(pmap, va, m); + } else { + pt2pg_pa = pte2_pa(pte2); + m = PHYS_TO_VM_PAGE(pt2pg_pa); } - return (1); -} + pt2_wirecount_inc(m, pte1_idx); + pt2_pa = page_pt2pa(pt2pg_pa, pte1_idx); + pte1_store(pte1p, PTE1_LINK(pt2_pa)); -/*************************************************** - * Superpage management routines. - ***************************************************/ + return (m); +} -static PMAP_INLINE struct pv_entry * -pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +static vm_page_t +pmap_allocpte2(pmap_t pmap, vm_offset_t va, u_int flags) { - pv_entry_t pv; + u_int pte1_idx; + pt1_entry_t *pte1p, pte1; + vm_page_t m; - rw_assert(&pvh_global_lock, RA_WLOCKED); + pte1_idx = pte1_index(va); +retry: + pte1p = pmap->pm_pt1 + pte1_idx; + pte1 = pte1_load(pte1p); - pv = pmap_find_pv(pvh, pmap, va); - if (pv != NULL) - TAILQ_REMOVE(&pvh->pv_list, pv, pv_list); + /* + * This supports switching from a 1MB page to a + * normal 4K page. + */ + if (pte1_is_section(pte1)) { + (void)pmap_demote_pte1(pmap, pte1p, va); + /* + * Reload pte1 after demotion. + * + * Note: Demotion can even fail as either PT2 is not find for + * the virtual address or PT2PG can not be allocated. + */ + pte1 = pte1_load(pte1p); + } - return (pv); + /* + * If the L2 page table page is mapped, we just increment the + * hold count, and activate it. + */ + if (pte1_is_link(pte1)) { + m = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); + pt2_wirecount_inc(m, pte1_idx); + } else { + /* + * Here if the PT2 isn't mapped, or if it has + * been deallocated. + */ + m = _pmap_allocpte2(pmap, va, flags); + if (m == NULL && (flags & PMAP_ENTER_NOSLEEP) == 0) + goto retry; + } + + return (m); } -static void -pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +static __inline void +pmap_free_zero_pages(struct spglist *free) { - pv_entry_t pv; + vm_page_t m; - pv = pmap_pvh_remove(pvh, pmap, va); - KASSERT(pv != NULL, ("pmap_pvh_free: pv not found")); - pmap_free_pv_entry(pmap, pv); + while ((m = SLIST_FIRST(free)) != NULL) { + SLIST_REMOVE_HEAD(free, plinks.s.ss); + /* Preserve the page's PG_ZERO setting. */ + vm_page_free_toq(m); + } } -static boolean_t -pmap_pv_insert_section(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) +/* + * Schedule the specified unused L2 page table page to be freed. Specifically, + * add the page to the specified list of pages that will be released to the + * physical memory manager after the TLB has been updated. + */ +static __inline void +pmap_add_delayed_free_list(vm_page_t m, struct spglist *free) { - struct md_page *pvh; - pv_entry_t pv; - rw_assert(&pvh_global_lock, RA_WLOCKED); - if (pv_entry_count < pv_entry_high_water && - (pv = pmap_get_pv_entry(pmap, TRUE)) != NULL) { - pv->pv_va = va; - pvh = pa_to_pvh(pa); - TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_list); - return (TRUE); - } else - return (FALSE); + /* + * Put page on a list so that it is released after + * *ALL* TLB shootdown is done + */ +#ifdef PMAP_DEBUG + pmap_zero_page_check(m); +#endif + m->flags |= PG_ZERO; + SLIST_INSERT_HEAD(free, m, plinks.s.ss); } /* - * Create the pv entries for each of the pages within a superpage. + * Unwire L2 page tables page. */ static void -pmap_pv_demote_section(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) +pmap_unwire_pt2pg(pmap_t pmap, vm_offset_t va, vm_page_t m) { - struct md_page *pvh; - pv_entry_t pve, pv; - vm_offset_t va_last; - vm_page_t m; + pt1_entry_t *pte1p, opte1 __unused; + pt2_entry_t *pte2p; + uint32_t i; - rw_assert(&pvh_global_lock, RA_WLOCKED); - KASSERT((pa & L1_S_OFFSET) == 0, - ("pmap_pv_demote_section: pa is not 1mpage aligned")); + KASSERT(pt2pg_is_empty(m), + ("%s: pmap %p PT2PG %p wired", __func__, pmap, m)); /* - * Transfer the 1mpage's pv entry for this mapping to the first - * page's pv list. + * Unmap all L2 page tables in the page from L1 page table. + * + * QQQ: Individual L2 page tables (except the last one) can be unmapped + * earlier. However, we are doing that this way. */ - pvh = pa_to_pvh(pa); - va = trunc_1mpage(va); - pv = pmap_pvh_remove(pvh, pmap, va); - KASSERT(pv != NULL, ("pmap_pv_demote_section: pv not found")); - m = PHYS_TO_VM_PAGE(pa); - TAILQ_INSERT_HEAD(&m->md.pv_list, pv, pv_list); - /* Instantiate the remaining pv entries. */ - va_last = L2_NEXT_BUCKET(va) - PAGE_SIZE; - do { - m++; - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_pv_demote_section: page %p is not managed", m)); - va += PAGE_SIZE; - pve = pmap_get_pv_entry(pmap, FALSE); - pmap_enter_pv(m, pve, pmap, va, pv->pv_flags); - } while (va < va_last); -} + KASSERT(m->pindex == (pte1_index(va) & ~PT2PG_MASK), + ("%s: pmap %p va %#x PT2PG %p bad index", __func__, pmap, va, m)); + pte1p = pmap->pm_pt1 + m->pindex; + for (i = 0; i < NPT2_IN_PG; i++, pte1p++) { + KASSERT(m->md.pt2_wirecount[i] == 0, + ("%s: pmap %p PT2 %u (PG %p) wired", __func__, pmap, i, m)); + opte1 = pte1_load(pte1p); + if (pte1_is_link(opte1)) { + pte1_clear(pte1p); + /* + * Flush intermediate TLB cache. + */ + pmap_tlb_flush(pmap, (m->pindex + i) << PTE1_SHIFT); + } +#ifdef INVARIANTS + else + KASSERT((opte1 == 0) || pte1_is_section(opte1), + ("%s: pmap %p va %#x bad pte1 %x at %u", __func__, + pmap, va, opte1, i)); +#endif + } -static void -pmap_pv_promote_section(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) -{ - struct md_page *pvh; - pv_entry_t pv; - vm_offset_t va_last; - vm_page_t m; + /* + * Unmap the page from PT2TAB. + */ + pte2p = pmap_pt2tab_entry(pmap, va); + (void)pt2tab_load_clear(pte2p); + pmap_tlb_flush(pmap, pt2map_pt2pg(va)); - rw_assert(&pvh_global_lock, RA_WLOCKED); - KASSERT((pa & L1_S_OFFSET) == 0, - ("pmap_pv_promote_section: pa is not 1mpage aligned")); + m->wire_count = 0; + pmap->pm_stats.resident_count--; /* - * Transfer the first page's pv entry for this mapping to the - * 1mpage's pv list. Aside from avoiding the cost of a call - * to get_pv_entry(), a transfer avoids the possibility that - * get_pv_entry() calls pmap_pv_reclaim() and that pmap_pv_reclaim() - * removes one of the mappings that is being promoted. + * This is a release store so that the ordinary store unmapping + * the L2 page table page is globally performed before TLB shoot- + * down is begun. */ - m = PHYS_TO_VM_PAGE(pa); - va = trunc_1mpage(va); - pv = pmap_pvh_remove(&m->md, pmap, va); - KASSERT(pv != NULL, ("pmap_pv_promote_section: pv not found")); - pvh = pa_to_pvh(pa); - TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_list); - /* Free the remaining pv entries in the newly mapped section pages */ - va_last = L2_NEXT_BUCKET(va) - PAGE_SIZE; - do { - m++; - va += PAGE_SIZE; - /* - * Don't care the flags, first pv contains sufficient - * information for all of the pages so nothing is really lost. - */ - pmap_pvh_free(&m->md, pmap, va); - } while (va < va_last); + atomic_subtract_rel_int(&vm_cnt.v_wire_count, 1); } /* - * Tries to create a 1MB page mapping. Returns TRUE if successful and - * FALSE otherwise. Fails if (1) page is unmanageg, kernel pmap or vectors - * page, (2) a mapping already exists at the specified virtual address, or - * (3) a pv entry cannot be allocated without reclaiming another pv entry. + * Decrements a L2 page table page's wire count, which is used to record the + * number of valid page table entries within the page. If the wire count + * drops to zero, then the page table page is unmapped. Returns TRUE if the + * page table page was unmapped and FALSE otherwise. */ -static boolean_t -pmap_enter_section(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) +static __inline boolean_t +pmap_unwire_pt2(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free) { - pd_entry_t *pl1pd; - vm_offset_t pa; - struct l2_bucket *l2b; - - rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_ASSERT_LOCKED(pmap); - - /* Skip kernel, vectors page and unmanaged mappings */ - if ((pmap == pmap_kernel()) || (L1_IDX(va) == L1_IDX(vector_page)) || - ((m->oflags & VPO_UNMANAGED) != 0)) { - CTR2(KTR_PMAP, "pmap_enter_section: failure for va %#lx" - " in pmap %p", va, pmap); + pt2_wirecount_dec(m, pte1_index(va)); + if (pt2pg_is_empty(m)) { + /* + * QQQ: Wire count is zero, so whole page should be zero and + * we can set PG_ZERO flag to it. + * Note that when promotion is enabled, it takes some + * more efforts. See pmap_unwire_pt2_all() below. + */ + pmap_unwire_pt2pg(pmap, va, m); + pmap_add_delayed_free_list(m, free); + return (TRUE); + } else return (FALSE); - } - /* - * Check whether this is a valid section superpage entry or - * there is a l2_bucket associated with that L1 page directory. - */ - va = trunc_1mpage(va); - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - l2b = pmap_get_l2_bucket(pmap, va); - if ((*pl1pd & L1_S_PROTO) || (l2b != NULL)) { - CTR2(KTR_PMAP, "pmap_enter_section: failure for va %#lx" - " in pmap %p", va, pmap); - return (FALSE); - } - pa = VM_PAGE_TO_PHYS(m); - /* - * Abort this mapping if its PV entry could not be created. - */ - if (!pmap_pv_insert_section(pmap, va, VM_PAGE_TO_PHYS(m))) { - CTR2(KTR_PMAP, "pmap_enter_section: failure for va %#lx" - " in pmap %p", va, pmap); - return (FALSE); - } - /* - * Increment counters. - */ - pmap->pm_stats.resident_count += L2_PTE_NUM_TOTAL; - /* - * Despite permissions, mark the superpage read-only. - */ - prot &= ~VM_PROT_WRITE; - /* - * Map the superpage. - */ - pmap_map_section(pmap, va, pa, prot, FALSE); - - pmap_section_mappings++; - CTR2(KTR_PMAP, "pmap_enter_section: success for va %#lx" - " in pmap %p", va, pmap); - return (TRUE); } /* - * pmap_remove_section: do the things to unmap a superpage in a process + * Drop a L2 page table page's wire count at once, which is used to record + * the number of valid L2 page table entries within the page. If the wire + * count drops to zero, then the L2 page table page is unmapped. */ -static void -pmap_remove_section(pmap_t pmap, vm_offset_t sva) +static __inline void +pmap_unwire_pt2_all(pmap_t pmap, vm_offset_t va, vm_page_t m, + struct spglist *free) { - struct md_page *pvh; - struct l2_bucket *l2b; - pd_entry_t *pl1pd, l1pd; - vm_offset_t eva, va; - vm_page_t m; + u_int pte1_idx = pte1_index(va); - PMAP_ASSERT_LOCKED(pmap); - if ((pmap == pmap_kernel()) || (L1_IDX(sva) == L1_IDX(vector_page))) - return; + KASSERT(m->pindex == (pte1_idx & ~PT2PG_MASK), + ("%s: PT2 page's pindex is wrong", __func__)); + KASSERT(m->wire_count > pt2_wirecount_get(m, pte1_idx), + ("%s: bad pt2 wire count %u > %u", __func__, m->wire_count, + pt2_wirecount_get(m, pte1_idx))); - KASSERT((sva & L1_S_OFFSET) == 0, - ("pmap_remove_section: sva is not 1mpage aligned")); + /* + * It's possible that the L2 page table was never used. + * It happened in case that a section was created without promotion. + */ + if (pt2_is_full(m, va)) { + pt2_wirecount_set(m, pte1_idx, 0); - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(sva)]; - l1pd = *pl1pd; - - m = PHYS_TO_VM_PAGE(l1pd & L1_S_FRAME); - KASSERT((m != NULL && ((m->oflags & VPO_UNMANAGED) == 0)), - ("pmap_remove_section: no corresponding vm_page or " - "page unmanaged")); - - pmap->pm_stats.resident_count -= L2_PTE_NUM_TOTAL; - pvh = pa_to_pvh(l1pd & L1_S_FRAME); - pmap_pvh_free(pvh, pmap, sva); - eva = L2_NEXT_BUCKET(sva); - for (va = sva, m = PHYS_TO_VM_PAGE(l1pd & L1_S_FRAME); - va < eva; va += PAGE_SIZE, m++) { /* - * Mark base pages referenced but skip marking them dirty. - * If the superpage is writeable, hence all base pages were - * already marked as dirty in pmap_fault_fixup() before - * promotion. Reference bit however, might not have been set - * for each base page when the superpage was created at once, - * not as a result of promotion. + * QQQ: We clear L2 page table now, so when L2 page table page + * is going to be freed, we can set it PG_ZERO flag ... + * This function is called only on section mappings, so + * hopefully it's not to big overload. + * + * XXX: If pmap is current, existing PT2MAP mapping could be + * used for zeroing. */ - if (L1_S_REFERENCED(l1pd)) - vm_page_aflag_set(m, PGA_REFERENCED); - if (TAILQ_EMPTY(&m->md.pv_list) && - TAILQ_EMPTY(&pvh->pv_list)) - vm_page_aflag_clear(m, PGA_WRITEABLE); + pmap_zero_page_area(m, page_pt2off(pte1_idx), NB_IN_PT2); } - - l2b = pmap_get_l2_bucket(pmap, sva); - if (l2b != NULL) { - KASSERT(l2b->l2b_occupancy == L2_PTE_NUM_TOTAL, - ("pmap_remove_section: l2_bucket occupancy error")); - pmap_free_l2_bucket(pmap, l2b, L2_PTE_NUM_TOTAL); - } - /* Now invalidate L1 slot */ - *pl1pd = 0; - PTE_SYNC(pl1pd); - if (L1_S_EXECUTABLE(l1pd)) - cpu_tlb_flushID_SE(sva); +#ifdef INVARIANTS else - cpu_tlb_flushD_SE(sva); - cpu_cpwait(); + KASSERT(pt2_is_empty(m, va), ("%s: PT2 is not empty (%u)", + __func__, pt2_wirecount_get(m, pte1_idx))); +#endif + if (pt2pg_is_empty(m)) { + pmap_unwire_pt2pg(pmap, va, m); + pmap_add_delayed_free_list(m, free); + } } /* - * Tries to promote the 256, contiguous 4KB page mappings that are - * within a single l2_bucket to a single 1MB section mapping. - * For promotion to occur, two conditions must be met: (1) the 4KB page - * mappings must map aligned, contiguous physical memory and (2) the 4KB page - * mappings must have identical characteristics. + * After removing a L2 page table entry, this routine is used to + * conditionally free the page, and manage the hold/wire counts. */ -static void -pmap_promote_section(pmap_t pmap, vm_offset_t va) +static boolean_t +pmap_unuse_pt2(pmap_t pmap, vm_offset_t va, struct spglist *free) { - pt_entry_t *firstptep, firstpte, oldpte, pa, *pte; - vm_page_t m, oldm; - vm_offset_t first_va, old_va; - struct l2_bucket *l2b = NULL; - vm_prot_t prot; - struct pv_entry *pve, *first_pve; + pt1_entry_t pte1; + vm_page_t mpte; - PMAP_ASSERT_LOCKED(pmap); + if (va >= VM_MAXUSER_ADDRESS) + return (FALSE); + pte1 = pte1_load(pmap_pte1(pmap, va)); + mpte = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); + return (pmap_unwire_pt2(pmap, va, mpte, free)); +} - prot = VM_PROT_ALL; - /* - * Skip promoting kernel pages. This is justified by following: - * 1. Kernel is already mapped using section mappings in each pmap - * 2. Managed mappings within the kernel are not to be promoted anyway - */ - if (pmap == pmap_kernel()) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for va %#x" - " in pmap %p", va, pmap); - return; - } - /* Do not attemp to promote vectors pages */ - if (L1_IDX(va) == L1_IDX(vector_page)) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for va %#x" - " in pmap %p", va, pmap); - return; - } - /* - * Examine the first PTE in the specified l2_bucket. Abort if this PTE - * is either invalid, unused, or does not map the first 4KB physical - * page within 1MB page. - */ - first_va = trunc_1mpage(va); - l2b = pmap_get_l2_bucket(pmap, first_va); - KASSERT(l2b != NULL, ("pmap_promote_section: trying to promote " - "not existing l2 bucket")); - firstptep = &l2b->l2b_kva[0]; +/************************************* + * + * Page management routines. + * + *************************************/ - firstpte = *firstptep; - if ((l2pte_pa(firstpte) & L1_S_OFFSET) != 0) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for va %#x" - " in pmap %p", va, pmap); - return; - } +CTASSERT(sizeof(struct pv_chunk) == PAGE_SIZE); +CTASSERT(_NPCM == 11); +CTASSERT(_NPCPV == 336); - if ((firstpte & (L2_S_PROTO | L2_S_REF)) != (L2_S_PROTO | L2_S_REF)) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for va %#x" - " in pmap %p", va, pmap); - return; - } - /* - * ARM uses pv_entry to mark particular mapping WIRED so don't promote - * unmanaged pages since it is impossible to determine, whether the - * page is wired or not if there is no corresponding pv_entry. - */ - m = PHYS_TO_VM_PAGE(l2pte_pa(firstpte)); - if (m && ((m->oflags & VPO_UNMANAGED) != 0)) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for va %#x" - " in pmap %p", va, pmap); - return; - } - first_pve = pmap_find_pv(&m->md, pmap, first_va); - /* - * PTE is modified only on write due to modified bit - * emulation. If the entry is referenced and writable - * then it is modified and we don't clear write enable. - * Otherwise, writing is disabled in PTE anyway and - * we just configure protections for the section mapping - * that is going to be created. - */ - if ((first_pve->pv_flags & PVF_WRITE) != 0) { - if (!L2_S_WRITABLE(firstpte)) { - first_pve->pv_flags &= ~PVF_WRITE; - prot &= ~VM_PROT_WRITE; - } - } else - prot &= ~VM_PROT_WRITE; +static __inline struct pv_chunk * +pv_to_chunk(pv_entry_t pv) +{ - if (!L2_S_EXECUTABLE(firstpte)) - prot &= ~VM_PROT_EXECUTE; + return ((struct pv_chunk *)((uintptr_t)pv & ~(uintptr_t)PAGE_MASK)); +} - /* - * Examine each of the other PTEs in the specified l2_bucket. - * Abort if this PTE maps an unexpected 4KB physical page or - * does not have identical characteristics to the first PTE. - */ - pa = l2pte_pa(firstpte) + ((L2_PTE_NUM_TOTAL - 1) * PAGE_SIZE); - old_va = L2_NEXT_BUCKET(first_va) - PAGE_SIZE; +#define PV_PMAP(pv) (pv_to_chunk(pv)->pc_pmap) - for (pte = (firstptep + L2_PTE_NUM_TOTAL - 1); pte > firstptep; pte--) { - oldpte = *pte; - if (l2pte_pa(oldpte) != pa) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for " - "va %#x in pmap %p", va, pmap); - return; - } - if ((oldpte & L2_S_PROMOTE) != (firstpte & L2_S_PROMOTE)) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for " - "va %#x in pmap %p", va, pmap); - return; - } - oldm = PHYS_TO_VM_PAGE(l2pte_pa(oldpte)); - if (oldm && ((oldm->oflags & VPO_UNMANAGED) != 0)) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for " - "va %#x in pmap %p", va, pmap); - return; - } +#define PC_FREE0_9 0xfffffffful /* Free values for index 0 through 9 */ +#define PC_FREE10 0x0000fffful /* Free values for index 10 */ - pve = pmap_find_pv(&oldm->md, pmap, old_va); - if (pve == NULL) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for " - "va %#x old_va %x - no pve", va, old_va); - return; - } +static const uint32_t pc_freemask[_NPCM] = { + PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, + PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, + PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, + PC_FREE0_9, PC_FREE10 +}; - if (!L2_S_WRITABLE(oldpte) && (pve->pv_flags & PVF_WRITE)) - pve->pv_flags &= ~PVF_WRITE; - if (pve->pv_flags != first_pve->pv_flags) { - pmap_section_p_failures++; - CTR2(KTR_PMAP, "pmap_promote_section: failure for " - "va %#x in pmap %p", va, pmap); - return; - } +SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_count, CTLFLAG_RD, &pv_entry_count, 0, + "Current number of pv entries"); - old_va -= PAGE_SIZE; - pa -= PAGE_SIZE; - } - /* - * Promote the pv entries. - */ - pmap_pv_promote_section(pmap, first_va, l2pte_pa(firstpte)); - /* - * Map the superpage. - */ - pmap_map_section(pmap, first_va, l2pte_pa(firstpte), prot, TRUE); - /* - * Invalidate all possible TLB mappings for small - * pages within the newly created superpage. - * Rely on the first PTE's attributes since they - * have to be consistent across all of the base pages - * within the superpage. If page is not executable it - * is at least referenced. - * The fastest way to do that is to invalidate whole - * TLB at once instead of executing 256 CP15 TLB - * invalidations by single entry. TLBs usually maintain - * several dozen entries so loss of unrelated entries is - * still a less agresive approach. - */ - if (L2_S_EXECUTABLE(firstpte)) - cpu_tlb_flushID(); - else - cpu_tlb_flushD(); - cpu_cpwait(); +#ifdef PV_STATS +static int pc_chunk_count, pc_chunk_allocs, pc_chunk_frees, pc_chunk_tryfail; - pmap_section_promotions++; - CTR2(KTR_PMAP, "pmap_promote_section: success for va %#x" - " in pmap %p", first_va, pmap); -} +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_count, CTLFLAG_RD, &pc_chunk_count, 0, + "Current number of pv entry chunks"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_allocs, CTLFLAG_RD, &pc_chunk_allocs, 0, + "Current number of pv entry chunks allocated"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_frees, CTLFLAG_RD, &pc_chunk_frees, 0, + "Current number of pv entry chunks frees"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_tryfail, CTLFLAG_RD, &pc_chunk_tryfail, + 0, "Number of times tried to get a chunk page but failed."); -/* - * Fills a l2_bucket with mappings to consecutive physical pages. - */ -static void -pmap_fill_l2b(struct l2_bucket *l2b, pt_entry_t newpte) -{ - pt_entry_t *ptep; - int i; +static long pv_entry_frees, pv_entry_allocs; +static int pv_entry_spare; - for (i = 0; i < L2_PTE_NUM_TOTAL; i++) { - ptep = &l2b->l2b_kva[i]; - *ptep = newpte; - PTE_SYNC(ptep); +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_frees, CTLFLAG_RD, &pv_entry_frees, 0, + "Current number of pv entry frees"); +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_allocs, CTLFLAG_RD, &pv_entry_allocs, + 0, "Current number of pv entry allocs"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_spare, CTLFLAG_RD, &pv_entry_spare, 0, + "Current number of spare pv entries"); +#endif - newpte += PAGE_SIZE; - } - - l2b->l2b_occupancy = L2_PTE_NUM_TOTAL; -} - /* - * Tries to demote a 1MB section mapping. If demotion fails, the - * 1MB section mapping is invalidated. + * Is given page managed? */ -static boolean_t -pmap_demote_section(pmap_t pmap, vm_offset_t va) +static __inline boolean_t +is_managed(vm_paddr_t pa) { - struct l2_bucket *l2b; - struct pv_entry *l1pdpve; - struct md_page *pvh; - pd_entry_t *pl1pd, l1pd, newl1pd; - pt_entry_t *firstptep, newpte; - vm_offset_t pa; + vm_offset_t pgnum; vm_page_t m; - PMAP_ASSERT_LOCKED(pmap); - /* - * According to assumptions described in pmap_promote_section, - * kernel is and always should be mapped using 1MB section mappings. - * What more, managed kernel pages were not to be promoted. - */ - KASSERT(pmap != pmap_kernel() && L1_IDX(va) != L1_IDX(vector_page), - ("pmap_demote_section: forbidden section mapping")); - - va = trunc_1mpage(va); - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - l1pd = *pl1pd; - KASSERT((l1pd & L1_TYPE_MASK) == L1_S_PROTO, - ("pmap_demote_section: not section or invalid section")); - - pa = l1pd & L1_S_FRAME; - m = PHYS_TO_VM_PAGE(pa); - KASSERT((m != NULL && (m->oflags & VPO_UNMANAGED) == 0), - ("pmap_demote_section: no vm_page for selected superpage or" - "unmanaged")); - - pvh = pa_to_pvh(pa); - l1pdpve = pmap_find_pv(pvh, pmap, va); - KASSERT(l1pdpve != NULL, ("pmap_demote_section: no pv entry for " - "managed page")); - - l2b = pmap_get_l2_bucket(pmap, va); - if (l2b == NULL) { - KASSERT((l1pdpve->pv_flags & PVF_WIRED) == 0, - ("pmap_demote_section: No l2_bucket for wired mapping")); - /* - * Invalidate the 1MB section mapping and return - * "failure" if the mapping was never accessed or the - * allocation of the new l2_bucket fails. - */ - if (!L1_S_REFERENCED(l1pd) || - (l2b = pmap_alloc_l2_bucket(pmap, va)) == NULL) { - /* Unmap and invalidate superpage. */ - pmap_remove_section(pmap, trunc_1mpage(va)); - CTR2(KTR_PMAP, "pmap_demote_section: failure for " - "va %#x in pmap %p", va, pmap); + pgnum = atop(pa); + if (pgnum >= first_page) { + m = PHYS_TO_VM_PAGE(pa); + if (m == NULL) return (FALSE); - } + if ((m->oflags & VPO_UNMANAGED) == 0) + return (TRUE); } + return (FALSE); +} - /* - * Now we should have corresponding l2_bucket available. - * Let's process it to recreate 256 PTEs for each base page - * within superpage. - */ - newpte = pa | L1_S_DEMOTE(l1pd); - if (m->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) - newpte |= pte_l2_s_cache_mode; +static __inline boolean_t +pte1_is_managed(pt1_entry_t pte1) +{ - /* - * If the l2_bucket is new, initialize it. - */ - if (l2b->l2b_occupancy == 0) - pmap_fill_l2b(l2b, newpte); - else { - firstptep = &l2b->l2b_kva[0]; - KASSERT(l2pte_pa(*firstptep) == (pa), - ("pmap_demote_section: firstpte and newpte map different " - "physical addresses")); - /* - * If the mapping has changed attributes, update the page table - * entries. - */ - if ((*firstptep & L2_S_PROMOTE) != (L1_S_DEMOTE(l1pd))) - pmap_fill_l2b(l2b, newpte); - } - /* Demote PV entry */ - pmap_pv_demote_section(pmap, va, pa); + return (is_managed(pte1_pa(pte1))); +} - /* Now fix-up L1 */ - newl1pd = l2b->l2b_phys | L1_C_DOM(pmap->pm_domain) | L1_C_PROTO; - *pl1pd = newl1pd; - PTE_SYNC(pl1pd); - /* Invalidate old TLB mapping */ - if (L1_S_EXECUTABLE(l1pd)) - cpu_tlb_flushID_SE(va); - else if (L1_S_REFERENCED(l1pd)) - cpu_tlb_flushD_SE(va); - cpu_cpwait(); +static __inline boolean_t +pte2_is_managed(pt2_entry_t pte2) +{ - pmap_section_demotions++; - CTR2(KTR_PMAP, "pmap_demote_section: success for va %#x" - " in pmap %p", va, pmap); - return (TRUE); + return (is_managed(pte2_pa(pte2))); } -/*************************************************** - * page management routines. - ***************************************************/ - /* - * We are in a serious low memory condition. Resort to - * drastic measures to free some pages so we can allocate - * another pv entry chunk. + * We are in a serious low memory condition. Resort to + * drastic measures to free some pages so we can allocate + * another pv entry chunk. */ static vm_page_t pmap_pv_reclaim(pmap_t locked_pmap) { struct pch newtail; struct pv_chunk *pc; - struct l2_bucket *l2b = NULL; + struct md_page *pvh; + pt1_entry_t *pte1p; pmap_t pmap; - pd_entry_t *pl1pd; - pt_entry_t *ptep; + pt2_entry_t *pte2p, tpte2; pv_entry_t pv; vm_offset_t va; - vm_page_t free, m, m_pc; + vm_page_t m, m_pc; + struct spglist free; uint32_t inuse; - int bit, field, freed, idx; + int bit, field, freed; - PMAP_ASSERT_LOCKED(locked_pmap); + PMAP_LOCK_ASSERT(locked_pmap, MA_OWNED); pmap = NULL; - free = m_pc = NULL; + m_pc = NULL; + SLIST_INIT(&free); TAILQ_INIT(&newtail); while ((pc = TAILQ_FIRST(&pv_chunks)) != NULL && (pv_vafree == 0 || - free == NULL)) { + SLIST_EMPTY(&free))) { TAILQ_REMOVE(&pv_chunks, pc, pc_lru); if (pmap != pc->pc_pmap) { if (pmap != NULL) { - cpu_tlb_flushID(); - cpu_cpwait(); if (pmap != locked_pmap) PMAP_UNLOCK(pmap); } pmap = pc->pc_pmap; /* Avoid deadlock and lock recursion. */ if (pmap > locked_pmap) PMAP_LOCK(pmap); else if (pmap != locked_pmap && !PMAP_TRYLOCK(pmap)) { pmap = NULL; TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); continue; } } /* * Destroy every non-wired, 4 KB page mapping in the chunk. */ freed = 0; for (field = 0; field < _NPCM; field++) { for (inuse = ~pc->pc_map[field] & pc_freemask[field]; inuse != 0; inuse &= ~(1UL << bit)) { bit = ffs(inuse) - 1; - idx = field * sizeof(inuse) * NBBY + bit; - pv = &pc->pc_pventry[idx]; + pv = &pc->pc_pventry[field * 32 + bit]; va = pv->pv_va; - - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - if ((*pl1pd & L1_TYPE_MASK) == L1_S_PROTO) + pte1p = pmap_pte1(pmap, va); + if (pte1_is_section(pte1_load(pte1p))) continue; - if (pv->pv_flags & PVF_WIRED) + pte2p = pmap_pte2(pmap, va); + tpte2 = pte2_load(pte2p); + if ((tpte2 & PTE2_W) == 0) + tpte2 = pte2_load_clear(pte2p); + pmap_pte2_release(pte2p); + if ((tpte2 & PTE2_W) != 0) continue; - - l2b = pmap_get_l2_bucket(pmap, va); - KASSERT(l2b != NULL, ("No l2 bucket")); - ptep = &l2b->l2b_kva[l2pte_index(va)]; - m = PHYS_TO_VM_PAGE(l2pte_pa(*ptep)); - KASSERT((vm_offset_t)m >= KERNBASE, - ("Trying to access non-existent page " - "va %x pte %x", va, *ptep)); - *ptep = 0; - PTE_SYNC(ptep); - TAILQ_REMOVE(&m->md.pv_list, pv, pv_list); - if (TAILQ_EMPTY(&m->md.pv_list)) - vm_page_aflag_clear(m, PGA_WRITEABLE); + KASSERT(tpte2 != 0, + ("pmap_pv_reclaim: pmap %p va %#x zero pte", + pmap, va)); + pmap_tlb_flush(pmap, va); + m = PHYS_TO_VM_PAGE(pte2_pa(tpte2)); + if (pte2_is_dirty(tpte2)) + vm_page_dirty(m); + if ((tpte2 & PTE2_A) != 0) + vm_page_aflag_set(m, PGA_REFERENCED); + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + if (TAILQ_EMPTY(&m->md.pv_list) && + (m->flags & PG_FICTITIOUS) == 0) { + pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); + if (TAILQ_EMPTY(&pvh->pv_list)) { + vm_page_aflag_clear(m, + PGA_WRITEABLE); + } + } pc->pc_map[field] |= 1UL << bit; + pmap_unuse_pt2(pmap, va, &free); freed++; } } - if (freed == 0) { TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); continue; } /* Every freed mapping is for a 4 KB page. */ pmap->pm_stats.resident_count -= freed; PV_STAT(pv_entry_frees += freed); PV_STAT(pv_entry_spare += freed); pv_entry_count -= freed; TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); for (field = 0; field < _NPCM; field++) if (pc->pc_map[field] != pc_freemask[field]) { TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); /* * One freed pv entry in locked_pmap is * sufficient. */ if (pmap == locked_pmap) goto out; break; } if (field == _NPCM) { PV_STAT(pv_entry_spare -= _NPCPV); PV_STAT(pc_chunk_count--); PV_STAT(pc_chunk_frees++); /* Entire chunk is free; return it. */ m_pc = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)pc)); pmap_qremove((vm_offset_t)pc, 1); - pmap_ptelist_free(&pv_vafree, (vm_offset_t)pc); + pmap_pte2list_free(&pv_vafree, (vm_offset_t)pc); break; } } out: TAILQ_CONCAT(&pv_chunks, &newtail, pc_lru); if (pmap != NULL) { - cpu_tlb_flushID(); - cpu_cpwait(); if (pmap != locked_pmap) PMAP_UNLOCK(pmap); } + if (m_pc == NULL && pv_vafree != 0 && SLIST_EMPTY(&free)) { + m_pc = SLIST_FIRST(&free); + SLIST_REMOVE_HEAD(&free, plinks.s.ss); + /* Recycle a freed page table page. */ + m_pc->wire_count = 1; + atomic_add_int(&vm_cnt.v_wire_count, 1); + } + pmap_free_zero_pages(&free); return (m_pc); } +static void +free_pv_chunk(struct pv_chunk *pc) +{ + vm_page_t m; + + TAILQ_REMOVE(&pv_chunks, pc, pc_lru); + PV_STAT(pv_entry_spare -= _NPCPV); + PV_STAT(pc_chunk_count--); + PV_STAT(pc_chunk_frees++); + /* entire chunk is free, return it */ + m = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)pc)); + pmap_qremove((vm_offset_t)pc, 1); + vm_page_unwire(m, PQ_NONE); + vm_page_free(m); + pmap_pte2list_free(&pv_vafree, (vm_offset_t)pc); +} + /* - * free the pv_entry back to the free list + * Free the pv_entry back to the free list. */ static void -pmap_free_pv_entry(pmap_t pmap, pv_entry_t pv) +free_pv_entry(pmap_t pmap, pv_entry_t pv) { struct pv_chunk *pc; - int bit, field, idx; + int idx, field, bit; rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_ASSERT_LOCKED(pmap); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); PV_STAT(pv_entry_frees++); PV_STAT(pv_entry_spare++); pv_entry_count--; pc = pv_to_chunk(pv); idx = pv - &pc->pc_pventry[0]; - field = idx / (sizeof(u_long) * NBBY); - bit = idx % (sizeof(u_long) * NBBY); + field = idx / 32; + bit = idx % 32; pc->pc_map[field] |= 1ul << bit; for (idx = 0; idx < _NPCM; idx++) if (pc->pc_map[idx] != pc_freemask[idx]) { /* * 98% of the time, pc is already at the head of the * list. If it isn't already, move it to the head. */ if (__predict_false(TAILQ_FIRST(&pmap->pm_pvchunk) != pc)) { TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); } return; } TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); - pmap_free_pv_chunk(pc); + free_pv_chunk(pc); } -static void -pmap_free_pv_chunk(struct pv_chunk *pc) -{ - vm_page_t m; - - TAILQ_REMOVE(&pv_chunks, pc, pc_lru); - PV_STAT(pv_entry_spare -= _NPCPV); - PV_STAT(pc_chunk_count--); - PV_STAT(pc_chunk_frees++); - /* entire chunk is free, return it */ - m = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)pc)); - pmap_qremove((vm_offset_t)pc, 1); - vm_page_unwire(m, PQ_NONE); - vm_page_free(m); - pmap_ptelist_free(&pv_vafree, (vm_offset_t)pc); - -} - +/* + * Get a new pv_entry, allocating a block from the system + * when needed. + */ static pv_entry_t -pmap_get_pv_entry(pmap_t pmap, boolean_t try) +get_pv_entry(pmap_t pmap, boolean_t try) { static const struct timeval printinterval = { 60, 0 }; static struct timeval lastprint; - struct pv_chunk *pc; + int bit, field; pv_entry_t pv; + struct pv_chunk *pc; vm_page_t m; - int bit, field, idx; rw_assert(&pvh_global_lock, RA_WLOCKED); - PMAP_ASSERT_LOCKED(pmap); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); PV_STAT(pv_entry_allocs++); pv_entry_count++; - if (pv_entry_count > pv_entry_high_water) if (ratecheck(&lastprint, &printinterval)) - printf("%s: Approaching the limit on PV entries.\n", - __func__); + printf("Approaching the limit on PV entries, consider " + "increasing either the vm.pmap.shpgperproc or the " + "vm.pmap.pv_entry_max tunable.\n"); retry: pc = TAILQ_FIRST(&pmap->pm_pvchunk); if (pc != NULL) { for (field = 0; field < _NPCM; field++) { if (pc->pc_map[field]) { bit = ffs(pc->pc_map[field]) - 1; break; } } if (field < _NPCM) { - idx = field * sizeof(pc->pc_map[field]) * NBBY + bit; - pv = &pc->pc_pventry[idx]; + pv = &pc->pc_pventry[field * 32 + bit]; pc->pc_map[field] &= ~(1ul << bit); /* If this was the last item, move it to tail */ for (field = 0; field < _NPCM; field++) if (pc->pc_map[field] != 0) { PV_STAT(pv_entry_spare--); return (pv); /* not full, return */ } TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); TAILQ_INSERT_TAIL(&pmap->pm_pvchunk, pc, pc_list); PV_STAT(pv_entry_spare--); return (pv); } } /* - * Access to the ptelist "pv_vafree" is synchronized by the pvh + * Access to the pte2list "pv_vafree" is synchronized by the pvh * global lock. If "pv_vafree" is currently non-empty, it will - * remain non-empty until pmap_ptelist_alloc() completes. + * remain non-empty until pmap_pte2list_alloc() completes. */ if (pv_vafree == 0 || (m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ | VM_ALLOC_WIRED)) == NULL) { if (try) { pv_entry_count--; PV_STAT(pc_chunk_tryfail++); return (NULL); } m = pmap_pv_reclaim(pmap); if (m == NULL) goto retry; } PV_STAT(pc_chunk_count++); PV_STAT(pc_chunk_allocs++); - pc = (struct pv_chunk *)pmap_ptelist_alloc(&pv_vafree); + pc = (struct pv_chunk *)pmap_pte2list_alloc(&pv_vafree); pmap_qenter((vm_offset_t)pc, &m, 1); pc->pc_pmap = pmap; pc->pc_map[0] = pc_freemask[0] & ~1ul; /* preallocated bit 0 */ for (field = 1; field < _NPCM; field++) pc->pc_map[field] = pc_freemask[field]; TAILQ_INSERT_TAIL(&pv_chunks, pc, pc_lru); pv = &pc->pc_pventry[0]; TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); PV_STAT(pv_entry_spare += _NPCPV - 1); return (pv); } /* - * Remove the given range of addresses from the specified map. - * - * It is assumed that the start and end are properly - * rounded to the page size. + * Create a pv entry for page at pa for + * (pmap, va). */ -#define PMAP_REMOVE_CLEAN_LIST_SIZE 3 -void -pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +static void +pmap_insert_entry(pmap_t pmap, vm_offset_t va, vm_page_t m) { - struct l2_bucket *l2b; - vm_offset_t next_bucket; - pd_entry_t l1pd; - pt_entry_t *ptep; - u_int total; - u_int mappings, is_exec, is_refd; - int flushall = 0; + pv_entry_t pv; + rw_assert(&pvh_global_lock, RA_WLOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + pv = get_pv_entry(pmap, FALSE); + pv->pv_va = va; + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); +} +static __inline pv_entry_t +pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +{ + pv_entry_t pv; + + rw_assert(&pvh_global_lock, RA_WLOCKED); + TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { + if (pmap == PV_PMAP(pv) && va == pv->pv_va) { + TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); + break; + } + } + return (pv); +} + +static void +pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +{ + pv_entry_t pv; + + pv = pmap_pvh_remove(pvh, pmap, va); + KASSERT(pv != NULL, ("pmap_pvh_free: pv not found")); + free_pv_entry(pmap, pv); +} + +static void +pmap_remove_entry(pmap_t pmap, vm_page_t m, vm_offset_t va) +{ + struct md_page *pvh; + + rw_assert(&pvh_global_lock, RA_WLOCKED); + pmap_pvh_free(&m->md, pmap, va); + if (TAILQ_EMPTY(&m->md.pv_list) && (m->flags & PG_FICTITIOUS) == 0) { + pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); + if (TAILQ_EMPTY(&pvh->pv_list)) + vm_page_aflag_clear(m, PGA_WRITEABLE); + } +} + +static void +pmap_pv_demote_pte1(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) +{ + struct md_page *pvh; + pv_entry_t pv; + vm_offset_t va_last; + vm_page_t m; + + rw_assert(&pvh_global_lock, RA_WLOCKED); + KASSERT((pa & PTE1_OFFSET) == 0, + ("pmap_pv_demote_pte1: pa is not 1mpage aligned")); + /* - * we lock in the pmap => pv_head direction + * Transfer the 1mpage's pv entry for this mapping to the first + * page's pv list. */ + pvh = pa_to_pvh(pa); + va = pte1_trunc(va); + pv = pmap_pvh_remove(pvh, pmap, va); + KASSERT(pv != NULL, ("pmap_pv_demote_pte1: pv not found")); + m = PHYS_TO_VM_PAGE(pa); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + /* Instantiate the remaining NPTE2_IN_PT2 - 1 pv entries. */ + va_last = va + PTE1_SIZE - PAGE_SIZE; + do { + m++; + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_pv_demote_pte1: page %p is not managed", m)); + va += PAGE_SIZE; + pmap_insert_entry(pmap, va, m); + } while (va < va_last); +} - rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap); - total = 0; - while (sva < eva) { - next_bucket = L2_NEXT_BUCKET(sva); +static void +pmap_pv_promote_pte1(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) +{ + struct md_page *pvh; + pv_entry_t pv; + vm_offset_t va_last; + vm_page_t m; + rw_assert(&pvh_global_lock, RA_WLOCKED); + KASSERT((pa & PTE1_OFFSET) == 0, + ("pmap_pv_promote_pte1: pa is not 1mpage aligned")); + + /* + * Transfer the first page's pv entry for this mapping to the + * 1mpage's pv list. Aside from avoiding the cost of a call + * to get_pv_entry(), a transfer avoids the possibility that + * get_pv_entry() calls pmap_pv_reclaim() and that pmap_pv_reclaim() + * removes one of the mappings that is being promoted. + */ + m = PHYS_TO_VM_PAGE(pa); + va = pte1_trunc(va); + pv = pmap_pvh_remove(&m->md, pmap, va); + KASSERT(pv != NULL, ("pmap_pv_promote_pte1: pv not found")); + pvh = pa_to_pvh(pa); + TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); + /* Free the remaining NPTE2_IN_PT2 - 1 pv entries. */ + va_last = va + PTE1_SIZE - PAGE_SIZE; + do { + m++; + va += PAGE_SIZE; + pmap_pvh_free(&m->md, pmap, va); + } while (va < va_last); +} + +/* + * Conditionally create a pv entry. + */ +static boolean_t +pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, vm_page_t m) +{ + pv_entry_t pv; + + rw_assert(&pvh_global_lock, RA_WLOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + if (pv_entry_count < pv_entry_high_water && + (pv = get_pv_entry(pmap, TRUE)) != NULL) { + pv->pv_va = va; + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + return (TRUE); + } else + return (FALSE); +} + +/* + * Create the pv entries for each of the pages within a section. + */ +static boolean_t +pmap_pv_insert_pte1(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) +{ + struct md_page *pvh; + pv_entry_t pv; + + rw_assert(&pvh_global_lock, RA_WLOCKED); + if (pv_entry_count < pv_entry_high_water && + (pv = get_pv_entry(pmap, TRUE)) != NULL) { + pv->pv_va = va; + pvh = pa_to_pvh(pa); + TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); + return (TRUE); + } else + return (FALSE); +} + +/* + * Tries to promote the NPTE2_IN_PT2, contiguous 4KB page mappings that are + * within a single page table page (PT2) to a single 1MB page mapping. + * For promotion to occur, two conditions must be met: (1) the 4KB page + * mappings must map aligned, contiguous physical memory and (2) the 4KB page + * mappings must have identical characteristics. + * + * Managed (PG_MANAGED) mappings within the kernel address space are not + * promoted. The reason is that kernel PTE1s are replicated in each pmap but + * pmap_remove_write(), pmap_clear_modify(), and pmap_clear_reference() only + * read the PTE1 from the kernel pmap. + */ +static void +pmap_promote_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t va) +{ + pt1_entry_t npte1; + pt2_entry_t *fpte2p, fpte2, fpte2_fav; + pt2_entry_t *pte2p, pte2; + vm_offset_t pteva __unused; + vm_page_t m __unused; + + PDEBUG(6, printf("%s(%p): try for va %#x pte1 %#x at %p\n", __func__, + pmap, va, pte1_load(pte1p), pte1p)); + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + + /* + * Examine the first PTE2 in the specified PT2. Abort if this PTE2 is + * either invalid, unused, or does not map the first 4KB physical page + * within a 1MB page. + */ + fpte2p = pmap_pte2_quick(pmap, pte1_trunc(va)); +setpte1: + fpte2 = pte2_load(fpte2p); + if ((fpte2 & ((PTE2_FRAME & PTE1_OFFSET) | PTE2_A | PTE2_V)) != + (PTE2_A | PTE2_V)) { + pmap_pte1_p_failures++; + CTR3(KTR_PMAP, "%s: failure(1) for va %#x in pmap %p", + __func__, va, pmap); + return; + } + if (pte2_is_managed(fpte2) && pmap == kernel_pmap) { + pmap_pte1_p_failures++; + CTR3(KTR_PMAP, "%s: failure(2) for va %#x in pmap %p", + __func__, va, pmap); + return; + } + if ((fpte2 & (PTE2_NM | PTE2_RO)) == PTE2_NM) { /* - * Check for large page. + * When page is not modified, PTE2_RO can be set without + * a TLB invalidation. + * + * Note: When modified bit is being set, then in hardware case, + * the TLB entry is re-read (updated) from PT2, and in + * software case (abort), the PTE2 is read from PT2 and + * TLB flushed if changed. The following cmpset() solves + * any race with setting this bit in both cases. */ - l1pd = pmap->pm_l1->l1_kva[L1_IDX(sva)]; - if ((l1pd & L1_TYPE_MASK) == L1_S_PROTO) { - KASSERT((l1pd & L1_S_DOM_MASK) != - L1_S_DOM(PMAP_DOMAIN_KERNEL), ("pmap_remove: " - "Trying to remove kernel section mapping")); + if (!pte2_cmpset(fpte2p, fpte2, fpte2 | PTE2_RO)) + goto setpte1; + fpte2 |= PTE2_RO; + } + + /* + * Examine each of the other PTE2s in the specified PT2. Abort if this + * PTE2 maps an unexpected 4KB physical page or does not have identical + * characteristics to the first PTE2. + */ + fpte2_fav = (fpte2 & (PTE2_FRAME | PTE2_A | PTE2_V)); + fpte2_fav += PTE1_SIZE - PTE2_SIZE; /* examine from the end */ + for (pte2p = fpte2p + NPTE2_IN_PT2 - 1; pte2p > fpte2p; pte2p--) { +setpte2: + pte2 = pte2_load(pte2p); + if ((pte2 & (PTE2_FRAME | PTE2_A | PTE2_V)) != fpte2_fav) { + pmap_pte1_p_failures++; + CTR3(KTR_PMAP, "%s: failure(3) for va %#x in pmap %p", + __func__, va, pmap); + return; + } + if ((pte2 & (PTE2_NM | PTE2_RO)) == PTE2_NM) { /* - * Are we removing the entire large page? If not, - * demote the mapping and fall through. + * When page is not modified, PTE2_RO can be set + * without a TLB invalidation. See note above. */ - if (sva + L1_S_SIZE == next_bucket && - eva >= next_bucket) { - pmap_remove_section(pmap, sva); - sva = next_bucket; - continue; - } else if (!pmap_demote_section(pmap, sva)) { - /* The large page mapping was destroyed. */ - sva = next_bucket; - continue; - } + if (!pte2_cmpset(pte2p, pte2, pte2 | PTE2_RO)) + goto setpte2; + pte2 |= PTE2_RO; + pteva = pte1_trunc(va) | (pte2 & PTE1_OFFSET & + PTE2_FRAME); + CTR3(KTR_PMAP, "%s: protect for va %#x in pmap %p", + __func__, pteva, pmap); } + if ((pte2 & PTE2_PROMOTE) != (fpte2 & PTE2_PROMOTE)) { + pmap_pte1_p_failures++; + CTR3(KTR_PMAP, "%s: failure(4) for va %#x in pmap %p", + __func__, va, pmap); + return; + } + + fpte2_fav -= PTE2_SIZE; + } + /* + * The page table page in its current state will stay in PT2TAB + * until the PTE1 mapping the section is demoted by pmap_demote_pte1() + * or destroyed by pmap_remove_pte1(). + * + * Note that L2 page table size is not equal to PAGE_SIZE. + */ + m = PHYS_TO_VM_PAGE(trunc_page(pte1_link_pa(pte1_load(pte1p)))); + KASSERT(m >= vm_page_array && m < &vm_page_array[vm_page_array_size], + ("%s: PT2 page is out of range", __func__)); + KASSERT(m->pindex == (pte1_index(va) & ~PT2PG_MASK), + ("%s: PT2 page's pindex is wrong", __func__)); + + /* + * Get pte1 from pte2 format. + */ + npte1 = (fpte2 & PTE1_FRAME) | ATTR_TO_L1(fpte2) | PTE1_V; + + /* + * Promote the pv entries. + */ + if (pte2_is_managed(fpte2)) + pmap_pv_promote_pte1(pmap, va, pte1_pa(npte1)); + + /* + * Map the section. + */ + if (pmap == kernel_pmap) + pmap_kenter_pte1(va, npte1); + else + pte1_store(pte1p, npte1); + /* + * Flush old small mappings. We call single pmap_tlb_flush() in + * pmap_demote_pte1() and pmap_remove_pte1(), so we must be sure that + * no small mappings survive. We assume that given pmap is current and + * don't play game with PTE2_NG. + */ + pmap_tlb_flush_range(pmap, pte1_trunc(va), PTE1_SIZE); + + pmap_pte1_promotions++; + CTR3(KTR_PMAP, "%s: success for va %#x in pmap %p", + __func__, va, pmap); + + PDEBUG(6, printf("%s(%p): success for va %#x pte1 %#x(%#x) at %p\n", + __func__, pmap, va, npte1, pte1_load(pte1p), pte1p)); +} + +/* + * Zero L2 page table page. + */ +static __inline void +pmap_clear_pt2(pt2_entry_t *fpte2p) +{ + pt2_entry_t *pte2p; + + for (pte2p = fpte2p; pte2p < fpte2p + NPTE2_IN_PT2; pte2p++) + pte2_clear(pte2p); + +} + +/* + * Removes a 1MB page mapping from the kernel pmap. + */ +static void +pmap_remove_kernel_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t va) +{ + vm_page_t m; + uint32_t pte1_idx; + pt2_entry_t *fpte2p; + vm_paddr_t pt2_pa; + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + m = pmap_pt2_page(pmap, va); + if (m == NULL) /* - * Do one L2 bucket's worth at a time. + * QQQ: Is this function called only on promoted pte1? + * We certainly do section mappings directly + * (without promotion) in kernel !!! */ - if (next_bucket > eva) - next_bucket = eva; + panic("%s: missing pt2 page", __func__); - l2b = pmap_get_l2_bucket(pmap, sva); - if (l2b == NULL) { - sva = next_bucket; - continue; + pte1_idx = pte1_index(va); + + /* + * Initialize the L2 page table. + */ + fpte2p = page_pt2(pt2map_pt2pg(va), pte1_idx); + pmap_clear_pt2(fpte2p); + + /* + * Remove the mapping. + */ + pt2_pa = page_pt2pa(VM_PAGE_TO_PHYS(m), pte1_idx); + pmap_kenter_pte1(va, PTE1_LINK(pt2_pa)); + + /* + * QQQ: We do not need to invalidate PT2MAP mapping + * as we did not change it. I.e. the L2 page table page + * was and still is mapped the same way. + */ +} + +/* + * Do the things to unmap a section in a process + */ +static void +pmap_remove_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t sva, + struct spglist *free) +{ + pt1_entry_t opte1; + struct md_page *pvh; + vm_offset_t eva, va; + vm_page_t m; + + PDEBUG(6, printf("%s(%p): va %#x pte1 %#x at %p\n", __func__, pmap, sva, + pte1_load(pte1p), pte1p)); + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + KASSERT((sva & PTE1_OFFSET) == 0, + ("%s: sva is not 1mpage aligned", __func__)); + + /* + * Clear and invalidate the mapping. It should occupy one and only TLB + * entry. So, pmap_tlb_flush() called with aligned address should be + * sufficient. + */ + opte1 = pte1_load_clear(pte1p); + pmap_tlb_flush(pmap, sva); + + if (pte1_is_wired(opte1)) + pmap->pm_stats.wired_count -= PTE1_SIZE / PAGE_SIZE; + pmap->pm_stats.resident_count -= PTE1_SIZE / PAGE_SIZE; + if (pte1_is_managed(opte1)) { + pvh = pa_to_pvh(pte1_pa(opte1)); + pmap_pvh_free(pvh, pmap, sva); + eva = sva + PTE1_SIZE; + for (va = sva, m = PHYS_TO_VM_PAGE(pte1_pa(opte1)); + va < eva; va += PAGE_SIZE, m++) { + if (pte1_is_dirty(opte1)) + vm_page_dirty(m); + if (opte1 & PTE1_A) + vm_page_aflag_set(m, PGA_REFERENCED); + if (TAILQ_EMPTY(&m->md.pv_list) && + TAILQ_EMPTY(&pvh->pv_list)) + vm_page_aflag_clear(m, PGA_WRITEABLE); } + } + if (pmap == kernel_pmap) { + /* + * L2 page table(s) can't be removed from kernel map as + * kernel counts on it (stuff around pmap_growkernel()). + */ + pmap_remove_kernel_pte1(pmap, pte1p, sva); + } else { + /* + * Get associated L2 page table page. + * It's possible that the page was never allocated. + */ + m = pmap_pt2_page(pmap, sva); + if (m != NULL) + pmap_unwire_pt2_all(pmap, sva, m, free); + } +} - ptep = &l2b->l2b_kva[l2pte_index(sva)]; - mappings = 0; +/* + * Fills L2 page table page with mappings to consecutive physical pages. + */ +static __inline void +pmap_fill_pt2(pt2_entry_t *fpte2p, pt2_entry_t npte2) +{ + pt2_entry_t *pte2p; - while (sva < next_bucket) { - struct vm_page *m; - pt_entry_t pte; - vm_paddr_t pa; + for (pte2p = fpte2p; pte2p < fpte2p + NPTE2_IN_PT2; pte2p++) { + pte2_store(pte2p, npte2); + npte2 += PTE2_SIZE; + } +} - pte = *ptep; +/* + * Tries to demote a 1MB page mapping. If demotion fails, the + * 1MB page mapping is invalidated. + */ +static boolean_t +pmap_demote_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t va) +{ + pt1_entry_t opte1, npte1; + pt2_entry_t *fpte2p, npte2; + vm_paddr_t pt2pg_pa, pt2_pa; + vm_page_t m; + struct spglist free; + uint32_t pte1_idx, isnew = 0; - if (pte == 0) { - /* - * Nothing here, move along - */ - sva += PAGE_SIZE; - ptep++; - continue; - } + PDEBUG(6, printf("%s(%p): try for va %#x pte1 %#x at %p\n", __func__, + pmap, va, pte1_load(pte1p), pte1p)); - pmap->pm_stats.resident_count--; - pa = l2pte_pa(pte); - is_exec = 0; - is_refd = 1; + PMAP_LOCK_ASSERT(pmap, MA_OWNED); - /* - * Update flags. In a number of circumstances, - * we could cluster a lot of these and do a - * number of sequential pages in one go. - */ - if ((m = PHYS_TO_VM_PAGE(pa)) != NULL) { - struct pv_entry *pve; + opte1 = pte1_load(pte1p); + KASSERT(pte1_is_section(opte1), ("%s: opte1 not a section", __func__)); - pve = pmap_remove_pv(m, pmap, sva); - if (pve) { - is_exec = PTE_BEEN_EXECD(pte); - is_refd = PTE_BEEN_REFD(pte); - pmap_free_pv_entry(pmap, pve); - } - } + if ((opte1 & PTE1_A) == 0 || (m = pmap_pt2_page(pmap, va)) == NULL) { + KASSERT(!pte1_is_wired(opte1), + ("%s: PT2 page for a wired mapping is missing", __func__)); - *ptep = 0; - PTE_SYNC(ptep); - if (pmap_is_current(pmap)) { - total++; - if (total < PMAP_REMOVE_CLEAN_LIST_SIZE) { - if (is_exec) - cpu_tlb_flushID_SE(sva); - else if (is_refd) - cpu_tlb_flushD_SE(sva); - } else if (total == PMAP_REMOVE_CLEAN_LIST_SIZE) - flushall = 1; - } + /* + * Invalidate the 1MB page mapping and return + * "failure" if the mapping was never accessed or the + * allocation of the new page table page fails. + */ + if ((opte1 & PTE1_A) == 0 || (m = vm_page_alloc(NULL, + pte1_index(va) & ~PT2PG_MASK, VM_ALLOC_NOOBJ | + VM_ALLOC_NORMAL | VM_ALLOC_WIRED)) == NULL) { + SLIST_INIT(&free); + pmap_remove_pte1(pmap, pte1p, pte1_trunc(va), &free); + pmap_free_zero_pages(&free); + CTR3(KTR_PMAP, "%s: failure for va %#x in pmap %p", + __func__, va, pmap); + return (FALSE); + } + if (va < VM_MAXUSER_ADDRESS) + pmap->pm_stats.resident_count++; - sva += PAGE_SIZE; - ptep++; - mappings++; + isnew = 1; + + /* + * We init all L2 page tables in the page even if + * we are going to change everything for one L2 page + * table in a while. + */ + pt2pg_pa = pmap_pt2pg_init(pmap, va, m); + } else { + if (va < VM_MAXUSER_ADDRESS) { + if (pt2_is_empty(m, va)) + isnew = 1; /* Demoting section w/o promotion. */ +#ifdef INVARIANTS + else + KASSERT(pt2_is_full(m, va), ("%s: bad PT2 wire" + " count %u", __func__, + pt2_wirecount_get(m, pte1_index(va)))); +#endif } + } - pmap_free_l2_bucket(pmap, l2b, mappings); + pt2pg_pa = VM_PAGE_TO_PHYS(m); + pte1_idx = pte1_index(va); + /* + * If the pmap is current, then the PT2MAP can provide access to + * the page table page (promoted L2 page tables are not unmapped). + * Otherwise, temporarily map the L2 page table page (m) into + * the kernel's address space at either PADDR1 or PADDR2. + * + * Note that L2 page table size is not equal to PAGE_SIZE. + */ + if (pmap_is_current(pmap)) + fpte2p = page_pt2(pt2map_pt2pg(va), pte1_idx); + else if (curthread->td_pinned > 0 && rw_wowned(&pvh_global_lock)) { + if (pte2_pa(pte2_load(PMAP1)) != pt2pg_pa) { + pte2_store(PMAP1, PTE2_KPT(pt2pg_pa)); +#ifdef SMP + PMAP1cpu = PCPU_GET(cpuid); +#endif + tlb_flush_local((vm_offset_t)PADDR1); + PMAP1changed++; + } else +#ifdef SMP + if (PMAP1cpu != PCPU_GET(cpuid)) { + PMAP1cpu = PCPU_GET(cpuid); + tlb_flush_local((vm_offset_t)PADDR1); + PMAP1changedcpu++; + } else +#endif + PMAP1unchanged++; + fpte2p = page_pt2((vm_offset_t)PADDR1, pte1_idx); + } else { + mtx_lock(&PMAP2mutex); + if (pte2_pa(pte2_load(PMAP2)) != pt2pg_pa) { + pte2_store(PMAP2, PTE2_KPT(pt2pg_pa)); + tlb_flush((vm_offset_t)PADDR2); + } + fpte2p = page_pt2((vm_offset_t)PADDR2, pte1_idx); } + pt2_pa = page_pt2pa(pt2pg_pa, pte1_idx); + npte1 = PTE1_LINK(pt2_pa); - rw_wunlock(&pvh_global_lock); - if (flushall) - cpu_tlb_flushID(); - cpu_cpwait(); + KASSERT((opte1 & PTE1_A) != 0, + ("%s: opte1 is missing PTE1_A", __func__)); + KASSERT((opte1 & (PTE1_NM | PTE1_RO)) != PTE1_NM, + ("%s: opte1 has PTE1_NM", __func__)); - PMAP_UNLOCK(pmap); + /* + * Get pte2 from pte1 format. + */ + npte2 = pte1_pa(opte1) | ATTR_TO_L2(opte1) | PTE2_V; + + /* + * If the L2 page table page is new, initialize it. If the mapping + * has changed attributes, update the page table entries. + */ + if (isnew != 0) { + pt2_wirecount_set(m, pte1_idx, NPTE2_IN_PT2); + pmap_fill_pt2(fpte2p, npte2); + } else if ((pte2_load(fpte2p) & PTE2_PROMOTE) != + (npte2 & PTE2_PROMOTE)) + pmap_fill_pt2(fpte2p, npte2); + + KASSERT(pte2_pa(pte2_load(fpte2p)) == pte2_pa(npte2), + ("%s: fpte2p and npte2 map different physical addresses", + __func__)); + + if (fpte2p == PADDR2) + mtx_unlock(&PMAP2mutex); + + /* + * Demote the mapping. This pmap is locked. The old PTE1 has + * PTE1_A set. If the old PTE1 has not PTE1_RO set, it also + * has not PTE1_NM set. Thus, there is no danger of a race with + * another processor changing the setting of PTE1_A and/or PTE1_NM + * between the read above and the store below. + */ + if (pmap == kernel_pmap) + pmap_kenter_pte1(va, npte1); + else + pte1_store(pte1p, npte1); + + /* + * Flush old big mapping. The mapping should occupy one and only + * TLB entry. So, pmap_tlb_flush() called with aligned address + * should be sufficient. + */ + pmap_tlb_flush(pmap, pte1_trunc(va)); + + /* + * Demote the pv entry. This depends on the earlier demotion + * of the mapping. Specifically, the (re)creation of a per- + * page pv entry might trigger the execution of pmap_pv_reclaim(), + * which might reclaim a newly (re)created per-page pv entry + * and destroy the associated mapping. In order to destroy + * the mapping, the PTE1 must have already changed from mapping + * the 1mpage to referencing the page table page. + */ + if (pte1_is_managed(opte1)) + pmap_pv_demote_pte1(pmap, va, pte1_pa(opte1)); + + pmap_pte1_demotions++; + CTR3(KTR_PMAP, "%s: success for va %#x in pmap %p", + __func__, va, pmap); + + PDEBUG(6, printf("%s(%p): success for va %#x pte1 %#x(%#x) at %p\n", + __func__, pmap, va, npte1, pte1_load(pte1p), pte1p)); + return (TRUE); } /* - * pmap_zero_page() + * Insert the given physical page (p) at + * the specified virtual address (v) in the + * target physical map with the protection requested. * - * Zero a given physical page by mapping it at a page hook point. - * In doing the zero page op, the page we zero is mapped cachable, as with - * StrongARM accesses to non-cached pages are non-burst making writing - * _any_ bulk data very slow. + * If specified, the page will be wired down, meaning + * that the related pte can not be reclaimed. + * + * NB: This is the only routine which MAY NOT lazy-evaluate + * or lose information. That is, this routine must actually + * insert this page into the given map NOW. */ -static void -pmap_zero_page_gen(vm_page_t m, int off, int size) +int +pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, + u_int flags, int8_t psind) { - struct czpages *czp; + pt1_entry_t *pte1p; + pt2_entry_t *pte2p; + pt2_entry_t npte2, opte2; + pv_entry_t pv; + vm_paddr_t opa, pa; + vm_page_t mpte2, om; + boolean_t wired; - KASSERT(TAILQ_EMPTY(&m->md.pv_list), - ("pmap_zero_page_gen: page has mappings")); + va = trunc_page(va); + mpte2 = NULL; + wired = (flags & PMAP_ENTER_WIRED) != 0; - vm_paddr_t phys = VM_PAGE_TO_PHYS(m); + KASSERT(va <= vm_max_kernel_address, ("%s: toobig", __func__)); + KASSERT(va < UPT2V_MIN_ADDRESS || va >= UPT2V_MAX_ADDRESS, + ("%s: invalid to pmap_enter page table pages (va: 0x%x)", __func__, + va)); + if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m)) + VM_OBJECT_ASSERT_LOCKED(m->object); + rw_wlock(&pvh_global_lock); + PMAP_LOCK(pmap); sched_pin(); - czp = &cpu_czpages[PCPU_GET(cpuid)]; - mtx_lock(&czp->lock); /* - * Hook in the page, zero it. + * In the case that a page table page is not + * resident, we are creating it here. */ - *czp->dstptep = L2_S_PROTO | phys | pte_l2_s_cache_mode | L2_S_REF; - pmap_set_prot(czp->dstptep, VM_PROT_WRITE, 0); - PTE_SYNC(czp->dstptep); - cpu_tlb_flushD_SE(czp->dstva); - cpu_cpwait(); + if (va < VM_MAXUSER_ADDRESS) { + mpte2 = pmap_allocpte2(pmap, va, flags); + if (mpte2 == NULL) { + KASSERT((flags & PMAP_ENTER_NOSLEEP) != 0, + ("pmap_allocpte2 failed with sleep allowed")); + sched_unpin(); + rw_wunlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + return (KERN_RESOURCE_SHORTAGE); + } + } + pte1p = pmap_pte1(pmap, va); + if (pte1_is_section(pte1_load(pte1p))) + panic("%s: attempted on 1MB page", __func__); + pte2p = pmap_pte2_quick(pmap, va); + if (pte2p == NULL) + panic("%s: invalid L1 page table entry va=%#x", __func__, va); - if (off || size != PAGE_SIZE) - bzero((void *)(czp->dstva + off), size); + om = NULL; + pa = VM_PAGE_TO_PHYS(m); + opte2 = pte2_load(pte2p); + opa = pte2_pa(opte2); + /* + * Mapping has not changed, must be protection or wiring change. + */ + if (pte2_is_valid(opte2) && (opa == pa)) { + /* + * Wiring change, just update stats. We don't worry about + * wiring PT2 pages as they remain resident as long as there + * are valid mappings in them. Hence, if a user page is wired, + * the PT2 page will be also. + */ + if (wired && !pte2_is_wired(opte2)) + pmap->pm_stats.wired_count++; + else if (!wired && pte2_is_wired(opte2)) + pmap->pm_stats.wired_count--; + + /* + * Remove extra pte2 reference + */ + if (mpte2) + pt2_wirecount_dec(mpte2, pte1_index(va)); + if (pte2_is_managed(opte2)) + om = m; + goto validate; + } + + /* + * QQQ: We think that changing physical address on writeable mapping + * is not safe. Well, maybe on kernel address space with correct + * locking, it can make a sense. However, we have no idea why + * anyone should do that on user address space. Are we wrong? + */ + KASSERT((opa == 0) || (opa == pa) || + !pte2_is_valid(opte2) || ((opte2 & PTE2_RO) != 0), + ("%s: pmap %p va %#x(%#x) opa %#x pa %#x - gotcha %#x %#x!", + __func__, pmap, va, opte2, opa, pa, flags, prot)); + + pv = NULL; + + /* + * Mapping has changed, invalidate old range and fall through to + * handle validating new mapping. + */ + if (opa) { + if (pte2_is_wired(opte2)) + pmap->pm_stats.wired_count--; + if (pte2_is_managed(opte2)) { + om = PHYS_TO_VM_PAGE(opa); + pv = pmap_pvh_remove(&om->md, pmap, va); + } + /* + * Remove extra pte2 reference + */ + if (mpte2 != NULL) + pt2_wirecount_dec(mpte2, va >> PTE1_SHIFT); + } else + pmap->pm_stats.resident_count++; + + /* + * Enter on the PV list if part of our managed memory. + */ + if ((m->oflags & VPO_UNMANAGED) == 0) { + KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva, + ("%s: managed mapping within the clean submap", __func__)); + if (pv == NULL) + pv = get_pv_entry(pmap, FALSE); + pv->pv_va = va; + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + } else if (pv != NULL) + free_pv_entry(pmap, pv); + + /* + * Increment counters + */ + if (wired) + pmap->pm_stats.wired_count++; + +validate: + /* + * Now validate mapping with desired protection/wiring. + */ + npte2 = PTE2(pa, PTE2_NM, m->md.pat_mode); + if (prot & VM_PROT_WRITE) { + if (pte2_is_managed(npte2)) + vm_page_aflag_set(m, PGA_WRITEABLE); + } else - bzero_page(czp->dstva); + npte2 |= PTE2_RO; + if ((prot & VM_PROT_EXECUTE) == 0) + npte2 |= PTE2_NX; + if (wired) + npte2 |= PTE2_W; + if (va < VM_MAXUSER_ADDRESS) + npte2 |= PTE2_U; + if (pmap != kernel_pmap) + npte2 |= PTE2_NG; /* - * Although aliasing is not possible, if we use temporary mappings with - * memory that will be mapped later as non-cached or with write-through - * caches, we might end up overwriting it when calling wbinv_all. So - * make sure caches are clean after the operation. + * If the mapping or permission bits are different, we need + * to update the pte2. + * + * QQQ: Think again and again what to do + * if the mapping is going to be changed! */ - cpu_idcache_wbinv_range(czp->dstva, size); - pmap_l2cache_wbinv_range(czp->dstva, phys, size); + if ((opte2 & ~(PTE2_NM | PTE2_A)) != (npte2 & ~(PTE2_NM | PTE2_A))) { + /* + * Sync icache if exec permission and attribute PTE2_ATTR_WB_WA + * is set. Do it now, before the mapping is stored and made + * valid for hardware table walk. If done later, there is a race + * for other threads of current process in lazy loading case. + * Don't do it for kernel memory which is mapped with exec + * permission even if the memory isn't going to hold executable + * code. The only time when icache sync is needed is after + * kernel module is loaded and the relocation info is processed. + * And it's done in elf_cpu_load_file(). + * + * QQQ: (1) Does it exist any better way where + * or how to sync icache? + * (2) Now, we do it on a page basis. + */ + if ((prot & VM_PROT_EXECUTE) && pmap != kernel_pmap && + m->md.pat_mode == PTE2_ATTR_WB_WA && + (opa != pa || (opte2 & PTE2_NX))) + cache_icache_sync_fresh(va, pa, PAGE_SIZE); - mtx_unlock(&czp->lock); + npte2 |= PTE2_A; + if (flags & VM_PROT_WRITE) + npte2 &= ~PTE2_NM; + if (opte2 & PTE2_V) { + /* Change mapping with break-before-make approach. */ + opte2 = pte2_load_clear(pte2p); + pmap_tlb_flush(pmap, va); + pte2_store(pte2p, npte2); + if (opte2 & PTE2_A) { + if (pte2_is_managed(opte2)) + vm_page_aflag_set(om, PGA_REFERENCED); + } + if (pte2_is_dirty(opte2)) { + if (pte2_is_managed(opte2)) + vm_page_dirty(om); + } + if (pte2_is_managed(opte2) && + TAILQ_EMPTY(&om->md.pv_list) && + ((om->flags & PG_FICTITIOUS) != 0 || + TAILQ_EMPTY(&pa_to_pvh(opa)->pv_list))) + vm_page_aflag_clear(om, PGA_WRITEABLE); + } else + pte2_store(pte2p, npte2); + } +#if 0 + else { + /* + * QQQ: In time when both access and not mofified bits are + * emulated by software, this should not happen. Some + * analysis is need, if this really happen. Missing + * tlb flush somewhere could be the reason. + */ + panic("%s: pmap %p va %#x opte2 %x npte2 %x !!", __func__, pmap, + va, opte2, npte2); + } +#endif + /* + * If both the L2 page table page and the reservation are fully + * populated, then attempt promotion. + */ + if ((mpte2 == NULL || pt2_is_full(mpte2, va)) && + sp_enabled && (m->flags & PG_FICTITIOUS) == 0 && + vm_reserv_level_iffullpop(m) == 0) + pmap_promote_pte1(pmap, pte1p, va); sched_unpin(); + rw_wunlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + return (KERN_SUCCESS); } /* - * pmap_zero_page zeros the specified hardware page by mapping - * the page into KVM and using bzero to clear its contents. + * Do the things to unmap a page in a process. */ -void -pmap_zero_page(vm_page_t m) +static int +pmap_remove_pte2(pmap_t pmap, pt2_entry_t *pte2p, vm_offset_t va, + struct spglist *free) { - pmap_zero_page_gen(m, 0, PAGE_SIZE); + pt2_entry_t opte2; + vm_page_t m; + + rw_assert(&pvh_global_lock, RA_WLOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + + /* Clear and invalidate the mapping. */ + opte2 = pte2_load_clear(pte2p); + pmap_tlb_flush(pmap, va); + + KASSERT(pte2_is_valid(opte2), ("%s: pmap %p va %#x not link pte2 %#x", + __func__, pmap, va, opte2)); + + if (opte2 & PTE2_W) + pmap->pm_stats.wired_count -= 1; + pmap->pm_stats.resident_count -= 1; + if (pte2_is_managed(opte2)) { + m = PHYS_TO_VM_PAGE(pte2_pa(opte2)); + if (pte2_is_dirty(opte2)) + vm_page_dirty(m); + if (opte2 & PTE2_A) + vm_page_aflag_set(m, PGA_REFERENCED); + pmap_remove_entry(pmap, m, va); + } + return (pmap_unuse_pt2(pmap, va, free)); } +/* + * Remove a single page from a process address space. + */ +static void +pmap_remove_page(pmap_t pmap, vm_offset_t va, struct spglist *free) +{ + pt2_entry_t *pte2p; + rw_assert(&pvh_global_lock, RA_WLOCKED); + KASSERT(curthread->td_pinned > 0, + ("%s: curthread not pinned", __func__)); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + if ((pte2p = pmap_pte2_quick(pmap, va)) == NULL || + !pte2_is_valid(pte2_load(pte2p))) + return; + pmap_remove_pte2(pmap, pte2p, va, free); +} + /* - * pmap_zero_page_area zeros the specified hardware page by mapping - * the page into KVM and using bzero to clear its contents. + * Remove the given range of addresses from the specified map. * - * off and size may not cover an area beyond a single hardware page. + * It is assumed that the start and end are properly + * rounded to the page size. */ void -pmap_zero_page_area(vm_page_t m, int off, int size) +pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { + vm_offset_t nextva; + pt1_entry_t *pte1p, pte1; + pt2_entry_t *pte2p, pte2; + struct spglist free; - pmap_zero_page_gen(m, off, size); -} + /* + * Perform an unsynchronized read. This is, however, safe. + */ + if (pmap->pm_stats.resident_count == 0) + return; + SLIST_INIT(&free); + rw_wlock(&pvh_global_lock); + sched_pin(); + PMAP_LOCK(pmap); + + /* + * Special handling of removing one page. A very common + * operation and easy to short circuit some code. + */ + if (sva + PAGE_SIZE == eva) { + pte1 = pte1_load(pmap_pte1(pmap, sva)); + if (pte1_is_link(pte1)) { + pmap_remove_page(pmap, sva, &free); + goto out; + } + } + + for (; sva < eva; sva = nextva) { + /* + * Calculate address for next L2 page table. + */ + nextva = pte1_trunc(sva + PTE1_SIZE); + if (nextva < sva) + nextva = eva; + if (pmap->pm_stats.resident_count == 0) + break; + + pte1p = pmap_pte1(pmap, sva); + pte1 = pte1_load(pte1p); + + /* + * Weed out invalid mappings. Note: we assume that the L1 page + * table is always allocated, and in kernel virtual. + */ + if (pte1 == 0) + continue; + + if (pte1_is_section(pte1)) { + /* + * Are we removing the entire large page? If not, + * demote the mapping and fall through. + */ + if (sva + PTE1_SIZE == nextva && eva >= nextva) { + pmap_remove_pte1(pmap, pte1p, sva, &free); + continue; + } else if (!pmap_demote_pte1(pmap, pte1p, sva)) { + /* The large page mapping was destroyed. */ + continue; + } +#ifdef INVARIANTS + else { + /* Update pte1 after demotion. */ + pte1 = pte1_load(pte1p); + } +#endif + } + + KASSERT(pte1_is_link(pte1), ("%s: pmap %p va %#x pte1 %#x at %p" + " is not link", __func__, pmap, sva, pte1, pte1p)); + + /* + * Limit our scan to either the end of the va represented + * by the current L2 page table page, or to the end of the + * range being removed. + */ + if (nextva > eva) + nextva = eva; + + for (pte2p = pmap_pte2_quick(pmap, sva); sva != nextva; + pte2p++, sva += PAGE_SIZE) { + pte2 = pte2_load(pte2p); + if (!pte2_is_valid(pte2)) + continue; + if (pmap_remove_pte2(pmap, pte2p, sva, &free)) + break; + } + } +out: + sched_unpin(); + rw_wunlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + pmap_free_zero_pages(&free); +} + /* - * pmap_zero_page_idle zeros the specified hardware page by mapping - * the page into KVM and using bzero to clear its contents. This - * is intended to be called from the vm_pagezero process only and - * outside of Giant. + * Routine: pmap_remove_all + * Function: + * Removes this physical page from + * all physical maps in which it resides. + * Reflects back modify bits to the pager. + * + * Notes: + * Original versions of this routine were very + * inefficient because they iteratively called + * pmap_remove (slow...) */ + void -pmap_zero_page_idle(vm_page_t m) +pmap_remove_all(vm_page_t m) { + struct md_page *pvh; + pv_entry_t pv; + pmap_t pmap; + pt2_entry_t *pte2p, opte2; + pt1_entry_t *pte1p; + vm_offset_t va; + struct spglist free; - pmap_zero_page(m); + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("%s: page %p is not managed", __func__, m)); + SLIST_INIT(&free); + rw_wlock(&pvh_global_lock); + sched_pin(); + if ((m->flags & PG_FICTITIOUS) != 0) + goto small_mappings; + pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); + while ((pv = TAILQ_FIRST(&pvh->pv_list)) != NULL) { + va = pv->pv_va; + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1p = pmap_pte1(pmap, va); + (void)pmap_demote_pte1(pmap, pte1p, va); + PMAP_UNLOCK(pmap); + } +small_mappings: + while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pmap->pm_stats.resident_count--; + pte1p = pmap_pte1(pmap, pv->pv_va); + KASSERT(!pte1_is_section(pte1_load(pte1p)), ("%s: found " + "a 1mpage in page %p's pv list", __func__, m)); + pte2p = pmap_pte2_quick(pmap, pv->pv_va); + opte2 = pte2_load_clear(pte2p); + pmap_tlb_flush(pmap, pv->pv_va); + KASSERT(pte2_is_valid(opte2), ("%s: pmap %p va %x zero pte2", + __func__, pmap, pv->pv_va)); + if (pte2_is_wired(opte2)) + pmap->pm_stats.wired_count--; + if (opte2 & PTE2_A) + vm_page_aflag_set(m, PGA_REFERENCED); + + /* + * Update the vm_page_t clean and reference bits. + */ + if (pte2_is_dirty(opte2)) + vm_page_dirty(m); + pmap_unuse_pt2(pmap, pv->pv_va, &free); + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + free_pv_entry(pmap, pv); + PMAP_UNLOCK(pmap); + } + vm_page_aflag_clear(m, PGA_WRITEABLE); + sched_unpin(); + rw_wunlock(&pvh_global_lock); + pmap_free_zero_pages(&free); } /* - * pmap_copy_page copies the specified (machine independent) - * page by mapping the page into virtual memory and using - * bcopy to copy the page, one machine dependent page at a - * time. + * Just subroutine for pmap_remove_pages() to reasonably satisfy + * good coding style, a.k.a. 80 character line width limit hell. */ +static __inline void +pmap_remove_pte1_quick(pmap_t pmap, pt1_entry_t pte1, pv_entry_t pv, + struct spglist *free) +{ + vm_paddr_t pa; + vm_page_t m, mt, mpt2pg; + struct md_page *pvh; + pa = pte1_pa(pte1); + m = PHYS_TO_VM_PAGE(pa); + + KASSERT(m->phys_addr == pa, ("%s: vm_page_t %p addr mismatch %#x %#x", + __func__, m, m->phys_addr, pa)); + KASSERT((m->flags & PG_FICTITIOUS) != 0 || + m < &vm_page_array[vm_page_array_size], + ("%s: bad pte1 %#x", __func__, pte1)); + + if (pte1_is_dirty(pte1)) { + for (mt = m; mt < &m[PTE1_SIZE / PAGE_SIZE]; mt++) + vm_page_dirty(mt); + } + + pmap->pm_stats.resident_count -= PTE1_SIZE / PAGE_SIZE; + pvh = pa_to_pvh(pa); + TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); + if (TAILQ_EMPTY(&pvh->pv_list)) { + for (mt = m; mt < &m[PTE1_SIZE / PAGE_SIZE]; mt++) + if (TAILQ_EMPTY(&mt->md.pv_list)) + vm_page_aflag_clear(mt, PGA_WRITEABLE); + } + mpt2pg = pmap_pt2_page(pmap, pv->pv_va); + if (mpt2pg != NULL) + pmap_unwire_pt2_all(pmap, pv->pv_va, mpt2pg, free); +} + /* - * pmap_copy_page() - * - * Copy one physical page into another, by mapping the pages into - * hook points. The same comment regarding cachability as in - * pmap_zero_page also applies here. + * Just subroutine for pmap_remove_pages() to reasonably satisfy + * good coding style, a.k.a. 80 character line width limit hell. */ +static __inline void +pmap_remove_pte2_quick(pmap_t pmap, pt2_entry_t pte2, pv_entry_t pv, + struct spglist *free) +{ + vm_paddr_t pa; + vm_page_t m; + struct md_page *pvh; + + pa = pte2_pa(pte2); + m = PHYS_TO_VM_PAGE(pa); + + KASSERT(m->phys_addr == pa, ("%s: vm_page_t %p addr mismatch %#x %#x", + __func__, m, m->phys_addr, pa)); + KASSERT((m->flags & PG_FICTITIOUS) != 0 || + m < &vm_page_array[vm_page_array_size], + ("%s: bad pte2 %#x", __func__, pte2)); + + if (pte2_is_dirty(pte2)) + vm_page_dirty(m); + + pmap->pm_stats.resident_count--; + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + if (TAILQ_EMPTY(&m->md.pv_list) && (m->flags & PG_FICTITIOUS) == 0) { + pvh = pa_to_pvh(pa); + if (TAILQ_EMPTY(&pvh->pv_list)) + vm_page_aflag_clear(m, PGA_WRITEABLE); + } + pmap_unuse_pt2(pmap, pv->pv_va, free); +} + +/* + * Remove all pages from specified address space this aids process + * exit speeds. Also, this code is special cased for current process + * only, but can have the more generic (and slightly slower) mode enabled. + * This is much faster than pmap_remove in the case of running down + * an entire address space. + */ void -pmap_copy_page_generic(vm_paddr_t src, vm_paddr_t dst) +pmap_remove_pages(pmap_t pmap) { - struct czpages *czp; + pt1_entry_t *pte1p, pte1; + pt2_entry_t *pte2p, pte2; + pv_entry_t pv; + struct pv_chunk *pc, *npc; + struct spglist free; + int field, idx; + int32_t bit; + uint32_t inuse, bitmask; + boolean_t allfree; + /* + * Assert that the given pmap is only active on the current + * CPU. Unfortunately, we cannot block another CPU from + * activating the pmap while this function is executing. + */ + KASSERT(pmap == vmspace_pmap(curthread->td_proc->p_vmspace), + ("%s: non-current pmap %p", __func__, pmap)); +#if defined(SMP) && defined(INVARIANTS) + { + cpuset_t other_cpus; + + sched_pin(); + other_cpus = pmap->pm_active; + CPU_CLR(PCPU_GET(cpuid), &other_cpus); + sched_unpin(); + KASSERT(CPU_EMPTY(&other_cpus), + ("%s: pmap %p active on other cpus", __func__, pmap)); + } +#endif + SLIST_INIT(&free); + rw_wlock(&pvh_global_lock); + PMAP_LOCK(pmap); sched_pin(); - czp = &cpu_czpages[PCPU_GET(cpuid)]; - mtx_lock(&czp->lock); + TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { + KASSERT(pc->pc_pmap == pmap, ("%s: wrong pmap %p %p", + __func__, pmap, pc->pc_pmap)); + allfree = TRUE; + for (field = 0; field < _NPCM; field++) { + inuse = (~(pc->pc_map[field])) & pc_freemask[field]; + while (inuse != 0) { + bit = ffs(inuse) - 1; + bitmask = 1UL << bit; + idx = field * 32 + bit; + pv = &pc->pc_pventry[idx]; + inuse &= ~bitmask; + /* + * Note that we cannot remove wired pages + * from a process' mapping at this time + */ + pte1p = pmap_pte1(pmap, pv->pv_va); + pte1 = pte1_load(pte1p); + if (pte1_is_section(pte1)) { + if (pte1_is_wired(pte1)) { + allfree = FALSE; + continue; + } + pte1_clear(pte1p); + pmap_remove_pte1_quick(pmap, pte1, pv, + &free); + } + else if (pte1_is_link(pte1)) { + pte2p = pt2map_entry(pv->pv_va); + pte2 = pte2_load(pte2p); + + if (!pte2_is_valid(pte2)) { + printf("%s: pmap %p va %#x " + "pte2 %#x\n", __func__, + pmap, pv->pv_va, pte2); + panic("bad pte2"); + } + + if (pte2_is_wired(pte2)) { + allfree = FALSE; + continue; + } + pte2_clear(pte2p); + pmap_remove_pte2_quick(pmap, pte2, pv, + &free); + } else { + printf("%s: pmap %p va %#x pte1 %#x\n", + __func__, pmap, pv->pv_va, pte1); + panic("bad pte1"); + } + + /* Mark free */ + PV_STAT(pv_entry_frees++); + PV_STAT(pv_entry_spare++); + pv_entry_count--; + pc->pc_map[field] |= bitmask; + } + } + if (allfree) { + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + free_pv_chunk(pc); + } + } + tlb_flush_all_ng_local(); + sched_unpin(); + rw_wunlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + pmap_free_zero_pages(&free); +} + +/* + * This code makes some *MAJOR* assumptions: + * 1. Current pmap & pmap exists. + * 2. Not wired. + * 3. Read access. + * 4. No L2 page table pages. + * but is *MUCH* faster than pmap_enter... + */ +static vm_page_t +pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, + vm_prot_t prot, vm_page_t mpt2pg) +{ + pt2_entry_t *pte2p, pte2; + vm_paddr_t pa; + struct spglist free; + uint32_t l2prot; + + KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva || + (m->oflags & VPO_UNMANAGED) != 0, + ("%s: managed mapping within the clean submap", __func__)); + rw_assert(&pvh_global_lock, RA_WLOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + /* - * Map the pages into the page hook points, copy them, and purge the - * cache for the appropriate page. + * In the case that a L2 page table page is not + * resident, we are creating it here. */ - *czp->srcptep = L2_S_PROTO | src | pte_l2_s_cache_mode | L2_S_REF; - pmap_set_prot(czp->srcptep, VM_PROT_READ, 0); - PTE_SYNC(czp->srcptep); - cpu_tlb_flushD_SE(czp->srcva); - *czp->dstptep = L2_S_PROTO | dst | pte_l2_s_cache_mode | L2_S_REF; - pmap_set_prot(czp->dstptep, VM_PROT_READ | VM_PROT_WRITE, 0); - PTE_SYNC(czp->dstptep); - cpu_tlb_flushD_SE(czp->dstva); - cpu_cpwait(); + if (va < VM_MAXUSER_ADDRESS) { + u_int pte1_idx; + pt1_entry_t pte1, *pte1p; + vm_paddr_t pt2_pa; - bcopy_page(czp->srcva, czp->dstva); + /* + * Get L1 page table things. + */ + pte1_idx = pte1_index(va); + pte1p = pmap_pte1(pmap, va); + pte1 = pte1_load(pte1p); + if (mpt2pg && (mpt2pg->pindex == (pte1_idx & ~PT2PG_MASK))) { + /* + * Each of NPT2_IN_PG L2 page tables on the page can + * come here. Make sure that associated L1 page table + * link is established. + * + * QQQ: It comes that we don't establish all links to + * L2 page tables for newly allocated L2 page + * tables page. + */ + KASSERT(!pte1_is_section(pte1), + ("%s: pte1 %#x is section", __func__, pte1)); + if (!pte1_is_link(pte1)) { + pt2_pa = page_pt2pa(VM_PAGE_TO_PHYS(mpt2pg), + pte1_idx); + pte1_store(pte1p, PTE1_LINK(pt2_pa)); + } + pt2_wirecount_inc(mpt2pg, pte1_idx); + } else { + /* + * If the L2 page table page is mapped, we just + * increment the hold count, and activate it. + */ + if (pte1_is_section(pte1)) { + return (NULL); + } else if (pte1_is_link(pte1)) { + mpt2pg = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); + pt2_wirecount_inc(mpt2pg, pte1_idx); + } else { + mpt2pg = _pmap_allocpte2(pmap, va, + PMAP_ENTER_NOSLEEP); + if (mpt2pg == NULL) + return (NULL); + } + } + } else { + mpt2pg = NULL; + } + /* - * Although aliasing is not possible, if we use temporary mappings with - * memory that will be mapped later as non-cached or with write-through - * caches, we might end up overwriting it when calling wbinv_all. So - * make sure caches are clean after the operation. + * This call to pt2map_entry() makes the assumption that we are + * entering the page into the current pmap. In order to support + * quick entry into any pmap, one would likely use pmap_pte2_quick(). + * But that isn't as quick as pt2map_entry(). */ - cpu_idcache_wbinv_range(czp->dstva, PAGE_SIZE); - pmap_l2cache_wbinv_range(czp->dstva, dst, PAGE_SIZE); + pte2p = pt2map_entry(va); + pte2 = pte2_load(pte2p); + if (pte2_is_valid(pte2)) { + if (mpt2pg != NULL) { + /* + * Remove extra pte2 reference + */ + pt2_wirecount_dec(mpt2pg, pte1_index(va)); + mpt2pg = NULL; + } + return (NULL); + } - mtx_unlock(&czp->lock); - sched_unpin(); -} + /* + * Enter on the PV list if part of our managed memory. + */ + if ((m->oflags & VPO_UNMANAGED) == 0 && + !pmap_try_insert_pv_entry(pmap, va, m)) { + if (mpt2pg != NULL) { + SLIST_INIT(&free); + if (pmap_unwire_pt2(pmap, va, mpt2pg, &free)) { + pmap_tlb_flush(pmap, va); + pmap_free_zero_pages(&free); + } -int unmapped_buf_allowed = 1; + mpt2pg = NULL; + } + return (NULL); + } + /* + * Increment counters + */ + pmap->pm_stats.resident_count++; + + /* + * Now validate mapping with RO protection + */ + pa = VM_PAGE_TO_PHYS(m); + l2prot = PTE2_RO | PTE2_NM; + if (va < VM_MAXUSER_ADDRESS) + l2prot |= PTE2_U | PTE2_NG; + if ((prot & VM_PROT_EXECUTE) == 0) + l2prot |= PTE2_NX; + else if (m->md.pat_mode == PTE2_ATTR_WB_WA && pmap != kernel_pmap) { + /* + * Sync icache if exec permission and attribute PTE2_ATTR_WB_WA + * is set. QQQ: For more info, see comments in pmap_enter(). + */ + cache_icache_sync_fresh(va, pa, PAGE_SIZE); + } + pte2_store(pte2p, PTE2(pa, l2prot, m->md.pat_mode)); + + return (mpt2pg); +} + void -pmap_copy_pages(vm_page_t ma[], vm_offset_t a_offset, vm_page_t mb[], - vm_offset_t b_offset, int xfersize) +pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) { - vm_page_t a_pg, b_pg; - vm_offset_t a_pg_offset, b_pg_offset; - int cnt; - struct czpages *czp; - sched_pin(); - czp = &cpu_czpages[PCPU_GET(cpuid)]; - mtx_lock(&czp->lock); + rw_wlock(&pvh_global_lock); + PMAP_LOCK(pmap); + (void)pmap_enter_quick_locked(pmap, va, m, prot, NULL); + rw_wunlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); +} - while (xfersize > 0) { - a_pg = ma[a_offset >> PAGE_SHIFT]; - a_pg_offset = a_offset & PAGE_MASK; - cnt = min(xfersize, PAGE_SIZE - a_pg_offset); - b_pg = mb[b_offset >> PAGE_SHIFT]; - b_pg_offset = b_offset & PAGE_MASK; - cnt = min(cnt, PAGE_SIZE - b_pg_offset); - *czp->srcptep = L2_S_PROTO | VM_PAGE_TO_PHYS(a_pg) | - pte_l2_s_cache_mode | L2_S_REF; - pmap_set_prot(czp->srcptep, VM_PROT_READ, 0); - PTE_SYNC(czp->srcptep); - cpu_tlb_flushD_SE(czp->srcva); - *czp->dstptep = L2_S_PROTO | VM_PAGE_TO_PHYS(b_pg) | - pte_l2_s_cache_mode | L2_S_REF; - pmap_set_prot(czp->dstptep, VM_PROT_READ | VM_PROT_WRITE, 0); - PTE_SYNC(czp->dstptep); - cpu_tlb_flushD_SE(czp->dstva); - cpu_cpwait(); - bcopy((char *)czp->srcva + a_pg_offset, (char *)czp->dstva + b_pg_offset, - cnt); - cpu_idcache_wbinv_range(czp->dstva + b_pg_offset, cnt); - pmap_l2cache_wbinv_range(czp->dstva + b_pg_offset, - VM_PAGE_TO_PHYS(b_pg) + b_pg_offset, cnt); - xfersize -= cnt; - a_offset += cnt; - b_offset += cnt; +/* + * Tries to create 1MB page mapping. Returns TRUE if successful and + * FALSE otherwise. Fails if (1) a page table page cannot be allocated without + * blocking, (2) a mapping already exists at the specified virtual address, or + * (3) a pv entry cannot be allocated without reclaiming another pv entry. + */ +static boolean_t +pmap_enter_pte1(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) +{ + pt1_entry_t *pte1p; + vm_paddr_t pa; + uint32_t l1prot; + + rw_assert(&pvh_global_lock, RA_WLOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + pte1p = pmap_pte1(pmap, va); + if (pte1_is_valid(pte1_load(pte1p))) { + CTR3(KTR_PMAP, "%s: failure for va %#lx in pmap %p", __func__, + va, pmap); + return (FALSE); } + if ((m->oflags & VPO_UNMANAGED) == 0) { + /* + * Abort this mapping if its PV entry could not be created. + */ + if (!pmap_pv_insert_pte1(pmap, va, VM_PAGE_TO_PHYS(m))) { + CTR3(KTR_PMAP, "%s: failure for va %#lx in pmap %p", + __func__, va, pmap); + return (FALSE); + } + } + /* + * Increment counters. + */ + pmap->pm_stats.resident_count += PTE1_SIZE / PAGE_SIZE; - mtx_unlock(&czp->lock); - sched_unpin(); + /* + * Map the section. + * + * QQQ: Why VM_PROT_WRITE is not evaluated and the mapping is + * made readonly? + */ + pa = VM_PAGE_TO_PHYS(m); + l1prot = PTE1_RO | PTE1_NM; + if (va < VM_MAXUSER_ADDRESS) + l1prot |= PTE1_U | PTE1_NG; + if ((prot & VM_PROT_EXECUTE) == 0) + l1prot |= PTE1_NX; + else if (m->md.pat_mode == PTE2_ATTR_WB_WA && pmap != kernel_pmap) { + /* + * Sync icache if exec permission and attribute PTE2_ATTR_WB_WA + * is set. QQQ: For more info, see comments in pmap_enter(). + */ + cache_icache_sync_fresh(va, pa, PTE1_SIZE); + } + pte1_store(pte1p, PTE1(pa, l1prot, ATTR_TO_L1(m->md.pat_mode))); + + pmap_pte1_mappings++; + CTR3(KTR_PMAP, "%s: success for va %#lx in pmap %p", __func__, va, + pmap); + return (TRUE); } +/* + * Maps a sequence of resident pages belonging to the same object. + * The sequence begins with the given page m_start. This page is + * mapped at the given virtual address start. Each subsequent page is + * mapped at a virtual address that is offset from start by the same + * amount as the page is offset from m_start within the object. The + * last page in the sequence is the page with the largest offset from + * m_start that can be mapped at a virtual address less than the given + * virtual address end. Not every virtual page between start and end + * is mapped; only those for which a resident page exists with the + * corresponding offset from m_start are mapped. + */ void -pmap_copy_page(vm_page_t src, vm_page_t dst) +pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, + vm_page_t m_start, vm_prot_t prot) { + vm_offset_t va; + vm_page_t m, mpt2pg; + vm_pindex_t diff, psize; - if (_arm_memcpy && PAGE_SIZE >= _min_memcpy_size && - _arm_memcpy((void *)VM_PAGE_TO_PHYS(dst), - (void *)VM_PAGE_TO_PHYS(src), PAGE_SIZE, IS_PHYSICAL) == 0) - return; + PDEBUG(6, printf("%s: pmap %p start %#x end %#x m %p prot %#x\n", + __func__, pmap, start, end, m_start, prot)); - pmap_copy_page_generic(VM_PAGE_TO_PHYS(src), VM_PAGE_TO_PHYS(dst)); + VM_OBJECT_ASSERT_LOCKED(m_start->object); + psize = atop(end - start); + mpt2pg = NULL; + m = m_start; + rw_wlock(&pvh_global_lock); + PMAP_LOCK(pmap); + while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { + va = start + ptoa(diff); + if ((va & PTE1_OFFSET) == 0 && va + PTE1_SIZE <= end && + m->psind == 1 && sp_enabled && + pmap_enter_pte1(pmap, va, m, prot)) + m = &m[PTE1_SIZE / PAGE_SIZE - 1]; + else + mpt2pg = pmap_enter_quick_locked(pmap, va, m, prot, + mpt2pg); + m = TAILQ_NEXT(m, listq); + } + rw_wunlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); } -vm_offset_t -pmap_quick_enter_page(vm_page_t m) +/* + * This code maps large physical mmap regions into the + * processor address space. Note that some shortcuts + * are taken, but the code works. + */ +void +pmap_object_init_pt(pmap_t pmap, vm_offset_t addr, vm_object_t object, + vm_pindex_t pindex, vm_size_t size) { - pt_entry_t *qmap_pte; - vm_offset_t qmap_addr; + pt1_entry_t *pte1p; + vm_paddr_t pa, pte2_pa; + vm_page_t p; + int pat_mode; + u_int l1attr, l1prot; - critical_enter(); + VM_OBJECT_ASSERT_WLOCKED(object); + KASSERT(object->type == OBJT_DEVICE || object->type == OBJT_SG, + ("%s: non-device object", __func__)); + if ((addr & PTE1_OFFSET) == 0 && (size & PTE1_OFFSET) == 0) { + if (!vm_object_populate(object, pindex, pindex + atop(size))) + return; + p = vm_page_lookup(object, pindex); + KASSERT(p->valid == VM_PAGE_BITS_ALL, + ("%s: invalid page %p", __func__, p)); + pat_mode = p->md.pat_mode; - qmap_addr = PCPU_GET(qmap_addr); - qmap_pte = PCPU_GET(qmap_pte); + /* + * Abort the mapping if the first page is not physically + * aligned to a 1MB page boundary. + */ + pte2_pa = VM_PAGE_TO_PHYS(p); + if (pte2_pa & PTE1_OFFSET) + return; - KASSERT(*qmap_pte == 0, ("pmap_quick_enter_page: PTE busy")); + /* + * Skip the first page. Abort the mapping if the rest of + * the pages are not physically contiguous or have differing + * memory attributes. + */ + p = TAILQ_NEXT(p, listq); + for (pa = pte2_pa + PAGE_SIZE; pa < pte2_pa + size; + pa += PAGE_SIZE) { + KASSERT(p->valid == VM_PAGE_BITS_ALL, + ("%s: invalid page %p", __func__, p)); + if (pa != VM_PAGE_TO_PHYS(p) || + pat_mode != p->md.pat_mode) + return; + p = TAILQ_NEXT(p, listq); + } - *qmap_pte = L2_S_PROTO | VM_PAGE_TO_PHYS(m) | L2_S_REF; - if (m->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) - *qmap_pte |= pte_l2_s_cache_mode; - pmap_set_prot(qmap_pte, VM_PROT_READ | VM_PROT_WRITE, 0); - PTE_SYNC(qmap_pte); - cpu_tlb_flushD_SE(qmap_addr); - cpu_cpwait(); + /* + * Map using 1MB pages. + * + * QQQ: Well, we are mapping a section, so same condition must + * be hold like during promotion. It looks that only RW mapping + * is done here, so readonly mapping must be done elsewhere. + */ + l1prot = PTE1_U | PTE1_NG | PTE1_RW | PTE1_M | PTE1_A; + l1attr = ATTR_TO_L1(pat_mode); + PMAP_LOCK(pmap); + for (pa = pte2_pa; pa < pte2_pa + size; pa += PTE1_SIZE) { + pte1p = pmap_pte1(pmap, addr); + if (!pte1_is_valid(pte1_load(pte1p))) { + pte1_store(pte1p, PTE1(pa, l1prot, l1attr)); + pmap->pm_stats.resident_count += PTE1_SIZE / + PAGE_SIZE; + pmap_pte1_mappings++; + } + /* Else continue on if the PTE1 is already valid. */ + addr += PTE1_SIZE; + } + PMAP_UNLOCK(pmap); + } +} - return (qmap_addr); +/* + * Do the things to protect a 1mpage in a process. + */ +static void +pmap_protect_pte1(pmap_t pmap, pt1_entry_t *pte1p, vm_offset_t sva, + vm_prot_t prot) +{ + pt1_entry_t npte1, opte1; + vm_offset_t eva, va; + vm_page_t m; + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + KASSERT((sva & PTE1_OFFSET) == 0, + ("%s: sva is not 1mpage aligned", __func__)); +retry: + opte1 = npte1 = pte1_load(pte1p); + if (pte1_is_managed(opte1)) { + eva = sva + PTE1_SIZE; + for (va = sva, m = PHYS_TO_VM_PAGE(pte1_pa(opte1)); + va < eva; va += PAGE_SIZE, m++) + if (pte1_is_dirty(opte1)) + vm_page_dirty(m); + } + if ((prot & VM_PROT_WRITE) == 0) + npte1 |= PTE1_RO | PTE1_NM; + if ((prot & VM_PROT_EXECUTE) == 0) + npte1 |= PTE1_NX; + + /* + * QQQ: Herein, execute permission is never set. + * It only can be cleared. So, no icache + * syncing is needed. + */ + + if (npte1 != opte1) { + if (!pte1_cmpset(pte1p, opte1, npte1)) + goto retry; + pmap_tlb_flush(pmap, sva); + } } +/* + * Set the physical protection on the + * specified range of this map as requested. + */ void -pmap_quick_remove_page(vm_offset_t addr) +pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) { - pt_entry_t *qmap_pte; + boolean_t pv_lists_locked; + vm_offset_t nextva; + pt1_entry_t *pte1p, pte1; + pt2_entry_t *pte2p, opte2, npte2; - qmap_pte = PCPU_GET(qmap_pte); + KASSERT((prot & ~VM_PROT_ALL) == 0, ("invalid prot %x", prot)); + if (prot == VM_PROT_NONE) { + pmap_remove(pmap, sva, eva); + return; + } - KASSERT(addr == PCPU_GET(qmap_addr), - ("pmap_quick_remove_page: invalid address")); - KASSERT(*qmap_pte != 0, - ("pmap_quick_remove_page: PTE not in use")); + if ((prot & (VM_PROT_WRITE | VM_PROT_EXECUTE)) == + (VM_PROT_WRITE | VM_PROT_EXECUTE)) + return; - cpu_idcache_wbinv_range(addr, PAGE_SIZE); - pmap_l2cache_wbinv_range(addr, *qmap_pte & L2_S_FRAME, PAGE_SIZE); - *qmap_pte = 0; - PTE_SYNC(qmap_pte); - critical_exit(); + if (pmap_is_current(pmap)) + pv_lists_locked = FALSE; + else { + pv_lists_locked = TRUE; +resume: + rw_wlock(&pvh_global_lock); + sched_pin(); + } + + PMAP_LOCK(pmap); + for (; sva < eva; sva = nextva) { + /* + * Calculate address for next L2 page table. + */ + nextva = pte1_trunc(sva + PTE1_SIZE); + if (nextva < sva) + nextva = eva; + + pte1p = pmap_pte1(pmap, sva); + pte1 = pte1_load(pte1p); + + /* + * Weed out invalid mappings. Note: we assume that L1 page + * page table is always allocated, and in kernel virtual. + */ + if (pte1 == 0) + continue; + + if (pte1_is_section(pte1)) { + /* + * Are we protecting the entire large page? If not, + * demote the mapping and fall through. + */ + if (sva + PTE1_SIZE == nextva && eva >= nextva) { + pmap_protect_pte1(pmap, pte1p, sva, prot); + continue; + } else { + if (!pv_lists_locked) { + pv_lists_locked = TRUE; + if (!rw_try_wlock(&pvh_global_lock)) { + PMAP_UNLOCK(pmap); + goto resume; + } + sched_pin(); + } + if (!pmap_demote_pte1(pmap, pte1p, sva)) { + /* + * The large page mapping + * was destroyed. + */ + continue; + } +#ifdef INVARIANTS + else { + /* Update pte1 after demotion */ + pte1 = pte1_load(pte1p); + } +#endif + } + } + + KASSERT(pte1_is_link(pte1), ("%s: pmap %p va %#x pte1 %#x at %p" + " is not link", __func__, pmap, sva, pte1, pte1p)); + + /* + * Limit our scan to either the end of the va represented + * by the current L2 page table page, or to the end of the + * range being protected. + */ + if (nextva > eva) + nextva = eva; + + for (pte2p = pmap_pte2_quick(pmap, sva); sva != nextva; pte2p++, + sva += PAGE_SIZE) { + vm_page_t m; +retry: + opte2 = npte2 = pte2_load(pte2p); + if (!pte2_is_valid(opte2)) + continue; + + if ((prot & VM_PROT_WRITE) == 0) { + if (pte2_is_managed(opte2) && + pte2_is_dirty(opte2)) { + m = PHYS_TO_VM_PAGE(pte2_pa(opte2)); + vm_page_dirty(m); + } + npte2 |= PTE2_RO | PTE2_NM; + } + + if ((prot & VM_PROT_EXECUTE) == 0) + npte2 |= PTE2_NX; + + /* + * QQQ: Herein, execute permission is never set. + * It only can be cleared. So, no icache + * syncing is needed. + */ + + if (npte2 != opte2) { + + if (!pte2_cmpset(pte2p, opte2, npte2)) + goto retry; + pmap_tlb_flush(pmap, sva); + } + } + } + if (pv_lists_locked) { + sched_unpin(); + rw_wunlock(&pvh_global_lock); + } + PMAP_UNLOCK(pmap); } /* - * this routine returns true if a physical page resides - * in the given pmap. + * pmap_pvh_wired_mappings: + * + * Return the updated number "count" of managed mappings that are wired. */ -boolean_t -pmap_page_exists_quick(pmap_t pmap, vm_page_t m) +static int +pmap_pvh_wired_mappings(struct md_page *pvh, int count) { - struct md_page *pvh; + pmap_t pmap; + pt1_entry_t pte1; + pt2_entry_t pte2; pv_entry_t pv; - int loops = 0; - boolean_t rv; - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_page_exists_quick: page %p is not managed", m)); - rv = FALSE; - rw_wlock(&pvh_global_lock); - TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) { - if (PV_PMAP(pv) == pmap) { - rv = TRUE; - break; + rw_assert(&pvh_global_lock, RA_WLOCKED); + sched_pin(); + TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1 = pte1_load(pmap_pte1(pmap, pv->pv_va)); + if (pte1_is_section(pte1)) { + if (pte1_is_wired(pte1)) + count++; + } else { + KASSERT(pte1_is_link(pte1), + ("%s: pte1 %#x is not link", __func__, pte1)); + pte2 = pte2_load(pmap_pte2_quick(pmap, pv->pv_va)); + if (pte2_is_wired(pte2)) + count++; } - loops++; - if (loops >= 16) - break; + PMAP_UNLOCK(pmap); } - if (!rv && loops < 16 && (m->flags & PG_FICTITIOUS) == 0) { - pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); - TAILQ_FOREACH(pv, &pvh->pv_list, pv_list) { - if (PV_PMAP(pv) == pmap) { - rv = TRUE; - break; - } - loops++; - if (loops >= 16) - break; - } - } - rw_wunlock(&pvh_global_lock); - return (rv); + sched_unpin(); + return (count); } /* * pmap_page_wired_mappings: * * Return the number of managed mappings to the given physical page * that are wired. */ int pmap_page_wired_mappings(vm_page_t m) { int count; count = 0; if ((m->oflags & VPO_UNMANAGED) != 0) return (count); rw_wlock(&pvh_global_lock); count = pmap_pvh_wired_mappings(&m->md, count); if ((m->flags & PG_FICTITIOUS) == 0) { - count = pmap_pvh_wired_mappings(pa_to_pvh(VM_PAGE_TO_PHYS(m)), - count); + count = pmap_pvh_wired_mappings(pa_to_pvh(VM_PAGE_TO_PHYS(m)), + count); } rw_wunlock(&pvh_global_lock); return (count); } /* - * pmap_pvh_wired_mappings: - * - * Return the updated number "count" of managed mappings that are wired. + * Returns TRUE if any of the given mappings were used to modify + * physical memory. Otherwise, returns FALSE. Both page and 1mpage + * mappings are supported. */ -static int -pmap_pvh_wired_mappings(struct md_page *pvh, int count) +static boolean_t +pmap_is_modified_pvh(struct md_page *pvh) { pv_entry_t pv; + pt1_entry_t pte1; + pt2_entry_t pte2; + pmap_t pmap; + boolean_t rv; rw_assert(&pvh_global_lock, RA_WLOCKED); - TAILQ_FOREACH(pv, &pvh->pv_list, pv_list) { - if ((pv->pv_flags & PVF_WIRED) != 0) - count++; + rv = FALSE; + sched_pin(); + TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1 = pte1_load(pmap_pte1(pmap, pv->pv_va)); + if (pte1_is_section(pte1)) { + rv = pte1_is_dirty(pte1); + } else { + KASSERT(pte1_is_link(pte1), + ("%s: pte1 %#x is not link", __func__, pte1)); + pte2 = pte2_load(pmap_pte2_quick(pmap, pv->pv_va)); + rv = pte2_is_dirty(pte2); + } + PMAP_UNLOCK(pmap); + if (rv) + break; } - return (count); + sched_unpin(); + return (rv); } /* - * Returns TRUE if any of the given mappings were referenced and FALSE - * otherwise. Both page and section mappings are supported. + * pmap_is_modified: + * + * Return whether or not the specified physical page was modified + * in any physical maps. */ +boolean_t +pmap_is_modified(vm_page_t m) +{ + boolean_t rv; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("%s: page %p is not managed", __func__, m)); + + /* + * If the page is not exclusive busied, then PGA_WRITEABLE cannot be + * concurrently set while the object is locked. Thus, if PGA_WRITEABLE + * is clear, no PTE2s can have PG_M set. + */ + VM_OBJECT_ASSERT_WLOCKED(m->object); + if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) + return (FALSE); + rw_wlock(&pvh_global_lock); + rv = pmap_is_modified_pvh(&m->md) || + ((m->flags & PG_FICTITIOUS) == 0 && + pmap_is_modified_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m)))); + rw_wunlock(&pvh_global_lock); + return (rv); +} + +/* + * pmap_is_prefaultable: + * + * Return whether or not the specified virtual address is eligible + * for prefault. + */ +boolean_t +pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr) +{ + pt1_entry_t pte1; + pt2_entry_t pte2; + boolean_t rv; + + rv = FALSE; + PMAP_LOCK(pmap); + pte1 = pte1_load(pmap_pte1(pmap, addr)); + if (pte1_is_link(pte1)) { + pte2 = pte2_load(pt2map_entry(addr)); + rv = !pte2_is_valid(pte2) ; + } + PMAP_UNLOCK(pmap); + return (rv); +} + +/* + * Returns TRUE if any of the given mappings were referenced and FALSE + * otherwise. Both page and 1mpage mappings are supported. + */ static boolean_t pmap_is_referenced_pvh(struct md_page *pvh) { - struct l2_bucket *l2b; + pv_entry_t pv; - pd_entry_t *pl1pd; - pt_entry_t *ptep; + pt1_entry_t pte1; + pt2_entry_t pte2; pmap_t pmap; boolean_t rv; rw_assert(&pvh_global_lock, RA_WLOCKED); rv = FALSE; - TAILQ_FOREACH(pv, &pvh->pv_list, pv_list) { + sched_pin(); + TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(pv->pv_va)]; - if ((*pl1pd & L1_TYPE_MASK) == L1_S_PROTO) - rv = L1_S_REFERENCED(*pl1pd); - else { - l2b = pmap_get_l2_bucket(pmap, pv->pv_va); - ptep = &l2b->l2b_kva[l2pte_index(pv->pv_va)]; - rv = L2_S_REFERENCED(*ptep); + pte1 = pte1_load(pmap_pte1(pmap, pv->pv_va)); + if (pte1_is_section(pte1)) { + rv = (pte1 & (PTE1_A | PTE1_V)) == (PTE1_A | PTE1_V); + } else { + pte2 = pte2_load(pmap_pte2_quick(pmap, pv->pv_va)); + rv = (pte2 & (PTE2_A | PTE2_V)) == (PTE2_A | PTE2_V); } PMAP_UNLOCK(pmap); if (rv) break; } + sched_unpin(); return (rv); } /* * pmap_is_referenced: * * Return whether or not the specified physical page was referenced * in any physical maps. */ boolean_t pmap_is_referenced(vm_page_t m) { boolean_t rv; KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_is_referenced: page %p is not managed", m)); + ("%s: page %p is not managed", __func__, m)); rw_wlock(&pvh_global_lock); rv = pmap_is_referenced_pvh(&m->md) || ((m->flags & PG_FICTITIOUS) == 0 && pmap_is_referenced_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m)))); rw_wunlock(&pvh_global_lock); return (rv); } +#define PMAP_TS_REFERENCED_MAX 5 + /* * pmap_ts_referenced: * - * Return the count of reference bits for a page, clearing all of them. + * Return a count of reference bits for a page, clearing those bits. + * It is not necessary for every reference bit to be cleared, but it + * is necessary that 0 only be returned when there are truly no + * reference bits set. + * + * XXX: The exact number of bits to check and clear is a matter that + * should be tested and standardized at some point in the future for + * optimal aging of shared pages. */ int pmap_ts_referenced(vm_page_t m) { + struct md_page *pvh; + pv_entry_t pv, pvf; + pmap_t pmap; + pt1_entry_t *pte1p, opte1; + pt2_entry_t *pte2p; + vm_paddr_t pa; + int rtval = 0; KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_ts_referenced: page %p is not managed", m)); - return (pmap_clearbit(m, PVF_REF)); + ("%s: page %p is not managed", __func__, m)); + pa = VM_PAGE_TO_PHYS(m); + pvh = pa_to_pvh(pa); + rw_wlock(&pvh_global_lock); + sched_pin(); + if ((m->flags & PG_FICTITIOUS) != 0 || + (pvf = TAILQ_FIRST(&pvh->pv_list)) == NULL) + goto small_mappings; + pv = pvf; + do { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1p = pmap_pte1(pmap, pv->pv_va); + opte1 = pte1_load(pte1p); + if ((opte1 & PTE1_A) != 0) { + /* + * Since this reference bit is shared by 256 4KB pages, + * it should not be cleared every time it is tested. + * Apply a simple "hash" function on the physical page + * number, the virtual section number, and the pmap + * address to select one 4KB page out of the 256 + * on which testing the reference bit will result + * in clearing that bit. This function is designed + * to avoid the selection of the same 4KB page + * for every 1MB page mapping. + * + * On demotion, a mapping that hasn't been referenced + * is simply destroyed. To avoid the possibility of a + * subsequent page fault on a demoted wired mapping, + * always leave its reference bit set. Moreover, + * since the section is wired, the current state of + * its reference bit won't affect page replacement. + */ + if ((((pa >> PAGE_SHIFT) ^ (pv->pv_va >> PTE1_SHIFT) ^ + (uintptr_t)pmap) & (NPTE2_IN_PG - 1)) == 0 && + !pte1_is_wired(opte1)) { + pte1_clear_bit(pte1p, PTE1_A); + pmap_tlb_flush(pmap, pv->pv_va); + } + rtval++; + } + PMAP_UNLOCK(pmap); + /* Rotate the PV list if it has more than one entry. */ + if (TAILQ_NEXT(pv, pv_next) != NULL) { + TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); + TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); + } + if (rtval >= PMAP_TS_REFERENCED_MAX) + goto out; + } while ((pv = TAILQ_FIRST(&pvh->pv_list)) != pvf); +small_mappings: + if ((pvf = TAILQ_FIRST(&m->md.pv_list)) == NULL) + goto out; + pv = pvf; + do { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1p = pmap_pte1(pmap, pv->pv_va); + KASSERT(pte1_is_link(pte1_load(pte1p)), + ("%s: not found a link in page %p's pv list", __func__, m)); + + pte2p = pmap_pte2_quick(pmap, pv->pv_va); + if ((pte2_load(pte2p) & PTE2_A) != 0) { + pte2_clear_bit(pte2p, PTE2_A); + pmap_tlb_flush(pmap, pv->pv_va); + rtval++; + } + PMAP_UNLOCK(pmap); + /* Rotate the PV list if it has more than one entry. */ + if (TAILQ_NEXT(pv, pv_next) != NULL) { + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + } + } while ((pv = TAILQ_FIRST(&m->md.pv_list)) != pvf && rtval < + PMAP_TS_REFERENCED_MAX); +out: + sched_unpin(); + rw_wunlock(&pvh_global_lock); + return (rtval); } /* - * Returns TRUE if any of the given mappings were used to modify - * physical memory. Otherwise, returns FALSE. Both page and 1MB section - * mappings are supported. + * Clear the wired attribute from the mappings for the specified range of + * addresses in the given pmap. Every valid mapping within that range + * must have the wired attribute set. In contrast, invalid mappings + * cannot have the wired attribute set, so they are ignored. + * + * The wired attribute of the page table entry is not a hardware feature, + * so there is no need to invalidate any TLB entries. */ -static boolean_t -pmap_is_modified_pvh(struct md_page *pvh) +void +pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { - pd_entry_t *pl1pd; - struct l2_bucket *l2b; - pv_entry_t pv; - pt_entry_t *ptep; - pmap_t pmap; - boolean_t rv; + vm_offset_t nextva; + pt1_entry_t *pte1p, pte1; + pt2_entry_t *pte2p, pte2; + boolean_t pv_lists_locked; - rw_assert(&pvh_global_lock, RA_WLOCKED); - rv = FALSE; + if (pmap_is_current(pmap)) + pv_lists_locked = FALSE; + else { + pv_lists_locked = TRUE; +resume: + rw_wlock(&pvh_global_lock); + sched_pin(); + } + PMAP_LOCK(pmap); + for (; sva < eva; sva = nextva) { + nextva = pte1_trunc(sva + PTE1_SIZE); + if (nextva < sva) + nextva = eva; - TAILQ_FOREACH(pv, &pvh->pv_list, pv_list) { - pmap = PV_PMAP(pv); - PMAP_LOCK(pmap); - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(pv->pv_va)]; - if ((*pl1pd & L1_TYPE_MASK) == L1_S_PROTO) - rv = L1_S_WRITABLE(*pl1pd); - else { - l2b = pmap_get_l2_bucket(pmap, pv->pv_va); - ptep = &l2b->l2b_kva[l2pte_index(pv->pv_va)]; - rv = L2_S_WRITABLE(*ptep); + pte1p = pmap_pte1(pmap, sva); + pte1 = pte1_load(pte1p); + + /* + * Weed out invalid mappings. Note: we assume that L1 page + * page table is always allocated, and in kernel virtual. + */ + if (pte1 == 0) + continue; + + if (pte1_is_section(pte1)) { + if (!pte1_is_wired(pte1)) + panic("%s: pte1 %#x not wired", __func__, pte1); + + /* + * Are we unwiring the entire large page? If not, + * demote the mapping and fall through. + */ + if (sva + PTE1_SIZE == nextva && eva >= nextva) { + pte1_clear_bit(pte1p, PTE1_W); + pmap->pm_stats.wired_count -= PTE1_SIZE / + PAGE_SIZE; + continue; + } else { + if (!pv_lists_locked) { + pv_lists_locked = TRUE; + if (!rw_try_wlock(&pvh_global_lock)) { + PMAP_UNLOCK(pmap); + /* Repeat sva. */ + goto resume; + } + sched_pin(); + } + if (!pmap_demote_pte1(pmap, pte1p, sva)) + panic("%s: demotion failed", __func__); +#ifdef INVARIANTS + else { + /* Update pte1 after demotion */ + pte1 = pte1_load(pte1p); + } +#endif + } } - PMAP_UNLOCK(pmap); - if (rv) - break; - } - return (rv); + KASSERT(pte1_is_link(pte1), ("%s: pmap %p va %#x pte1 %#x at %p" + " is not link", __func__, pmap, sva, pte1, pte1p)); + + /* + * Limit our scan to either the end of the va represented + * by the current L2 page table page, or to the end of the + * range being protected. + */ + if (nextva > eva) + nextva = eva; + + for (pte2p = pmap_pte2_quick(pmap, sva); sva != nextva; pte2p++, + sva += PAGE_SIZE) { + pte2 = pte2_load(pte2p); + if (!pte2_is_valid(pte2)) + continue; + if (!pte2_is_wired(pte2)) + panic("%s: pte2 %#x is missing PTE2_W", + __func__, pte2); + + /* + * PTE2_W must be cleared atomically. Although the pmap + * lock synchronizes access to PTE2_W, another processor + * could be changing PTE2_NM and/or PTE2_A concurrently. + */ + pte2_clear_bit(pte2p, PTE2_W); + pmap->pm_stats.wired_count--; + } + } + if (pv_lists_locked) { + sched_unpin(); + rw_wunlock(&pvh_global_lock); + } + PMAP_UNLOCK(pmap); } -boolean_t -pmap_is_modified(vm_page_t m) +/* + * Clear the write and modified bits in each of the given page's mappings. + */ +void +pmap_remove_write(vm_page_t m) { - boolean_t rv; + struct md_page *pvh; + pv_entry_t next_pv, pv; + pmap_t pmap; + pt1_entry_t *pte1p; + pt2_entry_t *pte2p, opte2; + vm_offset_t va; KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_is_modified: page %p is not managed", m)); + ("%s: page %p is not managed", __func__, m)); + /* * If the page is not exclusive busied, then PGA_WRITEABLE cannot be - * concurrently set while the object is locked. Thus, if PGA_WRITEABLE - * is clear, no PTEs can have APX cleared. + * set by another thread while the object is locked. Thus, + * if PGA_WRITEABLE is clear, no page table entries need updating. */ VM_OBJECT_ASSERT_WLOCKED(m->object); if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) - return (FALSE); + return; rw_wlock(&pvh_global_lock); - rv = pmap_is_modified_pvh(&m->md) || - ((m->flags & PG_FICTITIOUS) == 0 && - pmap_is_modified_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m)))); + sched_pin(); + if ((m->flags & PG_FICTITIOUS) != 0) + goto small_mappings; + pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); + TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_next, next_pv) { + va = pv->pv_va; + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1p = pmap_pte1(pmap, va); + if (!(pte1_load(pte1p) & PTE1_RO)) + (void)pmap_demote_pte1(pmap, pte1p, va); + PMAP_UNLOCK(pmap); + } +small_mappings: + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1p = pmap_pte1(pmap, pv->pv_va); + KASSERT(!pte1_is_section(pte1_load(pte1p)), ("%s: found" + " a section in page %p's pv list", __func__, m)); + pte2p = pmap_pte2_quick(pmap, pv->pv_va); +retry: + opte2 = pte2_load(pte2p); + if (!(opte2 & PTE2_RO)) { + if (!pte2_cmpset(pte2p, opte2, + opte2 | (PTE2_RO | PTE2_NM))) + goto retry; + if (pte2_is_dirty(opte2)) + vm_page_dirty(m); + pmap_tlb_flush(pmap, pv->pv_va); + } + PMAP_UNLOCK(pmap); + } + vm_page_aflag_clear(m, PGA_WRITEABLE); + sched_unpin(); rw_wunlock(&pvh_global_lock); - return (rv); } /* * Apply the given advice to the specified range of addresses within the * given pmap. Depending on the advice, clear the referenced and/or - * modified flags in each mapping. + * modified flags in each mapping and set the mapped page's dirty field. */ void pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice) { - struct l2_bucket *l2b; - struct pv_entry *pve; - pd_entry_t l1pd; - pt_entry_t *ptep, opte, pte; - vm_offset_t next_bucket; + pt1_entry_t *pte1p, opte1; + pt2_entry_t *pte2p, pte2; + vm_offset_t pdnxt; vm_page_t m; + boolean_t pv_lists_locked; if (advice != MADV_DONTNEED && advice != MADV_FREE) return; - rw_wlock(&pvh_global_lock); + if (pmap_is_current(pmap)) + pv_lists_locked = FALSE; + else { + pv_lists_locked = TRUE; +resume: + rw_wlock(&pvh_global_lock); + sched_pin(); + } PMAP_LOCK(pmap); - for (; sva < eva; sva = next_bucket) { - next_bucket = L2_NEXT_BUCKET(sva); - if (next_bucket < sva) - next_bucket = eva; - l1pd = pmap->pm_l1->l1_kva[L1_IDX(sva)]; - if ((l1pd & L1_TYPE_MASK) == L1_S_PROTO) { - if (pmap == pmap_kernel()) + for (; sva < eva; sva = pdnxt) { + pdnxt = pte1_trunc(sva + PTE1_SIZE); + if (pdnxt < sva) + pdnxt = eva; + pte1p = pmap_pte1(pmap, sva); + opte1 = pte1_load(pte1p); + if (!pte1_is_valid(opte1)) /* XXX */ + continue; + else if (pte1_is_section(opte1)) { + if (!pte1_is_managed(opte1)) continue; - if (!pmap_demote_section(pmap, sva)) { + if (!pv_lists_locked) { + pv_lists_locked = TRUE; + if (!rw_try_wlock(&pvh_global_lock)) { + PMAP_UNLOCK(pmap); + goto resume; + } + sched_pin(); + } + if (!pmap_demote_pte1(pmap, pte1p, sva)) { /* * The large page mapping was destroyed. */ continue; } + /* * Unless the page mappings are wired, remove the * mapping to a single page so that a subsequent - * access may repromote. Since the underlying - * l2_bucket is fully populated, this removal - * never frees an entire l2_bucket. + * access may repromote. Since the underlying L2 page + * table is fully populated, this removal never + * frees a L2 page table page. */ - l2b = pmap_get_l2_bucket(pmap, sva); - KASSERT(l2b != NULL, - ("pmap_advise: no l2 bucket for " - "va 0x%#x, pmap 0x%p", sva, pmap)); - ptep = &l2b->l2b_kva[l2pte_index(sva)]; - opte = *ptep; - m = PHYS_TO_VM_PAGE(l2pte_pa(*ptep)); - KASSERT(m != NULL, - ("pmap_advise: no vm_page for demoted superpage")); - pve = pmap_find_pv(&m->md, pmap, sva); - KASSERT(pve != NULL, - ("pmap_advise: no PV entry for managed mapping")); - if ((pve->pv_flags & PVF_WIRED) == 0) { - pmap_free_l2_bucket(pmap, l2b, 1); - pve = pmap_remove_pv(m, pmap, sva); - pmap_free_pv_entry(pmap, pve); - *ptep = 0; - PTE_SYNC(ptep); - if (pmap_is_current(pmap)) { - if (PTE_BEEN_EXECD(opte)) - cpu_tlb_flushID_SE(sva); - else if (PTE_BEEN_REFD(opte)) - cpu_tlb_flushD_SE(sva); - } + if (!pte1_is_wired(opte1)) { + pte2p = pmap_pte2_quick(pmap, sva); + KASSERT(pte2_is_valid(pte2_load(pte2p)), + ("%s: invalid PTE2", __func__)); + pmap_remove_pte2(pmap, pte2p, sva, NULL); } } - if (next_bucket > eva) - next_bucket = eva; - l2b = pmap_get_l2_bucket(pmap, sva); - if (l2b == NULL) - continue; - for (ptep = &l2b->l2b_kva[l2pte_index(sva)]; - sva != next_bucket; ptep++, sva += PAGE_SIZE) { - opte = pte = *ptep; - if ((opte & L2_S_PROTO) == 0) + if (pdnxt > eva) + pdnxt = eva; + for (pte2p = pmap_pte2_quick(pmap, sva); sva != pdnxt; pte2p++, + sva += PAGE_SIZE) { + pte2 = pte2_load(pte2p); + if (!pte2_is_valid(pte2) || !pte2_is_managed(pte2)) continue; - m = PHYS_TO_VM_PAGE(l2pte_pa(opte)); - if (m == NULL || (m->oflags & VPO_UNMANAGED) != 0) - continue; - else if (L2_S_WRITABLE(opte)) { + else if (pte2_is_dirty(pte2)) { if (advice == MADV_DONTNEED) { /* - * Don't need to mark the page - * dirty as it was already marked as - * such in pmap_fault_fixup() or - * pmap_enter_locked(). - * Just clear the state. + * Future calls to pmap_is_modified() + * can be avoided by making the page + * dirty now. */ - } else - pte |= L2_APX; - - pte &= ~L2_S_REF; - *ptep = pte; - PTE_SYNC(ptep); - } else if (L2_S_REFERENCED(opte)) { - pte &= ~L2_S_REF; - *ptep = pte; - PTE_SYNC(ptep); - } else + m = PHYS_TO_VM_PAGE(pte2_pa(pte2)); + vm_page_dirty(m); + } + pte2_set_bit(pte2p, PTE2_NM); + pte2_clear_bit(pte2p, PTE2_A); + } else if ((pte2 & PTE2_A) != 0) + pte2_clear_bit(pte2p, PTE2_A); + else continue; - if (pmap_is_current(pmap)) { - if (PTE_BEEN_EXECD(opte)) - cpu_tlb_flushID_SE(sva); - else if (PTE_BEEN_REFD(opte)) - cpu_tlb_flushD_SE(sva); - } + pmap_tlb_flush(pmap, sva); } } - cpu_cpwait(); - rw_wunlock(&pvh_global_lock); + if (pv_lists_locked) { + sched_unpin(); + rw_wunlock(&pvh_global_lock); + } PMAP_UNLOCK(pmap); } /* * Clear the modify bits on the specified physical page. */ void pmap_clear_modify(vm_page_t m) { + struct md_page *pvh; + pv_entry_t next_pv, pv; + pmap_t pmap; + pt1_entry_t *pte1p, opte1; + pt2_entry_t *pte2p, opte2; + vm_offset_t va; KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_clear_modify: page %p is not managed", m)); + ("%s: page %p is not managed", __func__, m)); VM_OBJECT_ASSERT_WLOCKED(m->object); KASSERT(!vm_page_xbusied(m), - ("pmap_clear_modify: page %p is exclusive busied", m)); + ("%s: page %p is exclusive busy", __func__, m)); /* - * If the page is not PGA_WRITEABLE, then no mappings can be modified. - * If the object containing the page is locked and the page is not - * exclusive busied, then PGA_WRITEABLE cannot be concurrently set. + * If the page is not PGA_WRITEABLE, then no PTE2s can have PTE2_NM + * cleared. If the object containing the page is locked and the page + * is not exclusive busied, then PGA_WRITEABLE cannot be concurrently + * set. */ - if ((m->aflags & PGA_WRITEABLE) == 0) + if ((m->flags & PGA_WRITEABLE) == 0) return; - if (pmap_is_modified(m)) - pmap_clearbit(m, PVF_MOD); + rw_wlock(&pvh_global_lock); + sched_pin(); + if ((m->flags & PG_FICTITIOUS) != 0) + goto small_mappings; + pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); + TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_next, next_pv) { + va = pv->pv_va; + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1p = pmap_pte1(pmap, va); + opte1 = pte1_load(pte1p); + if (!(opte1 & PTE1_RO)) { + if (pmap_demote_pte1(pmap, pte1p, va) && + !pte1_is_wired(opte1)) { + /* + * Write protect the mapping to a + * single page so that a subsequent + * write access may repromote. + */ + va += VM_PAGE_TO_PHYS(m) - pte1_pa(opte1); + pte2p = pmap_pte2_quick(pmap, va); + opte2 = pte2_load(pte2p); + if ((opte2 & PTE2_V)) { + pte2_set_bit(pte2p, PTE2_NM | PTE2_RO); + vm_page_dirty(m); + pmap_tlb_flush(pmap, va); + } + } + } + PMAP_UNLOCK(pmap); + } +small_mappings: + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pte1p = pmap_pte1(pmap, pv->pv_va); + KASSERT(!pte1_is_section(pte1_load(pte1p)), ("%s: found" + " a section in page %p's pv list", __func__, m)); + pte2p = pmap_pte2_quick(pmap, pv->pv_va); + if (pte2_is_dirty(pte2_load(pte2p))) { + pte2_set_bit(pte2p, PTE2_NM); + pmap_tlb_flush(pmap, pv->pv_va); + } + PMAP_UNLOCK(pmap); + } + sched_unpin(); + rw_wunlock(&pvh_global_lock); } /* - * Clear the write and modified bits in each of the given page's mappings. + * Sets the memory attribute for the specified page. */ void -pmap_remove_write(vm_page_t m) +pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma) { - KASSERT((m->oflags & VPO_UNMANAGED) == 0, - ("pmap_remove_write: page %p is not managed", m)); + struct sysmaps *sysmaps; + vm_memattr_t oma; + vm_paddr_t pa; + oma = m->md.pat_mode; + m->md.pat_mode = ma; + + CTR5(KTR_PMAP, "%s: page %p - 0x%08X oma: %d, ma: %d", __func__, m, + VM_PAGE_TO_PHYS(m), oma, ma); + if ((m->flags & PG_FICTITIOUS) != 0) + return; +#if 0 /* - * If the page is not exclusive busied, then PGA_WRITEABLE cannot be - * set by another thread while the object is locked. Thus, - * if PGA_WRITEABLE is clear, no page table entries need updating. + * If "m" is a normal page, flush it from the cache. + * + * First, try to find an existing mapping of the page by sf + * buffer. sf_buf_invalidate_cache() modifies mapping and + * flushes the cache. */ - VM_OBJECT_ASSERT_WLOCKED(m->object); - if (vm_page_xbusied(m) || (m->aflags & PGA_WRITEABLE) != 0) - pmap_clearbit(m, PVF_WRITE); + if (sf_buf_invalidate_cache(m, oma)) + return; +#endif + /* + * If page is not mapped by sf buffer, map the page + * transient and do invalidation. + */ + if (ma != oma) { + pa = VM_PAGE_TO_PHYS(m); + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (*sysmaps->CMAP2) + panic("%s: CMAP2 busy", __func__); + pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(pa, PTE2_AP_KRW, ma)); + dcache_wbinv_poc((vm_offset_t)sysmaps->CADDR2, pa, PAGE_SIZE); + pte2_clear(sysmaps->CMAP2); + tlb_flush((vm_offset_t)sysmaps->CADDR2); + sched_unpin(); + mtx_unlock(&sysmaps->lock); + } } +/* + * Miscellaneous support routines follow + */ /* - * perform the pmap work for mincore + * Returns TRUE if the given page is mapped individually or as part of + * a 1mpage. Otherwise, returns FALSE. */ +boolean_t +pmap_page_is_mapped(vm_page_t m) +{ + boolean_t rv; + + if ((m->oflags & VPO_UNMANAGED) != 0) + return (FALSE); + rw_wlock(&pvh_global_lock); + rv = !TAILQ_EMPTY(&m->md.pv_list) || + ((m->flags & PG_FICTITIOUS) == 0 && + !TAILQ_EMPTY(&pa_to_pvh(VM_PAGE_TO_PHYS(m))->pv_list)); + rw_wunlock(&pvh_global_lock); + return (rv); +} + +/* + * Returns true if the pmap's pv is one of the first + * 16 pvs linked to from this page. This count may + * be changed upwards or downwards in the future; it + * is only necessary that true be returned for a small + * subset of pmaps for proper page aging. + */ +boolean_t +pmap_page_exists_quick(pmap_t pmap, vm_page_t m) +{ + struct md_page *pvh; + pv_entry_t pv; + int loops = 0; + boolean_t rv; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("%s: page %p is not managed", __func__, m)); + rv = FALSE; + rw_wlock(&pvh_global_lock); + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + if (PV_PMAP(pv) == pmap) { + rv = TRUE; + break; + } + loops++; + if (loops >= 16) + break; + } + if (!rv && loops < 16 && (m->flags & PG_FICTITIOUS) == 0) { + pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); + TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { + if (PV_PMAP(pv) == pmap) { + rv = TRUE; + break; + } + loops++; + if (loops >= 16) + break; + } + } + rw_wunlock(&pvh_global_lock); + return (rv); +} + +/* + * pmap_zero_page zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. + */ +void +pmap_zero_page(vm_page_t m) +{ + struct sysmaps *sysmaps; + + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (pte2_load(sysmaps->CMAP2) != 0) + panic("%s: CMAP2 busy", __func__); + pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, + m->md.pat_mode)); + pagezero(sysmaps->CADDR2); + pte2_clear(sysmaps->CMAP2); + tlb_flush((vm_offset_t)sysmaps->CADDR2); + sched_unpin(); + mtx_unlock(&sysmaps->lock); +} + +/* + * pmap_zero_page_area zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. + * + * off and size may not cover an area beyond a single hardware page. + */ +void +pmap_zero_page_area(vm_page_t m, int off, int size) +{ + struct sysmaps *sysmaps; + + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (pte2_load(sysmaps->CMAP2) != 0) + panic("%s: CMAP2 busy", __func__); + pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, + m->md.pat_mode)); + if (off == 0 && size == PAGE_SIZE) + pagezero(sysmaps->CADDR2); + else + bzero(sysmaps->CADDR2 + off, size); + pte2_clear(sysmaps->CMAP2); + tlb_flush((vm_offset_t)sysmaps->CADDR2); + sched_unpin(); + mtx_unlock(&sysmaps->lock); +} + +/* + * pmap_zero_page_idle zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. This + * is intended to be called from the vm_pagezero process only and + * outside of Giant. + */ +void +pmap_zero_page_idle(vm_page_t m) +{ + + if (pte2_load(CMAP3) != 0) + panic("%s: CMAP3 busy", __func__); + sched_pin(); + pte2_store(CMAP3, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, + m->md.pat_mode)); + pagezero(CADDR3); + pte2_clear(CMAP3); + tlb_flush((vm_offset_t)CADDR3); + sched_unpin(); +} + +/* + * pmap_copy_page copies the specified (machine independent) + * page by mapping the page into virtual memory and using + * bcopy to copy the page, one machine dependent page at a + * time. + */ +void +pmap_copy_page(vm_page_t src, vm_page_t dst) +{ + struct sysmaps *sysmaps; + + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (pte2_load(sysmaps->CMAP1) != 0) + panic("%s: CMAP1 busy", __func__); + if (pte2_load(sysmaps->CMAP2) != 0) + panic("%s: CMAP2 busy", __func__); + pte2_store(sysmaps->CMAP1, PTE2_KERN_NG(VM_PAGE_TO_PHYS(src), + PTE2_AP_KR | PTE2_NM, src->md.pat_mode)); + pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(dst), + PTE2_AP_KRW, dst->md.pat_mode)); + bcopy(sysmaps->CADDR1, sysmaps->CADDR2, PAGE_SIZE); + pte2_clear(sysmaps->CMAP1); + tlb_flush((vm_offset_t)sysmaps->CADDR1); + pte2_clear(sysmaps->CMAP2); + tlb_flush((vm_offset_t)sysmaps->CADDR2); + sched_unpin(); + mtx_unlock(&sysmaps->lock); +} + +int unmapped_buf_allowed = 1; + +void +pmap_copy_pages(vm_page_t ma[], vm_offset_t a_offset, vm_page_t mb[], + vm_offset_t b_offset, int xfersize) +{ + struct sysmaps *sysmaps; + vm_page_t a_pg, b_pg; + char *a_cp, *b_cp; + vm_offset_t a_pg_offset, b_pg_offset; + int cnt; + + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (*sysmaps->CMAP1 != 0) + panic("pmap_copy_pages: CMAP1 busy"); + if (*sysmaps->CMAP2 != 0) + panic("pmap_copy_pages: CMAP2 busy"); + while (xfersize > 0) { + a_pg = ma[a_offset >> PAGE_SHIFT]; + a_pg_offset = a_offset & PAGE_MASK; + cnt = min(xfersize, PAGE_SIZE - a_pg_offset); + b_pg = mb[b_offset >> PAGE_SHIFT]; + b_pg_offset = b_offset & PAGE_MASK; + cnt = min(cnt, PAGE_SIZE - b_pg_offset); + pte2_store(sysmaps->CMAP1, PTE2_KERN_NG(VM_PAGE_TO_PHYS(a_pg), + PTE2_AP_KR | PTE2_NM, a_pg->md.pat_mode)); + tlb_flush_local((vm_offset_t)sysmaps->CADDR1); + pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(b_pg), + PTE2_AP_KRW, b_pg->md.pat_mode)); + tlb_flush_local((vm_offset_t)sysmaps->CADDR2); + a_cp = sysmaps->CADDR1 + a_pg_offset; + b_cp = sysmaps->CADDR2 + b_pg_offset; + bcopy(a_cp, b_cp, cnt); + a_offset += cnt; + b_offset += cnt; + xfersize -= cnt; + } + pte2_clear(sysmaps->CMAP1); + tlb_flush((vm_offset_t)sysmaps->CADDR1); + pte2_clear(sysmaps->CMAP2); + tlb_flush((vm_offset_t)sysmaps->CADDR2); + sched_unpin(); + mtx_unlock(&sysmaps->lock); +} + +vm_offset_t +pmap_quick_enter_page(vm_page_t m) +{ + pt2_entry_t *pte2p; + vm_offset_t qmap_addr; + + critical_enter(); + qmap_addr = PCPU_GET(qmap_addr); + pte2p = pt2map_entry(qmap_addr); + + KASSERT(pte2_load(pte2p) == 0, ("%s: PTE2 busy", __func__)); + + pte2_store(pte2p, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, + pmap_page_get_memattr(m))); + return (qmap_addr); +} + +void +pmap_quick_remove_page(vm_offset_t addr) +{ + pt2_entry_t *pte2p; + vm_offset_t qmap_addr; + + qmap_addr = PCPU_GET(qmap_addr); + pte2p = pt2map_entry(qmap_addr); + + KASSERT(addr == qmap_addr, ("%s: invalid address", __func__)); + KASSERT(pte2_load(pte2p) != 0, ("%s: PTE2 not in use", __func__)); + + pte2_clear(pte2p); + tlb_flush(qmap_addr); + critical_exit(); +} + +/* + * Copy the range specified by src_addr/len + * from the source map to the range dst_addr/len + * in the destination map. + * + * This routine is only advisory and need not do anything. + */ +void +pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t len, + vm_offset_t src_addr) +{ + struct spglist free; + vm_offset_t addr; + vm_offset_t end_addr = src_addr + len; + vm_offset_t nextva; + + if (dst_addr != src_addr) + return; + + if (!pmap_is_current(src_pmap)) + return; + + rw_wlock(&pvh_global_lock); + if (dst_pmap < src_pmap) { + PMAP_LOCK(dst_pmap); + PMAP_LOCK(src_pmap); + } else { + PMAP_LOCK(src_pmap); + PMAP_LOCK(dst_pmap); + } + sched_pin(); + for (addr = src_addr; addr < end_addr; addr = nextva) { + pt2_entry_t *src_pte2p, *dst_pte2p; + vm_page_t dst_mpt2pg, src_mpt2pg; + pt1_entry_t src_pte1; + u_int pte1_idx; + + KASSERT(addr < VM_MAXUSER_ADDRESS, + ("%s: invalid to pmap_copy page tables", __func__)); + + nextva = pte1_trunc(addr + PTE1_SIZE); + if (nextva < addr) + nextva = end_addr; + + pte1_idx = pte1_index(addr); + src_pte1 = src_pmap->pm_pt1[pte1_idx]; + if (pte1_is_section(src_pte1)) { + if ((addr & PTE1_OFFSET) != 0 || + (addr + PTE1_SIZE) > end_addr) + continue; + if (dst_pmap->pm_pt1[pte1_idx] == 0 && + (!pte1_is_managed(src_pte1) || + pmap_pv_insert_pte1(dst_pmap, addr, + pte1_pa(src_pte1)))) { + dst_pmap->pm_pt1[pte1_idx] = src_pte1 & + ~PTE1_W; + dst_pmap->pm_stats.resident_count += + PTE1_SIZE / PAGE_SIZE; + pmap_pte1_mappings++; + } + continue; + } else if (!pte1_is_link(src_pte1)) + continue; + + src_mpt2pg = PHYS_TO_VM_PAGE(pte1_link_pa(src_pte1)); + + /* + * We leave PT2s to be linked from PT1 even if they are not + * referenced until all PT2s in a page are without reference. + * + * QQQ: It could be changed ... + */ +#if 0 /* single_pt2_link_is_cleared */ + KASSERT(pt2_wirecount_get(src_mpt2pg, pte1_idx) > 0, + ("%s: source page table page is unused", __func__)); +#else + if (pt2_wirecount_get(src_mpt2pg, pte1_idx) == 0) + continue; +#endif + if (nextva > end_addr) + nextva = end_addr; + + src_pte2p = pt2map_entry(addr); + while (addr < nextva) { + pt2_entry_t temp_pte2; + temp_pte2 = pte2_load(src_pte2p); + /* + * we only virtual copy managed pages + */ + if (pte2_is_managed(temp_pte2)) { + dst_mpt2pg = pmap_allocpte2(dst_pmap, addr, + PMAP_ENTER_NOSLEEP); + if (dst_mpt2pg == NULL) + goto out; + dst_pte2p = pmap_pte2_quick(dst_pmap, addr); + if (!pte2_is_valid(pte2_load(dst_pte2p)) && + pmap_try_insert_pv_entry(dst_pmap, addr, + PHYS_TO_VM_PAGE(pte2_pa(temp_pte2)))) { + /* + * Clear the wired, modified, and + * accessed (referenced) bits + * during the copy. + */ + temp_pte2 &= ~(PTE2_W | PTE2_A); + temp_pte2 |= PTE2_NM; + pte2_store(dst_pte2p, temp_pte2); + dst_pmap->pm_stats.resident_count++; + } else { + SLIST_INIT(&free); + if (pmap_unwire_pt2(dst_pmap, addr, + dst_mpt2pg, &free)) { + pmap_tlb_flush(dst_pmap, addr); + pmap_free_zero_pages(&free); + } + goto out; + } + if (pt2_wirecount_get(dst_mpt2pg, pte1_idx) >= + pt2_wirecount_get(src_mpt2pg, pte1_idx)) + break; + } + addr += PAGE_SIZE; + src_pte2p++; + } + } +out: + sched_unpin(); + rw_wunlock(&pvh_global_lock); + PMAP_UNLOCK(src_pmap); + PMAP_UNLOCK(dst_pmap); +} + +/* + * Increase the starting virtual address of the given mapping if a + * different alignment might result in more section mappings. + */ +void +pmap_align_superpage(vm_object_t object, vm_ooffset_t offset, + vm_offset_t *addr, vm_size_t size) +{ + vm_offset_t pte1_offset; + + if (size < PTE1_SIZE) + return; + if (object != NULL && (object->flags & OBJ_COLORED) != 0) + offset += ptoa(object->pg_color); + pte1_offset = offset & PTE1_OFFSET; + if (size - ((PTE1_SIZE - pte1_offset) & PTE1_OFFSET) < PTE1_SIZE || + (*addr & PTE1_OFFSET) == pte1_offset) + return; + if ((*addr & PTE1_OFFSET) < pte1_offset) + *addr = pte1_trunc(*addr) + pte1_offset; + else + *addr = pte1_roundup(*addr) + pte1_offset; +} + +void +pmap_activate(struct thread *td) +{ + pmap_t pmap, oldpmap; + u_int cpuid, ttb; + + PDEBUG(9, printf("%s: td = %08x\n", __func__, (uint32_t)td)); + + critical_enter(); + pmap = vmspace_pmap(td->td_proc->p_vmspace); + oldpmap = PCPU_GET(curpmap); + cpuid = PCPU_GET(cpuid); + +#if defined(SMP) + CPU_CLR_ATOMIC(cpuid, &oldpmap->pm_active); + CPU_SET_ATOMIC(cpuid, &pmap->pm_active); +#else + CPU_CLR(cpuid, &oldpmap->pm_active); + CPU_SET(cpuid, &pmap->pm_active); +#endif + + ttb = pmap_ttb_get(pmap); + + /* + * pmap_activate is for the current thread on the current cpu + */ + td->td_pcb->pcb_pagedir = ttb; + cp15_ttbr_set(ttb); + PCPU_SET(curpmap, pmap); + critical_exit(); +} + +/* + * Perform the pmap work for mincore. + */ int pmap_mincore(pmap_t pmap, vm_offset_t addr, vm_paddr_t *locked_pa) { - struct l2_bucket *l2b; - pd_entry_t *pl1pd, l1pd; - pt_entry_t *ptep, pte; + pt1_entry_t *pte1p, pte1; + pt2_entry_t *pte2p, pte2; vm_paddr_t pa; - vm_page_t m; - int val; boolean_t managed; + int val; PMAP_LOCK(pmap); retry: - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(addr)]; - l1pd = *pl1pd; - if ((l1pd & L1_TYPE_MASK) == L1_S_PROTO) { - pa = (l1pd & L1_S_FRAME); + pte1p = pmap_pte1(pmap, addr); + pte1 = pte1_load(pte1p); + if (pte1_is_section(pte1)) { + pa = trunc_page(pte1_pa(pte1) | (addr & PTE1_OFFSET)); + managed = pte1_is_managed(pte1); val = MINCORE_SUPER | MINCORE_INCORE; - if (L1_S_WRITABLE(l1pd)) + if (pte1_is_dirty(pte1)) val |= MINCORE_MODIFIED | MINCORE_MODIFIED_OTHER; - managed = FALSE; - m = PHYS_TO_VM_PAGE(pa); - if (m != NULL && (m->oflags & VPO_UNMANAGED) == 0) - managed = TRUE; - if (managed) { - if (L1_S_REFERENCED(l1pd)) - val |= MINCORE_REFERENCED | - MINCORE_REFERENCED_OTHER; - } - } else { - l2b = pmap_get_l2_bucket(pmap, addr); - if (l2b == NULL) { - val = 0; - goto out; - } - ptep = &l2b->l2b_kva[l2pte_index(addr)]; - pte = *ptep; - if (!l2pte_valid(pte)) { - val = 0; - goto out; - } + if (pte1 & PTE1_A) + val |= MINCORE_REFERENCED | MINCORE_REFERENCED_OTHER; + } else if (pte1_is_link(pte1)) { + pte2p = pmap_pte2(pmap, addr); + pte2 = pte2_load(pte2p); + pmap_pte2_release(pte2p); + pa = pte2_pa(pte2); + managed = pte2_is_managed(pte2); val = MINCORE_INCORE; - if (L2_S_WRITABLE(pte)) + if (pte2_is_dirty(pte2)) val |= MINCORE_MODIFIED | MINCORE_MODIFIED_OTHER; + if (pte2 & PTE2_A) + val |= MINCORE_REFERENCED | MINCORE_REFERENCED_OTHER; + } else { managed = FALSE; - pa = l2pte_pa(pte); - m = PHYS_TO_VM_PAGE(pa); - if (m != NULL && (m->oflags & VPO_UNMANAGED) == 0) - managed = TRUE; - if (managed) { - if (L2_S_REFERENCED(pte)) - val |= MINCORE_REFERENCED | - MINCORE_REFERENCED_OTHER; - } + val = 0; } if ((val & (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER)) != (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER) && managed) { /* Ensure that "PHYS_TO_VM_PAGE(pa)->object" doesn't change. */ if (vm_page_pa_tryrelock(pmap, pa, locked_pa)) goto retry; } else -out: PA_UNLOCK_COND(*locked_pa); PMAP_UNLOCK(pmap); return (val); } void -pmap_sync_icache(pmap_t pmap, vm_offset_t va, vm_size_t sz) +pmap_kenter_device(vm_offset_t va, vm_size_t size, vm_paddr_t pa) { + vm_offset_t sva; + + KASSERT((size & PAGE_MASK) == 0, + ("%s: device mapping not page-sized", __func__)); + + sva = va; + while (size != 0) { + pmap_kenter_prot_attr(va, pa, PTE2_AP_KRW, PTE2_ATTR_DEVICE); + va += PAGE_SIZE; + pa += PAGE_SIZE; + size -= PAGE_SIZE; + } + tlb_flush_range(sva, va - sva); } -/* - * Increase the starting virtual address of the given mapping if a - * different alignment might result in more superpage mappings. - */ void -pmap_align_superpage(vm_object_t object, vm_ooffset_t offset, - vm_offset_t *addr, vm_size_t size) +pmap_kremove_device(vm_offset_t va, vm_size_t size) { - vm_offset_t superpage_offset; + vm_offset_t sva; - if (size < NBPDR) - return; - if (object != NULL && (object->flags & OBJ_COLORED) != 0) - offset += ptoa(object->pg_color); - superpage_offset = offset & PDRMASK; - if (size - ((NBPDR - superpage_offset) & PDRMASK) < NBPDR || - (*addr & PDRMASK) == superpage_offset) - return; - if ((*addr & PDRMASK) < superpage_offset) - *addr = (*addr & ~PDRMASK) + superpage_offset; - else - *addr = ((*addr + PDRMASK) & ~PDRMASK) + superpage_offset; + KASSERT((size & PAGE_MASK) == 0, + ("%s: device mapping not page-sized", __func__)); + + sva = va; + while (size != 0) { + pmap_kremove(va); + va += PAGE_SIZE; + size -= PAGE_SIZE; + } + tlb_flush_range(sva, va - sva); } -/* - * pmap_map_section: - * - * Create a single section mapping. - */ void -pmap_map_section(pmap_t pmap, vm_offset_t va, vm_offset_t pa, vm_prot_t prot, - boolean_t ref) +pmap_set_pcb_pagedir(pmap_t pmap, struct pcb *pcb) { - pd_entry_t *pl1pd, l1pd; - pd_entry_t fl; - KASSERT(((va | pa) & L1_S_OFFSET) == 0, - ("Not a valid section mapping")); + pcb->pcb_pagedir = pmap_ttb_get(pmap); +} - fl = pte_l1_s_cache_mode; - pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; - l1pd = L1_S_PROTO | pa | L1_S_PROT(PTE_USER, prot) | fl | - L1_S_DOM(pmap->pm_domain); +/* + * Clean L1 data cache range by physical address. + * The range must be within a single page. + */ +static void +pmap_dcache_wb_pou(vm_paddr_t pa, vm_size_t size, vm_memattr_t ma) +{ + struct sysmaps *sysmaps; - /* Mark page referenced if this section is a result of a promotion. */ - if (ref == TRUE) - l1pd |= L1_S_REF; -#ifdef SMP - l1pd |= L1_SHARED; -#endif - *pl1pd = l1pd; - PTE_SYNC(pl1pd); + KASSERT(((pa & PAGE_MASK) + size) <= PAGE_SIZE, + ("%s: not on single page", __func__)); + + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (*sysmaps->CMAP3) + panic("%s: CMAP3 busy", __func__); + pte2_store(sysmaps->CMAP3, PTE2_KERN_NG(pa, PTE2_AP_KRW, ma)); + dcache_wb_pou((vm_offset_t)sysmaps->CADDR3 + (pa & PAGE_MASK), size); + pte2_clear(sysmaps->CMAP3); + tlb_flush((vm_offset_t)sysmaps->CADDR3); + sched_unpin(); + mtx_unlock(&sysmaps->lock); } /* - * pmap_link_l2pt: - * - * Link the L2 page table specified by l2pv.pv_pa into the L1 - * page table at the slot for "va". + * Sync instruction cache range which is not mapped yet. */ void -pmap_link_l2pt(vm_offset_t l1pt, vm_offset_t va, struct pv_addr *l2pv) +cache_icache_sync_fresh(vm_offset_t va, vm_paddr_t pa, vm_size_t size) { - pd_entry_t *pde = (pd_entry_t *) l1pt, proto; - u_int slot = va >> L1_S_SHIFT; + uint32_t len, offset; + vm_page_t m; - proto = L1_S_DOM(PMAP_DOMAIN_KERNEL) | L1_C_PROTO; + /* Write back d-cache on given address range. */ + offset = pa & PAGE_MASK; + for ( ; size != 0; size -= len, pa += len, offset = 0) { + len = min(PAGE_SIZE - offset, size); + m = PHYS_TO_VM_PAGE(pa); + KASSERT(m != NULL, ("%s: vm_page_t is null for %#x", + __func__, pa)); + pmap_dcache_wb_pou(pa, len, m->md.pat_mode); + } + /* + * I-cache is VIPT. Only way how to flush all virtual mappings + * on given physical address is to invalidate all i-cache. + */ + icache_inv_all(); +} -#ifdef VERBOSE_INIT_ARM - printf("pmap_link_l2pt: pa=0x%x va=0x%x\n", l2pv->pv_pa, l2pv->pv_va); -#endif +void +pmap_sync_icache(pmap_t pmap, vm_offset_t va, vm_size_t size) +{ - pde[slot + 0] = proto | (l2pv->pv_pa + 0x000); - PTE_SYNC(&pde[slot]); + /* Write back d-cache on given address range. */ + if (va >= VM_MIN_KERNEL_ADDRESS) { + dcache_wb_pou(va, size); + } else { + uint32_t len, offset; + vm_paddr_t pa; + vm_page_t m; - SLIST_INSERT_HEAD(&kernel_pt_list, l2pv, pv_list); - + offset = va & PAGE_MASK; + for ( ; size != 0; size -= len, va += len, offset = 0) { + pa = pmap_extract(pmap, va); /* offset is preserved */ + len = min(PAGE_SIZE - offset, size); + m = PHYS_TO_VM_PAGE(pa); + KASSERT(m != NULL, ("%s: vm_page_t is null for %#x", + __func__, pa)); + pmap_dcache_wb_pou(pa, len, m->md.pat_mode); + } + } + /* + * I-cache is VIPT. Only way how to flush all virtual mappings + * on given physical address is to invalidate all i-cache. + */ + icache_inv_all(); } /* - * pmap_map_entry - * - * Create a single page mapping. + * The implementation of pmap_fault() uses IN_RANGE2() macro which + * depends on the fact that given range size is a power of 2. */ -void -pmap_map_entry(vm_offset_t l1pt, vm_offset_t va, vm_offset_t pa, int prot, - int cache) +CTASSERT(powerof2(NB_IN_PT1)); +CTASSERT(powerof2(PT2MAP_SIZE)); + +#define IN_RANGE2(addr, start, size) \ + ((vm_offset_t)(start) == ((vm_offset_t)(addr) & ~((size) - 1))) + +/* + * Handle access and R/W emulation faults. + */ +int +pmap_fault(pmap_t pmap, vm_offset_t far, uint32_t fsr, int idx, bool usermode) { - pd_entry_t *pde = (pd_entry_t *) l1pt; - pt_entry_t fl; - pt_entry_t *ptep; + pt1_entry_t *pte1p, pte1; + pt2_entry_t *pte2p, pte2; - KASSERT(((va | pa) & PAGE_MASK) == 0, ("ouin")); + if (pmap == NULL) + pmap = kernel_pmap; - fl = l2s_mem_types[cache]; + /* + * In kernel, we should never get abort with FAR which is in range of + * pmap->pm_pt1 or PT2MAP address spaces. If it happens, stop here + * and print out a useful abort message and even get to the debugger + * otherwise it likely ends with never ending loop of aborts. + */ + if (__predict_false(IN_RANGE2(far, pmap->pm_pt1, NB_IN_PT1))) { + /* + * All L1 tables should always be mapped and present. + * However, we check only current one herein. For user mode, + * only permission abort from malicious user is not fatal. + * And alignment abort as it may have higher priority. + */ + if (!usermode || (idx != FAULT_ALIGN && idx != FAULT_PERM_L2)) { + CTR4(KTR_PMAP, "%s: pmap %#x pm_pt1 %#x far %#x", + __func__, pmap, pmap->pm_pt1, far); + panic("%s: pm_pt1 abort", __func__); + } + return (KERN_INVALID_ADDRESS); + } + if (__predict_false(IN_RANGE2(far, PT2MAP, PT2MAP_SIZE))) { + /* + * PT2MAP should be always mapped and present in current + * L1 table. However, only existing L2 tables are mapped + * in PT2MAP. For user mode, only L2 translation abort and + * permission abort from malicious user is not fatal. + * And alignment abort as it may have higher priority. + */ + if (!usermode || (idx != FAULT_ALIGN && + idx != FAULT_TRAN_L2 && idx != FAULT_PERM_L2)) { + CTR4(KTR_PMAP, "%s: pmap %#x PT2MAP %#x far %#x", + __func__, pmap, PT2MAP, far); + panic("%s: PT2MAP abort", __func__); + } + return (KERN_INVALID_ADDRESS); + } - if ((pde[va >> L1_S_SHIFT] & L1_TYPE_MASK) != L1_TYPE_C) - panic("pmap_map_entry: no L2 table for VA 0x%08x", va); + /* + * Accesss bits for page and section. Note that the entry + * is not in TLB yet, so TLB flush is not necessary. + * + * QQQ: This is hardware emulation, we do not call userret() + * for aborts from user mode. + * We do not lock PMAP, so cmpset() is a need. Hopefully, + * no one removes the mapping when we are here. + */ + if (idx == FAULT_ACCESS_L2) { + pte2p = pt2map_entry(far); +pte2_seta: + pte2 = pte2_load(pte2p); + if (pte2_is_valid(pte2)) { + if (!pte2_cmpset(pte2p, pte2, pte2 | PTE2_A)) { + goto pte2_seta; + } + return (KERN_SUCCESS); + } + } + if (idx == FAULT_ACCESS_L1) { + pte1p = pmap_pte1(pmap, far); +pte1_seta: + pte1 = pte1_load(pte1p); + if (pte1_is_section(pte1)) { + if (!pte1_cmpset(pte1p, pte1, pte1 | PTE1_A)) { + goto pte1_seta; + } + return (KERN_SUCCESS); + } + } - ptep = (pt_entry_t *)kernel_pt_lookup(pde[L1_IDX(va)] & L1_C_ADDR_MASK); + /* + * Handle modify bits for page and section. Note that the modify + * bit is emulated by software. So PTEx_RO is software read only + * bit and PTEx_NM flag is real hardware read only bit. + * + * QQQ: This is hardware emulation, we do not call userret() + * for aborts from user mode. + * We do not lock PMAP, so cmpset() is a need. Hopefully, + * no one removes the mapping when we are here. + */ + if ((fsr & FSR_WNR) && (idx == FAULT_PERM_L2)) { + pte2p = pt2map_entry(far); +pte2_setrw: + pte2 = pte2_load(pte2p); + if (pte2_is_valid(pte2) && !(pte2 & PTE2_RO) && + (pte2 & PTE2_NM)) { + if (!pte2_cmpset(pte2p, pte2, pte2 & ~PTE2_NM)) { + goto pte2_setrw; + } + tlb_flush(trunc_page(far)); + return (KERN_SUCCESS); + } + } + if ((fsr & FSR_WNR) && (idx == FAULT_PERM_L1)) { + pte1p = pmap_pte1(pmap, far); +pte1_setrw: + pte1 = pte1_load(pte1p); + if (pte1_is_section(pte1) && !(pte1 & PTE1_RO) && + (pte1 & PTE1_NM)) { + if (!pte1_cmpset(pte1p, pte1, pte1 & ~PTE1_NM)) { + goto pte1_setrw; + } + tlb_flush(pte1_trunc(far)); + return (KERN_SUCCESS); + } + } - if (ptep == NULL) - panic("pmap_map_entry: can't find L2 table for VA 0x%08x", va); + /* + * QQQ: The previous code, mainly fast handling of access and + * modify bits aborts, could be moved to ASM. Now we are + * starting to deal with not fast aborts. + */ - ptep[l2pte_index(va)] = L2_S_PROTO | pa | fl | L2_S_REF; - pmap_set_prot(&ptep[l2pte_index(va)], prot, 0); - PTE_SYNC(&ptep[l2pte_index(va)]); +#ifdef INVARIANTS + /* + * Read an entry in PT2TAB associated with both pmap and far. + * It's safe because PT2TAB is always mapped. + * + * QQQ: We do not lock PMAP, so false positives could happen if + * the mapping is removed concurrently. + */ + pte2 = pt2tab_load(pmap_pt2tab_entry(pmap, far)); + if (pte2_is_valid(pte2)) { + /* + * Now, when we know that L2 page table is allocated, + * we can use PT2MAP to get L2 page table entry. + */ + pte2 = pte2_load(pt2map_entry(far)); + if (pte2_is_valid(pte2)) { + /* + * If L2 page table entry is valid, make sure that + * L1 page table entry is valid too. Note that we + * leave L2 page entries untouched when promoted. + */ + pte1 = pte1_load(pmap_pte1(pmap, far)); + if (!pte1_is_valid(pte1)) { + panic("%s: missing L1 page entry (%p, %#x)", + __func__, pmap, far); + } + } + } +#endif + return (KERN_FAILURE); } +/* !!!! REMOVE !!!! */ +void vector_page_setprot(int p) +{ +} + +#if defined(PMAP_DEBUG) /* - * pmap_map_chunk: - * - * Map a chunk of memory using the most efficient mappings - * possible (section. large page, small page) into the - * provided L1 and L2 tables at the specified virtual address. + * Reusing of KVA used in pmap_zero_page function !!! */ -vm_size_t -pmap_map_chunk(vm_offset_t l1pt, vm_offset_t va, vm_offset_t pa, - vm_size_t size, int prot, int type) +static void +pmap_zero_page_check(vm_page_t m) { - pd_entry_t *pde = (pd_entry_t *) l1pt; - pt_entry_t *ptep, f1, f2s, f2l; - vm_size_t resid; - int i; + uint32_t *p, *end; + struct sysmaps *sysmaps; - resid = (size + (PAGE_SIZE - 1)) & ~(PAGE_SIZE - 1); + sched_pin(); + sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; + mtx_lock(&sysmaps->lock); + if (pte2_load(sysmaps->CMAP2) != 0) + panic("%s: CMAP2 busy", __func__); + pte2_store(sysmaps->CMAP2, PTE2_KERN_NG(VM_PAGE_TO_PHYS(m), PTE2_AP_KRW, + m->md.pat_mode)); + end = (uint32_t*)(sysmaps->CADDR2 + PAGE_SIZE); + for (p = (uint32_t*)sysmaps->CADDR2; p < end; p++) + if (*p != 0) + panic("%s: page %p not zero, va: %p", __func__, m, + sysmaps->CADDR2); + pte2_clear(sysmaps->CMAP2); + tlb_flush((vm_offset_t)sysmaps->CADDR2); + sched_unpin(); + mtx_unlock(&sysmaps->lock); +} - if (l1pt == 0) - panic("pmap_map_chunk: no L1 table provided"); +int +pmap_pid_dump(int pid) +{ + pmap_t pmap; + struct proc *p; + int npte2 = 0; + int i, j, index; -#ifdef VERBOSE_INIT_ARM - printf("pmap_map_chunk: pa=0x%x va=0x%x size=0x%x resid=0x%x " - "prot=0x%x type=%d\n", pa, va, size, resid, prot, type); + sx_slock(&allproc_lock); + FOREACH_PROC_IN_SYSTEM(p) { + if (p->p_pid != pid || p->p_vmspace == NULL) + continue; + index = 0; + pmap = vmspace_pmap(p->p_vmspace); + for (i = 0; i < NPTE1_IN_PT1; i++) { + pt1_entry_t pte1; + pt2_entry_t *pte2p, pte2; + vm_offset_t base, va; + vm_paddr_t pa; + vm_page_t m; + + base = i << PTE1_SHIFT; + pte1 = pte1_load(&pmap->pm_pt1[i]); + + if (pte1_is_section(pte1)) { + /* + * QQQ: Do something here! + */ + } else if (pte1_is_link(pte1)) { + for (j = 0; j < NPTE2_IN_PT2; j++) { + va = base + (j << PAGE_SHIFT); + if (va >= VM_MIN_KERNEL_ADDRESS) { + if (index) { + index = 0; + printf("\n"); + } + sx_sunlock(&allproc_lock); + return (npte2); + } + pte2p = pmap_pte2(pmap, va); + pte2 = pte2_load(pte2p); + pmap_pte2_release(pte2p); + if (!pte2_is_valid(pte2)) + continue; + + pa = pte2_pa(pte2); + m = PHYS_TO_VM_PAGE(pa); + printf("va: 0x%x, pa: 0x%x, h: %d, w:" + " %d, f: 0x%x", va, pa, + m->hold_count, m->wire_count, + m->flags); + npte2++; + index++; + if (index >= 2) { + index = 0; + printf("\n"); + } else { + printf(" "); + } + } + } + } + } + sx_sunlock(&allproc_lock); + return (npte2); +} + #endif - f1 = l1_mem_types[type]; - f2l = l2l_mem_types[type]; - f2s = l2s_mem_types[type]; +#ifdef DDB +static pt2_entry_t * +pmap_pte2_ddb(pmap_t pmap, vm_offset_t va) +{ + pt1_entry_t pte1; + vm_paddr_t pt2pg_pa; - size = resid; + pte1 = pte1_load(pmap_pte1(pmap, va)); + if (!pte1_is_link(pte1)) + return (NULL); - while (resid > 0) { - /* See if we can use a section mapping. */ - if (L1_S_MAPPABLE_P(va, pa, resid)) { -#ifdef VERBOSE_INIT_ARM - printf("S"); + if (pmap_is_current(pmap)) + return (pt2map_entry(va)); + + /* Note that L2 page table size is not equal to PAGE_SIZE. */ + pt2pg_pa = trunc_page(pte1_link_pa(pte1)); + if (pte2_pa(pte2_load(PMAP3)) != pt2pg_pa) { + pte2_store(PMAP3, PTE2_KPT(pt2pg_pa)); +#ifdef SMP + PMAP3cpu = PCPU_GET(cpuid); #endif - pde[va >> L1_S_SHIFT] = L1_S_PROTO | pa | - L1_S_PROT(PTE_KERNEL, prot | VM_PROT_EXECUTE) | - f1 | L1_S_DOM(PMAP_DOMAIN_KERNEL) | L1_S_REF; - PTE_SYNC(&pde[va >> L1_S_SHIFT]); - va += L1_S_SIZE; - pa += L1_S_SIZE; - resid -= L1_S_SIZE; + tlb_flush_local((vm_offset_t)PADDR3); + } +#ifdef SMP + else if (PMAP3cpu != PCPU_GET(cpuid)) { + PMAP3cpu = PCPU_GET(cpuid); + tlb_flush_local((vm_offset_t)PADDR3); + } +#endif + return (PADDR3 + (arm32_btop(va) & (NPTE2_IN_PG - 1))); +} + +static void +dump_pmap(pmap_t pmap) +{ + + printf("pmap %p\n", pmap); + printf(" pm_pt1: %p\n", pmap->pm_pt1); + printf(" pm_pt2tab: %p\n", pmap->pm_pt2tab); + printf(" pm_active: 0x%08lX\n", pmap->pm_active.__bits[0]); +} + +DB_SHOW_COMMAND(pmaps, pmap_list_pmaps) +{ + + pmap_t pmap; + LIST_FOREACH(pmap, &allpmaps, pm_list) { + dump_pmap(pmap); + } +} + +static int +pte2_class(pt2_entry_t pte2) +{ + int cls; + + cls = (pte2 >> 2) & 0x03; + cls |= (pte2 >> 4) & 0x04; + return (cls); +} + +static void +dump_section(pmap_t pmap, uint32_t pte1_idx) +{ +} + +static void +dump_link(pmap_t pmap, uint32_t pte1_idx, boolean_t invalid_ok) +{ + uint32_t i; + vm_offset_t va; + pt2_entry_t *pte2p, pte2; + vm_page_t m; + + va = pte1_idx << PTE1_SHIFT; + pte2p = pmap_pte2_ddb(pmap, va); + for (i = 0; i < NPTE2_IN_PT2; i++, pte2p++, va += PAGE_SIZE) { + pte2 = pte2_load(pte2p); + if (pte2 == 0) continue; + if (!pte2_is_valid(pte2)) { + printf(" 0x%08X: 0x%08X", va, pte2); + if (!invalid_ok) + printf(" - not valid !!!"); + printf("\n"); + continue; } + m = PHYS_TO_VM_PAGE(pte2_pa(pte2)); + printf(" 0x%08X: 0x%08X, TEX%d, s:%d, g:%d, m:%p", va , pte2, + pte2_class(pte2), !!(pte2 & PTE2_S), !(pte2 & PTE2_NG), m); + if (m != NULL) { + printf(" v:%d h:%d w:%d f:0x%04X\n", m->valid, + m->hold_count, m->wire_count, m->flags); + } else { + printf("\n"); + } + } +} - /* - * Ok, we're going to use an L2 table. Make sure - * one is actually in the corresponding L1 slot - * for the current VA. - */ - if ((pde[va >> L1_S_SHIFT] & L1_TYPE_MASK) != L1_TYPE_C) - panic("pmap_map_chunk: no L2 table for VA 0x%08x", va); +static __inline boolean_t +is_pv_chunk_space(vm_offset_t va) +{ - ptep = (pt_entry_t *) kernel_pt_lookup( - pde[L1_IDX(va)] & L1_C_ADDR_MASK); - if (ptep == NULL) - panic("pmap_map_chunk: can't find L2 table for VA" - "0x%08x", va); - /* See if we can use a L2 large page mapping. */ - if (L2_L_MAPPABLE_P(va, pa, resid)) { -#ifdef VERBOSE_INIT_ARM - printf("L"); -#endif - for (i = 0; i < 16; i++) { - ptep[l2pte_index(va) + i] = - L2_L_PROTO | pa | - L2_L_PROT(PTE_KERNEL, prot) | f2l; - PTE_SYNC(&ptep[l2pte_index(va) + i]); - } - va += L2_L_SIZE; - pa += L2_L_SIZE; - resid -= L2_L_SIZE; - continue; + if ((((vm_offset_t)pv_chunkbase) <= va) && + (va < ((vm_offset_t)pv_chunkbase + PAGE_SIZE * pv_maxchunks))) + return (TRUE); + return (FALSE); +} + +DB_SHOW_COMMAND(pmap, pmap_pmap_print) +{ + /* XXX convert args. */ + pmap_t pmap = (pmap_t)addr; + pt1_entry_t pte1; + pt2_entry_t pte2; + vm_offset_t va, eva; + vm_page_t m; + uint32_t i; + boolean_t invalid_ok, dump_link_ok, dump_pv_chunk; + + if (have_addr) { + pmap_t pm; + + LIST_FOREACH(pm, &allpmaps, pm_list) + if (pm == pmap) break; + if (pm == NULL) { + printf("given pmap %p is not in allpmaps list\n", pmap); + return; } + } else + pmap = PCPU_GET(curpmap); - /* Use a small page mapping. */ -#ifdef VERBOSE_INIT_ARM - printf("P"); -#endif - ptep[l2pte_index(va)] = L2_S_PROTO | pa | f2s | L2_S_REF; - pmap_set_prot(&ptep[l2pte_index(va)], prot, 0); - PTE_SYNC(&ptep[l2pte_index(va)]); - va += PAGE_SIZE; - pa += PAGE_SIZE; - resid -= PAGE_SIZE; + eva = (modif[0] == 'u') ? VM_MAXUSER_ADDRESS : 0xFFFFFFFF; + dump_pv_chunk = FALSE; /* XXX evaluate from modif[] */ + + printf("pmap: 0x%08X\n", (uint32_t)pmap); + printf("PT2MAP: 0x%08X\n", (uint32_t)PT2MAP); + printf("pt2tab: 0x%08X\n", (uint32_t)pmap->pm_pt2tab); + + for(i = 0; i < NPTE1_IN_PT1; i++) { + pte1 = pte1_load(&pmap->pm_pt1[i]); + if (pte1 == 0) + continue; + va = i << PTE1_SHIFT; + if (va >= eva) + break; + + if (pte1_is_section(pte1)) { + printf("0x%08X: Section 0x%08X, s:%d g:%d\n", va, pte1, + !!(pte1 & PTE1_S), !(pte1 & PTE1_NG)); + dump_section(pmap, i); + } else if (pte1_is_link(pte1)) { + dump_link_ok = TRUE; + invalid_ok = FALSE; + pte2 = pte2_load(pmap_pt2tab_entry(pmap, va)); + m = PHYS_TO_VM_PAGE(pte1_link_pa(pte1)); + printf("0x%08X: Link 0x%08X, pt2tab: 0x%08X m: %p", + va, pte1, pte2, m); + if (is_pv_chunk_space(va)) { + printf(" - pv_chunk space"); + if (dump_pv_chunk) + invalid_ok = TRUE; + else + dump_link_ok = FALSE; + } + else if (m != NULL) + printf(" w:%d w2:%u", m->wire_count, + pt2_wirecount_get(m, pte1_index(va))); + if (pte2 == 0) + printf(" !!! pt2tab entry is ZERO"); + else if (pte2_pa(pte1) != pte2_pa(pte2)) + printf(" !!! pt2tab entry is DIFFERENT - m: %p", + PHYS_TO_VM_PAGE(pte2_pa(pte2))); + printf("\n"); + if (dump_link_ok) + dump_link(pmap, i, invalid_ok); + } else + printf("0x%08X: Invalid entry 0x%08X\n", va, pte1); } -#ifdef VERBOSE_INIT_ARM - printf("\n"); -#endif - return (size); +} +static void +dump_pt2tab(pmap_t pmap) +{ + uint32_t i; + pt2_entry_t pte2; + vm_offset_t va; + vm_paddr_t pa; + vm_page_t m; + + printf("PT2TAB:\n"); + for (i = 0; i < PT2TAB_ENTRIES; i++) { + pte2 = pte2_load(&pmap->pm_pt2tab[i]); + if (!pte2_is_valid(pte2)) + continue; + va = i << PT2TAB_SHIFT; + pa = pte2_pa(pte2); + m = PHYS_TO_VM_PAGE(pa); + printf(" 0x%08X: 0x%08X, TEX%d, s:%d, m:%p", va, pte2, + pte2_class(pte2), !!(pte2 & PTE2_S), m); + if (m != NULL) + printf(" , h: %d, w: %d, f: 0x%04X pidx: %lld", + m->hold_count, m->wire_count, m->flags, m->pindex); + printf("\n"); + } } -void -pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma) +DB_SHOW_COMMAND(pmap_pt2tab, pmap_pt2tab_print) { - /* - * Remember the memattr in a field that gets used to set the appropriate - * bits in the PTEs as mappings are established. - */ - m->md.pv_memattr = ma; + /* XXX convert args. */ + pmap_t pmap = (pmap_t)addr; + pt1_entry_t pte1; + pt2_entry_t pte2; + vm_offset_t va; + uint32_t i, start; - /* - * It appears that this function can only be called before any mappings - * for the page are established on ARM. If this ever changes, this code - * will need to walk the pv_list and make each of the existing mappings - * uncacheable, being careful to sync caches and PTEs (and maybe - * invalidate TLB?) for any current mapping it modifies. - */ - if (TAILQ_FIRST(&m->md.pv_list) != NULL) - panic("Can't change memattr on page with existing mappings"); + if (have_addr) { + printf("supported only on current pmap\n"); + return; + } + + pmap = PCPU_GET(curpmap); + printf("curpmap: 0x%08X\n", (uint32_t)pmap); + printf("PT2MAP: 0x%08X\n", (uint32_t)PT2MAP); + printf("pt2tab: 0x%08X\n", (uint32_t)pmap->pm_pt2tab); + + start = pte1_index((vm_offset_t)PT2MAP); + for (i = start; i < (start + NPT2_IN_PT2TAB); i++) { + pte1 = pte1_load(&pmap->pm_pt1[i]); + if (pte1 == 0) + continue; + va = i << PTE1_SHIFT; + if (pte1_is_section(pte1)) { + printf("0x%08X: Section 0x%08X, s:%d\n", va, pte1, + !!(pte1 & PTE1_S)); + dump_section(pmap, i); + } else if (pte1_is_link(pte1)) { + pte2 = pte2_load(pmap_pt2tab_entry(pmap, va)); + printf("0x%08X: Link 0x%08X, pt2tab: 0x%08X\n", va, + pte1, pte2); + if (pte2 == 0) + printf(" !!! pt2tab entry is ZERO\n"); + } else + printf("0x%08X: Invalid entry 0x%08X\n", va, pte1); + } + dump_pt2tab(pmap); } +#endif Index: user/ngie/make_check/sys/arm/arm/pmap.c =================================================================== --- user/ngie/make_check/sys/arm/arm/pmap.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/pmap.c (revision 295078) @@ -1,4874 +1,4874 @@ /* From: $NetBSD: pmap.c,v 1.148 2004/04/03 04:35:48 bsh Exp $ */ /*- * Copyright 2004 Olivier Houchard. * Copyright 2003 Wasabi Systems, Inc. * All rights reserved. * * Written by Steve C. Woodford for Wasabi Systems, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed for the NetBSD Project by * Wasabi Systems, Inc. * 4. The name of Wasabi Systems, Inc. may not be used to endorse * or promote products derived from this software without specific prior * written permission. * * THIS SOFTWARE IS PROVIDED BY WASABI SYSTEMS, INC. ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASABI SYSTEMS, INC * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 2002-2003 Wasabi Systems, Inc. * Copyright (c) 2001 Richard Earnshaw * Copyright (c) 2001-2002 Christopher Gilbert * All rights reserved. * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the company nor the name of the author may be used to * endorse or promote products derived from this software without specific * prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ /*- * Copyright (c) 1999 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Charles M. Hannum. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1994-1998 Mark Brinicombe. * Copyright (c) 1994 Brini. * All rights reserved. * * This code is derived from software written for Brini by Mark Brinicombe * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Mark Brinicombe. * 4. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * * RiscBSD kernel project * * pmap.c * * Machine dependant vm stuff * * Created : 20/09/94 */ /* * Special compilation symbols * PMAP_DEBUG - Build in pmap_debug_level code * * Note that pmap_mapdev() and pmap_unmapdev() are implemented in arm/devmap.c */ /* Include header files */ #include "opt_vm.h" #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef PMAP_DEBUG #define PDEBUG(_lev_,_stat_) \ if (pmap_debug_level >= (_lev_)) \ ((_stat_)) #define dprintf printf int pmap_debug_level = 0; #define PMAP_INLINE #else /* PMAP_DEBUG */ #define PDEBUG(_lev_,_stat_) /* Nothing */ #define dprintf(x, arg...) #define PMAP_INLINE __inline #endif /* PMAP_DEBUG */ extern struct pv_addr systempage; extern int last_fault_code; /* * Internal function prototypes */ static void pmap_free_pv_entry (pv_entry_t); static pv_entry_t pmap_get_pv_entry(void); static int pmap_enter_locked(pmap_t, vm_offset_t, vm_page_t, vm_prot_t, u_int); static vm_paddr_t pmap_extract_locked(pmap_t pmap, vm_offset_t va); static void pmap_fix_cache(struct vm_page *, pmap_t, vm_offset_t); static void pmap_alloc_l1(pmap_t); static void pmap_free_l1(pmap_t); static int pmap_clearbit(struct vm_page *, u_int); static struct l2_bucket *pmap_get_l2_bucket(pmap_t, vm_offset_t); static struct l2_bucket *pmap_alloc_l2_bucket(pmap_t, vm_offset_t); static void pmap_free_l2_bucket(pmap_t, struct l2_bucket *, u_int); static vm_offset_t kernel_pt_lookup(vm_paddr_t); static MALLOC_DEFINE(M_VMPMAP, "pmap", "PMAP L1"); vm_offset_t virtual_avail; /* VA of first avail page (after kernel bss) */ vm_offset_t virtual_end; /* VA of last avail page (end of kernel AS) */ vm_offset_t pmap_curmaxkvaddr; vm_paddr_t kernel_l1pa; vm_offset_t kernel_vm_end = 0; vm_offset_t vm_max_kernel_address; struct pmap kernel_pmap_store; static pt_entry_t *csrc_pte, *cdst_pte; static vm_offset_t csrcp, cdstp, qmap_addr; static struct mtx cmtx, qmap_mtx; static void pmap_init_l1(struct l1_ttable *, pd_entry_t *); /* * These routines are called when the CPU type is identified to set up * the PTE prototypes, cache modes, etc. * * The variables are always here, just in case LKMs need to reference * them (though, they shouldn't). */ pt_entry_t pte_l1_s_cache_mode; pt_entry_t pte_l1_s_cache_mode_pt; pt_entry_t pte_l1_s_cache_mask; pt_entry_t pte_l2_l_cache_mode; pt_entry_t pte_l2_l_cache_mode_pt; pt_entry_t pte_l2_l_cache_mask; pt_entry_t pte_l2_s_cache_mode; pt_entry_t pte_l2_s_cache_mode_pt; pt_entry_t pte_l2_s_cache_mask; pt_entry_t pte_l2_s_prot_u; pt_entry_t pte_l2_s_prot_w; pt_entry_t pte_l2_s_prot_mask; pt_entry_t pte_l1_s_proto; pt_entry_t pte_l1_c_proto; pt_entry_t pte_l2_s_proto; void (*pmap_copy_page_func)(vm_paddr_t, vm_paddr_t); void (*pmap_copy_page_offs_func)(vm_paddr_t a_phys, vm_offset_t a_offs, vm_paddr_t b_phys, vm_offset_t b_offs, int cnt); void (*pmap_zero_page_func)(vm_paddr_t, int, int); struct msgbuf *msgbufp = 0; /* * Crashdump maps. */ static caddr_t crashdumpmap; extern void bcopy_page(vm_offset_t, vm_offset_t); extern void bzero_page(vm_offset_t); extern vm_offset_t alloc_firstaddr; char *_tmppt; /* * Metadata for L1 translation tables. */ struct l1_ttable { /* Entry on the L1 Table list */ SLIST_ENTRY(l1_ttable) l1_link; /* Entry on the L1 Least Recently Used list */ TAILQ_ENTRY(l1_ttable) l1_lru; /* Track how many domains are allocated from this L1 */ volatile u_int l1_domain_use_count; /* * A free-list of domain numbers for this L1. * We avoid using ffs() and a bitmap to track domains since ffs() * is slow on ARM. */ u_int8_t l1_domain_first; u_int8_t l1_domain_free[PMAP_DOMAINS]; /* Physical address of this L1 page table */ vm_paddr_t l1_physaddr; /* KVA of this L1 page table */ pd_entry_t *l1_kva; }; /* * Convert a virtual address into its L1 table index. That is, the * index used to locate the L2 descriptor table pointer in an L1 table. * This is basically used to index l1->l1_kva[]. * * Each L2 descriptor table represents 1MB of VA space. */ #define L1_IDX(va) (((vm_offset_t)(va)) >> L1_S_SHIFT) /* * L1 Page Tables are tracked using a Least Recently Used list. * - New L1s are allocated from the HEAD. * - Freed L1s are added to the TAIl. * - Recently accessed L1s (where an 'access' is some change to one of * the userland pmaps which owns this L1) are moved to the TAIL. */ static TAILQ_HEAD(, l1_ttable) l1_lru_list; /* * A list of all L1 tables */ static SLIST_HEAD(, l1_ttable) l1_list; static struct mtx l1_lru_lock; /* * The l2_dtable tracks L2_BUCKET_SIZE worth of L1 slots. * * This is normally 16MB worth L2 page descriptors for any given pmap. * Reference counts are maintained for L2 descriptors so they can be * freed when empty. */ struct l2_dtable { /* The number of L2 page descriptors allocated to this l2_dtable */ u_int l2_occupancy; /* List of L2 page descriptors */ struct l2_bucket { pt_entry_t *l2b_kva; /* KVA of L2 Descriptor Table */ vm_paddr_t l2b_phys; /* Physical address of same */ u_short l2b_l1idx; /* This L2 table's L1 index */ u_short l2b_occupancy; /* How many active descriptors */ } l2_bucket[L2_BUCKET_SIZE]; }; /* pmap_kenter_internal flags */ #define KENTER_CACHE 0x1 #define KENTER_USER 0x2 /* * Given an L1 table index, calculate the corresponding l2_dtable index * and bucket index within the l2_dtable. */ #define L2_IDX(l1idx) (((l1idx) >> L2_BUCKET_LOG2) & \ (L2_SIZE - 1)) #define L2_BUCKET(l1idx) ((l1idx) & (L2_BUCKET_SIZE - 1)) /* * Given a virtual address, this macro returns the * virtual address required to drop into the next L2 bucket. */ #define L2_NEXT_BUCKET(va) (((va) & L1_S_FRAME) + L1_S_SIZE) /* * We try to map the page tables write-through, if possible. However, not * all CPUs have a write-through cache mode, so on those we have to sync * the cache when we frob page tables. * * We try to evaluate this at compile time, if possible. However, it's * not always possible to do that, hence this run-time var. */ int pmap_needs_pte_sync; /* * Macro to determine if a mapping might be resident in the * instruction cache and/or TLB */ #define PV_BEEN_EXECD(f) (((f) & (PVF_REF | PVF_EXEC)) == (PVF_REF | PVF_EXEC)) /* * Macro to determine if a mapping might be resident in the * data cache and/or TLB */ #define PV_BEEN_REFD(f) (((f) & PVF_REF) != 0) #ifndef PMAP_SHPGPERPROC #define PMAP_SHPGPERPROC 200 #endif -#define pmap_is_current(pm) ((pm) == pmap_kernel() || \ +#define pmap_is_current(pm) ((pm) == kernel_pmap || \ curproc->p_vmspace->vm_map.pmap == (pm)) static uma_zone_t pvzone = NULL; uma_zone_t l2zone; static uma_zone_t l2table_zone; static vm_offset_t pmap_kernel_l2dtable_kva; static vm_offset_t pmap_kernel_l2ptp_kva; static vm_paddr_t pmap_kernel_l2ptp_phys; static int pv_entry_count=0, pv_entry_max=0, pv_entry_high_water=0; static struct rwlock pvh_global_lock; void pmap_copy_page_offs_generic(vm_paddr_t a_phys, vm_offset_t a_offs, vm_paddr_t b_phys, vm_offset_t b_offs, int cnt); #if ARM_MMU_XSCALE == 1 void pmap_copy_page_offs_xscale(vm_paddr_t a_phys, vm_offset_t a_offs, vm_paddr_t b_phys, vm_offset_t b_offs, int cnt); #endif /* * This list exists for the benefit of pmap_map_chunk(). It keeps track * of the kernel L2 tables during bootstrap, so that pmap_map_chunk() can * find them as necessary. * * Note that the data on this list MUST remain valid after initarm() returns, * as pmap_bootstrap() uses it to contruct L2 table metadata. */ SLIST_HEAD(, pv_addr) kernel_pt_list = SLIST_HEAD_INITIALIZER(kernel_pt_list); static void pmap_init_l1(struct l1_ttable *l1, pd_entry_t *l1pt) { int i; l1->l1_kva = l1pt; l1->l1_domain_use_count = 0; l1->l1_domain_first = 0; for (i = 0; i < PMAP_DOMAINS; i++) l1->l1_domain_free[i] = i + 1; /* * Copy the kernel's L1 entries to each new L1. */ - if (l1pt != pmap_kernel()->pm_l1->l1_kva) - memcpy(l1pt, pmap_kernel()->pm_l1->l1_kva, L1_TABLE_SIZE); + if (l1pt != kernel_pmap->pm_l1->l1_kva) + memcpy(l1pt, kernel_pmap->pm_l1->l1_kva, L1_TABLE_SIZE); - if ((l1->l1_physaddr = pmap_extract(pmap_kernel(), (vm_offset_t)l1pt)) == 0) + if ((l1->l1_physaddr = pmap_extract(kernel_pmap, (vm_offset_t)l1pt)) == 0) panic("pmap_init_l1: can't get PA of L1 at %p", l1pt); SLIST_INSERT_HEAD(&l1_list, l1, l1_link); TAILQ_INSERT_TAIL(&l1_lru_list, l1, l1_lru); } static vm_offset_t kernel_pt_lookup(vm_paddr_t pa) { struct pv_addr *pv; SLIST_FOREACH(pv, &kernel_pt_list, pv_list) { if (pv->pv_pa == pa) return (pv->pv_va); } return (0); } #if ARM_MMU_GENERIC != 0 void pmap_pte_init_generic(void) { pte_l1_s_cache_mode = L1_S_B|L1_S_C; pte_l1_s_cache_mask = L1_S_CACHE_MASK_generic; pte_l2_l_cache_mode = L2_B|L2_C; pte_l2_l_cache_mask = L2_L_CACHE_MASK_generic; pte_l2_s_cache_mode = L2_B|L2_C; pte_l2_s_cache_mask = L2_S_CACHE_MASK_generic; /* * If we have a write-through cache, set B and C. If * we have a write-back cache, then we assume setting * only C will make those pages write-through. */ if (cpufuncs.cf_dcache_wb_range == (void *) cpufunc_nullop) { pte_l1_s_cache_mode_pt = L1_S_B|L1_S_C; pte_l2_l_cache_mode_pt = L2_B|L2_C; pte_l2_s_cache_mode_pt = L2_B|L2_C; } else { pte_l1_s_cache_mode_pt = L1_S_C; pte_l2_l_cache_mode_pt = L2_C; pte_l2_s_cache_mode_pt = L2_C; } pte_l2_s_prot_u = L2_S_PROT_U_generic; pte_l2_s_prot_w = L2_S_PROT_W_generic; pte_l2_s_prot_mask = L2_S_PROT_MASK_generic; pte_l1_s_proto = L1_S_PROTO_generic; pte_l1_c_proto = L1_C_PROTO_generic; pte_l2_s_proto = L2_S_PROTO_generic; pmap_copy_page_func = pmap_copy_page_generic; pmap_copy_page_offs_func = pmap_copy_page_offs_generic; pmap_zero_page_func = pmap_zero_page_generic; } #endif /* ARM_MMU_GENERIC != 0 */ #if ARM_MMU_XSCALE == 1 #if (ARM_NMMUS > 1) || defined (CPU_XSCALE_CORE3) static u_int xscale_use_minidata; #endif void pmap_pte_init_xscale(void) { uint32_t auxctl; int write_through = 0; pte_l1_s_cache_mode = L1_S_B|L1_S_C|L1_S_XSCALE_P; pte_l1_s_cache_mask = L1_S_CACHE_MASK_xscale; pte_l2_l_cache_mode = L2_B|L2_C; pte_l2_l_cache_mask = L2_L_CACHE_MASK_xscale; pte_l2_s_cache_mode = L2_B|L2_C; pte_l2_s_cache_mask = L2_S_CACHE_MASK_xscale; pte_l1_s_cache_mode_pt = L1_S_C; pte_l2_l_cache_mode_pt = L2_C; pte_l2_s_cache_mode_pt = L2_C; #ifdef XSCALE_CACHE_READ_WRITE_ALLOCATE /* * The XScale core has an enhanced mode where writes that * miss the cache cause a cache line to be allocated. This * is significantly faster than the traditional, write-through * behavior of this case. */ pte_l1_s_cache_mode |= L1_S_XSCALE_TEX(TEX_XSCALE_X); pte_l2_l_cache_mode |= L2_XSCALE_L_TEX(TEX_XSCALE_X); pte_l2_s_cache_mode |= L2_XSCALE_T_TEX(TEX_XSCALE_X); #endif /* XSCALE_CACHE_READ_WRITE_ALLOCATE */ #ifdef XSCALE_CACHE_WRITE_THROUGH /* * Some versions of the XScale core have various bugs in * their cache units, the work-around for which is to run * the cache in write-through mode. Unfortunately, this * has a major (negative) impact on performance. So, we * go ahead and run fast-and-loose, in the hopes that we * don't line up the planets in a way that will trip the * bugs. * * However, we give you the option to be slow-but-correct. */ write_through = 1; #elif defined(XSCALE_CACHE_WRITE_BACK) /* force write back cache mode */ write_through = 0; #elif defined(CPU_XSCALE_PXA2X0) /* * Intel PXA2[15]0 processors are known to have a bug in * write-back cache on revision 4 and earlier (stepping * A[01] and B[012]). Fixed for C0 and later. */ { uint32_t id, type; id = cpufunc_id(); type = id & ~(CPU_ID_XSCALE_COREREV_MASK|CPU_ID_REVISION_MASK); if (type == CPU_ID_PXA250 || type == CPU_ID_PXA210) { if ((id & CPU_ID_REVISION_MASK) < 5) { /* write through for stepping A0-1 and B0-2 */ write_through = 1; } } } #endif /* XSCALE_CACHE_WRITE_THROUGH */ if (write_through) { pte_l1_s_cache_mode = L1_S_C; pte_l2_l_cache_mode = L2_C; pte_l2_s_cache_mode = L2_C; } #if (ARM_NMMUS > 1) xscale_use_minidata = 1; #endif pte_l2_s_prot_u = L2_S_PROT_U_xscale; pte_l2_s_prot_w = L2_S_PROT_W_xscale; pte_l2_s_prot_mask = L2_S_PROT_MASK_xscale; pte_l1_s_proto = L1_S_PROTO_xscale; pte_l1_c_proto = L1_C_PROTO_xscale; pte_l2_s_proto = L2_S_PROTO_xscale; #ifdef CPU_XSCALE_CORE3 pmap_copy_page_func = pmap_copy_page_generic; pmap_copy_page_offs_func = pmap_copy_page_offs_generic; pmap_zero_page_func = pmap_zero_page_generic; xscale_use_minidata = 0; /* Make sure it is L2-cachable */ pte_l1_s_cache_mode |= L1_S_XSCALE_TEX(TEX_XSCALE_T); pte_l1_s_cache_mode_pt = pte_l1_s_cache_mode &~ L1_S_XSCALE_P; pte_l2_l_cache_mode |= L2_XSCALE_L_TEX(TEX_XSCALE_T) ; pte_l2_l_cache_mode_pt = pte_l1_s_cache_mode; pte_l2_s_cache_mode |= L2_XSCALE_T_TEX(TEX_XSCALE_T); pte_l2_s_cache_mode_pt = pte_l2_s_cache_mode; #else pmap_copy_page_func = pmap_copy_page_xscale; pmap_copy_page_offs_func = pmap_copy_page_offs_xscale; pmap_zero_page_func = pmap_zero_page_xscale; #endif /* * Disable ECC protection of page table access, for now. */ __asm __volatile("mrc p15, 0, %0, c1, c0, 1" : "=r" (auxctl)); auxctl &= ~XSCALE_AUXCTL_P; __asm __volatile("mcr p15, 0, %0, c1, c0, 1" : : "r" (auxctl)); } /* * xscale_setup_minidata: * * Set up the mini-data cache clean area. We require the * caller to allocate the right amount of physically and * virtually contiguous space. */ extern vm_offset_t xscale_minidata_clean_addr; extern vm_size_t xscale_minidata_clean_size; /* already initialized */ void xscale_setup_minidata(vm_offset_t l1pt, vm_offset_t va, vm_paddr_t pa) { pd_entry_t *pde = (pd_entry_t *) l1pt; pt_entry_t *pte; vm_size_t size; uint32_t auxctl; xscale_minidata_clean_addr = va; /* Round it to page size. */ size = (xscale_minidata_clean_size + L2_S_OFFSET) & L2_S_FRAME; for (; size != 0; va += L2_S_SIZE, pa += L2_S_SIZE, size -= L2_S_SIZE) { pte = (pt_entry_t *) kernel_pt_lookup( pde[L1_IDX(va)] & L1_C_ADDR_MASK); if (pte == NULL) panic("xscale_setup_minidata: can't find L2 table for " "VA 0x%08x", (u_int32_t) va); pte[l2pte_index(va)] = L2_S_PROTO | pa | L2_S_PROT(PTE_KERNEL, VM_PROT_READ) | L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X); } /* * Configure the mini-data cache for write-back with * read/write-allocate. * * NOTE: In order to reconfigure the mini-data cache, we must * make sure it contains no valid data! In order to do that, * we must issue a global data cache invalidate command! * * WE ASSUME WE ARE RUNNING UN-CACHED WHEN THIS ROUTINE IS CALLED! * THIS IS VERY IMPORTANT! */ /* Invalidate data and mini-data. */ __asm __volatile("mcr p15, 0, %0, c7, c6, 0" : : "r" (0)); __asm __volatile("mrc p15, 0, %0, c1, c0, 1" : "=r" (auxctl)); auxctl = (auxctl & ~XSCALE_AUXCTL_MD_MASK) | XSCALE_AUXCTL_MD_WB_RWA; __asm __volatile("mcr p15, 0, %0, c1, c0, 1" : : "r" (auxctl)); } #endif /* * Allocate an L1 translation table for the specified pmap. * This is called at pmap creation time. */ static void pmap_alloc_l1(pmap_t pm) { struct l1_ttable *l1; u_int8_t domain; /* * Remove the L1 at the head of the LRU list */ mtx_lock(&l1_lru_lock); l1 = TAILQ_FIRST(&l1_lru_list); TAILQ_REMOVE(&l1_lru_list, l1, l1_lru); /* * Pick the first available domain number, and update * the link to the next number. */ domain = l1->l1_domain_first; l1->l1_domain_first = l1->l1_domain_free[domain]; /* * If there are still free domain numbers in this L1, * put it back on the TAIL of the LRU list. */ if (++l1->l1_domain_use_count < PMAP_DOMAINS) TAILQ_INSERT_TAIL(&l1_lru_list, l1, l1_lru); mtx_unlock(&l1_lru_lock); /* * Fix up the relevant bits in the pmap structure */ pm->pm_l1 = l1; pm->pm_domain = domain + 1; } /* * Free an L1 translation table. * This is called at pmap destruction time. */ static void pmap_free_l1(pmap_t pm) { struct l1_ttable *l1 = pm->pm_l1; mtx_lock(&l1_lru_lock); /* * If this L1 is currently on the LRU list, remove it. */ if (l1->l1_domain_use_count < PMAP_DOMAINS) TAILQ_REMOVE(&l1_lru_list, l1, l1_lru); /* * Free up the domain number which was allocated to the pmap */ l1->l1_domain_free[pm->pm_domain - 1] = l1->l1_domain_first; l1->l1_domain_first = pm->pm_domain - 1; l1->l1_domain_use_count--; /* * The L1 now must have at least 1 free domain, so add * it back to the LRU list. If the use count is zero, * put it at the head of the list, otherwise it goes * to the tail. */ if (l1->l1_domain_use_count == 0) { TAILQ_INSERT_HEAD(&l1_lru_list, l1, l1_lru); } else TAILQ_INSERT_TAIL(&l1_lru_list, l1, l1_lru); mtx_unlock(&l1_lru_lock); } /* * Returns a pointer to the L2 bucket associated with the specified pmap * and VA, or NULL if no L2 bucket exists for the address. */ static PMAP_INLINE struct l2_bucket * pmap_get_l2_bucket(pmap_t pm, vm_offset_t va) { struct l2_dtable *l2; struct l2_bucket *l2b; u_short l1idx; l1idx = L1_IDX(va); if ((l2 = pm->pm_l2[L2_IDX(l1idx)]) == NULL || (l2b = &l2->l2_bucket[L2_BUCKET(l1idx)])->l2b_kva == NULL) return (NULL); return (l2b); } /* * Returns a pointer to the L2 bucket associated with the specified pmap * and VA. * * If no L2 bucket exists, perform the necessary allocations to put an L2 * bucket/page table in place. * * Note that if a new L2 bucket/page was allocated, the caller *must* * increment the bucket occupancy counter appropriately *before* * releasing the pmap's lock to ensure no other thread or cpu deallocates * the bucket/page in the meantime. */ static struct l2_bucket * pmap_alloc_l2_bucket(pmap_t pm, vm_offset_t va) { struct l2_dtable *l2; struct l2_bucket *l2b; u_short l1idx; l1idx = L1_IDX(va); PMAP_ASSERT_LOCKED(pm); rw_assert(&pvh_global_lock, RA_WLOCKED); if ((l2 = pm->pm_l2[L2_IDX(l1idx)]) == NULL) { /* * No mapping at this address, as there is * no entry in the L1 table. * Need to allocate a new l2_dtable. */ PMAP_UNLOCK(pm); rw_wunlock(&pvh_global_lock); if ((l2 = uma_zalloc(l2table_zone, M_NOWAIT)) == NULL) { rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); return (NULL); } rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); if (pm->pm_l2[L2_IDX(l1idx)] != NULL) { /* * Someone already allocated the l2_dtable while * we were doing the same. */ uma_zfree(l2table_zone, l2); l2 = pm->pm_l2[L2_IDX(l1idx)]; } else { bzero(l2, sizeof(*l2)); /* * Link it into the parent pmap */ pm->pm_l2[L2_IDX(l1idx)] = l2; } } l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; /* * Fetch pointer to the L2 page table associated with the address. */ if (l2b->l2b_kva == NULL) { pt_entry_t *ptep; /* * No L2 page table has been allocated. Chances are, this * is because we just allocated the l2_dtable, above. */ l2->l2_occupancy++; PMAP_UNLOCK(pm); rw_wunlock(&pvh_global_lock); ptep = uma_zalloc(l2zone, M_NOWAIT); rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); if (l2b->l2b_kva != 0) { /* We lost the race. */ l2->l2_occupancy--; uma_zfree(l2zone, ptep); return (l2b); } l2b->l2b_phys = vtophys(ptep); if (ptep == NULL) { /* * Oops, no more L2 page tables available at this * time. We may need to deallocate the l2_dtable * if we allocated a new one above. */ l2->l2_occupancy--; if (l2->l2_occupancy == 0) { pm->pm_l2[L2_IDX(l1idx)] = NULL; uma_zfree(l2table_zone, l2); } return (NULL); } l2b->l2b_kva = ptep; l2b->l2b_l1idx = l1idx; } return (l2b); } static PMAP_INLINE void #ifndef PMAP_INCLUDE_PTE_SYNC pmap_free_l2_ptp(pt_entry_t *l2) #else pmap_free_l2_ptp(boolean_t need_sync, pt_entry_t *l2) #endif { #ifdef PMAP_INCLUDE_PTE_SYNC /* * Note: With a write-back cache, we may need to sync this * L2 table before re-using it. * This is because it may have belonged to a non-current * pmap, in which case the cache syncs would have been * skipped when the pages were being unmapped. If the * L2 table were then to be immediately re-allocated to * the *current* pmap, it may well contain stale mappings * which have not yet been cleared by a cache write-back * and so would still be visible to the mmu. */ if (need_sync) PTE_SYNC_RANGE(l2, L2_TABLE_SIZE_REAL / sizeof(pt_entry_t)); #endif uma_zfree(l2zone, l2); } /* * One or more mappings in the specified L2 descriptor table have just been * invalidated. * * Garbage collect the metadata and descriptor table itself if necessary. * * The pmap lock must be acquired when this is called (not necessary * for the kernel pmap). */ static void pmap_free_l2_bucket(pmap_t pm, struct l2_bucket *l2b, u_int count) { struct l2_dtable *l2; pd_entry_t *pl1pd, l1pd; pt_entry_t *ptep; u_short l1idx; /* * Update the bucket's reference count according to how many * PTEs the caller has just invalidated. */ l2b->l2b_occupancy -= count; /* * Note: * * Level 2 page tables allocated to the kernel pmap are never freed * as that would require checking all Level 1 page tables and * removing any references to the Level 2 page table. See also the * comment elsewhere about never freeing bootstrap L2 descriptors. * * We make do with just invalidating the mapping in the L2 table. * * This isn't really a big deal in practice and, in fact, leads * to a performance win over time as we don't need to continually * alloc/free. */ - if (l2b->l2b_occupancy > 0 || pm == pmap_kernel()) + if (l2b->l2b_occupancy > 0 || pm == kernel_pmap) return; /* * There are no more valid mappings in this level 2 page table. * Go ahead and NULL-out the pointer in the bucket, then * free the page table. */ l1idx = l2b->l2b_l1idx; ptep = l2b->l2b_kva; l2b->l2b_kva = NULL; pl1pd = &pm->pm_l1->l1_kva[l1idx]; /* * If the L1 slot matches the pmap's domain * number, then invalidate it. */ l1pd = *pl1pd & (L1_TYPE_MASK | L1_C_DOM_MASK); if (l1pd == (L1_C_DOM(pm->pm_domain) | L1_TYPE_C)) { *pl1pd = 0; PTE_SYNC(pl1pd); } /* * Release the L2 descriptor table back to the pool cache. */ #ifndef PMAP_INCLUDE_PTE_SYNC pmap_free_l2_ptp(ptep); #else pmap_free_l2_ptp(!pmap_is_current(pm), ptep); #endif /* * Update the reference count in the associated l2_dtable */ l2 = pm->pm_l2[L2_IDX(l1idx)]; if (--l2->l2_occupancy > 0) return; /* * There are no more valid mappings in any of the Level 1 * slots managed by this l2_dtable. Go ahead and NULL-out * the pointer in the parent pmap and free the l2_dtable. */ pm->pm_l2[L2_IDX(l1idx)] = NULL; uma_zfree(l2table_zone, l2); } /* * Pool cache constructors for L2 descriptor tables, metadata and pmap * structures. */ static int pmap_l2ptp_ctor(void *mem, int size, void *arg, int flags) { #ifndef PMAP_INCLUDE_PTE_SYNC struct l2_bucket *l2b; pt_entry_t *ptep, pte; vm_offset_t va = (vm_offset_t)mem & ~PAGE_MASK; /* * The mappings for these page tables were initially made using * pmap_kenter() by the pool subsystem. Therefore, the cache- * mode will not be right for page table mappings. To avoid * polluting the pmap_kenter() code with a special case for * page tables, we simply fix up the cache-mode here if it's not * correct. */ - l2b = pmap_get_l2_bucket(pmap_kernel(), va); + l2b = pmap_get_l2_bucket(kernel_pmap, va); ptep = &l2b->l2b_kva[l2pte_index(va)]; pte = *ptep; if ((pte & L2_S_CACHE_MASK) != pte_l2_s_cache_mode_pt) { /* * Page tables must have the cache-mode set to * Write-Thru. */ *ptep = (pte & ~L2_S_CACHE_MASK) | pte_l2_s_cache_mode_pt; PTE_SYNC(ptep); cpu_tlb_flushD_SE(va); cpu_cpwait(); } #endif memset(mem, 0, L2_TABLE_SIZE_REAL); PTE_SYNC_RANGE(mem, L2_TABLE_SIZE_REAL / sizeof(pt_entry_t)); return (0); } /* * A bunch of routines to conditionally flush the caches/TLB depending * on whether the specified pmap actually needs to be flushed at any * given time. */ static PMAP_INLINE void pmap_tlb_flushID_SE(pmap_t pm, vm_offset_t va) { if (pmap_is_current(pm)) cpu_tlb_flushID_SE(va); } static PMAP_INLINE void pmap_tlb_flushD_SE(pmap_t pm, vm_offset_t va) { if (pmap_is_current(pm)) cpu_tlb_flushD_SE(va); } static PMAP_INLINE void pmap_tlb_flushID(pmap_t pm) { if (pmap_is_current(pm)) cpu_tlb_flushID(); } static PMAP_INLINE void pmap_tlb_flushD(pmap_t pm) { if (pmap_is_current(pm)) cpu_tlb_flushD(); } static int pmap_has_valid_mapping(pmap_t pm, vm_offset_t va) { pd_entry_t *pde; pt_entry_t *ptep; if (pmap_get_pde_pte(pm, va, &pde, &ptep) && ptep && ((*ptep & L2_TYPE_MASK) != L2_TYPE_INV)) return (1); return (0); } static PMAP_INLINE void pmap_idcache_wbinv_range(pmap_t pm, vm_offset_t va, vm_size_t len) { vm_size_t rest; CTR4(KTR_PMAP, "pmap_dcache_wbinv_range: pmap %p is_kernel %d va 0x%08x" - " len 0x%x ", pm, pm == pmap_kernel(), va, len); + " len 0x%x ", pm, pm == kernel_pmap, va, len); - if (pmap_is_current(pm) || pm == pmap_kernel()) { + if (pmap_is_current(pm) || pm == kernel_pmap) { rest = MIN(PAGE_SIZE - (va & PAGE_MASK), len); while (len > 0) { if (pmap_has_valid_mapping(pm, va)) { cpu_idcache_wbinv_range(va, rest); cpu_l2cache_wbinv_range(va, rest); } len -= rest; va += rest; rest = MIN(PAGE_SIZE, len); } } } static PMAP_INLINE void pmap_dcache_wb_range(pmap_t pm, vm_offset_t va, vm_size_t len, boolean_t do_inv, boolean_t rd_only) { vm_size_t rest; CTR4(KTR_PMAP, "pmap_dcache_wb_range: pmap %p is_kernel %d va 0x%08x " - "len 0x%x ", pm, pm == pmap_kernel(), va, len); + "len 0x%x ", pm, pm == kernel_pmap, va, len); CTR2(KTR_PMAP, " do_inv %d rd_only %d", do_inv, rd_only); if (pmap_is_current(pm)) { rest = MIN(PAGE_SIZE - (va & PAGE_MASK), len); while (len > 0) { if (pmap_has_valid_mapping(pm, va)) { if (do_inv && rd_only) { cpu_dcache_inv_range(va, rest); cpu_l2cache_inv_range(va, rest); } else if (do_inv) { cpu_dcache_wbinv_range(va, rest); cpu_l2cache_wbinv_range(va, rest); } else if (!rd_only) { cpu_dcache_wb_range(va, rest); cpu_l2cache_wb_range(va, rest); } } len -= rest; va += rest; rest = MIN(PAGE_SIZE, len); } } } static PMAP_INLINE void pmap_idcache_wbinv_all(pmap_t pm) { if (pmap_is_current(pm)) { cpu_idcache_wbinv_all(); cpu_l2cache_wbinv_all(); } } #ifdef notyet static PMAP_INLINE void pmap_dcache_wbinv_all(pmap_t pm) { if (pmap_is_current(pm)) { cpu_dcache_wbinv_all(); cpu_l2cache_wbinv_all(); } } #endif /* * PTE_SYNC_CURRENT: * * Make sure the pte is written out to RAM. * We need to do this for one of two cases: * - We're dealing with the kernel pmap * - There is no pmap active in the cache/tlb. * - The specified pmap is 'active' in the cache/tlb. */ #ifdef PMAP_INCLUDE_PTE_SYNC #define PTE_SYNC_CURRENT(pm, ptep) \ do { \ if (PMAP_NEEDS_PTE_SYNC && \ pmap_is_current(pm)) \ PTE_SYNC(ptep); \ } while (/*CONSTCOND*/0) #else #define PTE_SYNC_CURRENT(pm, ptep) /* nothing */ #endif /* * cacheable == -1 means we must make the entry uncacheable, 1 means * cacheable; */ static __inline void pmap_set_cache_entry(pv_entry_t pv, pmap_t pm, vm_offset_t va, int cacheable) { struct l2_bucket *l2b; pt_entry_t *ptep, pte; l2b = pmap_get_l2_bucket(pv->pv_pmap, pv->pv_va); ptep = &l2b->l2b_kva[l2pte_index(pv->pv_va)]; if (cacheable == 1) { pte = (*ptep & ~L2_S_CACHE_MASK) | pte_l2_s_cache_mode; if (l2pte_valid(pte)) { if (PV_BEEN_EXECD(pv->pv_flags)) { pmap_tlb_flushID_SE(pv->pv_pmap, pv->pv_va); } else if (PV_BEEN_REFD(pv->pv_flags)) { pmap_tlb_flushD_SE(pv->pv_pmap, pv->pv_va); } } } else { pte = *ptep &~ L2_S_CACHE_MASK; if ((va != pv->pv_va || pm != pv->pv_pmap) && l2pte_valid(pte)) { if (PV_BEEN_EXECD(pv->pv_flags)) { pmap_idcache_wbinv_range(pv->pv_pmap, pv->pv_va, PAGE_SIZE); pmap_tlb_flushID_SE(pv->pv_pmap, pv->pv_va); } else if (PV_BEEN_REFD(pv->pv_flags)) { pmap_dcache_wb_range(pv->pv_pmap, pv->pv_va, PAGE_SIZE, TRUE, (pv->pv_flags & PVF_WRITE) == 0); pmap_tlb_flushD_SE(pv->pv_pmap, pv->pv_va); } } } *ptep = pte; PTE_SYNC_CURRENT(pv->pv_pmap, ptep); } static void pmap_fix_cache(struct vm_page *pg, pmap_t pm, vm_offset_t va) { int pmwc = 0; int writable = 0, kwritable = 0, uwritable = 0; int entries = 0, kentries = 0, uentries = 0; struct pv_entry *pv; rw_assert(&pvh_global_lock, RA_WLOCKED); /* the cache gets written back/invalidated on context switch. * therefore, if a user page shares an entry in the same page or * with the kernel map and at least one is writable, then the * cache entry must be set write-through. */ TAILQ_FOREACH(pv, &pg->md.pv_list, pv_list) { /* generate a count of the pv_entry uses */ if (pv->pv_flags & PVF_WRITE) { - if (pv->pv_pmap == pmap_kernel()) + if (pv->pv_pmap == kernel_pmap) kwritable++; else if (pv->pv_pmap == pm) uwritable++; writable++; } - if (pv->pv_pmap == pmap_kernel()) + if (pv->pv_pmap == kernel_pmap) kentries++; else { if (pv->pv_pmap == pm) uentries++; entries++; } } /* * check if the user duplicate mapping has * been removed. */ - if ((pm != pmap_kernel()) && (((uentries > 1) && uwritable) || + if ((pm != kernel_pmap) && (((uentries > 1) && uwritable) || (uwritable > 1))) pmwc = 1; TAILQ_FOREACH(pv, &pg->md.pv_list, pv_list) { /* check for user uncachable conditions - order is important */ - if (pm != pmap_kernel() && - (pv->pv_pmap == pm || pv->pv_pmap == pmap_kernel())) { + if (pm != kernel_pmap && + (pv->pv_pmap == pm || pv->pv_pmap == kernel_pmap)) { if ((uentries > 1 && uwritable) || uwritable > 1) { /* user duplicate mapping */ - if (pv->pv_pmap != pmap_kernel()) + if (pv->pv_pmap != kernel_pmap) pv->pv_flags |= PVF_MWC; if (!(pv->pv_flags & PVF_NC)) { pv->pv_flags |= PVF_NC; pmap_set_cache_entry(pv, pm, va, -1); } continue; } else /* no longer a duplicate user */ pv->pv_flags &= ~PVF_MWC; } /* * check for kernel uncachable conditions * kernel writable or kernel readable with writable user entry */ if ((kwritable && (entries || kentries > 1)) || (kwritable > 1) || ((kwritable != writable) && kentries && - (pv->pv_pmap == pmap_kernel() || + (pv->pv_pmap == kernel_pmap || (pv->pv_flags & PVF_WRITE) || (pv->pv_flags & PVF_MWC)))) { if (!(pv->pv_flags & PVF_NC)) { pv->pv_flags |= PVF_NC; pmap_set_cache_entry(pv, pm, va, -1); } continue; } /* kernel and user are cachable */ - if ((pm == pmap_kernel()) && !(pv->pv_flags & PVF_MWC) && + if ((pm == kernel_pmap) && !(pv->pv_flags & PVF_MWC) && (pv->pv_flags & PVF_NC)) { pv->pv_flags &= ~PVF_NC; if (pg->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) pmap_set_cache_entry(pv, pm, va, 1); continue; } /* user is no longer sharable and writable */ - if (pm != pmap_kernel() && - (pv->pv_pmap == pm || pv->pv_pmap == pmap_kernel()) && + if (pm != kernel_pmap && + (pv->pv_pmap == pm || pv->pv_pmap == kernel_pmap) && !pmwc && (pv->pv_flags & PVF_NC)) { pv->pv_flags &= ~(PVF_NC | PVF_MWC); if (pg->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) pmap_set_cache_entry(pv, pm, va, 1); } } if ((kwritable == 0) && (writable == 0)) { pg->md.pvh_attrs &= ~PVF_MOD; vm_page_aflag_clear(pg, PGA_WRITEABLE); return; } } /* * Modify pte bits for all ptes corresponding to the given physical address. * We use `maskbits' rather than `clearbits' because we're always passing * constants and the latter would require an extra inversion at run-time. */ static int pmap_clearbit(struct vm_page *pg, u_int maskbits) { struct l2_bucket *l2b; struct pv_entry *pv; pt_entry_t *ptep, npte, opte; pmap_t pm; vm_offset_t va; u_int oflags; int count = 0; rw_wlock(&pvh_global_lock); if (maskbits & PVF_WRITE) maskbits |= PVF_MOD; /* * Clear saved attributes (modify, reference) */ pg->md.pvh_attrs &= ~(maskbits & (PVF_MOD | PVF_REF)); if (TAILQ_EMPTY(&pg->md.pv_list)) { rw_wunlock(&pvh_global_lock); return (0); } /* * Loop over all current mappings setting/clearing as appropos */ TAILQ_FOREACH(pv, &pg->md.pv_list, pv_list) { va = pv->pv_va; pm = pv->pv_pmap; oflags = pv->pv_flags; if (!(oflags & maskbits)) { if ((maskbits & PVF_WRITE) && (pv->pv_flags & PVF_NC)) { if (pg->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) { PMAP_LOCK(pm); l2b = pmap_get_l2_bucket(pm, va); ptep = &l2b->l2b_kva[l2pte_index(va)]; *ptep |= pte_l2_s_cache_mode; PTE_SYNC(ptep); PMAP_UNLOCK(pm); } pv->pv_flags &= ~(PVF_NC | PVF_MWC); } continue; } pv->pv_flags &= ~maskbits; PMAP_LOCK(pm); l2b = pmap_get_l2_bucket(pm, va); ptep = &l2b->l2b_kva[l2pte_index(va)]; npte = opte = *ptep; if (maskbits & (PVF_WRITE|PVF_MOD)) { if ((pv->pv_flags & PVF_NC)) { /* * Entry is not cacheable: * * Don't turn caching on again if this is a * modified emulation. This would be * inconsitent with the settings created by * pmap_fix_cache(). Otherwise, it's safe * to re-enable cacheing. * * There's no need to call pmap_fix_cache() * here: all pages are losing their write * permission. */ if (maskbits & PVF_WRITE) { if (pg->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) npte |= pte_l2_s_cache_mode; pv->pv_flags &= ~(PVF_NC | PVF_MWC); } } else if (opte & L2_S_PROT_W) { vm_page_dirty(pg); /* * Entry is writable/cacheable: check if pmap * is current if it is flush it, otherwise it * won't be in the cache */ if (PV_BEEN_EXECD(oflags)) pmap_idcache_wbinv_range(pm, pv->pv_va, PAGE_SIZE); else if (PV_BEEN_REFD(oflags)) pmap_dcache_wb_range(pm, pv->pv_va, PAGE_SIZE, (maskbits & PVF_REF) ? TRUE : FALSE, FALSE); } /* make the pte read only */ npte &= ~L2_S_PROT_W; } if (maskbits & PVF_REF) { if ((pv->pv_flags & PVF_NC) == 0 && (maskbits & (PVF_WRITE|PVF_MOD)) == 0) { /* * Check npte here; we may have already * done the wbinv above, and the validity * of the PTE is the same for opte and * npte. */ if (npte & L2_S_PROT_W) { if (PV_BEEN_EXECD(oflags)) pmap_idcache_wbinv_range(pm, pv->pv_va, PAGE_SIZE); else if (PV_BEEN_REFD(oflags)) pmap_dcache_wb_range(pm, pv->pv_va, PAGE_SIZE, TRUE, FALSE); } else if ((npte & L2_TYPE_MASK) != L2_TYPE_INV) { /* XXXJRT need idcache_inv_range */ if (PV_BEEN_EXECD(oflags)) pmap_idcache_wbinv_range(pm, pv->pv_va, PAGE_SIZE); else if (PV_BEEN_REFD(oflags)) pmap_dcache_wb_range(pm, pv->pv_va, PAGE_SIZE, TRUE, TRUE); } } /* * Make the PTE invalid so that we will take a * page fault the next time the mapping is * referenced. */ npte &= ~L2_TYPE_MASK; npte |= L2_TYPE_INV; } if (npte != opte) { count++; *ptep = npte; PTE_SYNC(ptep); /* Flush the TLB entry if a current pmap. */ if (PV_BEEN_EXECD(oflags)) pmap_tlb_flushID_SE(pm, pv->pv_va); else if (PV_BEEN_REFD(oflags)) pmap_tlb_flushD_SE(pm, pv->pv_va); } PMAP_UNLOCK(pm); } if (maskbits & PVF_WRITE) vm_page_aflag_clear(pg, PGA_WRITEABLE); rw_wunlock(&pvh_global_lock); return (count); } /* * main pv_entry manipulation functions: * pmap_enter_pv: enter a mapping onto a vm_page list * pmap_remove_pv: remove a mappiing from a vm_page list * * NOTE: pmap_enter_pv expects to lock the pvh itself * pmap_remove_pv expects the caller to lock the pvh before calling */ /* * pmap_enter_pv: enter a mapping onto a vm_page's PV list * * => caller should hold the proper lock on pvh_global_lock * => caller should have pmap locked * => we will (someday) gain the lock on the vm_page's PV list * => caller should adjust ptp's wire_count before calling * => caller should not adjust pmap's wire_count */ static void pmap_enter_pv(struct vm_page *pg, struct pv_entry *pve, pmap_t pm, vm_offset_t va, u_int flags) { rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_ASSERT_LOCKED(pm); if (pg->md.pv_kva != 0) { pve->pv_pmap = kernel_pmap; pve->pv_va = pg->md.pv_kva; pve->pv_flags = PVF_WRITE | PVF_UNMAN; if (pm != kernel_pmap) PMAP_LOCK(kernel_pmap); TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list); TAILQ_INSERT_HEAD(&kernel_pmap->pm_pvlist, pve, pv_plist); if (pm != kernel_pmap) PMAP_UNLOCK(kernel_pmap); pg->md.pv_kva = 0; if ((pve = pmap_get_pv_entry()) == NULL) panic("pmap_kenter_pv: no pv entries"); } pve->pv_pmap = pm; pve->pv_va = va; pve->pv_flags = flags; TAILQ_INSERT_HEAD(&pg->md.pv_list, pve, pv_list); TAILQ_INSERT_HEAD(&pm->pm_pvlist, pve, pv_plist); pg->md.pvh_attrs |= flags & (PVF_REF | PVF_MOD); if (pve->pv_flags & PVF_WIRED) ++pm->pm_stats.wired_count; vm_page_aflag_set(pg, PGA_REFERENCED); } /* * * pmap_find_pv: Find a pv entry * * => caller should hold lock on vm_page */ static PMAP_INLINE struct pv_entry * pmap_find_pv(struct vm_page *pg, pmap_t pm, vm_offset_t va) { struct pv_entry *pv; rw_assert(&pvh_global_lock, RA_WLOCKED); TAILQ_FOREACH(pv, &pg->md.pv_list, pv_list) if (pm == pv->pv_pmap && va == pv->pv_va) break; return (pv); } /* * vector_page_setprot: * * Manipulate the protection of the vector page. */ void vector_page_setprot(int prot) { struct l2_bucket *l2b; pt_entry_t *ptep; - l2b = pmap_get_l2_bucket(pmap_kernel(), vector_page); + l2b = pmap_get_l2_bucket(kernel_pmap, vector_page); ptep = &l2b->l2b_kva[l2pte_index(vector_page)]; *ptep = (*ptep & ~L1_S_PROT_MASK) | L2_S_PROT(PTE_KERNEL, prot); PTE_SYNC(ptep); cpu_tlb_flushD_SE(vector_page); cpu_cpwait(); } /* * pmap_remove_pv: try to remove a mapping from a pv_list * * => caller should hold proper lock on pmap_main_lock * => pmap should be locked * => caller should hold lock on vm_page [so that attrs can be adjusted] * => caller should adjust ptp's wire_count and free PTP if needed * => caller should NOT adjust pmap's wire_count * => we return the removed pve */ static void pmap_nuke_pv(struct vm_page *pg, pmap_t pm, struct pv_entry *pve) { struct pv_entry *pv; rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_ASSERT_LOCKED(pm); TAILQ_REMOVE(&pg->md.pv_list, pve, pv_list); TAILQ_REMOVE(&pm->pm_pvlist, pve, pv_plist); if (pve->pv_flags & PVF_WIRED) --pm->pm_stats.wired_count; if (pg->md.pvh_attrs & PVF_MOD) vm_page_dirty(pg); if (TAILQ_FIRST(&pg->md.pv_list) == NULL) pg->md.pvh_attrs &= ~PVF_REF; else vm_page_aflag_set(pg, PGA_REFERENCED); - if ((pve->pv_flags & PVF_NC) && ((pm == pmap_kernel()) || + if ((pve->pv_flags & PVF_NC) && ((pm == kernel_pmap) || (pve->pv_flags & PVF_WRITE) || !(pve->pv_flags & PVF_MWC))) pmap_fix_cache(pg, pm, 0); else if (pve->pv_flags & PVF_WRITE) { TAILQ_FOREACH(pve, &pg->md.pv_list, pv_list) if (pve->pv_flags & PVF_WRITE) break; if (!pve) { pg->md.pvh_attrs &= ~PVF_MOD; vm_page_aflag_clear(pg, PGA_WRITEABLE); } } pv = TAILQ_FIRST(&pg->md.pv_list); if (pv != NULL && (pv->pv_flags & PVF_UNMAN) && TAILQ_NEXT(pv, pv_list) == NULL) { pm = kernel_pmap; pg->md.pv_kva = pv->pv_va; /* a recursive pmap_nuke_pv */ TAILQ_REMOVE(&pg->md.pv_list, pv, pv_list); TAILQ_REMOVE(&pm->pm_pvlist, pv, pv_plist); if (pv->pv_flags & PVF_WIRED) --pm->pm_stats.wired_count; pg->md.pvh_attrs &= ~PVF_REF; pg->md.pvh_attrs &= ~PVF_MOD; vm_page_aflag_clear(pg, PGA_WRITEABLE); pmap_free_pv_entry(pv); } } static struct pv_entry * pmap_remove_pv(struct vm_page *pg, pmap_t pm, vm_offset_t va) { struct pv_entry *pve; rw_assert(&pvh_global_lock, RA_WLOCKED); pve = TAILQ_FIRST(&pg->md.pv_list); while (pve) { if (pve->pv_pmap == pm && pve->pv_va == va) { /* match? */ pmap_nuke_pv(pg, pm, pve); break; } pve = TAILQ_NEXT(pve, pv_list); } if (pve == NULL && pg->md.pv_kva == va) pg->md.pv_kva = 0; return(pve); /* return removed pve */ } /* * * pmap_modify_pv: Update pv flags * * => caller should hold lock on vm_page [so that attrs can be adjusted] * => caller should NOT adjust pmap's wire_count * => we return the old flags * * Modify a physical-virtual mapping in the pv table */ static u_int pmap_modify_pv(struct vm_page *pg, pmap_t pm, vm_offset_t va, u_int clr_mask, u_int set_mask) { struct pv_entry *npv; u_int flags, oflags; PMAP_ASSERT_LOCKED(pm); rw_assert(&pvh_global_lock, RA_WLOCKED); if ((npv = pmap_find_pv(pg, pm, va)) == NULL) return (0); /* * There is at least one VA mapping this page. */ if (clr_mask & (PVF_REF | PVF_MOD)) pg->md.pvh_attrs |= set_mask & (PVF_REF | PVF_MOD); oflags = npv->pv_flags; npv->pv_flags = flags = (oflags & ~clr_mask) | set_mask; if ((flags ^ oflags) & PVF_WIRED) { if (flags & PVF_WIRED) ++pm->pm_stats.wired_count; else --pm->pm_stats.wired_count; } if ((flags ^ oflags) & PVF_WRITE) pmap_fix_cache(pg, pm, 0); return (oflags); } /* Function to set the debug level of the pmap code */ #ifdef PMAP_DEBUG void pmap_debug(int level) { pmap_debug_level = level; dprintf("pmap_debug: level=%d\n", pmap_debug_level); } #endif /* PMAP_DEBUG */ void pmap_pinit0(struct pmap *pmap) { PDEBUG(1, printf("pmap_pinit0: pmap = %08x\n", (u_int32_t) pmap)); bcopy(kernel_pmap, pmap, sizeof(*pmap)); bzero(&pmap->pm_mtx, sizeof(pmap->pm_mtx)); PMAP_LOCK_INIT(pmap); } /* * Initialize a vm_page's machine-dependent fields. */ void pmap_page_init(vm_page_t m) { TAILQ_INIT(&m->md.pv_list); m->md.pv_memattr = VM_MEMATTR_DEFAULT; } /* * Initialize the pmap module. * Called by vm_init, to initialize any structures that the pmap * system needs to map virtual memory. */ void pmap_init(void) { int shpgperproc = PMAP_SHPGPERPROC; l2zone = uma_zcreate("L2 Table", L2_TABLE_SIZE_REAL, pmap_l2ptp_ctor, NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); l2table_zone = uma_zcreate("L2 Table", sizeof(struct l2_dtable), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); /* * Initialize the PV entry allocator. */ pvzone = uma_zcreate("PV ENTRY", sizeof (struct pv_entry), NULL, NULL, NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); TUNABLE_INT_FETCH("vm.pmap.shpgperproc", &shpgperproc); pv_entry_max = shpgperproc * maxproc + vm_cnt.v_page_count; uma_zone_reserve_kva(pvzone, pv_entry_max); pv_entry_high_water = 9 * (pv_entry_max / 10); /* * Now it is safe to enable pv_table recording. */ PDEBUG(1, printf("pmap_init: done!\n")); } int pmap_fault_fixup(pmap_t pm, vm_offset_t va, vm_prot_t ftype, int user) { struct l2_dtable *l2; struct l2_bucket *l2b; pd_entry_t *pl1pd, l1pd; pt_entry_t *ptep, pte; vm_paddr_t pa; u_int l1idx; int rv = 0; l1idx = L1_IDX(va); rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); /* * If there is no l2_dtable for this address, then the process * has no business accessing it. * * Note: This will catch userland processes trying to access * kernel addresses. */ l2 = pm->pm_l2[L2_IDX(l1idx)]; if (l2 == NULL) goto out; /* * Likewise if there is no L2 descriptor table */ l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; if (l2b->l2b_kva == NULL) goto out; /* * Check the PTE itself. */ ptep = &l2b->l2b_kva[l2pte_index(va)]; pte = *ptep; if (pte == 0) goto out; /* * Catch a userland access to the vector page mapped at 0x0 */ if (user && (pte & L2_S_PROT_U) == 0) goto out; if (va == vector_page) goto out; pa = l2pte_pa(pte); if ((ftype & VM_PROT_WRITE) && (pte & L2_S_PROT_W) == 0) { /* * This looks like a good candidate for "page modified" * emulation... */ struct pv_entry *pv; struct vm_page *pg; /* Extract the physical address of the page */ if ((pg = PHYS_TO_VM_PAGE(pa)) == NULL) { goto out; } /* Get the current flags for this page. */ pv = pmap_find_pv(pg, pm, va); if (pv == NULL) { goto out; } /* * Do the flags say this page is writable? If not then it * is a genuine write fault. If yes then the write fault is * our fault as we did not reflect the write access in the * PTE. Now we know a write has occurred we can correct this * and also set the modified bit */ if ((pv->pv_flags & PVF_WRITE) == 0) { goto out; } pg->md.pvh_attrs |= PVF_REF | PVF_MOD; vm_page_dirty(pg); pv->pv_flags |= PVF_REF | PVF_MOD; /* * Re-enable write permissions for the page. No need to call * pmap_fix_cache(), since this is just a * modified-emulation fault, and the PVF_WRITE bit isn't * changing. We've already set the cacheable bits based on * the assumption that we can write to this page. */ *ptep = (pte & ~L2_TYPE_MASK) | L2_S_PROTO | L2_S_PROT_W; PTE_SYNC(ptep); rv = 1; } else if ((pte & L2_TYPE_MASK) == L2_TYPE_INV) { /* * This looks like a good candidate for "page referenced" * emulation. */ struct pv_entry *pv; struct vm_page *pg; /* Extract the physical address of the page */ if ((pg = PHYS_TO_VM_PAGE(pa)) == NULL) goto out; /* Get the current flags for this page. */ pv = pmap_find_pv(pg, pm, va); if (pv == NULL) goto out; pg->md.pvh_attrs |= PVF_REF; pv->pv_flags |= PVF_REF; *ptep = (pte & ~L2_TYPE_MASK) | L2_S_PROTO; PTE_SYNC(ptep); rv = 1; } /* * We know there is a valid mapping here, so simply * fix up the L1 if necessary. */ pl1pd = &pm->pm_l1->l1_kva[l1idx]; l1pd = l2b->l2b_phys | L1_C_DOM(pm->pm_domain) | L1_C_PROTO; if (*pl1pd != l1pd) { *pl1pd = l1pd; PTE_SYNC(pl1pd); rv = 1; } #ifdef DEBUG /* * If 'rv == 0' at this point, it generally indicates that there is a * stale TLB entry for the faulting address. This happens when two or * more processes are sharing an L1. Since we don't flush the TLB on * a context switch between such processes, we can take domain faults * for mappings which exist at the same VA in both processes. EVEN IF * WE'VE RECENTLY FIXED UP THE CORRESPONDING L1 in pmap_enter(), for * example. * * This is extremely likely to happen if pmap_enter() updated the L1 * entry for a recently entered mapping. In this case, the TLB is * flushed for the new mapping, but there may still be TLB entries for * other mappings belonging to other processes in the 1MB range * covered by the L1 entry. * * Since 'rv == 0', we know that the L1 already contains the correct * value, so the fault must be due to a stale TLB entry. * * Since we always need to flush the TLB anyway in the case where we * fixed up the L1, or frobbed the L2 PTE, we effectively deal with * stale TLB entries dynamically. * * However, the above condition can ONLY happen if the current L1 is * being shared. If it happens when the L1 is unshared, it indicates * that other parts of the pmap are not doing their job WRT managing * the TLB. */ if (rv == 0 && pm->pm_l1->l1_domain_use_count == 1) { printf("fixup: pm %p, va 0x%lx, ftype %d - nothing to do!\n", pm, (u_long)va, ftype); printf("fixup: l2 %p, l2b %p, ptep %p, pl1pd %p\n", l2, l2b, ptep, pl1pd); printf("fixup: pte 0x%x, l1pd 0x%x, last code 0x%x\n", pte, l1pd, last_fault_code); #ifdef DDB Debugger(); #endif } #endif cpu_tlb_flushID_SE(va); cpu_cpwait(); rv = 1; out: rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pm); return (rv); } void pmap_postinit(void) { struct l2_bucket *l2b; struct l1_ttable *l1; pd_entry_t *pl1pt; pt_entry_t *ptep, pte; vm_offset_t va, eva; u_int loop, needed; needed = (maxproc / PMAP_DOMAINS) + ((maxproc % PMAP_DOMAINS) ? 1 : 0); needed -= 1; l1 = malloc(sizeof(*l1) * needed, M_VMPMAP, M_WAITOK); for (loop = 0; loop < needed; loop++, l1++) { /* Allocate a L1 page table */ va = (vm_offset_t)contigmalloc(L1_TABLE_SIZE, M_VMPMAP, 0, 0x0, 0xffffffff, L1_TABLE_SIZE, 0); if (va == 0) panic("Cannot allocate L1 KVM"); eva = va + L1_TABLE_SIZE; pl1pt = (pd_entry_t *)va; while (va < eva) { - l2b = pmap_get_l2_bucket(pmap_kernel(), va); + l2b = pmap_get_l2_bucket(kernel_pmap, va); ptep = &l2b->l2b_kva[l2pte_index(va)]; pte = *ptep; pte = (pte & ~L2_S_CACHE_MASK) | pte_l2_s_cache_mode_pt; *ptep = pte; PTE_SYNC(ptep); cpu_tlb_flushD_SE(va); va += PAGE_SIZE; } pmap_init_l1(l1, pl1pt); } #ifdef DEBUG printf("pmap_postinit: Allocated %d static L1 descriptor tables\n", needed); #endif } /* * This is used to stuff certain critical values into the PCB where they * can be accessed quickly from cpu_switch() et al. */ void pmap_set_pcb_pagedir(pmap_t pm, struct pcb *pcb) { struct l2_bucket *l2b; pcb->pcb_pagedir = pm->pm_l1->l1_physaddr; pcb->pcb_dacr = (DOMAIN_CLIENT << (PMAP_DOMAIN_KERNEL * 2)) | (DOMAIN_CLIENT << (pm->pm_domain * 2)); if (vector_page < KERNBASE) { pcb->pcb_pl1vec = &pm->pm_l1->l1_kva[L1_IDX(vector_page)]; l2b = pmap_get_l2_bucket(pm, vector_page); pcb->pcb_l1vec = l2b->l2b_phys | L1_C_PROTO | L1_C_DOM(pm->pm_domain) | L1_C_DOM(PMAP_DOMAIN_KERNEL); } else pcb->pcb_pl1vec = NULL; } void pmap_activate(struct thread *td) { pmap_t pm; struct pcb *pcb; pm = vmspace_pmap(td->td_proc->p_vmspace); pcb = td->td_pcb; critical_enter(); pmap_set_pcb_pagedir(pm, pcb); if (td == curthread) { u_int cur_dacr, cur_ttb; __asm __volatile("mrc p15, 0, %0, c2, c0, 0" : "=r"(cur_ttb)); __asm __volatile("mrc p15, 0, %0, c3, c0, 0" : "=r"(cur_dacr)); cur_ttb &= ~(L1_TABLE_SIZE - 1); if (cur_ttb == (u_int)pcb->pcb_pagedir && cur_dacr == pcb->pcb_dacr) { /* * No need to switch address spaces. */ critical_exit(); return; } /* * We MUST, I repeat, MUST fix up the L1 entry corresponding * to 'vector_page' in the incoming L1 table before switching * to it otherwise subsequent interrupts/exceptions (including * domain faults!) will jump into hyperspace. */ if (pcb->pcb_pl1vec) { *pcb->pcb_pl1vec = pcb->pcb_l1vec; /* * Don't need to PTE_SYNC() at this point since * cpu_setttb() is about to flush both the cache * and the TLB. */ } cpu_domains(pcb->pcb_dacr); cpu_setttb(pcb->pcb_pagedir); } critical_exit(); } static int pmap_set_pt_cache_mode(pd_entry_t *kl1, vm_offset_t va) { pd_entry_t *pdep, pde; pt_entry_t *ptep, pte; vm_offset_t pa; int rv = 0; /* * Make sure the descriptor itself has the correct cache mode */ pdep = &kl1[L1_IDX(va)]; pde = *pdep; if (l1pte_section_p(pde)) { if ((pde & L1_S_CACHE_MASK) != pte_l1_s_cache_mode_pt) { *pdep = (pde & ~L1_S_CACHE_MASK) | pte_l1_s_cache_mode_pt; PTE_SYNC(pdep); cpu_dcache_wbinv_range((vm_offset_t)pdep, sizeof(*pdep)); cpu_l2cache_wbinv_range((vm_offset_t)pdep, sizeof(*pdep)); rv = 1; } } else { pa = (vm_paddr_t)(pde & L1_C_ADDR_MASK); ptep = (pt_entry_t *)kernel_pt_lookup(pa); if (ptep == NULL) panic("pmap_bootstrap: No L2 for L2 @ va %p\n", ptep); ptep = &ptep[l2pte_index(va)]; pte = *ptep; if ((pte & L2_S_CACHE_MASK) != pte_l2_s_cache_mode_pt) { *ptep = (pte & ~L2_S_CACHE_MASK) | pte_l2_s_cache_mode_pt; PTE_SYNC(ptep); cpu_dcache_wbinv_range((vm_offset_t)ptep, sizeof(*ptep)); cpu_l2cache_wbinv_range((vm_offset_t)ptep, sizeof(*ptep)); rv = 1; } } return (rv); } static void pmap_alloc_specials(vm_offset_t *availp, int pages, vm_offset_t *vap, pt_entry_t **ptep) { vm_offset_t va = *availp; struct l2_bucket *l2b; if (ptep) { - l2b = pmap_get_l2_bucket(pmap_kernel(), va); + l2b = pmap_get_l2_bucket(kernel_pmap, va); if (l2b == NULL) panic("pmap_alloc_specials: no l2b for 0x%x", va); *ptep = &l2b->l2b_kva[l2pte_index(va)]; } *vap = va; *availp = va + (PAGE_SIZE * pages); } /* * Bootstrap the system enough to run with virtual memory. * * On the arm this is called after mapping has already been enabled * and just syncs the pmap module with what has already been done. * [We can't call it easily with mapping off since the kernel is not * mapped with PA == VA, hence we would have to relocate every address * from the linked base (virtual) address "KERNBASE" to the actual * (physical) address starting relative to 0] */ #define PMAP_STATIC_L2_SIZE 16 void pmap_bootstrap(vm_offset_t firstaddr, struct pv_addr *l1pt) { static struct l1_ttable static_l1; static struct l2_dtable static_l2[PMAP_STATIC_L2_SIZE]; struct l1_ttable *l1 = &static_l1; struct l2_dtable *l2; struct l2_bucket *l2b; pd_entry_t pde; pd_entry_t *kernel_l1pt = (pd_entry_t *)l1pt->pv_va; pt_entry_t *ptep; pt_entry_t *qmap_pte; vm_paddr_t pa; vm_offset_t va; vm_size_t size; int l1idx, l2idx, l2next = 0; PDEBUG(1, printf("firstaddr = %08x, lastaddr = %08x\n", firstaddr, vm_max_kernel_address)); virtual_avail = firstaddr; kernel_pmap->pm_l1 = l1; kernel_l1pa = l1pt->pv_pa; /* * Scan the L1 translation table created by initarm() and create * the required metadata for all valid mappings found in it. */ for (l1idx = 0; l1idx < (L1_TABLE_SIZE / sizeof(pd_entry_t)); l1idx++) { pde = kernel_l1pt[l1idx]; /* * We're only interested in Coarse mappings. * pmap_extract() can deal with section mappings without * recourse to checking L2 metadata. */ if ((pde & L1_TYPE_MASK) != L1_TYPE_C) continue; /* * Lookup the KVA of this L2 descriptor table */ pa = (vm_paddr_t)(pde & L1_C_ADDR_MASK); ptep = (pt_entry_t *)kernel_pt_lookup(pa); if (ptep == NULL) { panic("pmap_bootstrap: No L2 for va 0x%x, pa 0x%lx", (u_int)l1idx << L1_S_SHIFT, (long unsigned int)pa); } /* * Fetch the associated L2 metadata structure. * Allocate a new one if necessary. */ if ((l2 = kernel_pmap->pm_l2[L2_IDX(l1idx)]) == NULL) { if (l2next == PMAP_STATIC_L2_SIZE) panic("pmap_bootstrap: out of static L2s"); kernel_pmap->pm_l2[L2_IDX(l1idx)] = l2 = &static_l2[l2next++]; } /* * One more L1 slot tracked... */ l2->l2_occupancy++; /* * Fill in the details of the L2 descriptor in the * appropriate bucket. */ l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; l2b->l2b_kva = ptep; l2b->l2b_phys = pa; l2b->l2b_l1idx = l1idx; /* * Establish an initial occupancy count for this descriptor */ for (l2idx = 0; l2idx < (L2_TABLE_SIZE_REAL / sizeof(pt_entry_t)); l2idx++) { if ((ptep[l2idx] & L2_TYPE_MASK) != L2_TYPE_INV) { l2b->l2b_occupancy++; } } /* * Make sure the descriptor itself has the correct cache mode. * If not, fix it, but whine about the problem. Port-meisters * should consider this a clue to fix up their initarm() * function. :) */ if (pmap_set_pt_cache_mode(kernel_l1pt, (vm_offset_t)ptep)) { printf("pmap_bootstrap: WARNING! wrong cache mode for " "L2 pte @ %p\n", ptep); } } /* * Ensure the primary (kernel) L1 has the correct cache mode for * a page table. Bitch if it is not correctly set. */ for (va = (vm_offset_t)kernel_l1pt; va < ((vm_offset_t)kernel_l1pt + L1_TABLE_SIZE); va += PAGE_SIZE) { if (pmap_set_pt_cache_mode(kernel_l1pt, va)) printf("pmap_bootstrap: WARNING! wrong cache mode for " "primary L1 @ 0x%x\n", va); } cpu_dcache_wbinv_all(); cpu_l2cache_wbinv_all(); cpu_tlb_flushID(); cpu_cpwait(); PMAP_LOCK_INIT(kernel_pmap); CPU_FILL(&kernel_pmap->pm_active); kernel_pmap->pm_domain = PMAP_DOMAIN_KERNEL; TAILQ_INIT(&kernel_pmap->pm_pvlist); /* * Initialize the global pv list lock. */ rw_init_flags(&pvh_global_lock, "pmap pv global", RW_RECURSE); /* * Reserve some special page table entries/VA space for temporary * mapping of pages. */ pmap_alloc_specials(&virtual_avail, 1, &csrcp, &csrc_pte); pmap_set_pt_cache_mode(kernel_l1pt, (vm_offset_t)csrc_pte); pmap_alloc_specials(&virtual_avail, 1, &cdstp, &cdst_pte); pmap_set_pt_cache_mode(kernel_l1pt, (vm_offset_t)cdst_pte); pmap_alloc_specials(&virtual_avail, 1, &qmap_addr, &qmap_pte); pmap_set_pt_cache_mode(kernel_l1pt, (vm_offset_t)qmap_pte); size = ((vm_max_kernel_address - pmap_curmaxkvaddr) + L1_S_OFFSET) / L1_S_SIZE; pmap_alloc_specials(&virtual_avail, round_page(size * L2_TABLE_SIZE_REAL) / PAGE_SIZE, &pmap_kernel_l2ptp_kva, NULL); size = (size + (L2_BUCKET_SIZE - 1)) / L2_BUCKET_SIZE; pmap_alloc_specials(&virtual_avail, round_page(size * sizeof(struct l2_dtable)) / PAGE_SIZE, &pmap_kernel_l2dtable_kva, NULL); pmap_alloc_specials(&virtual_avail, 1, (vm_offset_t*)&_tmppt, NULL); pmap_alloc_specials(&virtual_avail, MAXDUMPPGS, (vm_offset_t *)&crashdumpmap, NULL); SLIST_INIT(&l1_list); TAILQ_INIT(&l1_lru_list); mtx_init(&l1_lru_lock, "l1 list lock", NULL, MTX_DEF); pmap_init_l1(l1, kernel_l1pt); cpu_dcache_wbinv_all(); cpu_l2cache_wbinv_all(); virtual_avail = round_page(virtual_avail); virtual_end = vm_max_kernel_address; kernel_vm_end = pmap_curmaxkvaddr; mtx_init(&cmtx, "TMP mappings mtx", NULL, MTX_DEF); mtx_init(&qmap_mtx, "quick mapping mtx", NULL, MTX_DEF); pmap_set_pcb_pagedir(kernel_pmap, thread0.td_pcb); } /*************************************************** * Pmap allocation/deallocation routines. ***************************************************/ /* * Release any resources held by the given physical map. * Called when a pmap initialized by pmap_pinit is being released. * Should only be called if the map contains no valid mappings. */ void pmap_release(pmap_t pmap) { struct pcb *pcb; pmap_idcache_wbinv_all(pmap); cpu_l2cache_wbinv_all(); pmap_tlb_flushID(pmap); cpu_cpwait(); if (vector_page < KERNBASE) { struct pcb *curpcb = PCPU_GET(curpcb); pcb = thread0.td_pcb; if (pmap_is_current(pmap)) { /* * Frob the L1 entry corresponding to the vector * page so that it contains the kernel pmap's domain * number. This will ensure pmap_remove() does not * pull the current vector page out from under us. */ critical_enter(); *pcb->pcb_pl1vec = pcb->pcb_l1vec; cpu_domains(pcb->pcb_dacr); cpu_setttb(pcb->pcb_pagedir); critical_exit(); } pmap_remove(pmap, vector_page, vector_page + PAGE_SIZE); /* * Make sure cpu_switch(), et al, DTRT. This is safe to do * since this process has no remaining mappings of its own. */ curpcb->pcb_pl1vec = pcb->pcb_pl1vec; curpcb->pcb_l1vec = pcb->pcb_l1vec; curpcb->pcb_dacr = pcb->pcb_dacr; curpcb->pcb_pagedir = pcb->pcb_pagedir; } pmap_free_l1(pmap); dprintf("pmap_release()\n"); } /* * Helper function for pmap_grow_l2_bucket() */ static __inline int pmap_grow_map(vm_offset_t va, pt_entry_t cache_mode, vm_paddr_t *pap) { struct l2_bucket *l2b; pt_entry_t *ptep; vm_paddr_t pa; struct vm_page *pg; pg = vm_page_alloc(NULL, 0, VM_ALLOC_NOOBJ | VM_ALLOC_WIRED); if (pg == NULL) return (1); pa = VM_PAGE_TO_PHYS(pg); if (pap) *pap = pa; - l2b = pmap_get_l2_bucket(pmap_kernel(), va); + l2b = pmap_get_l2_bucket(kernel_pmap, va); ptep = &l2b->l2b_kva[l2pte_index(va)]; *ptep = L2_S_PROTO | pa | cache_mode | L2_S_PROT(PTE_KERNEL, VM_PROT_READ | VM_PROT_WRITE); PTE_SYNC(ptep); return (0); } /* * This is the same as pmap_alloc_l2_bucket(), except that it is only * used by pmap_growkernel(). */ static __inline struct l2_bucket * pmap_grow_l2_bucket(pmap_t pm, vm_offset_t va) { struct l2_dtable *l2; struct l2_bucket *l2b; struct l1_ttable *l1; pd_entry_t *pl1pd; u_short l1idx; vm_offset_t nva; l1idx = L1_IDX(va); if ((l2 = pm->pm_l2[L2_IDX(l1idx)]) == NULL) { /* * No mapping at this address, as there is * no entry in the L1 table. * Need to allocate a new l2_dtable. */ nva = pmap_kernel_l2dtable_kva; if ((nva & PAGE_MASK) == 0) { /* * Need to allocate a backing page */ if (pmap_grow_map(nva, pte_l2_s_cache_mode, NULL)) return (NULL); } l2 = (struct l2_dtable *)nva; nva += sizeof(struct l2_dtable); if ((nva & PAGE_MASK) < (pmap_kernel_l2dtable_kva & PAGE_MASK)) { /* * The new l2_dtable straddles a page boundary. * Map in another page to cover it. */ if (pmap_grow_map(nva, pte_l2_s_cache_mode, NULL)) return (NULL); } pmap_kernel_l2dtable_kva = nva; /* * Link it into the parent pmap */ pm->pm_l2[L2_IDX(l1idx)] = l2; memset(l2, 0, sizeof(*l2)); } l2b = &l2->l2_bucket[L2_BUCKET(l1idx)]; /* * Fetch pointer to the L2 page table associated with the address. */ if (l2b->l2b_kva == NULL) { pt_entry_t *ptep; /* * No L2 page table has been allocated. Chances are, this * is because we just allocated the l2_dtable, above. */ nva = pmap_kernel_l2ptp_kva; ptep = (pt_entry_t *)nva; if ((nva & PAGE_MASK) == 0) { /* * Need to allocate a backing page */ if (pmap_grow_map(nva, pte_l2_s_cache_mode_pt, &pmap_kernel_l2ptp_phys)) return (NULL); PTE_SYNC_RANGE(ptep, PAGE_SIZE / sizeof(pt_entry_t)); } memset(ptep, 0, L2_TABLE_SIZE_REAL); l2->l2_occupancy++; l2b->l2b_kva = ptep; l2b->l2b_l1idx = l1idx; l2b->l2b_phys = pmap_kernel_l2ptp_phys; pmap_kernel_l2ptp_kva += L2_TABLE_SIZE_REAL; pmap_kernel_l2ptp_phys += L2_TABLE_SIZE_REAL; } /* Distribute new L1 entry to all other L1s */ SLIST_FOREACH(l1, &l1_list, l1_link) { pl1pd = &l1->l1_kva[L1_IDX(va)]; *pl1pd = l2b->l2b_phys | L1_C_DOM(PMAP_DOMAIN_KERNEL) | L1_C_PROTO; PTE_SYNC(pl1pd); } return (l2b); } /* * grow the number of kernel page table entries, if needed */ void pmap_growkernel(vm_offset_t addr) { - pmap_t kpm = pmap_kernel(); + pmap_t kpm = kernel_pmap; if (addr <= pmap_curmaxkvaddr) return; /* we are OK */ /* * whoops! we need to add kernel PTPs */ /* Map 1MB at a time */ for (; pmap_curmaxkvaddr < addr; pmap_curmaxkvaddr += L1_S_SIZE) pmap_grow_l2_bucket(kpm, pmap_curmaxkvaddr); /* * flush out the cache, expensive but growkernel will happen so * rarely */ cpu_dcache_wbinv_all(); cpu_l2cache_wbinv_all(); cpu_tlb_flushD(); cpu_cpwait(); kernel_vm_end = pmap_curmaxkvaddr; } /* * Remove all pages from specified address space * this aids process exit speeds. Also, this code * is special cased for current process only, but * can have the more generic (and slightly slower) * mode enabled. This is much faster than pmap_remove * in the case of running down an entire address space. */ void pmap_remove_pages(pmap_t pmap) { struct pv_entry *pv, *npv; struct l2_bucket *l2b = NULL; vm_page_t m; pt_entry_t *pt; rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); cpu_idcache_wbinv_all(); cpu_l2cache_wbinv_all(); for (pv = TAILQ_FIRST(&pmap->pm_pvlist); pv; pv = npv) { if (pv->pv_flags & PVF_WIRED || pv->pv_flags & PVF_UNMAN) { /* Cannot remove wired or unmanaged pages now. */ npv = TAILQ_NEXT(pv, pv_plist); continue; } pmap->pm_stats.resident_count--; l2b = pmap_get_l2_bucket(pmap, pv->pv_va); KASSERT(l2b != NULL, ("No L2 bucket in pmap_remove_pages")); pt = &l2b->l2b_kva[l2pte_index(pv->pv_va)]; m = PHYS_TO_VM_PAGE(*pt & L2_ADDR_MASK); KASSERT((vm_offset_t)m >= KERNBASE, ("Trying to access non-existent page va %x pte %x", pv->pv_va, *pt)); *pt = 0; PTE_SYNC(pt); npv = TAILQ_NEXT(pv, pv_plist); pmap_nuke_pv(m, pmap, pv); if (TAILQ_EMPTY(&m->md.pv_list)) vm_page_aflag_clear(m, PGA_WRITEABLE); pmap_free_pv_entry(pv); pmap_free_l2_bucket(pmap, l2b, 1); } rw_wunlock(&pvh_global_lock); cpu_tlb_flushID(); cpu_cpwait(); PMAP_UNLOCK(pmap); } /*************************************************** * Low level mapping routines..... ***************************************************/ #ifdef ARM_HAVE_SUPERSECTIONS /* Map a super section into the KVA. */ void pmap_kenter_supersection(vm_offset_t va, uint64_t pa, int flags) { pd_entry_t pd = L1_S_PROTO | L1_S_SUPERSEC | (pa & L1_SUP_FRAME) | (((pa >> 32) & 0xf) << 20) | L1_S_PROT(PTE_KERNEL, VM_PROT_READ|VM_PROT_WRITE) | L1_S_DOM(PMAP_DOMAIN_KERNEL); struct l1_ttable *l1; vm_offset_t va0, va_end; KASSERT(((va | pa) & L1_SUP_OFFSET) == 0, ("Not a valid super section mapping")); if (flags & SECTION_CACHE) pd |= pte_l1_s_cache_mode; else if (flags & SECTION_PT) pd |= pte_l1_s_cache_mode_pt; va0 = va & L1_SUP_FRAME; va_end = va + L1_SUP_SIZE; SLIST_FOREACH(l1, &l1_list, l1_link) { va = va0; for (; va < va_end; va += L1_S_SIZE) { l1->l1_kva[L1_IDX(va)] = pd; PTE_SYNC(&l1->l1_kva[L1_IDX(va)]); } } } #endif /* Map a section into the KVA. */ void pmap_kenter_section(vm_offset_t va, vm_offset_t pa, int flags) { pd_entry_t pd = L1_S_PROTO | pa | L1_S_PROT(PTE_KERNEL, VM_PROT_READ|VM_PROT_WRITE) | L1_S_DOM(PMAP_DOMAIN_KERNEL); struct l1_ttable *l1; KASSERT(((va | pa) & L1_S_OFFSET) == 0, ("Not a valid section mapping")); if (flags & SECTION_CACHE) pd |= pte_l1_s_cache_mode; else if (flags & SECTION_PT) pd |= pte_l1_s_cache_mode_pt; SLIST_FOREACH(l1, &l1_list, l1_link) { l1->l1_kva[L1_IDX(va)] = pd; PTE_SYNC(&l1->l1_kva[L1_IDX(va)]); } } /* * Make a temporary mapping for a physical address. This is only intended * to be used for panic dumps. */ void * pmap_kenter_temporary(vm_paddr_t pa, int i) { vm_offset_t va; va = (vm_offset_t)crashdumpmap + (i * PAGE_SIZE); pmap_kenter(va, pa); return ((void *)crashdumpmap); } /* * add a wired page to the kva * note that in order for the mapping to take effect -- you * should do a invltlb after doing the pmap_kenter... */ static PMAP_INLINE void pmap_kenter_internal(vm_offset_t va, vm_offset_t pa, int flags) { struct l2_bucket *l2b; pt_entry_t *pte; pt_entry_t opte; struct pv_entry *pve; vm_page_t m; PDEBUG(1, printf("pmap_kenter: va = %08x, pa = %08x\n", (uint32_t) va, (uint32_t) pa)); - l2b = pmap_get_l2_bucket(pmap_kernel(), va); + l2b = pmap_get_l2_bucket(kernel_pmap, va); if (l2b == NULL) - l2b = pmap_grow_l2_bucket(pmap_kernel(), va); + l2b = pmap_grow_l2_bucket(kernel_pmap, va); KASSERT(l2b != NULL, ("No L2 Bucket")); pte = &l2b->l2b_kva[l2pte_index(va)]; opte = *pte; PDEBUG(1, printf("pmap_kenter: pte = %08x, opte = %08x, npte = %08x\n", (uint32_t) pte, opte, *pte)); if (l2pte_valid(opte)) { pmap_kremove(va); } else { if (opte == 0) l2b->l2b_occupancy++; } *pte = L2_S_PROTO | pa | L2_S_PROT(PTE_KERNEL, VM_PROT_READ | VM_PROT_WRITE); if (flags & KENTER_CACHE) *pte |= pte_l2_s_cache_mode; if (flags & KENTER_USER) *pte |= L2_S_PROT_U; PTE_SYNC(pte); /* * A kernel mapping may not be the page's only mapping, so create a PV * entry to ensure proper caching. * * The existence test for the pvzone is used to delay the recording of * kernel mappings until the VM system is fully initialized. * * This expects the physical memory to have a vm_page_array entry. */ if (pvzone != NULL && (m = vm_phys_paddr_to_vm_page(pa)) != NULL) { rw_wlock(&pvh_global_lock); if (!TAILQ_EMPTY(&m->md.pv_list) || m->md.pv_kva != 0) { if ((pve = pmap_get_pv_entry()) == NULL) panic("pmap_kenter_internal: no pv entries"); - PMAP_LOCK(pmap_kernel()); - pmap_enter_pv(m, pve, pmap_kernel(), va, + PMAP_LOCK(kernel_pmap); + pmap_enter_pv(m, pve, kernel_pmap, va, PVF_WRITE | PVF_UNMAN); - pmap_fix_cache(m, pmap_kernel(), va); - PMAP_UNLOCK(pmap_kernel()); + pmap_fix_cache(m, kernel_pmap, va); + PMAP_UNLOCK(kernel_pmap); } else { m->md.pv_kva = va; } rw_wunlock(&pvh_global_lock); } } void pmap_kenter(vm_offset_t va, vm_paddr_t pa) { pmap_kenter_internal(va, pa, KENTER_CACHE); } void pmap_kenter_nocache(vm_offset_t va, vm_paddr_t pa) { pmap_kenter_internal(va, pa, 0); } void pmap_kenter_device(vm_offset_t va, vm_size_t size, vm_paddr_t pa) { vm_offset_t sva; KASSERT((size & PAGE_MASK) == 0, ("%s: device mapping not page-sized", __func__)); sva = va; while (size != 0) { pmap_kenter_internal(va, pa, 0); va += PAGE_SIZE; pa += PAGE_SIZE; size -= PAGE_SIZE; } } void pmap_kremove_device(vm_offset_t va, vm_size_t size) { vm_offset_t sva; KASSERT((size & PAGE_MASK) == 0, ("%s: device mapping not page-sized", __func__)); sva = va; while (size != 0) { pmap_kremove(va); va += PAGE_SIZE; size -= PAGE_SIZE; } } void pmap_kenter_user(vm_offset_t va, vm_paddr_t pa) { pmap_kenter_internal(va, pa, KENTER_CACHE|KENTER_USER); /* * Call pmap_fault_fixup now, to make sure we'll have no exception * at the first use of the new address, or bad things will happen, * as we use one of these addresses in the exception handlers. */ - pmap_fault_fixup(pmap_kernel(), va, VM_PROT_READ|VM_PROT_WRITE, 1); + pmap_fault_fixup(kernel_pmap, va, VM_PROT_READ|VM_PROT_WRITE, 1); } vm_paddr_t pmap_kextract(vm_offset_t va) { return (pmap_extract_locked(kernel_pmap, va)); } /* * remove a page from the kernel pagetables */ void pmap_kremove(vm_offset_t va) { struct l2_bucket *l2b; pt_entry_t *pte, opte; struct pv_entry *pve; vm_page_t m; vm_offset_t pa; - l2b = pmap_get_l2_bucket(pmap_kernel(), va); + l2b = pmap_get_l2_bucket(kernel_pmap, va); if (!l2b) return; KASSERT(l2b != NULL, ("No L2 Bucket")); pte = &l2b->l2b_kva[l2pte_index(va)]; opte = *pte; if (l2pte_valid(opte)) { /* pa = vtophs(va) taken from pmap_extract() */ if ((opte & L2_TYPE_MASK) == L2_TYPE_L) pa = (opte & L2_L_FRAME) | (va & L2_L_OFFSET); else pa = (opte & L2_S_FRAME) | (va & L2_S_OFFSET); /* note: should never have to remove an allocation * before the pvzone is initialized. */ rw_wlock(&pvh_global_lock); - PMAP_LOCK(pmap_kernel()); + PMAP_LOCK(kernel_pmap); if (pvzone != NULL && (m = vm_phys_paddr_to_vm_page(pa)) && - (pve = pmap_remove_pv(m, pmap_kernel(), va))) + (pve = pmap_remove_pv(m, kernel_pmap, va))) pmap_free_pv_entry(pve); - PMAP_UNLOCK(pmap_kernel()); + PMAP_UNLOCK(kernel_pmap); rw_wunlock(&pvh_global_lock); va = va & ~PAGE_MASK; cpu_dcache_wbinv_range(va, PAGE_SIZE); cpu_l2cache_wbinv_range(va, PAGE_SIZE); cpu_tlb_flushD_SE(va); cpu_cpwait(); *pte = 0; } } /* * Used to map a range of physical addresses into kernel * virtual address space. * * The value passed in '*virt' is a suggested virtual address for * the mapping. Architectures which can support a direct-mapped * physical to virtual region can return the appropriate address * within that region, leaving '*virt' unchanged. Other * architectures should map the pages starting at '*virt' and * update '*virt' with the first usable address after the mapped * region. */ vm_offset_t pmap_map(vm_offset_t *virt, vm_offset_t start, vm_offset_t end, int prot) { vm_offset_t sva = *virt; vm_offset_t va = sva; PDEBUG(1, printf("pmap_map: virt = %08x, start = %08x, end = %08x, " "prot = %d\n", (uint32_t) *virt, (uint32_t) start, (uint32_t) end, prot)); while (start < end) { pmap_kenter(va, start); va += PAGE_SIZE; start += PAGE_SIZE; } *virt = va; return (sva); } static void pmap_wb_page(vm_page_t m) { struct pv_entry *pv; TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) pmap_dcache_wb_range(pv->pv_pmap, pv->pv_va, PAGE_SIZE, FALSE, (pv->pv_flags & PVF_WRITE) == 0); } static void pmap_inv_page(vm_page_t m) { struct pv_entry *pv; TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) pmap_dcache_wb_range(pv->pv_pmap, pv->pv_va, PAGE_SIZE, TRUE, TRUE); } /* * Add a list of wired pages to the kva * this routine is only used for temporary * kernel mappings that do not need to have * page modification or references recorded. * Note that old mappings are simply written * over. The page *must* be wired. */ void pmap_qenter(vm_offset_t va, vm_page_t *m, int count) { int i; for (i = 0; i < count; i++) { pmap_wb_page(m[i]); pmap_kenter_internal(va, VM_PAGE_TO_PHYS(m[i]), KENTER_CACHE); va += PAGE_SIZE; } } /* * this routine jerks page mappings from the * kernel -- it is meant only for temporary mappings. */ void pmap_qremove(vm_offset_t va, int count) { vm_paddr_t pa; int i; for (i = 0; i < count; i++) { pa = vtophys(va); if (pa) { pmap_inv_page(PHYS_TO_VM_PAGE(pa)); pmap_kremove(va); } va += PAGE_SIZE; } } /* * pmap_object_init_pt preloads the ptes for a given object * into the specified pmap. This eliminates the blast of soft * faults on process startup and immediately after an mmap. */ void pmap_object_init_pt(pmap_t pmap, vm_offset_t addr, vm_object_t object, vm_pindex_t pindex, vm_size_t size) { VM_OBJECT_ASSERT_WLOCKED(object); KASSERT(object->type == OBJT_DEVICE || object->type == OBJT_SG, ("pmap_object_init_pt: non-device object")); } /* * pmap_is_prefaultable: * * Return whether or not the specified virtual address is elgible * for prefault. */ boolean_t pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr) { pd_entry_t *pde; pt_entry_t *pte; if (!pmap_get_pde_pte(pmap, addr, &pde, &pte)) return (FALSE); KASSERT(pte != NULL, ("Valid mapping but no pte ?")); if (*pte == 0) return (TRUE); return (FALSE); } /* * Fetch pointers to the PDE/PTE for the given pmap/VA pair. * Returns TRUE if the mapping exists, else FALSE. * * NOTE: This function is only used by a couple of arm-specific modules. * It is not safe to take any pmap locks here, since we could be right * in the middle of debugging the pmap anyway... * * It is possible for this routine to return FALSE even though a valid * mapping does exist. This is because we don't lock, so the metadata * state may be inconsistent. * * NOTE: We can return a NULL *ptp in the case where the L1 pde is * a "section" mapping. */ boolean_t pmap_get_pde_pte(pmap_t pm, vm_offset_t va, pd_entry_t **pdp, pt_entry_t **ptp) { struct l2_dtable *l2; pd_entry_t *pl1pd, l1pd; pt_entry_t *ptep; u_short l1idx; if (pm->pm_l1 == NULL) return (FALSE); l1idx = L1_IDX(va); *pdp = pl1pd = &pm->pm_l1->l1_kva[l1idx]; l1pd = *pl1pd; if (l1pte_section_p(l1pd)) { *ptp = NULL; return (TRUE); } if (pm->pm_l2 == NULL) return (FALSE); l2 = pm->pm_l2[L2_IDX(l1idx)]; if (l2 == NULL || (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) { return (FALSE); } *ptp = &ptep[l2pte_index(va)]; return (TRUE); } /* * Routine: pmap_remove_all * Function: * Removes this physical page from * all physical maps in which it resides. * Reflects back modify bits to the pager. * * Notes: * Original versions of this routine were very * inefficient because they iteratively called * pmap_remove (slow...) */ void pmap_remove_all(vm_page_t m) { pv_entry_t pv; pt_entry_t *ptep; struct l2_bucket *l2b; boolean_t flush = FALSE; pmap_t curpm; int flags = 0; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_remove_all: page %p is not managed", m)); if (TAILQ_EMPTY(&m->md.pv_list)) return; rw_wlock(&pvh_global_lock); /* * XXX This call shouldn't exist. Iterating over the PV list twice, * once in pmap_clearbit() and again below, is both unnecessary and * inefficient. The below code should itself write back the cache * entry before it destroys the mapping. */ pmap_clearbit(m, PVF_WRITE); curpm = vmspace_pmap(curproc->p_vmspace); while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { if (flush == FALSE && (pv->pv_pmap == curpm || - pv->pv_pmap == pmap_kernel())) + pv->pv_pmap == kernel_pmap)) flush = TRUE; PMAP_LOCK(pv->pv_pmap); /* * Cached contents were written-back in pmap_clearbit(), * but we still have to invalidate the cache entry to make * sure stale data are not retrieved when another page will be * mapped under this virtual address. */ if (pmap_is_current(pv->pv_pmap)) { cpu_dcache_inv_range(pv->pv_va, PAGE_SIZE); if (pmap_has_valid_mapping(pv->pv_pmap, pv->pv_va)) cpu_l2cache_inv_range(pv->pv_va, PAGE_SIZE); } if (pv->pv_flags & PVF_UNMAN) { /* remove the pv entry, but do not remove the mapping * and remember this is a kernel mapped page */ m->md.pv_kva = pv->pv_va; } else { /* remove the mapping and pv entry */ l2b = pmap_get_l2_bucket(pv->pv_pmap, pv->pv_va); KASSERT(l2b != NULL, ("No l2 bucket")); ptep = &l2b->l2b_kva[l2pte_index(pv->pv_va)]; *ptep = 0; PTE_SYNC_CURRENT(pv->pv_pmap, ptep); pmap_free_l2_bucket(pv->pv_pmap, l2b, 1); pv->pv_pmap->pm_stats.resident_count--; flags |= pv->pv_flags; } pmap_nuke_pv(m, pv->pv_pmap, pv); PMAP_UNLOCK(pv->pv_pmap); pmap_free_pv_entry(pv); } if (flush) { if (PV_BEEN_EXECD(flags)) pmap_tlb_flushID(curpm); else pmap_tlb_flushD(curpm); } vm_page_aflag_clear(m, PGA_WRITEABLE); rw_wunlock(&pvh_global_lock); } /* * Set the physical protection on the * specified range of this map as requested. */ void pmap_protect(pmap_t pm, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) { struct l2_bucket *l2b; pt_entry_t *ptep, pte; vm_offset_t next_bucket; u_int flags; int flush; CTR4(KTR_PMAP, "pmap_protect: pmap %p sva 0x%08x eva 0x%08x prot %x", pm, sva, eva, prot); if ((prot & VM_PROT_READ) == 0) { pmap_remove(pm, sva, eva); return; } if (prot & VM_PROT_WRITE) { /* * If this is a read->write transition, just ignore it and let * vm_fault() take care of it later. */ return; } rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); /* * OK, at this point, we know we're doing write-protect operation. * If the pmap is active, write-back the range. */ pmap_dcache_wb_range(pm, sva, eva - sva, FALSE, FALSE); flush = ((eva - sva) >= (PAGE_SIZE * 4)) ? 0 : -1; flags = 0; while (sva < eva) { next_bucket = L2_NEXT_BUCKET(sva); if (next_bucket > eva) next_bucket = eva; l2b = pmap_get_l2_bucket(pm, sva); if (l2b == NULL) { sva = next_bucket; continue; } ptep = &l2b->l2b_kva[l2pte_index(sva)]; while (sva < next_bucket) { if ((pte = *ptep) != 0 && (pte & L2_S_PROT_W) != 0) { struct vm_page *pg; u_int f; pg = PHYS_TO_VM_PAGE(l2pte_pa(pte)); pte &= ~L2_S_PROT_W; *ptep = pte; PTE_SYNC(ptep); if (!(pg->oflags & VPO_UNMANAGED)) { f = pmap_modify_pv(pg, pm, sva, PVF_WRITE, 0); if (f & PVF_WRITE) vm_page_dirty(pg); } else f = 0; if (flush >= 0) { flush++; flags |= f; } else if (PV_BEEN_EXECD(f)) pmap_tlb_flushID_SE(pm, sva); else if (PV_BEEN_REFD(f)) pmap_tlb_flushD_SE(pm, sva); } sva += PAGE_SIZE; ptep++; } } if (flush) { if (PV_BEEN_EXECD(flags)) pmap_tlb_flushID(pm); else if (PV_BEEN_REFD(flags)) pmap_tlb_flushD(pm); } rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pm); } /* * Insert the given physical page (p) at * the specified virtual address (v) in the * target physical map with the protection requested. * * If specified, the page will be wired down, meaning * that the related pte can not be reclaimed. * * NB: This is the only routine which MAY NOT lazy-evaluate * or lose information. That is, this routine must actually * insert this page into the given map NOW. */ int pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, u_int flags, int8_t psind __unused) { int rv; rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); rv = pmap_enter_locked(pmap, va, m, prot, flags); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); return (rv); } /* * The pvh global and pmap locks must be held. */ static int pmap_enter_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, u_int flags) { struct l2_bucket *l2b = NULL; struct vm_page *opg; struct pv_entry *pve = NULL; pt_entry_t *ptep, npte, opte; u_int nflags; u_int oflags; vm_paddr_t pa; PMAP_ASSERT_LOCKED(pmap); rw_assert(&pvh_global_lock, RA_WLOCKED); if (va == vector_page) { pa = systempage.pv_pa; m = NULL; } else { if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m)) VM_OBJECT_ASSERT_LOCKED(m->object); pa = VM_PAGE_TO_PHYS(m); } nflags = 0; if (prot & VM_PROT_WRITE) nflags |= PVF_WRITE; if (prot & VM_PROT_EXECUTE) nflags |= PVF_EXEC; if ((flags & PMAP_ENTER_WIRED) != 0) nflags |= PVF_WIRED; PDEBUG(1, printf("pmap_enter: pmap = %08x, va = %08x, m = %08x, prot = %x, " "flags = %x\n", (uint32_t) pmap, va, (uint32_t) m, prot, flags)); - if (pmap == pmap_kernel()) { + if (pmap == kernel_pmap) { l2b = pmap_get_l2_bucket(pmap, va); if (l2b == NULL) l2b = pmap_grow_l2_bucket(pmap, va); } else { do_l2b_alloc: l2b = pmap_alloc_l2_bucket(pmap, va); if (l2b == NULL) { if ((flags & PMAP_ENTER_NOSLEEP) == 0) { PMAP_UNLOCK(pmap); rw_wunlock(&pvh_global_lock); VM_WAIT; rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); goto do_l2b_alloc; } return (KERN_RESOURCE_SHORTAGE); } } ptep = &l2b->l2b_kva[l2pte_index(va)]; opte = *ptep; npte = pa; oflags = 0; if (opte) { /* * There is already a mapping at this address. * If the physical address is different, lookup the * vm_page. */ if (l2pte_pa(opte) != pa) opg = PHYS_TO_VM_PAGE(l2pte_pa(opte)); else opg = m; } else opg = NULL; if ((prot & (VM_PROT_ALL)) || (!m || m->md.pvh_attrs & PVF_REF)) { /* * - The access type indicates that we don't need * to do referenced emulation. * OR * - The physical page has already been referenced * so no need to re-do referenced emulation here. */ npte |= L2_S_PROTO; nflags |= PVF_REF; if (m && ((prot & VM_PROT_WRITE) != 0 || (m->md.pvh_attrs & PVF_MOD))) { /* * This is a writable mapping, and the * page's mod state indicates it has * already been modified. Make it * writable from the outset. */ nflags |= PVF_MOD; if (!(m->md.pvh_attrs & PVF_MOD)) vm_page_dirty(m); } if (m && opte) vm_page_aflag_set(m, PGA_REFERENCED); } else { /* * Need to do page referenced emulation. */ npte |= L2_TYPE_INV; } if (prot & VM_PROT_WRITE) { npte |= L2_S_PROT_W; if (m != NULL && (m->oflags & VPO_UNMANAGED) == 0) vm_page_aflag_set(m, PGA_WRITEABLE); } if (m->md.pv_memattr != VM_MEMATTR_UNCACHEABLE) npte |= pte_l2_s_cache_mode; if (m && m == opg) { /* * We're changing the attrs of an existing mapping. */ oflags = pmap_modify_pv(m, pmap, va, PVF_WRITE | PVF_EXEC | PVF_WIRED | PVF_MOD | PVF_REF, nflags); /* * We may need to flush the cache if we're * doing rw-ro... */ if (pmap_is_current(pmap) && (oflags & PVF_NC) == 0 && (opte & L2_S_PROT_W) != 0 && (prot & VM_PROT_WRITE) == 0 && (opte & L2_TYPE_MASK) != L2_TYPE_INV) { cpu_dcache_wb_range(va, PAGE_SIZE); cpu_l2cache_wb_range(va, PAGE_SIZE); } } else { /* * New mapping, or changing the backing page * of an existing mapping. */ if (opg) { /* * Replacing an existing mapping with a new one. * It is part of our managed memory so we * must remove it from the PV list */ if ((pve = pmap_remove_pv(opg, pmap, va))) { /* note for patch: the oflags/invalidation was moved * because PG_FICTITIOUS pages could free the pve */ oflags = pve->pv_flags; /* * If the old mapping was valid (ref/mod * emulation creates 'invalid' mappings * initially) then make sure to frob * the cache. */ if ((oflags & PVF_NC) == 0 && l2pte_valid(opte)) { if (PV_BEEN_EXECD(oflags)) { pmap_idcache_wbinv_range(pmap, va, PAGE_SIZE); } else if (PV_BEEN_REFD(oflags)) { pmap_dcache_wb_range(pmap, va, PAGE_SIZE, TRUE, (oflags & PVF_WRITE) == 0); } } /* free/allocate a pv_entry for UNMANAGED pages if * this physical page is not/is already mapped. */ if (m && (m->oflags & VPO_UNMANAGED) && !m->md.pv_kva && TAILQ_EMPTY(&m->md.pv_list)) { pmap_free_pv_entry(pve); pve = NULL; } } else if (m && (!(m->oflags & VPO_UNMANAGED) || m->md.pv_kva || !TAILQ_EMPTY(&m->md.pv_list))) pve = pmap_get_pv_entry(); } else if (m && (!(m->oflags & VPO_UNMANAGED) || m->md.pv_kva || !TAILQ_EMPTY(&m->md.pv_list))) pve = pmap_get_pv_entry(); if (m) { if ((m->oflags & VPO_UNMANAGED)) { if (!TAILQ_EMPTY(&m->md.pv_list) || m->md.pv_kva) { KASSERT(pve != NULL, ("No pv")); nflags |= PVF_UNMAN; pmap_enter_pv(m, pve, pmap, va, nflags); } else m->md.pv_kva = va; } else { KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva, ("pmap_enter: managed mapping within the clean submap")); KASSERT(pve != NULL, ("No pv")); pmap_enter_pv(m, pve, pmap, va, nflags); } } } /* * Make sure userland mappings get the right permissions */ - if (pmap != pmap_kernel() && va != vector_page) { + if (pmap != kernel_pmap && va != vector_page) { npte |= L2_S_PROT_U; } /* * Keep the stats up to date */ if (opte == 0) { l2b->l2b_occupancy++; pmap->pm_stats.resident_count++; } /* * If this is just a wiring change, the two PTEs will be * identical, so there's no need to update the page table. */ if (npte != opte) { boolean_t is_cached = pmap_is_current(pmap); *ptep = npte; if (is_cached) { /* * We only need to frob the cache/tlb if this pmap * is current */ PTE_SYNC(ptep); if (L1_IDX(va) != L1_IDX(vector_page) && l2pte_valid(npte)) { /* * This mapping is likely to be accessed as * soon as we return to userland. Fix up the * L1 entry to avoid taking another * page/domain fault. */ pd_entry_t *pl1pd, l1pd; pl1pd = &pmap->pm_l1->l1_kva[L1_IDX(va)]; l1pd = l2b->l2b_phys | L1_C_DOM(pmap->pm_domain) | L1_C_PROTO; if (*pl1pd != l1pd) { *pl1pd = l1pd; PTE_SYNC(pl1pd); } } } if (PV_BEEN_EXECD(oflags)) pmap_tlb_flushID_SE(pmap, va); else if (PV_BEEN_REFD(oflags)) pmap_tlb_flushD_SE(pmap, va); if (m) pmap_fix_cache(m, pmap, va); } return (KERN_SUCCESS); } /* * Maps a sequence of resident pages belonging to the same object. * The sequence begins with the given page m_start. This page is * mapped at the given virtual address start. Each subsequent page is * mapped at a virtual address that is offset from start by the same * amount as the page is offset from m_start within the object. The * last page in the sequence is the page with the largest offset from * m_start that can be mapped at a virtual address less than the given * virtual address end. Not every virtual page between start and end * is mapped; only those for which a resident page exists with the * corresponding offset from m_start are mapped. */ void pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, vm_page_t m_start, vm_prot_t prot) { vm_page_t m; vm_pindex_t diff, psize; VM_OBJECT_ASSERT_LOCKED(m_start->object); psize = atop(end - start); m = m_start; rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { pmap_enter_locked(pmap, start + ptoa(diff), m, prot & (VM_PROT_READ | VM_PROT_EXECUTE), PMAP_ENTER_NOSLEEP); m = TAILQ_NEXT(m, listq); } rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } /* * this code makes some *MAJOR* assumptions: * 1. Current pmap & pmap exists. * 2. Not wired. * 3. Read access. * 4. No page table pages. * but is *MUCH* faster than pmap_enter... */ void pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) { rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); pmap_enter_locked(pmap, va, m, prot & (VM_PROT_READ | VM_PROT_EXECUTE), PMAP_ENTER_NOSLEEP); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } /* * Clear the wired attribute from the mappings for the specified range of * addresses in the given pmap. Every valid mapping within that range * must have the wired attribute set. In contrast, invalid mappings * cannot have the wired attribute set, so they are ignored. * * XXX Wired mappings of unmanaged pages cannot be counted by this pmap * implementation. */ void pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { struct l2_bucket *l2b; pt_entry_t *ptep, pte; pv_entry_t pv; vm_offset_t next_bucket; vm_page_t m; rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); while (sva < eva) { next_bucket = L2_NEXT_BUCKET(sva); if (next_bucket > eva) next_bucket = eva; l2b = pmap_get_l2_bucket(pmap, sva); if (l2b == NULL) { sva = next_bucket; continue; } for (ptep = &l2b->l2b_kva[l2pte_index(sva)]; sva < next_bucket; sva += PAGE_SIZE, ptep++) { if ((pte = *ptep) == 0 || (m = PHYS_TO_VM_PAGE(l2pte_pa(pte))) == NULL || (m->oflags & VPO_UNMANAGED) != 0) continue; pv = pmap_find_pv(m, pmap, sva); if ((pv->pv_flags & PVF_WIRED) == 0) panic("pmap_unwire: pv %p isn't wired", pv); pv->pv_flags &= ~PVF_WIRED; pmap->pm_stats.wired_count--; } } rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } /* * Copy the range specified by src_addr/len * from the source map to the range dst_addr/len * in the destination map. * * This routine is only advisory and need not do anything. */ void pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t len, vm_offset_t src_addr) { } /* * Routine: pmap_extract * Function: * Extract the physical page address associated * with the given map/virtual_address pair. */ vm_paddr_t pmap_extract(pmap_t pmap, vm_offset_t va) { vm_paddr_t pa; PMAP_LOCK(pmap); pa = pmap_extract_locked(pmap, va); PMAP_UNLOCK(pmap); return (pa); } static vm_paddr_t pmap_extract_locked(pmap_t pmap, vm_offset_t va) { struct l2_dtable *l2; pd_entry_t l1pd; pt_entry_t *ptep, pte; vm_paddr_t pa; u_int l1idx; if (pmap != kernel_pmap) PMAP_ASSERT_LOCKED(pmap); l1idx = L1_IDX(va); l1pd = pmap->pm_l1->l1_kva[l1idx]; if (l1pte_section_p(l1pd)) { /* * These should only happen for the kernel pmap. */ KASSERT(pmap == kernel_pmap, ("unexpected section")); /* XXX: what to do about the bits > 32 ? */ if (l1pd & L1_S_SUPERSEC) pa = (l1pd & L1_SUP_FRAME) | (va & L1_SUP_OFFSET); else pa = (l1pd & L1_S_FRAME) | (va & L1_S_OFFSET); } else { /* * Note that we can't rely on the validity of the L1 * descriptor as an indication that a mapping exists. * We have to look it up in the L2 dtable. */ l2 = pmap->pm_l2[L2_IDX(l1idx)]; if (l2 == NULL || (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) return (0); pte = ptep[l2pte_index(va)]; if (pte == 0) return (0); if ((pte & L2_TYPE_MASK) == L2_TYPE_L) pa = (pte & L2_L_FRAME) | (va & L2_L_OFFSET); else pa = (pte & L2_S_FRAME) | (va & L2_S_OFFSET); } return (pa); } /* * Atomically extract and hold the physical page with the given * pmap and virtual address pair if that mapping permits the given * protection. * */ vm_page_t pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) { struct l2_dtable *l2; pd_entry_t l1pd; pt_entry_t *ptep, pte; vm_paddr_t pa, paddr; vm_page_t m = NULL; u_int l1idx; l1idx = L1_IDX(va); paddr = 0; PMAP_LOCK(pmap); retry: l1pd = pmap->pm_l1->l1_kva[l1idx]; if (l1pte_section_p(l1pd)) { /* - * These should only happen for pmap_kernel() + * These should only happen for kernel_pmap */ - KASSERT(pmap == pmap_kernel(), ("huh")); + KASSERT(pmap == kernel_pmap, ("huh")); /* XXX: what to do about the bits > 32 ? */ if (l1pd & L1_S_SUPERSEC) pa = (l1pd & L1_SUP_FRAME) | (va & L1_SUP_OFFSET); else pa = (l1pd & L1_S_FRAME) | (va & L1_S_OFFSET); if (vm_page_pa_tryrelock(pmap, pa & PG_FRAME, &paddr)) goto retry; if (l1pd & L1_S_PROT_W || (prot & VM_PROT_WRITE) == 0) { m = PHYS_TO_VM_PAGE(pa); vm_page_hold(m); } } else { /* * Note that we can't rely on the validity of the L1 * descriptor as an indication that a mapping exists. * We have to look it up in the L2 dtable. */ l2 = pmap->pm_l2[L2_IDX(l1idx)]; if (l2 == NULL || (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) { PMAP_UNLOCK(pmap); return (NULL); } ptep = &ptep[l2pte_index(va)]; pte = *ptep; if (pte == 0) { PMAP_UNLOCK(pmap); return (NULL); } if (pte & L2_S_PROT_W || (prot & VM_PROT_WRITE) == 0) { if ((pte & L2_TYPE_MASK) == L2_TYPE_L) pa = (pte & L2_L_FRAME) | (va & L2_L_OFFSET); else pa = (pte & L2_S_FRAME) | (va & L2_S_OFFSET); if (vm_page_pa_tryrelock(pmap, pa & PG_FRAME, &paddr)) goto retry; m = PHYS_TO_VM_PAGE(pa); vm_page_hold(m); } } PMAP_UNLOCK(pmap); PA_UNLOCK_COND(paddr); return (m); } vm_paddr_t pmap_dump_kextract(vm_offset_t va, pt2_entry_t *pte2p) { struct l2_dtable *l2; pd_entry_t l1pd; pt_entry_t *ptep, pte; vm_paddr_t pa; u_int l1idx; l1idx = L1_IDX(va); l1pd = kernel_pmap->pm_l1->l1_kva[l1idx]; if (l1pte_section_p(l1pd)) { if (l1pd & L1_S_SUPERSEC) pa = (l1pd & L1_SUP_FRAME) | (va & L1_SUP_OFFSET); else pa = (l1pd & L1_S_FRAME) | (va & L1_S_OFFSET); pte = L2_S_PROTO | pa | L2_S_PROT(PTE_KERNEL, VM_PROT_READ | VM_PROT_WRITE); } else { l2 = kernel_pmap->pm_l2[L2_IDX(l1idx)]; if (l2 == NULL || (ptep = l2->l2_bucket[L2_BUCKET(l1idx)].l2b_kva) == NULL) { pte = 0; pa = 0; goto out; } pte = ptep[l2pte_index(va)]; if (pte == 0) { pa = 0; goto out; } if ((pte & L2_TYPE_MASK) == L2_TYPE_L) pa = (pte & L2_L_FRAME) | (va & L2_L_OFFSET); else pa = (pte & L2_S_FRAME) | (va & L2_S_OFFSET); } out: if (pte2p != NULL) *pte2p = pte; return (pa); } /* * Initialize a preallocated and zeroed pmap structure, * such as one in a vmspace structure. */ int pmap_pinit(pmap_t pmap) { PDEBUG(1, printf("pmap_pinit: pmap = %08x\n", (uint32_t) pmap)); pmap_alloc_l1(pmap); bzero(pmap->pm_l2, sizeof(pmap->pm_l2)); CPU_ZERO(&pmap->pm_active); TAILQ_INIT(&pmap->pm_pvlist); bzero(&pmap->pm_stats, sizeof pmap->pm_stats); pmap->pm_stats.resident_count = 1; if (vector_page < KERNBASE) { pmap_enter(pmap, vector_page, PHYS_TO_VM_PAGE(systempage.pv_pa), VM_PROT_READ, PMAP_ENTER_WIRED | VM_PROT_READ, 0); } return (1); } /*************************************************** * page management routines. ***************************************************/ static void pmap_free_pv_entry(pv_entry_t pv) { pv_entry_count--; uma_zfree(pvzone, pv); } /* * get a new pv_entry, allocating a block from the system * when needed. * the memory allocation is performed bypassing the malloc code * because of the possibility of allocations at interrupt time. */ static pv_entry_t pmap_get_pv_entry(void) { pv_entry_t ret_value; pv_entry_count++; if (pv_entry_count > pv_entry_high_water) pagedaemon_wakeup(); ret_value = uma_zalloc(pvzone, M_NOWAIT); return ret_value; } /* * Remove the given range of addresses from the specified map. * * It is assumed that the start and end are properly * rounded to the page size. */ #define PMAP_REMOVE_CLEAN_LIST_SIZE 3 void pmap_remove(pmap_t pm, vm_offset_t sva, vm_offset_t eva) { struct l2_bucket *l2b; vm_offset_t next_bucket; pt_entry_t *ptep; u_int total; u_int mappings, is_exec, is_refd; int flushall = 0; /* * we lock in the pmap => pv_head direction */ rw_wlock(&pvh_global_lock); PMAP_LOCK(pm); total = 0; while (sva < eva) { /* * Do one L2 bucket's worth at a time. */ next_bucket = L2_NEXT_BUCKET(sva); if (next_bucket > eva) next_bucket = eva; l2b = pmap_get_l2_bucket(pm, sva); if (l2b == NULL) { sva = next_bucket; continue; } ptep = &l2b->l2b_kva[l2pte_index(sva)]; mappings = 0; while (sva < next_bucket) { struct vm_page *pg; pt_entry_t pte; vm_paddr_t pa; pte = *ptep; if (pte == 0) { /* * Nothing here, move along */ sva += PAGE_SIZE; ptep++; continue; } pm->pm_stats.resident_count--; pa = l2pte_pa(pte); is_exec = 0; is_refd = 1; /* * Update flags. In a number of circumstances, * we could cluster a lot of these and do a * number of sequential pages in one go. */ if ((pg = PHYS_TO_VM_PAGE(pa)) != NULL) { struct pv_entry *pve; pve = pmap_remove_pv(pg, pm, sva); if (pve) { is_exec = PV_BEEN_EXECD(pve->pv_flags); is_refd = PV_BEEN_REFD(pve->pv_flags); pmap_free_pv_entry(pve); } } if (l2pte_valid(pte) && pmap_is_current(pm)) { if (total < PMAP_REMOVE_CLEAN_LIST_SIZE) { total++; if (is_exec) { cpu_idcache_wbinv_range(sva, PAGE_SIZE); cpu_l2cache_wbinv_range(sva, PAGE_SIZE); cpu_tlb_flushID_SE(sva); } else if (is_refd) { cpu_dcache_wbinv_range(sva, PAGE_SIZE); cpu_l2cache_wbinv_range(sva, PAGE_SIZE); cpu_tlb_flushD_SE(sva); } } else if (total == PMAP_REMOVE_CLEAN_LIST_SIZE) { /* flushall will also only get set for * for a current pmap */ cpu_idcache_wbinv_all(); cpu_l2cache_wbinv_all(); flushall = 1; total++; } } *ptep = 0; PTE_SYNC(ptep); sva += PAGE_SIZE; ptep++; mappings++; } pmap_free_l2_bucket(pm, l2b, mappings); } rw_wunlock(&pvh_global_lock); if (flushall) cpu_tlb_flushID(); PMAP_UNLOCK(pm); } /* * pmap_zero_page() * * Zero a given physical page by mapping it at a page hook point. * In doing the zero page op, the page we zero is mapped cachable, as with * StrongARM accesses to non-cached pages are non-burst making writing * _any_ bulk data very slow. */ #if ARM_MMU_GENERIC != 0 || defined(CPU_XSCALE_CORE3) void pmap_zero_page_generic(vm_paddr_t phys, int off, int size) { if (_arm_bzero && size >= _min_bzero_size && _arm_bzero((void *)(phys + off), size, IS_PHYSICAL) == 0) return; mtx_lock(&cmtx); /* * Hook in the page, zero it, invalidate the TLB as needed. * * Note the temporary zero-page mapping must be a non-cached page in * order to work without corruption when write-allocate is enabled. */ *cdst_pte = L2_S_PROTO | phys | L2_S_PROT(PTE_KERNEL, VM_PROT_WRITE); PTE_SYNC(cdst_pte); cpu_tlb_flushD_SE(cdstp); cpu_cpwait(); if (off || size != PAGE_SIZE) bzero((void *)(cdstp + off), size); else bzero_page(cdstp); mtx_unlock(&cmtx); } #endif /* ARM_MMU_GENERIC != 0 */ #if ARM_MMU_XSCALE == 1 void pmap_zero_page_xscale(vm_paddr_t phys, int off, int size) { if (_arm_bzero && size >= _min_bzero_size && _arm_bzero((void *)(phys + off), size, IS_PHYSICAL) == 0) return; mtx_lock(&cmtx); /* * Hook in the page, zero it, and purge the cache for that * zeroed page. Invalidate the TLB as needed. */ *cdst_pte = L2_S_PROTO | phys | L2_S_PROT(PTE_KERNEL, VM_PROT_WRITE) | L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X); /* mini-data */ PTE_SYNC(cdst_pte); cpu_tlb_flushD_SE(cdstp); cpu_cpwait(); if (off || size != PAGE_SIZE) bzero((void *)(cdstp + off), size); else bzero_page(cdstp); mtx_unlock(&cmtx); xscale_cache_clean_minidata(); } /* * Change the PTEs for the specified kernel mappings such that they * will use the mini data cache instead of the main data cache. */ void pmap_use_minicache(vm_offset_t va, vm_size_t size) { struct l2_bucket *l2b; pt_entry_t *ptep, *sptep, pte; vm_offset_t next_bucket, eva; #if (ARM_NMMUS > 1) || defined(CPU_XSCALE_CORE3) if (xscale_use_minidata == 0) return; #endif eva = va + size; while (va < eva) { next_bucket = L2_NEXT_BUCKET(va); if (next_bucket > eva) next_bucket = eva; - l2b = pmap_get_l2_bucket(pmap_kernel(), va); + l2b = pmap_get_l2_bucket(kernel_pmap, va); sptep = ptep = &l2b->l2b_kva[l2pte_index(va)]; while (va < next_bucket) { pte = *ptep; if (!l2pte_minidata(pte)) { cpu_dcache_wbinv_range(va, PAGE_SIZE); cpu_tlb_flushD_SE(va); *ptep = pte & ~L2_B; } ptep++; va += PAGE_SIZE; } PTE_SYNC_RANGE(sptep, (u_int)(ptep - sptep)); } cpu_cpwait(); } #endif /* ARM_MMU_XSCALE == 1 */ /* * pmap_zero_page zeros the specified hardware page by mapping * the page into KVM and using bzero to clear its contents. */ void pmap_zero_page(vm_page_t m) { pmap_zero_page_func(VM_PAGE_TO_PHYS(m), 0, PAGE_SIZE); } /* * pmap_zero_page_area zeros the specified hardware page by mapping * the page into KVM and using bzero to clear its contents. * * off and size may not cover an area beyond a single hardware page. */ void pmap_zero_page_area(vm_page_t m, int off, int size) { pmap_zero_page_func(VM_PAGE_TO_PHYS(m), off, size); } /* * pmap_zero_page_idle zeros the specified hardware page by mapping * the page into KVM and using bzero to clear its contents. This * is intended to be called from the vm_pagezero process only and * outside of Giant. */ void pmap_zero_page_idle(vm_page_t m) { pmap_zero_page(m); } #if 0 /* * pmap_clean_page() * * This is a local function used to work out the best strategy to clean * a single page referenced by its entry in the PV table. It should be used by * pmap_copy_page, pmap_zero page and maybe some others later on. * * Its policy is effectively: * o If there are no mappings, we don't bother doing anything with the cache. * o If there is one mapping, we clean just that page. * o If there are multiple mappings, we clean the entire cache. * * So that some functions can be further optimised, it returns 0 if it didn't * clean the entire cache, or 1 if it did. * * XXX One bug in this routine is that if the pv_entry has a single page * mapped at 0x00000000 a whole cache clean will be performed rather than * just the 1 page. Since this should not occur in everyday use and if it does * it will just result in not the most efficient clean for the page. * * We don't yet use this function but may want to. */ static int pmap_clean_page(struct pv_entry *pv, boolean_t is_src) { pmap_t pm, pm_to_clean = NULL; struct pv_entry *npv; u_int cache_needs_cleaning = 0; u_int flags = 0; vm_offset_t page_to_clean = 0; if (pv == NULL) { /* nothing mapped in so nothing to flush */ return (0); } /* * Since we flush the cache each time we change to a different * user vmspace, we only need to flush the page if it is in the * current pmap. */ if (curthread) pm = vmspace_pmap(curproc->p_vmspace); else - pm = pmap_kernel(); + pm = kernel_pmap; for (npv = pv; npv; npv = TAILQ_NEXT(npv, pv_list)) { - if (npv->pv_pmap == pmap_kernel() || npv->pv_pmap == pm) { + if (npv->pv_pmap == kernel_pmap || npv->pv_pmap == pm) { flags |= npv->pv_flags; /* * The page is mapped non-cacheable in * this map. No need to flush the cache. */ if (npv->pv_flags & PVF_NC) { #ifdef DIAGNOSTIC if (cache_needs_cleaning) panic("pmap_clean_page: " "cache inconsistency"); #endif break; } else if (is_src && (npv->pv_flags & PVF_WRITE) == 0) continue; if (cache_needs_cleaning) { page_to_clean = 0; break; } else { page_to_clean = npv->pv_va; pm_to_clean = npv->pv_pmap; } cache_needs_cleaning = 1; } } if (page_to_clean) { if (PV_BEEN_EXECD(flags)) pmap_idcache_wbinv_range(pm_to_clean, page_to_clean, PAGE_SIZE); else pmap_dcache_wb_range(pm_to_clean, page_to_clean, PAGE_SIZE, !is_src, (flags & PVF_WRITE) == 0); } else if (cache_needs_cleaning) { if (PV_BEEN_EXECD(flags)) pmap_idcache_wbinv_all(pm); else pmap_dcache_wbinv_all(pm); return (1); } return (0); } #endif /* * pmap_copy_page copies the specified (machine independent) * page by mapping the page into virtual memory and using * bcopy to copy the page, one machine dependent page at a * time. */ /* * pmap_copy_page() * * Copy one physical page into another, by mapping the pages into * hook points. The same comment regarding cachability as in * pmap_zero_page also applies here. */ #if ARM_MMU_GENERIC != 0 || defined (CPU_XSCALE_CORE3) void pmap_copy_page_generic(vm_paddr_t src, vm_paddr_t dst) { #if 0 struct vm_page *src_pg = PHYS_TO_VM_PAGE(src); #endif /* * Clean the source page. Hold the source page's lock for * the duration of the copy so that no other mappings can * be created while we have a potentially aliased mapping. */ #if 0 /* * XXX: Not needed while we call cpu_dcache_wbinv_all() in * pmap_copy_page(). */ (void) pmap_clean_page(TAILQ_FIRST(&src_pg->md.pv_list), TRUE); #endif /* * Map the pages into the page hook points, copy them, and purge * the cache for the appropriate page. Invalidate the TLB * as required. */ mtx_lock(&cmtx); *csrc_pte = L2_S_PROTO | src | L2_S_PROT(PTE_KERNEL, VM_PROT_READ) | pte_l2_s_cache_mode; PTE_SYNC(csrc_pte); *cdst_pte = L2_S_PROTO | dst | L2_S_PROT(PTE_KERNEL, VM_PROT_WRITE) | pte_l2_s_cache_mode; PTE_SYNC(cdst_pte); cpu_tlb_flushD_SE(csrcp); cpu_tlb_flushD_SE(cdstp); cpu_cpwait(); bcopy_page(csrcp, cdstp); mtx_unlock(&cmtx); cpu_dcache_inv_range(csrcp, PAGE_SIZE); cpu_dcache_wbinv_range(cdstp, PAGE_SIZE); cpu_l2cache_inv_range(csrcp, PAGE_SIZE); cpu_l2cache_wbinv_range(cdstp, PAGE_SIZE); } void pmap_copy_page_offs_generic(vm_paddr_t a_phys, vm_offset_t a_offs, vm_paddr_t b_phys, vm_offset_t b_offs, int cnt) { mtx_lock(&cmtx); *csrc_pte = L2_S_PROTO | a_phys | L2_S_PROT(PTE_KERNEL, VM_PROT_READ) | pte_l2_s_cache_mode; PTE_SYNC(csrc_pte); *cdst_pte = L2_S_PROTO | b_phys | L2_S_PROT(PTE_KERNEL, VM_PROT_WRITE) | pte_l2_s_cache_mode; PTE_SYNC(cdst_pte); cpu_tlb_flushD_SE(csrcp); cpu_tlb_flushD_SE(cdstp); cpu_cpwait(); bcopy((char *)csrcp + a_offs, (char *)cdstp + b_offs, cnt); mtx_unlock(&cmtx); cpu_dcache_inv_range(csrcp + a_offs, cnt); cpu_dcache_wbinv_range(cdstp + b_offs, cnt); cpu_l2cache_inv_range(csrcp + a_offs, cnt); cpu_l2cache_wbinv_range(cdstp + b_offs, cnt); } #endif /* ARM_MMU_GENERIC != 0 */ #if ARM_MMU_XSCALE == 1 void pmap_copy_page_xscale(vm_paddr_t src, vm_paddr_t dst) { #if 0 /* XXX: Only needed for pmap_clean_page(), which is commented out. */ struct vm_page *src_pg = PHYS_TO_VM_PAGE(src); #endif /* * Clean the source page. Hold the source page's lock for * the duration of the copy so that no other mappings can * be created while we have a potentially aliased mapping. */ #if 0 /* * XXX: Not needed while we call cpu_dcache_wbinv_all() in * pmap_copy_page(). */ (void) pmap_clean_page(TAILQ_FIRST(&src_pg->md.pv_list), TRUE); #endif /* * Map the pages into the page hook points, copy them, and purge * the cache for the appropriate page. Invalidate the TLB * as required. */ mtx_lock(&cmtx); *csrc_pte = L2_S_PROTO | src | L2_S_PROT(PTE_KERNEL, VM_PROT_READ) | L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X); /* mini-data */ PTE_SYNC(csrc_pte); *cdst_pte = L2_S_PROTO | dst | L2_S_PROT(PTE_KERNEL, VM_PROT_WRITE) | L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X); /* mini-data */ PTE_SYNC(cdst_pte); cpu_tlb_flushD_SE(csrcp); cpu_tlb_flushD_SE(cdstp); cpu_cpwait(); bcopy_page(csrcp, cdstp); mtx_unlock(&cmtx); xscale_cache_clean_minidata(); } void pmap_copy_page_offs_xscale(vm_paddr_t a_phys, vm_offset_t a_offs, vm_paddr_t b_phys, vm_offset_t b_offs, int cnt) { mtx_lock(&cmtx); *csrc_pte = L2_S_PROTO | a_phys | L2_S_PROT(PTE_KERNEL, VM_PROT_READ) | L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X); PTE_SYNC(csrc_pte); *cdst_pte = L2_S_PROTO | b_phys | L2_S_PROT(PTE_KERNEL, VM_PROT_WRITE) | L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X); PTE_SYNC(cdst_pte); cpu_tlb_flushD_SE(csrcp); cpu_tlb_flushD_SE(cdstp); cpu_cpwait(); bcopy((char *)csrcp + a_offs, (char *)cdstp + b_offs, cnt); mtx_unlock(&cmtx); xscale_cache_clean_minidata(); } #endif /* ARM_MMU_XSCALE == 1 */ void pmap_copy_page(vm_page_t src, vm_page_t dst) { cpu_dcache_wbinv_all(); cpu_l2cache_wbinv_all(); if (_arm_memcpy && PAGE_SIZE >= _min_memcpy_size && _arm_memcpy((void *)VM_PAGE_TO_PHYS(dst), (void *)VM_PAGE_TO_PHYS(src), PAGE_SIZE, IS_PHYSICAL) == 0) return; pmap_copy_page_func(VM_PAGE_TO_PHYS(src), VM_PAGE_TO_PHYS(dst)); } /* * We have code to do unmapped I/O. However, it isn't quite right and * causes un-page-aligned I/O to devices to fail (most notably newfs * or fsck). We give up a little performance to not allow unmapped I/O * to gain stability. */ int unmapped_buf_allowed = 0; void pmap_copy_pages(vm_page_t ma[], vm_offset_t a_offset, vm_page_t mb[], vm_offset_t b_offset, int xfersize) { vm_page_t a_pg, b_pg; vm_offset_t a_pg_offset, b_pg_offset; int cnt; cpu_dcache_wbinv_all(); cpu_l2cache_wbinv_all(); while (xfersize > 0) { a_pg = ma[a_offset >> PAGE_SHIFT]; a_pg_offset = a_offset & PAGE_MASK; cnt = min(xfersize, PAGE_SIZE - a_pg_offset); b_pg = mb[b_offset >> PAGE_SHIFT]; b_pg_offset = b_offset & PAGE_MASK; cnt = min(cnt, PAGE_SIZE - b_pg_offset); pmap_copy_page_offs_func(VM_PAGE_TO_PHYS(a_pg), a_pg_offset, VM_PAGE_TO_PHYS(b_pg), b_pg_offset, cnt); xfersize -= cnt; a_offset += cnt; b_offset += cnt; } } vm_offset_t pmap_quick_enter_page(vm_page_t m) { /* * Don't bother with a PCPU pageframe, since we don't support * SMP for anything pre-armv7. Use pmap_kenter() to ensure * caching is handled correctly for multiple mappings of the * same physical page. */ mtx_assert(&qmap_mtx, MA_NOTOWNED); mtx_lock(&qmap_mtx); pmap_kenter(qmap_addr, VM_PAGE_TO_PHYS(m)); return (qmap_addr); } void pmap_quick_remove_page(vm_offset_t addr) { KASSERT(addr == qmap_addr, ("pmap_quick_remove_page: invalid address")); mtx_assert(&qmap_mtx, MA_OWNED); pmap_kremove(addr); mtx_unlock(&qmap_mtx); } /* * this routine returns true if a physical page resides * in the given pmap. */ boolean_t pmap_page_exists_quick(pmap_t pmap, vm_page_t m) { pv_entry_t pv; int loops = 0; boolean_t rv; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_page_exists_quick: page %p is not managed", m)); rv = FALSE; rw_wlock(&pvh_global_lock); TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) { if (pv->pv_pmap == pmap) { rv = TRUE; break; } loops++; if (loops >= 16) break; } rw_wunlock(&pvh_global_lock); return (rv); } /* * pmap_page_wired_mappings: * * Return the number of managed mappings to the given physical page * that are wired. */ int pmap_page_wired_mappings(vm_page_t m) { pv_entry_t pv; int count; count = 0; if ((m->oflags & VPO_UNMANAGED) != 0) return (count); rw_wlock(&pvh_global_lock); TAILQ_FOREACH(pv, &m->md.pv_list, pv_list) if ((pv->pv_flags & PVF_WIRED) != 0) count++; rw_wunlock(&pvh_global_lock); return (count); } /* * This function is advisory. */ void pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice) { } /* * pmap_ts_referenced: * * Return the count of reference bits for a page, clearing all of them. */ int pmap_ts_referenced(vm_page_t m) { KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_ts_referenced: page %p is not managed", m)); return (pmap_clearbit(m, PVF_REF)); } boolean_t pmap_is_modified(vm_page_t m) { KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_is_modified: page %p is not managed", m)); if (m->md.pvh_attrs & PVF_MOD) return (TRUE); return(FALSE); } /* * Clear the modify bits on the specified physical page. */ void pmap_clear_modify(vm_page_t m) { KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_clear_modify: page %p is not managed", m)); VM_OBJECT_ASSERT_WLOCKED(m->object); KASSERT(!vm_page_xbusied(m), ("pmap_clear_modify: page %p is exclusive busied", m)); /* * If the page is not PGA_WRITEABLE, then no mappings can be modified. * If the object containing the page is locked and the page is not * exclusive busied, then PGA_WRITEABLE cannot be concurrently set. */ if ((m->aflags & PGA_WRITEABLE) == 0) return; if (m->md.pvh_attrs & PVF_MOD) pmap_clearbit(m, PVF_MOD); } /* * pmap_is_referenced: * * Return whether or not the specified physical page was referenced * in any physical maps. */ boolean_t pmap_is_referenced(vm_page_t m) { KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_is_referenced: page %p is not managed", m)); return ((m->md.pvh_attrs & PVF_REF) != 0); } /* * Clear the write and modified bits in each of the given page's mappings. */ void pmap_remove_write(vm_page_t m) { KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_remove_write: page %p is not managed", m)); /* * If the page is not exclusive busied, then PGA_WRITEABLE cannot be * set by another thread while the object is locked. Thus, * if PGA_WRITEABLE is clear, no page table entries need updating. */ VM_OBJECT_ASSERT_WLOCKED(m->object); if (vm_page_xbusied(m) || (m->aflags & PGA_WRITEABLE) != 0) pmap_clearbit(m, PVF_WRITE); } /* * perform the pmap work for mincore */ int pmap_mincore(pmap_t pmap, vm_offset_t addr, vm_paddr_t *locked_pa) { struct l2_bucket *l2b; pt_entry_t *ptep, pte; vm_paddr_t pa; vm_page_t m; int val; boolean_t managed; PMAP_LOCK(pmap); retry: l2b = pmap_get_l2_bucket(pmap, addr); if (l2b == NULL) { val = 0; goto out; } ptep = &l2b->l2b_kva[l2pte_index(addr)]; pte = *ptep; if (!l2pte_valid(pte)) { val = 0; goto out; } val = MINCORE_INCORE; if (pte & L2_S_PROT_W) val |= MINCORE_MODIFIED | MINCORE_MODIFIED_OTHER; managed = false; pa = l2pte_pa(pte); m = PHYS_TO_VM_PAGE(pa); if (m != NULL && !(m->oflags & VPO_UNMANAGED)) managed = true; if (managed) { /* * The ARM pmap tries to maintain a per-mapping * reference bit. The trouble is that it's kept in * the PV entry, not the PTE, so it's costly to access * here. You would need to acquire the pvh global * lock, call pmap_find_pv(), and introduce a custom * version of vm_page_pa_tryrelock() that releases and * reacquires the pvh global lock. In the end, I * doubt it's worthwhile. This may falsely report * the given address as referenced. */ if ((m->md.pvh_attrs & PVF_REF) != 0) val |= MINCORE_REFERENCED | MINCORE_REFERENCED_OTHER; } if ((val & (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER)) != (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER) && managed) { /* Ensure that "PHYS_TO_VM_PAGE(pa)->object" doesn't change. */ if (vm_page_pa_tryrelock(pmap, pa, locked_pa)) goto retry; } else out: PA_UNLOCK_COND(*locked_pa); PMAP_UNLOCK(pmap); return (val); } void pmap_sync_icache(pmap_t pm, vm_offset_t va, vm_size_t sz) { } /* * Increase the starting virtual address of the given mapping if a * different alignment might result in more superpage mappings. */ void pmap_align_superpage(vm_object_t object, vm_ooffset_t offset, vm_offset_t *addr, vm_size_t size) { } #define BOOTSTRAP_DEBUG /* * pmap_map_section: * * Create a single section mapping. */ void pmap_map_section(vm_offset_t l1pt, vm_offset_t va, vm_offset_t pa, int prot, int cache) { pd_entry_t *pde = (pd_entry_t *) l1pt; pd_entry_t fl; KASSERT(((va | pa) & L1_S_OFFSET) == 0, ("ouin2")); switch (cache) { case PTE_NOCACHE: default: fl = 0; break; case PTE_CACHE: fl = pte_l1_s_cache_mode; break; case PTE_PAGETABLE: fl = pte_l1_s_cache_mode_pt; break; } pde[va >> L1_S_SHIFT] = L1_S_PROTO | pa | L1_S_PROT(PTE_KERNEL, prot) | fl | L1_S_DOM(PMAP_DOMAIN_KERNEL); PTE_SYNC(&pde[va >> L1_S_SHIFT]); } /* * pmap_link_l2pt: * * Link the L2 page table specified by l2pv.pv_pa into the L1 * page table at the slot for "va". */ void pmap_link_l2pt(vm_offset_t l1pt, vm_offset_t va, struct pv_addr *l2pv) { pd_entry_t *pde = (pd_entry_t *) l1pt, proto; u_int slot = va >> L1_S_SHIFT; proto = L1_S_DOM(PMAP_DOMAIN_KERNEL) | L1_C_PROTO; #ifdef VERBOSE_INIT_ARM printf("pmap_link_l2pt: pa=0x%x va=0x%x\n", l2pv->pv_pa, l2pv->pv_va); #endif pde[slot + 0] = proto | (l2pv->pv_pa + 0x000); PTE_SYNC(&pde[slot]); SLIST_INSERT_HEAD(&kernel_pt_list, l2pv, pv_list); } /* * pmap_map_entry * * Create a single page mapping. */ void pmap_map_entry(vm_offset_t l1pt, vm_offset_t va, vm_offset_t pa, int prot, int cache) { pd_entry_t *pde = (pd_entry_t *) l1pt; pt_entry_t fl; pt_entry_t *pte; KASSERT(((va | pa) & PAGE_MASK) == 0, ("ouin")); switch (cache) { case PTE_NOCACHE: default: fl = 0; break; case PTE_CACHE: fl = pte_l2_s_cache_mode; break; case PTE_PAGETABLE: fl = pte_l2_s_cache_mode_pt; break; } if ((pde[va >> L1_S_SHIFT] & L1_TYPE_MASK) != L1_TYPE_C) panic("pmap_map_entry: no L2 table for VA 0x%08x", va); pte = (pt_entry_t *) kernel_pt_lookup(pde[L1_IDX(va)] & L1_C_ADDR_MASK); if (pte == NULL) panic("pmap_map_entry: can't find L2 table for VA 0x%08x", va); pte[l2pte_index(va)] = L2_S_PROTO | pa | L2_S_PROT(PTE_KERNEL, prot) | fl; PTE_SYNC(&pte[l2pte_index(va)]); } /* * pmap_map_chunk: * * Map a chunk of memory using the most efficient mappings * possible (section. large page, small page) into the * provided L1 and L2 tables at the specified virtual address. */ vm_size_t pmap_map_chunk(vm_offset_t l1pt, vm_offset_t va, vm_offset_t pa, vm_size_t size, int prot, int cache) { pd_entry_t *pde = (pd_entry_t *) l1pt; pt_entry_t *pte, f1, f2s, f2l; vm_size_t resid; int i; resid = (size + (PAGE_SIZE - 1)) & ~(PAGE_SIZE - 1); if (l1pt == 0) panic("pmap_map_chunk: no L1 table provided"); #ifdef VERBOSE_INIT_ARM printf("pmap_map_chunk: pa=0x%x va=0x%x size=0x%x resid=0x%x " "prot=0x%x cache=%d\n", pa, va, size, resid, prot, cache); #endif switch (cache) { case PTE_NOCACHE: default: f1 = 0; f2l = 0; f2s = 0; break; case PTE_CACHE: f1 = pte_l1_s_cache_mode; f2l = pte_l2_l_cache_mode; f2s = pte_l2_s_cache_mode; break; case PTE_PAGETABLE: f1 = pte_l1_s_cache_mode_pt; f2l = pte_l2_l_cache_mode_pt; f2s = pte_l2_s_cache_mode_pt; break; } size = resid; while (resid > 0) { /* See if we can use a section mapping. */ if (L1_S_MAPPABLE_P(va, pa, resid)) { #ifdef VERBOSE_INIT_ARM printf("S"); #endif pde[va >> L1_S_SHIFT] = L1_S_PROTO | pa | L1_S_PROT(PTE_KERNEL, prot) | f1 | L1_S_DOM(PMAP_DOMAIN_KERNEL); PTE_SYNC(&pde[va >> L1_S_SHIFT]); va += L1_S_SIZE; pa += L1_S_SIZE; resid -= L1_S_SIZE; continue; } /* * Ok, we're going to use an L2 table. Make sure * one is actually in the corresponding L1 slot * for the current VA. */ if ((pde[va >> L1_S_SHIFT] & L1_TYPE_MASK) != L1_TYPE_C) panic("pmap_map_chunk: no L2 table for VA 0x%08x", va); pte = (pt_entry_t *) kernel_pt_lookup( pde[L1_IDX(va)] & L1_C_ADDR_MASK); if (pte == NULL) panic("pmap_map_chunk: can't find L2 table for VA" "0x%08x", va); /* See if we can use a L2 large page mapping. */ if (L2_L_MAPPABLE_P(va, pa, resid)) { #ifdef VERBOSE_INIT_ARM printf("L"); #endif for (i = 0; i < 16; i++) { pte[l2pte_index(va) + i] = L2_L_PROTO | pa | L2_L_PROT(PTE_KERNEL, prot) | f2l; PTE_SYNC(&pte[l2pte_index(va) + i]); } va += L2_L_SIZE; pa += L2_L_SIZE; resid -= L2_L_SIZE; continue; } /* Use a small page mapping. */ #ifdef VERBOSE_INIT_ARM printf("P"); #endif pte[l2pte_index(va)] = L2_S_PROTO | pa | L2_S_PROT(PTE_KERNEL, prot) | f2s; PTE_SYNC(&pte[l2pte_index(va)]); va += PAGE_SIZE; pa += PAGE_SIZE; resid -= PAGE_SIZE; } #ifdef VERBOSE_INIT_ARM printf("\n"); #endif return (size); } void pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma) { /* * Remember the memattr in a field that gets used to set the appropriate * bits in the PTEs as mappings are established. */ m->md.pv_memattr = ma; /* * It appears that this function can only be called before any mappings * for the page are established on ARM. If this ever changes, this code * will need to walk the pv_list and make each of the existing mappings * uncacheable, being careful to sync caches and PTEs (and maybe * invalidate TLB?) for any current mapping it modifies. */ if (m->md.pv_kva != 0 || TAILQ_FIRST(&m->md.pv_list) != NULL) panic("Can't change memattr on page with existing mappings"); } Index: user/ngie/make_check/sys/arm/arm/swtch-v4.S =================================================================== --- user/ngie/make_check/sys/arm/arm/swtch-v4.S (nonexistent) +++ user/ngie/make_check/sys/arm/arm/swtch-v4.S (revision 295078) @@ -0,0 +1,381 @@ +/* $NetBSD: cpuswitch.S,v 1.41 2003/11/15 08:44:18 scw Exp $ */ + +/*- + * Copyright 2003 Wasabi Systems, Inc. + * All rights reserved. + * + * Written by Steve C. Woodford for Wasabi Systems, Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed for the NetBSD Project by + * Wasabi Systems, Inc. + * 4. The name of Wasabi Systems, Inc. may not be used to endorse + * or promote products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY WASABI SYSTEMS, INC. ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASABI SYSTEMS, INC + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ +/*- + * Copyright (c) 1994-1998 Mark Brinicombe. + * Copyright (c) 1994 Brini. + * All rights reserved. + * + * This code is derived from software written for Brini by Mark Brinicombe + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by Brini. + * 4. The name of the company nor the name of the author may be used to + * endorse or promote products derived from this software without specific + * prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY BRINI ``AS IS'' AND ANY EXPRESS OR IMPLIED + * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + * IN NO EVENT SHALL BRINI OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * RiscBSD kernel project + * + * cpuswitch.S + * + * cpu switching functions + * + * Created : 15/10/94 + * + */ + +#include "assym.s" +#include "opt_sched.h" + +#include +#include +#include +#include +#include + +__FBSDID("$FreeBSD$"); + + +#define GET_PCPU(tmp, tmp2) \ + ldr tmp, .Lcurpcpu + +#ifdef VFP + .fpu vfp /* allow VFP instructions */ +#endif + +.Lcurpcpu: + .word _C_LABEL(__pcpu) + .word PCPU_SIZE +.Lblocked_lock: + .word _C_LABEL(blocked_lock) + + +#define DOMAIN_CLIENT 0x01 + +.Lcpufuncs: + .word _C_LABEL(cpufuncs) + +/* + * cpu_throw(oldtd, newtd) + * + * Remove current thread state, then select the next thread to run + * and load its state. + * r0 = oldtd + * r1 = newtd + */ +ENTRY(cpu_throw) + mov r5, r1 + + /* + * r0 = oldtd + * r5 = newtd + */ + +#ifdef VFP /* This thread is dying, disable */ + bl _C_LABEL(vfp_discard) /* VFP without preserving state. */ +#endif + + GET_PCPU(r7, r9) + ldr r7, [r5, #(TD_PCB)] /* r7 = new thread's PCB */ + + /* Switch to lwp0 context */ + + ldr r9, .Lcpufuncs + mov lr, pc + ldr pc, [r9, #CF_IDCACHE_WBINV_ALL] + ldr r0, [r7, #(PCB_PL1VEC)] + ldr r1, [r7, #(PCB_DACR)] + /* + * r0 = Pointer to L1 slot for vector_page (or NULL) + * r1 = lwp0's DACR + * r5 = lwp0 + * r7 = lwp0's PCB + * r9 = cpufuncs + */ + + /* + * Ensure the vector table is accessible by fixing up lwp0's L1 + */ + cmp r0, #0 /* No need to fixup vector table? */ + ldrne r3, [r0] /* But if yes, fetch current value */ + ldrne r2, [r7, #(PCB_L1VEC)] /* Fetch new vector_page value */ + mcr p15, 0, r1, c3, c0, 0 /* Update DACR for lwp0's context */ + cmpne r3, r2 /* Stuffing the same value? */ + strne r2, [r0] /* Store if not. */ + +#ifdef PMAP_INCLUDE_PTE_SYNC + /* + * Need to sync the cache to make sure that last store is + * visible to the MMU. + */ + movne r1, #4 + movne lr, pc + ldrne pc, [r9, #CF_DCACHE_WB_RANGE] +#endif /* PMAP_INCLUDE_PTE_SYNC */ + + /* + * Note: We don't do the same optimisation as cpu_switch() with + * respect to avoiding flushing the TLB if we're switching to + * the same L1 since this process' VM space may be about to go + * away, so we don't want *any* turds left in the TLB. + */ + + /* Switch the memory to the new process */ + ldr r0, [r7, #(PCB_PAGEDIR)] + mov lr, pc + ldr pc, [r9, #CF_CONTEXT_SWITCH] + + GET_PCPU(r6, r4) + /* Hook in a new pcb */ + str r7, [r6, #PC_CURPCB] + /* We have a new curthread now so make a note it */ + str r5, [r6, #PC_CURTHREAD] + + /* Set the new tp */ + ldr r6, [r5, #(TD_MD + MD_TP)] + ldr r4, =ARM_TP_ADDRESS + str r6, [r4] + ldr r6, [r5, #(TD_MD + MD_RAS_START)] + str r6, [r4, #4] /* ARM_RAS_START */ + ldr r6, [r5, #(TD_MD + MD_RAS_END)] + str r6, [r4, #8] /* ARM_RAS_END */ + + /* Restore all the saved registers and exit */ + add r3, r7, #PCB_R4 + ldmia r3, {r4-r12, sp, pc} +END(cpu_throw) + +/* + * cpu_switch(oldtd, newtd, lock) + * + * Save the current thread state, then select the next thread to run + * and load its state. + * r0 = oldtd + * r1 = newtd + * r2 = lock (new lock for old thread) + */ +ENTRY(cpu_switch) + /* Interrupts are disabled. */ + /* Save all the registers in the old thread's pcb. */ + ldr r3, [r0, #(TD_PCB)] + + /* Restore all the saved registers and exit */ + add r3, #(PCB_R4) + stmia r3, {r4-r12, sp, lr, pc} + + mov r6, r2 /* Save the mutex */ + + /* rem: r0 = old lwp */ + /* rem: interrupts are disabled */ + + /* Process is now on a processor. */ + /* We have a new curthread now so make a note it */ + GET_PCPU(r7, r2) + str r1, [r7, #PC_CURTHREAD] + + /* Hook in a new pcb */ + ldr r2, [r1, #TD_PCB] + str r2, [r7, #PC_CURPCB] + + /* Stage two : Save old context */ + + /* Get the user structure for the old thread. */ + ldr r2, [r0, #(TD_PCB)] + mov r4, r0 /* Save the old thread. */ + + /* Store the old tp; userland can change it on armv4. */ + ldr r3, =ARM_TP_ADDRESS + ldr r9, [r3] + str r9, [r0, #(TD_MD + MD_TP)] + ldr r9, [r3, #4] + str r9, [r0, #(TD_MD + MD_RAS_START)] + ldr r9, [r3, #8] + str r9, [r0, #(TD_MD + MD_RAS_END)] + + /* Set the new tp */ + ldr r9, [r1, #(TD_MD + MD_TP)] + str r9, [r3] + ldr r9, [r1, #(TD_MD + MD_RAS_START)] + str r9, [r3, #4] + ldr r9, [r1, #(TD_MD + MD_RAS_END)] + str r9, [r3, #8] + + /* Get the user structure for the new process in r9 */ + ldr r9, [r1, #(TD_PCB)] + + /* rem: r2 = old PCB */ + /* rem: r9 = new PCB */ + /* rem: interrupts are enabled */ + +#ifdef VFP + fmrx r0, fpexc /* If the VFP is enabled */ + tst r0, #(VFPEXC_EN) /* the current thread has */ + movne r1, #1 /* used it, so go save */ + addne r0, r2, #(PCB_VFPSTATE) /* the state into the PCB */ + blne _C_LABEL(vfp_store) /* and disable the VFP. */ +#endif + + /* r0-r3 now free! */ + + /* Third phase : restore saved context */ + + /* rem: r2 = old PCB */ + /* rem: r9 = new PCB */ + + ldr r5, [r9, #(PCB_DACR)] /* r5 = new DACR */ + mov r2, #DOMAIN_CLIENT + cmp r5, r2, lsl #(PMAP_DOMAIN_KERNEL * 2) /* Sw to kernel thread? */ + beq .Lcs_context_switched /* Yup. Don't flush cache */ + mrc p15, 0, r0, c3, c0, 0 /* r0 = old DACR */ + /* + * Get the new L1 table pointer into r11. If we're switching to + * an LWP with the same address space as the outgoing one, we can + * skip the cache purge and the TTB load. + * + * To avoid data dep stalls that would happen anyway, we try + * and get some useful work done in the mean time. + */ + mrc p15, 0, r10, c2, c0, 0 /* r10 = old L1 */ + ldr r11, [r9, #(PCB_PAGEDIR)] /* r11 = new L1 */ + + teq r10, r11 /* Same L1? */ + cmpeq r0, r5 /* Same DACR? */ + beq .Lcs_context_switched /* yes! */ + + /* + * Definately need to flush the cache. + */ + + ldr r1, .Lcpufuncs + mov lr, pc + ldr pc, [r1, #CF_IDCACHE_WBINV_ALL] + +.Lcs_cache_purge_skipped: + /* rem: r6 = lock */ + /* rem: r9 = new PCB */ + /* rem: r10 = old L1 */ + /* rem: r11 = new L1 */ + + mov r2, #0x00000000 + ldr r7, [r9, #(PCB_PL1VEC)] + + /* + * Ensure the vector table is accessible by fixing up the L1 + */ + cmp r7, #0 /* No need to fixup vector table? */ + ldrne r2, [r7] /* But if yes, fetch current value */ + ldrne r0, [r9, #(PCB_L1VEC)] /* Fetch new vector_page value */ + mcr p15, 0, r5, c3, c0, 0 /* Update DACR for new context */ + cmpne r2, r0 /* Stuffing the same value? */ +#ifndef PMAP_INCLUDE_PTE_SYNC + strne r0, [r7] /* Nope, update it */ +#else + beq .Lcs_same_vector + str r0, [r7] /* Otherwise, update it */ + + /* + * Need to sync the cache to make sure that last store is + * visible to the MMU. + */ + ldr r2, .Lcpufuncs + mov r0, r7 + mov r1, #4 + mov lr, pc + ldr pc, [r2, #CF_DCACHE_WB_RANGE] + +.Lcs_same_vector: +#endif /* PMAP_INCLUDE_PTE_SYNC */ + + cmp r10, r11 /* Switching to the same L1? */ + ldr r10, .Lcpufuncs + beq .Lcs_same_l1 /* Yup. */ + /* + * Do a full context switch, including full TLB flush. + */ + mov r0, r11 + mov lr, pc + ldr pc, [r10, #CF_CONTEXT_SWITCH] + + b .Lcs_context_switched + + /* + * We're switching to a different process in the same L1. + * In this situation, we only need to flush the TLB for the + * vector_page mapping, and even then only if r7 is non-NULL. + */ +.Lcs_same_l1: + cmp r7, #0 + movne r0, #0 /* We *know* vector_page's VA is 0x0 */ + movne lr, pc + ldrne pc, [r10, #CF_TLB_FLUSHID_SE] + +.Lcs_context_switched: + + /* Release the old thread */ + str r6, [r4, #TD_LOCK] + + /* XXXSCW: Safe to re-enable FIQs here */ + + /* rem: r9 = new PCB */ + + /* Restore all the saved registers and exit */ + add r3, r9, #PCB_R4 + ldmia r3, {r4-r12, sp, pc} +END(cpu_switch) + + + Property changes on: user/ngie/make_check/sys/arm/arm/swtch-v4.S ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/make_check/sys/arm/arm/swtch-v6.S =================================================================== --- user/ngie/make_check/sys/arm/arm/swtch-v6.S (nonexistent) +++ user/ngie/make_check/sys/arm/arm/swtch-v6.S (revision 295078) @@ -0,0 +1,484 @@ +/* $NetBSD: cpuswitch.S,v 1.41 2003/11/15 08:44:18 scw Exp $ */ + +/*- + * Copyright 2003 Wasabi Systems, Inc. + * All rights reserved. + * + * Written by Steve C. Woodford for Wasabi Systems, Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed for the NetBSD Project by + * Wasabi Systems, Inc. + * 4. The name of Wasabi Systems, Inc. may not be used to endorse + * or promote products derived from this software without specific prior + * written permission. + * + * THIS SOFTWARE IS PROVIDED BY WASABI SYSTEMS, INC. ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASABI SYSTEMS, INC + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ +/*- + * Copyright (c) 1994-1998 Mark Brinicombe. + * Copyright (c) 1994 Brini. + * All rights reserved. + * + * This code is derived from software written for Brini by Mark Brinicombe + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by Brini. + * 4. The name of the company nor the name of the author may be used to + * endorse or promote products derived from this software without specific + * prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY BRINI ``AS IS'' AND ANY EXPRESS OR IMPLIED + * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + * IN NO EVENT SHALL BRINI OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, + * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES + * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR + * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * RiscBSD kernel project + * + * cpuswitch.S + * + * cpu switching functions + * + * Created : 15/10/94 + * + */ + +#include "assym.s" +#include "opt_sched.h" + +#include +#include +#include +#include +#include + +__FBSDID("$FreeBSD$"); + +#if __ARM_ARCH >= 6 && defined(SMP) +#define GET_PCPU(tmp, tmp2) \ + mrc p15, 0, tmp, c0, c0, 5; \ + and tmp, tmp, #0xf; \ + ldr tmp2, .Lcurpcpu+4; \ + mul tmp, tmp, tmp2; \ + ldr tmp2, .Lcurpcpu; \ + add tmp, tmp, tmp2; +#else + +#define GET_PCPU(tmp, tmp2) \ + ldr tmp, .Lcurpcpu +#endif + +#ifdef VFP + .fpu vfp /* allow VFP instructions */ +#endif + +.Lcurpcpu: + .word _C_LABEL(__pcpu) + .word PCPU_SIZE +.Lblocked_lock: + .word _C_LABEL(blocked_lock) + + +#include + +ENTRY(cpu_context_switch) /* QQQ: What about macro instead of function? */ + DSB + mcr CP15_TTBR0(r0) /* set the new TTB */ + ISB + mov r0, #(CPU_ASID_KERNEL) + mcr CP15_TLBIASID(r0) /* flush not global TLBs */ + /* + * Flush entire Branch Target Cache because of the branch predictor + * is not architecturally invisible. See ARM Architecture Reference + * Manual ARMv7-A and ARMv7-R edition, page B2-1264(65), Branch + * predictors and Requirements for branch predictor maintenance + * operations sections. + * + * QQQ: The predictor is virtually addressed and holds virtual target + * addresses. Therefore, if mapping is changed, the predictor cache + * must be flushed.The flush is part of entire i-cache invalidation + * what is always called when code mapping is changed. So herein, + * it's the only place where standalone predictor flush must be + * executed in kernel (except self modifying code case). + */ + mcr CP15_BPIALL /* and flush entire Branch Target Cache */ + DSB + mov pc, lr +END(cpu_context_switch) + +/* + * cpu_throw(oldtd, newtd) + * + * Remove current thread state, then select the next thread to run + * and load its state. + * r0 = oldtd + * r1 = newtd + */ +ENTRY(cpu_throw) + mov r10, r0 /* r10 = oldtd */ + mov r11, r1 /* r11 = newtd */ + +#ifdef VFP /* This thread is dying, disable */ + bl _C_LABEL(vfp_discard) /* VFP without preserving state. */ +#endif + GET_PCPU(r8, r9) /* r8 = current pcpu */ + ldr r4, [r8, #PC_CPUID] /* r4 = current cpu id */ + + cmp r10, #0 /* old thread? */ + beq 2f /* no, skip */ + + /* Remove this CPU from the active list. */ + ldr r5, [r8, #PC_CURPMAP] + mov r0, #(PM_ACTIVE) + add r5, r0 /* r5 = old pm_active */ + + /* Compute position and mask. */ +#if _NCPUWORDS > 1 + lsr r0, r4, #3 + bic r0, #3 + add r5, r0 /* r5 = position in old pm_active */ + mov r2, #1 + and r0, r4, #31 + lsl r2, r0 /* r2 = mask */ +#else + mov r2, #1 + lsl r2, r4 /* r2 = mask */ +#endif + /* Clear cpu from old active list. */ +#ifdef SMP +1: ldrex r0, [r5] + bic r0, r2 + strex r1, r0, [r5] + teq r1, #0 + bne 1b +#else + ldr r0, [r5] + bic r0, r2 + str r0, [r5] +#endif + +2: +#ifdef INVARIANTS + cmp r11, #0 /* new thread? */ + beq badsw1 /* no, panic */ +#endif + ldr r7, [r11, #(TD_PCB)] /* r7 = new PCB */ + + /* + * Registers at this point + * r4 = current cpu id + * r7 = new PCB + * r8 = current pcpu + * r11 = newtd + */ + + /* MMU switch to new thread. */ + ldr r0, [r7, #(PCB_PAGEDIR)] +#ifdef INVARIANTS + cmp r0, #0 /* new thread? */ + beq badsw4 /* no, panic */ +#endif + bl _C_LABEL(cpu_context_switch) + + /* + * Set new PMAP as current one. + * Insert cpu to new active list. + */ + + ldr r6, [r11, #(TD_PROC)] /* newtd->proc */ + ldr r6, [r6, #(P_VMSPACE)] /* newtd->proc->vmspace */ + add r6, #VM_PMAP /* newtd->proc->vmspace->pmap */ + str r6, [r8, #PC_CURPMAP] /* store to curpmap */ + + mov r0, #PM_ACTIVE + add r6, r0 /* r6 = new pm_active */ + + /* compute position and mask */ +#if _NCPUWORDS > 1 + lsr r0, r4, #3 + bic r0, #3 + add r6, r0 /* r6 = position in new pm_active */ + mov r2, #1 + and r0, r4, #31 + lsl r2, r0 /* r2 = mask */ +#else + mov r2, #1 + lsl r2, r4 /* r2 = mask */ +#endif + /* Set cpu to new active list. */ +#ifdef SMP +1: ldrex r0, [r6] + orr r0, r2 + strex r1, r0, [r6] + teq r1, #0 + bne 1b +#else + ldr r0, [r6] + orr r0, r2 + str r0, [r6] +#endif + /* + * Registers at this point. + * r7 = new PCB + * r8 = current pcpu + * r11 = newtd + * They must match the ones in sw1 position !!! + */ + DMB + b sw1 /* share new thread init with cpu_switch() */ +END(cpu_throw) + +/* + * cpu_switch(oldtd, newtd, lock) + * + * Save the current thread state, then select the next thread to run + * and load its state. + * r0 = oldtd + * r1 = newtd + * r2 = lock (new lock for old thread) + */ +ENTRY(cpu_switch) + /* Interrupts are disabled. */ +#ifdef INVARIANTS + cmp r0, #0 /* old thread? */ + beq badsw2 /* no, panic */ +#endif + /* Save all the registers in the old thread's pcb. */ + ldr r3, [r0, #(TD_PCB)] + add r3, #(PCB_R4) + stmia r3, {r4-r12, sp, lr, pc} + +#ifdef INVARIANTS + cmp r1, #0 /* new thread? */ + beq badsw3 /* no, panic */ +#endif + /* + * Save arguments. Note that we can now use r0-r14 until + * it is time to restore them for the new thread. However, + * some registers are not safe over function call. + */ + mov r9, r2 /* r9 = lock */ + mov r10, r0 /* r10 = oldtd */ + mov r11, r1 /* r11 = newtd */ + + GET_PCPU(r8, r3) /* r8 = current PCPU */ + ldr r7, [r11, #(TD_PCB)] /* r7 = newtd->td_pcb */ + + + +#ifdef VFP + ldr r3, [r10, #(TD_PCB)] + fmrx r0, fpexc /* If the VFP is enabled */ + tst r0, #(VFPEXC_EN) /* the current thread has */ + movne r1, #1 /* used it, so go save */ + addne r0, r3, #(PCB_VFPSTATE) /* the state into the PCB */ + blne _C_LABEL(vfp_store) /* and disable the VFP. */ +#endif + + /* + * MMU switch. If we're switching to a thread with the same + * address space as the outgoing one, we can skip the MMU switch. + */ + mrc CP15_TTBR0(r1) /* r1 = old TTB */ + ldr r0, [r7, #(PCB_PAGEDIR)] /* r0 = new TTB */ + cmp r0, r1 /* Switching to the TTB? */ + beq sw0 /* same TTB, skip */ + +#ifdef INVARIANTS + cmp r0, #0 /* new thread? */ + beq badsw4 /* no, panic */ +#endif + + bl cpu_context_switch /* new TTB as argument */ + + /* + * Registers at this point + * r7 = new PCB + * r8 = current pcpu + * r9 = lock + * r10 = oldtd + * r11 = newtd + */ + + /* + * Set new PMAP as current one. + * Update active list on PMAPs. + */ + ldr r6, [r11, #TD_PROC] /* newtd->proc */ + ldr r6, [r6, #P_VMSPACE] /* newtd->proc->vmspace */ + add r6, #VM_PMAP /* newtd->proc->vmspace->pmap */ + + ldr r5, [r8, #PC_CURPMAP] /* get old curpmap */ + str r6, [r8, #PC_CURPMAP] /* and save new one */ + + mov r0, #PM_ACTIVE + add r5, r0 /* r5 = old pm_active */ + add r6, r0 /* r6 = new pm_active */ + + /* Compute position and mask. */ + ldr r4, [r8, #PC_CPUID] +#if _NCPUWORDS > 1 + lsr r0, r4, #3 + bic r0, #3 + add r5, r0 /* r5 = position in old pm_active */ + add r6, r0 /* r6 = position in new pm_active */ + mov r2, #1 + and r0, r4, #31 + lsl r2, r0 /* r2 = mask */ +#else + mov r2, #1 + lsl r2, r4 /* r2 = mask */ +#endif + /* Clear cpu from old active list. */ +#ifdef SMP +1: ldrex r0, [r5] + bic r0, r2 + strex r1, r0, [r5] + teq r1, #0 + bne 1b +#else + ldr r0, [r5] + bic r0, r2 + str r0, [r5] +#endif + /* Set cpu to new active list. */ +#ifdef SMP +1: ldrex r0, [r6] + orr r0, r2 + strex r1, r0, [r6] + teq r1, #0 + bne 1b +#else + ldr r0, [r6] + orr r0, r2 + str r0, [r6] +#endif + +sw0: + /* + * Registers at this point + * r7 = new PCB + * r8 = current pcpu + * r9 = lock + * r10 = oldtd + * r11 = newtd + */ + + /* Change the old thread lock. */ + add r5, r10, #TD_LOCK + DMB +1: ldrex r0, [r5] + strex r1, r9, [r5] + teq r1, #0 + bne 1b + DMB + +sw1: + clrex + /* + * Registers at this point + * r7 = new PCB + * r8 = current pcpu + * r11 = newtd + */ + +#if defined(SMP) && defined(SCHED_ULE) + /* + * 386 and amd64 do the blocked lock test only for SMP and SCHED_ULE + * QQQ: What does it mean in reality and why is it done? + */ + ldr r6, =blocked_lock +1: + ldr r3, [r11, #TD_LOCK] /* atomic write regular read */ + cmp r3, r6 + beq 1b +#endif + /* Set the new tls */ + ldr r0, [r11, #(TD_MD + MD_TP)] + mcr CP15_TPIDRURO(r0) /* write tls thread reg 2 */ + + /* We have a new curthread now so make a note it */ + str r11, [r8, #PC_CURTHREAD] + mcr CP15_TPIDRPRW(r11) + + /* store pcb in per cpu structure */ + str r7, [r8, #PC_CURPCB] + + /* + * Restore all saved registers and return. Note that some saved + * registers can be changed when either cpu_fork(), cpu_set_upcall(), + * cpu_set_fork_handler(), or makectx() was called. + */ + add r3, r7, #PCB_R4 + ldmia r3, {r4-r12, sp, pc} + +#ifdef INVARIANTS +badsw1: + ldr r0, =sw1_panic_str + bl _C_LABEL(panic) +1: nop + b 1b + +badsw2: + ldr r0, =sw2_panic_str + bl _C_LABEL(panic) +1: nop + b 1b + +badsw3: + ldr r0, =sw3_panic_str + bl _C_LABEL(panic) +1: nop + b 1b + +badsw4: + ldr r0, =sw4_panic_str + bl _C_LABEL(panic) +1: nop + b 1b + +sw1_panic_str: + .asciz "cpu_throw: no newthread supplied.\n" +sw2_panic_str: + .asciz "cpu_switch: no curthread supplied.\n" +sw3_panic_str: + .asciz "cpu_switch: no newthread supplied.\n" +sw4_panic_str: + .asciz "cpu_switch: new pagedir is NULL.\n" +#endif +END(cpu_switch) Property changes on: user/ngie/make_check/sys/arm/arm/swtch-v6.S ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/make_check/sys/arm/arm/swtch.S =================================================================== --- user/ngie/make_check/sys/arm/arm/swtch.S (revision 295077) +++ user/ngie/make_check/sys/arm/arm/swtch.S (revision 295078) @@ -1,821 +1,121 @@ /* $NetBSD: cpuswitch.S,v 1.41 2003/11/15 08:44:18 scw Exp $ */ /*- * Copyright 2003 Wasabi Systems, Inc. * All rights reserved. * * Written by Steve C. Woodford for Wasabi Systems, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed for the NetBSD Project by * Wasabi Systems, Inc. * 4. The name of Wasabi Systems, Inc. may not be used to endorse * or promote products derived from this software without specific prior * written permission. * * THIS SOFTWARE IS PROVIDED BY WASABI SYSTEMS, INC. ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL WASABI SYSTEMS, INC * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ /*- * Copyright (c) 1994-1998 Mark Brinicombe. * Copyright (c) 1994 Brini. * All rights reserved. * * This code is derived from software written for Brini by Mark Brinicombe * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Brini. * 4. The name of the company nor the name of the author may be used to * endorse or promote products derived from this software without specific * prior written permission. * * THIS SOFTWARE IS PROVIDED BY BRINI ``AS IS'' AND ANY EXPRESS OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL BRINI OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * RiscBSD kernel project * * cpuswitch.S * * cpu switching functions * * Created : 15/10/94 * */ #include "assym.s" -#include "opt_sched.h" -#include #include #include #include #include __FBSDID("$FreeBSD$"); -#if __ARM_ARCH >= 6 && defined(SMP) -#define GET_PCPU(tmp, tmp2) \ - mrc p15, 0, tmp, c0, c0, 5; \ - and tmp, tmp, #0xf; \ - ldr tmp2, .Lcurpcpu+4; \ - mul tmp, tmp, tmp2; \ - ldr tmp2, .Lcurpcpu; \ - add tmp, tmp, tmp2; -#else - -#define GET_PCPU(tmp, tmp2) \ - ldr tmp, .Lcurpcpu -#endif - #ifdef VFP .fpu vfp /* allow VFP instructions */ #endif - -.Lcurpcpu: - .word _C_LABEL(__pcpu) - .word PCPU_SIZE -.Lblocked_lock: - .word _C_LABEL(blocked_lock) - - -#ifndef ARM_NEW_PMAP - -#define DOMAIN_CLIENT 0x01 - -.Lcpufuncs: - .word _C_LABEL(cpufuncs) - -/* - * cpu_throw(oldtd, newtd) - * - * Remove current thread state, then select the next thread to run - * and load its state. - * r0 = oldtd - * r1 = newtd - */ -ENTRY(cpu_throw) - mov r5, r1 - - /* - * r0 = oldtd - * r5 = newtd - */ - -#ifdef VFP /* This thread is dying, disable */ - bl _C_LABEL(vfp_discard) /* VFP without preserving state. */ -#endif - - GET_PCPU(r7, r9) - ldr r7, [r5, #(TD_PCB)] /* r7 = new thread's PCB */ - - /* Switch to lwp0 context */ - - ldr r9, .Lcpufuncs -#if !defined(CPU_ARM11) && !defined(CPU_CORTEXA) && !defined(CPU_MV_PJ4B) && !defined(CPU_KRAIT) - mov lr, pc - ldr pc, [r9, #CF_IDCACHE_WBINV_ALL] -#endif - ldr r0, [r7, #(PCB_PL1VEC)] - ldr r1, [r7, #(PCB_DACR)] - /* - * r0 = Pointer to L1 slot for vector_page (or NULL) - * r1 = lwp0's DACR - * r5 = lwp0 - * r7 = lwp0's PCB - * r9 = cpufuncs - */ - - /* - * Ensure the vector table is accessible by fixing up lwp0's L1 - */ - cmp r0, #0 /* No need to fixup vector table? */ - ldrne r3, [r0] /* But if yes, fetch current value */ - ldrne r2, [r7, #(PCB_L1VEC)] /* Fetch new vector_page value */ - mcr p15, 0, r1, c3, c0, 0 /* Update DACR for lwp0's context */ - cmpne r3, r2 /* Stuffing the same value? */ - strne r2, [r0] /* Store if not. */ - -#ifdef PMAP_INCLUDE_PTE_SYNC - /* - * Need to sync the cache to make sure that last store is - * visible to the MMU. - */ - movne r1, #4 - movne lr, pc - ldrne pc, [r9, #CF_DCACHE_WB_RANGE] -#endif /* PMAP_INCLUDE_PTE_SYNC */ - - /* - * Note: We don't do the same optimisation as cpu_switch() with - * respect to avoiding flushing the TLB if we're switching to - * the same L1 since this process' VM space may be about to go - * away, so we don't want *any* turds left in the TLB. - */ - - /* Switch the memory to the new process */ - ldr r0, [r7, #(PCB_PAGEDIR)] - mov lr, pc - ldr pc, [r9, #CF_CONTEXT_SWITCH] - - GET_PCPU(r6, r4) - /* Hook in a new pcb */ - str r7, [r6, #PC_CURPCB] - /* We have a new curthread now so make a note it */ - str r5, [r6, #PC_CURTHREAD] -#if __ARM_ARCH >= 6 - mcr p15, 0, r5, c13, c0, 4 -#endif - /* Set the new tp */ - ldr r6, [r5, #(TD_MD + MD_TP)] -#if __ARM_ARCH >= 6 - mcr p15, 0, r6, c13, c0, 3 -#else - ldr r4, =ARM_TP_ADDRESS - str r6, [r4] - ldr r6, [r5, #(TD_MD + MD_RAS_START)] - str r6, [r4, #4] /* ARM_RAS_START */ - ldr r6, [r5, #(TD_MD + MD_RAS_END)] - str r6, [r4, #8] /* ARM_RAS_END */ -#endif - /* Restore all the saved registers and exit */ - add r3, r7, #PCB_R4 - ldmia r3, {r4-r12, sp, pc} -END(cpu_throw) - -/* - * cpu_switch(oldtd, newtd, lock) - * - * Save the current thread state, then select the next thread to run - * and load its state. - * r0 = oldtd - * r1 = newtd - * r2 = lock (new lock for old thread) - */ -ENTRY(cpu_switch) - /* Interrupts are disabled. */ - /* Save all the registers in the old thread's pcb. */ - ldr r3, [r0, #(TD_PCB)] - - /* Restore all the saved registers and exit */ - add r3, #(PCB_R4) - stmia r3, {r4-r12, sp, lr, pc} - - mov r6, r2 /* Save the mutex */ - - /* rem: r0 = old lwp */ - /* rem: interrupts are disabled */ - - /* Process is now on a processor. */ - /* We have a new curthread now so make a note it */ - GET_PCPU(r7, r2) - str r1, [r7, #PC_CURTHREAD] -#if __ARM_ARCH >= 6 - mcr p15, 0, r1, c13, c0, 4 -#endif - - /* Hook in a new pcb */ - ldr r2, [r1, #TD_PCB] - str r2, [r7, #PC_CURPCB] - - /* Stage two : Save old context */ - - /* Get the user structure for the old thread. */ - ldr r2, [r0, #(TD_PCB)] - mov r4, r0 /* Save the old thread. */ - -#if __ARM_ARCH >= 6 - /* - * Set new tp. No need to store the old one first, userland can't - * change it directly on armv6. - */ - ldr r9, [r1, #(TD_MD + MD_TP)] - mcr p15, 0, r9, c13, c0, 3 -#else - /* Store the old tp; userland can change it on armv4. */ - ldr r3, =ARM_TP_ADDRESS - ldr r9, [r3] - str r9, [r0, #(TD_MD + MD_TP)] - ldr r9, [r3, #4] - str r9, [r0, #(TD_MD + MD_RAS_START)] - ldr r9, [r3, #8] - str r9, [r0, #(TD_MD + MD_RAS_END)] - - /* Set the new tp */ - ldr r9, [r1, #(TD_MD + MD_TP)] - str r9, [r3] - ldr r9, [r1, #(TD_MD + MD_RAS_START)] - str r9, [r3, #4] - ldr r9, [r1, #(TD_MD + MD_RAS_END)] - str r9, [r3, #8] -#endif - - /* Get the user structure for the new process in r9 */ - ldr r9, [r1, #(TD_PCB)] - - /* rem: r2 = old PCB */ - /* rem: r9 = new PCB */ - /* rem: interrupts are enabled */ - -#ifdef VFP - fmrx r0, fpexc /* If the VFP is enabled */ - tst r0, #(VFPEXC_EN) /* the current thread has */ - movne r1, #1 /* used it, so go save */ - addne r0, r2, #(PCB_VFPSTATE) /* the state into the PCB */ - blne _C_LABEL(vfp_store) /* and disable the VFP. */ -#endif - - /* r0-r3 now free! */ - - /* Third phase : restore saved context */ - - /* rem: r2 = old PCB */ - /* rem: r9 = new PCB */ - - ldr r5, [r9, #(PCB_DACR)] /* r5 = new DACR */ - mov r2, #DOMAIN_CLIENT - cmp r5, r2, lsl #(PMAP_DOMAIN_KERNEL * 2) /* Sw to kernel thread? */ - beq .Lcs_context_switched /* Yup. Don't flush cache */ - mrc p15, 0, r0, c3, c0, 0 /* r0 = old DACR */ - /* - * Get the new L1 table pointer into r11. If we're switching to - * an LWP with the same address space as the outgoing one, we can - * skip the cache purge and the TTB load. - * - * To avoid data dep stalls that would happen anyway, we try - * and get some useful work done in the mean time. - */ - mrc p15, 0, r10, c2, c0, 0 /* r10 = old L1 */ - ldr r11, [r9, #(PCB_PAGEDIR)] /* r11 = new L1 */ - - teq r10, r11 /* Same L1? */ - cmpeq r0, r5 /* Same DACR? */ - beq .Lcs_context_switched /* yes! */ - -#if !defined(CPU_ARM11) && !defined(CPU_CORTEXA) && !defined(CPU_MV_PJ4B) && !defined(CPU_KRAIT) - /* - * Definately need to flush the cache. - */ - - ldr r1, .Lcpufuncs - mov lr, pc - ldr pc, [r1, #CF_IDCACHE_WBINV_ALL] -#endif -.Lcs_cache_purge_skipped: - /* rem: r6 = lock */ - /* rem: r9 = new PCB */ - /* rem: r10 = old L1 */ - /* rem: r11 = new L1 */ - - mov r2, #0x00000000 - ldr r7, [r9, #(PCB_PL1VEC)] - - /* - * Ensure the vector table is accessible by fixing up the L1 - */ - cmp r7, #0 /* No need to fixup vector table? */ - ldrne r2, [r7] /* But if yes, fetch current value */ - ldrne r0, [r9, #(PCB_L1VEC)] /* Fetch new vector_page value */ - mcr p15, 0, r5, c3, c0, 0 /* Update DACR for new context */ - cmpne r2, r0 /* Stuffing the same value? */ -#ifndef PMAP_INCLUDE_PTE_SYNC - strne r0, [r7] /* Nope, update it */ -#else - beq .Lcs_same_vector - str r0, [r7] /* Otherwise, update it */ - - /* - * Need to sync the cache to make sure that last store is - * visible to the MMU. - */ - ldr r2, .Lcpufuncs - mov r0, r7 - mov r1, #4 - mov lr, pc - ldr pc, [r2, #CF_DCACHE_WB_RANGE] - -.Lcs_same_vector: -#endif /* PMAP_INCLUDE_PTE_SYNC */ - - cmp r10, r11 /* Switching to the same L1? */ - ldr r10, .Lcpufuncs - beq .Lcs_same_l1 /* Yup. */ - /* - * Do a full context switch, including full TLB flush. - */ - mov r0, r11 - mov lr, pc - ldr pc, [r10, #CF_CONTEXT_SWITCH] - - b .Lcs_context_switched - - /* - * We're switching to a different process in the same L1. - * In this situation, we only need to flush the TLB for the - * vector_page mapping, and even then only if r7 is non-NULL. - */ -.Lcs_same_l1: - cmp r7, #0 - movne r0, #0 /* We *know* vector_page's VA is 0x0 */ - movne lr, pc - ldrne pc, [r10, #CF_TLB_FLUSHID_SE] - -.Lcs_context_switched: - - /* Release the old thread */ - str r6, [r4, #TD_LOCK] -#if defined(SCHED_ULE) && defined(SMP) - ldr r6, .Lblocked_lock - GET_CURTHREAD_PTR(r3) -1: - ldr r4, [r3, #TD_LOCK] - cmp r4, r6 - beq 1b -#endif - - /* XXXSCW: Safe to re-enable FIQs here */ - - /* rem: r9 = new PCB */ - - /* Restore all the saved registers and exit */ - add r3, r9, #PCB_R4 - ldmia r3, {r4-r12, sp, pc} -END(cpu_switch) - - -#else /* !ARM_NEW_PMAP */ -#include - -ENTRY(cpu_context_switch) /* QQQ: What about macro instead of function? */ - DSB - mcr CP15_TTBR0(r0) /* set the new TTB */ - ISB - mov r0, #(CPU_ASID_KERNEL) - mcr CP15_TLBIASID(r0) /* flush not global TLBs */ - /* - * Flush entire Branch Target Cache because of the branch predictor - * is not architecturally invisible. See ARM Architecture Reference - * Manual ARMv7-A and ARMv7-R edition, page B2-1264(65), Branch - * predictors and Requirements for branch predictor maintenance - * operations sections. - * - * QQQ: The predictor is virtually addressed and holds virtual target - * addresses. Therefore, if mapping is changed, the predictor cache - * must be flushed.The flush is part of entire i-cache invalidation - * what is always called when code mapping is changed. So herein, - * it's the only place where standalone predictor flush must be - * executed in kernel (except self modifying code case). - */ - mcr CP15_BPIALL /* and flush entire Branch Target Cache */ - DSB - mov pc, lr -END(cpu_context_switch) - -/* - * cpu_throw(oldtd, newtd) - * - * Remove current thread state, then select the next thread to run - * and load its state. - * r0 = oldtd - * r1 = newtd - */ -ENTRY(cpu_throw) - mov r10, r0 /* r10 = oldtd */ - mov r11, r1 /* r11 = newtd */ - -#ifdef VFP /* This thread is dying, disable */ - bl _C_LABEL(vfp_discard) /* VFP without preserving state. */ -#endif - GET_PCPU(r8, r9) /* r8 = current pcpu */ - ldr r4, [r8, #PC_CPUID] /* r4 = current cpu id */ - - cmp r10, #0 /* old thread? */ - beq 2f /* no, skip */ - - /* Remove this CPU from the active list. */ - ldr r5, [r8, #PC_CURPMAP] - mov r0, #(PM_ACTIVE) - add r5, r0 /* r5 = old pm_active */ - - /* Compute position and mask. */ -#if _NCPUWORDS > 1 - lsr r0, r4, #3 - bic r0, #3 - add r5, r0 /* r5 = position in old pm_active */ - mov r2, #1 - and r0, r4, #31 - lsl r2, r0 /* r2 = mask */ -#else - mov r2, #1 - lsl r2, r4 /* r2 = mask */ -#endif - /* Clear cpu from old active list. */ -#ifdef SMP -1: ldrex r0, [r5] - bic r0, r2 - strex r1, r0, [r5] - teq r1, #0 - bne 1b -#else - ldr r0, [r5] - bic r0, r2 - str r0, [r5] -#endif - -2: -#ifdef INVARIANTS - cmp r11, #0 /* new thread? */ - beq badsw1 /* no, panic */ -#endif - ldr r7, [r11, #(TD_PCB)] /* r7 = new PCB */ - - /* - * Registers at this point - * r4 = current cpu id - * r7 = new PCB - * r8 = current pcpu - * r11 = newtd - */ - - /* MMU switch to new thread. */ - ldr r0, [r7, #(PCB_PAGEDIR)] -#ifdef INVARIANTS - cmp r0, #0 /* new thread? */ - beq badsw4 /* no, panic */ -#endif - bl _C_LABEL(cpu_context_switch) - - /* - * Set new PMAP as current one. - * Insert cpu to new active list. - */ - - ldr r6, [r11, #(TD_PROC)] /* newtd->proc */ - ldr r6, [r6, #(P_VMSPACE)] /* newtd->proc->vmspace */ - add r6, #VM_PMAP /* newtd->proc->vmspace->pmap */ - str r6, [r8, #PC_CURPMAP] /* store to curpmap */ - - mov r0, #PM_ACTIVE - add r6, r0 /* r6 = new pm_active */ - - /* compute position and mask */ -#if _NCPUWORDS > 1 - lsr r0, r4, #3 - bic r0, #3 - add r6, r0 /* r6 = position in new pm_active */ - mov r2, #1 - and r0, r4, #31 - lsl r2, r0 /* r2 = mask */ -#else - mov r2, #1 - lsl r2, r4 /* r2 = mask */ -#endif - /* Set cpu to new active list. */ -#ifdef SMP -1: ldrex r0, [r6] - orr r0, r2 - strex r1, r0, [r6] - teq r1, #0 - bne 1b -#else - ldr r0, [r6] - orr r0, r2 - str r0, [r6] -#endif - /* - * Registers at this point. - * r7 = new PCB - * r8 = current pcpu - * r11 = newtd - * They must match the ones in sw1 position !!! - */ - DMB - b sw1 /* share new thread init with cpu_switch() */ -END(cpu_throw) - -/* - * cpu_switch(oldtd, newtd, lock) - * - * Save the current thread state, then select the next thread to run - * and load its state. - * r0 = oldtd - * r1 = newtd - * r2 = lock (new lock for old thread) - */ -ENTRY(cpu_switch) - /* Interrupts are disabled. */ -#ifdef INVARIANTS - cmp r0, #0 /* old thread? */ - beq badsw2 /* no, panic */ -#endif - /* Save all the registers in the old thread's pcb. */ - ldr r3, [r0, #(TD_PCB)] - add r3, #(PCB_R4) - stmia r3, {r4-r12, sp, lr, pc} - -#ifdef INVARIANTS - cmp r1, #0 /* new thread? */ - beq badsw3 /* no, panic */ -#endif - /* - * Save arguments. Note that we can now use r0-r14 until - * it is time to restore them for the new thread. However, - * some registers are not safe over function call. - */ - mov r9, r2 /* r9 = lock */ - mov r10, r0 /* r10 = oldtd */ - mov r11, r1 /* r11 = newtd */ - - GET_PCPU(r8, r3) /* r8 = current PCPU */ - ldr r7, [r11, #(TD_PCB)] /* r7 = newtd->td_pcb */ - - - -#ifdef VFP - ldr r3, [r10, #(TD_PCB)] - fmrx r0, fpexc /* If the VFP is enabled */ - tst r0, #(VFPEXC_EN) /* the current thread has */ - movne r1, #1 /* used it, so go save */ - addne r0, r3, #(PCB_VFPSTATE) /* the state into the PCB */ - blne _C_LABEL(vfp_store) /* and disable the VFP. */ -#endif - - /* - * MMU switch. If we're switching to a thread with the same - * address space as the outgoing one, we can skip the MMU switch. - */ - mrc CP15_TTBR0(r1) /* r1 = old TTB */ - ldr r0, [r7, #(PCB_PAGEDIR)] /* r0 = new TTB */ - cmp r0, r1 /* Switching to the TTB? */ - beq sw0 /* same TTB, skip */ - -#ifdef INVARIANTS - cmp r0, #0 /* new thread? */ - beq badsw4 /* no, panic */ -#endif - - bl cpu_context_switch /* new TTB as argument */ - - /* - * Registers at this point - * r7 = new PCB - * r8 = current pcpu - * r9 = lock - * r10 = oldtd - * r11 = newtd - */ - - /* - * Set new PMAP as current one. - * Update active list on PMAPs. - */ - ldr r6, [r11, #TD_PROC] /* newtd->proc */ - ldr r6, [r6, #P_VMSPACE] /* newtd->proc->vmspace */ - add r6, #VM_PMAP /* newtd->proc->vmspace->pmap */ - - ldr r5, [r8, #PC_CURPMAP] /* get old curpmap */ - str r6, [r8, #PC_CURPMAP] /* and save new one */ - - mov r0, #PM_ACTIVE - add r5, r0 /* r5 = old pm_active */ - add r6, r0 /* r6 = new pm_active */ - - /* Compute position and mask. */ - ldr r4, [r8, #PC_CPUID] -#if _NCPUWORDS > 1 - lsr r0, r4, #3 - bic r0, #3 - add r5, r0 /* r5 = position in old pm_active */ - add r6, r0 /* r6 = position in new pm_active */ - mov r2, #1 - and r0, r4, #31 - lsl r2, r0 /* r2 = mask */ -#else - mov r2, #1 - lsl r2, r4 /* r2 = mask */ -#endif - /* Clear cpu from old active list. */ -#ifdef SMP -1: ldrex r0, [r5] - bic r0, r2 - strex r1, r0, [r5] - teq r1, #0 - bne 1b -#else - ldr r0, [r5] - bic r0, r2 - str r0, [r5] -#endif - /* Set cpu to new active list. */ -#ifdef SMP -1: ldrex r0, [r6] - orr r0, r2 - strex r1, r0, [r6] - teq r1, #0 - bne 1b -#else - ldr r0, [r6] - orr r0, r2 - str r0, [r6] -#endif - -sw0: - /* - * Registers at this point - * r7 = new PCB - * r8 = current pcpu - * r9 = lock - * r10 = oldtd - * r11 = newtd - */ - - /* Change the old thread lock. */ - add r5, r10, #TD_LOCK - DMB -1: ldrex r0, [r5] - strex r1, r9, [r5] - teq r1, #0 - bne 1b - DMB - -sw1: - clrex - /* - * Registers at this point - * r7 = new PCB - * r8 = current pcpu - * r11 = newtd - */ - -#if defined(SMP) && defined(SCHED_ULE) - /* - * 386 and amd64 do the blocked lock test only for SMP and SCHED_ULE - * QQQ: What does it mean in reality and why is it done? - */ - ldr r6, =blocked_lock -1: - ldr r3, [r11, #TD_LOCK] /* atomic write regular read */ - cmp r3, r6 - beq 1b -#endif - /* Set the new tls */ - ldr r0, [r11, #(TD_MD + MD_TP)] - mcr CP15_TPIDRURO(r0) /* write tls thread reg 2 */ - - /* We have a new curthread now so make a note it */ - str r11, [r8, #PC_CURTHREAD] - mcr CP15_TPIDRPRW(r11) - - /* store pcb in per cpu structure */ - str r7, [r8, #PC_CURPCB] - - /* - * Restore all saved registers and return. Note that some saved - * registers can be changed when either cpu_fork(), cpu_set_upcall(), - * cpu_set_fork_handler(), or makectx() was called. - */ - add r3, r7, #PCB_R4 - ldmia r3, {r4-r12, sp, pc} - -#ifdef INVARIANTS -badsw1: - ldr r0, =sw1_panic_str - bl _C_LABEL(panic) -1: nop - b 1b - -badsw2: - ldr r0, =sw2_panic_str - bl _C_LABEL(panic) -1: nop - b 1b - -badsw3: - ldr r0, =sw3_panic_str - bl _C_LABEL(panic) -1: nop - b 1b - -badsw4: - ldr r0, =sw4_panic_str - bl _C_LABEL(panic) -1: nop - b 1b - -sw1_panic_str: - .asciz "cpu_throw: no newthread supplied.\n" -sw2_panic_str: - .asciz "cpu_switch: no curthread supplied.\n" -sw3_panic_str: - .asciz "cpu_switch: no newthread supplied.\n" -sw4_panic_str: - .asciz "cpu_switch: new pagedir is NULL.\n" -#endif -END(cpu_switch) - - -#endif /* !ARM_NEW_PMAP */ ENTRY(savectx) stmfd sp!, {lr} sub sp, sp, #4 /* Store all the registers in the thread's pcb */ add r3, r0, #(PCB_R4) stmia r3, {r4-r12, sp, lr, pc} #ifdef VFP fmrx r2, fpexc /* If the VFP is enabled */ tst r2, #(VFPEXC_EN) /* the current thread has */ movne r1, #1 /* used it, so go save */ addne r0, r0, #(PCB_VFPSTATE) /* the state into the PCB */ blne _C_LABEL(vfp_store) /* and disable the VFP. */ #endif add sp, sp, #4; ldmfd sp!, {pc} END(savectx) ENTRY(fork_trampoline) STOP_UNWINDING /* EABI: Don't unwind beyond the thread enty point. */ mov fp, #0 /* OABI: Stack traceback via fp stops here. */ mov r2, sp mov r1, r5 mov r0, r4 ldr lr, =swi_exit /* Go finish forking, then return */ b _C_LABEL(fork_exit) /* to userland via swi_exit code. */ END(fork_trampoline) Index: user/ngie/make_check/sys/arm/arm/trap-v6.c =================================================================== --- user/ngie/make_check/sys/arm/arm/trap-v6.c (revision 295077) +++ user/ngie/make_check/sys/arm/arm/trap-v6.c (revision 295078) @@ -1,666 +1,657 @@ /*- * Copyright 2014 Olivier Houchard * Copyright 2014 Svatopluk Kraus * Copyright 2014 Michal Meloun * Copyright 2014 Andrew Turner * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include "opt_ktrace.h" #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #ifdef KTRACE #include #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef KDB #include #include #endif #ifdef KDTRACE_HOOKS #include #endif extern char fusubailout[]; extern char cachebailout[]; #ifdef DEBUG int last_fault_code; /* For the benefit of pmap_fault_fixup() */ #endif struct ksig { int sig; u_long code; vm_offset_t addr; }; typedef int abort_func_t(struct trapframe *, u_int, u_int, u_int, u_int, struct thread *, struct ksig *); static abort_func_t abort_fatal; static abort_func_t abort_align; static abort_func_t abort_icache; struct abort { abort_func_t *func; const char *desc; }; /* * How are the aborts handled? * * Undefined Code: * - Always fatal as we do not know what does it mean. * Imprecise External Abort: * - Always fatal, but can be handled somehow in the future. * Now, due to PCIe buggy hardware, ignored. * Precise External Abort: * - Always fatal, but who knows in the future??? * Debug Event: * - Special handling. * External Translation Abort (L1 & L2) * - Always fatal as something is screwed up in page tables or hardware. * Domain Fault (L1 & L2): * - Always fatal as we do not play game with domains. * Alignment Fault: * - Everything should be aligned in kernel with exception of user to kernel * and vice versa data copying, so if pcb_onfault is not set, it's fatal. * We generate signal in case of abort from user mode. * Instruction cache maintenance: * - According to manual, this is translation fault during cache maintenance * operation. So, it could be really complex in SMP case and fuzzy too * for cache operations working on virtual addresses. For now, we will * consider this abort as fatal. In fact, no cache maintenance on * not mapped virtual addresses should be called. As cache maintenance * operation (except DMB, DSB, and Flush Prefetch Buffer) are priviledged, * the abort is fatal for user mode as well for now. (This is good place to * note that cache maintenance on virtual address fill TLB.) * Acces Bit (L1 & L2): * - Fast hardware emulation for kernel and user mode. * Translation Fault (L1 & L2): * - Standard fault mechanism is held including vm_fault(). * Permission Fault (L1 & L2): * - Fast hardware emulation of modify bits and in other cases, standard * fault mechanism is held including vm_fault(). */ static const struct abort aborts[] = { {abort_fatal, "Undefined Code (0x000)"}, {abort_align, "Alignment Fault"}, {abort_fatal, "Debug Event"}, {NULL, "Access Bit (L1)"}, {NULL, "Instruction cache maintenance"}, {NULL, "Translation Fault (L1)"}, {NULL, "Access Bit (L2)"}, {NULL, "Translation Fault (L2)"}, {abort_fatal, "External Abort"}, {abort_fatal, "Domain Fault (L1)"}, {abort_fatal, "Undefined Code (0x00A)"}, {abort_fatal, "Domain Fault (L2)"}, {abort_fatal, "External Translation Abort (L1)"}, {NULL, "Permission Fault (L1)"}, {abort_fatal, "External Translation Abort (L2)"}, {NULL, "Permission Fault (L2)"}, {abort_fatal, "TLB Conflict Abort"}, {abort_fatal, "Undefined Code (0x401)"}, {abort_fatal, "Undefined Code (0x402)"}, {abort_fatal, "Undefined Code (0x403)"}, {abort_fatal, "Undefined Code (0x404)"}, {abort_fatal, "Undefined Code (0x405)"}, {abort_fatal, "Asynchronous External Abort"}, {abort_fatal, "Undefined Code (0x407)"}, {abort_fatal, "Asynchronous Parity Error on Memory Access"}, {abort_fatal, "Parity Error on Memory Access"}, {abort_fatal, "Undefined Code (0x40A)"}, {abort_fatal, "Undefined Code (0x40B)"}, {abort_fatal, "Parity Error on Translation (L1)"}, {abort_fatal, "Undefined Code (0x40D)"}, {abort_fatal, "Parity Error on Translation (L2)"}, {abort_fatal, "Undefined Code (0x40F)"} }; static __inline void call_trapsignal(struct thread *td, int sig, int code, vm_offset_t addr) { ksiginfo_t ksi; CTR4(KTR_TRAP, "%s: addr: %#x, sig: %d, code: %d", __func__, addr, sig, code); /* * TODO: some info would be nice to know * if we are serving data or prefetch abort. */ ksiginfo_init_trap(&ksi); ksi.ksi_signo = sig; ksi.ksi_code = code; ksi.ksi_addr = (void *)addr; trapsignal(td, &ksi); } /* * abort_imprecise() handles the following abort: * * FAULT_EA_IMPREC - Imprecise External Abort * * The imprecise means that we don't know where the abort happened, * thus FAR is undefined. The abort should not never fire, but hot * plugging or accidental hardware failure can be the cause of it. * If the abort happens, it can even be on different (thread) context. * Without any additional support, the abort is fatal, as we do not * know what really happened. * * QQQ: Some additional functionality, like pcb_onfault but global, * can be implemented. Imprecise handlers could be registered * which tell us if the abort is caused by something they know * about. They should return one of three codes like: * FAULT_IS_MINE, * FAULT_CAN_BE_MINE, * FAULT_IS_NOT_MINE. * The handlers should be called until some of them returns * FAULT_IS_MINE value or all was called. If all handlers return * FAULT_IS_NOT_MINE value, then the abort is fatal. */ static __inline void abort_imprecise(struct trapframe *tf, u_int fsr, u_int prefetch, bool usermode) { /* * XXX - We can got imprecise abort as result of access * to not-present PCI/PCIe configuration space. */ #if 0 goto out; #endif abort_fatal(tf, FAULT_EA_IMPREC, fsr, 0, prefetch, curthread, NULL); /* * Returning from this function means that we ignore * the abort for good reason. Note that imprecise abort * could fire any time even in user mode. */ #if 0 out: if (usermode) userret(curthread, tf); #endif } /* * abort_debug() handles the following abort: * * FAULT_DEBUG - Debug Event * */ static __inline void abort_debug(struct trapframe *tf, u_int fsr, u_int prefetch, bool usermode, u_int far) { if (usermode) { struct thread *td; td = curthread; call_trapsignal(td, SIGTRAP, TRAP_BRKPT, far); userret(td, tf); } else { #ifdef KDB kdb_trap((prefetch) ? T_BREAKPOINT : T_WATCHPOINT, 0, tf); #else printf("No debugger in kernel.\n"); #endif } } /* * Abort handler. * * FAR, FSR, and everything what can be lost after enabling * interrupts must be grabbed before the interrupts will be * enabled. Note that when interrupts will be enabled, we * could even migrate to another CPU ... * * TODO: move quick cases to ASM */ void abort_handler(struct trapframe *tf, int prefetch) { struct thread *td; vm_offset_t far, va; int idx, rv; uint32_t fsr; struct ksig ksig; struct proc *p; struct pcb *pcb; struct vm_map *map; struct vmspace *vm; vm_prot_t ftype; bool usermode; #ifdef INVARIANTS void *onfault; #endif td = curthread; fsr = (prefetch) ? cp15_ifsr_get(): cp15_dfsr_get(); #if __ARM_ARCH >= 7 far = (prefetch) ? cp15_ifar_get() : cp15_dfar_get(); #else far = (prefetch) ? TRAPF_PC(tf) : cp15_dfar_get(); #endif idx = FSR_TO_FAULT(fsr); usermode = TRAPF_USERMODE(tf); /* Abort came from user mode? */ if (usermode) td->td_frame = tf; CTR6(KTR_TRAP, "%s: fsr %#x (idx %u) far %#x prefetch %u usermode %d", __func__, fsr, idx, far, prefetch, usermode); /* * Firstly, handle aborts that are not directly related to mapping. */ if (__predict_false(idx == FAULT_EA_IMPREC)) { abort_imprecise(tf, fsr, prefetch, usermode); return; } if (__predict_false(idx == FAULT_DEBUG)) { abort_debug(tf, fsr, prefetch, usermode, far); return; } /* * ARM has a set of unprivileged load and store instructions * (LDRT/LDRBT/STRT/STRBT ...) which are supposed to be used in other * than user mode and OS should recognize their aborts and behave * appropriately. However, there is no way how to do that reasonably * in general unless we restrict the handling somehow. * * For now, these instructions are used only in copyin()/copyout() * like functions where usermode buffers are checked in advance that * they are not from KVA space. Thus, no action is needed here. */ -#ifdef ARM_NEW_PMAP rv = pmap_fault(PCPU_GET(curpmap), far, fsr, idx, usermode); if (rv == KERN_SUCCESS) return; if (rv == KERN_INVALID_ADDRESS) goto nogo; -#endif /* * Now, when we handled imprecise and debug aborts, the rest of * aborts should be really related to mapping. */ PCPU_INC(cnt.v_trap); #ifdef KDB if (kdb_active) { kdb_reenter(); goto out; } #endif if (__predict_false((td->td_pflags & TDP_NOFAULTING) != 0)) { /* * Due to both processor errata and lazy TLB invalidation when * access restrictions are removed from virtual pages, memory * accesses that are allowed by the physical mapping layer may * nonetheless cause one spurious page fault per virtual page. * When the thread is executing a "no faulting" section that * is bracketed by vm_fault_{disable,enable}_pagefaults(), * every page fault is treated as a spurious page fault, * unless it accesses the same virtual address as the most * recent page fault within the same "no faulting" section. */ if (td->td_md.md_spurflt_addr != far || (td->td_pflags & TDP_RESETSPUR) != 0) { td->td_md.md_spurflt_addr = far; td->td_pflags &= ~TDP_RESETSPUR; tlb_flush_local(far & ~PAGE_MASK); return; } } else { /* * If we get a page fault while in a critical section, then * it is most likely a fatal kernel page fault. The kernel * is already going to panic trying to get a sleep lock to * do the VM lookup, so just consider it a fatal trap so the * kernel can print out a useful trap message and even get * to the debugger. * * If we get a page fault while holding a non-sleepable * lock, then it is most likely a fatal kernel page fault. * If WITNESS is enabled, then it's going to whine about * bogus LORs with various VM locks, so just skip to the * fatal trap handling directly. */ if (td->td_critnest != 0 || WITNESS_CHECK(WARN_SLEEPOK | WARN_GIANTOK, NULL, "Kernel page fault") != 0) { abort_fatal(tf, idx, fsr, far, prefetch, td, &ksig); return; } } /* Re-enable interrupts if they were enabled previously. */ if (td->td_md.md_spinlock_count == 0) { if (__predict_true(tf->tf_spsr & PSR_I) == 0) enable_interrupts(PSR_I); if (__predict_true(tf->tf_spsr & PSR_F) == 0) enable_interrupts(PSR_F); } p = td->td_proc; if (usermode) { td->td_pticks = 0; if (td->td_cowgen != p->p_cowgen) thread_cow_update(td); } /* Invoke the appropriate handler, if necessary. */ if (__predict_false(aborts[idx].func != NULL)) { if ((aborts[idx].func)(tf, idx, fsr, far, prefetch, td, &ksig)) goto do_trapsignal; goto out; } /* * Don't pass faulting cache operation to vm_fault(). We don't want * to handle all vm stuff at this moment. */ pcb = td->td_pcb; if (__predict_false(pcb->pcb_onfault == cachebailout)) { tf->tf_r0 = far; /* return failing address */ tf->tf_pc = (register_t)pcb->pcb_onfault; return; } /* Handle remaining I-cache aborts. */ if (idx == FAULT_ICACHE) { if (abort_icache(tf, idx, fsr, far, prefetch, td, &ksig)) goto do_trapsignal; goto out; } /* * At this point, we're dealing with one of the following aborts: * * FAULT_TRAN_xx - Translation * FAULT_PERM_xx - Permission * * These are the main virtual memory-related faults signalled by * the MMU. */ /* fusubailout is used by [fs]uswintr to avoid page faulting. */ if (__predict_false(pcb->pcb_onfault == fusubailout)) { tf->tf_r0 = EFAULT; tf->tf_pc = (register_t)pcb->pcb_onfault; return; } va = trunc_page(far); if (va >= KERNBASE) { /* * Don't allow user-mode faults in kernel address space. */ if (usermode) goto nogo; map = kernel_map; } else { /* * This is a fault on non-kernel virtual memory. If curproc * is NULL or curproc->p_vmspace is NULL the fault is fatal. */ vm = (p != NULL) ? p->p_vmspace : NULL; if (vm == NULL) goto nogo; map = &vm->vm_map; if (!usermode && (td->td_intr_nesting_level != 0 || pcb->pcb_onfault == NULL)) { abort_fatal(tf, idx, fsr, far, prefetch, td, &ksig); return; } } ftype = (fsr & FSR_WNR) ? VM_PROT_WRITE : VM_PROT_READ; if (prefetch) ftype |= VM_PROT_EXECUTE; #ifdef DEBUG last_fault_code = fsr; -#endif - -#ifndef ARM_NEW_PMAP - if (pmap_fault_fixup(vmspace_pmap(td->td_proc->p_vmspace), va, ftype, - usermode)) { - goto out; - } #endif #ifdef INVARIANTS onfault = pcb->pcb_onfault; pcb->pcb_onfault = NULL; #endif /* Fault in the page. */ rv = vm_fault(map, va, ftype, VM_FAULT_NORMAL); #ifdef INVARIANTS pcb->pcb_onfault = onfault; #endif if (__predict_true(rv == KERN_SUCCESS)) goto out; nogo: if (!usermode) { if (td->td_intr_nesting_level == 0 && pcb->pcb_onfault != NULL) { tf->tf_r0 = rv; tf->tf_pc = (int)pcb->pcb_onfault; return; } CTR2(KTR_TRAP, "%s: vm_fault() failed with %d", __func__, rv); abort_fatal(tf, idx, fsr, far, prefetch, td, &ksig); return; } ksig.sig = SIGSEGV; ksig.code = (rv == KERN_PROTECTION_FAILURE) ? SEGV_ACCERR : SEGV_MAPERR; ksig.addr = far; do_trapsignal: call_trapsignal(td, ksig.sig, ksig.code, ksig.addr); out: if (usermode) userret(td, tf); } /* * abort_fatal() handles the following data aborts: * * FAULT_DEBUG - Debug Event * FAULT_ACCESS_xx - Acces Bit * FAULT_EA_PREC - Precise External Abort * FAULT_DOMAIN_xx - Domain Fault * FAULT_EA_TRAN_xx - External Translation Abort * FAULT_EA_IMPREC - Imprecise External Abort * + all undefined codes for ABORT * * We should never see these on a properly functioning system. * * This function is also called by the other handlers if they * detect a fatal problem. * * Note: If 'l' is NULL, we assume we're dealing with a prefetch abort. */ static int abort_fatal(struct trapframe *tf, u_int idx, u_int fsr, u_int far, u_int prefetch, struct thread *td, struct ksig *ksig) { bool usermode; const char *mode; const char *rw_mode; usermode = TRAPF_USERMODE(tf); #ifdef KDTRACE_HOOKS if (!usermode) { if (dtrace_trap_func != NULL && (*dtrace_trap_func)(tf, far)) return (0); } #endif mode = usermode ? "user" : "kernel"; rw_mode = fsr & FSR_WNR ? "write" : "read"; disable_interrupts(PSR_I|PSR_F); if (td != NULL) { printf("Fatal %s mode data abort: '%s' on %s\n", mode, aborts[idx].desc, rw_mode); printf("trapframe: %p\nFSR=%08x, FAR=", tf, fsr); if (idx != FAULT_EA_IMPREC) printf("%08x, ", far); else printf("Invalid, "); printf("spsr=%08x\n", tf->tf_spsr); } else { printf("Fatal %s mode prefetch abort at 0x%08x\n", mode, tf->tf_pc); printf("trapframe: %p, spsr=%08x\n", tf, tf->tf_spsr); } printf("r0 =%08x, r1 =%08x, r2 =%08x, r3 =%08x\n", tf->tf_r0, tf->tf_r1, tf->tf_r2, tf->tf_r3); printf("r4 =%08x, r5 =%08x, r6 =%08x, r7 =%08x\n", tf->tf_r4, tf->tf_r5, tf->tf_r6, tf->tf_r7); printf("r8 =%08x, r9 =%08x, r10=%08x, r11=%08x\n", tf->tf_r8, tf->tf_r9, tf->tf_r10, tf->tf_r11); printf("r12=%08x, ", tf->tf_r12); if (usermode) printf("usp=%08x, ulr=%08x", tf->tf_usr_sp, tf->tf_usr_lr); else printf("ssp=%08x, slr=%08x", tf->tf_svc_sp, tf->tf_svc_lr); printf(", pc =%08x\n\n", tf->tf_pc); #ifdef KDB if (debugger_on_panic || kdb_active) kdb_trap(fsr, 0, tf); #endif panic("Fatal abort"); /*NOTREACHED*/ } /* * abort_align() handles the following data abort: * * FAULT_ALIGN - Alignment fault * * Everything should be aligned in kernel with exception of user to kernel * and vice versa data copying, so if pcb_onfault is not set, it's fatal. * We generate signal in case of abort from user mode. */ static int abort_align(struct trapframe *tf, u_int idx, u_int fsr, u_int far, u_int prefetch, struct thread *td, struct ksig *ksig) { bool usermode; usermode = TRAPF_USERMODE(tf); if (!usermode) { if (td->td_intr_nesting_level == 0 && td != NULL && td->td_pcb->pcb_onfault != NULL) { tf->tf_r0 = EFAULT; tf->tf_pc = (int)td->td_pcb->pcb_onfault; return (0); } abort_fatal(tf, idx, fsr, far, prefetch, td, ksig); } /* Deliver a bus error signal to the process */ ksig->code = BUS_ADRALN; ksig->sig = SIGBUS; ksig->addr = far; return (1); } /* * abort_icache() handles the following data abort: * * FAULT_ICACHE - Instruction cache maintenance * * According to manual, FAULT_ICACHE is translation fault during cache * maintenance operation. In fact, no cache maintenance operation on * not mapped virtual addresses should be called. As cache maintenance * operation (except DMB, DSB, and Flush Prefetch Buffer) are priviledged, * the abort is concider as fatal for now. However, all the matter with * cache maintenance operation on virtual addresses could be really complex * and fuzzy in SMP case, so maybe in future standard fault mechanism * should be held here including vm_fault() calling. */ static int abort_icache(struct trapframe *tf, u_int idx, u_int fsr, u_int far, u_int prefetch, struct thread *td, struct ksig *ksig) { abort_fatal(tf, idx, fsr, far, prefetch, td, ksig); return(0); } Index: user/ngie/make_check/sys/arm/conf/std.armv6 =================================================================== --- user/ngie/make_check/sys/arm/conf/std.armv6 (revision 295077) +++ user/ngie/make_check/sys/arm/conf/std.armv6 (revision 295078) @@ -1,45 +1,43 @@ # Standard kernel config items for all ARMv6/v7 systems. # # $FreeBSD$ options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options IPSEC # IP (v4/v6) security options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options QUOTA # Enable disk quotas for UFS options NFSCL # Network Filesystem Client options NFSLOCKD # Network Lock Manager options NFS_ROOT # NFS usable as /, requires NFSCL options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options TMPFS # Efficient memory filesystem options GEOM_PART_GPT # GUID Partition Tables options GEOM_PART_BSD # BSD partition scheme options GEOM_PART_MBR # MBR partition scheme options GEOM_LABEL # Provides labelization options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options FREEBSD_BOOT_LOADER # Process metadata passed from loader(8) options VFP # Enable floating point hardware support -options ARM_NEW_PMAP # Use new pmap code. - # DTrace support options KDTRACE_HOOKS # Kernel DTrace hooks options DDB_CTF # all architectures - kernel ELF linker loads CTF data makeoptions WITH_CTF=1 Index: user/ngie/make_check/sys/arm/include/cpufunc.h =================================================================== --- user/ngie/make_check/sys/arm/include/cpufunc.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/cpufunc.h (revision 295078) @@ -1,627 +1,580 @@ /* $NetBSD: cpufunc.h,v 1.29 2003/09/06 09:08:35 rearnsha Exp $ */ /*- * Copyright (c) 1997 Mark Brinicombe. * Copyright (c) 1997 Causality Limited * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Causality Limited. * 4. The name of Causality Limited may not be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY CAUSALITY LIMITED ``AS IS'' AND ANY EXPRESS * OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE * DISCLAIMED. IN NO EVENT SHALL CAUSALITY LIMITED BE LIABLE FOR ANY DIRECT, * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * RiscBSD kernel project * * cpufunc.h * * Prototypes for cpu, mmu and tlb related functions. * * $FreeBSD$ */ #ifndef _MACHINE_CPUFUNC_H_ #define _MACHINE_CPUFUNC_H_ #ifdef _KERNEL #include #include #include static __inline void breakpoint(void) { __asm(".word 0xe7ffffff"); } struct cpu_functions { /* CPU functions */ u_int (*cf_id) (void); void (*cf_cpwait) (void); /* MMU functions */ u_int (*cf_control) (u_int bic, u_int eor); void (*cf_domains) (u_int domains); void (*cf_setttb) (u_int ttb); u_int (*cf_faultstatus) (void); u_int (*cf_faultaddress) (void); /* TLB functions */ void (*cf_tlb_flushID) (void); void (*cf_tlb_flushID_SE) (u_int va); void (*cf_tlb_flushI) (void); void (*cf_tlb_flushI_SE) (u_int va); void (*cf_tlb_flushD) (void); void (*cf_tlb_flushD_SE) (u_int va); /* * Cache operations: * * We define the following primitives: * * icache_sync_all Synchronize I-cache * icache_sync_range Synchronize I-cache range * * dcache_wbinv_all Write-back and Invalidate D-cache * dcache_wbinv_range Write-back and Invalidate D-cache range * dcache_inv_range Invalidate D-cache range * dcache_wb_range Write-back D-cache range * * idcache_wbinv_all Write-back and Invalidate D-cache, * Invalidate I-cache * idcache_wbinv_range Write-back and Invalidate D-cache, * Invalidate I-cache range * * Note that the ARM term for "write-back" is "clean". We use * the term "write-back" since it's a more common way to describe * the operation. * * There are some rules that must be followed: * * ID-cache Invalidate All: * Unlike other functions, this one must never write back. * It is used to intialize the MMU when it is in an unknown * state (such as when it may have lines tagged as valid * that belong to a previous set of mappings). * * I-cache Synch (all or range): * The goal is to synchronize the instruction stream, * so you may beed to write-back dirty D-cache blocks * first. If a range is requested, and you can't * synchronize just a range, you have to hit the whole * thing. * * D-cache Write-Back and Invalidate range: * If you can't WB-Inv a range, you must WB-Inv the * entire D-cache. * * D-cache Invalidate: * If you can't Inv the D-cache, you must Write-Back * and Invalidate. Code that uses this operation * MUST NOT assume that the D-cache will not be written * back to memory. * * D-cache Write-Back: * If you can't Write-back without doing an Inv, * that's fine. Then treat this as a WB-Inv. * Skipping the invalidate is merely an optimization. * * All operations: * Valid virtual addresses must be passed to each * cache operation. */ void (*cf_icache_sync_all) (void); void (*cf_icache_sync_range) (vm_offset_t, vm_size_t); void (*cf_dcache_wbinv_all) (void); void (*cf_dcache_wbinv_range) (vm_offset_t, vm_size_t); void (*cf_dcache_inv_range) (vm_offset_t, vm_size_t); void (*cf_dcache_wb_range) (vm_offset_t, vm_size_t); void (*cf_idcache_inv_all) (void); void (*cf_idcache_wbinv_all) (void); void (*cf_idcache_wbinv_range) (vm_offset_t, vm_size_t); void (*cf_l2cache_wbinv_all) (void); void (*cf_l2cache_wbinv_range) (vm_offset_t, vm_size_t); void (*cf_l2cache_inv_range) (vm_offset_t, vm_size_t); void (*cf_l2cache_wb_range) (vm_offset_t, vm_size_t); void (*cf_l2cache_drain_writebuf) (void); /* Other functions */ void (*cf_flush_prefetchbuf) (void); void (*cf_drain_writebuf) (void); void (*cf_flush_brnchtgt_C) (void); void (*cf_flush_brnchtgt_E) (u_int va); void (*cf_sleep) (int mode); /* Soft functions */ int (*cf_dataabt_fixup) (void *arg); int (*cf_prefetchabt_fixup) (void *arg); void (*cf_context_switch) (void); void (*cf_setup) (void); }; extern struct cpu_functions cpufuncs; extern u_int cputype; #define cpu_ident() cpufuncs.cf_id() #define cpu_cpwait() cpufuncs.cf_cpwait() #define cpu_control(c, e) cpufuncs.cf_control(c, e) #define cpu_domains(d) cpufuncs.cf_domains(d) #define cpu_setttb(t) cpufuncs.cf_setttb(t) #define cpu_faultstatus() cpufuncs.cf_faultstatus() #define cpu_faultaddress() cpufuncs.cf_faultaddress() -#ifndef SMP - #define cpu_tlb_flushID() cpufuncs.cf_tlb_flushID() #define cpu_tlb_flushID_SE(e) cpufuncs.cf_tlb_flushID_SE(e) #define cpu_tlb_flushI() cpufuncs.cf_tlb_flushI() #define cpu_tlb_flushI_SE(e) cpufuncs.cf_tlb_flushI_SE(e) #define cpu_tlb_flushD() cpufuncs.cf_tlb_flushD() #define cpu_tlb_flushD_SE(e) cpufuncs.cf_tlb_flushD_SE(e) - -#else -void tlb_broadcast(int); - -#if defined(CPU_CORTEXA) || defined(CPU_MV_PJ4B) || defined(CPU_KRAIT) -#define TLB_BROADCAST /* No need to explicitely send an IPI */ -#else -#define TLB_BROADCAST tlb_broadcast(7) -#endif - -#define cpu_tlb_flushID() do { \ - cpufuncs.cf_tlb_flushID(); \ - TLB_BROADCAST; \ -} while(0) - -#define cpu_tlb_flushID_SE(e) do { \ - cpufuncs.cf_tlb_flushID_SE(e); \ - TLB_BROADCAST; \ -} while(0) - - -#define cpu_tlb_flushI() do { \ - cpufuncs.cf_tlb_flushI(); \ - TLB_BROADCAST; \ -} while(0) - - -#define cpu_tlb_flushI_SE(e) do { \ - cpufuncs.cf_tlb_flushI_SE(e); \ - TLB_BROADCAST; \ -} while(0) - - -#define cpu_tlb_flushD() do { \ - cpufuncs.cf_tlb_flushD(); \ - TLB_BROADCAST; \ -} while(0) - - -#define cpu_tlb_flushD_SE(e) do { \ - cpufuncs.cf_tlb_flushD_SE(e); \ - TLB_BROADCAST; \ -} while(0) - -#endif #define cpu_icache_sync_all() cpufuncs.cf_icache_sync_all() #define cpu_icache_sync_range(a, s) cpufuncs.cf_icache_sync_range((a), (s)) #define cpu_dcache_wbinv_all() cpufuncs.cf_dcache_wbinv_all() #define cpu_dcache_wbinv_range(a, s) cpufuncs.cf_dcache_wbinv_range((a), (s)) #define cpu_dcache_inv_range(a, s) cpufuncs.cf_dcache_inv_range((a), (s)) #define cpu_dcache_wb_range(a, s) cpufuncs.cf_dcache_wb_range((a), (s)) #define cpu_idcache_inv_all() cpufuncs.cf_idcache_inv_all() #define cpu_idcache_wbinv_all() cpufuncs.cf_idcache_wbinv_all() #define cpu_idcache_wbinv_range(a, s) cpufuncs.cf_idcache_wbinv_range((a), (s)) #define cpu_l2cache_wbinv_all() cpufuncs.cf_l2cache_wbinv_all() #define cpu_l2cache_wb_range(a, s) cpufuncs.cf_l2cache_wb_range((a), (s)) #define cpu_l2cache_inv_range(a, s) cpufuncs.cf_l2cache_inv_range((a), (s)) #define cpu_l2cache_wbinv_range(a, s) cpufuncs.cf_l2cache_wbinv_range((a), (s)) #define cpu_l2cache_drain_writebuf() cpufuncs.cf_l2cache_drain_writebuf() #define cpu_flush_prefetchbuf() cpufuncs.cf_flush_prefetchbuf() #define cpu_drain_writebuf() cpufuncs.cf_drain_writebuf() #define cpu_flush_brnchtgt_C() cpufuncs.cf_flush_brnchtgt_C() #define cpu_flush_brnchtgt_E(e) cpufuncs.cf_flush_brnchtgt_E(e) #define cpu_sleep(m) cpufuncs.cf_sleep(m) #define cpu_dataabt_fixup(a) cpufuncs.cf_dataabt_fixup(a) #define cpu_prefetchabt_fixup(a) cpufuncs.cf_prefetchabt_fixup(a) #define ABORT_FIXUP_OK 0 /* fixup succeeded */ #define ABORT_FIXUP_FAILED 1 /* fixup failed */ #define ABORT_FIXUP_RETURN 2 /* abort handler should return */ #define cpu_setup() cpufuncs.cf_setup() int set_cpufuncs (void); #define ARCHITECTURE_NOT_PRESENT 1 /* known but not configured */ #define ARCHITECTURE_NOT_SUPPORTED 2 /* not known */ void cpufunc_nullop (void); int cpufunc_null_fixup (void *); int early_abort_fixup (void *); int late_abort_fixup (void *); u_int cpufunc_id (void); u_int cpufunc_cpuid (void); u_int cpufunc_control (u_int clear, u_int bic); void cpufunc_domains (u_int domains); u_int cpufunc_faultstatus (void); u_int cpufunc_faultaddress (void); u_int cpu_pfr (int); #if defined(CPU_FA526) void fa526_setup (void); void fa526_setttb (u_int ttb); void fa526_context_switch (void); void fa526_cpu_sleep (int); void fa526_tlb_flushI_SE (u_int); void fa526_tlb_flushID_SE (u_int); void fa526_flush_prefetchbuf (void); void fa526_flush_brnchtgt_E (u_int); void fa526_icache_sync_all (void); void fa526_icache_sync_range(vm_offset_t start, vm_size_t end); void fa526_dcache_wbinv_all (void); void fa526_dcache_wbinv_range(vm_offset_t start, vm_size_t end); void fa526_dcache_inv_range (vm_offset_t start, vm_size_t end); void fa526_dcache_wb_range (vm_offset_t start, vm_size_t end); void fa526_idcache_wbinv_all(void); void fa526_idcache_wbinv_range(vm_offset_t start, vm_size_t end); #endif #ifdef CPU_ARM9 void arm9_setttb (u_int); void arm9_tlb_flushID_SE (u_int va); void arm9_icache_sync_all (void); void arm9_icache_sync_range (vm_offset_t, vm_size_t); void arm9_dcache_wbinv_all (void); void arm9_dcache_wbinv_range (vm_offset_t, vm_size_t); void arm9_dcache_inv_range (vm_offset_t, vm_size_t); void arm9_dcache_wb_range (vm_offset_t, vm_size_t); void arm9_idcache_wbinv_all (void); void arm9_idcache_wbinv_range (vm_offset_t, vm_size_t); void arm9_context_switch (void); void arm9_setup (void); extern unsigned arm9_dcache_sets_max; extern unsigned arm9_dcache_sets_inc; extern unsigned arm9_dcache_index_max; extern unsigned arm9_dcache_index_inc; #endif #if defined(CPU_ARM9E) void arm10_tlb_flushID_SE (u_int); void arm10_tlb_flushI_SE (u_int); void arm10_context_switch (void); void arm10_setup (void); u_int sheeva_control_ext (u_int, u_int); void sheeva_cpu_sleep (int); void sheeva_setttb (u_int); void sheeva_dcache_wbinv_range (vm_offset_t, vm_size_t); void sheeva_dcache_inv_range (vm_offset_t, vm_size_t); void sheeva_dcache_wb_range (vm_offset_t, vm_size_t); void sheeva_idcache_wbinv_range (vm_offset_t, vm_size_t); void sheeva_l2cache_wbinv_range (vm_offset_t, vm_size_t); void sheeva_l2cache_inv_range (vm_offset_t, vm_size_t); void sheeva_l2cache_wb_range (vm_offset_t, vm_size_t); void sheeva_l2cache_wbinv_all (void); #endif #if defined(CPU_MV_PJ4B) void armv6_idcache_wbinv_all (void); #endif #if defined(CPU_MV_PJ4B) || defined(CPU_CORTEXA) || defined(CPU_KRAIT) void armv7_setttb (u_int); void armv7_tlb_flushID (void); void armv7_tlb_flushID_SE (u_int); void armv7_icache_sync_all (void); void armv7_icache_sync_range (vm_offset_t, vm_size_t); void armv7_idcache_wbinv_range (vm_offset_t, vm_size_t); void armv7_idcache_inv_all (void); void armv7_dcache_wbinv_all (void); void armv7_idcache_wbinv_all (void); void armv7_dcache_wbinv_range (vm_offset_t, vm_size_t); void armv7_dcache_inv_range (vm_offset_t, vm_size_t); void armv7_dcache_wb_range (vm_offset_t, vm_size_t); void armv7_cpu_sleep (int); void armv7_setup (void); void armv7_context_switch (void); void armv7_drain_writebuf (void); void armv7_sev (void); u_int armv7_auxctrl (u_int, u_int); void armadaxp_idcache_wbinv_all (void); void cortexa_setup (void); #endif #if defined(CPU_MV_PJ4B) void pj4b_config (void); void pj4bv7_setup (void); #endif #if defined(CPU_ARM1176) void arm11_tlb_flushID (void); void arm11_tlb_flushID_SE (u_int); void arm11_tlb_flushI (void); void arm11_tlb_flushI_SE (u_int); void arm11_tlb_flushD (void); void arm11_tlb_flushD_SE (u_int va); void arm11_context_switch (void); void arm11_drain_writebuf (void); void armv6_dcache_wbinv_range (vm_offset_t, vm_size_t); void armv6_dcache_inv_range (vm_offset_t, vm_size_t); void armv6_dcache_wb_range (vm_offset_t, vm_size_t); void armv6_idcache_inv_all (void); void arm11x6_setttb (u_int); void arm11x6_idcache_wbinv_all (void); void arm11x6_dcache_wbinv_all (void); void arm11x6_icache_sync_all (void); void arm11x6_flush_prefetchbuf (void); void arm11x6_icache_sync_range (vm_offset_t, vm_size_t); void arm11x6_idcache_wbinv_range (vm_offset_t, vm_size_t); void arm11x6_setup (void); void arm11x6_sleep (int); /* no ref. for errata */ #endif #if defined(CPU_ARM9E) void armv5_ec_setttb(u_int); void armv5_ec_icache_sync_all(void); void armv5_ec_icache_sync_range(vm_offset_t, vm_size_t); void armv5_ec_dcache_wbinv_all(void); void armv5_ec_dcache_wbinv_range(vm_offset_t, vm_size_t); void armv5_ec_dcache_inv_range(vm_offset_t, vm_size_t); void armv5_ec_dcache_wb_range(vm_offset_t, vm_size_t); void armv5_ec_idcache_wbinv_all(void); void armv5_ec_idcache_wbinv_range(vm_offset_t, vm_size_t); #endif #if defined(CPU_ARM9) || defined(CPU_ARM9E) || \ defined(CPU_XSCALE_80321) || \ defined(CPU_FA526) || \ defined(CPU_XSCALE_PXA2X0) || defined(CPU_XSCALE_IXP425) || \ defined(CPU_XSCALE_80219) || defined(CPU_XSCALE_81342) void armv4_tlb_flushID (void); void armv4_tlb_flushI (void); void armv4_tlb_flushD (void); void armv4_tlb_flushD_SE (u_int va); void armv4_drain_writebuf (void); void armv4_idcache_inv_all (void); #endif #if defined(CPU_XSCALE_80321) || \ defined(CPU_XSCALE_PXA2X0) || defined(CPU_XSCALE_IXP425) || \ defined(CPU_XSCALE_80219) || defined(CPU_XSCALE_81342) void xscale_cpwait (void); void xscale_cpu_sleep (int mode); u_int xscale_control (u_int clear, u_int bic); void xscale_setttb (u_int ttb); void xscale_tlb_flushID_SE (u_int va); void xscale_cache_flushID (void); void xscale_cache_flushI (void); void xscale_cache_flushD (void); void xscale_cache_flushD_SE (u_int entry); void xscale_cache_cleanID (void); void xscale_cache_cleanD (void); void xscale_cache_cleanD_E (u_int entry); void xscale_cache_clean_minidata (void); void xscale_cache_purgeID (void); void xscale_cache_purgeID_E (u_int entry); void xscale_cache_purgeD (void); void xscale_cache_purgeD_E (u_int entry); void xscale_cache_syncI (void); void xscale_cache_cleanID_rng (vm_offset_t start, vm_size_t end); void xscale_cache_cleanD_rng (vm_offset_t start, vm_size_t end); void xscale_cache_purgeID_rng (vm_offset_t start, vm_size_t end); void xscale_cache_purgeD_rng (vm_offset_t start, vm_size_t end); void xscale_cache_syncI_rng (vm_offset_t start, vm_size_t end); void xscale_cache_flushD_rng (vm_offset_t start, vm_size_t end); void xscale_context_switch (void); void xscale_setup (void); #endif /* CPU_XSCALE_80321 || CPU_XSCALE_PXA2X0 || CPU_XSCALE_IXP425 CPU_XSCALE_80219 */ #ifdef CPU_XSCALE_81342 void xscalec3_l2cache_purge (void); void xscalec3_cache_purgeID (void); void xscalec3_cache_purgeD (void); void xscalec3_cache_cleanID (void); void xscalec3_cache_cleanD (void); void xscalec3_cache_syncI (void); void xscalec3_cache_purgeID_rng (vm_offset_t start, vm_size_t end); void xscalec3_cache_purgeD_rng (vm_offset_t start, vm_size_t end); void xscalec3_cache_cleanID_rng (vm_offset_t start, vm_size_t end); void xscalec3_cache_cleanD_rng (vm_offset_t start, vm_size_t end); void xscalec3_cache_syncI_rng (vm_offset_t start, vm_size_t end); void xscalec3_l2cache_flush_rng (vm_offset_t, vm_size_t); void xscalec3_l2cache_clean_rng (vm_offset_t start, vm_size_t end); void xscalec3_l2cache_purge_rng (vm_offset_t start, vm_size_t end); void xscalec3_setttb (u_int ttb); void xscalec3_context_switch (void); #endif /* CPU_XSCALE_81342 */ #define setttb cpu_setttb #define drain_writebuf cpu_drain_writebuf /* * Macros for manipulating CPU interrupts */ #if __ARM_ARCH < 6 #define __ARM_INTR_BITS (PSR_I | PSR_F) #else #define __ARM_INTR_BITS (PSR_I | PSR_F | PSR_A) #endif static __inline uint32_t __set_cpsr(uint32_t bic, uint32_t eor) { uint32_t tmp, ret; __asm __volatile( "mrs %0, cpsr\n" /* Get the CPSR */ "bic %1, %0, %2\n" /* Clear bits */ "eor %1, %1, %3\n" /* XOR bits */ "msr cpsr_xc, %1\n" /* Set the CPSR */ : "=&r" (ret), "=&r" (tmp) : "r" (bic), "r" (eor) : "memory"); return ret; } static __inline uint32_t disable_interrupts(uint32_t mask) { return (__set_cpsr(mask & __ARM_INTR_BITS, mask & __ARM_INTR_BITS)); } static __inline uint32_t enable_interrupts(uint32_t mask) { return (__set_cpsr(mask & __ARM_INTR_BITS, 0)); } static __inline uint32_t restore_interrupts(uint32_t old_cpsr) { return (__set_cpsr(__ARM_INTR_BITS, old_cpsr & __ARM_INTR_BITS)); } static __inline register_t intr_disable(void) { return (disable_interrupts(PSR_I | PSR_F)); } static __inline void intr_restore(register_t s) { restore_interrupts(s); } #undef __ARM_INTR_BITS /* * Functions to manipulate cpu r13 * (in arm/arm32/setstack.S) */ void set_stackptr (u_int mode, u_int address); u_int get_stackptr (u_int mode); /* * Miscellany */ int get_pc_str_offset (void); /* * CPU functions from locore.S */ void cpu_reset (void) __attribute__((__noreturn__)); /* * Cache info variables. */ /* PRIMARY CACHE VARIABLES */ extern int arm_picache_size; extern int arm_picache_line_size; extern int arm_picache_ways; extern int arm_pdcache_size; /* and unified */ extern int arm_pdcache_line_size; extern int arm_pdcache_ways; extern int arm_pcache_type; extern int arm_pcache_unified; extern int arm_dcache_align; extern int arm_dcache_align_mask; extern u_int arm_cache_level; extern u_int arm_cache_loc; extern u_int arm_cache_type[14]; #endif /* _KERNEL */ #endif /* _MACHINE_CPUFUNC_H_ */ /* End of cpufunc.h */ Index: user/ngie/make_check/sys/arm/include/machdep.h =================================================================== --- user/ngie/make_check/sys/arm/include/machdep.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/machdep.h (revision 295078) @@ -1,49 +1,51 @@ /* $NetBSD: machdep.h,v 1.7 2002/02/21 02:52:21 thorpej Exp $ */ /* $FreeBSD$ */ #ifndef _MACHDEP_BOOT_MACHDEP_H_ #define _MACHDEP_BOOT_MACHDEP_H_ +#include + /* Structs that need to be initialised by initarm */ -#ifdef ARM_NEW_PMAP +#if __ARM_ARCH >= 6 extern vm_offset_t irqstack; extern vm_offset_t undstack; extern vm_offset_t abtstack; #else struct pv_addr; extern struct pv_addr irqstack; extern struct pv_addr undstack; extern struct pv_addr abtstack; #endif /* Define various stack sizes in pages */ #define IRQ_STACK_SIZE 1 #define ABT_STACK_SIZE 1 #define UND_STACK_SIZE 1 /* misc prototypes used by the many arm machdeps */ struct trapframe; void arm_lock_cache_line(vm_offset_t); void init_proc0(vm_offset_t kstack); void halt(void); void abort_handler(struct trapframe *, int ); void set_stackptrs(int cpu); void undefinedinstruction_bounce(struct trapframe *); /* Early boot related helper functions */ struct arm_boot_params; vm_offset_t default_parse_boot_param(struct arm_boot_params *abp); vm_offset_t freebsd_parse_boot_param(struct arm_boot_params *abp); vm_offset_t linux_parse_boot_param(struct arm_boot_params *abp); vm_offset_t fake_preload_metadata(struct arm_boot_params *abp); vm_offset_t parse_boot_param(struct arm_boot_params *abp); void arm_generic_initclocks(void); /* Board-specific attributes */ void board_set_serial(uint64_t); void board_set_revision(uint32_t); int arm_predict_branch(void *, u_int, register_t, register_t *, u_int (*)(void*, int), u_int (*)(void*, vm_offset_t, u_int*)); #endif /* !_MACHINE_MACHDEP_H_ */ Index: user/ngie/make_check/sys/arm/include/param.h =================================================================== --- user/ngie/make_check/sys/arm/include/param.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/param.h (revision 295078) @@ -1,152 +1,151 @@ /*- * Copyright (c) 2001 David E. O'Brien * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)param.h 5.8 (Berkeley) 6/28/91 * $FreeBSD$ */ #ifndef _ARM_INCLUDE_PARAM_H_ #define _ARM_INCLUDE_PARAM_H_ /* * Machine dependent constants for StrongARM */ #include #include #define STACKALIGNBYTES (8 - 1) #define STACKALIGN(p) ((u_int)(p) & ~STACKALIGNBYTES) #define __PCI_REROUTE_INTERRUPT #if __ARM_ARCH >= 6 #define _V6_SUFFIX "v6" #else #define _V6_SUFFIX "" #endif #ifdef __ARM_PCS_VFP #define _HF_SUFFIX "hf" #else #define _HF_SUFFIX "" #endif #ifdef __ARM_BIG_ENDIAN #define _EB_SUFFIX "eb" #else #define _EB_SUFFIX "" #endif #ifndef MACHINE #define MACHINE "arm" #endif #ifndef MACHINE_ARCH #define MACHINE_ARCH "arm" _V6_SUFFIX _HF_SUFFIX _EB_SUFFIX #endif #if defined(SMP) || defined(KLD_MODULE) #ifndef MAXCPU #define MAXCPU 4 #endif #else #define MAXCPU 1 #endif /* SMP || KLD_MODULE */ #ifndef MAXMEMDOM #define MAXMEMDOM 1 #endif #define ALIGNBYTES _ALIGNBYTES #define ALIGN(p) _ALIGN(p) /* * ALIGNED_POINTER is a boolean macro that checks whether an address * is valid to fetch data elements of type t from on this architecture. * This does not reflect the optimal alignment, just the possibility * (within reasonable limits). */ #define ALIGNED_POINTER(p, t) ((((unsigned)(p)) & (sizeof(t)-1)) == 0) /* * CACHE_LINE_SIZE is the compile-time maximum cache line size for an * architecture. It should be used with appropriate caution. */ #define CACHE_LINE_SHIFT 6 #define CACHE_LINE_SIZE (1 << CACHE_LINE_SHIFT) #define PAGE_SHIFT 12 #define PAGE_SIZE (1 << PAGE_SHIFT) /* Page size */ #define PAGE_MASK (PAGE_SIZE - 1) -#define NPTEPG (PAGE_SIZE/(sizeof (pt_entry_t))) #define PDR_SHIFT 20 /* log2(NBPDR) */ #define NBPDR (1 << PDR_SHIFT) #define PDRMASK (NBPDR - 1) #define NPDEPG (1 << (32 - PDR_SHIFT)) #define MAXPAGESIZES 2 /* maximum number of supported page sizes */ #ifndef KSTACK_PAGES #define KSTACK_PAGES 2 #endif /* !KSTACK_PAGES */ #ifndef FPCONTEXTSIZE #define FPCONTEXTSIZE (0x100) #endif #ifndef KSTACK_GUARD_PAGES #define KSTACK_GUARD_PAGES 1 #endif /* !KSTACK_GUARD_PAGES */ #define USPACE_SVC_STACK_TOP (kstack_pages * PAGE_SIZE) /* * Mach derived conversion macros */ #define trunc_page(x) ((x) & ~PAGE_MASK) #define round_page(x) (((x) + PAGE_MASK) & ~PAGE_MASK) #define trunc_1mpage(x) ((unsigned)(x) & ~PDRMASK) #define round_1mpage(x) ((((unsigned)(x)) + PDRMASK) & ~PDRMASK) #define atop(x) ((unsigned)(x) >> PAGE_SHIFT) #define ptoa(x) ((unsigned)(x) << PAGE_SHIFT) #define arm32_btop(x) ((unsigned)(x) >> PAGE_SHIFT) #define arm32_ptob(x) ((unsigned)(x) << PAGE_SHIFT) #define pgtok(x) ((x) * (PAGE_SIZE / 1024)) #endif /* !_ARM_INCLUDE_PARAM_H_ */ Index: user/ngie/make_check/sys/arm/include/pmap-v6.h =================================================================== --- user/ngie/make_check/sys/arm/include/pmap-v6.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/pmap-v6.h (revision 295078) @@ -1,309 +1,299 @@ /*- * Copyright 2014 Svatopluk Kraus * Copyright 2014 Michal Meloun * Copyright (c) 1991 Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department and William Jolitz of UUNET Technologies Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * The ARM version of this file was more or less based on the i386 version, * which has the following provenance... * * Derived from hp300 version by Mike Hibler, this version by William * Jolitz uses a recursive map [a pde points to the page directory] to * map the page tables using the pagetables themselves. This is done to * reduce the impact on kernel virtual memory for lots of sparse address * space, and to reduce the cost of memory to each process. * * from: hp300: @(#)pmap.h 7.2 (Berkeley) 12/16/90 * from: @(#)pmap.h 7.4 (Berkeley) 5/12/91 * from: FreeBSD: src/sys/i386/include/pmap.h,v 1.70 2000/11/30 * * $FreeBSD$ */ #ifndef _MACHINE_PMAP_H_ #define _MACHINE_PMAP_H_ #include #include #include #include typedef uint32_t pt1_entry_t; /* L1 table entry */ typedef uint32_t pt2_entry_t; /* L2 table entry */ typedef uint32_t ttb_entry_t; /* TTB entry */ #ifdef _KERNEL #if 0 #define PMAP_PTE_NOCACHE // Use uncached page tables #endif /* * (1) During pmap bootstrap, physical pages for L2 page tables are * allocated in advance which are used for KVA continuous mapping * starting from KERNBASE. This makes things more simple. * (2) During vm subsystem initialization, only vm subsystem itself can * allocate physical memory safely. As pmap_map() is called during * this initialization, we must be prepared for that and have some * preallocated physical pages for L2 page tables. * * Note that some more pages for L2 page tables are preallocated too * for mappings laying above VM_MAX_KERNEL_ADDRESS. */ #ifndef NKPT2PG /* * The optimal way is to define this in board configuration as * definition here must be safe enough. It means really big. * * 1 GB KVA <=> 256 kernel L2 page table pages * * From real platforms: * 1 GB physical memory <=> 10 pages is enough * 2 GB physical memory <=> 21 pages is enough */ #define NKPT2PG 32 #endif extern vm_paddr_t phys_avail[]; extern vm_paddr_t dump_avail[]; extern char *_tmppt; /* poor name! */ extern vm_offset_t virtual_avail; extern vm_offset_t virtual_end; /* * Pmap stuff */ /* * This structure is used to hold a virtual<->physical address * association and is used mostly by bootstrap code */ struct pv_addr { SLIST_ENTRY(pv_addr) pv_list; vm_offset_t pv_va; vm_paddr_t pv_pa; }; #endif struct pv_entry; struct pv_chunk; struct md_page { TAILQ_HEAD(,pv_entry) pv_list; uint16_t pt2_wirecount[4]; int pat_mode; }; struct pmap { struct mtx pm_mtx; pt1_entry_t *pm_pt1; /* KVA of pt1 */ pt2_entry_t *pm_pt2tab; /* KVA of pt2 pages table */ TAILQ_HEAD(,pv_chunk) pm_pvchunk; /* list of mappings in pmap */ cpuset_t pm_active; /* active on cpus */ struct pmap_statistics pm_stats; /* pmap statictics */ LIST_ENTRY(pmap) pm_list; /* List of all pmaps */ }; typedef struct pmap *pmap_t; #ifdef _KERNEL extern struct pmap kernel_pmap_store; #define kernel_pmap (&kernel_pmap_store) #define PMAP_LOCK(pmap) mtx_lock(&(pmap)->pm_mtx) #define PMAP_LOCK_ASSERT(pmap, type) \ mtx_assert(&(pmap)->pm_mtx, (type)) #define PMAP_LOCK_DESTROY(pmap) mtx_destroy(&(pmap)->pm_mtx) #define PMAP_LOCK_INIT(pmap) mtx_init(&(pmap)->pm_mtx, "pmap", \ NULL, MTX_DEF | MTX_DUPOK) #define PMAP_LOCKED(pmap) mtx_owned(&(pmap)->pm_mtx) #define PMAP_MTX(pmap) (&(pmap)->pm_mtx) #define PMAP_TRYLOCK(pmap) mtx_trylock(&(pmap)->pm_mtx) #define PMAP_UNLOCK(pmap) mtx_unlock(&(pmap)->pm_mtx) #endif /* * For each vm_page_t, there is a list of all currently valid virtual * mappings of that page. An entry is a pv_entry_t, the list is pv_list. */ typedef struct pv_entry { vm_offset_t pv_va; /* virtual address for mapping */ TAILQ_ENTRY(pv_entry) pv_next; } *pv_entry_t; /* * pv_entries are allocated in chunks per-process. This avoids the * need to track per-pmap assignments. */ #define _NPCM 11 #define _NPCPV 336 struct pv_chunk { pmap_t pc_pmap; TAILQ_ENTRY(pv_chunk) pc_list; uint32_t pc_map[_NPCM]; /* bitmap; 1 = free */ TAILQ_ENTRY(pv_chunk) pc_lru; struct pv_entry pc_pventry[_NPCPV]; }; #ifdef _KERNEL struct pcb; extern ttb_entry_t pmap_kern_ttb; /* TTB for kernel pmap */ #define pmap_page_get_memattr(m) ((vm_memattr_t)(m)->md.pat_mode) #define pmap_page_is_write_mapped(m) (((m)->aflags & PGA_WRITEABLE) != 0) /* * Only the following functions or macros may be used before pmap_bootstrap() * is called: pmap_kenter(), pmap_kextract(), pmap_kremove(), vtophys(), and * vtopte2(). */ void pmap_bootstrap(vm_offset_t ); void pmap_kenter(vm_offset_t , vm_paddr_t ); void *pmap_kenter_temporary(vm_paddr_t , int ); void pmap_kremove(vm_offset_t); void *pmap_mapdev(vm_paddr_t, vm_size_t); void *pmap_mapdev_attr(vm_paddr_t, vm_size_t, int); boolean_t pmap_page_is_mapped(vm_page_t ); void pmap_page_set_memattr(vm_page_t , vm_memattr_t ); void pmap_unmapdev(vm_offset_t, vm_size_t); void pmap_kenter_device(vm_offset_t, vm_size_t, vm_paddr_t); void pmap_kremove_device(vm_offset_t, vm_size_t); void pmap_set_pcb_pagedir(pmap_t , struct pcb *); void pmap_tlb_flush(pmap_t , vm_offset_t ); void pmap_tlb_flush_range(pmap_t , vm_offset_t , vm_size_t ); void pmap_dcache_wb_range(vm_paddr_t , vm_size_t , vm_memattr_t ); vm_paddr_t pmap_kextract(vm_offset_t ); vm_paddr_t pmap_dump_kextract(vm_offset_t, pt2_entry_t *); int pmap_fault(pmap_t , vm_offset_t , uint32_t , int , bool); #define vtophys(va) pmap_kextract((vm_offset_t)(va)) void pmap_set_tex(void); void reinit_mmu(ttb_entry_t ttb, u_int aux_clr, u_int aux_set); /* * Pre-bootstrap epoch functions set. */ void pmap_bootstrap_prepare(vm_paddr_t ); vm_paddr_t pmap_preboot_get_pages(u_int ); void pmap_preboot_map_pages(vm_paddr_t , vm_offset_t , u_int ); vm_offset_t pmap_preboot_reserve_pages(u_int ); vm_offset_t pmap_preboot_get_vpages(u_int ); void pmap_preboot_map_attr(vm_paddr_t , vm_offset_t , vm_size_t , int , int ); static __inline void pmap_map_chunk(vm_offset_t l1pt, vm_offset_t va, vm_offset_t pa, vm_size_t size, int prot, int cache) { pmap_preboot_map_attr(pa, va, size, prot, cache); } /* * This structure is used by machine-dependent code to describe * static mappings of devices, created at bootstrap time. */ struct pmap_devmap { vm_offset_t pd_va; /* virtual address */ vm_paddr_t pd_pa; /* physical address */ vm_size_t pd_size; /* size of region */ vm_prot_t pd_prot; /* protection code */ int pd_cache; /* cache attributes */ }; void pmap_devmap_bootstrap(const struct pmap_devmap *); #endif /* _KERNEL */ // ----------------- TO BE DELETED --------------------------------------------- #include #ifdef _KERNEL /* * sys/arm/arm/elf_trampoline.c * sys/arm/arm/genassym.c * sys/arm/arm/machdep.c * sys/arm/arm/mp_machdep.c * sys/arm/arm/locore.S * sys/arm/arm/pmap.c * sys/arm/arm/swtch.S * sys/arm/at91/at91_machdep.c * sys/arm/cavium/cns11xx/econa_machdep.c * sys/arm/s3c2xx0/s3c24x0_machdep.c * sys/arm/xscale/ixp425/avila_machdep.c * sys/arm/xscale/i8134x/crb_machdep.c * sys/arm/xscale/i80321/ep80219_machdep.c * sys/arm/xscale/i80321/iq31244_machdep.c * sys/arm/xscale/pxa/pxa_machdep.c */ #define PMAP_DOMAIN_KERNEL 0 /* The kernel uses domain #0 */ /* * sys/arm/arm/cpufunc.c */ -void pmap_pte_init_mmu_v6(void); void vector_page_setprot(int); - - -/* - * sys/arm/arm/db_interface.c - * sys/arm/arm/machdep.c - * sys/arm/arm/minidump_machdep.c - * sys/arm/arm/pmap.c - */ -#define pmap_kernel() kernel_pmap /* * sys/arm/arm/bus_space_generic.c (just comment) * sys/arm/arm/devmap.c * sys/arm/arm/pmap.c (just comment) * sys/arm/at91/at91_machdep.c * sys/arm/cavium/cns11xx/econa_machdep.c * sys/arm/freescale/imx/imx6_machdep.c (just comment) * sys/arm/mv/orion/db88f5xxx.c * sys/arm/mv/mv_localbus.c * sys/arm/mv/mv_machdep.c * sys/arm/mv/mv_pci.c * sys/arm/s3c2xx0/s3c24x0_machdep.c * sys/arm/versatile/versatile_machdep.c * sys/arm/xscale/ixp425/avila_machdep.c * sys/arm/xscale/i8134x/crb_machdep.c * sys/arm/xscale/i80321/ep80219_machdep.c * sys/arm/xscale/i80321/iq31244_machdep.c * sys/arm/xscale/pxa/pxa_machdep.c */ #define PTE_DEVICE PTE2_ATTR_DEVICE #endif /* _KERNEL */ // ----------------------------------------------------------------------------- #endif /* !_MACHINE_PMAP_H_ */ Index: user/ngie/make_check/sys/arm/include/pmap.h =================================================================== --- user/ngie/make_check/sys/arm/include/pmap.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/pmap.h (revision 295078) @@ -1,707 +1,705 @@ /*- * Copyright (c) 1991 Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department and William Jolitz of UUNET Technologies Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * Derived from hp300 version by Mike Hibler, this version by William * Jolitz uses a recursive map [a pde points to the page directory] to * map the page tables using the pagetables themselves. This is done to * reduce the impact on kernel virtual memory for lots of sparse address * space, and to reduce the cost of memory to each process. * * from: hp300: @(#)pmap.h 7.2 (Berkeley) 12/16/90 * from: @(#)pmap.h 7.4 (Berkeley) 5/12/91 * from: FreeBSD: src/sys/i386/include/pmap.h,v 1.70 2000/11/30 * * $FreeBSD$ */ -#ifdef ARM_NEW_PMAP + #include + +#if __ARM_ARCH >= 6 #include -#else /* ARM_NEW_PMAP */ +#else /* __ARM_ARCH >= 6 */ #ifndef _MACHINE_PMAP_H_ #define _MACHINE_PMAP_H_ #include #include /* * Pte related macros */ #if ARM_ARCH_6 || ARM_ARCH_7A #ifdef SMP #define PTE_NOCACHE 2 #else #define PTE_NOCACHE 1 #endif #define PTE_CACHE 6 #define PTE_DEVICE 2 #define PTE_PAGETABLE 6 #else #define PTE_NOCACHE 1 #define PTE_CACHE 2 #define PTE_DEVICE PTE_NOCACHE #define PTE_PAGETABLE 3 #endif enum mem_type { STRONG_ORD = 0, DEVICE_NOSHARE, DEVICE_SHARE, NRML_NOCACHE, NRML_IWT_OWT, NRML_IWB_OWB, NRML_IWBA_OWBA }; #ifndef LOCORE #include #include #include #include #define PDESIZE sizeof(pd_entry_t) /* for assembly files */ #define PTESIZE sizeof(pt_entry_t) /* for assembly files */ #ifdef _KERNEL #define vtophys(va) pmap_kextract((vm_offset_t)(va)) #endif #define pmap_page_get_memattr(m) ((m)->md.pv_memattr) #define pmap_page_is_write_mapped(m) (((m)->aflags & PGA_WRITEABLE) != 0) #if (ARM_MMU_V6 + ARM_MMU_V7) > 0 boolean_t pmap_page_is_mapped(vm_page_t); #else #define pmap_page_is_mapped(m) (!TAILQ_EMPTY(&(m)->md.pv_list)) #endif void pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma); /* * Pmap stuff */ /* * This structure is used to hold a virtual<->physical address * association and is used mostly by bootstrap code */ struct pv_addr { SLIST_ENTRY(pv_addr) pv_list; vm_offset_t pv_va; vm_paddr_t pv_pa; }; struct pv_entry; struct pv_chunk; struct md_page { int pvh_attrs; vm_memattr_t pv_memattr; #if (ARM_MMU_V6 + ARM_MMU_V7) == 0 vm_offset_t pv_kva; /* first kernel VA mapping */ #endif TAILQ_HEAD(,pv_entry) pv_list; }; struct l1_ttable; struct l2_dtable; /* * The number of L2 descriptor tables which can be tracked by an l2_dtable. * A bucket size of 16 provides for 16MB of contiguous virtual address * space per l2_dtable. Most processes will, therefore, require only two or * three of these to map their whole working set. */ #define L2_BUCKET_LOG2 4 #define L2_BUCKET_SIZE (1 << L2_BUCKET_LOG2) /* * Given the above "L2-descriptors-per-l2_dtable" constant, the number * of l2_dtable structures required to track all possible page descriptors * mappable by an L1 translation table is given by the following constants: */ #define L2_LOG2 ((32 - L1_S_SHIFT) - L2_BUCKET_LOG2) #define L2_SIZE (1 << L2_LOG2) struct pmap { struct mtx pm_mtx; u_int8_t pm_domain; struct l1_ttable *pm_l1; struct l2_dtable *pm_l2[L2_SIZE]; cpuset_t pm_active; /* active on cpus */ struct pmap_statistics pm_stats; /* pmap statictics */ #if (ARM_MMU_V6 + ARM_MMU_V7) != 0 TAILQ_HEAD(,pv_chunk) pm_pvchunk; /* list of mappings in pmap */ #else TAILQ_HEAD(,pv_entry) pm_pvlist; /* list of mappings in pmap */ #endif }; typedef struct pmap *pmap_t; #ifdef _KERNEL extern struct pmap kernel_pmap_store; #define kernel_pmap (&kernel_pmap_store) -#define pmap_kernel() kernel_pmap #define PMAP_ASSERT_LOCKED(pmap) \ mtx_assert(&(pmap)->pm_mtx, MA_OWNED) #define PMAP_LOCK(pmap) mtx_lock(&(pmap)->pm_mtx) #define PMAP_LOCK_DESTROY(pmap) mtx_destroy(&(pmap)->pm_mtx) #define PMAP_LOCK_INIT(pmap) mtx_init(&(pmap)->pm_mtx, "pmap", \ NULL, MTX_DEF | MTX_DUPOK) #define PMAP_OWNED(pmap) mtx_owned(&(pmap)->pm_mtx) #define PMAP_MTX(pmap) (&(pmap)->pm_mtx) #define PMAP_TRYLOCK(pmap) mtx_trylock(&(pmap)->pm_mtx) #define PMAP_UNLOCK(pmap) mtx_unlock(&(pmap)->pm_mtx) #endif /* * For each vm_page_t, there is a list of all currently valid virtual * mappings of that page. An entry is a pv_entry_t, the list is pv_list. */ typedef struct pv_entry { vm_offset_t pv_va; /* virtual address for mapping */ TAILQ_ENTRY(pv_entry) pv_list; int pv_flags; /* flags (wired, etc...) */ #if (ARM_MMU_V6 + ARM_MMU_V7) == 0 pmap_t pv_pmap; /* pmap where mapping lies */ TAILQ_ENTRY(pv_entry) pv_plist; #endif } *pv_entry_t; /* * pv_entries are allocated in chunks per-process. This avoids the * need to track per-pmap assignments. */ #define _NPCM 8 #define _NPCPV 252 struct pv_chunk { pmap_t pc_pmap; TAILQ_ENTRY(pv_chunk) pc_list; uint32_t pc_map[_NPCM]; /* bitmap; 1 = free */ uint32_t pc_dummy[3]; /* aligns pv_chunk to 4KB */ TAILQ_ENTRY(pv_chunk) pc_lru; struct pv_entry pc_pventry[_NPCPV]; }; #ifdef _KERNEL boolean_t pmap_get_pde_pte(pmap_t, vm_offset_t, pd_entry_t **, pt_entry_t **); /* * virtual address to page table entry and * to physical address. Likewise for alternate address space. * Note: these work recursively, thus vtopte of a pte will give * the corresponding pde that in turn maps it. */ /* * The current top of kernel VM. */ extern vm_offset_t pmap_curmaxkvaddr; struct pcb; void pmap_set_pcb_pagedir(pmap_t, struct pcb *); /* Virtual address to page table entry */ static __inline pt_entry_t * vtopte(vm_offset_t va) { pd_entry_t *pdep; pt_entry_t *ptep; - if (pmap_get_pde_pte(pmap_kernel(), va, &pdep, &ptep) == FALSE) + if (pmap_get_pde_pte(kernel_pmap, va, &pdep, &ptep) == FALSE) return (NULL); return (ptep); } extern vm_paddr_t phys_avail[]; extern vm_offset_t virtual_avail; extern vm_offset_t virtual_end; void pmap_bootstrap(vm_offset_t firstaddr, struct pv_addr *l1pt); int pmap_change_attr(vm_offset_t, vm_size_t, int); void pmap_kenter(vm_offset_t va, vm_paddr_t pa); void pmap_kenter_nocache(vm_offset_t va, vm_paddr_t pa); void pmap_kenter_device(vm_offset_t, vm_size_t, vm_paddr_t); void pmap_kremove_device(vm_offset_t, vm_size_t); void *pmap_kenter_temporary(vm_paddr_t pa, int i); void pmap_kenter_user(vm_offset_t va, vm_paddr_t pa); vm_paddr_t pmap_kextract(vm_offset_t va); vm_paddr_t pmap_dump_kextract(vm_offset_t, pt2_entry_t *); void pmap_kremove(vm_offset_t); void *pmap_mapdev(vm_offset_t, vm_size_t); void pmap_unmapdev(vm_offset_t, vm_size_t); vm_page_t pmap_use_pt(pmap_t, vm_offset_t); void pmap_debug(int); #if (ARM_MMU_V6 + ARM_MMU_V7) == 0 void pmap_map_section(vm_offset_t, vm_offset_t, vm_offset_t, int, int); #endif void pmap_link_l2pt(vm_offset_t, vm_offset_t, struct pv_addr *); vm_size_t pmap_map_chunk(vm_offset_t, vm_offset_t, vm_offset_t, vm_size_t, int, int); void pmap_map_entry(vm_offset_t l1pt, vm_offset_t va, vm_offset_t pa, int prot, int cache); int pmap_fault_fixup(pmap_t, vm_offset_t, vm_prot_t, int); /* * Definitions for MMU domains */ #define PMAP_DOMAINS 15 /* 15 'user' domains (1-15) */ #define PMAP_DOMAIN_KERNEL 0 /* The kernel uses domain #0 */ /* * The new pmap ensures that page-tables are always mapping Write-Thru. * Thus, on some platforms we can run fast and loose and avoid syncing PTEs * on every change. * * Unfortunately, not all CPUs have a write-through cache mode. So we * define PMAP_NEEDS_PTE_SYNC for C code to conditionally do PTE syncs, * and if there is the chance for PTE syncs to be needed, we define * PMAP_INCLUDE_PTE_SYNC so e.g. assembly code can include (and run) * the code. */ extern int pmap_needs_pte_sync; /* * These macros define the various bit masks in the PTE. * * We use these macros since we use different bits on different processor * models. */ #define L1_S_CACHE_MASK_generic (L1_S_B|L1_S_C) #define L1_S_CACHE_MASK_xscale (L1_S_B|L1_S_C|L1_S_XSCALE_TEX(TEX_XSCALE_X)|\ L1_S_XSCALE_TEX(TEX_XSCALE_T)) #define L2_L_CACHE_MASK_generic (L2_B|L2_C) #define L2_L_CACHE_MASK_xscale (L2_B|L2_C|L2_XSCALE_L_TEX(TEX_XSCALE_X) | \ L2_XSCALE_L_TEX(TEX_XSCALE_T)) #define L2_S_PROT_U_generic (L2_AP(AP_U)) #define L2_S_PROT_W_generic (L2_AP(AP_W)) #define L2_S_PROT_MASK_generic (L2_S_PROT_U|L2_S_PROT_W) #define L2_S_PROT_U_xscale (L2_AP0(AP_U)) #define L2_S_PROT_W_xscale (L2_AP0(AP_W)) #define L2_S_PROT_MASK_xscale (L2_S_PROT_U|L2_S_PROT_W) #define L2_S_CACHE_MASK_generic (L2_B|L2_C) #define L2_S_CACHE_MASK_xscale (L2_B|L2_C|L2_XSCALE_T_TEX(TEX_XSCALE_X)| \ L2_XSCALE_T_TEX(TEX_XSCALE_X)) #define L1_S_PROTO_generic (L1_TYPE_S | L1_S_IMP) #define L1_S_PROTO_xscale (L1_TYPE_S) #define L1_C_PROTO_generic (L1_TYPE_C | L1_C_IMP2) #define L1_C_PROTO_xscale (L1_TYPE_C) #define L2_L_PROTO (L2_TYPE_L) #define L2_S_PROTO_generic (L2_TYPE_S) #define L2_S_PROTO_xscale (L2_TYPE_XSCALE_XS) /* * User-visible names for the ones that vary with MMU class. */ #if (ARM_MMU_V6 + ARM_MMU_V7) != 0 #define L2_AP(x) (L2_AP0(x)) #else #define L2_AP(x) (L2_AP0(x) | L2_AP1(x) | L2_AP2(x) | L2_AP3(x)) #endif #if (ARM_MMU_V6 + ARM_MMU_V7) != 0 /* * AP[2:1] access permissions model: * * AP[2](APX) - Write Disable * AP[1] - User Enable * AP[0] - Reference Flag * * AP[2] AP[1] Kernel User * 0 0 R/W N * 0 1 R/W R/W * 1 0 R N * 1 1 R R * */ #define L2_S_PROT_R (0) /* kernel read */ #define L2_S_PROT_U (L2_AP0(2)) /* user read */ #define L2_S_REF (L2_AP0(1)) /* reference flag */ #define L2_S_PROT_MASK (L2_S_PROT_U|L2_S_PROT_R|L2_APX) #define L2_S_EXECUTABLE(pte) (!(pte & L2_XN)) #define L2_S_WRITABLE(pte) (!(pte & L2_APX)) #define L2_S_REFERENCED(pte) (!!(pte & L2_S_REF)) #ifndef SMP #define L1_S_CACHE_MASK (L1_S_TEX_MASK|L1_S_B|L1_S_C) #define L2_L_CACHE_MASK (L2_L_TEX_MASK|L2_B|L2_C) #define L2_S_CACHE_MASK (L2_S_TEX_MASK|L2_B|L2_C) #else #define L1_S_CACHE_MASK (L1_S_TEX_MASK|L1_S_B|L1_S_C|L1_SHARED) #define L2_L_CACHE_MASK (L2_L_TEX_MASK|L2_B|L2_C|L2_SHARED) #define L2_S_CACHE_MASK (L2_S_TEX_MASK|L2_B|L2_C|L2_SHARED) #endif /* SMP */ #define L1_S_PROTO (L1_TYPE_S) #define L1_C_PROTO (L1_TYPE_C) #define L2_S_PROTO (L2_TYPE_S) /* * Promotion to a 1MB (SECTION) mapping requires that the corresponding * 4KB (SMALL) page mappings have identical settings for the following fields: */ #define L2_S_PROMOTE (L2_S_REF | L2_SHARED | L2_S_PROT_MASK | \ L2_XN | L2_S_PROTO) /* * In order to compare 1MB (SECTION) entry settings with the 4KB (SMALL) * page mapping it is necessary to read and shift appropriate bits from * L1 entry to positions of the corresponding bits in the L2 entry. */ #define L1_S_DEMOTE(l1pd) ((((l1pd) & L1_S_PROTO) >> 0) | \ (((l1pd) & L1_SHARED) >> 6) | \ (((l1pd) & L1_S_REF) >> 6) | \ (((l1pd) & L1_S_PROT_MASK) >> 6) | \ (((l1pd) & L1_S_XN) >> 4)) #ifndef SMP #define ARM_L1S_STRONG_ORD (0) #define ARM_L1S_DEVICE_NOSHARE (L1_S_TEX(2)) #define ARM_L1S_DEVICE_SHARE (L1_S_B) #define ARM_L1S_NRML_NOCACHE (L1_S_TEX(1)) #define ARM_L1S_NRML_IWT_OWT (L1_S_C) #define ARM_L1S_NRML_IWB_OWB (L1_S_C|L1_S_B) #define ARM_L1S_NRML_IWBA_OWBA (L1_S_TEX(1)|L1_S_C|L1_S_B) #define ARM_L2L_STRONG_ORD (0) #define ARM_L2L_DEVICE_NOSHARE (L2_L_TEX(2)) #define ARM_L2L_DEVICE_SHARE (L2_B) #define ARM_L2L_NRML_NOCACHE (L2_L_TEX(1)) #define ARM_L2L_NRML_IWT_OWT (L2_C) #define ARM_L2L_NRML_IWB_OWB (L2_C|L2_B) #define ARM_L2L_NRML_IWBA_OWBA (L2_L_TEX(1)|L2_C|L2_B) #define ARM_L2S_STRONG_ORD (0) #define ARM_L2S_DEVICE_NOSHARE (L2_S_TEX(2)) #define ARM_L2S_DEVICE_SHARE (L2_B) #define ARM_L2S_NRML_NOCACHE (L2_S_TEX(1)) #define ARM_L2S_NRML_IWT_OWT (L2_C) #define ARM_L2S_NRML_IWB_OWB (L2_C|L2_B) #define ARM_L2S_NRML_IWBA_OWBA (L2_S_TEX(1)|L2_C|L2_B) #else #define ARM_L1S_STRONG_ORD (0) #define ARM_L1S_DEVICE_NOSHARE (L1_S_TEX(2)) #define ARM_L1S_DEVICE_SHARE (L1_S_B) #define ARM_L1S_NRML_NOCACHE (L1_S_TEX(1)|L1_SHARED) #define ARM_L1S_NRML_IWT_OWT (L1_S_C|L1_SHARED) #define ARM_L1S_NRML_IWB_OWB (L1_S_C|L1_S_B|L1_SHARED) #define ARM_L1S_NRML_IWBA_OWBA (L1_S_TEX(1)|L1_S_C|L1_S_B|L1_SHARED) #define ARM_L2L_STRONG_ORD (0) #define ARM_L2L_DEVICE_NOSHARE (L2_L_TEX(2)) #define ARM_L2L_DEVICE_SHARE (L2_B) #define ARM_L2L_NRML_NOCACHE (L2_L_TEX(1)|L2_SHARED) #define ARM_L2L_NRML_IWT_OWT (L2_C|L2_SHARED) #define ARM_L2L_NRML_IWB_OWB (L2_C|L2_B|L2_SHARED) #define ARM_L2L_NRML_IWBA_OWBA (L2_L_TEX(1)|L2_C|L2_B|L2_SHARED) #define ARM_L2S_STRONG_ORD (0) #define ARM_L2S_DEVICE_NOSHARE (L2_S_TEX(2)) #define ARM_L2S_DEVICE_SHARE (L2_B) #define ARM_L2S_NRML_NOCACHE (L2_S_TEX(1)|L2_SHARED) #define ARM_L2S_NRML_IWT_OWT (L2_C|L2_SHARED) #define ARM_L2S_NRML_IWB_OWB (L2_C|L2_B|L2_SHARED) #define ARM_L2S_NRML_IWBA_OWBA (L2_S_TEX(1)|L2_C|L2_B|L2_SHARED) #endif /* SMP */ #elif ARM_NMMUS > 1 /* More than one MMU class configured; use variables. */ #define L2_S_PROT_U pte_l2_s_prot_u #define L2_S_PROT_W pte_l2_s_prot_w #define L2_S_PROT_MASK pte_l2_s_prot_mask #define L1_S_CACHE_MASK pte_l1_s_cache_mask #define L2_L_CACHE_MASK pte_l2_l_cache_mask #define L2_S_CACHE_MASK pte_l2_s_cache_mask #define L1_S_PROTO pte_l1_s_proto #define L1_C_PROTO pte_l1_c_proto #define L2_S_PROTO pte_l2_s_proto #elif ARM_MMU_GENERIC != 0 #define L2_S_PROT_U L2_S_PROT_U_generic #define L2_S_PROT_W L2_S_PROT_W_generic #define L2_S_PROT_MASK L2_S_PROT_MASK_generic #define L1_S_CACHE_MASK L1_S_CACHE_MASK_generic #define L2_L_CACHE_MASK L2_L_CACHE_MASK_generic #define L2_S_CACHE_MASK L2_S_CACHE_MASK_generic #define L1_S_PROTO L1_S_PROTO_generic #define L1_C_PROTO L1_C_PROTO_generic #define L2_S_PROTO L2_S_PROTO_generic #elif ARM_MMU_XSCALE == 1 #define L2_S_PROT_U L2_S_PROT_U_xscale #define L2_S_PROT_W L2_S_PROT_W_xscale #define L2_S_PROT_MASK L2_S_PROT_MASK_xscale #define L1_S_CACHE_MASK L1_S_CACHE_MASK_xscale #define L2_L_CACHE_MASK L2_L_CACHE_MASK_xscale #define L2_S_CACHE_MASK L2_S_CACHE_MASK_xscale #define L1_S_PROTO L1_S_PROTO_xscale #define L1_C_PROTO L1_C_PROTO_xscale #define L2_S_PROTO L2_S_PROTO_xscale #endif /* ARM_NMMUS > 1 */ #if defined(CPU_XSCALE_81342) || ARM_ARCH_6 || ARM_ARCH_7A #define PMAP_NEEDS_PTE_SYNC 1 #define PMAP_INCLUDE_PTE_SYNC #else #define PMAP_NEEDS_PTE_SYNC 0 #endif /* * These macros return various bits based on kernel/user and protection. * Note that the compiler will usually fold these at compile time. */ #if (ARM_MMU_V6 + ARM_MMU_V7) == 0 #define L1_S_PROT_U (L1_S_AP(AP_U)) #define L1_S_PROT_W (L1_S_AP(AP_W)) #define L1_S_PROT_MASK (L1_S_PROT_U|L1_S_PROT_W) #define L1_S_WRITABLE(pd) ((pd) & L1_S_PROT_W) #define L1_S_PROT(ku, pr) ((((ku) == PTE_USER) ? L1_S_PROT_U : 0) | \ (((pr) & VM_PROT_WRITE) ? L1_S_PROT_W : 0)) #define L2_L_PROT_U (L2_AP(AP_U)) #define L2_L_PROT_W (L2_AP(AP_W)) #define L2_L_PROT_MASK (L2_L_PROT_U|L2_L_PROT_W) #define L2_L_PROT(ku, pr) ((((ku) == PTE_USER) ? L2_L_PROT_U : 0) | \ (((pr) & VM_PROT_WRITE) ? L2_L_PROT_W : 0)) #define L2_S_PROT(ku, pr) ((((ku) == PTE_USER) ? L2_S_PROT_U : 0) | \ (((pr) & VM_PROT_WRITE) ? L2_S_PROT_W : 0)) #else #define L1_S_PROT_U (L1_S_AP(AP_U)) #define L1_S_PROT_W (L1_S_APX) /* Write disable */ #define L1_S_PROT_MASK (L1_S_PROT_W|L1_S_PROT_U) #define L1_S_REF (L1_S_AP(AP_REF)) /* Reference flag */ #define L1_S_WRITABLE(pd) (!((pd) & L1_S_PROT_W)) #define L1_S_EXECUTABLE(pd) (!((pd) & L1_S_XN)) #define L1_S_REFERENCED(pd) ((pd) & L1_S_REF) #define L1_S_PROT(ku, pr) (((((ku) == PTE_KERNEL) ? 0 : L1_S_PROT_U) | \ (((pr) & VM_PROT_WRITE) ? 0 : L1_S_PROT_W) | \ (((pr) & VM_PROT_EXECUTE) ? 0 : L1_S_XN))) #define L2_L_PROT_MASK (L2_APX|L2_AP0(0x3)) #define L2_L_PROT(ku, pr) (L2_L_PROT_MASK & ~((((ku) == PTE_KERNEL) ? L2_S_PROT_U : 0) | \ (((pr) & VM_PROT_WRITE) ? L2_APX : 0))) #define L2_S_PROT(ku, pr) (L2_S_PROT_MASK & ~((((ku) == PTE_KERNEL) ? L2_S_PROT_U : 0) | \ (((pr) & VM_PROT_WRITE) ? L2_APX : 0))) #endif /* * Macros to test if a mapping is mappable with an L1 Section mapping * or an L2 Large Page mapping. */ #define L1_S_MAPPABLE_P(va, pa, size) \ ((((va) | (pa)) & L1_S_OFFSET) == 0 && (size) >= L1_S_SIZE) #define L2_L_MAPPABLE_P(va, pa, size) \ ((((va) | (pa)) & L2_L_OFFSET) == 0 && (size) >= L2_L_SIZE) /* * Provide a fallback in case we were not able to determine it at * compile-time. */ #ifndef PMAP_NEEDS_PTE_SYNC #define PMAP_NEEDS_PTE_SYNC pmap_needs_pte_sync #define PMAP_INCLUDE_PTE_SYNC #endif #ifdef ARM_L2_PIPT #define _sync_l2(pte, size) cpu_l2cache_wb_range(vtophys(pte), size) #else #define _sync_l2(pte, size) cpu_l2cache_wb_range(pte, size) #endif #define PTE_SYNC(pte) \ do { \ if (PMAP_NEEDS_PTE_SYNC) { \ cpu_dcache_wb_range((vm_offset_t)(pte), sizeof(pt_entry_t));\ cpu_drain_writebuf(); \ _sync_l2((vm_offset_t)(pte), sizeof(pt_entry_t));\ } else \ cpu_drain_writebuf(); \ } while (/*CONSTCOND*/0) #define PTE_SYNC_RANGE(pte, cnt) \ do { \ if (PMAP_NEEDS_PTE_SYNC) { \ cpu_dcache_wb_range((vm_offset_t)(pte), \ (cnt) << 2); /* * sizeof(pt_entry_t) */ \ cpu_drain_writebuf(); \ _sync_l2((vm_offset_t)(pte), \ (cnt) << 2); /* * sizeof(pt_entry_t) */ \ } else \ cpu_drain_writebuf(); \ } while (/*CONSTCOND*/0) extern pt_entry_t pte_l1_s_cache_mode; extern pt_entry_t pte_l1_s_cache_mask; extern pt_entry_t pte_l2_l_cache_mode; extern pt_entry_t pte_l2_l_cache_mask; extern pt_entry_t pte_l2_s_cache_mode; extern pt_entry_t pte_l2_s_cache_mask; extern pt_entry_t pte_l1_s_cache_mode_pt; extern pt_entry_t pte_l2_l_cache_mode_pt; extern pt_entry_t pte_l2_s_cache_mode_pt; extern pt_entry_t pte_l2_s_prot_u; extern pt_entry_t pte_l2_s_prot_w; extern pt_entry_t pte_l2_s_prot_mask; extern pt_entry_t pte_l1_s_proto; extern pt_entry_t pte_l1_c_proto; extern pt_entry_t pte_l2_s_proto; extern void (*pmap_copy_page_func)(vm_paddr_t, vm_paddr_t); extern void (*pmap_copy_page_offs_func)(vm_paddr_t a_phys, vm_offset_t a_offs, vm_paddr_t b_phys, vm_offset_t b_offs, int cnt); extern void (*pmap_zero_page_func)(vm_paddr_t, int, int); #if (ARM_MMU_GENERIC + ARM_MMU_V6 + ARM_MMU_V7) != 0 || defined(CPU_XSCALE_81342) void pmap_copy_page_generic(vm_paddr_t, vm_paddr_t); void pmap_zero_page_generic(vm_paddr_t, int, int); void pmap_pte_init_generic(void); -#if (ARM_MMU_V6 + ARM_MMU_V7) != 0 -void pmap_pte_init_mmu_v6(void); -#endif /* (ARM_MMU_V6 + ARM_MMU_V7) != 0 */ #endif /* (ARM_MMU_GENERIC + ARM_MMU_V6 + ARM_MMU_V7) != 0 */ #if ARM_MMU_XSCALE == 1 void pmap_copy_page_xscale(vm_paddr_t, vm_paddr_t); void pmap_zero_page_xscale(vm_paddr_t, int, int); void pmap_pte_init_xscale(void); void xscale_setup_minidata(vm_offset_t, vm_offset_t, vm_offset_t); void pmap_use_minicache(vm_offset_t, vm_size_t); #endif /* ARM_MMU_XSCALE == 1 */ #if defined(CPU_XSCALE_81342) #define ARM_HAVE_SUPERSECTIONS #endif #define PTE_KERNEL 0 #define PTE_USER 1 #define l1pte_valid(pde) ((pde) != 0) #define l1pte_section_p(pde) (((pde) & L1_TYPE_MASK) == L1_TYPE_S) #define l1pte_page_p(pde) (((pde) & L1_TYPE_MASK) == L1_TYPE_C) #define l1pte_fpage_p(pde) (((pde) & L1_TYPE_MASK) == L1_TYPE_F) #define l2pte_index(v) (((v) & L2_ADDR_BITS) >> L2_S_SHIFT) #define l2pte_valid(pte) ((pte) != 0) #define l2pte_pa(pte) ((pte) & L2_S_FRAME) #define l2pte_minidata(pte) (((pte) & \ (L2_B | L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X)))\ == (L2_C | L2_XSCALE_T_TEX(TEX_XSCALE_X))) /* L1 and L2 page table macros */ #define pmap_pde_v(pde) l1pte_valid(*(pde)) #define pmap_pde_section(pde) l1pte_section_p(*(pde)) #define pmap_pde_page(pde) l1pte_page_p(*(pde)) #define pmap_pde_fpage(pde) l1pte_fpage_p(*(pde)) #define pmap_pte_v(pte) l2pte_valid(*(pte)) #define pmap_pte_pa(pte) l2pte_pa(*(pte)) /* * Flags that indicate attributes of pages or mappings of pages. * * The PVF_MOD and PVF_REF flags are stored in the mdpage for each * page. PVF_WIRED, PVF_WRITE, and PVF_NC are kept in individual * pv_entry's for each page. They live in the same "namespace" so * that we can clear multiple attributes at a time. * * Note the "non-cacheable" flag generally means the page has * multiple mappings in a given address space. */ #define PVF_MOD 0x01 /* page is modified */ #define PVF_REF 0x02 /* page is referenced */ #define PVF_WIRED 0x04 /* mapping is wired */ #define PVF_WRITE 0x08 /* mapping is writable */ #define PVF_EXEC 0x10 /* mapping is executable */ #define PVF_NC 0x20 /* mapping is non-cacheable */ #define PVF_MWC 0x40 /* mapping is used multiple times in userland */ #define PVF_UNMAN 0x80 /* mapping is unmanaged */ void vector_page_setprot(int); #define SECTION_CACHE 0x1 #define SECTION_PT 0x2 void pmap_kenter_section(vm_offset_t, vm_paddr_t, int flags); #ifdef ARM_HAVE_SUPERSECTIONS void pmap_kenter_supersection(vm_offset_t, uint64_t, int flags); #endif extern char *_tmppt; void pmap_postinit(void); extern vm_paddr_t dump_avail[]; #endif /* _KERNEL */ #endif /* !LOCORE */ #endif /* !_MACHINE_PMAP_H_ */ -#endif /* !ARM_NEW_PMAP */ +#endif /* __ARM_ARCH >= 6 */ Index: user/ngie/make_check/sys/arm/include/pte.h =================================================================== --- user/ngie/make_check/sys/arm/include/pte.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/pte.h (revision 295078) @@ -1,361 +1,363 @@ /* $NetBSD: pte.h,v 1.1 2001/11/23 17:39:04 thorpej Exp $ */ /*- * Copyright (c) 1994 Mark Brinicombe. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the RiscBSD team. * 4. The name "RiscBSD" nor the name of the author may be used to * endorse or promote products derived from this software without specific * prior written permission. * * THIS SOFTWARE IS PROVIDED BY RISCBSD ``AS IS'' AND ANY EXPRESS OR IMPLIED * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL RISCBSD OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, * INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ -#ifdef ARM_NEW_PMAP +#include + +#if __ARM_ARCH >= 6 #include -#else /* ARM_NEW_PMAP */ +#else /* __ARM_ARCH >= 6 */ #ifndef _MACHINE_PTE_H_ #define _MACHINE_PTE_H_ #ifndef LOCORE typedef uint32_t pd_entry_t; /* page directory entry */ typedef uint32_t pt_entry_t; /* page table entry */ typedef pt_entry_t pt2_entry_t; /* compatibility with v6 */ #endif #define PG_FRAME 0xfffff000 /* The PT_SIZE definition is misleading... A page table is only 0x400 * bytes long. But since VM mapping can only be done to 0x1000 a single * 1KB blocks cannot be steered to a va by itself. Therefore the * pages tables are allocated in blocks of 4. i.e. if a 1 KB block * was allocated for a PT then the other 3KB would also get mapped * whenever the 1KB was mapped. */ #define PT_RSIZE 0x0400 /* Real page table size */ #define PT_SIZE 0x1000 #define PD_SIZE 0x4000 /* Page table types and masks */ #define L1_PAGE 0x01 /* L1 page table mapping */ #define L1_SECTION 0x02 /* L1 section mapping */ #define L1_FPAGE 0x03 /* L1 fine page mapping */ #define L1_MASK 0x03 /* Mask for L1 entry type */ #define L2_LPAGE 0x01 /* L2 large page (64KB) */ #define L2_SPAGE 0x02 /* L2 small page (4KB) */ #define L2_MASK 0x03 /* Mask for L2 entry type */ #define L2_INVAL 0x00 /* L2 invalid type */ /* L1 and L2 address masks */ #define L1_ADDR_MASK 0xfffffc00 #define L2_ADDR_MASK 0xfffff000 /* * The ARM MMU architecture was introduced with ARM v3 (previous ARM * architecture versions used an optional off-CPU memory controller * to perform address translation). * * The ARM MMU consists of a TLB and translation table walking logic. * There is typically one TLB per memory interface (or, put another * way, one TLB per software-visible cache). * * The ARM MMU is capable of mapping memory in the following chunks: * * 1M Sections (L1 table) * * 64K Large Pages (L2 table) * * 4K Small Pages (L2 table) * * 1K Tiny Pages (L2 table) * * There are two types of L2 tables: Coarse Tables and Fine Tables. * Coarse Tables can map Large and Small Pages. Fine Tables can * map Tiny Pages. * * Coarse Tables can define 4 Subpages within Large and Small pages. * Subpages define different permissions for each Subpage within * a Page. * * Coarse Tables are 1K in length. Fine tables are 4K in length. * * The Translation Table Base register holds the pointer to the * L1 Table. The L1 Table is a 16K contiguous chunk of memory * aligned to a 16K boundary. Each entry in the L1 Table maps * 1M of virtual address space, either via a Section mapping or * via an L2 Table. * * In addition, the Fast Context Switching Extension (FCSE) is available * on some ARM v4 and ARM v5 processors. FCSE is a way of eliminating * TLB/cache flushes on context switch by use of a smaller address space * and a "process ID" that modifies the virtual address before being * presented to the translation logic. */ /* ARMv6 super-sections. */ #define L1_SUP_SIZE 0x01000000 /* 16M */ #define L1_SUP_OFFSET (L1_SUP_SIZE - 1) #define L1_SUP_FRAME (~L1_SUP_OFFSET) #define L1_SUP_SHIFT 24 #define L1_S_SIZE 0x00100000 /* 1M */ #define L1_S_OFFSET (L1_S_SIZE - 1) #define L1_S_FRAME (~L1_S_OFFSET) #define L1_S_SHIFT 20 #define L2_L_SIZE 0x00010000 /* 64K */ #define L2_L_OFFSET (L2_L_SIZE - 1) #define L2_L_FRAME (~L2_L_OFFSET) #define L2_L_SHIFT 16 #define L2_S_SIZE 0x00001000 /* 4K */ #define L2_S_OFFSET (L2_S_SIZE - 1) #define L2_S_FRAME (~L2_S_OFFSET) #define L2_S_SHIFT 12 #define L2_T_SIZE 0x00000400 /* 1K */ #define L2_T_OFFSET (L2_T_SIZE - 1) #define L2_T_FRAME (~L2_T_OFFSET) #define L2_T_SHIFT 10 /* * The NetBSD VM implementation only works on whole pages (4K), * whereas the ARM MMU's Coarse tables are sized in terms of 1K * (16K L1 table, 1K L2 table). * * So, we allocate L2 tables 4 at a time, thus yielding a 4K L2 * table. */ #define L1_ADDR_BITS 0xfff00000 /* L1 PTE address bits */ #define L2_ADDR_BITS 0x000ff000 /* L2 PTE address bits */ #define L1_TABLE_SIZE 0x4000 /* 16K */ #define L2_TABLE_SIZE 0x1000 /* 4K */ /* * The new pmap deals with the 1KB coarse L2 tables by * allocating them from a pool. Until every port has been converted, * keep the old L2_TABLE_SIZE define lying around. Converted ports * should use L2_TABLE_SIZE_REAL until then. */ #define L2_TABLE_SIZE_REAL 0x400 /* 1K */ /* Total number of page table entries in L2 table */ #define L2_PTE_NUM_TOTAL (L2_TABLE_SIZE_REAL / sizeof(pt_entry_t)) /* * ARM L1 Descriptors */ #define L1_TYPE_INV 0x00 /* Invalid (fault) */ #define L1_TYPE_C 0x01 /* Coarse L2 */ #define L1_TYPE_S 0x02 /* Section */ #define L1_TYPE_F 0x03 /* Fine L2 */ #define L1_TYPE_MASK 0x03 /* mask of type bits */ /* L1 Section Descriptor */ #define L1_S_B 0x00000004 /* bufferable Section */ #define L1_S_C 0x00000008 /* cacheable Section */ #define L1_S_IMP 0x00000010 /* implementation defined */ #define L1_S_XN (1 << 4) /* execute not */ #define L1_S_DOM(x) ((x) << 5) /* domain */ #define L1_S_DOM_MASK L1_S_DOM(0xf) #define L1_S_AP(x) ((x) << 10) /* access permissions */ #define L1_S_ADDR_MASK 0xfff00000 /* phys address of section */ #define L1_S_TEX(x) (((x) & 0x7) << 12) /* Type Extension */ #define L1_S_TEX_MASK (0x7 << 12) /* Type Extension */ #define L1_S_APX (1 << 15) #define L1_SHARED (1 << 16) #define L1_S_XSCALE_P 0x00000200 /* ECC enable for this section */ #define L1_S_XSCALE_TEX(x) ((x) << 12) /* Type Extension */ #define L1_S_SUPERSEC ((1) << 18) /* Section is a super-section. */ /* L1 Coarse Descriptor */ #define L1_C_IMP0 0x00000004 /* implementation defined */ #define L1_C_IMP1 0x00000008 /* implementation defined */ #define L1_C_IMP2 0x00000010 /* implementation defined */ #define L1_C_DOM(x) ((x) << 5) /* domain */ #define L1_C_DOM_MASK L1_C_DOM(0xf) #define L1_C_ADDR_MASK 0xfffffc00 /* phys address of L2 Table */ #define L1_C_XSCALE_P 0x00000200 /* ECC enable for this section */ /* L1 Fine Descriptor */ #define L1_F_IMP0 0x00000004 /* implementation defined */ #define L1_F_IMP1 0x00000008 /* implementation defined */ #define L1_F_IMP2 0x00000010 /* implementation defined */ #define L1_F_DOM(x) ((x) << 5) /* domain */ #define L1_F_DOM_MASK L1_F_DOM(0xf) #define L1_F_ADDR_MASK 0xfffff000 /* phys address of L2 Table */ #define L1_F_XSCALE_P 0x00000200 /* ECC enable for this section */ /* * ARM L2 Descriptors */ #define L2_TYPE_INV 0x00 /* Invalid (fault) */ #define L2_TYPE_L 0x01 /* Large Page */ #define L2_TYPE_S 0x02 /* Small Page */ #define L2_TYPE_T 0x03 /* Tiny Page */ #define L2_TYPE_MASK 0x03 /* mask of type bits */ /* * This L2 Descriptor type is available on XScale processors * when using a Coarse L1 Descriptor. The Extended Small * Descriptor has the same format as the XScale Tiny Descriptor, * but describes a 4K page, rather than a 1K page. */ #define L2_TYPE_XSCALE_XS 0x03 /* XScale Extended Small Page */ #define L2_B 0x00000004 /* Bufferable page */ #define L2_C 0x00000008 /* Cacheable page */ #define L2_AP0(x) ((x) << 4) /* access permissions (sp 0) */ #define L2_AP1(x) ((x) << 6) /* access permissions (sp 1) */ #define L2_AP2(x) ((x) << 8) /* access permissions (sp 2) */ #define L2_AP3(x) ((x) << 10) /* access permissions (sp 3) */ #define L2_SHARED (1 << 10) #define L2_APX (1 << 9) #define L2_XN (1 << 0) #define L2_L_TEX_MASK (0x7 << 12) /* Type Extension */ #define L2_L_TEX(x) (((x) & 0x7) << 12) #define L2_S_TEX_MASK (0x7 << 6) /* Type Extension */ #define L2_S_TEX(x) (((x) & 0x7) << 6) #define L2_XSCALE_L_TEX(x) ((x) << 12) /* Type Extension */ #define L2_XSCALE_L_S(x) (1 << 15) /* Shared */ #define L2_XSCALE_T_TEX(x) ((x) << 6) /* Type Extension */ /* * Access Permissions for L1 and L2 Descriptors. */ #define AP_W 0x01 /* writable */ #define AP_REF 0x01 /* referenced flag */ #define AP_U 0x02 /* user */ /* * Short-hand for common AP_* constants. * * Note: These values assume the S (System) bit is set and * the R (ROM) bit is clear in CP15 register 1. */ #define AP_KR 0x00 /* kernel read */ #define AP_KRW 0x01 /* kernel read/write */ #define AP_KRWUR 0x02 /* kernel read/write usr read */ #define AP_KRWURW 0x03 /* kernel read/write usr read/write */ /* * Domain Types for the Domain Access Control Register. */ #define DOMAIN_FAULT 0x00 /* no access */ #define DOMAIN_CLIENT 0x01 /* client */ #define DOMAIN_RESERVED 0x02 /* reserved */ #define DOMAIN_MANAGER 0x03 /* manager */ /* * Type Extension bits for XScale processors. * * Behavior of C and B when X == 0: * * C B Cacheable Bufferable Write Policy Line Allocate Policy * 0 0 N N - - * 0 1 N Y - - * 1 0 Y Y Write-through Read Allocate * 1 1 Y Y Write-back Read Allocate * * Behavior of C and B when X == 1: * C B Cacheable Bufferable Write Policy Line Allocate Policy * 0 0 - - - - DO NOT USE * 0 1 N Y - - * 1 0 Mini-Data - - - * 1 1 Y Y Write-back R/W Allocate */ #define TEX_XSCALE_X 0x01 /* X modifies C and B */ #define TEX_XSCALE_E 0x02 #define TEX_XSCALE_T 0x04 /* Xscale core 3 */ /* * * Cache attributes with L2 present, S = 0 * T E X C B L1 i-cache L1 d-cache L1 DC WP L2 cacheable write coalesce * 0 0 0 0 0 N N - N N * 0 0 0 0 1 N N - N Y * 0 0 0 1 0 Y Y WT N Y * 0 0 0 1 1 Y Y WB Y Y * 0 0 1 0 0 N N - Y Y * 0 0 1 0 1 N N - N N * 0 0 1 1 0 Y Y - - N * 0 0 1 1 1 Y Y WT Y Y * 0 1 0 0 0 N N - N N * 0 1 0 0 1 N/A N/A N/A N/A N/A * 0 1 0 1 0 N/A N/A N/A N/A N/A * 0 1 0 1 1 N/A N/A N/A N/A N/A * 0 1 1 X X N/A N/A N/A N/A N/A * 1 X 0 0 0 N N - N Y * 1 X 0 0 1 Y N WB N Y * 1 X 0 1 0 Y N WT N Y * 1 X 0 1 1 Y N WB Y Y * 1 X 1 0 0 N N - Y Y * 1 X 1 0 1 Y Y WB Y Y * 1 X 1 1 0 Y Y WT Y Y * 1 X 1 1 1 Y Y WB Y Y * * * * * Cache attributes with L2 present, S = 1 * T E X C B L1 i-cache L1 d-cache L1 DC WP L2 cacheable write coalesce * 0 0 0 0 0 N N - N N * 0 0 0 0 1 N N - N Y * 0 0 0 1 0 Y Y - N Y * 0 0 0 1 1 Y Y WT Y Y * 0 0 1 0 0 N N - Y Y * 0 0 1 0 1 N N - N N * 0 0 1 1 0 Y Y - - N * 0 0 1 1 1 Y Y WT Y Y * 0 1 0 0 0 N N - N N * 0 1 0 0 1 N/A N/A N/A N/A N/A * 0 1 0 1 0 N/A N/A N/A N/A N/A * 0 1 0 1 1 N/A N/A N/A N/A N/A * 0 1 1 X X N/A N/A N/A N/A N/A * 1 X 0 0 0 N N - N Y * 1 X 0 0 1 Y N - N Y * 1 X 0 1 0 Y N - N Y * 1 X 0 1 1 Y N - Y Y * 1 X 1 0 0 N N - Y Y * 1 X 1 0 1 Y Y WT Y Y * 1 X 1 1 0 Y Y WT Y Y * 1 X 1 1 1 Y Y WT Y Y */ #endif /* !_MACHINE_PTE_H_ */ -#endif /* !ARM_NEW_PMAP */ +#endif /* __ARM_ARCH >= 6 */ /* End of pte.h */ Index: user/ngie/make_check/sys/arm/include/sf_buf.h =================================================================== --- user/ngie/make_check/sys/arm/include/sf_buf.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/sf_buf.h (revision 295078) @@ -1,54 +1,56 @@ /*- * Copyright (c) 2003 Alan L. Cox * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _MACHINE_SF_BUF_H_ #define _MACHINE_SF_BUF_H_ +#include + static inline void sf_buf_map(struct sf_buf *sf, int flags) { -#ifdef ARM_NEW_PMAP +#if __ARM_ARCH >= 6 pmap_qenter(sf->kva, &(sf->m), 1); #else pmap_kenter(sf->kva, VM_PAGE_TO_PHYS(sf->m)); #endif } static inline int sf_buf_unmap(struct sf_buf *sf) { -#ifdef ARM_NEW_PMAP +#if __ARM_ARCH >= 6 pmap_qremove(sf->kva, 1); #else pmap_kremove(sf->kva); #endif return (1); } #endif /* !_MACHINE_SF_BUF_H_ */ Index: user/ngie/make_check/sys/arm/include/smp.h =================================================================== --- user/ngie/make_check/sys/arm/include/smp.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/smp.h (revision 295078) @@ -1,57 +1,57 @@ /* $FreeBSD$ */ #ifndef _MACHINE_SMP_H_ #define _MACHINE_SMP_H_ #include #include #ifdef ARM_INTRNG enum { IPI_AST, IPI_PREEMPT, IPI_RENDEZVOUS, IPI_STOP, IPI_STOP_HARD = IPI_STOP, /* These are synonyms on arm. */ IPI_HARDCLOCK, - IPI_TLB, - IPI_CACHE, + IPI_TLB, /* Not used now, but keep it reserved. */ + IPI_CACHE, /* Not used now, but keep it reserved. */ INTR_IPI_COUNT }; #else #define IPI_AST 0 #define IPI_PREEMPT 2 #define IPI_RENDEZVOUS 3 #define IPI_STOP 4 #define IPI_STOP_HARD 4 #define IPI_HARDCLOCK 6 -#define IPI_TLB 7 -#define IPI_CACHE 8 +#define IPI_TLB 7 /* Not used now, but keep it reserved. */ +#define IPI_CACHE 8 /* Not used now, but keep it reserved. */ #endif /* INTRNG */ void init_secondary(int cpu); void mpentry(void); void ipi_all_but_self(u_int ipi); void ipi_cpu(int cpu, u_int ipi); void ipi_selected(cpuset_t cpus, u_int ipi); /* PIC interface */ void pic_ipi_send(cpuset_t cpus, u_int ipi); #ifndef ARM_INTRNG void pic_ipi_clear(int ipi); int pic_ipi_read(int arg); #endif /* Platform interface */ void platform_mp_setmaxid(void); int platform_mp_probe(void); void platform_mp_start_ap(void); void platform_mp_init_secondary(void); void platform_ipi_send(cpuset_t cpus, u_int ipi); /* global data in mp_machdep.c */ extern struct pcb stoppcbs[]; #endif /* !_MACHINE_SMP_H_ */ Index: user/ngie/make_check/sys/arm/include/vm.h =================================================================== --- user/ngie/make_check/sys/arm/include/vm.h (revision 295077) +++ user/ngie/make_check/sys/arm/include/vm.h (revision 295078) @@ -1,52 +1,56 @@ /*- * Copyright (c) 2009 Alan L. Cox * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _MACHINE_VM_H_ #define _MACHINE_VM_H_ -#ifdef ARM_NEW_PMAP +#include + +#if __ARM_ARCH >= 6 #include #define VM_MEMATTR_WB_WA ((vm_memattr_t)PTE2_ATTR_WB_WA) #define VM_MEMATTR_NOCACHE ((vm_memattr_t)PTE2_ATTR_NOCACHE) #define VM_MEMATTR_DEVICE ((vm_memattr_t)PTE2_ATTR_DEVICE) #define VM_MEMATTR_SO ((vm_memattr_t)PTE2_ATTR_SO) #define VM_MEMATTR_WT ((vm_memattr_t)PTE2_ATTR_WT) #define VM_MEMATTR_DEFAULT VM_MEMATTR_WB_WA #define VM_MEMATTR_UNCACHEABLE VM_MEMATTR_SO /* misused by DMA */ +#ifdef _KERNEL +/* Don't export aliased VM_MEMATTR to userland */ #define VM_MEMATTR_WRITE_COMBINING VM_MEMATTR_WT /* for DRM */ #define VM_MEMATTR_WRITE_BACK VM_MEMATTR_WB_WA /* for DRM */ - +#endif #else /* Memory attribute configuration. */ #define VM_MEMATTR_DEFAULT 0 #define VM_MEMATTR_UNCACHEABLE 1 #endif #endif /* !_MACHINE_VM_H_ */ Index: user/ngie/make_check/sys/arm64/arm64/db_disasm.c =================================================================== --- user/ngie/make_check/sys/arm64/arm64/db_disasm.c (revision 295077) +++ user/ngie/make_check/sys/arm64/arm64/db_disasm.c (revision 295078) @@ -1,41 +1,70 @@ /*- * Copyright (c) 2015 The FreeBSD Foundation * All rights reserved. * * This software was developed by Semihalf under * the sponsorship of the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include +#include +#include +#include + +static u_int db_disasm_read_word(vm_offset_t); +static void db_disasm_printaddr(vm_offset_t); + +/* Glue code to interface db_disasm to the generic ARM disassembler */ +static const struct disasm_interface db_disasm_interface = { + db_disasm_read_word, + db_disasm_printaddr, + db_printf +}; + +static u_int +db_disasm_read_word(vm_offset_t address) +{ + + return (db_get_value(address, INSN_SIZE, 0)); +} + +static void +db_disasm_printaddr(vm_offset_t address) +{ + + db_printsym((db_addr_t)address, DB_STGY_ANY); +} + vm_offset_t db_disasm(vm_offset_t loc, bool altfmt) { - return 0; + + return (disasm(&db_disasm_interface, loc, altfmt)); } /* End of db_disasm.c */ Index: user/ngie/make_check/sys/arm64/arm64/disassem.c =================================================================== --- user/ngie/make_check/sys/arm64/arm64/disassem.c (nonexistent) +++ user/ngie/make_check/sys/arm64/arm64/disassem.c (revision 295078) @@ -0,0 +1,330 @@ +/*- + * Copyright (c) 2016 Cavium + * All rights reserved. + * + * This software was developed by Semihalf. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); +#include + +#include +#include +#include +#include + +#define ARM64_MAX_TOKEN_LEN 8 +#define ARM64_MAX_TOKEN_CNT 10 + +static const char *w_reg[] = { + "w0", "w1", "w2", "w3", "w4", "w5", "w6", "w7", + "w8", "w9", "w10", "w11", "w12", "w13", "w14", "w15", + "w16", "w17", "w18", "w19", "w20", "w21", "w22", "w23", + "w24", "w25", "w26", "w27", "w28", "w29", "w30", "wSP", +}; + +static const char *x_reg[] = { + "x0", "x1", "x2", "x3", "x4", "x5", "x6", "x7", + "x8", "x9", "x10", "x11", "x12", "x13", "x14", "x15", + "x16", "x17", "x18", "x19", "x20", "x21", "x22", "x23", + "x24", "x25", "x26", "x27", "x28", "x29", "LR", "SP", +}; + +static const char *shift_2[] = { + "LSL", "LSR", "ASR", "RSV" +}; + +/* + * Structure representing single token (operand) inside instruction. + * name - name of operand + * pos - position within the instruction (in bits) + * len - operand length (in bits) + */ +struct arm64_insn_token { + char name[ARM64_MAX_TOKEN_LEN]; + int pos; + int len; +}; + +/* + * Define generic types for instruction printing. + */ +enum arm64_format_type { + TYPE_01, /* OP , , {, #} SF32/64 + OP , , #{, } SF32/64 */ +}; + +/* + * Structure representing single parsed instruction format. + * name - opcode name + * format - opcode format in a human-readable way + * type - syntax type for printing + * special_ops - special options passed to a printer (if any) + * mask - bitmask for instruction matching + * pattern - pattern to look for + * tokens - array of tokens (operands) inside instruction + */ +struct arm64_insn { + char* name; + char* format; + enum arm64_format_type type; + uint64_t special_ops; + uint32_t mask; + uint32_t pattern; + struct arm64_insn_token tokens[ARM64_MAX_TOKEN_CNT]; +}; + +/* + * Specify instruction opcode format in a human-readable way. Use notation + * obtained from ARM Architecture Reference Manual for ARMv8-A. + * + * Format string description: + * Each group must be separated by "|". Group made of 0/1 is used to + * generate mask and pattern for instruction matching. Groups containing + * an operand token (in format NAME(length_bits)) are used to retrieve any + * operand data from the instruction. Names here must be meaningful + * and match the one described in the Manual. + * + * Token description: + * SF - "0" represents 32-bit access, "1" represents 64-bit access + * SHIFT - type of shift (instruction dependent) + * IMM - immediate value + * Rx - register number + */ +static struct arm64_insn arm64_i[] = { + { "add", "SF(1)|0001011|SHIFT(2)|0|RM(5)|IMM(6)|RN(5)|RD(5)", TYPE_01, 0 }, + { "mov", "SF(1)|001000100000000000000|RN(5)|RD(5)", TYPE_01, 0 }, + { "add", "SF(1)|0010001|SHIFT(2)|IMM(12)|RN(5)|RD(5)", TYPE_01, 0 }, + { NULL, NULL } +}; + +static void +arm64_disasm_generate_masks(struct arm64_insn *tab) +{ + uint32_t mask, val; + int a, i; + int len, ret; + int token = 0; + char *format; + int error; + + while (tab->name != NULL) { + mask = 0; + val = 0; + format = tab->format; + token = 0; + error = 0; + + /* + * For each entry analyze format strings from the + * left (i.e. from the MSB). + */ + a = (INSN_SIZE * NBBY) - 1; + while (*format != '\0' && (a >= 0)) { + switch(*format) { + case '0': + /* Bit is 0, add to mask and pattern */ + mask |= (1 << a); + a--; + format++; + break; + case '1': + /* Bit is 1, add to mask and pattern */ + mask |= (1 << a); + val |= (1 << a); + a--; + format++; + break; + case '|': + /* skip */ + format++; + break; + default: + /* Token found, copy the name */ + memset(tab->tokens[token].name, 0, + sizeof(tab->tokens[token].name)); + i = 0; + while (*format != '(') { + tab->tokens[token].name[i] = *format; + i++; + format++; + if (i >= ARM64_MAX_TOKEN_LEN) { + printf("ERROR: token too long in op %s\n", + tab->name); + error = 1; + break; + } + } + if (error != 0) + break; + + /* Read the length value */ + ret = sscanf(format, "(%d)", &len); + if (ret == 1) { + if (token >= ARM64_MAX_TOKEN_CNT) { + printf("ERROR: to many tokens in op %s\n", + tab->name); + error = 1; + break; + } + + a -= len; + tab->tokens[token].pos = a + 1; + tab->tokens[token].len = len; + token++; + } + + /* Skip to the end of the token */ + while (*format != 0 && *format != '|') + format++; + } + } + + /* Write mask and pattern to the instruction array */ + tab->mask = mask; + tab->pattern = val; + + /* + * If we got here, format string must be parsed and "a" + * should point to -1. If it's not, wrong number of bits + * in format string. Mark this as invalid and prevent + * from being matched. + */ + if (*format != 0 || (a != -1) || (error != 0)) { + tab->mask = 0; + tab->pattern = 0xffffffff; + printf("ERROR: skipping instruction op %s\n", + tab->name); + } + + tab++; + } +} + +static int +arm64_disasm_read_token(struct arm64_insn *insn, u_int opcode, + const char *token, int *val) +{ + int i; + + for (i = 0; i < ARM64_MAX_TOKEN_CNT; i++) { + if (strcmp(insn->tokens[i].name, token) == 0) { + *val = (opcode >> insn->tokens[i].pos & + ((1 << insn->tokens[i].len) - 1)); + return (0); + } + } + + return (EINVAL); +} + +static const char * +arm64_reg(int b64, int num) +{ + + if (b64 != 0) + return (x_reg[num]); + + return (w_reg[num]); +} + +vm_offset_t +disasm(const struct disasm_interface *di, vm_offset_t loc, int altfmt) +{ + struct arm64_insn *i_ptr = arm64_i; + uint32_t insn; + int matchp; + int ret; + int shift, rm, rd, rn, imm, sf; + int rm_absent; + + /* Initialize defaults, all are 0 except SF indicating 64bit access */ + shift = rd = rm = rn = imm = 0; + sf = 1; + + matchp = 0; + insn = di->di_readword(loc); + while (i_ptr->name) { + /* If mask is 0 then the parser was not initialized yet */ + if ((i_ptr->mask != 0) && + ((insn & i_ptr->mask) == i_ptr->pattern)) { + matchp = 1; + break; + } + i_ptr++; + } + if (matchp == 0) + goto undefined; + + switch (i_ptr->type) { + case TYPE_01: + /* OP , , {, #} SF32/64 + OP , , #{, } SF32/64 */ + + /* Mandatory tokens */ + ret = arm64_disasm_read_token(i_ptr, insn, "SF", &sf); + ret |= arm64_disasm_read_token(i_ptr, insn, "RD", &rd); + ret |= arm64_disasm_read_token(i_ptr, insn, "RN", &rn); + if (ret != 0) { + printf("ERROR: Missing mandatory token for op %s type %d\n", + i_ptr->name, i_ptr->type); + goto undefined; + } + + /* Optional tokens */ + arm64_disasm_read_token(i_ptr, insn, "IMM", &imm); + arm64_disasm_read_token(i_ptr, insn, "SHIFT", &shift); + rm_absent = arm64_disasm_read_token(i_ptr, insn, "RM", &rm); + + di->di_printf("%s\t%s, %s", i_ptr->name, arm64_reg(sf, rd), + arm64_reg(sf, rn)); + + /* If RM is present use it, otherwise use immediate notation */ + if (rm_absent == 0) { + di->di_printf(", %s", arm64_reg(sf, rm)); + if (imm != 0) + di->di_printf(", %s #%d", shift_2[shift], imm); + } else { + if (imm != 0 || shift != 0) + di->di_printf(", #0x%x", imm); + if (shift != 0) + di->di_printf(" LSL #12"); + } + break; + default: + goto undefined; + } + + di->di_printf("\n"); + return(loc + INSN_SIZE); + +undefined: + di->di_printf("undefined\t%08x\n", insn); + return(loc + INSN_SIZE); +} + +/* Parse format strings at the very beginning */ +SYSINIT(arm64_disasm_generate_masks, SI_SUB_DDB_SERVICES, + SI_ORDER_FIRST, arm64_disasm_generate_masks, arm64_i); Property changes on: user/ngie/make_check/sys/arm64/arm64/disassem.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/arm64/arm64/intr_machdep.c =================================================================== --- user/ngie/make_check/sys/arm64/arm64/intr_machdep.c (revision 295077) +++ user/ngie/make_check/sys/arm64/arm64/intr_machdep.c (revision 295078) @@ -1,502 +1,499 @@ /*- * Copyright (c) 1991 The Regents of the University of California. * Copyright (c) 2002 Benno Rice. * Copyright (c) 2014 The FreeBSD Foundation * All rights reserved. * * This software was developed by Semihalf under * the sponsorship of the FreeBSD Foundation. * * This code is derived from software contributed by * William Jolitz (Berkeley) and Benno Rice. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * form: src/sys/powerpc/powerpc/intr_machdep.c, r271712 2014/09/17 */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef SMP #include #endif #include "pic_if.h" #define MAX_STRAY_LOG 5 #define INTRNAME_LEN (MAXCOMLEN + 1) #define NIRQS 1024 /* Maximum number of interrupts in the system */ static MALLOC_DEFINE(M_INTR, "intr", "Interrupt Services"); /* * Linked list of interrupts that have been set-up. * Each element holds the interrupt description * and has to be allocated and freed dynamically. */ static SLIST_HEAD(, arm64_intr_entry) irq_slist_head = SLIST_HEAD_INITIALIZER(irq_slist_head); struct arm64_intr_entry { SLIST_ENTRY(arm64_intr_entry) entries; struct intr_event *i_event; enum intr_trigger i_trig; enum intr_polarity i_pol; u_int i_hw_irq; /* Physical interrupt number */ u_int i_cntidx; /* Index in intrcnt table */ u_int i_handlers; /* Allocated handlers */ u_long *i_cntp; /* Interrupt hit counter */ }; /* Counts and names for statistics - see sys/sys/interrupt.h */ /* Tables are indexed by i_cntidx */ u_long intrcnt[NIRQS]; char intrnames[NIRQS * INTRNAME_LEN]; size_t sintrcnt = sizeof(intrcnt); size_t sintrnames = sizeof(intrnames); static u_int intrcntidx; /* Current index into intrcnt table */ static u_int arm64_nintrs; /* Max interrupts number of the root PIC */ static u_int arm64_nstray; /* Number of received stray interrupts */ static device_t root_pic; /* PIC device for all incoming interrupts */ static device_t msi_pic; /* Device which handles MSI/MSI-X interrupts */ static struct mtx intr_list_lock; static void intr_init(void *dummy __unused) { mtx_init(&intr_list_lock, "intr sources lock", NULL, MTX_DEF); } SYSINIT(intr_init, SI_SUB_INTR, SI_ORDER_FIRST, intr_init, NULL); /* * Helper routines. */ /* Set interrupt name for statistics */ static void intrcnt_setname(const char *name, u_int idx) { snprintf(&intrnames[idx * INTRNAME_LEN], INTRNAME_LEN, "%-*s", INTRNAME_LEN - 1, name); } /* * Get intr structure for the given interrupt number. * Allocate one if this is the first time. * (Similar to ppc's intr_lookup() but without actual * lookup since irq number is an index in arm64_intrs[]). */ static struct arm64_intr_entry * intr_acquire(u_int hw_irq) { struct arm64_intr_entry *intr; mtx_lock(&intr_list_lock); SLIST_FOREACH(intr, &irq_slist_head, entries) { if (intr->i_hw_irq == hw_irq) { break; } } if (intr != NULL) goto out; /* Do not alloc another intr when max number of IRQs has been reached */ if (intrcntidx >= NIRQS) goto out; intr = malloc(sizeof(*intr), M_INTR, M_NOWAIT); if (intr == NULL) goto out; intr->i_event = NULL; intr->i_handlers = 0; intr->i_trig = INTR_TRIGGER_CONFORM; intr->i_pol = INTR_POLARITY_CONFORM; intr->i_cntidx = atomic_fetchadd_int(&intrcntidx, 1); intr->i_cntp = &intrcnt[intr->i_cntidx]; intr->i_hw_irq = hw_irq; SLIST_INSERT_HEAD(&irq_slist_head, intr, entries); out: mtx_unlock(&intr_list_lock); return intr; } static void intr_pre_ithread(void *arg) { struct arm64_intr_entry *intr = arg; PIC_PRE_ITHREAD(root_pic, intr->i_hw_irq); } static void intr_post_ithread(void *arg) { struct arm64_intr_entry *intr = arg; PIC_POST_ITHREAD(root_pic, intr->i_hw_irq); } static void intr_post_filter(void *arg) { struct arm64_intr_entry *intr = arg; PIC_POST_FILTER(root_pic, intr->i_hw_irq); } /* * Register PIC driver. * This is intended to be called by the very first PIC driver * at the end of the successful attach. * Note that during boot this can be called after first references * to bus_setup_intr() so it is required to not use root_pic if it * is not 100% safe. */ void arm_register_root_pic(device_t dev, u_int nirq) { KASSERT(root_pic == NULL, ("Unable to set the pic twice")); KASSERT(nirq <= NIRQS, ("PIC is trying to handle too many IRQs")); arm64_nintrs = NIRQS; /* Number of IRQs limited only by array size */ root_pic = dev; } /* Register device which allocates MSI interrupts */ void arm_register_msi_pic(device_t dev) { KASSERT(msi_pic == NULL, ("Unable to set msi_pic twice")); msi_pic = dev; } int arm_alloc_msi(device_t pci, device_t child, int count, int maxcount, int *irqs) { return (PIC_ALLOC_MSI(msi_pic, child, count, irqs)); } int arm_release_msi(device_t pci, device_t child, int count, int *irqs) { return (PIC_RELEASE_MSI(msi_pic, child, count, irqs)); } int arm_map_msi(device_t pci, device_t child, int irq, uint64_t *addr, uint32_t *data) { return (PIC_MAP_MSI(msi_pic, child, irq, addr, data)); } int arm_alloc_msix(device_t pci, device_t child, int *irq) { return (PIC_ALLOC_MSIX(msi_pic, child, irq)); } int arm_release_msix(device_t pci, device_t child, int irq) { return (PIC_RELEASE_MSIX(msi_pic, child, irq)); } /* * Finalize interrupts bring-up (should be called from configure_final()). * Enables all interrupts registered by bus_setup_intr() during boot * as well as unlocks interrups reception on primary CPU. */ int arm_enable_intr(void) { struct arm64_intr_entry *intr; if (root_pic == NULL) panic("Cannot enable interrupts. No PIC configured"); /* * Iterate through all possible interrupts and perform * configuration if the interrupt is registered. */ SLIST_FOREACH(intr, &irq_slist_head, entries) { /* * XXX: In case we allowed to set up interrupt whose number * exceeds maximum number of interrupts for the root PIC * disable it and print proper error message. * * This can happen only when calling bus_setup_intr() * before the interrupt controller is attached. */ if (intr->i_cntidx >= arm64_nintrs) { /* Better fail when IVARIANTS enabled */ KASSERT(0, ("%s: Interrupt %u cannot be handled by the " "registered PIC. Max interrupt number: %u", __func__, intr->i_cntidx, arm64_nintrs - 1)); /* Print message and disable otherwise */ printf("ERROR: Cannot enable irq %u. Disabling.\n", intr->i_cntidx); PIC_MASK(root_pic, intr->i_hw_irq); } if (intr->i_trig != INTR_TRIGGER_CONFORM || intr->i_pol != INTR_POLARITY_CONFORM) { PIC_CONFIG(root_pic, intr->i_hw_irq, intr->i_trig, intr->i_pol); } if (intr->i_handlers > 0) PIC_UNMASK(root_pic, intr->i_hw_irq); } /* Enable interrupt reception on this CPU */ intr_enable(); return (0); } int arm_setup_intr(const char *name, driver_filter_t *filt, driver_intr_t handler, void *arg, u_int hw_irq, enum intr_type flags, void **cookiep) { struct arm64_intr_entry *intr; int error; intr = intr_acquire(hw_irq); if (intr == NULL) return (ENOMEM); /* * Watch out for interrupts' numbers. * If this is a system boot then don't allow to overfill interrupts * table (the interrupts will be deconfigured in arm_enable_intr()). */ if (intr->i_cntidx >= NIRQS) return (EINVAL); if (intr->i_event == NULL) { error = intr_event_create(&intr->i_event, (void *)intr, 0, hw_irq, intr_pre_ithread, intr_post_ithread, intr_post_filter, NULL, "irq%u", hw_irq); if (error) return (error); } error = intr_event_add_handler(intr->i_event, name, filt, handler, arg, intr_priority(flags), flags, cookiep); if (!error) { mtx_lock(&intr_list_lock); intrcnt_setname(intr->i_event->ie_fullname, intr->i_cntidx); intr->i_handlers++; if (!cold && intr->i_handlers == 1) { if (intr->i_trig != INTR_TRIGGER_CONFORM || intr->i_pol != INTR_POLARITY_CONFORM) { PIC_CONFIG(root_pic, intr->i_hw_irq, intr->i_trig, intr->i_pol); } PIC_UNMASK(root_pic, intr->i_hw_irq); } mtx_unlock(&intr_list_lock); } return (error); } int arm_teardown_intr(void *cookie) { struct arm64_intr_entry *intr; int error; intr = intr_handler_source(cookie); error = intr_event_remove_handler(cookie); if (!error) { mtx_lock(&intr_list_lock); intr->i_handlers--; if (intr->i_handlers == 0) PIC_MASK(root_pic, intr->i_hw_irq); intrcnt_setname(intr->i_event->ie_fullname, intr->i_cntidx); mtx_unlock(&intr_list_lock); } return (error); } int arm_config_intr(u_int hw_irq, enum intr_trigger trig, enum intr_polarity pol) { struct arm64_intr_entry *intr; intr = intr_acquire(hw_irq); if (intr == NULL) return (ENOMEM); intr->i_trig = trig; intr->i_pol = pol; if (!cold && root_pic != NULL) PIC_CONFIG(root_pic, intr->i_hw_irq, trig, pol); return (0); } void arm_dispatch_intr(u_int hw_irq, struct trapframe *tf) { struct arm64_intr_entry *intr; SLIST_FOREACH(intr, &irq_slist_head, entries) { if (intr->i_hw_irq == hw_irq) { break; } } if (intr == NULL) goto stray; (*intr->i_cntp)++; if (!intr_event_handle(intr->i_event, tf)) return; stray: if (arm64_nstray < MAX_STRAY_LOG) { arm64_nstray++; printf("Stray IRQ %u\n", hw_irq); if (arm64_nstray >= MAX_STRAY_LOG) { printf("Got %d stray IRQs. Not logging anymore.\n", MAX_STRAY_LOG); } } if (intr != NULL) PIC_MASK(root_pic, intr->i_hw_irq); #ifdef HWPMC_HOOKS if (pmc_hook && (PCPU_GET(curthread)->td_pflags & TDP_CALLCHAIN)) pmc_hook(PCPU_GET(curthread), PMC_FN_USER_CALLCHAIN, tf); #endif } void arm_cpu_intr(struct trapframe *tf) { critical_enter(); PIC_DISPATCH(root_pic, tf); critical_exit(); } #ifdef SMP void arm_setup_ipihandler(driver_filter_t *filt, u_int ipi) { arm_setup_intr("ipi", filt, NULL, (void *)((uintptr_t)ipi | 1<<16), ipi, INTR_TYPE_MISC | INTR_EXCL, NULL); arm_unmask_ipi(ipi); } void arm_unmask_ipi(u_int ipi) { PIC_UNMASK(root_pic, ipi); } void arm_init_secondary(void) { PIC_INIT_SECONDARY(root_pic); } /* Sending IPI */ void ipi_all_but_self(u_int ipi) { cpuset_t other_cpus; other_cpus = all_cpus; CPU_CLR(PCPU_GET(cpuid), &other_cpus); - /* ARM64TODO: This will be fixed with arm_intrng */ - ipi += 16; - CTR2(KTR_SMP, "%s: ipi: %x", __func__, ipi); PIC_IPI_SEND(root_pic, other_cpus, ipi); } void ipi_cpu(int cpu, u_int ipi) { cpuset_t cpus; CPU_ZERO(&cpus); CPU_SET(cpu, &cpus); CTR2(KTR_SMP, "ipi_cpu: cpu: %d, ipi: %x", cpu, ipi); PIC_IPI_SEND(root_pic, cpus, ipi); } void ipi_selected(cpuset_t cpus, u_int ipi) { CTR1(KTR_SMP, "ipi_selected: ipi: %x", ipi); PIC_IPI_SEND(root_pic, cpus, ipi); } #endif Index: user/ngie/make_check/sys/arm64/include/armreg.h =================================================================== --- user/ngie/make_check/sys/arm64/include/armreg.h (revision 295077) +++ user/ngie/make_check/sys/arm64/include/armreg.h (revision 295078) @@ -1,392 +1,394 @@ /*- * Copyright (c) 2013, 2014 Andrew Turner * Copyright (c) 2015 The FreeBSD Foundation * All rights reserved. * * This software was developed by Andrew Turner under * sponsorship from the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _MACHINE_ARMREG_H_ #define _MACHINE_ARMREG_H_ +#define INSN_SIZE 4 + #define READ_SPECIALREG(reg) \ ({ uint64_t val; \ __asm __volatile("mrs %0, " __STRING(reg) : "=&r" (val)); \ val; \ }) #define WRITE_SPECIALREG(reg, val) \ __asm __volatile("msr " __STRING(reg) ", %0" : : "r"((uint64_t)val)) /* CNTHCTL_EL2 - Counter-timer Hypervisor Control register */ #define CNTHCTL_EVNTI_MASK (0xf << 4) /* Bit to trigger event stream */ #define CNTHCTL_EVNTDIR (1 << 3) /* Control transition trigger bit */ #define CNTHCTL_EVNTEN (1 << 2) /* Enable event stream */ #define CNTHCTL_EL1PCEN (1 << 1) /* Allow EL0/1 physical timer access */ #define CNTHCTL_EL1PCTEN (1 << 0) /*Allow EL0/1 physical counter access*/ /* CPACR_EL1 */ #define CPACR_FPEN_MASK (0x3 << 20) #define CPACR_FPEN_TRAP_ALL1 (0x0 << 20) /* Traps from EL0 and EL1 */ #define CPACR_FPEN_TRAP_EL0 (0x1 << 20) /* Traps from EL0 */ #define CPACR_FPEN_TRAP_ALL2 (0x2 << 20) /* Traps from EL0 and EL1 */ #define CPACR_FPEN_TRAP_NONE (0x3 << 20) /* No traps */ #define CPACR_TTA (0x1 << 28) /* CTR_EL0 - Cache Type Register */ #define CTR_DLINE_SHIFT 16 #define CTR_DLINE_MASK (0xf << CTR_DLINE_SHIFT) #define CTR_DLINE_SIZE(reg) (((reg) & CTR_DLINE_MASK) >> CTR_DLINE_SHIFT) #define CTR_ILINE_SHIFT 0 #define CTR_ILINE_MASK (0xf << CTR_ILINE_SHIFT) #define CTR_ILINE_SIZE(reg) (((reg) & CTR_ILINE_MASK) >> CTR_ILINE_SHIFT) /* ESR_ELx */ #define ESR_ELx_ISS_MASK 0x00ffffff #define ISS_INSN_FnV (0x01 << 10) #define ISS_INSN_EA (0x01 << 9) #define ISS_INSN_S1PTW (0x01 << 7) #define ISS_INSN_IFSC_MASK (0x1f << 0) #define ISS_DATA_ISV (0x01 << 24) #define ISS_DATA_SAS_MASK (0x03 << 22) #define ISS_DATA_SSE (0x01 << 21) #define ISS_DATA_SRT_MASK (0x1f << 16) #define ISS_DATA_SF (0x01 << 15) #define ISS_DATA_AR (0x01 << 14) #define ISS_DATA_FnV (0x01 << 10) #define ISS_DATa_EA (0x01 << 9) #define ISS_DATa_CM (0x01 << 8) #define ISS_INSN_S1PTW (0x01 << 7) #define ISS_DATa_WnR (0x01 << 6) #define ISS_DATA_DFSC_MASK (0x1f << 0) #define ESR_ELx_IL (0x01 << 25) #define ESR_ELx_EC_SHIFT 26 #define ESR_ELx_EC_MASK (0x3f << 26) #define ESR_ELx_EXCEPTION(esr) (((esr) & ESR_ELx_EC_MASK) >> ESR_ELx_EC_SHIFT) #define EXCP_UNKNOWN 0x00 /* Unkwn exception */ #define EXCP_FP_SIMD 0x07 /* VFP/SIMD trap */ #define EXCP_ILL_STATE 0x0e /* Illegal execution state */ #define EXCP_SVC 0x15 /* SVC trap */ #define EXCP_MSR 0x18 /* MSR/MRS trap */ #define EXCP_INSN_ABORT_L 0x20 /* Instruction abort, from lower EL */ #define EXCP_INSN_ABORT 0x21 /* Instruction abort, from same EL */ #define EXCP_PC_ALIGN 0x22 /* PC alignment fault */ #define EXCP_DATA_ABORT_L 0x24 /* Data abort, from lower EL */ #define EXCP_DATA_ABORT 0x25 /* Data abort, from same EL */ #define EXCP_SP_ALIGN 0x26 /* SP slignment fault */ #define EXCP_TRAP_FP 0x2c /* Trapped FP exception */ #define EXCP_SERROR 0x2f /* SError interrupt */ #define EXCP_SOFTSTP_EL1 0x33 /* Software Step, from same EL */ #define EXCP_WATCHPT_EL1 0x35 /* Watchpoint, from same EL */ #define EXCP_BRK 0x3c /* Breakpoint */ /* ICC_CTLR_EL1 */ #define ICC_CTLR_EL1_EOIMODE (1U << 1) /* ICC_IAR1_EL1 */ #define ICC_IAR1_EL1_SPUR (0x03ff) /* ICC_IGRPEN0_EL1 */ #define ICC_IGRPEN0_EL1_EN (1U << 0) /* ICC_PMR_EL1 */ #define ICC_PMR_EL1_PRIO_MASK (0xFFUL) /* ICC_SRE_EL1 */ #define ICC_SRE_EL1_SRE (1U << 0) /* ICC_SRE_EL2 */ #define ICC_SRE_EL2_EN (1U << 3) /* ID_AA64DFR0_EL1 */ #define ID_AA64DFR0_MASK 0xf0f0ffff #define ID_AA64DFR0_DEBUG_VER_SHIFT 0 #define ID_AA64DFR0_DEBUG_VER_MASK (0xf << ID_AA64DFR0_DEBUG_VER_SHIFT) #define ID_AA64DFR0_DEBUG_VER(x) ((x) & ID_AA64DFR0_DEBUG_VER_MASK) #define ID_AA64DFR0_DEBUG_VER_8 (0x6 << ID_AA64DFR0_DEBUG_VER_SHIFT) #define ID_AA64DFR0_TRACE_VER_SHIFT 4 #define ID_AA64DFR0_TRACE_VER_MASK (0xf << ID_AA64DFR0_TRACE_VER_SHIFT) #define ID_AA64DFR0_TRACE_VER(x) ((x) & ID_AA64DFR0_TRACE_VER_MASK) #define ID_AA64DFR0_TRACE_VER_NONE (0x0 << ID_AA64DFR0_TRACE_VER_SHIFT) #define ID_AA64DFR0_TRACE_VER_IMPL (0x1 << ID_AA64DFR0_TRACE_VER_SHIFT) #define ID_AA64DFR0_PMU_VER_SHIFT 8 #define ID_AA64DFR0_PMU_VER_MASK (0xf << ID_AA64DFR0_PMU_VER_SHIFT) #define ID_AA64DFR0_PMU_VER(x) ((x) & ID_AA64DFR0_PMU_VER_MASK) #define ID_AA64DFR0_PMU_VER_NONE (0x0 << ID_AA64DFR0_PMU_VER_SHIFT) #define ID_AA64DFR0_PMU_VER_3 (0x1 << ID_AA64DFR0_PMU_VER_SHIFT) #define ID_AA64DFR0_PMU_VER_IMPL (0xf << ID_AA64DFR0_PMU_VER_SHIFT) #define ID_AA64DFR0_BRPS_SHIFT 12 #define ID_AA64DFR0_BRPS_MASK (0xf << ID_AA64DFR0_BRPS_SHIFT) #define ID_AA64DFR0_BRPS(x) \ ((((x) >> ID_AA64DFR0_BRPS_SHIFT) & 0xf) + 1) #define ID_AA64DFR0_WRPS_SHIFT 20 #define ID_AA64DFR0_WRPS_MASK (0xf << ID_AA64DFR0_WRPS_SHIFT) #define ID_AA64DFR0_WRPS(x) \ ((((x) >> ID_AA64DFR0_WRPS_SHIFT) & 0xf) + 1) #define ID_AA64DFR0_CTX_CMPS_SHIFT 28 #define ID_AA64DFR0_CTX_CMPS_MASK (0xf << ID_AA64DFR0_CTX_CMPS_SHIFT) #define ID_AA64DFR0_CTX_CMPS(x) \ ((((x) >> ID_AA64DFR0_CTX_CMPS_SHIFT) & 0xf) + 1) /* ID_AA64ISAR0_EL1 */ #define ID_AA64ISAR0_MASK 0x000ffff0 #define ID_AA64ISAR0_AES_SHIFT 4 #define ID_AA64ISAR0_AES_MASK (0xf << ID_AA64ISAR0_AES_SHIFT) #define ID_AA64ISAR0_AES(x) ((x) & ID_AA64ISAR0_AES_MASK) #define ID_AA64ISAR0_AES_NONE (0x0 << ID_AA64ISAR0_AES_SHIFT) #define ID_AA64ISAR0_AES_BASE (0x1 << ID_AA64ISAR0_AES_SHIFT) #define ID_AA64ISAR0_AES_PMULL (0x2 << ID_AA64ISAR0_AES_SHIFT) #define ID_AA64ISAR0_SHA1_SHIFT 8 #define ID_AA64ISAR0_SHA1_MASK (0xf << ID_AA64ISAR0_SHA1_SHIFT) #define ID_AA64ISAR0_SHA1(x) ((x) & ID_AA64ISAR0_SHA1_MASK) #define ID_AA64ISAR0_SHA1_NONE (0x0 << ID_AA64ISAR0_SHA1_SHIFT) #define ID_AA64ISAR0_SHA1_BASE (0x1 << ID_AA64ISAR0_SHA1_SHIFT) #define ID_AA64ISAR0_SHA2_SHIFT 12 #define ID_AA64ISAR0_SHA2_MASK (0xf << ID_AA64ISAR0_SHA2_SHIFT) #define ID_AA64ISAR0_SHA2(x) ((x) & ID_AA64ISAR0_SHA2_MASK) #define ID_AA64ISAR0_SHA2_NONE (0x0 << ID_AA64ISAR0_SHA2_SHIFT) #define ID_AA64ISAR0_SHA2_BASE (0x1 << ID_AA64ISAR0_SHA2_SHIFT) #define ID_AA64ISAR0_CRC32_SHIFT 16 #define ID_AA64ISAR0_CRC32_MASK (0xf << ID_AA64ISAR0_CRC32_SHIFT) #define ID_AA64ISAR0_CRC32(x) ((x) & ID_AA64ISAR0_CRC32_MASK) #define ID_AA64ISAR0_CRC32_NONE (0x0 << ID_AA64ISAR0_CRC32_SHIFT) #define ID_AA64ISAR0_CRC32_BASE (0x1 << ID_AA64ISAR0_CRC32_SHIFT) /* ID_AA64MMFR0_EL1 */ #define ID_AA64MMFR0_MASK 0xffffffff #define ID_AA64MMFR0_PA_RANGE_SHIFT 0 #define ID_AA64MMFR0_PA_RANGE_MASK (0xf << ID_AA64MMFR0_PA_RANGE_SHIFT) #define ID_AA64MMFR0_PA_RANGE(x) ((x) & ID_AA64MMFR0_PA_RANGE_MASK) #define ID_AA64MMFR0_PA_RANGE_4G (0x0 << ID_AA64MMFR0_PA_RANGE_SHIFT) #define ID_AA64MMFR0_PA_RANGE_64G (0x1 << ID_AA64MMFR0_PA_RANGE_SHIFT) #define ID_AA64MMFR0_PA_RANGE_1T (0x2 << ID_AA64MMFR0_PA_RANGE_SHIFT) #define ID_AA64MMFR0_PA_RANGE_4T (0x3 << ID_AA64MMFR0_PA_RANGE_SHIFT) #define ID_AA64MMFR0_PA_RANGE_16T (0x4 << ID_AA64MMFR0_PA_RANGE_SHIFT) #define ID_AA64MMFR0_PA_RANGE_256T (0x5 << ID_AA64MMFR0_PA_RANGE_SHIFT) #define ID_AA64MMFR0_ASID_BITS_SHIFT 4 #define ID_AA64MMFR0_ASID_BITS_MASK (0xf << ID_AA64MMFR0_ASID_BITS_SHIFT) #define ID_AA64MMFR0_ASID_BITS(x) ((x) & ID_AA64MMFR0_ASID_BITS_MASK) #define ID_AA64MMFR0_ASID_BITS_8 (0x0 << ID_AA64MMFR0_ASID_BITS_SHIFT) #define ID_AA64MMFR0_ASID_BITS_16 (0x2 << ID_AA64MMFR0_ASID_BITS_SHIFT) #define ID_AA64MMFR0_BIGEND_SHIFT 8 #define ID_AA64MMFR0_BIGEND_MASK (0xf << ID_AA64MMFR0_BIGEND_SHIFT) #define ID_AA64MMFR0_BIGEND(x) ((x) & ID_AA64MMFR0_BIGEND_MASK) #define ID_AA64MMFR0_BIGEND_FIXED (0x0 << ID_AA64MMFR0_BIGEND_SHIFT) #define ID_AA64MMFR0_BIGEND_MIXED (0x1 << ID_AA64MMFR0_BIGEND_SHIFT) #define ID_AA64MMFR0_S_NS_MEM_SHIFT 12 #define ID_AA64MMFR0_S_NS_MEM_MASK (0xf << ID_AA64MMFR0_S_NS_MEM_SHIFT) #define ID_AA64MMFR0_S_NS_MEM(x) ((x) & ID_AA64MMFR0_S_NS_MEM_MASK) #define ID_AA64MMFR0_S_NS_MEM_NONE (0x0 << ID_AA64MMFR0_S_NS_MEM_SHIFT) #define ID_AA64MMFR0_S_NS_MEM_DISTINCT (0x1 << ID_AA64MMFR0_S_NS_MEM_SHIFT) #define ID_AA64MMFR0_BIGEND_EL0_SHIFT 16 #define ID_AA64MMFR0_BIGEND_EL0_MASK (0xf << ID_AA64MMFR0_BIGEND_EL0_SHIFT) #define ID_AA64MMFR0_BIGEND_EL0(x) ((x) & ID_AA64MMFR0_BIGEND_EL0_MASK) #define ID_AA64MMFR0_BIGEND_EL0_FIXED (0x0 << ID_AA64MMFR0_BIGEND_EL0_SHIFT) #define ID_AA64MMFR0_BIGEND_EL0_MIXED (0x1 << ID_AA64MMFR0_BIGEND_EL0_SHIFT) #define ID_AA64MMFR0_TGRAN16_SHIFT 20 #define ID_AA64MMFR0_TGRAN16_MASK (0xf << ID_AA64MMFR0_TGRAN16_SHIFT) #define ID_AA64MMFR0_TGRAN16(x) ((x) & ID_AA64MMFR0_TGRAN16_MASK) #define ID_AA64MMFR0_TGRAN16_NONE (0x0 << ID_AA64MMFR0_TGRAN16_SHIFT) #define ID_AA64MMFR0_TGRAN16_IMPL (0x1 << ID_AA64MMFR0_TGRAN16_SHIFT) #define ID_AA64MMFR0_TGRAN64_SHIFT 24 #define ID_AA64MMFR0_TGRAN64_MASK (0xf << ID_AA64MMFR0_TGRAN64_SHIFT) #define ID_AA64MMFR0_TGRAN64(x) ((x) & ID_AA64MMFR0_TGRAN64_MASK) #define ID_AA64MMFR0_TGRAN64_IMPL (0x0 << ID_AA64MMFR0_TGRAN64_SHIFT) #define ID_AA64MMFR0_TGRAN64_NONE (0xf << ID_AA64MMFR0_TGRAN64_SHIFT) #define ID_AA64MMFR0_TGRAN4_SHIFT 28 #define ID_AA64MMFR0_TGRAN4_MASK (0xf << ID_AA64MMFR0_TGRAN4_SHIFT) #define ID_AA64MMFR0_TGRAN4(x) ((x) & ID_AA64MMFR0_TGRAN4_MASK) #define ID_AA64MMFR0_TGRAN4_IMPL (0x0 << ID_AA64MMFR0_TGRAN4_SHIFT) #define ID_AA64MMFR0_TGRAN4_NONE (0xf << ID_AA64MMFR0_TGRAN4_SHIFT) /* ID_AA64PFR0_EL1 */ #define ID_AA64PFR0_MASK 0x0fffffff #define ID_AA64PFR0_EL0_SHIFT 0 #define ID_AA64PFR0_EL0_MASK (0xf << ID_AA64PFR0_EL0_SHIFT) #define ID_AA64PFR0_EL0(x) ((x) & ID_AA64PFR0_EL0_MASK) #define ID_AA64PFR0_EL0_64 (1 << ID_AA64PFR0_EL0_SHIFT) #define ID_AA64PFR0_EL0_64_32 (2 << ID_AA64PFR0_EL0_SHIFT) #define ID_AA64PFR0_EL1_SHIFT 4 #define ID_AA64PFR0_EL1_MASK (0xf << ID_AA64PFR0_EL1_SHIFT) #define ID_AA64PFR0_EL1(x) ((x) & ID_AA64PFR0_EL1_MASK) #define ID_AA64PFR0_EL1_64 (1 << ID_AA64PFR0_EL1_SHIFT) #define ID_AA64PFR0_EL1_64_32 (2 << ID_AA64PFR0_EL1_SHIFT) #define ID_AA64PFR0_EL2_SHIFT 8 #define ID_AA64PFR0_EL2_MASK (0xf << ID_AA64PFR0_EL2_SHIFT) #define ID_AA64PFR0_EL2(x) ((x) & ID_AA64PFR0_EL2_MASK) #define ID_AA64PFR0_EL2_NONE (0 << ID_AA64PFR0_EL2_SHIFT) #define ID_AA64PFR0_EL2_64 (1 << ID_AA64PFR0_EL2_SHIFT) #define ID_AA64PFR0_EL2_64_32 (2 << ID_AA64PFR0_EL2_SHIFT) #define ID_AA64PFR0_EL3_SHIFT 12 #define ID_AA64PFR0_EL3_MASK (0xf << ID_AA64PFR0_EL3_SHIFT) #define ID_AA64PFR0_EL3(x) ((x) & ID_AA64PFR0_EL3_MASK) #define ID_AA64PFR0_EL3_NONE (0 << ID_AA64PFR0_EL3_SHIFT) #define ID_AA64PFR0_EL3_64 (1 << ID_AA64PFR0_EL3_SHIFT) #define ID_AA64PFR0_EL3_64_32 (2 << ID_AA64PFR0_EL3_SHIFT) #define ID_AA64PFR0_FP_SHIFT 16 #define ID_AA64PFR0_FP_MASK (0xf << ID_AA64PFR0_FP_SHIFT) #define ID_AA64PFR0_FP(x) ((x) & ID_AA64PFR0_FP_MASK) #define ID_AA64PFR0_FP_IMPL (0x0 << ID_AA64PFR0_FP_SHIFT) #define ID_AA64PFR0_FP_NONE (0xf << ID_AA64PFR0_FP_SHIFT) #define ID_AA64PFR0_ADV_SIMD_SHIFT 20 #define ID_AA64PFR0_ADV_SIMD_MASK (0xf << ID_AA64PFR0_ADV_SIMD_SHIFT) #define ID_AA64PFR0_ADV_SIMD(x) ((x) & ID_AA64PFR0_ADV_SIMD_MASK) #define ID_AA64PFR0_ADV_SIMD_IMPL (0x0 << ID_AA64PFR0_ADV_SIMD_SHIFT) #define ID_AA64PFR0_ADV_SIMD_NONE (0xf << ID_AA64PFR0_ADV_SIMD_SHIFT) #define ID_AA64PFR0_GIC_BITS 0x4 /* Number of bits in GIC field */ #define ID_AA64PFR0_GIC_SHIFT 24 #define ID_AA64PFR0_GIC_MASK (0xf << ID_AA64PFR0_GIC_SHIFT) #define ID_AA64PFR0_GIC(x) ((x) & ID_AA64PFR0_GIC_MASK) #define ID_AA64PFR0_GIC_CPUIF_NONE (0x0 << ID_AA64PFR0_GIC_SHIFT) #define ID_AA64PFR0_GIC_CPUIF_EN (0x1 << ID_AA64PFR0_GIC_SHIFT) /* MAIR_EL1 - Memory Attribute Indirection Register */ #define MAIR_ATTR_MASK(idx) (0xff << ((n)* 8)) #define MAIR_ATTR(attr, idx) ((attr) << ((idx) * 8)) /* SCTLR_EL1 - System Control Register */ #define SCTLR_RES0 0xc8222400 /* Reserved, write 0 */ #define SCTLR_RES1 0x30d00800 /* Reserved, write 1 */ #define SCTLR_M 0x00000001 #define SCTLR_A 0x00000002 #define SCTLR_C 0x00000004 #define SCTLR_SA 0x00000008 #define SCTLR_SA0 0x00000010 #define SCTLR_CP15BEN 0x00000020 #define SCTLR_THEE 0x00000040 #define SCTLR_ITD 0x00000080 #define SCTLR_SED 0x00000100 #define SCTLR_UMA 0x00000200 #define SCTLR_I 0x00001000 #define SCTLR_DZE 0x00004000 #define SCTLR_UCT 0x00008000 #define SCTLR_nTWI 0x00010000 #define SCTLR_nTWE 0x00040000 #define SCTLR_WXN 0x00080000 #define SCTLR_EOE 0x01000000 #define SCTLR_EE 0x02000000 #define SCTLR_UCI 0x04000000 /* SPSR_EL1 */ /* * When the exception is taken in AArch64: * M[4] is 0 for AArch64 mode * M[3:2] is the exception level * M[1] is unused * M[0] is the SP select: * 0: always SP0 * 1: current ELs SP */ #define PSR_M_EL0t 0x00000000 #define PSR_M_EL1t 0x00000004 #define PSR_M_EL1h 0x00000005 #define PSR_M_EL2t 0x00000008 #define PSR_M_EL2h 0x00000009 #define PSR_M_MASK 0x0000001f #define PSR_F 0x00000040 #define PSR_I 0x00000080 #define PSR_A 0x00000100 #define PSR_D 0x00000200 #define PSR_IL 0x00100000 #define PSR_SS 0x00200000 #define PSR_V 0x10000000 #define PSR_C 0x20000000 #define PSR_Z 0x40000000 #define PSR_N 0x80000000 /* TCR_EL1 - Translation Control Register */ #define TCR_ASID_16 (1 << 36) #define TCR_IPS_SHIFT 32 #define TCR_IPS_32BIT (0 << TCR_IPS_SHIFT) #define TCR_IPS_36BIT (1 << TCR_IPS_SHIFT) #define TCR_IPS_40BIT (2 << TCR_IPS_SHIFT) #define TCR_IPS_42BIT (3 << TCR_IPS_SHIFT) #define TCR_IPS_44BIT (4 << TCR_IPS_SHIFT) #define TCR_IPS_48BIT (5 << TCR_IPS_SHIFT) #define TCR_TG1_SHIFT 30 #define TCR_TG1_16K (1 << TCR_TG1_SHIFT) #define TCR_TG1_4K (2 << TCR_TG1_SHIFT) #define TCR_TG1_64K (3 << TCR_TG1_SHIFT) #define TCR_SH1_SHIFT 28 #define TCR_SH1_IS (0x3UL << TCR_SH1_SHIFT) #define TCR_ORGN1_SHIFT 26 #define TCR_ORGN1_WBWA (0x1UL << TCR_ORGN1_SHIFT) #define TCR_IRGN1_SHIFT 24 #define TCR_IRGN1_WBWA (0x1UL << TCR_IRGN1_SHIFT) #define TCR_SH0_SHIFT 12 #define TCR_SH0_IS (0x3UL << TCR_SH0_SHIFT) #define TCR_ORGN0_SHIFT 10 #define TCR_ORGN0_WBWA (0x1UL << TCR_ORGN0_SHIFT) #define TCR_IRGN0_SHIFT 8 #define TCR_IRGN0_WBWA (0x1UL << TCR_IRGN0_SHIFT) #define TCR_CACHE_ATTRS ((TCR_IRGN0_WBWA | TCR_IRGN1_WBWA) |\ (TCR_ORGN0_WBWA | TCR_ORGN1_WBWA)) #ifdef SMP #define TCR_SMP_ATTRS (TCR_SH0_IS | TCR_SH1_IS) #else #define TCR_SMP_ATTRS 0 #endif #define TCR_T1SZ_SHIFT 16 #define TCR_T0SZ_SHIFT 0 #define TCR_T1SZ(x) ((x) << TCR_T1SZ_SHIFT) #define TCR_T0SZ(x) ((x) << TCR_T0SZ_SHIFT) #define TCR_TxSZ(x) (TCR_T1SZ(x) | TCR_T0SZ(x)) /* Saved Program Status Register */ #define DBG_SPSR_SS (0x1 << 21) /* Monitor Debug System Control Register */ #define DBG_MDSCR_SS (0x1 << 0) #define DBG_MDSCR_KDE (0x1 << 13) #define DBG_MDSCR_MDE (0x1 << 15) /* Perfomance Monitoring Counters */ #define PMCR_E (1 << 0) /* Enable all counters */ #define PMCR_P (1 << 1) /* Reset all counters */ #define PMCR_C (1 << 2) /* Clock counter reset */ #define PMCR_D (1 << 3) /* CNTR counts every 64 clk cycles */ #define PMCR_X (1 << 4) /* Export to ext. monitoring (ETM) */ #define PMCR_DP (1 << 5) /* Disable CCNT if non-invasive debug*/ #define PMCR_LC (1 << 6) /* Long cycle count enable */ #define PMCR_IMP_SHIFT 24 /* Implementer code */ #define PMCR_IMP_MASK (0xff << PMCR_IMP_SHIFT) #define PMCR_IDCODE_SHIFT 16 /* Identification code */ #define PMCR_IDCODE_MASK (0xff << PMCR_IDCODE_SHIFT) #define PMCR_IDCODE_CORTEX_A57 0x01 #define PMCR_IDCODE_CORTEX_A72 0x02 #define PMCR_IDCODE_CORTEX_A53 0x03 #define PMCR_N_SHIFT 11 /* Number of counters implemented */ #define PMCR_N_MASK (0x1f << PMCR_N_SHIFT) #endif /* !_MACHINE_ARMREG_H_ */ Index: user/ngie/make_check/sys/arm64/include/disassem.h =================================================================== --- user/ngie/make_check/sys/arm64/include/disassem.h (nonexistent) +++ user/ngie/make_check/sys/arm64/include/disassem.h (revision 295078) @@ -0,0 +1,42 @@ +/*- + * Copyright (c) 2016 Cavium + * All rights reserved. + * + * This software was developed by Semihalf. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef __DISASSEM_H_ +#define __DISASSEM_H_ + +struct disasm_interface { + u_int (*di_readword)(vm_offset_t); + void (*di_printaddr)(vm_offset_t); + int (*di_printf)(const char *, ...) __printflike(1, 2); +}; + +vm_offset_t disasm(const struct disasm_interface *, vm_offset_t, int); + +#endif /* __DISASSEM_H_ */ Property changes on: user/ngie/make_check/sys/arm64/include/disassem.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/boot/fdt/dts/riscv/spike.dts =================================================================== --- user/ngie/make_check/sys/boot/fdt/dts/riscv/spike.dts (nonexistent) +++ user/ngie/make_check/sys/boot/fdt/dts/riscv/spike.dts (revision 295078) @@ -0,0 +1,92 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +/dts-v1/; + +/ { + model = "UC Berkeley Spike Simulator RV64I"; + compatible = "riscv,rv64i"; + #address-cells = <1>; + #size-cells = <1>; + #interrupt-cells = <1>; + + aliases { + console0 = &console0; + }; + + memory { + device_type = "memory"; + reg = <0x0 0x8000000>; /* 128MB at 0x0 */ + }; + + soc { + #address-cells = <2>; + #size-cells = <2>; + #interrupt-cells = <1>; + + compatible = "simple-bus"; + ranges; + + pic0: pic@0 { + compatible = "riscv,pic"; + interrupt-controller; + }; + + timer0: timer@0 { + compatible = "riscv,timer"; + interrupts = < 1 >; + interrupt-parent = < &pic0 >; + clock-frequency = < 1000000 >; + }; + + htif0: htif@0 { + compatible = "riscv,htif"; + interrupts = < 0 >; + interrupt-parent = < &pic0 >; + + console0: console@0 { + compatible = "htif,console"; + status = "okay"; + }; + }; + }; + + chosen { + bootargs = "-v"; + stdin = "console0"; + stdout = "console0"; + }; +}; Property changes on: user/ngie/make_check/sys/boot/fdt/dts/riscv/spike.dts ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/boot/ficl/riscv/sysdep.c =================================================================== --- user/ngie/make_check/sys/boot/ficl/riscv/sysdep.c (nonexistent) +++ user/ngie/make_check/sys/boot/ficl/riscv/sysdep.c (revision 295078) @@ -0,0 +1,99 @@ +/******************************************************************* +** s y s d e p . c +** Forth Inspired Command Language +** Author: John Sadler (john_sadler@alum.mit.edu) +** Created: 16 Oct 1997 +** Implementations of FICL external interface functions... +** +*******************************************************************/ + +/* $FreeBSD$ */ + +#ifdef TESTMAIN +#include +#include +#else +#include +#endif +#include "ficl.h" + +/* +******************* FreeBSD P O R T B E G I N S H E R E ******************** Michael Smith +*/ + +#if PORTABLE_LONGMULDIV == 0 +DPUNS ficlLongMul(FICL_UNS x, FICL_UNS y) +{ + DPUNS q; + u_int64_t qx; + + qx = (u_int64_t)x * (u_int64_t) y; + + q.hi = (u_int32_t)( qx >> 32 ); + q.lo = (u_int32_t)( qx & 0xFFFFFFFFL); + + return q; +} + +UNSQR ficlLongDiv(DPUNS q, FICL_UNS y) +{ + UNSQR result; + u_int64_t qx, qh; + + qh = q.hi; + qx = (qh << 32) | q.lo; + + result.quot = qx / y; + result.rem = qx % y; + + return result; +} +#endif + +void ficlTextOut(FICL_VM *pVM, char *msg, int fNewline) +{ + IGNORE(pVM); + + while(*msg != 0) + putchar(*(msg++)); + if (fNewline) + putchar('\n'); + + return; +} + +void *ficlMalloc (size_t size) +{ + return malloc(size); +} + +void *ficlRealloc (void *p, size_t size) +{ + return realloc(p, size); +} + +void ficlFree (void *p) +{ + free(p); +} + + +/* +** Stub function for dictionary access control - does nothing +** by default, user can redefine to guarantee exclusive dict +** access to a single thread for updates. All dict update code +** is guaranteed to be bracketed as follows: +** ficlLockDictionary(TRUE); +** +** ficlLockDictionary(FALSE); +** +** Returns zero if successful, nonzero if unable to acquire lock +** befor timeout (optional - could also block forever) +*/ +#if FICL_MULTITHREAD +int ficlLockDictionary(short fLock) +{ + IGNORE(fLock); + return 0; +} +#endif /* FICL_MULTITHREAD */ Property changes on: user/ngie/make_check/sys/boot/ficl/riscv/sysdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/boot/ficl/riscv/sysdep.h =================================================================== --- user/ngie/make_check/sys/boot/ficl/riscv/sysdep.h (nonexistent) +++ user/ngie/make_check/sys/boot/ficl/riscv/sysdep.h (revision 295078) @@ -0,0 +1,411 @@ +/******************************************************************* + s y s d e p . h +** Forth Inspired Command Language +** Author: John Sadler (john_sadler@alum.mit.edu) +** Created: 16 Oct 1997 +** Ficl system dependent types and prototypes... +** +** Note: Ficl also depends on the use of "assert" when +** FICL_ROBUST is enabled. This may require some consideration +** in firmware systems since assert often +** assumes stderr/stdout. +** $Id: sysdep.h,v 1.6 2001-04-26 21:41:55-07 jsadler Exp jsadler $ +*******************************************************************/ +/* +** Copyright (c) 1997-2001 John Sadler (john_sadler@alum.mit.edu) +** All rights reserved. +** +** Get the latest Ficl release at http://ficl.sourceforge.net +** +** L I C E N S E and D I S C L A I M E R +** +** Redistribution and use in source and binary forms, with or without +** modification, are permitted provided that the following conditions +** are met: +** 1. Redistributions of source code must retain the above copyright +** notice, this list of conditions and the following disclaimer. +** 2. Redistributions in binary form must reproduce the above copyright +** notice, this list of conditions and the following disclaimer in the +** documentation and/or other materials provided with the distribution. +** +** THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +** ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +** IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +** ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +** FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +** DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +** OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +** HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +** LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +** OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +** SUCH DAMAGE. +** +** I am interested in hearing from anyone who uses ficl. If you have +** a problem, a success story, a defect, an enhancement request, or +** if you would like to contribute to the ficl release, please send +** contact me by email at the address above. +** +** $Id: sysdep.h,v 1.6 2001-04-26 21:41:55-07 jsadler Exp jsadler $ +** $FreeBSD$ +*/ + +#if !defined (__SYSDEP_H__) +#define __SYSDEP_H__ + +#include + +#include /* size_t, NULL */ +#include +#include + +#if !defined IGNORE /* Macro to silence unused param warnings */ +#define IGNORE(x) (void)(x) +#endif + +/* +** TRUE and FALSE for C boolean operations, and +** portable 32 bit types for CELLs +** +*/ +#if !defined TRUE +#define TRUE 1 +#endif +#if !defined FALSE +#define FALSE 0 +#endif + + +/* +** System dependent data type declarations... +*/ +#if !defined INT32 +#define INT32 int +#endif + +#if !defined UNS32 +#define UNS32 unsigned int +#endif + +#if !defined UNS16 +#define UNS16 unsigned short +#endif + +#if !defined UNS8 +#define UNS8 unsigned char +#endif + +#if !defined NULL +#define NULL ((void *)0) +#endif + +/* +** FICL_UNS and FICL_INT must have the same size as a void* on +** the target system. A CELL is a union of void*, FICL_UNS, and +** FICL_INT. +** (11/2000: same for FICL_FLOAT) +*/ +#if !defined FICL_INT +#define FICL_INT long +#endif + +#if !defined FICL_UNS +#define FICL_UNS unsigned long +#endif + +#if !defined FICL_FLOAT +#define FICL_FLOAT float +#endif + +/* +** Ficl presently supports values of 32 and 64 for BITS_PER_CELL +*/ +#if !defined BITS_PER_CELL +#define BITS_PER_CELL 64 +#endif + +#if ((BITS_PER_CELL != 32) && (BITS_PER_CELL != 64)) + Error! +#endif + +typedef struct +{ + FICL_UNS hi; + FICL_UNS lo; +} DPUNS; + +typedef struct +{ + FICL_UNS quot; + FICL_UNS rem; +} UNSQR; + +typedef struct +{ + FICL_INT hi; + FICL_INT lo; +} DPINT; + +typedef struct +{ + FICL_INT quot; + FICL_INT rem; +} INTQR; + + +/* +** B U I L D C O N T R O L S +*/ + +#if !defined (FICL_MINIMAL) +#define FICL_MINIMAL 0 +#endif +#if (FICL_MINIMAL) +#define FICL_WANT_SOFTWORDS 0 +#define FICL_WANT_FLOAT 0 +#define FICL_WANT_USER 0 +#define FICL_WANT_LOCALS 0 +#define FICL_WANT_DEBUGGER 0 +#define FICL_WANT_OOP 0 +#define FICL_PLATFORM_EXTEND 0 +#define FICL_MULTITHREAD 0 +#define FICL_ROBUST 0 +#define FICL_EXTENDED_PREFIX 0 +#endif + +/* +** FICL_PLATFORM_EXTEND +** Includes words defined in ficlCompilePlatform +*/ +#if !defined (FICL_PLATFORM_EXTEND) +#define FICL_PLATFORM_EXTEND 1 +#endif + +/* +** FICL_WANT_FLOAT +** Includes a floating point stack for the VM, and words to do float operations. +** Contributed by Guy Carver +*/ +#if !defined (FICL_WANT_FLOAT) +#define FICL_WANT_FLOAT 0 +#endif + +/* +** FICL_WANT_DEBUGGER +** Inludes a simple source level debugger +*/ +#if !defined (FICL_WANT_DEBUGGER) +#define FICL_WANT_DEBUGGER 1 +#endif + +/* +** User variables: per-instance variables bound to the VM. +** Kinda like thread-local storage. Could be implemented in a +** VM private dictionary, but I've chosen the lower overhead +** approach of an array of CELLs instead. +*/ +#if !defined FICL_WANT_USER +#define FICL_WANT_USER 1 +#endif + +#if !defined FICL_USER_CELLS +#define FICL_USER_CELLS 16 +#endif + +/* +** FICL_WANT_LOCALS controls the creation of the LOCALS wordset and +** a private dictionary for local variable compilation. +*/ +#if !defined FICL_WANT_LOCALS +#define FICL_WANT_LOCALS 1 +#endif + +/* Max number of local variables per definition */ +#if !defined FICL_MAX_LOCALS +#define FICL_MAX_LOCALS 16 +#endif + +/* +** FICL_WANT_OOP +** Inludes object oriented programming support (in softwords) +** OOP support requires locals and user variables! +*/ +#if !(FICL_WANT_LOCALS) || !(FICL_WANT_USER) +#if !defined (FICL_WANT_OOP) +#define FICL_WANT_OOP 0 +#endif +#endif + +#if !defined (FICL_WANT_OOP) +#define FICL_WANT_OOP 1 +#endif + +/* +** FICL_WANT_SOFTWORDS +** Controls inclusion of all softwords in softcore.c +*/ +#if !defined (FICL_WANT_SOFTWORDS) +#define FICL_WANT_SOFTWORDS 1 +#endif + +/* +** FICL_MULTITHREAD enables dictionary mutual exclusion +** wia the ficlLockDictionary system dependent function. +** Note: this implementation is experimental and poorly +** tested. Further, it's unnecessary unless you really +** intend to have multiple SESSIONS (poor choice of name +** on my part) - that is, threads that modify the dictionary +** at the same time. +*/ +#if !defined FICL_MULTITHREAD +#define FICL_MULTITHREAD 0 +#endif + +/* +** PORTABLE_LONGMULDIV causes ficlLongMul and ficlLongDiv to be +** defined in C in sysdep.c. Use this if you cannot easily +** generate an inline asm definition +*/ +#if !defined (PORTABLE_LONGMULDIV) +#define PORTABLE_LONGMULDIV 0 +#endif + +/* +** INLINE_INNER_LOOP causes the inner interpreter to be inline code +** instead of a function call. This is mainly because MS VC++ 5 +** chokes with an internal compiler error on the function version. +** in release mode. Sheesh. +*/ +#if !defined INLINE_INNER_LOOP +#if defined _DEBUG +#define INLINE_INNER_LOOP 0 +#else +#define INLINE_INNER_LOOP 1 +#endif +#endif + +/* +** FICL_ROBUST enables bounds checking of stacks and the dictionary. +** This will detect stack over and underflows and dictionary overflows. +** Any exceptional condition will result in an assertion failure. +** (As generated by the ANSI assert macro) +** FICL_ROBUST == 1 --> stack checking in the outer interpreter +** FICL_ROBUST == 2 also enables checking in many primitives +*/ + +#if !defined FICL_ROBUST +#define FICL_ROBUST 2 +#endif + +/* +** FICL_DEFAULT_STACK Specifies the default size (in CELLs) of +** a new virtual machine's stacks, unless overridden at +** create time. +*/ +#if !defined FICL_DEFAULT_STACK +#define FICL_DEFAULT_STACK 128 +#endif + +/* +** FICL_DEFAULT_DICT specifies the number of CELLs to allocate +** for the system dictionary by default. The value +** can be overridden at startup time as well. +** FICL_DEFAULT_ENV specifies the number of cells to allot +** for the environment-query dictionary. +*/ +#if !defined FICL_DEFAULT_DICT +#define FICL_DEFAULT_DICT 12288 +#endif + +#if !defined FICL_DEFAULT_ENV +#define FICL_DEFAULT_ENV 260 +#endif + +/* +** FICL_DEFAULT_VOCS specifies the maximum number of wordlists in +** the dictionary search order. See Forth DPANS sec 16.3.3 +** (file://dpans16.htm#16.3.3) +*/ +#if !defined FICL_DEFAULT_VOCS +#define FICL_DEFAULT_VOCS 16 +#endif + +/* +** FICL_MAX_PARSE_STEPS controls the size of an array in the FICL_SYSTEM structure +** that stores pointers to parser extension functions. I would never expect to have +** more than 8 of these, so that's the default limit. Too many of these functions +** will probably exact a nasty performance penalty. +*/ +#if !defined FICL_MAX_PARSE_STEPS +#define FICL_MAX_PARSE_STEPS 8 +#endif + +/* +** FICL_EXTENDED_PREFIX enables a bunch of extra prefixes in prefix.c and prefix.fr (if +** included as part of softcore.c) +*/ +#if !defined FICL_EXTENDED_PREFIX +#define FICL_EXTENDED_PREFIX 0 +#endif + +/* +** FICL_ALIGN is the power of two to which the dictionary +** pointer address must be aligned. This value is usually +** either 1 or 2, depending on the memory architecture +** of the target system; 2 is safe on any 16 or 32 bit +** machine. 3 would be appropriate for a 64 bit machine. +*/ +#if !defined FICL_ALIGN +#define FICL_ALIGN 3 +#define FICL_ALIGN_ADD ((1 << FICL_ALIGN) - 1) +#endif + +/* +** System dependent routines -- +** edit the implementations in sysdep.c to be compatible +** with your runtime environment... +** ficlTextOut sends a NULL terminated string to the +** default output device - used for system error messages +** ficlMalloc and ficlFree have the same semantics as malloc and free +** in standard C +** ficlLongMul multiplies two UNS32s and returns a 64 bit unsigned +** product +** ficlLongDiv divides an UNS64 by an UNS32 and returns UNS32 quotient +** and remainder +*/ +struct vm; +void ficlTextOut(struct vm *pVM, char *msg, int fNewline); +void *ficlMalloc (size_t size); +void ficlFree (void *p); +void *ficlRealloc(void *p, size_t size); +/* +** Stub function for dictionary access control - does nothing +** by default, user can redefine to guarantee exclusive dict +** access to a single thread for updates. All dict update code +** must be bracketed as follows: +** ficlLockDictionary(TRUE); +** +** ficlLockDictionary(FALSE); +** +** Returns zero if successful, nonzero if unable to acquire lock +** before timeout (optional - could also block forever) +** +** NOTE: this function must be implemented with lock counting +** semantics: nested calls must behave properly. +*/ +#if FICL_MULTITHREAD +int ficlLockDictionary(short fLock); +#else +#define ficlLockDictionary(x) 0 /* ignore */ +#endif + +/* +** 64 bit integer math support routines: multiply two UNS32s +** to get a 64 bit product, & divide the product by an UNS32 +** to get an UNS32 quotient and remainder. Much easier in asm +** on a 32 bit CPU than in C, which usually doesn't support +** the double length result (but it should). +*/ +DPUNS ficlLongMul(FICL_UNS x, FICL_UNS y); +UNSQR ficlLongDiv(DPUNS q, FICL_UNS y); + +#endif /*__SYSDEP_H__*/ Property changes on: user/ngie/make_check/sys/boot/ficl/riscv/sysdep.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/boot =================================================================== --- user/ngie/make_check/sys/boot (revision 295077) +++ user/ngie/make_check/sys/boot (revision 295078) Property changes on: user/ngie/make_check/sys/boot ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/boot:r295034-295077 Index: user/ngie/make_check/sys/cddl/compat/opensolaris/sys/atomic.h =================================================================== --- user/ngie/make_check/sys/cddl/compat/opensolaris/sys/atomic.h (revision 295077) +++ user/ngie/make_check/sys/cddl/compat/opensolaris/sys/atomic.h (revision 295078) @@ -1,142 +1,142 @@ /*- * Copyright (c) 2007 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _OPENSOLARIS_SYS_ATOMIC_H_ #define _OPENSOLARIS_SYS_ATOMIC_H_ #include #include #define casptr(_a, _b, _c) \ atomic_cmpset_ptr((volatile uintptr_t *)(_a), (uintptr_t)(_b), (uintptr_t) (_c)) #define cas32 atomic_cmpset_32 #if !defined(__LP64__) && !defined(__mips_n32) && !defined(ARM_HAVE_ATOMIC64) extern void atomic_add_64(volatile uint64_t *target, int64_t delta); extern void atomic_dec_64(volatile uint64_t *target); #endif #ifndef __sparc64__ extern uint32_t atomic_cas_32(volatile uint32_t *target, uint32_t cmp, uint32_t newval); extern uint64_t atomic_cas_64(volatile uint64_t *target, uint64_t cmp, uint64_t newval); #endif extern uint64_t atomic_add_64_nv(volatile uint64_t *target, int64_t delta); extern uint8_t atomic_or_8_nv(volatile uint8_t *target, uint8_t value); extern void membar_producer(void); #if defined(__sparc64__) || defined(__powerpc__) || defined(__arm__) || \ - defined(__mips__) || defined(__aarch64__) + defined(__mips__) || defined(__aarch64__) || defined(__riscv__) extern void atomic_or_8(volatile uint8_t *target, uint8_t value); #else static __inline void atomic_or_8(volatile uint8_t *target, uint8_t value) { atomic_set_8(target, value); } #endif static __inline uint32_t atomic_add_32_nv(volatile uint32_t *target, int32_t delta) { return (atomic_fetchadd_32(target, delta) + delta); } static __inline u_int atomic_add_int_nv(volatile u_int *target, int delta) { return (atomic_add_32_nv(target, delta)); } static __inline void atomic_dec_32(volatile uint32_t *target) { atomic_subtract_32(target, 1); } static __inline uint32_t atomic_dec_32_nv(volatile uint32_t *target) { return (atomic_fetchadd_32(target, -1) - 1); } #if defined(__LP64__) || defined(__mips_n32) || defined(ARM_HAVE_ATOMIC64) static __inline void atomic_dec_64(volatile uint64_t *target) { atomic_subtract_64(target, 1); } #endif static __inline void atomic_inc_32(volatile uint32_t *target) { atomic_add_32(target, 1); } static __inline uint32_t atomic_inc_32_nv(volatile uint32_t *target) { return (atomic_add_32_nv(target, 1)); } static __inline void atomic_inc_64(volatile uint64_t *target) { atomic_add_64(target, 1); } static __inline uint64_t atomic_inc_64_nv(volatile uint64_t *target) { return (atomic_add_64_nv(target, 1)); } static __inline uint64_t atomic_dec_64_nv(volatile uint64_t *target) { return (atomic_add_64_nv(target, -1)); } #if !defined(COMPAT_32BIT) && defined(__LP64__) static __inline void * atomic_cas_ptr(volatile void *target, void *cmp, void *newval) { return ((void *)atomic_cas_64((volatile uint64_t *)target, (uint64_t)cmp, (uint64_t)newval)); } #else static __inline void * atomic_cas_ptr(volatile void *target, void *cmp, void *newval) { return ((void *)atomic_cas_32((volatile uint32_t *)target, (uint32_t)cmp, (uint32_t)newval)); } #endif /* !defined(COMPAT_32BIT) && defined(__LP64__) */ #endif /* !_OPENSOLARIS_SYS_ATOMIC_H_ */ Index: user/ngie/make_check/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c =================================================================== --- user/ngie/make_check/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c (revision 295077) +++ user/ngie/make_check/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.c (revision 295078) @@ -1,3187 +1,3209 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved. * * Copyright (c) 2006-2010 Pawel Jakub Dawidek * All rights reserved. * * Portions Copyright 2010 Robert Milkowski * * Copyright 2011 Nexenta Systems, Inc. All rights reserved. * Copyright (c) 2012, 2014 by Delphix. All rights reserved. * Copyright (c) 2013, Joyent, Inc. All rights reserved. */ /* Portions Copyright 2011 Martin Matuska */ /* * ZFS volume emulation driver. * * Makes a DMU object look like a volume of arbitrary size, up to 2^64 bytes. * Volumes are accessed through the symbolic links named: * * /dev/zvol/dsk// * /dev/zvol/rdsk// * * These links are created by the /dev filesystem (sdev_zvolops.c). * Volumes are persistent through reboot. No user command needs to be * run before opening and using a device. * * FreeBSD notes. * On FreeBSD ZVOLs are simply GEOM providers like any other storage device * in the system. */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "zfs_namecheck.h" #ifndef illumos struct g_class zfs_zvol_class = { .name = "ZFS::ZVOL", .version = G_VERSION, }; DECLARE_GEOM_CLASS(zfs_zvol_class, zfs_zvol); #endif void *zfsdev_state; static char *zvol_tag = "zvol_tag"; #define ZVOL_DUMPSIZE "dumpsize" /* * This lock protects the zfsdev_state structure from being modified * while it's being used, e.g. an open that comes in before a create * finishes. It also protects temporary opens of the dataset so that, * e.g., an open doesn't get a spurious EBUSY. */ #ifdef illumos kmutex_t zfsdev_state_lock; #else /* * In FreeBSD we've replaced the upstream zfsdev_state_lock with the * spa_namespace_lock in the ZVOL code. */ #define zfsdev_state_lock spa_namespace_lock #endif static uint32_t zvol_minors; #ifndef illumos SYSCTL_DECL(_vfs_zfs); SYSCTL_NODE(_vfs_zfs, OID_AUTO, vol, CTLFLAG_RW, 0, "ZFS VOLUME"); static int volmode = ZFS_VOLMODE_GEOM; SYSCTL_INT(_vfs_zfs_vol, OID_AUTO, mode, CTLFLAG_RWTUN, &volmode, 0, "Expose as GEOM providers (1), device files (2) or neither"); +static boolean_t zpool_on_zvol = B_FALSE; +SYSCTL_INT(_vfs_zfs_vol, OID_AUTO, recursive, CTLFLAG_RWTUN, &zpool_on_zvol, 0, + "Allow zpools to use zvols as vdevs (DANGEROUS)"); #endif typedef struct zvol_extent { list_node_t ze_node; dva_t ze_dva; /* dva associated with this extent */ uint64_t ze_nblks; /* number of blocks in extent */ } zvol_extent_t; /* * The in-core state of each volume. */ typedef struct zvol_state { #ifndef illumos LIST_ENTRY(zvol_state) zv_links; #endif char zv_name[MAXPATHLEN]; /* pool/dd name */ uint64_t zv_volsize; /* amount of space we advertise */ uint64_t zv_volblocksize; /* volume block size */ #ifdef illumos minor_t zv_minor; /* minor number */ #else struct cdev *zv_dev; /* non-GEOM device */ struct g_provider *zv_provider; /* GEOM provider */ #endif uint8_t zv_min_bs; /* minimum addressable block shift */ uint8_t zv_flags; /* readonly, dumpified, etc. */ objset_t *zv_objset; /* objset handle */ #ifdef illumos uint32_t zv_open_count[OTYPCNT]; /* open counts */ #endif uint32_t zv_total_opens; /* total open count */ zilog_t *zv_zilog; /* ZIL handle */ list_t zv_extents; /* List of extents for dump */ znode_t zv_znode; /* for range locking */ dmu_buf_t *zv_dbuf; /* bonus handle */ #ifndef illumos int zv_state; int zv_volmode; /* Provide GEOM or cdev */ struct bio_queue_head zv_queue; struct mtx zv_queue_mtx; /* zv_queue mutex */ #endif } zvol_state_t; #ifndef illumos static LIST_HEAD(, zvol_state) all_zvols; #endif /* * zvol specific flags */ #define ZVOL_RDONLY 0x1 #define ZVOL_DUMPIFIED 0x2 #define ZVOL_EXCL 0x4 #define ZVOL_WCE 0x8 /* * zvol maximum transfer in one DMU tx. */ int zvol_maxphys = DMU_MAX_ACCESS/2; /* * Toggle unmap functionality. */ boolean_t zvol_unmap_enabled = B_TRUE; #ifndef illumos SYSCTL_INT(_vfs_zfs_vol, OID_AUTO, unmap_enabled, CTLFLAG_RWTUN, &zvol_unmap_enabled, 0, "Enable UNMAP functionality"); static d_open_t zvol_d_open; static d_close_t zvol_d_close; static d_read_t zvol_read; static d_write_t zvol_write; static d_ioctl_t zvol_d_ioctl; static d_strategy_t zvol_strategy; static struct cdevsw zvol_cdevsw = { .d_version = D_VERSION, .d_open = zvol_d_open, .d_close = zvol_d_close, .d_read = zvol_read, .d_write = zvol_write, .d_ioctl = zvol_d_ioctl, .d_strategy = zvol_strategy, .d_name = "zvol", .d_flags = D_DISK | D_TRACKCLOSE, }; static void zvol_geom_run(zvol_state_t *zv); static void zvol_geom_destroy(zvol_state_t *zv); static int zvol_geom_access(struct g_provider *pp, int acr, int acw, int ace); static void zvol_geom_start(struct bio *bp); static void zvol_geom_worker(void *arg); static void zvol_log_truncate(zvol_state_t *zv, dmu_tx_t *tx, uint64_t off, uint64_t len, boolean_t sync); #endif /* !illumos */ extern int zfs_set_prop_nvlist(const char *, zprop_source_t, nvlist_t *, nvlist_t *); static int zvol_remove_zv(zvol_state_t *); static int zvol_get_data(void *arg, lr_write_t *lr, char *buf, zio_t *zio); static int zvol_dumpify(zvol_state_t *zv); static int zvol_dump_fini(zvol_state_t *zv); static int zvol_dump_init(zvol_state_t *zv, boolean_t resize); static void zvol_size_changed(zvol_state_t *zv, uint64_t volsize) { #ifdef illumos dev_t dev = makedevice(ddi_driver_major(zfs_dip), zv->zv_minor); zv->zv_volsize = volsize; VERIFY(ddi_prop_update_int64(dev, zfs_dip, "Size", volsize) == DDI_SUCCESS); VERIFY(ddi_prop_update_int64(dev, zfs_dip, "Nblocks", lbtodb(volsize)) == DDI_SUCCESS); /* Notify specfs to invalidate the cached size */ spec_size_invalidate(dev, VBLK); spec_size_invalidate(dev, VCHR); #else /* !illumos */ zv->zv_volsize = volsize; if (zv->zv_volmode == ZFS_VOLMODE_GEOM) { struct g_provider *pp; pp = zv->zv_provider; if (pp == NULL) return; g_topology_lock(); g_resize_provider(pp, zv->zv_volsize); g_topology_unlock(); } #endif /* illumos */ } int zvol_check_volsize(uint64_t volsize, uint64_t blocksize) { if (volsize == 0) return (SET_ERROR(EINVAL)); if (volsize % blocksize != 0) return (SET_ERROR(EINVAL)); #ifdef _ILP32 if (volsize - 1 > SPEC_MAXOFFSET_T) return (SET_ERROR(EOVERFLOW)); #endif return (0); } int zvol_check_volblocksize(uint64_t volblocksize) { if (volblocksize < SPA_MINBLOCKSIZE || volblocksize > SPA_OLD_MAXBLOCKSIZE || !ISP2(volblocksize)) return (SET_ERROR(EDOM)); return (0); } int zvol_get_stats(objset_t *os, nvlist_t *nv) { int error; dmu_object_info_t doi; uint64_t val; error = zap_lookup(os, ZVOL_ZAP_OBJ, "size", 8, 1, &val); if (error) return (error); dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_VOLSIZE, val); error = dmu_object_info(os, ZVOL_OBJ, &doi); if (error == 0) { dsl_prop_nvlist_add_uint64(nv, ZFS_PROP_VOLBLOCKSIZE, doi.doi_data_block_size); } return (error); } static zvol_state_t * zvol_minor_lookup(const char *name) { #ifdef illumos minor_t minor; #endif zvol_state_t *zv; ASSERT(MUTEX_HELD(&zfsdev_state_lock)); #ifdef illumos for (minor = 1; minor <= ZFSDEV_MAX_MINOR; minor++) { zv = zfsdev_get_soft_state(minor, ZSST_ZVOL); if (zv == NULL) continue; #else LIST_FOREACH(zv, &all_zvols, zv_links) { #endif if (strcmp(zv->zv_name, name) == 0) return (zv); } return (NULL); } /* extent mapping arg */ struct maparg { zvol_state_t *ma_zv; uint64_t ma_blks; }; /*ARGSUSED*/ static int zvol_map_block(spa_t *spa, zilog_t *zilog, const blkptr_t *bp, const zbookmark_phys_t *zb, const dnode_phys_t *dnp, void *arg) { struct maparg *ma = arg; zvol_extent_t *ze; int bs = ma->ma_zv->zv_volblocksize; if (bp == NULL || BP_IS_HOLE(bp) || zb->zb_object != ZVOL_OBJ || zb->zb_level != 0) return (0); VERIFY(!BP_IS_EMBEDDED(bp)); VERIFY3U(ma->ma_blks, ==, zb->zb_blkid); ma->ma_blks++; /* Abort immediately if we have encountered gang blocks */ if (BP_IS_GANG(bp)) return (SET_ERROR(EFRAGS)); /* * See if the block is at the end of the previous extent. */ ze = list_tail(&ma->ma_zv->zv_extents); if (ze && DVA_GET_VDEV(BP_IDENTITY(bp)) == DVA_GET_VDEV(&ze->ze_dva) && DVA_GET_OFFSET(BP_IDENTITY(bp)) == DVA_GET_OFFSET(&ze->ze_dva) + ze->ze_nblks * bs) { ze->ze_nblks++; return (0); } dprintf_bp(bp, "%s", "next blkptr:"); /* start a new extent */ ze = kmem_zalloc(sizeof (zvol_extent_t), KM_SLEEP); ze->ze_dva = bp->blk_dva[0]; /* structure assignment */ ze->ze_nblks = 1; list_insert_tail(&ma->ma_zv->zv_extents, ze); return (0); } static void zvol_free_extents(zvol_state_t *zv) { zvol_extent_t *ze; while (ze = list_head(&zv->zv_extents)) { list_remove(&zv->zv_extents, ze); kmem_free(ze, sizeof (zvol_extent_t)); } } static int zvol_get_lbas(zvol_state_t *zv) { objset_t *os = zv->zv_objset; struct maparg ma; int err; ma.ma_zv = zv; ma.ma_blks = 0; zvol_free_extents(zv); /* commit any in-flight changes before traversing the dataset */ txg_wait_synced(dmu_objset_pool(os), 0); err = traverse_dataset(dmu_objset_ds(os), 0, TRAVERSE_PRE | TRAVERSE_PREFETCH_METADATA, zvol_map_block, &ma); if (err || ma.ma_blks != (zv->zv_volsize / zv->zv_volblocksize)) { zvol_free_extents(zv); return (err ? err : EIO); } return (0); } /* ARGSUSED */ void zvol_create_cb(objset_t *os, void *arg, cred_t *cr, dmu_tx_t *tx) { zfs_creat_t *zct = arg; nvlist_t *nvprops = zct->zct_props; int error; uint64_t volblocksize, volsize; VERIFY(nvlist_lookup_uint64(nvprops, zfs_prop_to_name(ZFS_PROP_VOLSIZE), &volsize) == 0); if (nvlist_lookup_uint64(nvprops, zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE), &volblocksize) != 0) volblocksize = zfs_prop_default_numeric(ZFS_PROP_VOLBLOCKSIZE); /* * These properties must be removed from the list so the generic * property setting step won't apply to them. */ VERIFY(nvlist_remove_all(nvprops, zfs_prop_to_name(ZFS_PROP_VOLSIZE)) == 0); (void) nvlist_remove_all(nvprops, zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE)); error = dmu_object_claim(os, ZVOL_OBJ, DMU_OT_ZVOL, volblocksize, DMU_OT_NONE, 0, tx); ASSERT(error == 0); error = zap_create_claim(os, ZVOL_ZAP_OBJ, DMU_OT_ZVOL_PROP, DMU_OT_NONE, 0, tx); ASSERT(error == 0); error = zap_update(os, ZVOL_ZAP_OBJ, "size", 8, 1, &volsize, tx); ASSERT(error == 0); } /* * Replay a TX_TRUNCATE ZIL transaction if asked. TX_TRUNCATE is how we * implement DKIOCFREE/free-long-range. */ static int zvol_replay_truncate(zvol_state_t *zv, lr_truncate_t *lr, boolean_t byteswap) { uint64_t offset, length; if (byteswap) byteswap_uint64_array(lr, sizeof (*lr)); offset = lr->lr_offset; length = lr->lr_length; return (dmu_free_long_range(zv->zv_objset, ZVOL_OBJ, offset, length)); } /* * Replay a TX_WRITE ZIL transaction that didn't get committed * after a system failure */ static int zvol_replay_write(zvol_state_t *zv, lr_write_t *lr, boolean_t byteswap) { objset_t *os = zv->zv_objset; char *data = (char *)(lr + 1); /* data follows lr_write_t */ uint64_t offset, length; dmu_tx_t *tx; int error; if (byteswap) byteswap_uint64_array(lr, sizeof (*lr)); offset = lr->lr_offset; length = lr->lr_length; /* If it's a dmu_sync() block, write the whole block */ if (lr->lr_common.lrc_reclen == sizeof (lr_write_t)) { uint64_t blocksize = BP_GET_LSIZE(&lr->lr_blkptr); if (length < blocksize) { offset -= offset % blocksize; length = blocksize; } } tx = dmu_tx_create(os); dmu_tx_hold_write(tx, ZVOL_OBJ, offset, length); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); } else { dmu_write(os, ZVOL_OBJ, offset, length, data, tx); dmu_tx_commit(tx); } return (error); } /* ARGSUSED */ static int zvol_replay_err(zvol_state_t *zv, lr_t *lr, boolean_t byteswap) { return (SET_ERROR(ENOTSUP)); } /* * Callback vectors for replaying records. * Only TX_WRITE and TX_TRUNCATE are needed for zvol. */ zil_replay_func_t *zvol_replay_vector[TX_MAX_TYPE] = { zvol_replay_err, /* 0 no such transaction type */ zvol_replay_err, /* TX_CREATE */ zvol_replay_err, /* TX_MKDIR */ zvol_replay_err, /* TX_MKXATTR */ zvol_replay_err, /* TX_SYMLINK */ zvol_replay_err, /* TX_REMOVE */ zvol_replay_err, /* TX_RMDIR */ zvol_replay_err, /* TX_LINK */ zvol_replay_err, /* TX_RENAME */ zvol_replay_write, /* TX_WRITE */ zvol_replay_truncate, /* TX_TRUNCATE */ zvol_replay_err, /* TX_SETATTR */ zvol_replay_err, /* TX_ACL */ zvol_replay_err, /* TX_CREATE_ACL */ zvol_replay_err, /* TX_CREATE_ATTR */ zvol_replay_err, /* TX_CREATE_ACL_ATTR */ zvol_replay_err, /* TX_MKDIR_ACL */ zvol_replay_err, /* TX_MKDIR_ATTR */ zvol_replay_err, /* TX_MKDIR_ACL_ATTR */ zvol_replay_err, /* TX_WRITE2 */ }; #ifdef illumos int zvol_name2minor(const char *name, minor_t *minor) { zvol_state_t *zv; mutex_enter(&zfsdev_state_lock); zv = zvol_minor_lookup(name); if (minor && zv) *minor = zv->zv_minor; mutex_exit(&zfsdev_state_lock); return (zv ? 0 : -1); } #endif /* illumos */ /* * Create a minor node (plus a whole lot more) for the specified volume. */ int zvol_create_minor(const char *name) { zfs_soft_state_t *zs; zvol_state_t *zv; objset_t *os; dmu_object_info_t doi; #ifdef illumos minor_t minor = 0; char chrbuf[30], blkbuf[30]; #else struct cdev *dev; struct g_provider *pp; struct g_geom *gp; uint64_t volsize, mode; #endif int error; #ifndef illumos ZFS_LOG(1, "Creating ZVOL %s...", name); #endif mutex_enter(&zfsdev_state_lock); if (zvol_minor_lookup(name) != NULL) { mutex_exit(&zfsdev_state_lock); return (SET_ERROR(EEXIST)); } /* lie and say we're read-only */ error = dmu_objset_own(name, DMU_OST_ZVOL, B_TRUE, FTAG, &os); if (error) { mutex_exit(&zfsdev_state_lock); return (error); } #ifdef illumos if ((minor = zfsdev_minor_alloc()) == 0) { dmu_objset_disown(os, FTAG); mutex_exit(&zfsdev_state_lock); return (SET_ERROR(ENXIO)); } if (ddi_soft_state_zalloc(zfsdev_state, minor) != DDI_SUCCESS) { dmu_objset_disown(os, FTAG); mutex_exit(&zfsdev_state_lock); return (SET_ERROR(EAGAIN)); } (void) ddi_prop_update_string(minor, zfs_dip, ZVOL_PROP_NAME, (char *)name); (void) snprintf(chrbuf, sizeof (chrbuf), "%u,raw", minor); if (ddi_create_minor_node(zfs_dip, chrbuf, S_IFCHR, minor, DDI_PSEUDO, 0) == DDI_FAILURE) { ddi_soft_state_free(zfsdev_state, minor); dmu_objset_disown(os, FTAG); mutex_exit(&zfsdev_state_lock); return (SET_ERROR(EAGAIN)); } (void) snprintf(blkbuf, sizeof (blkbuf), "%u", minor); if (ddi_create_minor_node(zfs_dip, blkbuf, S_IFBLK, minor, DDI_PSEUDO, 0) == DDI_FAILURE) { ddi_remove_minor_node(zfs_dip, chrbuf); ddi_soft_state_free(zfsdev_state, minor); dmu_objset_disown(os, FTAG); mutex_exit(&zfsdev_state_lock); return (SET_ERROR(EAGAIN)); } zs = ddi_get_soft_state(zfsdev_state, minor); zs->zss_type = ZSST_ZVOL; zv = zs->zss_data = kmem_zalloc(sizeof (zvol_state_t), KM_SLEEP); #else /* !illumos */ zv = kmem_zalloc(sizeof(*zv), KM_SLEEP); zv->zv_state = 0; error = zap_lookup(os, ZVOL_ZAP_OBJ, "size", 8, 1, &volsize); if (error) { kmem_free(zv, sizeof(*zv)); dmu_objset_disown(os, zvol_tag); mutex_exit(&zfsdev_state_lock); return (error); } error = dsl_prop_get_integer(name, zfs_prop_to_name(ZFS_PROP_VOLMODE), &mode, NULL); if (error != 0 || mode == ZFS_VOLMODE_DEFAULT) mode = volmode; DROP_GIANT(); zv->zv_volsize = volsize; zv->zv_volmode = mode; if (zv->zv_volmode == ZFS_VOLMODE_GEOM) { g_topology_lock(); gp = g_new_geomf(&zfs_zvol_class, "zfs::zvol::%s", name); gp->start = zvol_geom_start; gp->access = zvol_geom_access; pp = g_new_providerf(gp, "%s/%s", ZVOL_DRIVER, name); pp->flags |= G_PF_DIRECT_RECEIVE | G_PF_DIRECT_SEND; pp->sectorsize = DEV_BSIZE; pp->mediasize = zv->zv_volsize; pp->private = zv; zv->zv_provider = pp; bioq_init(&zv->zv_queue); mtx_init(&zv->zv_queue_mtx, "zvol", NULL, MTX_DEF); } else if (zv->zv_volmode == ZFS_VOLMODE_DEV) { if (make_dev_p(MAKEDEV_CHECKNAME | MAKEDEV_WAITOK, &dev, &zvol_cdevsw, NULL, UID_ROOT, GID_OPERATOR, 0640, "%s/%s", ZVOL_DRIVER, name) != 0) { kmem_free(zv, sizeof(*zv)); dmu_objset_disown(os, FTAG); mutex_exit(&zfsdev_state_lock); return (SET_ERROR(ENXIO)); } zv->zv_dev = dev; dev->si_iosize_max = MAXPHYS; dev->si_drv2 = zv; } LIST_INSERT_HEAD(&all_zvols, zv, zv_links); #endif /* illumos */ (void) strlcpy(zv->zv_name, name, MAXPATHLEN); zv->zv_min_bs = DEV_BSHIFT; #ifdef illumos zv->zv_minor = minor; #endif zv->zv_objset = os; if (dmu_objset_is_snapshot(os) || !spa_writeable(dmu_objset_spa(os))) zv->zv_flags |= ZVOL_RDONLY; mutex_init(&zv->zv_znode.z_range_lock, NULL, MUTEX_DEFAULT, NULL); avl_create(&zv->zv_znode.z_range_avl, zfs_range_compare, sizeof (rl_t), offsetof(rl_t, r_node)); list_create(&zv->zv_extents, sizeof (zvol_extent_t), offsetof(zvol_extent_t, ze_node)); /* get and cache the blocksize */ error = dmu_object_info(os, ZVOL_OBJ, &doi); ASSERT(error == 0); zv->zv_volblocksize = doi.doi_data_block_size; if (spa_writeable(dmu_objset_spa(os))) { if (zil_replay_disable) zil_destroy(dmu_objset_zil(os), B_FALSE); else zil_replay(os, zv, zvol_replay_vector); } dmu_objset_disown(os, FTAG); zv->zv_objset = NULL; zvol_minors++; mutex_exit(&zfsdev_state_lock); #ifndef illumos if (zv->zv_volmode == ZFS_VOLMODE_GEOM) { zvol_geom_run(zv); g_topology_unlock(); } PICKUP_GIANT(); ZFS_LOG(1, "ZVOL %s created.", name); #endif return (0); } /* * Remove minor node for the specified volume. */ static int zvol_remove_zv(zvol_state_t *zv) { #ifdef illumos char nmbuf[20]; minor_t minor = zv->zv_minor; #endif ASSERT(MUTEX_HELD(&zfsdev_state_lock)); if (zv->zv_total_opens != 0) return (SET_ERROR(EBUSY)); #ifdef illumos (void) snprintf(nmbuf, sizeof (nmbuf), "%u,raw", minor); ddi_remove_minor_node(zfs_dip, nmbuf); (void) snprintf(nmbuf, sizeof (nmbuf), "%u", minor); ddi_remove_minor_node(zfs_dip, nmbuf); #else ZFS_LOG(1, "ZVOL %s destroyed.", zv->zv_name); LIST_REMOVE(zv, zv_links); if (zv->zv_volmode == ZFS_VOLMODE_GEOM) { g_topology_lock(); zvol_geom_destroy(zv); g_topology_unlock(); } else if (zv->zv_volmode == ZFS_VOLMODE_DEV) destroy_dev(zv->zv_dev); #endif avl_destroy(&zv->zv_znode.z_range_avl); mutex_destroy(&zv->zv_znode.z_range_lock); kmem_free(zv, sizeof (zvol_state_t)); #ifdef illumos ddi_soft_state_free(zfsdev_state, minor); #endif zvol_minors--; return (0); } int zvol_remove_minor(const char *name) { zvol_state_t *zv; int rc; mutex_enter(&zfsdev_state_lock); if ((zv = zvol_minor_lookup(name)) == NULL) { mutex_exit(&zfsdev_state_lock); return (SET_ERROR(ENXIO)); } rc = zvol_remove_zv(zv); mutex_exit(&zfsdev_state_lock); return (rc); } int zvol_first_open(zvol_state_t *zv) { objset_t *os; uint64_t volsize; int error; uint64_t readonly; /* lie and say we're read-only */ error = dmu_objset_own(zv->zv_name, DMU_OST_ZVOL, B_TRUE, zvol_tag, &os); if (error) return (error); zv->zv_objset = os; error = zap_lookup(os, ZVOL_ZAP_OBJ, "size", 8, 1, &volsize); if (error) { ASSERT(error == 0); dmu_objset_disown(os, zvol_tag); return (error); } error = dmu_bonus_hold(os, ZVOL_OBJ, zvol_tag, &zv->zv_dbuf); if (error) { dmu_objset_disown(os, zvol_tag); return (error); } zvol_size_changed(zv, volsize); zv->zv_zilog = zil_open(os, zvol_get_data); VERIFY(dsl_prop_get_integer(zv->zv_name, "readonly", &readonly, NULL) == 0); if (readonly || dmu_objset_is_snapshot(os) || !spa_writeable(dmu_objset_spa(os))) zv->zv_flags |= ZVOL_RDONLY; else zv->zv_flags &= ~ZVOL_RDONLY; return (error); } void zvol_last_close(zvol_state_t *zv) { zil_close(zv->zv_zilog); zv->zv_zilog = NULL; dmu_buf_rele(zv->zv_dbuf, zvol_tag); zv->zv_dbuf = NULL; /* * Evict cached data */ if (dsl_dataset_is_dirty(dmu_objset_ds(zv->zv_objset)) && !(zv->zv_flags & ZVOL_RDONLY)) txg_wait_synced(dmu_objset_pool(zv->zv_objset), 0); dmu_objset_evict_dbufs(zv->zv_objset); dmu_objset_disown(zv->zv_objset, zvol_tag); zv->zv_objset = NULL; } #ifdef illumos int zvol_prealloc(zvol_state_t *zv) { objset_t *os = zv->zv_objset; dmu_tx_t *tx; uint64_t refd, avail, usedobjs, availobjs; uint64_t resid = zv->zv_volsize; uint64_t off = 0; /* Check the space usage before attempting to allocate the space */ dmu_objset_space(os, &refd, &avail, &usedobjs, &availobjs); if (avail < zv->zv_volsize) return (SET_ERROR(ENOSPC)); /* Free old extents if they exist */ zvol_free_extents(zv); while (resid != 0) { int error; uint64_t bytes = MIN(resid, SPA_OLD_MAXBLOCKSIZE); tx = dmu_tx_create(os); dmu_tx_hold_write(tx, ZVOL_OBJ, off, bytes); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); (void) dmu_free_long_range(os, ZVOL_OBJ, 0, off); return (error); } dmu_prealloc(os, ZVOL_OBJ, off, bytes, tx); dmu_tx_commit(tx); off += bytes; resid -= bytes; } txg_wait_synced(dmu_objset_pool(os), 0); return (0); } #endif /* illumos */ static int zvol_update_volsize(objset_t *os, uint64_t volsize) { dmu_tx_t *tx; int error; ASSERT(MUTEX_HELD(&zfsdev_state_lock)); tx = dmu_tx_create(os); dmu_tx_hold_zap(tx, ZVOL_ZAP_OBJ, TRUE, NULL); dmu_tx_mark_netfree(tx); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); return (error); } error = zap_update(os, ZVOL_ZAP_OBJ, "size", 8, 1, &volsize, tx); dmu_tx_commit(tx); if (error == 0) error = dmu_free_long_range(os, ZVOL_OBJ, volsize, DMU_OBJECT_END); return (error); } void zvol_remove_minors(const char *name) { #ifdef illumos zvol_state_t *zv; char *namebuf; minor_t minor; namebuf = kmem_zalloc(strlen(name) + 2, KM_SLEEP); (void) strncpy(namebuf, name, strlen(name)); (void) strcat(namebuf, "/"); mutex_enter(&zfsdev_state_lock); for (minor = 1; minor <= ZFSDEV_MAX_MINOR; minor++) { zv = zfsdev_get_soft_state(minor, ZSST_ZVOL); if (zv == NULL) continue; if (strncmp(namebuf, zv->zv_name, strlen(namebuf)) == 0) (void) zvol_remove_zv(zv); } kmem_free(namebuf, strlen(name) + 2); mutex_exit(&zfsdev_state_lock); #else /* !illumos */ zvol_state_t *zv, *tzv; size_t namelen; namelen = strlen(name); DROP_GIANT(); mutex_enter(&zfsdev_state_lock); LIST_FOREACH_SAFE(zv, &all_zvols, zv_links, tzv) { if (strcmp(zv->zv_name, name) == 0 || (strncmp(zv->zv_name, name, namelen) == 0 && strlen(zv->zv_name) > namelen && (zv->zv_name[namelen] == '/' || zv->zv_name[namelen] == '@'))) { (void) zvol_remove_zv(zv); } } mutex_exit(&zfsdev_state_lock); PICKUP_GIANT(); #endif /* illumos */ } static int zvol_update_live_volsize(zvol_state_t *zv, uint64_t volsize) { uint64_t old_volsize = 0ULL; int error = 0; ASSERT(MUTEX_HELD(&zfsdev_state_lock)); /* * Reinitialize the dump area to the new size. If we * failed to resize the dump area then restore it back to * its original size. We must set the new volsize prior * to calling dumpvp_resize() to ensure that the devices' * size(9P) is not visible by the dump subsystem. */ old_volsize = zv->zv_volsize; zvol_size_changed(zv, volsize); #ifdef ZVOL_DUMP if (zv->zv_flags & ZVOL_DUMPIFIED) { if ((error = zvol_dumpify(zv)) != 0 || (error = dumpvp_resize()) != 0) { int dumpify_error; (void) zvol_update_volsize(zv->zv_objset, old_volsize); zvol_size_changed(zv, old_volsize); dumpify_error = zvol_dumpify(zv); error = dumpify_error ? dumpify_error : error; } } #endif /* ZVOL_DUMP */ #ifdef illumos /* * Generate a LUN expansion event. */ if (error == 0) { sysevent_id_t eid; nvlist_t *attr; char *physpath = kmem_zalloc(MAXPATHLEN, KM_SLEEP); (void) snprintf(physpath, MAXPATHLEN, "%s%u", ZVOL_PSEUDO_DEV, zv->zv_minor); VERIFY(nvlist_alloc(&attr, NV_UNIQUE_NAME, KM_SLEEP) == 0); VERIFY(nvlist_add_string(attr, DEV_PHYS_PATH, physpath) == 0); (void) ddi_log_sysevent(zfs_dip, SUNW_VENDOR, EC_DEV_STATUS, ESC_DEV_DLE, attr, &eid, DDI_SLEEP); nvlist_free(attr); kmem_free(physpath, MAXPATHLEN); } #endif /* illumos */ return (error); } int zvol_set_volsize(const char *name, uint64_t volsize) { zvol_state_t *zv = NULL; objset_t *os; int error; dmu_object_info_t doi; uint64_t readonly; boolean_t owned = B_FALSE; error = dsl_prop_get_integer(name, zfs_prop_to_name(ZFS_PROP_READONLY), &readonly, NULL); if (error != 0) return (error); if (readonly) return (SET_ERROR(EROFS)); mutex_enter(&zfsdev_state_lock); zv = zvol_minor_lookup(name); if (zv == NULL || zv->zv_objset == NULL) { if ((error = dmu_objset_own(name, DMU_OST_ZVOL, B_FALSE, FTAG, &os)) != 0) { mutex_exit(&zfsdev_state_lock); return (error); } owned = B_TRUE; if (zv != NULL) zv->zv_objset = os; } else { os = zv->zv_objset; } if ((error = dmu_object_info(os, ZVOL_OBJ, &doi)) != 0 || (error = zvol_check_volsize(volsize, doi.doi_data_block_size)) != 0) goto out; error = zvol_update_volsize(os, volsize); if (error == 0 && zv != NULL) error = zvol_update_live_volsize(zv, volsize); out: if (owned) { dmu_objset_disown(os, FTAG); if (zv != NULL) zv->zv_objset = NULL; } mutex_exit(&zfsdev_state_lock); return (error); } /*ARGSUSED*/ #ifdef illumos int zvol_open(dev_t *devp, int flag, int otyp, cred_t *cr) #else static int zvol_open(struct g_provider *pp, int flag, int count) #endif { zvol_state_t *zv; int err = 0; #ifdef illumos mutex_enter(&zfsdev_state_lock); zv = zfsdev_get_soft_state(getminor(*devp), ZSST_ZVOL); if (zv == NULL) { mutex_exit(&zfsdev_state_lock); return (SET_ERROR(ENXIO)); } if (zv->zv_total_opens == 0) err = zvol_first_open(zv); if (err) { mutex_exit(&zfsdev_state_lock); return (err); } #else /* !illumos */ - if (tsd_get(zfs_geom_probe_vdev_key) != NULL) { + boolean_t locked = B_FALSE; + + if (!zpool_on_zvol && tsd_get(zfs_geom_probe_vdev_key) != NULL) { /* * if zfs_geom_probe_vdev_key is set, that means that zfs is * attempting to probe geom providers while looking for a * replacement for a missing VDEV. In this case, the * spa_namespace_lock will not be held, but it is still illegal * to use a zvol as a vdev. Deadlocks can result if another * thread has spa_namespace_lock */ return (EOPNOTSUPP); } + /* + * Protect against recursively entering spa_namespace_lock + * when spa_open() is used for a pool on a (local) ZVOL(s). + * This is needed since we replaced upstream zfsdev_state_lock + * with spa_namespace_lock in the ZVOL code. + * We are using the same trick as spa_open(). + * Note that calls in zvol_first_open which need to resolve + * pool name to a spa object will enter spa_open() + * recursively, but that function already has all the + * necessary protection. + */ + if (!MUTEX_HELD(&zfsdev_state_lock)) { + mutex_enter(&zfsdev_state_lock); + locked = B_TRUE; + } - mutex_enter(&zfsdev_state_lock); - zv = pp->private; if (zv == NULL) { - mutex_exit(&zfsdev_state_lock); + if (locked) + mutex_exit(&zfsdev_state_lock); return (SET_ERROR(ENXIO)); } if (zv->zv_total_opens == 0) { err = zvol_first_open(zv); if (err) { - mutex_exit(&zfsdev_state_lock); + if (locked) + mutex_exit(&zfsdev_state_lock); return (err); } pp->mediasize = zv->zv_volsize; pp->stripeoffset = 0; pp->stripesize = zv->zv_volblocksize; } #endif /* illumos */ if ((flag & FWRITE) && (zv->zv_flags & ZVOL_RDONLY)) { err = SET_ERROR(EROFS); goto out; } if (zv->zv_flags & ZVOL_EXCL) { err = SET_ERROR(EBUSY); goto out; } #ifdef FEXCL if (flag & FEXCL) { if (zv->zv_total_opens != 0) { err = SET_ERROR(EBUSY); goto out; } zv->zv_flags |= ZVOL_EXCL; } #endif #ifdef illumos if (zv->zv_open_count[otyp] == 0 || otyp == OTYP_LYR) { zv->zv_open_count[otyp]++; zv->zv_total_opens++; } mutex_exit(&zfsdev_state_lock); #else zv->zv_total_opens += count; - mutex_exit(&zfsdev_state_lock); + if (locked) + mutex_exit(&zfsdev_state_lock); #endif return (err); out: if (zv->zv_total_opens == 0) zvol_last_close(zv); #ifdef illumos mutex_exit(&zfsdev_state_lock); #else - mutex_exit(&zfsdev_state_lock); + if (locked) + mutex_exit(&zfsdev_state_lock); #endif return (err); } /*ARGSUSED*/ #ifdef illumos int zvol_close(dev_t dev, int flag, int otyp, cred_t *cr) { minor_t minor = getminor(dev); zvol_state_t *zv; int error = 0; mutex_enter(&zfsdev_state_lock); zv = zfsdev_get_soft_state(minor, ZSST_ZVOL); if (zv == NULL) { mutex_exit(&zfsdev_state_lock); #else /* !illumos */ static int zvol_close(struct g_provider *pp, int flag, int count) { zvol_state_t *zv; int error = 0; boolean_t locked = B_FALSE; /* See comment in zvol_open(). */ if (!MUTEX_HELD(&zfsdev_state_lock)) { mutex_enter(&zfsdev_state_lock); locked = B_TRUE; } zv = pp->private; if (zv == NULL) { if (locked) mutex_exit(&zfsdev_state_lock); #endif /* illumos */ return (SET_ERROR(ENXIO)); } if (zv->zv_flags & ZVOL_EXCL) { ASSERT(zv->zv_total_opens == 1); zv->zv_flags &= ~ZVOL_EXCL; } /* * If the open count is zero, this is a spurious close. * That indicates a bug in the kernel / DDI framework. */ #ifdef illumos ASSERT(zv->zv_open_count[otyp] != 0); #endif ASSERT(zv->zv_total_opens != 0); /* * You may get multiple opens, but only one close. */ #ifdef illumos zv->zv_open_count[otyp]--; zv->zv_total_opens--; #else zv->zv_total_opens -= count; #endif if (zv->zv_total_opens == 0) zvol_last_close(zv); #ifdef illumos mutex_exit(&zfsdev_state_lock); #else if (locked) mutex_exit(&zfsdev_state_lock); #endif return (error); } static void zvol_get_done(zgd_t *zgd, int error) { if (zgd->zgd_db) dmu_buf_rele(zgd->zgd_db, zgd); zfs_range_unlock(zgd->zgd_rl); if (error == 0 && zgd->zgd_bp) zil_add_block(zgd->zgd_zilog, zgd->zgd_bp); kmem_free(zgd, sizeof (zgd_t)); } /* * Get data to generate a TX_WRITE intent log record. */ static int zvol_get_data(void *arg, lr_write_t *lr, char *buf, zio_t *zio) { zvol_state_t *zv = arg; objset_t *os = zv->zv_objset; uint64_t object = ZVOL_OBJ; uint64_t offset = lr->lr_offset; uint64_t size = lr->lr_length; /* length of user data */ blkptr_t *bp = &lr->lr_blkptr; dmu_buf_t *db; zgd_t *zgd; int error; ASSERT(zio != NULL); ASSERT(size != 0); zgd = kmem_zalloc(sizeof (zgd_t), KM_SLEEP); zgd->zgd_zilog = zv->zv_zilog; zgd->zgd_rl = zfs_range_lock(&zv->zv_znode, offset, size, RL_READER); /* * Write records come in two flavors: immediate and indirect. * For small writes it's cheaper to store the data with the * log record (immediate); for large writes it's cheaper to * sync the data and get a pointer to it (indirect) so that * we don't have to write the data twice. */ if (buf != NULL) { /* immediate write */ error = dmu_read(os, object, offset, size, buf, DMU_READ_NO_PREFETCH); } else { size = zv->zv_volblocksize; offset = P2ALIGN(offset, size); error = dmu_buf_hold(os, object, offset, zgd, &db, DMU_READ_NO_PREFETCH); if (error == 0) { blkptr_t *obp = dmu_buf_get_blkptr(db); if (obp) { ASSERT(BP_IS_HOLE(bp)); *bp = *obp; } zgd->zgd_db = db; zgd->zgd_bp = bp; ASSERT(db->db_offset == offset); ASSERT(db->db_size == size); error = dmu_sync(zio, lr->lr_common.lrc_txg, zvol_get_done, zgd); if (error == 0) return (0); } } zvol_get_done(zgd, error); return (error); } /* * zvol_log_write() handles synchronous writes using TX_WRITE ZIL transactions. * * We store data in the log buffers if it's small enough. * Otherwise we will later flush the data out via dmu_sync(). */ ssize_t zvol_immediate_write_sz = 32768; static void zvol_log_write(zvol_state_t *zv, dmu_tx_t *tx, offset_t off, ssize_t resid, boolean_t sync) { uint32_t blocksize = zv->zv_volblocksize; zilog_t *zilog = zv->zv_zilog; boolean_t slogging; ssize_t immediate_write_sz; if (zil_replaying(zilog, tx)) return; immediate_write_sz = (zilog->zl_logbias == ZFS_LOGBIAS_THROUGHPUT) ? 0 : zvol_immediate_write_sz; slogging = spa_has_slogs(zilog->zl_spa) && (zilog->zl_logbias == ZFS_LOGBIAS_LATENCY); while (resid) { itx_t *itx; lr_write_t *lr; ssize_t len; itx_wr_state_t write_state; /* * Unlike zfs_log_write() we can be called with * upto DMU_MAX_ACCESS/2 (5MB) writes. */ if (blocksize > immediate_write_sz && !slogging && resid >= blocksize && off % blocksize == 0) { write_state = WR_INDIRECT; /* uses dmu_sync */ len = blocksize; } else if (sync) { write_state = WR_COPIED; len = MIN(ZIL_MAX_LOG_DATA, resid); } else { write_state = WR_NEED_COPY; len = MIN(ZIL_MAX_LOG_DATA, resid); } itx = zil_itx_create(TX_WRITE, sizeof (*lr) + (write_state == WR_COPIED ? len : 0)); lr = (lr_write_t *)&itx->itx_lr; if (write_state == WR_COPIED && dmu_read(zv->zv_objset, ZVOL_OBJ, off, len, lr + 1, DMU_READ_NO_PREFETCH) != 0) { zil_itx_destroy(itx); itx = zil_itx_create(TX_WRITE, sizeof (*lr)); lr = (lr_write_t *)&itx->itx_lr; write_state = WR_NEED_COPY; } itx->itx_wr_state = write_state; if (write_state == WR_NEED_COPY) itx->itx_sod += len; lr->lr_foid = ZVOL_OBJ; lr->lr_offset = off; lr->lr_length = len; lr->lr_blkoff = 0; BP_ZERO(&lr->lr_blkptr); itx->itx_private = zv; itx->itx_sync = sync; zil_itx_assign(zilog, itx, tx); off += len; resid -= len; } } #ifdef illumos static int zvol_dumpio_vdev(vdev_t *vd, void *addr, uint64_t offset, uint64_t origoffset, uint64_t size, boolean_t doread, boolean_t isdump) { vdev_disk_t *dvd; int c; int numerrors = 0; if (vd->vdev_ops == &vdev_mirror_ops || vd->vdev_ops == &vdev_replacing_ops || vd->vdev_ops == &vdev_spare_ops) { for (c = 0; c < vd->vdev_children; c++) { int err = zvol_dumpio_vdev(vd->vdev_child[c], addr, offset, origoffset, size, doread, isdump); if (err != 0) { numerrors++; } else if (doread) { break; } } } if (!vd->vdev_ops->vdev_op_leaf && vd->vdev_ops != &vdev_raidz_ops) return (numerrors < vd->vdev_children ? 0 : EIO); if (doread && !vdev_readable(vd)) return (SET_ERROR(EIO)); else if (!doread && !vdev_writeable(vd)) return (SET_ERROR(EIO)); if (vd->vdev_ops == &vdev_raidz_ops) { return (vdev_raidz_physio(vd, addr, size, offset, origoffset, doread, isdump)); } offset += VDEV_LABEL_START_SIZE; if (ddi_in_panic() || isdump) { ASSERT(!doread); if (doread) return (SET_ERROR(EIO)); dvd = vd->vdev_tsd; ASSERT3P(dvd, !=, NULL); return (ldi_dump(dvd->vd_lh, addr, lbtodb(offset), lbtodb(size))); } else { dvd = vd->vdev_tsd; ASSERT3P(dvd, !=, NULL); return (vdev_disk_ldi_physio(dvd->vd_lh, addr, size, offset, doread ? B_READ : B_WRITE)); } } static int zvol_dumpio(zvol_state_t *zv, void *addr, uint64_t offset, uint64_t size, boolean_t doread, boolean_t isdump) { vdev_t *vd; int error; zvol_extent_t *ze; spa_t *spa = dmu_objset_spa(zv->zv_objset); /* Must be sector aligned, and not stradle a block boundary. */ if (P2PHASE(offset, DEV_BSIZE) || P2PHASE(size, DEV_BSIZE) || P2BOUNDARY(offset, size, zv->zv_volblocksize)) { return (SET_ERROR(EINVAL)); } ASSERT(size <= zv->zv_volblocksize); /* Locate the extent this belongs to */ ze = list_head(&zv->zv_extents); while (offset >= ze->ze_nblks * zv->zv_volblocksize) { offset -= ze->ze_nblks * zv->zv_volblocksize; ze = list_next(&zv->zv_extents, ze); } if (ze == NULL) return (SET_ERROR(EINVAL)); if (!ddi_in_panic()) spa_config_enter(spa, SCL_STATE, FTAG, RW_READER); vd = vdev_lookup_top(spa, DVA_GET_VDEV(&ze->ze_dva)); offset += DVA_GET_OFFSET(&ze->ze_dva); error = zvol_dumpio_vdev(vd, addr, offset, DVA_GET_OFFSET(&ze->ze_dva), size, doread, isdump); if (!ddi_in_panic()) spa_config_exit(spa, SCL_STATE, FTAG); return (error); } int zvol_strategy(buf_t *bp) { zfs_soft_state_t *zs = NULL; #else /* !illumos */ void zvol_strategy(struct bio *bp) { #endif /* illumos */ zvol_state_t *zv; uint64_t off, volsize; size_t resid; char *addr; objset_t *os; rl_t *rl; int error = 0; #ifdef illumos boolean_t doread = bp->b_flags & B_READ; #else boolean_t doread = 0; #endif boolean_t is_dumpified; boolean_t sync; #ifdef illumos if (getminor(bp->b_edev) == 0) { error = SET_ERROR(EINVAL); } else { zs = ddi_get_soft_state(zfsdev_state, getminor(bp->b_edev)); if (zs == NULL) error = SET_ERROR(ENXIO); else if (zs->zss_type != ZSST_ZVOL) error = SET_ERROR(EINVAL); } if (error) { bioerror(bp, error); biodone(bp); return (0); } zv = zs->zss_data; if (!(bp->b_flags & B_READ) && (zv->zv_flags & ZVOL_RDONLY)) { bioerror(bp, EROFS); biodone(bp); return (0); } off = ldbtob(bp->b_blkno); #else /* !illumos */ if (bp->bio_to) zv = bp->bio_to->private; else zv = bp->bio_dev->si_drv2; if (zv == NULL) { error = SET_ERROR(ENXIO); goto out; } if (bp->bio_cmd != BIO_READ && (zv->zv_flags & ZVOL_RDONLY)) { error = SET_ERROR(EROFS); goto out; } switch (bp->bio_cmd) { case BIO_FLUSH: goto sync; case BIO_READ: doread = 1; case BIO_WRITE: case BIO_DELETE: break; default: error = EOPNOTSUPP; goto out; } off = bp->bio_offset; #endif /* illumos */ volsize = zv->zv_volsize; os = zv->zv_objset; ASSERT(os != NULL); #ifdef illumos bp_mapin(bp); addr = bp->b_un.b_addr; resid = bp->b_bcount; if (resid > 0 && (off < 0 || off >= volsize)) { bioerror(bp, EIO); biodone(bp); return (0); } is_dumpified = zv->zv_flags & ZVOL_DUMPIFIED; sync = ((!(bp->b_flags & B_ASYNC) && !(zv->zv_flags & ZVOL_WCE)) || (zv->zv_objset->os_sync == ZFS_SYNC_ALWAYS)) && !doread && !is_dumpified; #else /* !illumos */ addr = bp->bio_data; resid = bp->bio_length; if (resid > 0 && (off < 0 || off >= volsize)) { error = SET_ERROR(EIO); goto out; } is_dumpified = B_FALSE; sync = !doread && !is_dumpified && zv->zv_objset->os_sync == ZFS_SYNC_ALWAYS; #endif /* illumos */ /* * There must be no buffer changes when doing a dmu_sync() because * we can't change the data whilst calculating the checksum. */ rl = zfs_range_lock(&zv->zv_znode, off, resid, doread ? RL_READER : RL_WRITER); #ifndef illumos if (bp->bio_cmd == BIO_DELETE) { dmu_tx_t *tx = dmu_tx_create(zv->zv_objset); error = dmu_tx_assign(tx, TXG_WAIT); if (error != 0) { dmu_tx_abort(tx); } else { zvol_log_truncate(zv, tx, off, resid, B_TRUE); dmu_tx_commit(tx); error = dmu_free_long_range(zv->zv_objset, ZVOL_OBJ, off, resid); resid = 0; } goto unlock; } #endif while (resid != 0 && off < volsize) { size_t size = MIN(resid, zvol_maxphys); #ifdef illumos if (is_dumpified) { size = MIN(size, P2END(off, zv->zv_volblocksize) - off); error = zvol_dumpio(zv, addr, off, size, doread, B_FALSE); } else if (doread) { #else if (doread) { #endif error = dmu_read(os, ZVOL_OBJ, off, size, addr, DMU_READ_PREFETCH); } else { dmu_tx_t *tx = dmu_tx_create(os); dmu_tx_hold_write(tx, ZVOL_OBJ, off, size); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); } else { dmu_write(os, ZVOL_OBJ, off, size, addr, tx); zvol_log_write(zv, tx, off, size, sync); dmu_tx_commit(tx); } } if (error) { /* convert checksum errors into IO errors */ if (error == ECKSUM) error = SET_ERROR(EIO); break; } off += size; addr += size; resid -= size; } #ifndef illumos unlock: #endif zfs_range_unlock(rl); #ifdef illumos if ((bp->b_resid = resid) == bp->b_bcount) bioerror(bp, off > volsize ? EINVAL : error); if (sync) zil_commit(zv->zv_zilog, ZVOL_OBJ); biodone(bp); return (0); #else /* !illumos */ bp->bio_completed = bp->bio_length - resid; if (bp->bio_completed < bp->bio_length && off > volsize) error = EINVAL; if (sync) { sync: zil_commit(zv->zv_zilog, ZVOL_OBJ); } out: if (bp->bio_to) g_io_deliver(bp, error); else biofinish(bp, NULL, error); #endif /* illumos */ } #ifdef illumos /* * Set the buffer count to the zvol maximum transfer. * Using our own routine instead of the default minphys() * means that for larger writes we write bigger buffers on X86 * (128K instead of 56K) and flush the disk write cache less often * (every zvol_maxphys - currently 1MB) instead of minphys (currently * 56K on X86 and 128K on sparc). */ void zvol_minphys(struct buf *bp) { if (bp->b_bcount > zvol_maxphys) bp->b_bcount = zvol_maxphys; } int zvol_dump(dev_t dev, caddr_t addr, daddr_t blkno, int nblocks) { minor_t minor = getminor(dev); zvol_state_t *zv; int error = 0; uint64_t size; uint64_t boff; uint64_t resid; zv = zfsdev_get_soft_state(minor, ZSST_ZVOL); if (zv == NULL) return (SET_ERROR(ENXIO)); if ((zv->zv_flags & ZVOL_DUMPIFIED) == 0) return (SET_ERROR(EINVAL)); boff = ldbtob(blkno); resid = ldbtob(nblocks); VERIFY3U(boff + resid, <=, zv->zv_volsize); while (resid) { size = MIN(resid, P2END(boff, zv->zv_volblocksize) - boff); error = zvol_dumpio(zv, addr, boff, size, B_FALSE, B_TRUE); if (error) break; boff += size; addr += size; resid -= size; } return (error); } /*ARGSUSED*/ int zvol_read(dev_t dev, uio_t *uio, cred_t *cr) { minor_t minor = getminor(dev); #else /* !illumos */ int zvol_read(struct cdev *dev, struct uio *uio, int ioflag) { #endif /* illumos */ zvol_state_t *zv; uint64_t volsize; rl_t *rl; int error = 0; #ifdef illumos zv = zfsdev_get_soft_state(minor, ZSST_ZVOL); if (zv == NULL) return (SET_ERROR(ENXIO)); #else zv = dev->si_drv2; #endif volsize = zv->zv_volsize; /* uio_loffset == volsize isn't an error as its required for EOF processing. */ if (uio->uio_resid > 0 && (uio->uio_loffset < 0 || uio->uio_loffset > volsize)) return (SET_ERROR(EIO)); #ifdef illumos if (zv->zv_flags & ZVOL_DUMPIFIED) { error = physio(zvol_strategy, NULL, dev, B_READ, zvol_minphys, uio); return (error); } #endif rl = zfs_range_lock(&zv->zv_znode, uio->uio_loffset, uio->uio_resid, RL_READER); while (uio->uio_resid > 0 && uio->uio_loffset < volsize) { uint64_t bytes = MIN(uio->uio_resid, DMU_MAX_ACCESS >> 1); /* don't read past the end */ if (bytes > volsize - uio->uio_loffset) bytes = volsize - uio->uio_loffset; error = dmu_read_uio_dbuf(zv->zv_dbuf, uio, bytes); if (error) { /* convert checksum errors into IO errors */ if (error == ECKSUM) error = SET_ERROR(EIO); break; } } zfs_range_unlock(rl); return (error); } #ifdef illumos /*ARGSUSED*/ int zvol_write(dev_t dev, uio_t *uio, cred_t *cr) { minor_t minor = getminor(dev); #else /* !illumos */ int zvol_write(struct cdev *dev, struct uio *uio, int ioflag) { #endif /* illumos */ zvol_state_t *zv; uint64_t volsize; rl_t *rl; int error = 0; boolean_t sync; #ifdef illumos zv = zfsdev_get_soft_state(minor, ZSST_ZVOL); if (zv == NULL) return (SET_ERROR(ENXIO)); #else zv = dev->si_drv2; #endif volsize = zv->zv_volsize; /* uio_loffset == volsize isn't an error as its required for EOF processing. */ if (uio->uio_resid > 0 && (uio->uio_loffset < 0 || uio->uio_loffset > volsize)) return (SET_ERROR(EIO)); #ifdef illumos if (zv->zv_flags & ZVOL_DUMPIFIED) { error = physio(zvol_strategy, NULL, dev, B_WRITE, zvol_minphys, uio); return (error); } sync = !(zv->zv_flags & ZVOL_WCE) || #else sync = (ioflag & IO_SYNC) || #endif (zv->zv_objset->os_sync == ZFS_SYNC_ALWAYS); rl = zfs_range_lock(&zv->zv_znode, uio->uio_loffset, uio->uio_resid, RL_WRITER); while (uio->uio_resid > 0 && uio->uio_loffset < volsize) { uint64_t bytes = MIN(uio->uio_resid, DMU_MAX_ACCESS >> 1); uint64_t off = uio->uio_loffset; dmu_tx_t *tx = dmu_tx_create(zv->zv_objset); if (bytes > volsize - off) /* don't write past the end */ bytes = volsize - off; dmu_tx_hold_write(tx, ZVOL_OBJ, off, bytes); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); break; } error = dmu_write_uio_dbuf(zv->zv_dbuf, uio, bytes, tx); if (error == 0) zvol_log_write(zv, tx, off, bytes, sync); dmu_tx_commit(tx); if (error) break; } zfs_range_unlock(rl); if (sync) zil_commit(zv->zv_zilog, ZVOL_OBJ); return (error); } #ifdef illumos int zvol_getefi(void *arg, int flag, uint64_t vs, uint8_t bs) { struct uuid uuid = EFI_RESERVED; efi_gpe_t gpe = { 0 }; uint32_t crc; dk_efi_t efi; int length; char *ptr; if (ddi_copyin(arg, &efi, sizeof (dk_efi_t), flag)) return (SET_ERROR(EFAULT)); ptr = (char *)(uintptr_t)efi.dki_data_64; length = efi.dki_length; /* * Some clients may attempt to request a PMBR for the * zvol. Currently this interface will return EINVAL to * such requests. These requests could be supported by * adding a check for lba == 0 and consing up an appropriate * PMBR. */ if (efi.dki_lba < 1 || efi.dki_lba > 2 || length <= 0) return (SET_ERROR(EINVAL)); gpe.efi_gpe_StartingLBA = LE_64(34ULL); gpe.efi_gpe_EndingLBA = LE_64((vs >> bs) - 1); UUID_LE_CONVERT(gpe.efi_gpe_PartitionTypeGUID, uuid); if (efi.dki_lba == 1) { efi_gpt_t gpt = { 0 }; gpt.efi_gpt_Signature = LE_64(EFI_SIGNATURE); gpt.efi_gpt_Revision = LE_32(EFI_VERSION_CURRENT); gpt.efi_gpt_HeaderSize = LE_32(sizeof (gpt)); gpt.efi_gpt_MyLBA = LE_64(1ULL); gpt.efi_gpt_FirstUsableLBA = LE_64(34ULL); gpt.efi_gpt_LastUsableLBA = LE_64((vs >> bs) - 1); gpt.efi_gpt_PartitionEntryLBA = LE_64(2ULL); gpt.efi_gpt_NumberOfPartitionEntries = LE_32(1); gpt.efi_gpt_SizeOfPartitionEntry = LE_32(sizeof (efi_gpe_t)); CRC32(crc, &gpe, sizeof (gpe), -1U, crc32_table); gpt.efi_gpt_PartitionEntryArrayCRC32 = LE_32(~crc); CRC32(crc, &gpt, sizeof (gpt), -1U, crc32_table); gpt.efi_gpt_HeaderCRC32 = LE_32(~crc); if (ddi_copyout(&gpt, ptr, MIN(sizeof (gpt), length), flag)) return (SET_ERROR(EFAULT)); ptr += sizeof (gpt); length -= sizeof (gpt); } if (length > 0 && ddi_copyout(&gpe, ptr, MIN(sizeof (gpe), length), flag)) return (SET_ERROR(EFAULT)); return (0); } /* * BEGIN entry points to allow external callers access to the volume. */ /* * Return the volume parameters needed for access from an external caller. * These values are invariant as long as the volume is held open. */ int zvol_get_volume_params(minor_t minor, uint64_t *blksize, uint64_t *max_xfer_len, void **minor_hdl, void **objset_hdl, void **zil_hdl, void **rl_hdl, void **bonus_hdl) { zvol_state_t *zv; zv = zfsdev_get_soft_state(minor, ZSST_ZVOL); if (zv == NULL) return (SET_ERROR(ENXIO)); if (zv->zv_flags & ZVOL_DUMPIFIED) return (SET_ERROR(ENXIO)); ASSERT(blksize && max_xfer_len && minor_hdl && objset_hdl && zil_hdl && rl_hdl && bonus_hdl); *blksize = zv->zv_volblocksize; *max_xfer_len = (uint64_t)zvol_maxphys; *minor_hdl = zv; *objset_hdl = zv->zv_objset; *zil_hdl = zv->zv_zilog; *rl_hdl = &zv->zv_znode; *bonus_hdl = zv->zv_dbuf; return (0); } /* * Return the current volume size to an external caller. * The size can change while the volume is open. */ uint64_t zvol_get_volume_size(void *minor_hdl) { zvol_state_t *zv = minor_hdl; return (zv->zv_volsize); } /* * Return the current WCE setting to an external caller. * The WCE setting can change while the volume is open. */ int zvol_get_volume_wce(void *minor_hdl) { zvol_state_t *zv = minor_hdl; return ((zv->zv_flags & ZVOL_WCE) ? 1 : 0); } /* * Entry point for external callers to zvol_log_write */ void zvol_log_write_minor(void *minor_hdl, dmu_tx_t *tx, offset_t off, ssize_t resid, boolean_t sync) { zvol_state_t *zv = minor_hdl; zvol_log_write(zv, tx, off, resid, sync); } /* * END entry points to allow external callers access to the volume. */ #endif /* illumos */ /* * Log a DKIOCFREE/free-long-range to the ZIL with TX_TRUNCATE. */ static void zvol_log_truncate(zvol_state_t *zv, dmu_tx_t *tx, uint64_t off, uint64_t len, boolean_t sync) { itx_t *itx; lr_truncate_t *lr; zilog_t *zilog = zv->zv_zilog; if (zil_replaying(zilog, tx)) return; itx = zil_itx_create(TX_TRUNCATE, sizeof (*lr)); lr = (lr_truncate_t *)&itx->itx_lr; lr->lr_foid = ZVOL_OBJ; lr->lr_offset = off; lr->lr_length = len; itx->itx_sync = sync; zil_itx_assign(zilog, itx, tx); } #ifdef illumos /* * Dirtbag ioctls to support mkfs(1M) for UFS filesystems. See dkio(7I). * Also a dirtbag dkio ioctl for unmap/free-block functionality. */ /*ARGSUSED*/ int zvol_ioctl(dev_t dev, int cmd, intptr_t arg, int flag, cred_t *cr, int *rvalp) { zvol_state_t *zv; struct dk_callback *dkc; int error = 0; rl_t *rl; mutex_enter(&zfsdev_state_lock); zv = zfsdev_get_soft_state(getminor(dev), ZSST_ZVOL); if (zv == NULL) { mutex_exit(&zfsdev_state_lock); return (SET_ERROR(ENXIO)); } ASSERT(zv->zv_total_opens > 0); switch (cmd) { case DKIOCINFO: { struct dk_cinfo dki; bzero(&dki, sizeof (dki)); (void) strcpy(dki.dki_cname, "zvol"); (void) strcpy(dki.dki_dname, "zvol"); dki.dki_ctype = DKC_UNKNOWN; dki.dki_unit = getminor(dev); dki.dki_maxtransfer = 1 << (SPA_OLD_MAXBLOCKSHIFT - zv->zv_min_bs); mutex_exit(&zfsdev_state_lock); if (ddi_copyout(&dki, (void *)arg, sizeof (dki), flag)) error = SET_ERROR(EFAULT); return (error); } case DKIOCGMEDIAINFO: { struct dk_minfo dkm; bzero(&dkm, sizeof (dkm)); dkm.dki_lbsize = 1U << zv->zv_min_bs; dkm.dki_capacity = zv->zv_volsize >> zv->zv_min_bs; dkm.dki_media_type = DK_UNKNOWN; mutex_exit(&zfsdev_state_lock); if (ddi_copyout(&dkm, (void *)arg, sizeof (dkm), flag)) error = SET_ERROR(EFAULT); return (error); } case DKIOCGMEDIAINFOEXT: { struct dk_minfo_ext dkmext; bzero(&dkmext, sizeof (dkmext)); dkmext.dki_lbsize = 1U << zv->zv_min_bs; dkmext.dki_pbsize = zv->zv_volblocksize; dkmext.dki_capacity = zv->zv_volsize >> zv->zv_min_bs; dkmext.dki_media_type = DK_UNKNOWN; mutex_exit(&zfsdev_state_lock); if (ddi_copyout(&dkmext, (void *)arg, sizeof (dkmext), flag)) error = SET_ERROR(EFAULT); return (error); } case DKIOCGETEFI: { uint64_t vs = zv->zv_volsize; uint8_t bs = zv->zv_min_bs; mutex_exit(&zfsdev_state_lock); error = zvol_getefi((void *)arg, flag, vs, bs); return (error); } case DKIOCFLUSHWRITECACHE: dkc = (struct dk_callback *)arg; mutex_exit(&zfsdev_state_lock); zil_commit(zv->zv_zilog, ZVOL_OBJ); if ((flag & FKIOCTL) && dkc != NULL && dkc->dkc_callback) { (*dkc->dkc_callback)(dkc->dkc_cookie, error); error = 0; } return (error); case DKIOCGETWCE: { int wce = (zv->zv_flags & ZVOL_WCE) ? 1 : 0; if (ddi_copyout(&wce, (void *)arg, sizeof (int), flag)) error = SET_ERROR(EFAULT); break; } case DKIOCSETWCE: { int wce; if (ddi_copyin((void *)arg, &wce, sizeof (int), flag)) { error = SET_ERROR(EFAULT); break; } if (wce) { zv->zv_flags |= ZVOL_WCE; mutex_exit(&zfsdev_state_lock); } else { zv->zv_flags &= ~ZVOL_WCE; mutex_exit(&zfsdev_state_lock); zil_commit(zv->zv_zilog, ZVOL_OBJ); } return (0); } case DKIOCGGEOM: case DKIOCGVTOC: /* * commands using these (like prtvtoc) expect ENOTSUP * since we're emulating an EFI label */ error = SET_ERROR(ENOTSUP); break; case DKIOCDUMPINIT: rl = zfs_range_lock(&zv->zv_znode, 0, zv->zv_volsize, RL_WRITER); error = zvol_dumpify(zv); zfs_range_unlock(rl); break; case DKIOCDUMPFINI: if (!(zv->zv_flags & ZVOL_DUMPIFIED)) break; rl = zfs_range_lock(&zv->zv_znode, 0, zv->zv_volsize, RL_WRITER); error = zvol_dump_fini(zv); zfs_range_unlock(rl); break; case DKIOCFREE: { dkioc_free_t df; dmu_tx_t *tx; if (!zvol_unmap_enabled) break; if (ddi_copyin((void *)arg, &df, sizeof (df), flag)) { error = SET_ERROR(EFAULT); break; } /* * Apply Postel's Law to length-checking. If they overshoot, * just blank out until the end, if there's a need to blank * out anything. */ if (df.df_start >= zv->zv_volsize) break; /* No need to do anything... */ mutex_exit(&zfsdev_state_lock); rl = zfs_range_lock(&zv->zv_znode, df.df_start, df.df_length, RL_WRITER); tx = dmu_tx_create(zv->zv_objset); dmu_tx_mark_netfree(tx); error = dmu_tx_assign(tx, TXG_WAIT); if (error != 0) { dmu_tx_abort(tx); } else { zvol_log_truncate(zv, tx, df.df_start, df.df_length, B_TRUE); dmu_tx_commit(tx); error = dmu_free_long_range(zv->zv_objset, ZVOL_OBJ, df.df_start, df.df_length); } zfs_range_unlock(rl); if (error == 0) { /* * If the write-cache is disabled or 'sync' property * is set to 'always' then treat this as a synchronous * operation (i.e. commit to zil). */ if (!(zv->zv_flags & ZVOL_WCE) || (zv->zv_objset->os_sync == ZFS_SYNC_ALWAYS)) zil_commit(zv->zv_zilog, ZVOL_OBJ); /* * If the caller really wants synchronous writes, and * can't wait for them, don't return until the write * is done. */ if (df.df_flags & DF_WAIT_SYNC) { txg_wait_synced( dmu_objset_pool(zv->zv_objset), 0); } } return (error); } default: error = SET_ERROR(ENOTTY); break; } mutex_exit(&zfsdev_state_lock); return (error); } #endif /* illumos */ int zvol_busy(void) { return (zvol_minors != 0); } void zvol_init(void) { VERIFY(ddi_soft_state_init(&zfsdev_state, sizeof (zfs_soft_state_t), 1) == 0); #ifdef illumos mutex_init(&zfsdev_state_lock, NULL, MUTEX_DEFAULT, NULL); #else ZFS_LOG(1, "ZVOL Initialized."); #endif } void zvol_fini(void) { #ifdef illumos mutex_destroy(&zfsdev_state_lock); #endif ddi_soft_state_fini(&zfsdev_state); ZFS_LOG(1, "ZVOL Deinitialized."); } #ifdef illumos /*ARGSUSED*/ static int zfs_mvdev_dump_feature_check(void *arg, dmu_tx_t *tx) { spa_t *spa = dmu_tx_pool(tx)->dp_spa; if (spa_feature_is_active(spa, SPA_FEATURE_MULTI_VDEV_CRASH_DUMP)) return (1); return (0); } /*ARGSUSED*/ static void zfs_mvdev_dump_activate_feature_sync(void *arg, dmu_tx_t *tx) { spa_t *spa = dmu_tx_pool(tx)->dp_spa; spa_feature_incr(spa, SPA_FEATURE_MULTI_VDEV_CRASH_DUMP, tx); } static int zvol_dump_init(zvol_state_t *zv, boolean_t resize) { dmu_tx_t *tx; int error; objset_t *os = zv->zv_objset; spa_t *spa = dmu_objset_spa(os); vdev_t *vd = spa->spa_root_vdev; nvlist_t *nv = NULL; uint64_t version = spa_version(spa); uint64_t checksum, compress, refresrv, vbs, dedup; ASSERT(MUTEX_HELD(&zfsdev_state_lock)); ASSERT(vd->vdev_ops == &vdev_root_ops); error = dmu_free_long_range(zv->zv_objset, ZVOL_OBJ, 0, DMU_OBJECT_END); if (error != 0) return (error); /* wait for dmu_free_long_range to actually free the blocks */ txg_wait_synced(dmu_objset_pool(zv->zv_objset), 0); /* * If the pool on which the dump device is being initialized has more * than one child vdev, check that the MULTI_VDEV_CRASH_DUMP feature is * enabled. If so, bump that feature's counter to indicate that the * feature is active. We also check the vdev type to handle the * following case: * # zpool create test raidz disk1 disk2 disk3 * Now have spa_root_vdev->vdev_children == 1 (the raidz vdev), * the raidz vdev itself has 3 children. */ if (vd->vdev_children > 1 || vd->vdev_ops == &vdev_raidz_ops) { if (!spa_feature_is_enabled(spa, SPA_FEATURE_MULTI_VDEV_CRASH_DUMP)) return (SET_ERROR(ENOTSUP)); (void) dsl_sync_task(spa_name(spa), zfs_mvdev_dump_feature_check, zfs_mvdev_dump_activate_feature_sync, NULL, 2, ZFS_SPACE_CHECK_RESERVED); } if (!resize) { error = dsl_prop_get_integer(zv->zv_name, zfs_prop_to_name(ZFS_PROP_COMPRESSION), &compress, NULL); if (error == 0) { error = dsl_prop_get_integer(zv->zv_name, zfs_prop_to_name(ZFS_PROP_CHECKSUM), &checksum, NULL); } if (error == 0) { error = dsl_prop_get_integer(zv->zv_name, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), &refresrv, NULL); } if (error == 0) { error = dsl_prop_get_integer(zv->zv_name, zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE), &vbs, NULL); } if (version >= SPA_VERSION_DEDUP && error == 0) { error = dsl_prop_get_integer(zv->zv_name, zfs_prop_to_name(ZFS_PROP_DEDUP), &dedup, NULL); } } if (error != 0) return (error); tx = dmu_tx_create(os); dmu_tx_hold_zap(tx, ZVOL_ZAP_OBJ, TRUE, NULL); dmu_tx_hold_bonus(tx, ZVOL_OBJ); error = dmu_tx_assign(tx, TXG_WAIT); if (error != 0) { dmu_tx_abort(tx); return (error); } /* * If we are resizing the dump device then we only need to * update the refreservation to match the newly updated * zvolsize. Otherwise, we save off the original state of the * zvol so that we can restore them if the zvol is ever undumpified. */ if (resize) { error = zap_update(os, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), 8, 1, &zv->zv_volsize, tx); } else { error = zap_update(os, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_COMPRESSION), 8, 1, &compress, tx); if (error == 0) { error = zap_update(os, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_CHECKSUM), 8, 1, &checksum, tx); } if (error == 0) { error = zap_update(os, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), 8, 1, &refresrv, tx); } if (error == 0) { error = zap_update(os, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE), 8, 1, &vbs, tx); } if (error == 0) { error = dmu_object_set_blocksize( os, ZVOL_OBJ, SPA_OLD_MAXBLOCKSIZE, 0, tx); } if (version >= SPA_VERSION_DEDUP && error == 0) { error = zap_update(os, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_DEDUP), 8, 1, &dedup, tx); } if (error == 0) zv->zv_volblocksize = SPA_OLD_MAXBLOCKSIZE; } dmu_tx_commit(tx); /* * We only need update the zvol's property if we are initializing * the dump area for the first time. */ if (error == 0 && !resize) { /* * If MULTI_VDEV_CRASH_DUMP is active, use the NOPARITY checksum * function. Otherwise, use the old default -- OFF. */ checksum = spa_feature_is_active(spa, SPA_FEATURE_MULTI_VDEV_CRASH_DUMP) ? ZIO_CHECKSUM_NOPARITY : ZIO_CHECKSUM_OFF; VERIFY(nvlist_alloc(&nv, NV_UNIQUE_NAME, KM_SLEEP) == 0); VERIFY(nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), 0) == 0); VERIFY(nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_COMPRESSION), ZIO_COMPRESS_OFF) == 0); VERIFY(nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_CHECKSUM), checksum) == 0); if (version >= SPA_VERSION_DEDUP) { VERIFY(nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_DEDUP), ZIO_CHECKSUM_OFF) == 0); } error = zfs_set_prop_nvlist(zv->zv_name, ZPROP_SRC_LOCAL, nv, NULL); nvlist_free(nv); } /* Allocate the space for the dump */ if (error == 0) error = zvol_prealloc(zv); return (error); } static int zvol_dumpify(zvol_state_t *zv) { int error = 0; uint64_t dumpsize = 0; dmu_tx_t *tx; objset_t *os = zv->zv_objset; if (zv->zv_flags & ZVOL_RDONLY) return (SET_ERROR(EROFS)); if (zap_lookup(zv->zv_objset, ZVOL_ZAP_OBJ, ZVOL_DUMPSIZE, 8, 1, &dumpsize) != 0 || dumpsize != zv->zv_volsize) { boolean_t resize = (dumpsize > 0); if ((error = zvol_dump_init(zv, resize)) != 0) { (void) zvol_dump_fini(zv); return (error); } } /* * Build up our lba mapping. */ error = zvol_get_lbas(zv); if (error) { (void) zvol_dump_fini(zv); return (error); } tx = dmu_tx_create(os); dmu_tx_hold_zap(tx, ZVOL_ZAP_OBJ, TRUE, NULL); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); (void) zvol_dump_fini(zv); return (error); } zv->zv_flags |= ZVOL_DUMPIFIED; error = zap_update(os, ZVOL_ZAP_OBJ, ZVOL_DUMPSIZE, 8, 1, &zv->zv_volsize, tx); dmu_tx_commit(tx); if (error) { (void) zvol_dump_fini(zv); return (error); } txg_wait_synced(dmu_objset_pool(os), 0); return (0); } static int zvol_dump_fini(zvol_state_t *zv) { dmu_tx_t *tx; objset_t *os = zv->zv_objset; nvlist_t *nv; int error = 0; uint64_t checksum, compress, refresrv, vbs, dedup; uint64_t version = spa_version(dmu_objset_spa(zv->zv_objset)); /* * Attempt to restore the zvol back to its pre-dumpified state. * This is a best-effort attempt as it's possible that not all * of these properties were initialized during the dumpify process * (i.e. error during zvol_dump_init). */ tx = dmu_tx_create(os); dmu_tx_hold_zap(tx, ZVOL_ZAP_OBJ, TRUE, NULL); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); return (error); } (void) zap_remove(os, ZVOL_ZAP_OBJ, ZVOL_DUMPSIZE, tx); dmu_tx_commit(tx); (void) zap_lookup(zv->zv_objset, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_CHECKSUM), 8, 1, &checksum); (void) zap_lookup(zv->zv_objset, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_COMPRESSION), 8, 1, &compress); (void) zap_lookup(zv->zv_objset, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), 8, 1, &refresrv); (void) zap_lookup(zv->zv_objset, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_VOLBLOCKSIZE), 8, 1, &vbs); VERIFY(nvlist_alloc(&nv, NV_UNIQUE_NAME, KM_SLEEP) == 0); (void) nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_CHECKSUM), checksum); (void) nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_COMPRESSION), compress); (void) nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_REFRESERVATION), refresrv); if (version >= SPA_VERSION_DEDUP && zap_lookup(zv->zv_objset, ZVOL_ZAP_OBJ, zfs_prop_to_name(ZFS_PROP_DEDUP), 8, 1, &dedup) == 0) { (void) nvlist_add_uint64(nv, zfs_prop_to_name(ZFS_PROP_DEDUP), dedup); } (void) zfs_set_prop_nvlist(zv->zv_name, ZPROP_SRC_LOCAL, nv, NULL); nvlist_free(nv); zvol_free_extents(zv); zv->zv_flags &= ~ZVOL_DUMPIFIED; (void) dmu_free_long_range(os, ZVOL_OBJ, 0, DMU_OBJECT_END); /* wait for dmu_free_long_range to actually free the blocks */ txg_wait_synced(dmu_objset_pool(zv->zv_objset), 0); tx = dmu_tx_create(os); dmu_tx_hold_bonus(tx, ZVOL_OBJ); error = dmu_tx_assign(tx, TXG_WAIT); if (error) { dmu_tx_abort(tx); return (error); } if (dmu_object_set_blocksize(os, ZVOL_OBJ, vbs, 0, tx) == 0) zv->zv_volblocksize = vbs; dmu_tx_commit(tx); return (0); } #else /* !illumos */ static void zvol_geom_run(zvol_state_t *zv) { struct g_provider *pp; pp = zv->zv_provider; g_error_provider(pp, 0); kproc_kthread_add(zvol_geom_worker, zv, &zfsproc, NULL, 0, 0, "zfskern", "zvol %s", pp->name + sizeof(ZVOL_DRIVER)); } static void zvol_geom_destroy(zvol_state_t *zv) { struct g_provider *pp; g_topology_assert(); mtx_lock(&zv->zv_queue_mtx); zv->zv_state = 1; wakeup_one(&zv->zv_queue); while (zv->zv_state != 2) msleep(&zv->zv_state, &zv->zv_queue_mtx, 0, "zvol:w", 0); mtx_destroy(&zv->zv_queue_mtx); pp = zv->zv_provider; zv->zv_provider = NULL; pp->private = NULL; g_wither_geom(pp->geom, ENXIO); } static int zvol_geom_access(struct g_provider *pp, int acr, int acw, int ace) { int count, error, flags; g_topology_assert(); /* * To make it easier we expect either open or close, but not both * at the same time. */ KASSERT((acr >= 0 && acw >= 0 && ace >= 0) || (acr <= 0 && acw <= 0 && ace <= 0), ("Unsupported access request to %s (acr=%d, acw=%d, ace=%d).", pp->name, acr, acw, ace)); if (pp->private == NULL) { if (acr <= 0 && acw <= 0 && ace <= 0) return (0); return (pp->error); } /* * We don't pass FEXCL flag to zvol_open()/zvol_close() if ace != 0, * because GEOM already handles that and handles it a bit differently. * GEOM allows for multiple read/exclusive consumers and ZFS allows * only one exclusive consumer, no matter if it is reader or writer. * I like better the way GEOM works so I'll leave it for GEOM to * decide what to do. */ count = acr + acw + ace; if (count == 0) return (0); flags = 0; if (acr != 0 || ace != 0) flags |= FREAD; if (acw != 0) flags |= FWRITE; g_topology_unlock(); if (count > 0) error = zvol_open(pp, flags, count); else error = zvol_close(pp, flags, -count); g_topology_lock(); return (error); } static void zvol_geom_start(struct bio *bp) { zvol_state_t *zv; boolean_t first; zv = bp->bio_to->private; ASSERT(zv != NULL); switch (bp->bio_cmd) { case BIO_FLUSH: if (!THREAD_CAN_SLEEP()) goto enqueue; zil_commit(zv->zv_zilog, ZVOL_OBJ); g_io_deliver(bp, 0); break; case BIO_READ: case BIO_WRITE: case BIO_DELETE: if (!THREAD_CAN_SLEEP()) goto enqueue; zvol_strategy(bp); break; case BIO_GETATTR: { spa_t *spa = dmu_objset_spa(zv->zv_objset); uint64_t refd, avail, usedobjs, availobjs, val; if (g_handleattr_int(bp, "GEOM::candelete", 1)) return; if (strcmp(bp->bio_attribute, "blocksavail") == 0) { dmu_objset_space(zv->zv_objset, &refd, &avail, &usedobjs, &availobjs); if (g_handleattr_off_t(bp, "blocksavail", avail / DEV_BSIZE)) return; } else if (strcmp(bp->bio_attribute, "blocksused") == 0) { dmu_objset_space(zv->zv_objset, &refd, &avail, &usedobjs, &availobjs); if (g_handleattr_off_t(bp, "blocksused", refd / DEV_BSIZE)) return; } else if (strcmp(bp->bio_attribute, "poolblocksavail") == 0) { avail = metaslab_class_get_space(spa_normal_class(spa)); avail -= metaslab_class_get_alloc(spa_normal_class(spa)); if (g_handleattr_off_t(bp, "poolblocksavail", avail / DEV_BSIZE)) return; } else if (strcmp(bp->bio_attribute, "poolblocksused") == 0) { refd = metaslab_class_get_alloc(spa_normal_class(spa)); if (g_handleattr_off_t(bp, "poolblocksused", refd / DEV_BSIZE)) return; } /* FALLTHROUGH */ } default: g_io_deliver(bp, EOPNOTSUPP); break; } return; enqueue: mtx_lock(&zv->zv_queue_mtx); first = (bioq_first(&zv->zv_queue) == NULL); bioq_insert_tail(&zv->zv_queue, bp); mtx_unlock(&zv->zv_queue_mtx); if (first) wakeup_one(&zv->zv_queue); } static void zvol_geom_worker(void *arg) { zvol_state_t *zv; struct bio *bp; thread_lock(curthread); sched_prio(curthread, PRIBIO); thread_unlock(curthread); zv = arg; for (;;) { mtx_lock(&zv->zv_queue_mtx); bp = bioq_takefirst(&zv->zv_queue); if (bp == NULL) { if (zv->zv_state == 1) { zv->zv_state = 2; wakeup(&zv->zv_state); mtx_unlock(&zv->zv_queue_mtx); kthread_exit(); } msleep(&zv->zv_queue, &zv->zv_queue_mtx, PRIBIO | PDROP, "zvol:io", 0); continue; } mtx_unlock(&zv->zv_queue_mtx); switch (bp->bio_cmd) { case BIO_FLUSH: zil_commit(zv->zv_zilog, ZVOL_OBJ); g_io_deliver(bp, 0); break; case BIO_READ: case BIO_WRITE: case BIO_DELETE: zvol_strategy(bp); break; default: g_io_deliver(bp, EOPNOTSUPP); break; } } } extern boolean_t dataset_name_hidden(const char *name); static int zvol_create_snapshots(objset_t *os, const char *name) { uint64_t cookie, obj; char *sname; int error, len; cookie = obj = 0; sname = kmem_alloc(MAXPATHLEN, KM_SLEEP); #if 0 (void) dmu_objset_find(name, dmu_objset_prefetch, NULL, DS_FIND_SNAPSHOTS); #endif for (;;) { len = snprintf(sname, MAXPATHLEN, "%s@", name); if (len >= MAXPATHLEN) { dmu_objset_rele(os, FTAG); error = ENAMETOOLONG; break; } dsl_pool_config_enter(dmu_objset_pool(os), FTAG); error = dmu_snapshot_list_next(os, MAXPATHLEN - len, sname + len, &obj, &cookie, NULL); dsl_pool_config_exit(dmu_objset_pool(os), FTAG); if (error != 0) { if (error == ENOENT) error = 0; break; } if ((error = zvol_create_minor(sname)) != 0) { printf("ZFS WARNING: Unable to create ZVOL %s (error=%d).\n", sname, error); break; } } kmem_free(sname, MAXPATHLEN); return (error); } int zvol_create_minors(const char *name) { uint64_t cookie; objset_t *os; char *osname, *p; int error, len; if (dataset_name_hidden(name)) return (0); if ((error = dmu_objset_hold(name, FTAG, &os)) != 0) { printf("ZFS WARNING: Unable to put hold on %s (error=%d).\n", name, error); return (error); } if (dmu_objset_type(os) == DMU_OST_ZVOL) { dsl_dataset_long_hold(os->os_dsl_dataset, FTAG); dsl_pool_rele(dmu_objset_pool(os), FTAG); error = zvol_create_minor(name); if (error == 0 || error == EEXIST) { error = zvol_create_snapshots(os, name); } else { printf("ZFS WARNING: Unable to create ZVOL %s (error=%d).\n", name, error); } dsl_dataset_long_rele(os->os_dsl_dataset, FTAG); dsl_dataset_rele(os->os_dsl_dataset, FTAG); return (error); } if (dmu_objset_type(os) != DMU_OST_ZFS) { dmu_objset_rele(os, FTAG); return (0); } osname = kmem_alloc(MAXPATHLEN, KM_SLEEP); if (snprintf(osname, MAXPATHLEN, "%s/", name) >= MAXPATHLEN) { dmu_objset_rele(os, FTAG); kmem_free(osname, MAXPATHLEN); return (ENOENT); } p = osname + strlen(osname); len = MAXPATHLEN - (p - osname); #if 0 /* Prefetch the datasets. */ cookie = 0; while (dmu_dir_list_next(os, len, p, NULL, &cookie) == 0) { if (!dataset_name_hidden(osname)) (void) dmu_objset_prefetch(osname, NULL); } #endif cookie = 0; while (dmu_dir_list_next(os, MAXPATHLEN - (p - osname), p, NULL, &cookie) == 0) { dmu_objset_rele(os, FTAG); (void)zvol_create_minors(osname); if ((error = dmu_objset_hold(name, FTAG, &os)) != 0) { printf("ZFS WARNING: Unable to put hold on %s (error=%d).\n", name, error); return (error); } } dmu_objset_rele(os, FTAG); kmem_free(osname, MAXPATHLEN); return (0); } static void zvol_rename_minor(zvol_state_t *zv, const char *newname) { struct g_geom *gp; struct g_provider *pp; struct cdev *dev; ASSERT(MUTEX_HELD(&zfsdev_state_lock)); if (zv->zv_volmode == ZFS_VOLMODE_GEOM) { g_topology_lock(); pp = zv->zv_provider; ASSERT(pp != NULL); gp = pp->geom; ASSERT(gp != NULL); zv->zv_provider = NULL; g_wither_provider(pp, ENXIO); pp = g_new_providerf(gp, "%s/%s", ZVOL_DRIVER, newname); pp->flags |= G_PF_DIRECT_RECEIVE | G_PF_DIRECT_SEND; pp->sectorsize = DEV_BSIZE; pp->mediasize = zv->zv_volsize; pp->private = zv; zv->zv_provider = pp; g_error_provider(pp, 0); g_topology_unlock(); } else if (zv->zv_volmode == ZFS_VOLMODE_DEV) { dev = zv->zv_dev; ASSERT(dev != NULL); zv->zv_dev = NULL; destroy_dev(dev); if (make_dev_p(MAKEDEV_CHECKNAME | MAKEDEV_WAITOK, &dev, &zvol_cdevsw, NULL, UID_ROOT, GID_OPERATOR, 0640, "%s/%s", ZVOL_DRIVER, newname) == 0) { zv->zv_dev = dev; dev->si_iosize_max = MAXPHYS; dev->si_drv2 = zv; } } strlcpy(zv->zv_name, newname, sizeof(zv->zv_name)); } void zvol_rename_minors(const char *oldname, const char *newname) { char name[MAXPATHLEN]; struct g_provider *pp; struct g_geom *gp; size_t oldnamelen, newnamelen; zvol_state_t *zv; char *namebuf; boolean_t locked = B_FALSE; oldnamelen = strlen(oldname); newnamelen = strlen(newname); DROP_GIANT(); /* See comment in zvol_open(). */ if (!MUTEX_HELD(&zfsdev_state_lock)) { mutex_enter(&zfsdev_state_lock); locked = B_TRUE; } LIST_FOREACH(zv, &all_zvols, zv_links) { if (strcmp(zv->zv_name, oldname) == 0) { zvol_rename_minor(zv, newname); } else if (strncmp(zv->zv_name, oldname, oldnamelen) == 0 && (zv->zv_name[oldnamelen] == '/' || zv->zv_name[oldnamelen] == '@')) { snprintf(name, sizeof(name), "%s%c%s", newname, zv->zv_name[oldnamelen], zv->zv_name + oldnamelen + 1); zvol_rename_minor(zv, name); } } if (locked) mutex_exit(&zfsdev_state_lock); PICKUP_GIANT(); } static int zvol_d_open(struct cdev *dev, int flags, int fmt, struct thread *td) { zvol_state_t *zv; int err = 0; mutex_enter(&zfsdev_state_lock); zv = dev->si_drv2; if (zv == NULL) { mutex_exit(&zfsdev_state_lock); return(ENXIO); /* zvol_create_minor() not done yet */ } if (zv->zv_total_opens == 0) err = zvol_first_open(zv); if (err) { mutex_exit(&zfsdev_state_lock); return (err); } if ((flags & FWRITE) && (zv->zv_flags & ZVOL_RDONLY)) { err = SET_ERROR(EROFS); goto out; } if (zv->zv_flags & ZVOL_EXCL) { err = SET_ERROR(EBUSY); goto out; } #ifdef FEXCL if (flags & FEXCL) { if (zv->zv_total_opens != 0) { err = SET_ERROR(EBUSY); goto out; } zv->zv_flags |= ZVOL_EXCL; } #endif zv->zv_total_opens++; mutex_exit(&zfsdev_state_lock); return (err); out: if (zv->zv_total_opens == 0) zvol_last_close(zv); mutex_exit(&zfsdev_state_lock); return (err); } static int zvol_d_close(struct cdev *dev, int flags, int fmt, struct thread *td) { zvol_state_t *zv; int err = 0; mutex_enter(&zfsdev_state_lock); zv = dev->si_drv2; if (zv == NULL) { mutex_exit(&zfsdev_state_lock); return(ENXIO); } if (zv->zv_flags & ZVOL_EXCL) { ASSERT(zv->zv_total_opens == 1); zv->zv_flags &= ~ZVOL_EXCL; } /* * If the open count is zero, this is a spurious close. * That indicates a bug in the kernel / DDI framework. */ ASSERT(zv->zv_total_opens != 0); /* * You may get multiple opens, but only one close. */ zv->zv_total_opens--; if (zv->zv_total_opens == 0) zvol_last_close(zv); mutex_exit(&zfsdev_state_lock); return (0); } static int zvol_d_ioctl(struct cdev *dev, u_long cmd, caddr_t data, int fflag, struct thread *td) { zvol_state_t *zv; rl_t *rl; off_t offset, length, chunk; int i, error; u_int u; zv = dev->si_drv2; error = 0; KASSERT(zv->zv_total_opens > 0, ("Device with zero access count in zvol_d_ioctl")); i = IOCPARM_LEN(cmd); switch (cmd) { case DIOCGSECTORSIZE: *(u_int *)data = DEV_BSIZE; break; case DIOCGMEDIASIZE: *(off_t *)data = zv->zv_volsize; break; case DIOCGFLUSH: zil_commit(zv->zv_zilog, ZVOL_OBJ); break; case DIOCGDELETE: if (!zvol_unmap_enabled) break; offset = ((off_t *)data)[0]; length = ((off_t *)data)[1]; if ((offset % DEV_BSIZE) != 0 || (length % DEV_BSIZE) != 0 || offset < 0 || offset >= zv->zv_volsize || length <= 0) { printf("%s: offset=%jd length=%jd\n", __func__, offset, length); error = EINVAL; break; } rl = zfs_range_lock(&zv->zv_znode, offset, length, RL_WRITER); dmu_tx_t *tx = dmu_tx_create(zv->zv_objset); error = dmu_tx_assign(tx, TXG_WAIT); if (error != 0) { dmu_tx_abort(tx); } else { zvol_log_truncate(zv, tx, offset, length, B_TRUE); dmu_tx_commit(tx); error = dmu_free_long_range(zv->zv_objset, ZVOL_OBJ, offset, length); } zfs_range_unlock(rl); if (zv->zv_objset->os_sync == ZFS_SYNC_ALWAYS) zil_commit(zv->zv_zilog, ZVOL_OBJ); break; case DIOCGSTRIPESIZE: *(off_t *)data = zv->zv_volblocksize; break; case DIOCGSTRIPEOFFSET: *(off_t *)data = 0; break; case DIOCGATTR: { spa_t *spa = dmu_objset_spa(zv->zv_objset); struct diocgattr_arg *arg = (struct diocgattr_arg *)data; uint64_t refd, avail, usedobjs, availobjs; if (strcmp(arg->name, "GEOM::candelete") == 0) arg->value.i = 1; else if (strcmp(arg->name, "blocksavail") == 0) { dmu_objset_space(zv->zv_objset, &refd, &avail, &usedobjs, &availobjs); arg->value.off = avail / DEV_BSIZE; } else if (strcmp(arg->name, "blocksused") == 0) { dmu_objset_space(zv->zv_objset, &refd, &avail, &usedobjs, &availobjs); arg->value.off = refd / DEV_BSIZE; } else if (strcmp(arg->name, "poolblocksavail") == 0) { avail = metaslab_class_get_space(spa_normal_class(spa)); avail -= metaslab_class_get_alloc(spa_normal_class(spa)); arg->value.off = avail / DEV_BSIZE; } else if (strcmp(arg->name, "poolblocksused") == 0) { refd = metaslab_class_get_alloc(spa_normal_class(spa)); arg->value.off = refd / DEV_BSIZE; } else error = ENOIOCTL; break; } case FIOSEEKHOLE: case FIOSEEKDATA: { off_t *off = (off_t *)data; uint64_t noff; boolean_t hole; hole = (cmd == FIOSEEKHOLE); noff = *off; error = dmu_offset_next(zv->zv_objset, ZVOL_OBJ, hole, &noff); *off = noff; break; } default: error = ENOIOCTL; } return (error); } #endif /* illumos */ Index: user/ngie/make_check/sys/cddl/contrib/opensolaris/uts/common/sys/isa_defs.h =================================================================== --- user/ngie/make_check/sys/cddl/contrib/opensolaris/uts/common/sys/isa_defs.h (revision 295077) +++ user/ngie/make_check/sys/cddl/contrib/opensolaris/uts/common/sys/isa_defs.h (revision 295078) @@ -1,653 +1,695 @@ /* * CDDL HEADER START * * The contents of this file are subject to the terms of the * Common Development and Distribution License (the "License"). * You may not use this file except in compliance with the License. * * You can obtain a copy of the license at usr/src/OPENSOLARIS.LICENSE * or http://www.opensolaris.org/os/licensing. * See the License for the specific language governing permissions * and limitations under the License. * * * When distributing Covered Code, include this CDDL HEADER in each * file and include the License file at usr/src/OPENSOLARIS.LICENSE. * If applicable, add the following below this CDDL HEADER, with the * fields enclosed by brackets "[]" replaced with your own identifying * information: Portions Copyright [yyyy] [name of copyright owner] * * CDDL HEADER END */ /* * Copyright 2008 Sun Microsystems, Inc. All rights reserved. * Use is subject to license terms. */ #ifndef _SYS_ISA_DEFS_H #define _SYS_ISA_DEFS_H /* * This header file serves to group a set of well known defines and to * set these for each instruction set architecture. These defines may * be divided into two groups; characteristics of the processor and * implementation choices for Solaris on a processor. * * Processor Characteristics: * * _LITTLE_ENDIAN / _BIG_ENDIAN: * The natural byte order of the processor. A pointer to an int points * to the least/most significant byte of that int. * * _STACK_GROWS_UPWARD / _STACK_GROWS_DOWNWARD: * The processor specific direction of stack growth. A push onto the * stack increases/decreases the stack pointer, so it stores data at * successively higher/lower addresses. (Stackless machines ignored * without regrets). * * _LONG_LONG_HTOL / _LONG_LONG_LTOH: * A pointer to a long long points to the most/least significant long * within that long long. * * _BIT_FIELDS_HTOL / _BIT_FIELDS_LTOH: * The C compiler assigns bit fields from the high/low to the low/high end * of an int (most to least significant vs. least to most significant). * * _IEEE_754: * The processor (or supported implementations of the processor) * supports the ieee-754 floating point standard. No other floating * point standards are supported (or significant). Any other supported * floating point formats are expected to be cased on the ISA processor * symbol. * * _CHAR_IS_UNSIGNED / _CHAR_IS_SIGNED: * The C Compiler implements objects of type `char' as `unsigned' or * `signed' respectively. This is really an implementation choice of * the compiler writer, but it is specified in the ABI and tends to * be uniform across compilers for an instruction set architecture. * Hence, it has the properties of a processor characteristic. * * _CHAR_ALIGNMENT / _SHORT_ALIGNMENT / _INT_ALIGNMENT / _LONG_ALIGNMENT / * _LONG_LONG_ALIGNMENT / _DOUBLE_ALIGNMENT / _LONG_DOUBLE_ALIGNMENT / * _POINTER_ALIGNMENT / _FLOAT_ALIGNMENT: * The ABI defines alignment requirements of each of the primitive * object types. Some, if not all, may be hardware requirements as * well. The values are expressed in "byte-alignment" units. * * _MAX_ALIGNMENT: * The most stringent alignment requirement as specified by the ABI. * Equal to the maximum of all the above _XXX_ALIGNMENT values. * * _ALIGNMENT_REQUIRED: * True or false (1 or 0) whether or not the hardware requires the ABI * alignment. * * _LONG_LONG_ALIGNMENT_32 * The 32-bit ABI supported by a 64-bit kernel may have different * alignment requirements for primitive object types. The value of this * identifier is expressed in "byte-alignment" units. * * _HAVE_CPUID_INSN * This indicates that the architecture supports the 'cpuid' * instruction as defined by Intel. (Intel allows other vendors * to extend the instruction for their own purposes.) * * * Implementation Choices: * * _ILP32 / _LP64: * This specifies the compiler data type implementation as specified in * the relevant ABI. The choice between these is strongly influenced * by the underlying hardware, but is not absolutely tied to it. * Currently only two data type models are supported: * * _ILP32: * Int/Long/Pointer are 32 bits. This is the historical UNIX * and Solaris implementation. Due to its historical standing, * this is the default case. * * _LP64: * Long/Pointer are 64 bits, Int is 32 bits. This is the chosen * implementation for 64-bit ABIs such as SPARC V9. * * _I32LPx: * A compilation environment where 'int' is 32-bit, and * longs and pointers are simply the same size. * * In all cases, Char is 8 bits and Short is 16 bits. * * _SUNOS_VTOC_8 / _SUNOS_VTOC_16 / _SVR4_VTOC_16: * This specifies the form of the disk VTOC (or label): * * _SUNOS_VTOC_8: * This is a VTOC form which is upwardly compatible with the * SunOS 4.x disk label and allows 8 partitions per disk. * * _SUNOS_VTOC_16: * In this format the incore vtoc image matches the ondisk * version. It allows 16 slices per disk, and is not * compatible with the SunOS 4.x disk label. * * Note that these are not the only two VTOC forms possible and * additional forms may be added. One possible form would be the * SVr4 VTOC form. The symbol for that is reserved now, although * it is not implemented. * * _SVR4_VTOC_16: * This VTOC form is compatible with the System V Release 4 * VTOC (as implemented on the SVr4 Intel and 3b ports) with * 16 partitions per disk. * * * _DMA_USES_PHYSADDR / _DMA_USES_VIRTADDR * This describes the type of addresses used by system DMA: * * _DMA_USES_PHYSADDR: * This type of DMA, used in the x86 implementation, * requires physical addresses for DMA buffers. The 24-bit * addresses used by some legacy boards is the source of the * "low-memory" (<16MB) requirement for some devices using DMA. * * _DMA_USES_VIRTADDR: * This method of DMA allows the use of virtual addresses for * DMA transfers. * * _FIRMWARE_NEEDS_FDISK / _NO_FDISK_PRESENT * This indicates the presence/absence of an fdisk table. * * _FIRMWARE_NEEDS_FDISK * The fdisk table is required by system firmware. If present, * it allows a disk to be subdivided into multiple fdisk * partitions, each of which is equivalent to a separate, * virtual disk. This enables the co-existence of multiple * operating systems on a shared hard disk. * * _NO_FDISK_PRESENT * If the fdisk table is absent, it is assumed that the entire * media is allocated for a single operating system. * * _HAVE_TEM_FIRMWARE * Defined if this architecture has the (fallback) option of * using prom_* calls for doing I/O if a suitable kernel driver * is not available to do it. * * _DONT_USE_1275_GENERIC_NAMES * Controls whether or not device tree node names should * comply with the IEEE 1275 "Generic Names" Recommended * Practice. With _DONT_USE_GENERIC_NAMES, device-specific * names identifying the particular device will be used. * * __i386_COMPAT * This indicates whether the i386 ABI is supported as a *non-native* * mode for the platform. When this symbol is defined: * - 32-bit xstat-style system calls are enabled * - 32-bit xmknod-style system calls are enabled * - 32-bit system calls use i386 sizes -and- alignments * * Note that this is NOT defined for the i386 native environment! * * __x86 * This is ONLY a synonym for defined(__i386) || defined(__amd64) * which is useful only insofar as these two architectures share * common attributes. Analogous to __sparc. * * _PSM_MODULES * This indicates whether or not the implementation uses PSM * modules for processor support, reading /etc/mach from inside * the kernel to extract a list. * * _RTC_CONFIG * This indicates whether or not the implementation uses /etc/rtc_config * to configure the real-time clock in the kernel. * * _UNIX_KRTLD * This indicates that the implementation uses a dynamically * linked unix + krtld to form the core kernel image at boot * time, or (in the absence of this symbol) a prelinked kernel image. * * _OBP * This indicates the firmware interface is OBP. * * _SOFT_HOSTID * This indicates that the implementation obtains the hostid * from the file /etc/hostid, rather than from hardware. */ #ifdef __cplusplus extern "C" { #endif /* * The following set of definitions characterize Solaris on AMD's * 64-bit systems. */ #if defined(__x86_64) || defined(__amd64) #if !defined(__amd64) #define __amd64 /* preferred guard */ #endif #if !defined(__x86) #define __x86 #endif /* * Define the appropriate "processor characteristics" */ #ifdef illumos #define _LITTLE_ENDIAN #endif #define _STACK_GROWS_DOWNWARD #define _LONG_LONG_LTOH #define _BIT_FIELDS_LTOH #define _IEEE_754 #define _CHAR_IS_SIGNED #define _BOOL_ALIGNMENT 1 #define _CHAR_ALIGNMENT 1 #define _SHORT_ALIGNMENT 2 #define _INT_ALIGNMENT 4 #define _FLOAT_ALIGNMENT 4 #define _FLOAT_COMPLEX_ALIGNMENT 4 #define _LONG_ALIGNMENT 8 #define _LONG_LONG_ALIGNMENT 8 #define _DOUBLE_ALIGNMENT 8 #define _DOUBLE_COMPLEX_ALIGNMENT 8 #define _LONG_DOUBLE_ALIGNMENT 16 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 16 #define _POINTER_ALIGNMENT 8 #define _MAX_ALIGNMENT 16 #define _ALIGNMENT_REQUIRED 1 /* * Different alignment constraints for the i386 ABI in compatibility mode */ #define _LONG_LONG_ALIGNMENT_32 4 /* * Define the appropriate "implementation choices". */ #if !defined(_LP64) #define _LP64 #endif #if !defined(_I32LPx) && defined(_KERNEL) #define _I32LPx #endif #define _MULTI_DATAMODEL #define _SUNOS_VTOC_16 #define _DMA_USES_PHYSADDR #define _FIRMWARE_NEEDS_FDISK #define __i386_COMPAT #define _PSM_MODULES #define _RTC_CONFIG #define _SOFT_HOSTID #define _DONT_USE_1275_GENERIC_NAMES #define _HAVE_CPUID_INSN /* * The feature test macro __i386 is generic for all processors implementing * the Intel 386 instruction set or a superset of it. Specifically, this * includes all members of the 386, 486, and Pentium family of processors. */ #elif defined(__i386) || defined(__i386__) #if !defined(__i386) #define __i386 #endif #if !defined(__x86) #define __x86 #endif /* * Define the appropriate "processor characteristics" */ #ifdef illumos #define _LITTLE_ENDIAN #endif #define _STACK_GROWS_DOWNWARD #define _LONG_LONG_LTOH #define _BIT_FIELDS_LTOH #define _IEEE_754 #define _CHAR_IS_SIGNED #define _BOOL_ALIGNMENT 1 #define _CHAR_ALIGNMENT 1 #define _SHORT_ALIGNMENT 2 #define _INT_ALIGNMENT 4 #define _FLOAT_ALIGNMENT 4 #define _FLOAT_COMPLEX_ALIGNMENT 4 #define _LONG_ALIGNMENT 4 #define _LONG_LONG_ALIGNMENT 4 #define _DOUBLE_ALIGNMENT 4 #define _DOUBLE_COMPLEX_ALIGNMENT 4 #define _LONG_DOUBLE_ALIGNMENT 4 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 4 #define _POINTER_ALIGNMENT 4 #define _MAX_ALIGNMENT 4 #define _ALIGNMENT_REQUIRED 0 #define _LONG_LONG_ALIGNMENT_32 _LONG_LONG_ALIGNMENT /* * Define the appropriate "implementation choices". */ #if !defined(_ILP32) #define _ILP32 #endif #if !defined(_I32LPx) && defined(_KERNEL) #define _I32LPx #endif #define _SUNOS_VTOC_16 #define _DMA_USES_PHYSADDR #define _FIRMWARE_NEEDS_FDISK #define _PSM_MODULES #define _RTC_CONFIG #define _SOFT_HOSTID #define _DONT_USE_1275_GENERIC_NAMES #define _HAVE_CPUID_INSN #elif defined(__aarch64__) /* * Define the appropriate "processor characteristics" */ #define _STACK_GROWS_DOWNWARD #define _LONG_LONG_LTOH #define _BIT_FIELDS_LTOH #define _IEEE_754 #define _CHAR_IS_UNSIGNED #define _BOOL_ALIGNMENT 1 #define _CHAR_ALIGNMENT 1 #define _SHORT_ALIGNMENT 2 #define _INT_ALIGNMENT 4 #define _FLOAT_ALIGNMENT 4 #define _FLOAT_COMPLEX_ALIGNMENT 4 #define _LONG_ALIGNMENT 8 #define _LONG_LONG_ALIGNMENT 8 #define _DOUBLE_ALIGNMENT 8 #define _DOUBLE_COMPLEX_ALIGNMENT 8 #define _LONG_DOUBLE_ALIGNMENT 16 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 16 #define _POINTER_ALIGNMENT 8 #define _MAX_ALIGNMENT 16 #define _ALIGNMENT_REQUIRED 1 #define _LONG_LONG_ALIGNMENT_32 _LONG_LONG_ALIGNMENT /* * Define the appropriate "implementation choices" */ #if !defined(_LP64) #define _LP64 #endif #define _SUNOS_VTOC_16 #define _DMA_USES_PHYSADDR #define _FIRMWARE_NEEDS_FDISK #define _PSM_MODULES #define _RTC_CONFIG #define _DONT_USE_1275_GENERIC_NAMES #define _HAVE_CPUID_INSN +#elif defined(__riscv__) + +/* + * Define the appropriate "processor characteristics" + */ +#define _STACK_GROWS_DOWNWARD +#define _LONG_LONG_LTOH +#define _BIT_FIELDS_LTOH +#define _IEEE_754 +#define _CHAR_IS_UNSIGNED +#define _BOOL_ALIGNMENT 1 +#define _CHAR_ALIGNMENT 1 +#define _SHORT_ALIGNMENT 2 +#define _INT_ALIGNMENT 4 +#define _FLOAT_ALIGNMENT 4 +#define _FLOAT_COMPLEX_ALIGNMENT 4 +#define _LONG_ALIGNMENT 8 +#define _LONG_LONG_ALIGNMENT 8 +#define _DOUBLE_ALIGNMENT 8 +#define _DOUBLE_COMPLEX_ALIGNMENT 8 +#define _LONG_DOUBLE_ALIGNMENT 16 +#define _LONG_DOUBLE_COMPLEX_ALIGNMENT 16 +#define _POINTER_ALIGNMENT 8 +#define _MAX_ALIGNMENT 16 +#define _ALIGNMENT_REQUIRED 1 + +#define _LONG_LONG_ALIGNMENT_32 _LONG_LONG_ALIGNMENT + +/* + * Define the appropriate "implementation choices" + */ +#if !defined(_LP64) +#define _LP64 +#endif +#define _SUNOS_VTOC_16 +#define _DMA_USES_PHYSADDR +#define _FIRMWARE_NEEDS_FDISK +#define _PSM_MODULES +#define _RTC_CONFIG +#define _DONT_USE_1275_GENERIC_NAMES +#define _HAVE_CPUID_INSN + #elif defined(__arm__) /* * Define the appropriate "processor characteristics" */ #define _STACK_GROWS_DOWNWARD #define _LONG_LONG_LTOH #define _BIT_FIELDS_LTOH #define _IEEE_754 #define _CHAR_IS_SIGNED #define _BOOL_ALIGNMENT 1 #define _CHAR_ALIGNMENT 1 #define _SHORT_ALIGNMENT 2 #define _INT_ALIGNMENT 4 #define _FLOAT_ALIGNMENT 4 #define _FLOAT_COMPLEX_ALIGNMENT 4 #define _LONG_ALIGNMENT 4 #define _LONG_LONG_ALIGNMENT 4 #define _DOUBLE_ALIGNMENT 4 #define _DOUBLE_COMPLEX_ALIGNMENT 4 #define _LONG_DOUBLE_ALIGNMENT 4 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 4 #define _POINTER_ALIGNMENT 4 #define _MAX_ALIGNMENT 4 #define _ALIGNMENT_REQUIRED 0 #define _LONG_LONG_ALIGNMENT_32 _LONG_LONG_ALIGNMENT /* * Define the appropriate "implementation choices". */ #if !defined(_ILP32) #define _ILP32 #endif #if !defined(_I32LPx) && defined(_KERNEL) #define _I32LPx #endif #define _SUNOS_VTOC_16 #define _DMA_USES_PHYSADDR #define _FIRMWARE_NEEDS_FDISK #define _PSM_MODULES #define _RTC_CONFIG #define _DONT_USE_1275_GENERIC_NAMES #define _HAVE_CPUID_INSN #elif defined(__mips__) /* * Define the appropriate "processor characteristics" */ #define _STACK_GROWS_DOWNWARD #define _LONG_LONG_LTOH #define _BIT_FIELDS_LTOH #define _IEEE_754 #define _CHAR_IS_SIGNED #define _BOOL_ALIGNMENT 1 #define _CHAR_ALIGNMENT 1 #define _SHORT_ALIGNMENT 2 #define _INT_ALIGNMENT 4 #define _FLOAT_ALIGNMENT 4 #define _FLOAT_COMPLEX_ALIGNMENT 4 #if defined(__mips_n64) #define _LONG_ALIGNMENT 8 #define _LONG_LONG_ALIGNMENT 8 #define _DOUBLE_ALIGNMENT 8 #define _DOUBLE_COMPLEX_ALIGNMENT 8 #define _LONG_DOUBLE_ALIGNMENT 8 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 8 #define _POINTER_ALIGNMENT 8 #define _MAX_ALIGNMENT 8 #define _ALIGNMENT_REQUIRED 0 #define _LONG_LONG_ALIGNMENT_32 _INT_ALIGNMENT /* * Define the appropriate "implementation choices". */ #if !defined(_LP64) #define _LP64 #endif #else #define _LONG_ALIGNMENT 4 #define _LONG_LONG_ALIGNMENT 4 #define _DOUBLE_ALIGNMENT 4 #define _DOUBLE_COMPLEX_ALIGNMENT 4 #define _LONG_DOUBLE_ALIGNMENT 4 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 4 #define _POINTER_ALIGNMENT 4 #define _MAX_ALIGNMENT 4 #define _ALIGNMENT_REQUIRED 0 #define _LONG_LONG_ALIGNMENT_32 _LONG_LONG_ALIGNMENT /* * Define the appropriate "implementation choices". */ #define _ILP32 #if !defined(_I32LPx) && defined(_KERNEL) #define _I32LPx #endif #endif #define _SUNOS_VTOC_16 #define _DMA_USES_PHYSADDR #define _FIRMWARE_NEEDS_FDISK #define _PSM_MODULES #define _RTC_CONFIG #define _DONT_USE_1275_GENERIC_NAMES #define _HAVE_CPUID_INSN #elif defined(__powerpc__) #if defined(__BIG_ENDIAN__) #define _BIT_FIELDS_HTOL #else #define _BIT_FIELDS_LTOH #endif /* * The following set of definitions characterize the Solaris on SPARC systems. * * The symbol __sparc indicates any of the SPARC family of processor * architectures. This includes SPARC V7, SPARC V8 and SPARC V9. * * The symbol __sparcv8 indicates the 32-bit SPARC V8 architecture as defined * by Version 8 of the SPARC Architecture Manual. (SPARC V7 is close enough * to SPARC V8 for the former to be subsumed into the latter definition.) * * The symbol __sparcv9 indicates the 64-bit SPARC V9 architecture as defined * by Version 9 of the SPARC Architecture Manual. * * The symbols __sparcv8 and __sparcv9 are mutually exclusive, and are only * relevant when the symbol __sparc is defined. */ /* * XXX Due to the existence of 5110166, "defined(__sparcv9)" needs to be added * to support backwards builds. This workaround should be removed in s10_71. */ #elif defined(__sparc) || defined(__sparcv9) || defined(__sparc__) #if !defined(__sparc) #define __sparc #endif /* * You can be 32-bit or 64-bit, but not both at the same time. */ #if defined(__sparcv8) && defined(__sparcv9) #error "SPARC Versions 8 and 9 are mutually exclusive choices" #endif /* * Existing compilers do not set __sparcv8. Years will transpire before * the compilers can be depended on to set the feature test macro. In * the interim, we'll set it here on the basis of historical behaviour; * if you haven't asked for SPARC V9, then you must've meant SPARC V8. */ #if !defined(__sparcv9) && !defined(__sparcv8) #define __sparcv8 #endif /* * Define the appropriate "processor characteristics" shared between * all Solaris on SPARC systems. */ #ifdef illumos #define _BIG_ENDIAN #endif #define _STACK_GROWS_DOWNWARD #define _LONG_LONG_HTOL #define _BIT_FIELDS_HTOL #define _IEEE_754 #define _CHAR_IS_SIGNED #define _BOOL_ALIGNMENT 1 #define _CHAR_ALIGNMENT 1 #define _SHORT_ALIGNMENT 2 #define _INT_ALIGNMENT 4 #define _FLOAT_ALIGNMENT 4 #define _FLOAT_COMPLEX_ALIGNMENT 4 #define _LONG_LONG_ALIGNMENT 8 #define _DOUBLE_ALIGNMENT 8 #define _DOUBLE_COMPLEX_ALIGNMENT 8 #define _ALIGNMENT_REQUIRED 1 /* * Define the appropriate "implementation choices" shared between versions. */ #define _SUNOS_VTOC_8 #define _DMA_USES_VIRTADDR #define _NO_FDISK_PRESENT #define _HAVE_TEM_FIRMWARE #define _OBP /* * The following set of definitions characterize the implementation of * 32-bit Solaris on SPARC V8 systems. */ #if defined(__sparcv8) /* * Define the appropriate "processor characteristics" */ #define _LONG_ALIGNMENT 4 #define _LONG_DOUBLE_ALIGNMENT 8 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 8 #define _POINTER_ALIGNMENT 4 #define _MAX_ALIGNMENT 8 #define _LONG_LONG_ALIGNMENT_32 _LONG_LONG_ALIGNMENT /* * Define the appropriate "implementation choices" */ #define _ILP32 #if !defined(_I32LPx) && defined(_KERNEL) #define _I32LPx #endif /* * The following set of definitions characterize the implementation of * 64-bit Solaris on SPARC V9 systems. */ #elif defined(__sparcv9) /* * Define the appropriate "processor characteristics" */ #define _LONG_ALIGNMENT 8 #define _LONG_DOUBLE_ALIGNMENT 16 #define _LONG_DOUBLE_COMPLEX_ALIGNMENT 16 #define _POINTER_ALIGNMENT 8 #define _MAX_ALIGNMENT 16 #define _LONG_LONG_ALIGNMENT_32 _LONG_LONG_ALIGNMENT /* * Define the appropriate "implementation choices" */ #if !defined(_LP64) #define _LP64 #endif #if !defined(_I32LPx) #define _I32LPx #endif #define _MULTI_DATAMODEL #else #error "unknown SPARC version" #endif /* * #error is strictly ansi-C, but works as well as anything for K&R systems. */ #else #error "ISA not supported" #endif #if defined(_ILP32) && defined(_LP64) #error "Both _ILP32 and _LP64 are defined" #endif #ifdef __cplusplus } #endif #endif /* _SYS_ISA_DEFS_H */ Index: user/ngie/make_check/sys/cddl/contrib/opensolaris =================================================================== --- user/ngie/make_check/sys/cddl/contrib/opensolaris (revision 295077) +++ user/ngie/make_check/sys/cddl/contrib/opensolaris (revision 295078) Property changes on: user/ngie/make_check/sys/cddl/contrib/opensolaris ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/cddl/contrib/opensolaris:r295034-295077 Index: user/ngie/make_check/sys/conf/Makefile.riscv =================================================================== --- user/ngie/make_check/sys/conf/Makefile.riscv (nonexistent) +++ user/ngie/make_check/sys/conf/Makefile.riscv (revision 295078) @@ -0,0 +1,49 @@ +# Makefile.riscv -- with config changes. +# Copyright 1990 W. Jolitz +# from: @(#)Makefile.i386 7.1 5/10/91 +# from FreeBSD: src/sys/conf/Makefile.i386,v 1.255 2002/02/20 23:35:49 +# $FreeBSD$ +# +# Makefile for FreeBSD +# +# RISCVTODO: copy pasted from aarch64, needs to be +# constructed from a machine description: +# config machineid +# Most changes should be made in the machine description +# /sys/riscv/conf/``machineid'' +# after which you should do +# config machineid +# Generic makefile changes should be made in +# /sys/conf/Makefile.riscv +# after which config should be rerun for all machines. +# + +# Which version of config(8) is required. +%VERSREQ= 600012 + +.if !defined(S) +S= ../../.. +.endif +.include "$S/conf/kern.pre.mk" + +INCLUDES+= -I$S/contrib/libfdt + +.if !empty(DDB_ENABLED) +CFLAGS += -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer +.endif + +%BEFORE_DEPEND + +%OBJS + +%FILES.c + +%FILES.s + +%FILES.m + +%CLEAN + +%RULES + +.include "$S/conf/kern.post.mk" Property changes on: user/ngie/make_check/sys/conf/Makefile.riscv ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/conf/files.arm =================================================================== --- user/ngie/make_check/sys/conf/files.arm (revision 295077) +++ user/ngie/make_check/sys/conf/files.arm (revision 295078) @@ -1,139 +1,140 @@ # $FreeBSD$ arm/arm/autoconf.c standard arm/arm/bcopy_page.S standard arm/arm/bcopyinout.S standard arm/arm/blockio.S standard arm/arm/bus_space_asm_generic.S standard arm/arm/bus_space_base.c optional fdt arm/arm/bus_space_generic.c standard arm/arm/busdma_machdep.c optional !armv6 arm/arm/busdma_machdep-v6.c optional armv6 arm/arm/copystr.S standard arm/arm/cpufunc.c standard arm/arm/cpufunc_asm.S standard arm/arm/cpufunc_asm_arm9.S optional cpu_arm9 arm/arm/cpufunc_asm_arm10.S optional cpu_arm9e arm/arm/cpufunc_asm_arm11.S optional cpu_arm1176 arm/arm/cpufunc_asm_arm11x6.S optional cpu_arm1176 arm/arm/cpufunc_asm_armv4.S optional cpu_arm9 | cpu_arm9e | cpu_fa526 | cpu_xscale_80321 | cpu_xscale_pxa2x0 | cpu_xscale_ixp425 | cpu_xscale_80219 | cpu_xscale_81342 arm/arm/cpufunc_asm_armv5_ec.S optional cpu_arm9e arm/arm/cpufunc_asm_armv6.S optional cpu_arm1176 arm/arm/cpufunc_asm_armv7.S optional cpu_cortexa | cpu_krait | cpu_mv_pj4b arm/arm/cpufunc_asm_fa526.S optional cpu_fa526 arm/arm/cpufunc_asm_pj4b.S optional cpu_mv_pj4b arm/arm/cpufunc_asm_sheeva.S optional cpu_arm9e arm/arm/cpufunc_asm_xscale.S optional cpu_xscale_80321 | cpu_xscale_pxa2x0 | cpu_xscale_ixp425 | cpu_xscale_80219 | cpu_xscale_81342 arm/arm/cpufunc_asm_xscale_c3.S optional cpu_xscale_81342 arm/arm/cpuinfo.c standard arm/arm/cpu_asm-v6.S optional armv6 arm/arm/db_disasm.c optional ddb arm/arm/db_interface.c optional ddb arm/arm/db_trace.c optional ddb arm/arm/debug_monitor.c optional ddb armv6 arm/arm/devmap.c standard arm/arm/disassem.c optional ddb arm/arm/dump_machdep.c standard arm/arm/elf_machdep.c standard arm/arm/elf_note.S standard arm/arm/exception.S standard arm/arm/fiq.c standard arm/arm/fiq_subr.S standard arm/arm/fusu.S standard arm/arm/gdb_machdep.c optional gdb arm/arm/generic_timer.c optional generic_timer arm/arm/gic.c optional gic arm/arm/hdmi_if.m optional hdmi arm/arm/identcpu.c standard arm/arm/in_cksum.c optional inet | inet6 arm/arm/in_cksum_arm.S optional inet | inet6 arm/arm/intr.c optional !arm_intrng kern/subr_intr.c optional arm_intrng arm/arm/locore.S standard no-obj arm/arm/machdep.c standard arm/arm/machdep_intr.c standard arm/arm/mem.c optional mem arm/arm/minidump_machdep.c optional mem arm/arm/mp_machdep.c optional smp arm/arm/mpcore_timer.c optional mpcore_timer arm/arm/nexus.c standard arm/arm/ofw_machdep.c optional fdt arm/arm/physmem.c standard kern/pic_if.m optional arm_intrng arm/arm/pl190.c optional pl190 arm/arm/pl310.c optional pl310 arm/arm/platform.c optional platform arm/arm/platform_if.m optional platform arm/arm/pmap.c optional !armv6 -arm/arm/pmap-v6.c optional armv6 !arm_new_pmap -arm/arm/pmap-v6-new.c optional armv6 arm_new_pmap +arm/arm/pmap-v6.c optional armv6 arm/arm/pmu.c optional pmu | fdt hwpmc arm/arm/sc_machdep.c optional sc arm/arm/setcpsr.S standard arm/arm/setstack.s standard arm/arm/stack_machdep.c optional ddb | stack arm/arm/stdatomic.c standard \ compile-with "${NORMAL_C:N-Wmissing-prototypes}" arm/arm/support.S standard arm/arm/swtch.S standard +arm/arm/swtch-v4.S optional !armv6 +arm/arm/swtch-v6.S optional armv6 arm/arm/sys_machdep.c standard arm/arm/syscall.c standard arm/arm/trap.c optional !armv6 arm/arm/trap-v6.c optional armv6 arm/arm/uio_machdep.c standard arm/arm/undefined.c standard arm/arm/unwind.c optional ddb | kdtrace_hooks arm/arm/vm_machdep.c standard arm/arm/vfp.c standard board_id.h standard \ dependency "$S/arm/conf/genboardid.awk $S/arm/conf/mach-types" \ compile-with "${AWK} -f $S/arm/conf/genboardid.awk $S/arm/conf/mach-types > board_id.h" \ no-obj no-implicit-rule before-depend \ clean "board_id.h" cddl/compat/opensolaris/kern/opensolaris_atomic.c optional zfs | dtrace compile-with "${CDDL_C}" cddl/dev/dtrace/arm/dtrace_asm.S optional dtrace compile-with "${DTRACE_S}" cddl/dev/dtrace/arm/dtrace_subr.c optional dtrace compile-with "${DTRACE_C}" cddl/dev/fbt/arm/fbt_isa.c optional dtrace_fbt | dtraceall compile-with "${FBT_C}" crypto/blowfish/bf_enc.c optional crypto | ipsec crypto/des/des_enc.c optional crypto | ipsec | netsmb dev/dwc/if_dwc.c optional dwc dev/dwc/if_dwc_if.m optional dwc dev/fb/fb.c optional sc dev/fdt/fdt_arm_platform.c optional platform fdt dev/hwpmc/hwpmc_arm.c optional hwpmc dev/hwpmc/hwpmc_armv7.c optional hwpmc armv6 dev/psci/psci.c optional psci dev/psci/psci_arm.S optional psci dev/syscons/scgfbrndr.c optional sc dev/syscons/scterm-teken.c optional sc dev/syscons/scvtb.c optional sc dev/uart/uart_cpu_fdt.c optional uart fdt font.h optional sc \ compile-with "uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x16.fnt && file2c 'u_char dflt_font_16[16*256] = {' '};' < ${SC_DFLT_FONT}-8x16 > font.h && uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x14.fnt && file2c 'u_char dflt_font_14[14*256] = {' '};' < ${SC_DFLT_FONT}-8x14 >> font.h && uudecode < /usr/share/syscons/fonts/${SC_DFLT_FONT}-8x8.fnt && file2c 'u_char dflt_font_8[8*256] = {' '};' < ${SC_DFLT_FONT}-8x8 >> font.h" \ no-obj no-implicit-rule before-depend \ clean "font.h ${SC_DFLT_FONT}-8x14 ${SC_DFLT_FONT}-8x16 ${SC_DFLT_FONT}-8x8" kern/subr_busdma_bufalloc.c standard kern/subr_sfbuf.c standard libkern/arm/aeabi_unwind.c standard libkern/arm/divsi3.S standard libkern/arm/ffs.S standard libkern/arm/ldivmod.S standard libkern/arm/ldivmod_helper.c standard libkern/arm/memclr.S standard libkern/arm/memcpy.S standard libkern/arm/memset.S standard libkern/arm/muldi3.c standard libkern/ashldi3.c standard libkern/ashrdi3.c standard libkern/divdi3.c standard libkern/ffsl.c standard libkern/ffsll.c standard libkern/fls.c standard libkern/flsl.c standard libkern/flsll.c standard libkern/lshrdi3.c standard libkern/moddi3.c standard libkern/qdivrem.c standard libkern/ucmpdi2.c standard libkern/udivdi3.c standard libkern/umoddi3.c standard Index: user/ngie/make_check/sys/conf/files.arm64 =================================================================== --- user/ngie/make_check/sys/conf/files.arm64 (revision 295077) +++ user/ngie/make_check/sys/conf/files.arm64 (revision 295078) @@ -1,96 +1,97 @@ # $FreeBSD$ arm/arm/devmap.c standard arm/arm/generic_timer.c standard arm/arm/pmu.c standard arm64/acpica/acpi_machdep.c optional acpi arm64/acpica/OsdEnvironment.c optional acpi arm64/acpica/acpi_wakeup.c optional acpi arm64/acpica/pci_cfgreg.c optional acpi pci arm64/arm64/autoconf.c standard arm64/arm64/bcopy.c standard arm64/arm64/bus_machdep.c standard arm64/arm64/bus_space_asm.S standard arm64/arm64/busdma_bounce.c standard arm64/arm64/busdma_machdep.c standard arm64/arm64/clock.c standard arm64/arm64/copyinout.S standard arm64/arm64/copystr.c standard arm64/arm64/cpufunc_asm.S standard arm64/arm64/db_disasm.c optional ddb arm64/arm64/db_interface.c optional ddb arm64/arm64/db_trace.c optional ddb arm64/arm64/debug_monitor.c optional kdb +arm64/arm64/disassem.c optional ddb arm64/arm64/dump_machdep.c standard arm64/arm64/elf_machdep.c standard arm64/arm64/exception.S standard arm64/arm64/gic.c standard arm64/arm64/gic_acpi.c optional acpi arm64/arm64/gic_fdt.c optional fdt arm64/arm64/gic_v3.c standard arm64/arm64/gic_v3_fdt.c optional fdt arm64/arm64/gic_v3_its.c standard arm64/arm64/identcpu.c standard arm64/arm64/intr_machdep.c standard arm64/arm64/in_cksum.c optional inet | inet6 arm64/arm64/locore.S standard no-obj arm64/arm64/machdep.c standard arm64/arm64/mem.c standard arm64/arm64/minidump_machdep.c standard arm64/arm64/mp_machdep.c optional smp arm64/arm64/nexus.c standard arm64/arm64/ofw_machdep.c optional fdt arm64/arm64/pic_if.m standard arm64/arm64/pmap.c standard arm64/arm64/stack_machdep.c optional ddb | stack arm64/arm64/support.S standard arm64/arm64/swtch.S standard arm64/arm64/sys_machdep.c standard arm64/arm64/trap.c standard arm64/arm64/uio_machdep.c standard arm64/arm64/uma_machdep.c standard arm64/arm64/unwind.c optional ddb | kdtrace_hooks | stack arm64/arm64/vfp.c standard arm64/arm64/vm_machdep.c standard arm64/cavium/thunder_pcie.c optional soc_cavm_thunderx pci arm64/cavium/thunder_pcie_fdt.c optional soc_cavm_thunderx pci fdt arm64/cavium/thunder_pcie_pem.c optional soc_cavm_thunderx pci arm64/cavium/thunder_pcie_common.c optional soc_cavm_thunderx pci arm64/cloudabi64/cloudabi64_sysvec.c optional compat_cloudabi64 crypto/blowfish/bf_enc.c optional crypto | ipsec crypto/des/des_enc.c optional crypto | ipsec | netsmb dev/acpica/acpi_if.m optional acpi dev/ahci/ahci_generic.c optional ahci fdt dev/hwpmc/hwpmc_arm64.c optional hwpmc dev/hwpmc/hwpmc_arm64_md.c optional hwpmc dev/mmc/host/dwmmc.c optional dwmmc dev/mmc/host/dwmmc_hisi.c optional dwmmc soc_hisi_hi6220 dev/ofw/ofw_cpu.c optional fdt dev/pci/pci_host_generic.c optional pci fdt dev/psci/psci.c optional psci dev/psci/psci_arm64.S optional psci dev/uart/uart_cpu_fdt.c optional uart fdt dev/uart/uart_dev_pl011.c optional uart pl011 dev/usb/controller/dwc_otg_hisi.c optional dwcotg soc_hisi_hi6220 dev/vnic/mrml_bridge.c optional vnic fdt dev/vnic/nic_main.c optional vnic pci dev/vnic/nicvf_main.c optional vnic pci pci_iov dev/vnic/nicvf_queues.c optional vnic pci pci_iov dev/vnic/thunder_bgx_fdt.c optional vnic fdt dev/vnic/thunder_bgx.c optional vnic pci dev/vnic/thunder_mdio_fdt.c optional vnic fdt dev/vnic/thunder_mdio.c optional vnic dev/vnic/lmac_if.m optional vnic kern/kern_clocksource.c standard libkern/bcmp.c standard libkern/ffs.c standard libkern/ffsl.c standard libkern/ffsll.c standard libkern/fls.c standard libkern/flsl.c standard libkern/flsll.c standard libkern/memmove.c standard libkern/memset.c standard cddl/contrib/opensolaris/common/atomic/aarch64/opensolaris_atomic.S optional zfs | dtrace compile-with "${CDDL_C}" cddl/dev/dtrace/aarch64/dtrace_asm.S optional dtrace compile-with "${DTRACE_S}" cddl/dev/dtrace/aarch64/dtrace_subr.c optional dtrace compile-with "${DTRACE_C}" cddl/dev/fbt/aarch64/fbt_isa.c optional dtrace_fbt | dtraceall compile-with "${FBT_C}" Index: user/ngie/make_check/sys/conf/files.riscv =================================================================== --- user/ngie/make_check/sys/conf/files.riscv (nonexistent) +++ user/ngie/make_check/sys/conf/files.riscv (revision 295078) @@ -0,0 +1,44 @@ +# $FreeBSD$ +crypto/blowfish/bf_enc.c optional crypto | ipsec +crypto/des/des_enc.c optional crypto | ipsec | netsmb +kern/kern_clocksource.c standard +kern/subr_dummy_vdso_tc.c standard +libkern/bcmp.c standard +libkern/ffs.c standard +libkern/ffsl.c standard +libkern/fls.c standard +libkern/flsl.c standard +libkern/flsll.c standard +libkern/memmove.c standard +libkern/memset.c standard +riscv/htif/htif.c standard +riscv/htif/htif_block.c standard +riscv/htif/htif_console.c standard +riscv/riscv/autoconf.c standard +riscv/riscv/bcopy.c standard +riscv/riscv/bus_machdep.c standard +riscv/riscv/busdma_machdep.c standard +riscv/riscv/clock.c standard +riscv/riscv/copyinout.S standard +riscv/riscv/copystr.c standard +riscv/riscv/cpufunc_asm.S standard +riscv/riscv/devmap.c standard +riscv/riscv/dump_machdep.c standard +riscv/riscv/elf_machdep.c standard +riscv/riscv/intr_machdep.c standard +riscv/riscv/in_cksum.c optional inet | inet6 +riscv/riscv/identcpu.c standard +riscv/riscv/locore.S standard no-obj +riscv/riscv/minidump_machdep.c standard +riscv/riscv/machdep.c standard +riscv/riscv/mem.c standard +riscv/riscv/nexus.c standard +riscv/riscv/pmap.c standard +riscv/riscv/sys_machdep.c standard +riscv/riscv/support.S standard +riscv/riscv/swtch.S standard +riscv/riscv/trap.c standard +riscv/riscv/timer.c standard +riscv/riscv/uio_machdep.c standard +riscv/riscv/uma_machdep.c standard +riscv/riscv/vm_machdep.c standard Property changes on: user/ngie/make_check/sys/conf/files.riscv ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/conf/kern.mk =================================================================== --- user/ngie/make_check/sys/conf/kern.mk (revision 295077) +++ user/ngie/make_check/sys/conf/kern.mk (revision 295078) @@ -1,229 +1,233 @@ # $FreeBSD$ # # Warning flags for compiling the kernel and components of the kernel: # CWARNFLAGS?= -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes \ -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual \ -Wundef -Wno-pointer-sign ${FORMAT_EXTENSIONS} \ -Wmissing-include-dirs -fdiagnostics-show-option \ -Wno-unknown-pragmas \ ${CWARNEXTRA} # # The following flags are next up for working on: # -Wextra # Disable a few warnings for clang, since there are several places in the # kernel where fixing them is more trouble than it is worth, or where there is # a false positive. .if ${COMPILER_TYPE} == "clang" NO_WCONSTANT_CONVERSION= -Wno-constant-conversion NO_WSHIFT_COUNT_NEGATIVE= -Wno-shift-count-negative NO_WSHIFT_COUNT_OVERFLOW= -Wno-shift-count-overflow NO_WSELF_ASSIGN= -Wno-self-assign NO_WUNNEEDED_INTERNAL_DECL= -Wno-unneeded-internal-declaration NO_WSOMETIMES_UNINITIALIZED= -Wno-error-sometimes-uninitialized NO_WCAST_QUAL= -Wno-cast-qual # Several other warnings which might be useful in some cases, but not severe # enough to error out the whole kernel build. Display them anyway, so there is # some incentive to fix them eventually. CWARNEXTRA?= -Wno-error-tautological-compare -Wno-error-empty-body \ -Wno-error-parentheses-equality -Wno-error-unused-function \ -Wno-error-pointer-sign .if ${COMPILER_VERSION} >= 30700 CWARNEXTRA+= -Wno-error-shift-negative-value .endif CLANG_NO_IAS= -no-integrated-as .if ${COMPILER_VERSION} < 30500 # XXX: clang < 3.5 integrated-as doesn't grok .codeNN directives CLANG_NO_IAS34= -no-integrated-as .endif .endif .if ${COMPILER_TYPE} == "gcc" .if ${COMPILER_VERSION} >= 40300 # Catch-all for all the things that are in our tree, but for which we're # not yet ready for this compiler. Note: we likely only really "support" # building with gcc 4.8 and newer. Nothing older has been tested. CWARNEXTRA?= -Wno-error=inline -Wno-error=enum-compare -Wno-error=unused-but-set-variable \ -Wno-error=aggressive-loop-optimizations -Wno-error=maybe-uninitialized \ -Wno-error=array-bounds -Wno-error=address \ -Wno-error=cast-qual -Wno-error=sequence-point -Wno-error=attributes \ -Wno-error=strict-overflow -Wno-error=overflow .else # For gcc 4.2, eliminate the too-often-wrong warnings about uninitialized vars. CWARNEXTRA?= -Wno-uninitialized .endif .endif # External compilers may not support our format extensions. Allow them # to be disabled. WARNING: format checking is disabled in this case. .if ${MK_FORMAT_EXTENSIONS} == "no" FORMAT_EXTENSIONS= -Wno-format .elif ${COMPILER_TYPE} == "clang" && ${COMPILER_VERSION} >= 30600 FORMAT_EXTENSIONS= -D__printf__=__freebsd_kprintf__ .else FORMAT_EXTENSIONS= -fformat-extensions .endif # # On i386, do not align the stack to 16-byte boundaries. Otherwise GCC 2.95 # and above adds code to the entry and exit point of every function to align the # stack to 16-byte boundaries -- thus wasting approximately 12 bytes of stack # per function call. While the 16-byte alignment may benefit micro benchmarks, # it is probably an overall loss as it makes the code bigger (less efficient # use of code cache tag lines) and uses more stack (less efficient use of data # cache tag lines). Explicitly prohibit the use of FPU, SSE and other SIMD # operations inside the kernel itself. These operations are exclusively # reserved for user applications. # # gcc: # Setting -mno-mmx implies -mno-3dnow # Setting -mno-sse implies -mno-sse2, -mno-sse3 and -mno-ssse3 # # clang: # Setting -mno-mmx implies -mno-3dnow and -mno-3dnowa # Setting -mno-sse implies -mno-sse2, -mno-sse3, -mno-ssse3, -mno-sse41 and -mno-sse42 # .if ${MACHINE_CPUARCH} == "i386" CFLAGS.gcc+= -mno-align-long-strings -mpreferred-stack-boundary=2 CFLAGS.clang+= -mno-aes -mno-avx CFLAGS+= -mno-mmx -mno-sse -msoft-float INLINE_LIMIT?= 8000 .endif .if ${MACHINE_CPUARCH} == "arm" INLINE_LIMIT?= 8000 .endif .if ${MACHINE_CPUARCH} == "aarch64" # We generally don't want fpu instructions in the kernel. CFLAGS += -mgeneral-regs-only # Reserve x18 for pcpu data CFLAGS += -ffixed-x18 .endif +.if ${MACHINE_CPUARCH} == "riscv" +INLINE_LIMIT?= 8000 +.endif + # # For sparc64 we want the medany code model so modules may be located # anywhere in the 64-bit address space. We also tell GCC to use floating # point emulation. This avoids using floating point registers for integer # operations which it has a tendency to do. # .if ${MACHINE_CPUARCH} == "sparc64" CFLAGS.clang+= -mcmodel=large -fno-dwarf2-cfi-asm CFLAGS.gcc+= -mcmodel=medany -msoft-float INLINE_LIMIT?= 15000 .endif # # For AMD64, we explicitly prohibit the use of FPU, SSE and other SIMD # operations inside the kernel itself. These operations are exclusively # reserved for user applications. # # gcc: # Setting -mno-mmx implies -mno-3dnow # Setting -mno-sse implies -mno-sse2, -mno-sse3, -mno-ssse3 and -mfpmath=387 # # clang: # Setting -mno-mmx implies -mno-3dnow and -mno-3dnowa # Setting -mno-sse implies -mno-sse2, -mno-sse3, -mno-ssse3, -mno-sse41 and -mno-sse42 # (-mfpmath= is not supported) # .if ${MACHINE_CPUARCH} == "amd64" CFLAGS.clang+= -mno-aes -mno-avx CFLAGS+= -mcmodel=kernel -mno-red-zone -mno-mmx -mno-sse -msoft-float \ -fno-asynchronous-unwind-tables INLINE_LIMIT?= 8000 .endif # # For PowerPC we tell gcc to use floating point emulation. This avoids using # floating point registers for integer operations which it has a tendency to do. # Also explicitly disable Altivec instructions inside the kernel. # .if ${MACHINE_CPUARCH} == "powerpc" CFLAGS+= -mno-altivec CFLAGS.clang+= -mllvm -disable-ppc-float-in-variadic=true CFLAGS.gcc+= -msoft-float INLINE_LIMIT?= 15000 .endif # # Use dot symbols on powerpc64 to make ddb happy # .if ${MACHINE_ARCH} == "powerpc64" CFLAGS.gcc+= -mcall-aixdesc .endif # # For MIPS we also tell gcc to use floating point emulation # .if ${MACHINE_CPUARCH} == "mips" CFLAGS+= -msoft-float INLINE_LIMIT?= 8000 .endif # # GCC 3.0 and above like to do certain optimizations based on the # assumption that the program is linked against libc. Stop this. # CFLAGS+= -ffreestanding # # The C standard leaves signed integer overflow behavior undefined. # gcc and clang opimizers take advantage of this. The kernel makes # use of signed integer wraparound mechanics so we need the compiler # to treat it as a wraparound and not take shortcuts. # CFLAGS+= -fwrapv # # GCC SSP support # .if ${MK_SSP} != "no" && \ ${MACHINE_CPUARCH} != "arm" && ${MACHINE_CPUARCH} != "mips" CFLAGS+= -fstack-protector .endif # # Add -gdwarf-2 when compiling -g. The default starting in clang v3.4 # and gcc 4.8 is to generate DWARF version 4. However, our tools don't # cope well with DWARF 4, so force it to genereate DWARF2, which they # understand. Do this unconditionally as it is harmless when not needed, # but critical for these newer versions. # .if ${CFLAGS:M-g} != "" && ${CFLAGS:M-gdwarf*} == "" CFLAGS+= -gdwarf-2 .endif CFLAGS+= ${CWARNFLAGS} ${CWARNFLAGS.${.IMPSRC:T}} CFLAGS+= ${CFLAGS.${COMPILER_TYPE}} ${CFLAGS.${.IMPSRC:T}} # Tell bmake not to mistake standard targets for things to be searched for # or expect to ever be up-to-date. PHONY_NOTMAIN = afterdepend afterinstall all beforedepend beforeinstall \ beforelinking build build-tools buildfiles buildincludes \ checkdpadd clean cleandepend cleandir cleanobj configure \ depend dependall distclean distribute exe \ html includes install installfiles installincludes lint \ obj objlink objs objwarn realall realdepend \ realinstall regress subdir-all subdir-depend subdir-install \ tags whereobj .PHONY: ${PHONY_NOTMAIN} .NOTMAIN: ${PHONY_NOTMAIN} CSTD= c99 .if ${CSTD} == "k&r" CFLAGS+= -traditional .elif ${CSTD} == "c89" || ${CSTD} == "c90" CFLAGS+= -std=iso9899:1990 .elif ${CSTD} == "c94" || ${CSTD} == "c95" CFLAGS+= -std=iso9899:199409 .elif ${CSTD} == "c99" CFLAGS+= -std=iso9899:1999 .else # CSTD CFLAGS+= -std=${CSTD} .endif # CSTD Index: user/ngie/make_check/sys/conf/kern.pre.mk =================================================================== --- user/ngie/make_check/sys/conf/kern.pre.mk (revision 295077) +++ user/ngie/make_check/sys/conf/kern.pre.mk (revision 295078) @@ -1,260 +1,261 @@ # $FreeBSD$ # Part of a unified Makefile for building kernels. This part contains all # of the definitions that need to be before %BEFORE_DEPEND. # Allow user to configure things that only effect src tree builds. # Note: This is duplicated from src.sys.mk to ensure that we include # /etc/src.conf when building the kernel. Kernels can be built without # the rest of /usr/src, but they still always process SRCCONF even though # the normal mechanisms to prevent that (compiling out of tree) won't # work. To ensure they do work, we have to duplicate thee few lines here. SRCCONF?= /etc/src.conf .if (exists(${SRCCONF}) || ${SRCCONF} != "/etc/src.conf") && !target(_srcconf_included_) .include "${SRCCONF}" _srcconf_included_: .endif .include .include .include "kern.opts.mk" # Can be overridden by makeoptions or /etc/make.conf KERNEL_KO?= kernel KERNEL?= kernel KODIR?= /boot/${KERNEL} LDSCRIPT_NAME?= ldscript.$M LDSCRIPT?= $S/conf/${LDSCRIPT_NAME} M= ${MACHINE} AWK?= awk CP?= cp LINT?= lint NM?= nm OBJCOPY?= objcopy SIZE?= size .if defined(DEBUG) _MINUS_O= -O CTFFLAGS+= -g .else .if ${MACHINE_CPUARCH} == "powerpc" _MINUS_O= -O # gcc miscompiles some code at -O2 .else _MINUS_O= -O2 .endif .endif .if ${MACHINE_CPUARCH} == "amd64" .if ${COMPILER_TYPE} == "clang" COPTFLAGS?=-O2 -pipe .else COPTFLAGS?=-O2 -frename-registers -pipe .endif .else COPTFLAGS?=${_MINUS_O} -pipe .endif .if !empty(COPTFLAGS:M-O[23s]) && empty(COPTFLAGS:M-fno-strict-aliasing) COPTFLAGS+= -fno-strict-aliasing .endif .if !defined(NO_CPU_COPTFLAGS) COPTFLAGS+= ${_CPUCFLAGS} .endif NOSTDINC= -nostdinc INCLUDES= ${NOSTDINC} ${INCLMAGIC} -I. -I$S .if ${MK_FAST_DEPEND} == "no" && (make(depend) || make(kernel-depend)) # This hack lets us use the ipfilter code without spamming a new # include path into contrib'ed source files. INCLUDES+= -I$S/contrib/ipfilter # ... and the same for ath INCLUDES+= -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath_hal # ... and the same for the NgATM stuff INCLUDES+= -I$S/contrib/ngatm # ... and the same for vchiq INCLUDES+= -I$S/contrib/vchiq # ... and the same for twa INCLUDES+= -I$S/dev/twa # ... and the same for cxgb and cxgbe INCLUDES+= -I$S/dev/cxgb -I$S/dev/cxgbe .endif CFLAGS= ${COPTFLAGS} ${DEBUG} CFLAGS+= ${INCLUDES} -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h CFLAGS_PARAM_INLINE_UNIT_GROWTH?=100 CFLAGS_PARAM_LARGE_FUNCTION_GROWTH?=1000 .if ${MACHINE_CPUARCH} == "mips" CFLAGS_ARCH_PARAMS?=--param max-inline-insns-single=1000 .endif CFLAGS.gcc+= -fno-common -fms-extensions -finline-limit=${INLINE_LIMIT} CFLAGS.gcc+= --param inline-unit-growth=${CFLAGS_PARAM_INLINE_UNIT_GROWTH} CFLAGS.gcc+= --param large-function-growth=${CFLAGS_PARAM_LARGE_FUNCTION_GROWTH} .if defined(CFLAGS_ARCH_PARAMS) CFLAGS.gcc+=${CFLAGS_ARCH_PARAMS} .endif WERROR?= -Werror # XXX LOCORE means "don't declare C stuff" not "for locore.s". ASM_CFLAGS= -x assembler-with-cpp -DLOCORE ${CFLAGS} ${ASM_CFLAGS.${.IMPSRC:T}} .if defined(PROFLEVEL) && ${PROFLEVEL} >= 1 CFLAGS+= -DGPROF CFLAGS.gcc+= -falign-functions=16 .if ${PROFLEVEL} >= 2 CFLAGS+= -DGPROF4 -DGUPROF PROF= -pg .if ${COMPILER_TYPE} == "gcc" PROF+= -mprofiler-epilogue .endif .else PROF= -pg .endif .endif DEFINED_PROF= ${PROF} # Put configuration-specific C flags last (except for ${PROF}) so that they # can override the others. CFLAGS+= ${CONF_CFLAGS} # Optional linting. This can be overridden in /etc/make.conf. LINTFLAGS= ${LINTOBJKERNFLAGS} NORMAL_C= ${CC} -c ${CFLAGS} ${WERROR} ${PROF} ${.IMPSRC} NORMAL_S= ${CC:N${CCACHE_BIN}} -c ${ASM_CFLAGS} ${WERROR} ${.IMPSRC} PROFILE_C= ${CC} -c ${CFLAGS} ${WERROR} ${.IMPSRC} NORMAL_C_NOWERROR= ${CC} -c ${CFLAGS} ${PROF} ${.IMPSRC} NORMAL_M= ${AWK} -f $S/tools/makeobjops.awk ${.IMPSRC} -c ; \ ${CC} -c ${CFLAGS} ${WERROR} ${PROF} ${.PREFIX}.c NORMAL_FW= uudecode -o ${.TARGET} ${.ALLSRC} NORMAL_FWO= ${LD} -b binary --no-warn-mismatch -d -warn-common -r \ -o ${.TARGET} ${.ALLSRC:M*.fw} # Common for dtrace / zfs CDDL_CFLAGS= -DFREEBSD_NAMECACHE -nostdinc -I$S/cddl/compat/opensolaris -I$S/cddl/contrib/opensolaris/uts/common -I$S -I$S/cddl/contrib/opensolaris/common ${CFLAGS} -Wno-unknown-pragmas -Wno-missing-prototypes -Wno-undef -Wno-strict-prototypes -Wno-cast-qual -Wno-parentheses -Wno-redundant-decls -Wno-missing-braces -Wno-uninitialized -Wno-unused -Wno-inline -Wno-switch -Wno-pointer-arith -Wno-unknown-pragmas CDDL_CFLAGS+= -include $S/cddl/compat/opensolaris/sys/debug_compat.h CDDL_C= ${CC} -c ${CDDL_CFLAGS} ${WERROR} ${PROF} ${.IMPSRC} # Special flags for managing the compat compiles for ZFS ZFS_CFLAGS= -DBUILDING_ZFS -I$S/cddl/contrib/opensolaris/uts/common/fs/zfs -I$S/cddl/contrib/opensolaris/uts/common/zmod -I$S/cddl/contrib/opensolaris/common/zfs ${CDDL_CFLAGS} ZFS_ASM_CFLAGS= -x assembler-with-cpp -DLOCORE ${ZFS_CFLAGS} ZFS_C= ${CC} -c ${ZFS_CFLAGS} ${WERROR} ${PROF} ${.IMPSRC} ZFS_S= ${CC} -c ${ZFS_ASM_CFLAGS} ${WERROR} ${.IMPSRC} # Special flags for managing the compat compiles for DTrace DTRACE_CFLAGS= -DBUILDING_DTRACE ${CDDL_CFLAGS} -I$S/cddl/dev/dtrace -I$S/cddl/dev/dtrace/${MACHINE_CPUARCH} .if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386" DTRACE_CFLAGS+= -I$S/cddl/contrib/opensolaris/uts/intel -I$S/cddl/dev/dtrace/x86 .endif DTRACE_CFLAGS+= -I$S/cddl/contrib/opensolaris/common/util -I$S -DDIS_MEM -DSMP DTRACE_ASM_CFLAGS= -x assembler-with-cpp -DLOCORE ${DTRACE_CFLAGS} DTRACE_C= ${CC} -c ${DTRACE_CFLAGS} ${WERROR} ${PROF} ${.IMPSRC} DTRACE_S= ${CC} -c ${DTRACE_ASM_CFLAGS} ${WERROR} ${.IMPSRC} # Special flags for managing the compat compiles for DTrace/FBT FBT_CFLAGS= -DBUILDING_DTRACE -nostdinc -I$S/cddl/dev/fbt/${MACHINE_CPUARCH} -I$S/cddl/dev/fbt -I$S/cddl/compat/opensolaris -I$S/cddl/contrib/opensolaris/uts/common -I$S ${CDDL_CFLAGS} .if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386" FBT_CFLAGS+= -I$S/cddl/dev/fbt/x86 .endif FBT_C= ${CC} -c ${FBT_CFLAGS} ${WERROR} ${PROF} ${.IMPSRC} .if ${MK_CTF} != "no" NORMAL_CTFCONVERT= ${CTFCONVERT} ${CTFFLAGS} ${.TARGET} .elif ${MAKE_VERSION} >= 5201111300 NORMAL_CTFCONVERT= .else NORMAL_CTFCONVERT= @: .endif NORMAL_LINT= ${LINT} ${LINTFLAGS} ${CFLAGS:M-[DIU]*} ${.IMPSRC} # Linux Kernel Programming Interface C-flags LINUXKPI_INCLUDES= -I$S/compat/linuxkpi/common/include LINUXKPI_C= ${NORMAL_C} ${LINUXKPI_INCLUDES} # Infiniband C flags. Correct include paths and omit errors that linux # does not honor. OFEDINCLUDES= -I$S/ofed/include ${LINUXKPI_INCLUDES} OFEDNOERR= -Wno-cast-qual -Wno-pointer-arith OFEDCFLAGS= ${CFLAGS:N-I*} ${OFEDINCLUDES} ${CFLAGS:M-I*} ${OFEDNOERR} OFED_C_NOIMP= ${CC} -c -o ${.TARGET} ${OFEDCFLAGS} ${WERROR} ${PROF} OFED_C= ${OFED_C_NOIMP} ${.IMPSRC} GEN_CFILES= $S/$M/$M/genassym.c ${MFILES:T:S/.m$/.c/} SYSTEM_CFILES= config.c env.c hints.c vnode_if.c SYSTEM_DEP= Makefile ${SYSTEM_OBJS} SYSTEM_OBJS= locore.o ${MDOBJS} ${OBJS} SYSTEM_OBJS+= ${SYSTEM_CFILES:.c=.o} SYSTEM_OBJS+= hack.So .if ${MFS_IMAGE:Uno} != "no" SYSTEM_OBJS+= embedfs_${MFS_IMAGE:T:R}.o .endif SYSTEM_LD= @${LD} -Bdynamic -T ${LDSCRIPT} ${_LDFLAGS} --no-warn-mismatch \ --warn-common --export-dynamic --dynamic-linker /red/herring \ -o ${.TARGET} -X ${SYSTEM_OBJS} vers.o SYSTEM_LD_TAIL= @${OBJCOPY} --strip-symbol gcc2_compiled. ${.TARGET} ; \ ${SIZE} ${.TARGET} ; chmod 755 ${.TARGET} SYSTEM_DEP+= ${LDSCRIPT} # Calculate path for .m files early, if needed. .if !defined(__MPATH) __MPATH!=find ${S:tA}/ -name \*_if.m .endif # MKMODULESENV is set here so that port makefiles can augment # them. MKMODULESENV+= MAKEOBJDIRPREFIX=${.OBJDIR}/modules KMODDIR=${KODIR} MKMODULESENV+= MACHINE_CPUARCH=${MACHINE_CPUARCH} MKMODULESENV+= MACHINE=${MACHINE} MACHINE_ARCH=${MACHINE_ARCH} MKMODULESENV+= MODULES_EXTRA="${MODULES_EXTRA}" WITHOUT_MODULES="${WITHOUT_MODULES}" .if (${KERN_IDENT} == LINT) MKMODULESENV+= ALL_MODULES=LINT .endif .if defined(MODULES_OVERRIDE) MKMODULESENV+= MODULES_OVERRIDE="${MODULES_OVERRIDE}" .endif .if defined(DEBUG) MKMODULESENV+= DEBUG_FLAGS="${DEBUG}" .endif MKMODULESENV+= __MPATH="${__MPATH}" # Architecture and output format arguments for objdump to convert image to # object file .if ${MFS_IMAGE:Uno} != "no" .if !defined(EMBEDFS_FORMAT.${MACHINE_ARCH}) EMBEDFS_FORMAT.${MACHINE_ARCH}!= awk -F'"' '/OUTPUT_FORMAT/ {print $$2}' ${LDSCRIPT} .if empty(EMBEDFS_FORMAT.${MACHINE_ARCH}) .undef EMBEDFS_FORMAT.${MACHINE_ARCH} .endif .endif .if !defined(EMBEDFS_ARCH.${MACHINE_ARCH}) EMBEDFS_ARCH.${MACHINE_ARCH}!= sed -n '/OUTPUT_ARCH/s/.*(\(.*\)).*/\1/p' ${LDSCRIPT} .if empty(EMBEDFS_ARCH.${MACHINE_ARCH}) .undef EMBEDFS_ARCH.${MACHINE_ARCH} .endif .endif EMBEDFS_FORMAT.arm?= elf32-littlearm EMBEDFS_FORMAT.armv6?= elf32-littlearm EMBEDFS_FORMAT.mips?= elf32-tradbigmips EMBEDFS_FORMAT.mipsel?= elf32-tradlittlemips EMBEDFS_FORMAT.mips64?= elf64-tradbigmips EMBEDFS_FORMAT.mips64el?= elf64-tradlittlemips +EMBEDFS_FORMAT.riscv?= elf64-littleriscv .endif # Detect kernel config options that force stack frames to be turned on. DDB_ENABLED!= grep DDB opt_ddb.h || true ; echo DTR_ENABLED!= grep KDTRACE_FRAME opt_kdtrace.h || true ; echo HWPMC_ENABLED!= grep HWPMC opt_hwpmc_hooks.h || true ; echo Index: user/ngie/make_check/sys/conf/ldscript.riscv =================================================================== --- user/ngie/make_check/sys/conf/ldscript.riscv (nonexistent) +++ user/ngie/make_check/sys/conf/ldscript.riscv (revision 295078) @@ -0,0 +1,136 @@ +/* $FreeBSD$ */ +OUTPUT_ARCH(riscv) +ENTRY(_start) + +SEARCH_DIR(/usr/lib); +SECTIONS +{ + /* Read-only sections, merged into text segment: */ + . = kernbase + 0x100; + .text : AT(ADDR(.text) - kernbase) + { + *(.text) + *(.stub) + /* .gnu.warning sections are handled specially by elf32.em. */ + *(.gnu.warning) + *(.gnu.linkonce.t*) + } =0x9090 + _etext = .; + PROVIDE (etext = .); + .fini : { *(.fini) } =0x9090 + .rodata : { *(.rodata) *(.gnu.linkonce.r*) } + .rodata1 : { *(.rodata1) } + .interp : { *(.interp) } + .hash : { *(.hash) } + .dynsym : { *(.dynsym) } + .dynstr : { *(.dynstr) } + .gnu.version : { *(.gnu.version) } + .gnu.version_d : { *(.gnu.version_d) } + .gnu.version_r : { *(.gnu.version_r) } + .rel.text : + { *(.rel.text) *(.rel.gnu.linkonce.t*) } + .rela.text : + { *(.rela.text) *(.rela.gnu.linkonce.t*) } + .rel.data : + { *(.rel.data) *(.rel.gnu.linkonce.d*) } + .rela.data : + { *(.rela.data) *(.rela.gnu.linkonce.d*) } + .rel.rodata : + { *(.rel.rodata) *(.rel.gnu.linkonce.r*) } + .rela.rodata : + { *(.rela.rodata) *(.rela.gnu.linkonce.r*) } + .rel.got : { *(.rel.got) } + .rela.got : { *(.rela.got) } + .rel.ctors : { *(.rel.ctors) } + .rela.ctors : { *(.rela.ctors) } + .rel.dtors : { *(.rel.dtors) } + .rela.dtors : { *(.rela.dtors) } + .rel.init : { *(.rel.init) } + .rela.init : { *(.rela.init) } + .rel.fini : { *(.rel.fini) } + .rela.fini : { *(.rela.fini) } + .rel.bss : { *(.rel.bss) } + .rela.bss : { *(.rela.bss) } + .rel.plt : { *(.rel.plt) } + .rela.plt : { *(.rela.plt) } + .init : { *(.init) } =0x9090 + .plt : { *(.plt) } + + /* Adjust the address for the data segment. We want to adjust up to + the same address within the page on the next page up. */ + . = ALIGN(0x1000) + (. & (0x1000 - 1)) ; + .data : + { + *(.data) + *(.gnu.linkonce.d*) + CONSTRUCTORS + } + .data1 : { *(.data1) } + . = ALIGN(32 / 8); + _start_ctors = .; + PROVIDE (start_ctors = .); + .ctors : + { + *(.ctors) + } + _stop_ctors = .; + PROVIDE (stop_ctors = .); + .dtors : + { + *(.dtors) + } + .got : { *(.got.plt) *(.got) } + .dynamic : { *(.dynamic) } + /* We want the small data sections together, so single-instruction offsets + can access them all, and initialized data all before uninitialized, so + we can shorten the on-disk segment size. */ + . = ALIGN(8); + .sdata : { *(.sdata) } + _edata = .; + PROVIDE (edata = .); + __bss_start = .; + .sbss : { *(.sbss) *(.scommon) } + .bss : + { + *(.dynbss) + *(.bss) + *(COMMON) + } + . = ALIGN(8); + _end = . ; + PROVIDE (end = .); + /* Stabs debugging sections. */ + .stab 0 : { *(.stab) } + .stabstr 0 : { *(.stabstr) } + .stab.excl 0 : { *(.stab.excl) } + .stab.exclstr 0 : { *(.stab.exclstr) } + .stab.index 0 : { *(.stab.index) } + .stab.indexstr 0 : { *(.stab.indexstr) } + .comment 0 : { *(.comment) } + /* DWARF debug sections. + Symbols in the DWARF debugging sections are relative to the beginning + of the section so we begin them at 0. */ + /* DWARF 1 */ + .debug 0 : { *(.debug) } + .line 0 : { *(.line) } + /* GNU DWARF 1 extensions */ + .debug_srcinfo 0 : { *(.debug_srcinfo) } + .debug_sfnames 0 : { *(.debug_sfnames) } + /* DWARF 1.1 and DWARF 2 */ + .debug_aranges 0 : { *(.debug_aranges) } + .debug_pubnames 0 : { *(.debug_pubnames) } + /* DWARF 2 */ + .debug_info 0 : { *(.debug_info) } + .debug_abbrev 0 : { *(.debug_abbrev) } + .debug_line 0 : { *(.debug_line) } + .debug_frame 0 : { *(.debug_frame) } + .debug_str 0 : { *(.debug_str) } + .debug_loc 0 : { *(.debug_loc) } + .debug_macinfo 0 : { *(.debug_macinfo) } + /* SGI/MIPS DWARF 2 extensions */ + .debug_weaknames 0 : { *(.debug_weaknames) } + .debug_funcnames 0 : { *(.debug_funcnames) } + .debug_typenames 0 : { *(.debug_typenames) } + .debug_varnames 0 : { *(.debug_varnames) } + /* These must appear regardless of . */ +} Property changes on: user/ngie/make_check/sys/conf/ldscript.riscv ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/make_check/sys/conf/options.arm =================================================================== --- user/ngie/make_check/sys/conf/options.arm (revision 295077) +++ user/ngie/make_check/sys/conf/options.arm (revision 295078) @@ -1,74 +1,73 @@ #$FreeBSD$ ARMV6 opt_global.h ARM_CACHE_LOCK_ENABLE opt_global.h ARM_INTRNG opt_global.h ARM_KERN_DIRECTMAP opt_vm.h ARM_L2_PIPT opt_global.h ARM_MANY_BOARD opt_global.h -ARM_NEW_PMAP opt_global.h NKPT2PG opt_pmap.h ARM_WANT_TP_ADDRESS opt_global.h COUNTS_PER_SEC opt_timer.h CPU_ARM9 opt_global.h CPU_ARM9E opt_global.h CPU_ARM1176 opt_global.h CPU_CORTEXA opt_global.h CPU_KRAIT opt_global.h CPU_FA526 opt_global.h CPU_MV_PJ4B opt_global.h CPU_XSCALE_80219 opt_global.h CPU_XSCALE_80321 opt_global.h CPU_XSCALE_81342 opt_global.h CPU_XSCALE_IXP425 opt_global.h CPU_XSCALE_IXP435 opt_global.h CPU_XSCALE_PXA2X0 opt_global.h DEV_GIC opt_global.h DEV_PMU opt_global.h EFI opt_platform.h FLASHADDR opt_global.h GIC_DEFAULT_ICFGR_INIT opt_global.h IPI_IRQ_START opt_smp.h IPI_IRQ_END opt_smp.h FREEBSD_BOOT_LOADER opt_global.h IXP4XX_FLASH_SIZE opt_global.h KERNBASE opt_global.h KERNVIRTADDR opt_global.h LINUX_BOOT_ABI opt_global.h LOADERRAMADDR opt_global.h PHYSADDR opt_global.h PLATFORM opt_global.h SOCDEV_PA opt_global.h SOCDEV_VA opt_global.h PV_STATS opt_pmap.h QEMU_WORKAROUNDS opt_global.h SOC_BCM2835 opt_global.h SOC_BCM2836 opt_global.h SOC_IMX51 opt_global.h SOC_IMX53 opt_global.h SOC_IMX6 opt_global.h SOC_MV_ARMADAXP opt_global.h SOC_MV_ARMADA38X opt_global.h SOC_MV_DISCOVERY opt_global.h SOC_MV_DOVE opt_global.h SOC_MV_FREY opt_global.h SOC_MV_KIRKWOOD opt_global.h SOC_MV_LOKIPLUS opt_global.h SOC_MV_ORION opt_global.h SOC_OMAP3 opt_global.h SOC_OMAP4 opt_global.h SOC_TI_AM335X opt_global.h SOC_TEGRA2 opt_global.h XSCALE_CACHE_READ_WRITE_ALLOCATE opt_global.h XSACLE_DISABLE_CCNT opt_timer.h VERBOSE_INIT_ARM opt_global.h VM_MAXUSER_ADDRESS opt_global.h AT91_ATE_USE_RMII opt_at91.h AT91_MACB_USE_RMII opt_at91.h AT91_MCI_ALLOW_OVERCLOCK opt_at91.h AT91_MCI_HAS_4WIRE opt_at91.h AT91_MCI_SLOT_B opt_at91.h GFB_DEBUG opt_gfb.h GFB_NO_FONT_LOADING opt_gfb.h GFB_NO_MODE_CHANGE opt_gfb.h AT91C_MAIN_CLOCK opt_at91.h VFP opt_global.h Index: user/ngie/make_check/sys/conf/options.riscv =================================================================== --- user/ngie/make_check/sys/conf/options.riscv (nonexistent) +++ user/ngie/make_check/sys/conf/options.riscv (revision 295078) @@ -0,0 +1,4 @@ +# $FreeBSD$ + +RISCV opt_global.h +VFP opt_global.h Property changes on: user/ngie/make_check/sys/conf/options.riscv ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/make_check/sys/conf =================================================================== --- user/ngie/make_check/sys/conf (revision 295077) +++ user/ngie/make_check/sys/conf (revision 295078) Property changes on: user/ngie/make_check/sys/conf ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/conf:r295034-295077 Index: user/ngie/make_check/sys/dev/hwpmc/hwpmc_riscv.h =================================================================== --- user/ngie/make_check/sys/dev/hwpmc/hwpmc_riscv.h (nonexistent) +++ user/ngie/make_check/sys/dev/hwpmc/hwpmc_riscv.h (revision 295078) @@ -0,0 +1,51 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * This software was developed by the University of Cambridge Computer + * Laboratory with support from ARM Ltd. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _DEV_HWPMC_RISCV_H_ +#define _DEV_HWPMC_RISCV_H_ + +#define RISCV_PMC_CAPS (PMC_CAP_INTERRUPT | PMC_CAP_USER | \ + PMC_CAP_SYSTEM | PMC_CAP_EDGE | \ + PMC_CAP_THRESHOLD | PMC_CAP_READ | \ + PMC_CAP_WRITE | PMC_CAP_INVERT | \ + PMC_CAP_QUALIFIER) + +#define RISCV_RELOAD_COUNT_TO_PERFCTR_VALUE(R) (-(R)) +#define RISCV_PERFCTR_VALUE_TO_RELOAD_COUNT(P) (-(P)) +#define EVENT_ID_MASK 0xFF + +#ifdef _KERNEL +/* MD extension for 'struct pmc' */ +struct pmc_md_riscv_pmc { + uint32_t pm_riscv_evsel; +}; +#endif /* _KERNEL */ +#endif /* _DEV_HWPMC_RISCV_H_ */ Property changes on: user/ngie/make_check/sys/dev/hwpmc/hwpmc_riscv.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/dev/ixl/if_ixl.c =================================================================== --- user/ngie/make_check/sys/dev/ixl/if_ixl.c (revision 295077) +++ user/ngie/make_check/sys/dev/ixl/if_ixl.c (revision 295078) @@ -1,6684 +1,6684 @@ /****************************************************************************** Copyright (c) 2013-2015, Intel Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the Intel Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. ******************************************************************************/ /*$FreeBSD$*/ #ifndef IXL_STANDALONE_BUILD #include "opt_inet.h" #include "opt_inet6.h" #include "opt_rss.h" #endif #include "ixl.h" #include "ixl_pf.h" #ifdef RSS #include #endif /********************************************************************* * Driver version *********************************************************************/ char ixl_driver_version[] = "1.4.3"; /********************************************************************* * PCI Device ID Table * * Used by probe to select devices to load on * Last field stores an index into ixl_strings * Last entry must be all 0s * * { Vendor ID, Device ID, SubVendor ID, SubDevice ID, String Index } *********************************************************************/ static ixl_vendor_info_t ixl_vendor_info_array[] = { {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_SFP_XL710, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_KX_A, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_KX_B, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_KX_C, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_QSFP_A, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_QSFP_B, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_QSFP_C, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_10G_BASE_T, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_10G_BASE_T4, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_20G_KR2, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_20G_KR2_A, 0, 0, 0}, #ifdef X722_SUPPORT {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_SFP_X722, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_1G_BASE_T_X722, 0, 0, 0}, {I40E_INTEL_VENDOR_ID, I40E_DEV_ID_10G_BASE_T_X722, 0, 0, 0}, #endif /* required last entry */ {0, 0, 0, 0, 0} }; /********************************************************************* * Table of branding strings *********************************************************************/ static char *ixl_strings[] = { "Intel(R) Ethernet Connection XL710 Driver" }; /********************************************************************* * Function prototypes *********************************************************************/ static int ixl_probe(device_t); static int ixl_attach(device_t); static int ixl_detach(device_t); static int ixl_shutdown(device_t); static int ixl_get_hw_capabilities(struct ixl_pf *); static void ixl_cap_txcsum_tso(struct ixl_vsi *, struct ifnet *, int); static int ixl_ioctl(struct ifnet *, u_long, caddr_t); static void ixl_init(void *); static void ixl_init_locked(struct ixl_pf *); static void ixl_stop(struct ixl_pf *); static void ixl_media_status(struct ifnet *, struct ifmediareq *); static int ixl_media_change(struct ifnet *); static void ixl_update_link_status(struct ixl_pf *); static int ixl_allocate_pci_resources(struct ixl_pf *); static u16 ixl_get_bus_info(struct i40e_hw *, device_t); static int ixl_setup_stations(struct ixl_pf *); static int ixl_switch_config(struct ixl_pf *); static int ixl_initialize_vsi(struct ixl_vsi *); static int ixl_assign_vsi_msix(struct ixl_pf *); static int ixl_assign_vsi_legacy(struct ixl_pf *); static int ixl_init_msix(struct ixl_pf *); static void ixl_configure_msix(struct ixl_pf *); static void ixl_configure_itr(struct ixl_pf *); static void ixl_configure_legacy(struct ixl_pf *); static void ixl_free_pci_resources(struct ixl_pf *); static void ixl_local_timer(void *); static int ixl_setup_interface(device_t, struct ixl_vsi *); static void ixl_link_event(struct ixl_pf *, struct i40e_arq_event_info *); static void ixl_config_rss(struct ixl_vsi *); static void ixl_set_queue_rx_itr(struct ixl_queue *); static void ixl_set_queue_tx_itr(struct ixl_queue *); static int ixl_set_advertised_speeds(struct ixl_pf *, int); static int ixl_enable_rings(struct ixl_vsi *); static int ixl_disable_rings(struct ixl_vsi *); static void ixl_enable_intr(struct ixl_vsi *); static void ixl_disable_intr(struct ixl_vsi *); static void ixl_disable_rings_intr(struct ixl_vsi *); static void ixl_enable_adminq(struct i40e_hw *); static void ixl_disable_adminq(struct i40e_hw *); static void ixl_enable_queue(struct i40e_hw *, int); static void ixl_disable_queue(struct i40e_hw *, int); static void ixl_enable_legacy(struct i40e_hw *); static void ixl_disable_legacy(struct i40e_hw *); static void ixl_set_promisc(struct ixl_vsi *); static void ixl_add_multi(struct ixl_vsi *); static void ixl_del_multi(struct ixl_vsi *); static void ixl_register_vlan(void *, struct ifnet *, u16); static void ixl_unregister_vlan(void *, struct ifnet *, u16); static void ixl_setup_vlan_filters(struct ixl_vsi *); static void ixl_init_filters(struct ixl_vsi *); static void ixl_reconfigure_filters(struct ixl_vsi *vsi); static void ixl_add_filter(struct ixl_vsi *, u8 *, s16 vlan); static void ixl_del_filter(struct ixl_vsi *, u8 *, s16 vlan); static void ixl_add_hw_filters(struct ixl_vsi *, int, int); static void ixl_del_hw_filters(struct ixl_vsi *, int); static struct ixl_mac_filter * ixl_find_filter(struct ixl_vsi *, u8 *, s16); static void ixl_add_mc_filter(struct ixl_vsi *, u8 *); static void ixl_free_mac_filters(struct ixl_vsi *vsi); /* Sysctl debug interface */ #ifdef IXL_DEBUG_SYSCTL static int ixl_debug_info(SYSCTL_HANDLER_ARGS); static void ixl_print_debug_info(struct ixl_pf *); #endif /* The MSI/X Interrupt handlers */ static void ixl_intr(void *); static void ixl_msix_que(void *); static void ixl_msix_adminq(void *); static void ixl_handle_mdd_event(struct ixl_pf *); /* Deferred interrupt tasklets */ static void ixl_do_adminq(void *, int); /* Sysctl handlers */ static int ixl_set_flowcntl(SYSCTL_HANDLER_ARGS); static int ixl_set_advertise(SYSCTL_HANDLER_ARGS); static int ixl_current_speed(SYSCTL_HANDLER_ARGS); static int ixl_sysctl_show_fw(SYSCTL_HANDLER_ARGS); /* Statistics */ static void ixl_add_hw_stats(struct ixl_pf *); static void ixl_add_sysctls_mac_stats(struct sysctl_ctx_list *, struct sysctl_oid_list *, struct i40e_hw_port_stats *); static void ixl_add_sysctls_eth_stats(struct sysctl_ctx_list *, struct sysctl_oid_list *, struct i40e_eth_stats *); static void ixl_update_stats_counters(struct ixl_pf *); static void ixl_update_eth_stats(struct ixl_vsi *); static void ixl_update_vsi_stats(struct ixl_vsi *); static void ixl_pf_reset_stats(struct ixl_pf *); static void ixl_vsi_reset_stats(struct ixl_vsi *); static void ixl_stat_update48(struct i40e_hw *, u32, u32, bool, u64 *, u64 *); static void ixl_stat_update32(struct i40e_hw *, u32, bool, u64 *, u64 *); #ifdef IXL_DEBUG_SYSCTL static int ixl_sysctl_link_status(SYSCTL_HANDLER_ARGS); static int ixl_sysctl_phy_abilities(SYSCTL_HANDLER_ARGS); static int ixl_sysctl_sw_filter_list(SYSCTL_HANDLER_ARGS); static int ixl_sysctl_hw_res_alloc(SYSCTL_HANDLER_ARGS); static int ixl_sysctl_switch_config(SYSCTL_HANDLER_ARGS); #endif #ifdef PCI_IOV static int ixl_adminq_err_to_errno(enum i40e_admin_queue_err err); static int ixl_iov_init(device_t dev, uint16_t num_vfs, const nvlist_t*); static void ixl_iov_uninit(device_t dev); static int ixl_add_vf(device_t dev, uint16_t vfnum, const nvlist_t*); static void ixl_handle_vf_msg(struct ixl_pf *, struct i40e_arq_event_info *); static void ixl_handle_vflr(void *arg, int pending); static void ixl_reset_vf(struct ixl_pf *pf, struct ixl_vf *vf); static void ixl_reinit_vf(struct ixl_pf *pf, struct ixl_vf *vf); #endif /********************************************************************* * FreeBSD Device Interface Entry Points *********************************************************************/ static device_method_t ixl_methods[] = { /* Device interface */ DEVMETHOD(device_probe, ixl_probe), DEVMETHOD(device_attach, ixl_attach), DEVMETHOD(device_detach, ixl_detach), DEVMETHOD(device_shutdown, ixl_shutdown), #ifdef PCI_IOV DEVMETHOD(pci_iov_init, ixl_iov_init), DEVMETHOD(pci_iov_uninit, ixl_iov_uninit), DEVMETHOD(pci_iov_add_vf, ixl_add_vf), #endif {0, 0} }; static driver_t ixl_driver = { "ixl", ixl_methods, sizeof(struct ixl_pf), }; devclass_t ixl_devclass; DRIVER_MODULE(ixl, pci, ixl_driver, ixl_devclass, 0, 0); MODULE_DEPEND(ixl, pci, 1, 1, 1); MODULE_DEPEND(ixl, ether, 1, 1, 1); #ifdef DEV_NETMAP MODULE_DEPEND(ixl, netmap, 1, 1, 1); #endif /* DEV_NETMAP */ /* ** Global reset mutex */ static struct mtx ixl_reset_mtx; /* ** TUNEABLE PARAMETERS: */ static SYSCTL_NODE(_hw, OID_AUTO, ixl, CTLFLAG_RD, 0, "IXL driver parameters"); /* * MSIX should be the default for best performance, * but this allows it to be forced off for testing. */ static int ixl_enable_msix = 1; TUNABLE_INT("hw.ixl.enable_msix", &ixl_enable_msix); SYSCTL_INT(_hw_ixl, OID_AUTO, enable_msix, CTLFLAG_RDTUN, &ixl_enable_msix, 0, "Enable MSI-X interrupts"); /* ** Number of descriptors per ring: ** - TX and RX are the same size */ static int ixl_ringsz = DEFAULT_RING; TUNABLE_INT("hw.ixl.ringsz", &ixl_ringsz); SYSCTL_INT(_hw_ixl, OID_AUTO, ring_size, CTLFLAG_RDTUN, &ixl_ringsz, 0, "Descriptor Ring Size"); /* ** This can be set manually, if left as 0 the ** number of queues will be calculated based ** on cpus and msix vectors available. */ int ixl_max_queues = 0; TUNABLE_INT("hw.ixl.max_queues", &ixl_max_queues); SYSCTL_INT(_hw_ixl, OID_AUTO, max_queues, CTLFLAG_RDTUN, &ixl_max_queues, 0, "Number of Queues"); /* ** Controls for Interrupt Throttling ** - true/false for dynamic adjustment ** - default values for static ITR */ int ixl_dynamic_rx_itr = 0; TUNABLE_INT("hw.ixl.dynamic_rx_itr", &ixl_dynamic_rx_itr); SYSCTL_INT(_hw_ixl, OID_AUTO, dynamic_rx_itr, CTLFLAG_RDTUN, &ixl_dynamic_rx_itr, 0, "Dynamic RX Interrupt Rate"); int ixl_dynamic_tx_itr = 0; TUNABLE_INT("hw.ixl.dynamic_tx_itr", &ixl_dynamic_tx_itr); SYSCTL_INT(_hw_ixl, OID_AUTO, dynamic_tx_itr, CTLFLAG_RDTUN, &ixl_dynamic_tx_itr, 0, "Dynamic TX Interrupt Rate"); int ixl_rx_itr = IXL_ITR_8K; TUNABLE_INT("hw.ixl.rx_itr", &ixl_rx_itr); SYSCTL_INT(_hw_ixl, OID_AUTO, rx_itr, CTLFLAG_RDTUN, &ixl_rx_itr, 0, "RX Interrupt Rate"); int ixl_tx_itr = IXL_ITR_4K; TUNABLE_INT("hw.ixl.tx_itr", &ixl_tx_itr); SYSCTL_INT(_hw_ixl, OID_AUTO, tx_itr, CTLFLAG_RDTUN, &ixl_tx_itr, 0, "TX Interrupt Rate"); #ifdef IXL_FDIR static int ixl_enable_fdir = 1; TUNABLE_INT("hw.ixl.enable_fdir", &ixl_enable_fdir); /* Rate at which we sample */ int ixl_atr_rate = 20; TUNABLE_INT("hw.ixl.atr_rate", &ixl_atr_rate); #endif #ifdef DEV_NETMAP #define NETMAP_IXL_MAIN /* only bring in one part of the netmap code */ #include #endif /* DEV_NETMAP */ static char *ixl_fc_string[6] = { "None", "Rx", "Tx", "Full", "Priority", "Default" }; static MALLOC_DEFINE(M_IXL, "ixl", "ixl driver allocations"); static uint8_t ixl_bcast_addr[ETHER_ADDR_LEN] = {0xff, 0xff, 0xff, 0xff, 0xff, 0xff}; /********************************************************************* * Device identification routine * * ixl_probe determines if the driver should be loaded on * the hardware based on PCI vendor/device id of the device. * * return BUS_PROBE_DEFAULT on success, positive on failure *********************************************************************/ static int ixl_probe(device_t dev) { ixl_vendor_info_t *ent; u16 pci_vendor_id, pci_device_id; u16 pci_subvendor_id, pci_subdevice_id; char device_name[256]; static bool lock_init = FALSE; INIT_DEBUGOUT("ixl_probe: begin"); pci_vendor_id = pci_get_vendor(dev); if (pci_vendor_id != I40E_INTEL_VENDOR_ID) return (ENXIO); pci_device_id = pci_get_device(dev); pci_subvendor_id = pci_get_subvendor(dev); pci_subdevice_id = pci_get_subdevice(dev); ent = ixl_vendor_info_array; while (ent->vendor_id != 0) { if ((pci_vendor_id == ent->vendor_id) && (pci_device_id == ent->device_id) && ((pci_subvendor_id == ent->subvendor_id) || (ent->subvendor_id == 0)) && ((pci_subdevice_id == ent->subdevice_id) || (ent->subdevice_id == 0))) { sprintf(device_name, "%s, Version - %s", ixl_strings[ent->index], ixl_driver_version); device_set_desc_copy(dev, device_name); /* One shot mutex init */ if (lock_init == FALSE) { lock_init = TRUE; mtx_init(&ixl_reset_mtx, "ixl_reset", "IXL RESET Lock", MTX_DEF); } return (BUS_PROBE_DEFAULT); } ent++; } return (ENXIO); } /********************************************************************* * Device initialization routine * * The attach entry point is called when the driver is being loaded. * This routine identifies the type of hardware, allocates all resources * and initializes the hardware. * * return 0 on success, positive on failure *********************************************************************/ static int ixl_attach(device_t dev) { struct ixl_pf *pf; struct i40e_hw *hw; struct ixl_vsi *vsi; u16 bus; int error = 0; #ifdef PCI_IOV nvlist_t *pf_schema, *vf_schema; int iov_error; #endif INIT_DEBUGOUT("ixl_attach: begin"); /* Allocate, clear, and link in our primary soft structure */ pf = device_get_softc(dev); pf->dev = pf->osdep.dev = dev; hw = &pf->hw; /* ** Note this assumes we have a single embedded VSI, ** this could be enhanced later to allocate multiple */ vsi = &pf->vsi; vsi->dev = pf->dev; /* Core Lock Init*/ IXL_PF_LOCK_INIT(pf, device_get_nameunit(dev)); /* Set up the timer callout */ callout_init_mtx(&pf->timer, &pf->pf_mtx, 0); /* Set up sysctls */ SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "fc", CTLTYPE_INT | CTLFLAG_RW, pf, 0, ixl_set_flowcntl, "I", "Flow Control"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "advertise_speed", CTLTYPE_INT | CTLFLAG_RW, pf, 0, ixl_set_advertise, "I", "Advertised Speed"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "current_speed", CTLTYPE_STRING | CTLFLAG_RD, pf, 0, ixl_current_speed, "A", "Current Port Speed"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "fw_version", CTLTYPE_STRING | CTLFLAG_RD, pf, 0, ixl_sysctl_show_fw, "A", "Firmware version"); SYSCTL_ADD_INT(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "rx_itr", CTLFLAG_RW, &ixl_rx_itr, IXL_ITR_8K, "RX ITR"); SYSCTL_ADD_INT(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "dynamic_rx_itr", CTLFLAG_RW, &ixl_dynamic_rx_itr, 0, "Dynamic RX ITR"); SYSCTL_ADD_INT(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "tx_itr", CTLFLAG_RW, &ixl_tx_itr, IXL_ITR_4K, "TX ITR"); SYSCTL_ADD_INT(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "dynamic_tx_itr", CTLFLAG_RW, &ixl_dynamic_tx_itr, 0, "Dynamic TX ITR"); #ifdef IXL_DEBUG_SYSCTL SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "debug", CTLTYPE_INT|CTLFLAG_RW, pf, 0, ixl_debug_info, "I", "Debug Information"); /* Debug shared-code message level */ SYSCTL_ADD_UINT(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "debug_mask", CTLFLAG_RW, &pf->hw.debug_mask, 0, "Debug Message Level"); SYSCTL_ADD_UINT(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "vc_debug_level", CTLFLAG_RW, &pf->vc_debug_lvl, 0, "PF/VF Virtual Channel debug level"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "link_status", CTLTYPE_STRING | CTLFLAG_RD, pf, 0, ixl_sysctl_link_status, "A", "Current Link Status"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "phy_abilities", CTLTYPE_STRING | CTLFLAG_RD, pf, 0, ixl_sysctl_phy_abilities, "A", "PHY Abilities"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "filter_list", CTLTYPE_STRING | CTLFLAG_RD, pf, 0, ixl_sysctl_sw_filter_list, "A", "SW Filter List"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "hw_res_alloc", CTLTYPE_STRING | CTLFLAG_RD, pf, 0, ixl_sysctl_hw_res_alloc, "A", "HW Resource Allocation"); SYSCTL_ADD_PROC(device_get_sysctl_ctx(dev), SYSCTL_CHILDREN(device_get_sysctl_tree(dev)), OID_AUTO, "switch_config", CTLTYPE_STRING | CTLFLAG_RD, pf, 0, ixl_sysctl_switch_config, "A", "HW Switch Configuration"); #endif /* Save off the PCI information */ hw->vendor_id = pci_get_vendor(dev); hw->device_id = pci_get_device(dev); hw->revision_id = pci_read_config(dev, PCIR_REVID, 1); hw->subsystem_vendor_id = pci_read_config(dev, PCIR_SUBVEND_0, 2); hw->subsystem_device_id = pci_read_config(dev, PCIR_SUBDEV_0, 2); hw->bus.device = pci_get_slot(dev); hw->bus.func = pci_get_function(dev); pf->vc_debug_lvl = 1; /* Do PCI setup - map BAR0, etc */ if (ixl_allocate_pci_resources(pf)) { device_printf(dev, "Allocation of PCI resources failed\n"); error = ENXIO; goto err_out; } /* Establish a clean starting point */ i40e_clear_hw(hw); error = i40e_pf_reset(hw); if (error) { device_printf(dev,"PF reset failure %x\n", error); error = EIO; goto err_out; } /* Set admin queue parameters */ hw->aq.num_arq_entries = IXL_AQ_LEN; hw->aq.num_asq_entries = IXL_AQ_LEN; hw->aq.arq_buf_size = IXL_AQ_BUFSZ; hw->aq.asq_buf_size = IXL_AQ_BUFSZ; /* Initialize the shared code */ error = i40e_init_shared_code(hw); if (error) { device_printf(dev,"Unable to initialize the shared code\n"); error = EIO; goto err_out; } /* Set up the admin queue */ error = i40e_init_adminq(hw); if (error) { device_printf(dev, "The driver for the device stopped " "because the NVM image is newer than expected.\n" "You must install the most recent version of " " the network driver.\n"); goto err_out; } device_printf(dev, "%s\n", ixl_fw_version_str(hw)); if (hw->aq.api_maj_ver == I40E_FW_API_VERSION_MAJOR && hw->aq.api_min_ver > I40E_FW_API_VERSION_MINOR) device_printf(dev, "The driver for the device detected " "a newer version of the NVM image than expected.\n" "Please install the most recent version of the network driver.\n"); else if (hw->aq.api_maj_ver < I40E_FW_API_VERSION_MAJOR || hw->aq.api_min_ver < (I40E_FW_API_VERSION_MINOR - 1)) device_printf(dev, "The driver for the device detected " "an older version of the NVM image than expected.\n" "Please update the NVM image.\n"); /* Clear PXE mode */ i40e_clear_pxe_mode(hw); /* Get capabilities from the device */ error = ixl_get_hw_capabilities(pf); if (error) { device_printf(dev, "HW capabilities failure!\n"); goto err_get_cap; } /* Set up host memory cache */ error = i40e_init_lan_hmc(hw, hw->func_caps.num_tx_qp, hw->func_caps.num_rx_qp, 0, 0); if (error) { device_printf(dev, "init_lan_hmc failed: %d\n", error); goto err_get_cap; } error = i40e_configure_lan_hmc(hw, I40E_HMC_MODEL_DIRECT_ONLY); if (error) { device_printf(dev, "configure_lan_hmc failed: %d\n", error); goto err_mac_hmc; } /* Disable LLDP from the firmware */ i40e_aq_stop_lldp(hw, TRUE, NULL); i40e_get_mac_addr(hw, hw->mac.addr); error = i40e_validate_mac_addr(hw->mac.addr); if (error) { device_printf(dev, "validate_mac_addr failed: %d\n", error); goto err_mac_hmc; } bcopy(hw->mac.addr, hw->mac.perm_addr, ETHER_ADDR_LEN); i40e_get_port_mac_addr(hw, hw->mac.port_addr); /* Set up VSI and queues */ if (ixl_setup_stations(pf) != 0) { device_printf(dev, "setup stations failed!\n"); error = ENOMEM; goto err_mac_hmc; } /* Initialize mac filter list for VSI */ SLIST_INIT(&vsi->ftl); /* Set up interrupt routing here */ if (pf->msix > 1) error = ixl_assign_vsi_msix(pf); else error = ixl_assign_vsi_legacy(pf); if (error) goto err_late; if (((hw->aq.fw_maj_ver == 4) && (hw->aq.fw_min_ver < 33)) || (hw->aq.fw_maj_ver < 4)) { i40e_msec_delay(75); error = i40e_aq_set_link_restart_an(hw, TRUE, NULL); if (error) device_printf(dev, "link restart failed, aq_err=%d\n", pf->hw.aq.asq_last_status); } /* Determine link state */ i40e_aq_get_link_info(hw, TRUE, NULL, NULL); i40e_get_link_status(hw, &pf->link_up); /* Setup OS specific network interface */ if (ixl_setup_interface(dev, vsi) != 0) { device_printf(dev, "interface setup failed!\n"); error = EIO; goto err_late; } error = ixl_switch_config(pf); if (error) { device_printf(dev, "Initial switch config failed: %d\n", error); goto err_mac_hmc; } /* Limit phy interrupts to link and modules failure */ - error = i40e_aq_set_phy_int_mask(hw, - I40E_AQ_EVENT_LINK_UPDOWN | I40E_AQ_EVENT_MODULE_QUAL_FAIL, NULL); - if (error) + error = i40e_aq_set_phy_int_mask(hw, ~(I40E_AQ_EVENT_LINK_UPDOWN | + I40E_AQ_EVENT_MODULE_QUAL_FAIL), NULL); + if (error) device_printf(dev, "set phy mask failed: %d\n", error); /* Get the bus configuration and set the shared code */ bus = ixl_get_bus_info(hw, dev); i40e_set_pci_config_data(hw, bus); /* Initialize statistics */ ixl_pf_reset_stats(pf); ixl_update_stats_counters(pf); ixl_add_hw_stats(pf); /* Register for VLAN events */ vsi->vlan_attach = EVENTHANDLER_REGISTER(vlan_config, ixl_register_vlan, vsi, EVENTHANDLER_PRI_FIRST); vsi->vlan_detach = EVENTHANDLER_REGISTER(vlan_unconfig, ixl_unregister_vlan, vsi, EVENTHANDLER_PRI_FIRST); #ifdef PCI_IOV /* SR-IOV is only supported when MSI-X is in use. */ if (pf->msix > 1) { pf_schema = pci_iov_schema_alloc_node(); vf_schema = pci_iov_schema_alloc_node(); pci_iov_schema_add_unicast_mac(vf_schema, "mac-addr", 0, NULL); pci_iov_schema_add_bool(vf_schema, "mac-anti-spoof", IOV_SCHEMA_HASDEFAULT, TRUE); pci_iov_schema_add_bool(vf_schema, "allow-set-mac", IOV_SCHEMA_HASDEFAULT, FALSE); pci_iov_schema_add_bool(vf_schema, "allow-promisc", IOV_SCHEMA_HASDEFAULT, FALSE); iov_error = pci_iov_attach(dev, pf_schema, vf_schema); if (iov_error != 0) device_printf(dev, "Failed to initialize SR-IOV (error=%d)\n", iov_error); } #endif #ifdef DEV_NETMAP ixl_netmap_attach(vsi); #endif /* DEV_NETMAP */ INIT_DEBUGOUT("ixl_attach: end"); return (0); err_late: if (vsi->ifp != NULL) if_free(vsi->ifp); err_mac_hmc: i40e_shutdown_lan_hmc(hw); err_get_cap: i40e_shutdown_adminq(hw); err_out: ixl_free_pci_resources(pf); ixl_free_vsi(vsi); IXL_PF_LOCK_DESTROY(pf); return (error); } /********************************************************************* * Device removal routine * * The detach entry point is called when the driver is being removed. * This routine stops the adapter and deallocates all the resources * that were allocated for driver operation. * * return 0 on success, positive on failure *********************************************************************/ static int ixl_detach(device_t dev) { struct ixl_pf *pf = device_get_softc(dev); struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; i40e_status status; #ifdef PCI_IOV int error; #endif INIT_DEBUGOUT("ixl_detach: begin"); /* Make sure VLANS are not using driver */ if (vsi->ifp->if_vlantrunk != NULL) { device_printf(dev,"Vlan in use, detach first\n"); return (EBUSY); } #ifdef PCI_IOV error = pci_iov_detach(dev); if (error != 0) { device_printf(dev, "SR-IOV in use; detach first.\n"); return (error); } #endif ether_ifdetach(vsi->ifp); if (vsi->ifp->if_drv_flags & IFF_DRV_RUNNING) { IXL_PF_LOCK(pf); ixl_stop(pf); IXL_PF_UNLOCK(pf); } for (int i = 0; i < vsi->num_queues; i++, que++) { if (que->tq) { taskqueue_drain(que->tq, &que->task); taskqueue_drain(que->tq, &que->tx_task); taskqueue_free(que->tq); } } /* Shutdown LAN HMC */ status = i40e_shutdown_lan_hmc(hw); if (status) device_printf(dev, "Shutdown LAN HMC failed with code %d\n", status); /* Shutdown admin queue */ status = i40e_shutdown_adminq(hw); if (status) device_printf(dev, "Shutdown Admin queue failed with code %d\n", status); /* Unregister VLAN events */ if (vsi->vlan_attach != NULL) EVENTHANDLER_DEREGISTER(vlan_config, vsi->vlan_attach); if (vsi->vlan_detach != NULL) EVENTHANDLER_DEREGISTER(vlan_unconfig, vsi->vlan_detach); callout_drain(&pf->timer); #ifdef DEV_NETMAP netmap_detach(vsi->ifp); #endif /* DEV_NETMAP */ ixl_free_pci_resources(pf); bus_generic_detach(dev); if_free(vsi->ifp); ixl_free_vsi(vsi); IXL_PF_LOCK_DESTROY(pf); return (0); } /********************************************************************* * * Shutdown entry point * **********************************************************************/ static int ixl_shutdown(device_t dev) { struct ixl_pf *pf = device_get_softc(dev); IXL_PF_LOCK(pf); ixl_stop(pf); IXL_PF_UNLOCK(pf); return (0); } /********************************************************************* * * Get the hardware capabilities * **********************************************************************/ static int ixl_get_hw_capabilities(struct ixl_pf *pf) { struct i40e_aqc_list_capabilities_element_resp *buf; struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; int error, len; u16 needed; bool again = TRUE; len = 40 * sizeof(struct i40e_aqc_list_capabilities_element_resp); retry: if (!(buf = (struct i40e_aqc_list_capabilities_element_resp *) malloc(len, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate cap memory\n"); return (ENOMEM); } /* This populates the hw struct */ error = i40e_aq_discover_capabilities(hw, buf, len, &needed, i40e_aqc_opc_list_func_capabilities, NULL); free(buf, M_DEVBUF); if ((pf->hw.aq.asq_last_status == I40E_AQ_RC_ENOMEM) && (again == TRUE)) { /* retry once with a larger buffer */ again = FALSE; len = needed; goto retry; } else if (pf->hw.aq.asq_last_status != I40E_AQ_RC_OK) { device_printf(dev, "capability discovery failed: %d\n", pf->hw.aq.asq_last_status); return (ENODEV); } /* Capture this PF's starting queue pair */ pf->qbase = hw->func_caps.base_queue; #ifdef IXL_DEBUG device_printf(dev,"pf_id=%d, num_vfs=%d, msix_pf=%d, " "msix_vf=%d, fd_g=%d, fd_b=%d, tx_qp=%d rx_qp=%d qbase=%d\n", hw->pf_id, hw->func_caps.num_vfs, hw->func_caps.num_msix_vectors, hw->func_caps.num_msix_vectors_vf, hw->func_caps.fd_filters_guaranteed, hw->func_caps.fd_filters_best_effort, hw->func_caps.num_tx_qp, hw->func_caps.num_rx_qp, hw->func_caps.base_queue); #endif return (error); } static void ixl_cap_txcsum_tso(struct ixl_vsi *vsi, struct ifnet *ifp, int mask) { device_t dev = vsi->dev; /* Enable/disable TXCSUM/TSO4 */ if (!(ifp->if_capenable & IFCAP_TXCSUM) && !(ifp->if_capenable & IFCAP_TSO4)) { if (mask & IFCAP_TXCSUM) { ifp->if_capenable |= IFCAP_TXCSUM; /* enable TXCSUM, restore TSO if previously enabled */ if (vsi->flags & IXL_FLAGS_KEEP_TSO4) { vsi->flags &= ~IXL_FLAGS_KEEP_TSO4; ifp->if_capenable |= IFCAP_TSO4; } } else if (mask & IFCAP_TSO4) { ifp->if_capenable |= (IFCAP_TXCSUM | IFCAP_TSO4); vsi->flags &= ~IXL_FLAGS_KEEP_TSO4; device_printf(dev, "TSO4 requires txcsum, enabling both...\n"); } } else if((ifp->if_capenable & IFCAP_TXCSUM) && !(ifp->if_capenable & IFCAP_TSO4)) { if (mask & IFCAP_TXCSUM) ifp->if_capenable &= ~IFCAP_TXCSUM; else if (mask & IFCAP_TSO4) ifp->if_capenable |= IFCAP_TSO4; } else if((ifp->if_capenable & IFCAP_TXCSUM) && (ifp->if_capenable & IFCAP_TSO4)) { if (mask & IFCAP_TXCSUM) { vsi->flags |= IXL_FLAGS_KEEP_TSO4; ifp->if_capenable &= ~(IFCAP_TXCSUM | IFCAP_TSO4); device_printf(dev, "TSO4 requires txcsum, disabling both...\n"); } else if (mask & IFCAP_TSO4) ifp->if_capenable &= ~IFCAP_TSO4; } /* Enable/disable TXCSUM_IPV6/TSO6 */ if (!(ifp->if_capenable & IFCAP_TXCSUM_IPV6) && !(ifp->if_capenable & IFCAP_TSO6)) { if (mask & IFCAP_TXCSUM_IPV6) { ifp->if_capenable |= IFCAP_TXCSUM_IPV6; if (vsi->flags & IXL_FLAGS_KEEP_TSO6) { vsi->flags &= ~IXL_FLAGS_KEEP_TSO6; ifp->if_capenable |= IFCAP_TSO6; } } else if (mask & IFCAP_TSO6) { ifp->if_capenable |= (IFCAP_TXCSUM_IPV6 | IFCAP_TSO6); vsi->flags &= ~IXL_FLAGS_KEEP_TSO6; device_printf(dev, "TSO6 requires txcsum6, enabling both...\n"); } } else if((ifp->if_capenable & IFCAP_TXCSUM_IPV6) && !(ifp->if_capenable & IFCAP_TSO6)) { if (mask & IFCAP_TXCSUM_IPV6) ifp->if_capenable &= ~IFCAP_TXCSUM_IPV6; else if (mask & IFCAP_TSO6) ifp->if_capenable |= IFCAP_TSO6; } else if ((ifp->if_capenable & IFCAP_TXCSUM_IPV6) && (ifp->if_capenable & IFCAP_TSO6)) { if (mask & IFCAP_TXCSUM_IPV6) { vsi->flags |= IXL_FLAGS_KEEP_TSO6; ifp->if_capenable &= ~(IFCAP_TXCSUM_IPV6 | IFCAP_TSO6); device_printf(dev, "TSO6 requires txcsum6, disabling both...\n"); } else if (mask & IFCAP_TSO6) ifp->if_capenable &= ~IFCAP_TSO6; } } /********************************************************************* * Ioctl entry point * * ixl_ioctl is called when the user wants to configure the * interface. * * return 0 on success, positive on failure **********************************************************************/ static int ixl_ioctl(struct ifnet * ifp, u_long command, caddr_t data) { struct ixl_vsi *vsi = ifp->if_softc; struct ixl_pf *pf = vsi->back; struct ifreq *ifr = (struct ifreq *) data; #if defined(INET) || defined(INET6) struct ifaddr *ifa = (struct ifaddr *)data; bool avoid_reset = FALSE; #endif int error = 0; switch (command) { case SIOCSIFADDR: #ifdef INET if (ifa->ifa_addr->sa_family == AF_INET) avoid_reset = TRUE; #endif #ifdef INET6 if (ifa->ifa_addr->sa_family == AF_INET6) avoid_reset = TRUE; #endif #if defined(INET) || defined(INET6) /* ** Calling init results in link renegotiation, ** so we avoid doing it when possible. */ if (avoid_reset) { ifp->if_flags |= IFF_UP; if (!(ifp->if_drv_flags & IFF_DRV_RUNNING)) ixl_init(pf); #ifdef INET if (!(ifp->if_flags & IFF_NOARP)) arp_ifinit(ifp, ifa); #endif } else error = ether_ioctl(ifp, command, data); break; #endif case SIOCSIFMTU: IOCTL_DEBUGOUT("ioctl: SIOCSIFMTU (Set Interface MTU)"); if (ifr->ifr_mtu > IXL_MAX_FRAME - ETHER_HDR_LEN - ETHER_CRC_LEN - ETHER_VLAN_ENCAP_LEN) { error = EINVAL; } else { IXL_PF_LOCK(pf); ifp->if_mtu = ifr->ifr_mtu; vsi->max_frame_size = ifp->if_mtu + ETHER_HDR_LEN + ETHER_CRC_LEN + ETHER_VLAN_ENCAP_LEN; ixl_init_locked(pf); IXL_PF_UNLOCK(pf); } break; case SIOCSIFFLAGS: IOCTL_DEBUGOUT("ioctl: SIOCSIFFLAGS (Set Interface Flags)"); IXL_PF_LOCK(pf); if (ifp->if_flags & IFF_UP) { if ((ifp->if_drv_flags & IFF_DRV_RUNNING)) { if ((ifp->if_flags ^ pf->if_flags) & (IFF_PROMISC | IFF_ALLMULTI)) { ixl_set_promisc(vsi); } } else ixl_init_locked(pf); } else if (ifp->if_drv_flags & IFF_DRV_RUNNING) ixl_stop(pf); pf->if_flags = ifp->if_flags; IXL_PF_UNLOCK(pf); break; case SIOCADDMULTI: IOCTL_DEBUGOUT("ioctl: SIOCADDMULTI"); if (ifp->if_drv_flags & IFF_DRV_RUNNING) { IXL_PF_LOCK(pf); ixl_disable_intr(vsi); ixl_add_multi(vsi); ixl_enable_intr(vsi); IXL_PF_UNLOCK(pf); } break; case SIOCDELMULTI: IOCTL_DEBUGOUT("ioctl: SIOCDELMULTI"); if (ifp->if_drv_flags & IFF_DRV_RUNNING) { IXL_PF_LOCK(pf); ixl_disable_intr(vsi); ixl_del_multi(vsi); ixl_enable_intr(vsi); IXL_PF_UNLOCK(pf); } break; case SIOCSIFMEDIA: case SIOCGIFMEDIA: #ifdef IFM_ETH_XTYPE case SIOCGIFXMEDIA: #endif IOCTL_DEBUGOUT("ioctl: SIOCxIFMEDIA (Get/Set Interface Media)"); error = ifmedia_ioctl(ifp, ifr, &vsi->media, command); break; case SIOCSIFCAP: { int mask = ifr->ifr_reqcap ^ ifp->if_capenable; IOCTL_DEBUGOUT("ioctl: SIOCSIFCAP (Set Capabilities)"); ixl_cap_txcsum_tso(vsi, ifp, mask); if (mask & IFCAP_RXCSUM) ifp->if_capenable ^= IFCAP_RXCSUM; if (mask & IFCAP_RXCSUM_IPV6) ifp->if_capenable ^= IFCAP_RXCSUM_IPV6; if (mask & IFCAP_LRO) ifp->if_capenable ^= IFCAP_LRO; if (mask & IFCAP_VLAN_HWTAGGING) ifp->if_capenable ^= IFCAP_VLAN_HWTAGGING; if (mask & IFCAP_VLAN_HWFILTER) ifp->if_capenable ^= IFCAP_VLAN_HWFILTER; if (mask & IFCAP_VLAN_HWTSO) ifp->if_capenable ^= IFCAP_VLAN_HWTSO; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { IXL_PF_LOCK(pf); ixl_init_locked(pf); IXL_PF_UNLOCK(pf); } VLAN_CAPABILITIES(ifp); break; } default: IOCTL_DEBUGOUT("ioctl: UNKNOWN (0x%X)\n", (int)command); error = ether_ioctl(ifp, command, data); break; } return (error); } /********************************************************************* * Init entry point * * This routine is used in two ways. It is used by the stack as * init entry point in network interface structure. It is also used * by the driver as a hw/sw initialization routine to get to a * consistent state. * * return 0 on success, positive on failure **********************************************************************/ static void ixl_init_locked(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct ifnet *ifp = vsi->ifp; device_t dev = pf->dev; struct i40e_filter_control_settings filter; u8 tmpaddr[ETHER_ADDR_LEN]; int ret; mtx_assert(&pf->pf_mtx, MA_OWNED); INIT_DEBUGOUT("ixl_init: begin"); ixl_stop(pf); /* Get the latest mac address... User might use a LAA */ bcopy(IF_LLADDR(vsi->ifp), tmpaddr, I40E_ETH_LENGTH_OF_ADDRESS); if (!cmp_etheraddr(hw->mac.addr, tmpaddr) && (i40e_validate_mac_addr(tmpaddr) == I40E_SUCCESS)) { ixl_del_filter(vsi, hw->mac.addr, IXL_VLAN_ANY); bcopy(tmpaddr, hw->mac.addr, I40E_ETH_LENGTH_OF_ADDRESS); ret = i40e_aq_mac_address_write(hw, I40E_AQC_WRITE_TYPE_LAA_ONLY, hw->mac.addr, NULL); if (ret) { device_printf(dev, "LLA address" "change failed!!\n"); return; } else { ixl_add_filter(vsi, hw->mac.addr, IXL_VLAN_ANY); } } /* Set the various hardware offload abilities */ ifp->if_hwassist = 0; if (ifp->if_capenable & IFCAP_TSO) ifp->if_hwassist |= CSUM_TSO; if (ifp->if_capenable & IFCAP_TXCSUM) ifp->if_hwassist |= (CSUM_TCP | CSUM_UDP); if (ifp->if_capenable & IFCAP_TXCSUM_IPV6) ifp->if_hwassist |= (CSUM_TCP_IPV6 | CSUM_UDP_IPV6); /* Set up the device filtering */ bzero(&filter, sizeof(filter)); filter.enable_ethtype = TRUE; filter.enable_macvlan = TRUE; #ifdef IXL_FDIR filter.enable_fdir = TRUE; #endif if (i40e_set_filter_control(hw, &filter)) device_printf(dev, "set_filter_control() failed\n"); /* Set up RSS */ ixl_config_rss(vsi); /* ** Prepare the VSI: rings, hmc contexts, etc... */ if (ixl_initialize_vsi(vsi)) { device_printf(dev, "initialize vsi failed!!\n"); return; } /* Add protocol filters to list */ ixl_init_filters(vsi); /* Setup vlan's if needed */ ixl_setup_vlan_filters(vsi); /* Start the local timer */ callout_reset(&pf->timer, hz, ixl_local_timer, pf); /* Set up MSI/X routing and the ITR settings */ if (ixl_enable_msix) { ixl_configure_msix(pf); ixl_configure_itr(pf); } else ixl_configure_legacy(pf); ixl_enable_rings(vsi); i40e_aq_set_default_vsi(hw, vsi->seid, NULL); ixl_reconfigure_filters(vsi); /* Set MTU in hardware*/ int aq_error = i40e_aq_set_mac_config(hw, vsi->max_frame_size, TRUE, 0, NULL); if (aq_error) device_printf(vsi->dev, "aq_set_mac_config in init error, code %d\n", aq_error); /* And now turn on interrupts */ ixl_enable_intr(vsi); /* Now inform the stack we're ready */ ifp->if_drv_flags |= IFF_DRV_RUNNING; ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; return; } static void ixl_init(void *arg) { struct ixl_pf *pf = arg; IXL_PF_LOCK(pf); ixl_init_locked(pf); IXL_PF_UNLOCK(pf); return; } /* ** ** MSIX Interrupt Handlers and Tasklets ** */ static void ixl_handle_que(void *context, int pending) { struct ixl_queue *que = context; struct ixl_vsi *vsi = que->vsi; struct i40e_hw *hw = vsi->hw; struct tx_ring *txr = &que->txr; struct ifnet *ifp = vsi->ifp; bool more; if (ifp->if_drv_flags & IFF_DRV_RUNNING) { more = ixl_rxeof(que, IXL_RX_LIMIT); IXL_TX_LOCK(txr); ixl_txeof(que); if (!drbr_empty(ifp, txr->br)) ixl_mq_start_locked(ifp, txr); IXL_TX_UNLOCK(txr); if (more) { taskqueue_enqueue(que->tq, &que->task); return; } } /* Reenable this interrupt - hmmm */ ixl_enable_queue(hw, que->me); return; } /********************************************************************* * * Legacy Interrupt Service routine * **********************************************************************/ void ixl_intr(void *arg) { struct ixl_pf *pf = arg; struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; struct ifnet *ifp = vsi->ifp; struct tx_ring *txr = &que->txr; u32 reg, icr0, mask; bool more_tx, more_rx; ++que->irqs; /* Protect against spurious interrupts */ if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) return; icr0 = rd32(hw, I40E_PFINT_ICR0); reg = rd32(hw, I40E_PFINT_DYN_CTL0); reg = reg | I40E_PFINT_DYN_CTL0_CLEARPBA_MASK; wr32(hw, I40E_PFINT_DYN_CTL0, reg); mask = rd32(hw, I40E_PFINT_ICR0_ENA); #ifdef PCI_IOV if (icr0 & I40E_PFINT_ICR0_VFLR_MASK) taskqueue_enqueue(pf->tq, &pf->vflr_task); #endif if (icr0 & I40E_PFINT_ICR0_ADMINQ_MASK) { taskqueue_enqueue(pf->tq, &pf->adminq); return; } more_rx = ixl_rxeof(que, IXL_RX_LIMIT); IXL_TX_LOCK(txr); more_tx = ixl_txeof(que); if (!drbr_empty(vsi->ifp, txr->br)) more_tx = 1; IXL_TX_UNLOCK(txr); /* re-enable other interrupt causes */ wr32(hw, I40E_PFINT_ICR0_ENA, mask); /* And now the queues */ reg = rd32(hw, I40E_QINT_RQCTL(0)); reg |= I40E_QINT_RQCTL_CAUSE_ENA_MASK; wr32(hw, I40E_QINT_RQCTL(0), reg); reg = rd32(hw, I40E_QINT_TQCTL(0)); reg |= I40E_QINT_TQCTL_CAUSE_ENA_MASK; reg &= ~I40E_PFINT_ICR0_INTEVENT_MASK; wr32(hw, I40E_QINT_TQCTL(0), reg); ixl_enable_legacy(hw); return; } /********************************************************************* * * MSIX VSI Interrupt Service routine * **********************************************************************/ void ixl_msix_que(void *arg) { struct ixl_queue *que = arg; struct ixl_vsi *vsi = que->vsi; struct i40e_hw *hw = vsi->hw; struct tx_ring *txr = &que->txr; bool more_tx, more_rx; /* Protect against spurious interrupts */ if (!(vsi->ifp->if_drv_flags & IFF_DRV_RUNNING)) return; ++que->irqs; more_rx = ixl_rxeof(que, IXL_RX_LIMIT); IXL_TX_LOCK(txr); more_tx = ixl_txeof(que); /* ** Make certain that if the stack ** has anything queued the task gets ** scheduled to handle it. */ if (!drbr_empty(vsi->ifp, txr->br)) more_tx = 1; IXL_TX_UNLOCK(txr); ixl_set_queue_rx_itr(que); ixl_set_queue_tx_itr(que); if (more_tx || more_rx) taskqueue_enqueue(que->tq, &que->task); else ixl_enable_queue(hw, que->me); return; } /********************************************************************* * * MSIX Admin Queue Interrupt Service routine * **********************************************************************/ static void ixl_msix_adminq(void *arg) { struct ixl_pf *pf = arg; struct i40e_hw *hw = &pf->hw; u32 reg, mask; ++pf->admin_irq; reg = rd32(hw, I40E_PFINT_ICR0); mask = rd32(hw, I40E_PFINT_ICR0_ENA); /* Check on the cause */ if (reg & I40E_PFINT_ICR0_ADMINQ_MASK) mask &= ~I40E_PFINT_ICR0_ENA_ADMINQ_MASK; if (reg & I40E_PFINT_ICR0_MAL_DETECT_MASK) { ixl_handle_mdd_event(pf); mask &= ~I40E_PFINT_ICR0_ENA_MAL_DETECT_MASK; } #ifdef PCI_IOV if (reg & I40E_PFINT_ICR0_VFLR_MASK) { mask &= ~I40E_PFINT_ICR0_ENA_VFLR_MASK; taskqueue_enqueue(pf->tq, &pf->vflr_task); } #endif reg = rd32(hw, I40E_PFINT_DYN_CTL0); reg = reg | I40E_PFINT_DYN_CTL0_CLEARPBA_MASK; wr32(hw, I40E_PFINT_DYN_CTL0, reg); taskqueue_enqueue(pf->tq, &pf->adminq); return; } /********************************************************************* * * Media Ioctl callback * * This routine is called whenever the user queries the status of * the interface using ifconfig. * **********************************************************************/ static void ixl_media_status(struct ifnet * ifp, struct ifmediareq * ifmr) { struct ixl_vsi *vsi = ifp->if_softc; struct ixl_pf *pf = vsi->back; struct i40e_hw *hw = &pf->hw; INIT_DEBUGOUT("ixl_media_status: begin"); IXL_PF_LOCK(pf); hw->phy.get_link_info = TRUE; i40e_get_link_status(hw, &pf->link_up); ixl_update_link_status(pf); ifmr->ifm_status = IFM_AVALID; ifmr->ifm_active = IFM_ETHER; if (!pf->link_up) { IXL_PF_UNLOCK(pf); return; } ifmr->ifm_status |= IFM_ACTIVE; /* Hardware is always full-duplex */ ifmr->ifm_active |= IFM_FDX; switch (hw->phy.link_info.phy_type) { /* 100 M */ case I40E_PHY_TYPE_100BASE_TX: ifmr->ifm_active |= IFM_100_TX; break; /* 1 G */ case I40E_PHY_TYPE_1000BASE_T: ifmr->ifm_active |= IFM_1000_T; break; case I40E_PHY_TYPE_1000BASE_SX: ifmr->ifm_active |= IFM_1000_SX; break; case I40E_PHY_TYPE_1000BASE_LX: ifmr->ifm_active |= IFM_1000_LX; break; /* 10 G */ case I40E_PHY_TYPE_10GBASE_SFPP_CU: ifmr->ifm_active |= IFM_10G_TWINAX; break; case I40E_PHY_TYPE_10GBASE_SR: ifmr->ifm_active |= IFM_10G_SR; break; case I40E_PHY_TYPE_10GBASE_LR: ifmr->ifm_active |= IFM_10G_LR; break; case I40E_PHY_TYPE_10GBASE_T: ifmr->ifm_active |= IFM_10G_T; break; /* 40 G */ case I40E_PHY_TYPE_40GBASE_CR4: case I40E_PHY_TYPE_40GBASE_CR4_CU: ifmr->ifm_active |= IFM_40G_CR4; break; case I40E_PHY_TYPE_40GBASE_SR4: ifmr->ifm_active |= IFM_40G_SR4; break; case I40E_PHY_TYPE_40GBASE_LR4: ifmr->ifm_active |= IFM_40G_LR4; break; #ifndef IFM_ETH_XTYPE case I40E_PHY_TYPE_1000BASE_KX: ifmr->ifm_active |= IFM_1000_CX; break; case I40E_PHY_TYPE_10GBASE_CR1_CU: case I40E_PHY_TYPE_10GBASE_CR1: ifmr->ifm_active |= IFM_10G_TWINAX; break; case I40E_PHY_TYPE_10GBASE_KX4: ifmr->ifm_active |= IFM_10G_CX4; break; case I40E_PHY_TYPE_10GBASE_KR: ifmr->ifm_active |= IFM_10G_SR; break; case I40E_PHY_TYPE_40GBASE_KR4: case I40E_PHY_TYPE_XLPPI: ifmr->ifm_active |= IFM_40G_SR4; break; #else case I40E_PHY_TYPE_1000BASE_KX: ifmr->ifm_active |= IFM_1000_KX; break; /* ERJ: What's the difference between these? */ case I40E_PHY_TYPE_10GBASE_CR1_CU: case I40E_PHY_TYPE_10GBASE_CR1: ifmr->ifm_active |= IFM_10G_CR1; break; case I40E_PHY_TYPE_10GBASE_KX4: ifmr->ifm_active |= IFM_10G_KX4; break; case I40E_PHY_TYPE_10GBASE_KR: ifmr->ifm_active |= IFM_10G_KR; break; case I40E_PHY_TYPE_20GBASE_KR2: ifmr->ifm_active |= IFM_20G_KR2; break; case I40E_PHY_TYPE_40GBASE_KR4: ifmr->ifm_active |= IFM_40G_KR4; break; case I40E_PHY_TYPE_XLPPI: ifmr->ifm_active |= IFM_40G_XLPPI; break; #endif default: ifmr->ifm_active |= IFM_UNKNOWN; break; } /* Report flow control status as well */ if (hw->phy.link_info.an_info & I40E_AQ_LINK_PAUSE_TX) ifmr->ifm_active |= IFM_ETH_TXPAUSE; if (hw->phy.link_info.an_info & I40E_AQ_LINK_PAUSE_RX) ifmr->ifm_active |= IFM_ETH_RXPAUSE; IXL_PF_UNLOCK(pf); return; } /********************************************************************* * * Media Ioctl callback * * This routine is called when the user changes speed/duplex using * media/mediopt option with ifconfig. * **********************************************************************/ static int ixl_media_change(struct ifnet * ifp) { struct ixl_vsi *vsi = ifp->if_softc; struct ifmedia *ifm = &vsi->media; INIT_DEBUGOUT("ixl_media_change: begin"); if (IFM_TYPE(ifm->ifm_media) != IFM_ETHER) return (EINVAL); if_printf(ifp, "Media change is currently not supported.\n"); return (ENODEV); } #ifdef IXL_FDIR /* ** ATR: Application Targetted Receive - creates a filter ** based on TX flow info that will keep the receive ** portion of the flow on the same queue. Based on the ** implementation this is only available for TCP connections */ void ixl_atr(struct ixl_queue *que, struct tcphdr *th, int etype) { struct ixl_vsi *vsi = que->vsi; struct tx_ring *txr = &que->txr; struct i40e_filter_program_desc *FDIR; u32 ptype, dtype; int idx; /* check if ATR is enabled and sample rate */ if ((!ixl_enable_fdir) || (!txr->atr_rate)) return; /* ** We sample all TCP SYN/FIN packets, ** or at the selected sample rate */ txr->atr_count++; if (((th->th_flags & (TH_FIN | TH_SYN)) == 0) && (txr->atr_count < txr->atr_rate)) return; txr->atr_count = 0; /* Get a descriptor to use */ idx = txr->next_avail; FDIR = (struct i40e_filter_program_desc *) &txr->base[idx]; if (++idx == que->num_desc) idx = 0; txr->avail--; txr->next_avail = idx; ptype = (que->me << I40E_TXD_FLTR_QW0_QINDEX_SHIFT) & I40E_TXD_FLTR_QW0_QINDEX_MASK; ptype |= (etype == ETHERTYPE_IP) ? (I40E_FILTER_PCTYPE_NONF_IPV4_TCP << I40E_TXD_FLTR_QW0_PCTYPE_SHIFT) : (I40E_FILTER_PCTYPE_NONF_IPV6_TCP << I40E_TXD_FLTR_QW0_PCTYPE_SHIFT); ptype |= vsi->id << I40E_TXD_FLTR_QW0_DEST_VSI_SHIFT; dtype = I40E_TX_DESC_DTYPE_FILTER_PROG; /* ** We use the TCP TH_FIN as a trigger to remove ** the filter, otherwise its an update. */ dtype |= (th->th_flags & TH_FIN) ? (I40E_FILTER_PROGRAM_DESC_PCMD_REMOVE << I40E_TXD_FLTR_QW1_PCMD_SHIFT) : (I40E_FILTER_PROGRAM_DESC_PCMD_ADD_UPDATE << I40E_TXD_FLTR_QW1_PCMD_SHIFT); dtype |= I40E_FILTER_PROGRAM_DESC_DEST_DIRECT_PACKET_QINDEX << I40E_TXD_FLTR_QW1_DEST_SHIFT; dtype |= I40E_FILTER_PROGRAM_DESC_FD_STATUS_FD_ID << I40E_TXD_FLTR_QW1_FD_STATUS_SHIFT; FDIR->qindex_flex_ptype_vsi = htole32(ptype); FDIR->dtype_cmd_cntindex = htole32(dtype); return; } #endif static void ixl_set_promisc(struct ixl_vsi *vsi) { struct ifnet *ifp = vsi->ifp; struct i40e_hw *hw = vsi->hw; int err, mcnt = 0; bool uni = FALSE, multi = FALSE; if (ifp->if_flags & IFF_ALLMULTI) multi = TRUE; else { /* Need to count the multicast addresses */ struct ifmultiaddr *ifma; if_maddr_rlock(ifp); TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; if (mcnt == MAX_MULTICAST_ADDR) break; mcnt++; } if_maddr_runlock(ifp); } if (mcnt >= MAX_MULTICAST_ADDR) multi = TRUE; if (ifp->if_flags & IFF_PROMISC) uni = TRUE; err = i40e_aq_set_vsi_unicast_promiscuous(hw, vsi->seid, uni, NULL); err = i40e_aq_set_vsi_multicast_promiscuous(hw, vsi->seid, multi, NULL); return; } /********************************************************************* * Filter Routines * * Routines for multicast and vlan filter management. * *********************************************************************/ static void ixl_add_multi(struct ixl_vsi *vsi) { struct ifmultiaddr *ifma; struct ifnet *ifp = vsi->ifp; struct i40e_hw *hw = vsi->hw; int mcnt = 0, flags; IOCTL_DEBUGOUT("ixl_add_multi: begin"); if_maddr_rlock(ifp); /* ** First just get a count, to decide if we ** we simply use multicast promiscuous. */ TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; mcnt++; } if_maddr_runlock(ifp); if (__predict_false(mcnt >= MAX_MULTICAST_ADDR)) { /* delete existing MC filters */ ixl_del_hw_filters(vsi, mcnt); i40e_aq_set_vsi_multicast_promiscuous(hw, vsi->seid, TRUE, NULL); return; } mcnt = 0; if_maddr_rlock(ifp); TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; ixl_add_mc_filter(vsi, (u8*)LLADDR((struct sockaddr_dl *) ifma->ifma_addr)); mcnt++; } if_maddr_runlock(ifp); if (mcnt > 0) { flags = (IXL_FILTER_ADD | IXL_FILTER_USED | IXL_FILTER_MC); ixl_add_hw_filters(vsi, flags, mcnt); } IOCTL_DEBUGOUT("ixl_add_multi: end"); return; } static void ixl_del_multi(struct ixl_vsi *vsi) { struct ifnet *ifp = vsi->ifp; struct ifmultiaddr *ifma; struct ixl_mac_filter *f; int mcnt = 0; bool match = FALSE; IOCTL_DEBUGOUT("ixl_del_multi: begin"); /* Search for removed multicast addresses */ if_maddr_rlock(ifp); SLIST_FOREACH(f, &vsi->ftl, next) { if ((f->flags & IXL_FILTER_USED) && (f->flags & IXL_FILTER_MC)) { match = FALSE; TAILQ_FOREACH(ifma, &ifp->if_multiaddrs, ifma_link) { if (ifma->ifma_addr->sa_family != AF_LINK) continue; u8 *mc_addr = (u8 *)LLADDR((struct sockaddr_dl *)ifma->ifma_addr); if (cmp_etheraddr(f->macaddr, mc_addr)) { match = TRUE; break; } } if (match == FALSE) { f->flags |= IXL_FILTER_DEL; mcnt++; } } } if_maddr_runlock(ifp); if (mcnt > 0) ixl_del_hw_filters(vsi, mcnt); } /********************************************************************* * Timer routine * * This routine checks for link status,updates statistics, * and runs the watchdog check. * **********************************************************************/ static void ixl_local_timer(void *arg) { struct ixl_pf *pf = arg; struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; device_t dev = pf->dev; int hung = 0; u32 mask; mtx_assert(&pf->pf_mtx, MA_OWNED); /* Fire off the adminq task */ taskqueue_enqueue(pf->tq, &pf->adminq); /* Update stats */ ixl_update_stats_counters(pf); /* ** Check status of the queues */ mask = (I40E_PFINT_DYN_CTLN_INTENA_MASK | I40E_PFINT_DYN_CTLN_SWINT_TRIG_MASK); for (int i = 0; i < vsi->num_queues; i++,que++) { /* Any queues with outstanding work get a sw irq */ if (que->busy) wr32(hw, I40E_PFINT_DYN_CTLN(que->me), mask); /* ** Each time txeof runs without cleaning, but there ** are uncleaned descriptors it increments busy. If ** we get to 5 we declare it hung. */ if (que->busy == IXL_QUEUE_HUNG) { ++hung; /* Mark the queue as inactive */ vsi->active_queues &= ~((u64)1 << que->me); continue; } else { /* Check if we've come back from hung */ if ((vsi->active_queues & ((u64)1 << que->me)) == 0) vsi->active_queues |= ((u64)1 << que->me); } if (que->busy >= IXL_MAX_TX_BUSY) { #ifdef IXL_DEBUG device_printf(dev,"Warning queue %d " "appears to be hung!\n", i); #endif que->busy = IXL_QUEUE_HUNG; ++hung; } } /* Only reinit if all queues show hung */ if (hung == vsi->num_queues) goto hung; callout_reset(&pf->timer, hz, ixl_local_timer, pf); return; hung: device_printf(dev, "Local Timer: HANG DETECT - Resetting!!\n"); ixl_init_locked(pf); } /* ** Note: this routine updates the OS on the link state ** the real check of the hardware only happens with ** a link interrupt. */ static void ixl_update_link_status(struct ixl_pf *pf) { struct ixl_vsi *vsi = &pf->vsi; struct i40e_hw *hw = &pf->hw; struct ifnet *ifp = vsi->ifp; device_t dev = pf->dev; if (pf->link_up){ if (vsi->link_active == FALSE) { pf->fc = hw->fc.current_mode; if (bootverbose) { device_printf(dev,"Link is up %d Gbps %s," " Flow Control: %s\n", ((pf->link_speed == I40E_LINK_SPEED_40GB)? 40:10), "Full Duplex", ixl_fc_string[pf->fc]); } vsi->link_active = TRUE; /* ** Warn user if link speed on NPAR enabled ** partition is not at least 10GB */ if (hw->func_caps.npar_enable && (hw->phy.link_info.link_speed == I40E_LINK_SPEED_1GB || hw->phy.link_info.link_speed == I40E_LINK_SPEED_100MB)) device_printf(dev, "The partition detected" "link speed that is less than 10Gbps\n"); if_link_state_change(ifp, LINK_STATE_UP); } } else { /* Link down */ if (vsi->link_active == TRUE) { if (bootverbose) device_printf(dev,"Link is Down\n"); if_link_state_change(ifp, LINK_STATE_DOWN); vsi->link_active = FALSE; } } return; } /********************************************************************* * * This routine disables all traffic on the adapter by issuing a * global reset on the MAC and deallocates TX/RX buffers. * **********************************************************************/ static void ixl_stop(struct ixl_pf *pf) { struct ixl_vsi *vsi = &pf->vsi; struct ifnet *ifp = vsi->ifp; mtx_assert(&pf->pf_mtx, MA_OWNED); INIT_DEBUGOUT("ixl_stop: begin\n"); if (pf->num_vfs == 0) ixl_disable_intr(vsi); else ixl_disable_rings_intr(vsi); ixl_disable_rings(vsi); /* Tell the stack that the interface is no longer active */ ifp->if_drv_flags &= ~(IFF_DRV_RUNNING | IFF_DRV_OACTIVE); /* Stop the local timer */ callout_stop(&pf->timer); return; } /********************************************************************* * * Setup MSIX Interrupt resources and handlers for the VSI * **********************************************************************/ static int ixl_assign_vsi_legacy(struct ixl_pf *pf) { device_t dev = pf->dev; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; int error, rid = 0; if (pf->msix == 1) rid = 1; pf->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (pf->res == NULL) { device_printf(dev,"Unable to allocate" " bus resource: vsi legacy/msi interrupt\n"); return (ENXIO); } /* Set the handler function */ error = bus_setup_intr(dev, pf->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, ixl_intr, pf, &pf->tag); if (error) { pf->res = NULL; device_printf(dev, "Failed to register legacy/msi handler"); return (error); } bus_describe_intr(dev, pf->res, pf->tag, "irq0"); TASK_INIT(&que->tx_task, 0, ixl_deferred_mq_start, que); TASK_INIT(&que->task, 0, ixl_handle_que, que); que->tq = taskqueue_create_fast("ixl_que", M_NOWAIT, taskqueue_thread_enqueue, &que->tq); taskqueue_start_threads(&que->tq, 1, PI_NET, "%s que", device_get_nameunit(dev)); TASK_INIT(&pf->adminq, 0, ixl_do_adminq, pf); #ifdef PCI_IOV TASK_INIT(&pf->vflr_task, 0, ixl_handle_vflr, pf); #endif pf->tq = taskqueue_create_fast("ixl_adm", M_NOWAIT, taskqueue_thread_enqueue, &pf->tq); taskqueue_start_threads(&pf->tq, 1, PI_NET, "%s adminq", device_get_nameunit(dev)); return (0); } /********************************************************************* * * Setup MSIX Interrupt resources and handlers for the VSI * **********************************************************************/ static int ixl_assign_vsi_msix(struct ixl_pf *pf) { device_t dev = pf->dev; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; struct tx_ring *txr; int error, rid, vector = 0; #ifdef RSS cpuset_t cpu_mask; #endif /* Admin Que is vector 0*/ rid = vector + 1; pf->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (!pf->res) { device_printf(dev,"Unable to allocate" " bus resource: Adminq interrupt [%d]\n", rid); return (ENXIO); } /* Set the adminq vector and handler */ error = bus_setup_intr(dev, pf->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, ixl_msix_adminq, pf, &pf->tag); if (error) { pf->res = NULL; device_printf(dev, "Failed to register Admin que handler"); return (error); } bus_describe_intr(dev, pf->res, pf->tag, "aq"); pf->admvec = vector; /* Tasklet for Admin Queue */ TASK_INIT(&pf->adminq, 0, ixl_do_adminq, pf); #ifdef PCI_IOV TASK_INIT(&pf->vflr_task, 0, ixl_handle_vflr, pf); #endif pf->tq = taskqueue_create_fast("ixl_adm", M_NOWAIT, taskqueue_thread_enqueue, &pf->tq); taskqueue_start_threads(&pf->tq, 1, PI_NET, "%s adminq", device_get_nameunit(pf->dev)); ++vector; /* Now set up the stations */ for (int i = 0; i < vsi->num_queues; i++, vector++, que++) { int cpu_id = i; rid = vector + 1; txr = &que->txr; que->res = bus_alloc_resource_any(dev, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (que->res == NULL) { device_printf(dev,"Unable to allocate" " bus resource: que interrupt [%d]\n", vector); return (ENXIO); } /* Set the handler function */ error = bus_setup_intr(dev, que->res, INTR_TYPE_NET | INTR_MPSAFE, NULL, ixl_msix_que, que, &que->tag); if (error) { que->res = NULL; device_printf(dev, "Failed to register que handler"); return (error); } bus_describe_intr(dev, que->res, que->tag, "q%d", i); /* Bind the vector to a CPU */ #ifdef RSS cpu_id = rss_getcpu(i % rss_getnumbuckets()); #endif bus_bind_intr(dev, que->res, cpu_id); que->msix = vector; TASK_INIT(&que->tx_task, 0, ixl_deferred_mq_start, que); TASK_INIT(&que->task, 0, ixl_handle_que, que); que->tq = taskqueue_create_fast("ixl_que", M_NOWAIT, taskqueue_thread_enqueue, &que->tq); #ifdef RSS CPU_SETOF(cpu_id, &cpu_mask); taskqueue_start_threads_cpuset(&que->tq, 1, PI_NET, &cpu_mask, "%s (bucket %d)", device_get_nameunit(dev), cpu_id); #else taskqueue_start_threads(&que->tq, 1, PI_NET, "%s que", device_get_nameunit(dev)); #endif } return (0); } /* * Allocate MSI/X vectors */ static int ixl_init_msix(struct ixl_pf *pf) { device_t dev = pf->dev; int rid, want, vectors, queues, available; /* Override by tuneable */ if (ixl_enable_msix == 0) goto msi; /* ** When used in a virtualized environment ** PCI BUSMASTER capability may not be set ** so explicity set it here and rewrite ** the ENABLE in the MSIX control register ** at this point to cause the host to ** successfully initialize us. */ { u16 pci_cmd_word; int msix_ctrl; pci_cmd_word = pci_read_config(dev, PCIR_COMMAND, 2); pci_cmd_word |= PCIM_CMD_BUSMASTEREN; pci_write_config(dev, PCIR_COMMAND, pci_cmd_word, 2); pci_find_cap(dev, PCIY_MSIX, &rid); rid += PCIR_MSIX_CTRL; msix_ctrl = pci_read_config(dev, rid, 2); msix_ctrl |= PCIM_MSIXCTRL_MSIX_ENABLE; pci_write_config(dev, rid, msix_ctrl, 2); } /* First try MSI/X */ rid = PCIR_BAR(IXL_BAR); pf->msix_mem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (!pf->msix_mem) { /* May not be enabled */ device_printf(pf->dev, "Unable to map MSIX table \n"); goto msi; } available = pci_msix_count(dev); if (available == 0) { /* system has msix disabled */ bus_release_resource(dev, SYS_RES_MEMORY, rid, pf->msix_mem); pf->msix_mem = NULL; goto msi; } /* Figure out a reasonable auto config value */ queues = (mp_ncpus > (available - 1)) ? (available - 1) : mp_ncpus; /* Override with hardcoded value if sane */ if ((ixl_max_queues != 0) && (ixl_max_queues <= queues)) queues = ixl_max_queues; #ifdef RSS /* If we're doing RSS, clamp at the number of RSS buckets */ if (queues > rss_getnumbuckets()) queues = rss_getnumbuckets(); #endif /* ** Want one vector (RX/TX pair) per queue ** plus an additional for the admin queue. */ want = queues + 1; if (want <= available) /* Have enough */ vectors = want; else { device_printf(pf->dev, "MSIX Configuration Problem, " "%d vectors available but %d wanted!\n", available, want); return (0); /* Will go to Legacy setup */ } if (pci_alloc_msix(dev, &vectors) == 0) { device_printf(pf->dev, "Using MSIX interrupts with %d vectors\n", vectors); pf->msix = vectors; pf->vsi.num_queues = queues; #ifdef RSS /* * If we're doing RSS, the number of queues needs to * match the number of RSS buckets that are configured. * * + If there's more queues than RSS buckets, we'll end * up with queues that get no traffic. * * + If there's more RSS buckets than queues, we'll end * up having multiple RSS buckets map to the same queue, * so there'll be some contention. */ if (queues != rss_getnumbuckets()) { device_printf(dev, "%s: queues (%d) != RSS buckets (%d)" "; performance will be impacted.\n", __func__, queues, rss_getnumbuckets()); } #endif return (vectors); } msi: vectors = pci_msi_count(dev); pf->vsi.num_queues = 1; pf->msix = 1; ixl_max_queues = 1; ixl_enable_msix = 0; if (vectors == 1 && pci_alloc_msi(dev, &vectors) == 0) device_printf(pf->dev,"Using an MSI interrupt\n"); else { pf->msix = 0; device_printf(pf->dev,"Using a Legacy interrupt\n"); } return (vectors); } /* * Plumb MSI/X vectors */ static void ixl_configure_msix(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; u32 reg; u16 vector = 1; /* First set up the adminq - vector 0 */ wr32(hw, I40E_PFINT_ICR0_ENA, 0); /* disable all */ rd32(hw, I40E_PFINT_ICR0); /* read to clear */ reg = I40E_PFINT_ICR0_ENA_ECC_ERR_MASK | I40E_PFINT_ICR0_ENA_GRST_MASK | I40E_PFINT_ICR0_HMC_ERR_MASK | I40E_PFINT_ICR0_ENA_ADMINQ_MASK | I40E_PFINT_ICR0_ENA_MAL_DETECT_MASK | I40E_PFINT_ICR0_ENA_VFLR_MASK | I40E_PFINT_ICR0_ENA_PCI_EXCEPTION_MASK; wr32(hw, I40E_PFINT_ICR0_ENA, reg); wr32(hw, I40E_PFINT_LNKLST0, 0x7FF); wr32(hw, I40E_PFINT_ITR0(IXL_RX_ITR), 0x003E); wr32(hw, I40E_PFINT_DYN_CTL0, I40E_PFINT_DYN_CTL0_SW_ITR_INDX_MASK | I40E_PFINT_DYN_CTL0_INTENA_MSK_MASK); wr32(hw, I40E_PFINT_STAT_CTL0, 0); /* Next configure the queues */ for (int i = 0; i < vsi->num_queues; i++, vector++) { wr32(hw, I40E_PFINT_DYN_CTLN(i), i); wr32(hw, I40E_PFINT_LNKLSTN(i), i); reg = I40E_QINT_RQCTL_CAUSE_ENA_MASK | (IXL_RX_ITR << I40E_QINT_RQCTL_ITR_INDX_SHIFT) | (vector << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) | (i << I40E_QINT_RQCTL_NEXTQ_INDX_SHIFT) | (I40E_QUEUE_TYPE_TX << I40E_QINT_RQCTL_NEXTQ_TYPE_SHIFT); wr32(hw, I40E_QINT_RQCTL(i), reg); reg = I40E_QINT_TQCTL_CAUSE_ENA_MASK | (IXL_TX_ITR << I40E_QINT_TQCTL_ITR_INDX_SHIFT) | (vector << I40E_QINT_TQCTL_MSIX_INDX_SHIFT) | ((i+1) << I40E_QINT_TQCTL_NEXTQ_INDX_SHIFT) | (I40E_QUEUE_TYPE_RX << I40E_QINT_TQCTL_NEXTQ_TYPE_SHIFT); if (i == (vsi->num_queues - 1)) reg |= (IXL_QUEUE_EOL << I40E_QINT_TQCTL_NEXTQ_INDX_SHIFT); wr32(hw, I40E_QINT_TQCTL(i), reg); } } /* * Configure for MSI single vector operation */ static void ixl_configure_legacy(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; u32 reg; wr32(hw, I40E_PFINT_ITR0(0), 0); wr32(hw, I40E_PFINT_ITR0(1), 0); /* Setup "other" causes */ reg = I40E_PFINT_ICR0_ENA_ECC_ERR_MASK | I40E_PFINT_ICR0_ENA_MAL_DETECT_MASK | I40E_PFINT_ICR0_ENA_GRST_MASK | I40E_PFINT_ICR0_ENA_PCI_EXCEPTION_MASK | I40E_PFINT_ICR0_ENA_GPIO_MASK | I40E_PFINT_ICR0_ENA_LINK_STAT_CHANGE_MASK | I40E_PFINT_ICR0_ENA_HMC_ERR_MASK | I40E_PFINT_ICR0_ENA_PE_CRITERR_MASK | I40E_PFINT_ICR0_ENA_VFLR_MASK | I40E_PFINT_ICR0_ENA_ADMINQ_MASK ; wr32(hw, I40E_PFINT_ICR0_ENA, reg); /* SW_ITR_IDX = 0, but don't change INTENA */ wr32(hw, I40E_PFINT_DYN_CTL0, I40E_PFINT_DYN_CTLN_SW_ITR_INDX_MASK | I40E_PFINT_DYN_CTLN_INTENA_MSK_MASK); /* SW_ITR_IDX = 0, OTHER_ITR_IDX = 0 */ wr32(hw, I40E_PFINT_STAT_CTL0, 0); /* FIRSTQ_INDX = 0, FIRSTQ_TYPE = 0 (rx) */ wr32(hw, I40E_PFINT_LNKLST0, 0); /* Associate the queue pair to the vector and enable the q int */ reg = I40E_QINT_RQCTL_CAUSE_ENA_MASK | (IXL_RX_ITR << I40E_QINT_RQCTL_ITR_INDX_SHIFT) | (I40E_QUEUE_TYPE_TX << I40E_QINT_TQCTL_NEXTQ_TYPE_SHIFT); wr32(hw, I40E_QINT_RQCTL(0), reg); reg = I40E_QINT_TQCTL_CAUSE_ENA_MASK | (IXL_TX_ITR << I40E_QINT_TQCTL_ITR_INDX_SHIFT) | (IXL_QUEUE_EOL << I40E_QINT_TQCTL_NEXTQ_INDX_SHIFT); wr32(hw, I40E_QINT_TQCTL(0), reg); /* Next enable the queue pair */ reg = rd32(hw, I40E_QTX_ENA(0)); reg |= I40E_QTX_ENA_QENA_REQ_MASK; wr32(hw, I40E_QTX_ENA(0), reg); reg = rd32(hw, I40E_QRX_ENA(0)); reg |= I40E_QRX_ENA_QENA_REQ_MASK; wr32(hw, I40E_QRX_ENA(0), reg); } /* * Set the Initial ITR state */ static void ixl_configure_itr(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; vsi->rx_itr_setting = ixl_rx_itr; if (ixl_dynamic_rx_itr) vsi->rx_itr_setting |= IXL_ITR_DYNAMIC; vsi->tx_itr_setting = ixl_tx_itr; if (ixl_dynamic_tx_itr) vsi->tx_itr_setting |= IXL_ITR_DYNAMIC; for (int i = 0; i < vsi->num_queues; i++, que++) { struct tx_ring *txr = &que->txr; struct rx_ring *rxr = &que->rxr; wr32(hw, I40E_PFINT_ITRN(IXL_RX_ITR, i), vsi->rx_itr_setting); rxr->itr = vsi->rx_itr_setting; rxr->latency = IXL_AVE_LATENCY; wr32(hw, I40E_PFINT_ITRN(IXL_TX_ITR, i), vsi->tx_itr_setting); txr->itr = vsi->tx_itr_setting; txr->latency = IXL_AVE_LATENCY; } } static int ixl_allocate_pci_resources(struct ixl_pf *pf) { int rid; device_t dev = pf->dev; rid = PCIR_BAR(0); pf->pci_mem = bus_alloc_resource_any(dev, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (!(pf->pci_mem)) { device_printf(dev,"Unable to allocate bus resource: memory\n"); return (ENXIO); } pf->osdep.mem_bus_space_tag = rman_get_bustag(pf->pci_mem); pf->osdep.mem_bus_space_handle = rman_get_bushandle(pf->pci_mem); pf->osdep.mem_bus_space_size = rman_get_size(pf->pci_mem); pf->osdep.flush_reg = I40E_GLGEN_STAT; pf->hw.hw_addr = (u8 *) &pf->osdep.mem_bus_space_handle; pf->hw.back = &pf->osdep; /* ** Now setup MSI or MSI/X, should ** return us the number of supported ** vectors. (Will be 1 for MSI) */ pf->msix = ixl_init_msix(pf); return (0); } static void ixl_free_pci_resources(struct ixl_pf * pf) { struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; device_t dev = pf->dev; int rid, memrid; memrid = PCIR_BAR(IXL_BAR); /* We may get here before stations are setup */ if ((!ixl_enable_msix) || (que == NULL)) goto early; /* ** Release all msix VSI resources: */ for (int i = 0; i < vsi->num_queues; i++, que++) { rid = que->msix + 1; if (que->tag != NULL) { bus_teardown_intr(dev, que->res, que->tag); que->tag = NULL; } if (que->res != NULL) bus_release_resource(dev, SYS_RES_IRQ, rid, que->res); } early: /* Clean the AdminQ interrupt last */ if (pf->admvec) /* we are doing MSIX */ rid = pf->admvec + 1; else (pf->msix != 0) ? (rid = 1):(rid = 0); if (pf->tag != NULL) { bus_teardown_intr(dev, pf->res, pf->tag); pf->tag = NULL; } if (pf->res != NULL) bus_release_resource(dev, SYS_RES_IRQ, rid, pf->res); if (pf->msix) pci_release_msi(dev); if (pf->msix_mem != NULL) bus_release_resource(dev, SYS_RES_MEMORY, memrid, pf->msix_mem); if (pf->pci_mem != NULL) bus_release_resource(dev, SYS_RES_MEMORY, PCIR_BAR(0), pf->pci_mem); return; } static void ixl_add_ifmedia(struct ixl_vsi *vsi, u32 phy_type) { /* Display supported media types */ if (phy_type & (1 << I40E_PHY_TYPE_100BASE_TX)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_100_TX, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_1000BASE_T)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_1000_T, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_1000BASE_SX)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_1000_SX, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_1000BASE_LX)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_1000_LX, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_XAUI) || phy_type & (1 << I40E_PHY_TYPE_XFI) || phy_type & (1 << I40E_PHY_TYPE_10GBASE_SFPP_CU)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_TWINAX, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_SR)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_SR, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_LR)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_LR, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_T)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_T, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_40GBASE_CR4) || phy_type & (1 << I40E_PHY_TYPE_40GBASE_CR4_CU) || phy_type & (1 << I40E_PHY_TYPE_40GBASE_AOC) || phy_type & (1 << I40E_PHY_TYPE_XLAUI) || phy_type & (1 << I40E_PHY_TYPE_40GBASE_KR4)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_40G_CR4, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_40GBASE_SR4)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_40G_SR4, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_40GBASE_LR4)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_40G_LR4, 0, NULL); #ifndef IFM_ETH_XTYPE if (phy_type & (1 << I40E_PHY_TYPE_1000BASE_KX)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_1000_CX, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_CR1_CU) || phy_type & (1 << I40E_PHY_TYPE_10GBASE_CR1) || phy_type & (1 << I40E_PHY_TYPE_10GBASE_AOC) || phy_type & (1 << I40E_PHY_TYPE_SFI)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_TWINAX, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_KX4)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_CX4, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_KR)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_SR, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_40GBASE_KR4)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_40G_SR4, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_XLPPI)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_40G_CR4, 0, NULL); #else if (phy_type & (1 << I40E_PHY_TYPE_1000BASE_KX)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_1000_KX, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_CR1_CU) || phy_type & (1 << I40E_PHY_TYPE_10GBASE_CR1)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_CR1, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_AOC)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_TWINAX_LONG, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_SFI)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_SFI, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_KX4)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_KX4, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_10GBASE_KR)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_10G_KR, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_20GBASE_KR2)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_20G_KR2, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_40GBASE_KR4)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_40G_KR4, 0, NULL); if (phy_type & (1 << I40E_PHY_TYPE_XLPPI)) ifmedia_add(&vsi->media, IFM_ETHER | IFM_40G_XLPPI, 0, NULL); #endif } /********************************************************************* * * Setup networking device structure and register an interface. * **********************************************************************/ static int ixl_setup_interface(device_t dev, struct ixl_vsi *vsi) { struct ifnet *ifp; struct i40e_hw *hw = vsi->hw; struct ixl_queue *que = vsi->queues; struct i40e_aq_get_phy_abilities_resp abilities; enum i40e_status_code aq_error = 0; INIT_DEBUGOUT("ixl_setup_interface: begin"); ifp = vsi->ifp = if_alloc(IFT_ETHER); if (ifp == NULL) { device_printf(dev, "can not allocate ifnet structure\n"); return (-1); } if_initname(ifp, device_get_name(dev), device_get_unit(dev)); ifp->if_mtu = ETHERMTU; ifp->if_baudrate = IF_Gbps(40); ifp->if_init = ixl_init; ifp->if_softc = vsi; ifp->if_flags = IFF_BROADCAST | IFF_SIMPLEX | IFF_MULTICAST; ifp->if_ioctl = ixl_ioctl; #if __FreeBSD_version >= 1100036 if_setgetcounterfn(ifp, ixl_get_counter); #endif ifp->if_transmit = ixl_mq_start; ifp->if_qflush = ixl_qflush; ifp->if_snd.ifq_maxlen = que->num_desc - 2; vsi->max_frame_size = ifp->if_mtu + ETHER_HDR_LEN + ETHER_CRC_LEN + ETHER_VLAN_ENCAP_LEN; /* * Tell the upper layer(s) we support long frames. */ ifp->if_hdrlen = sizeof(struct ether_vlan_header); ifp->if_capabilities |= IFCAP_HWCSUM; ifp->if_capabilities |= IFCAP_HWCSUM_IPV6; ifp->if_capabilities |= IFCAP_TSO; ifp->if_capabilities |= IFCAP_JUMBO_MTU; ifp->if_capabilities |= IFCAP_LRO; /* VLAN capabilties */ ifp->if_capabilities |= IFCAP_VLAN_HWTAGGING | IFCAP_VLAN_HWTSO | IFCAP_VLAN_MTU | IFCAP_VLAN_HWCSUM; ifp->if_capenable = ifp->if_capabilities; /* ** Don't turn this on by default, if vlans are ** created on another pseudo device (eg. lagg) ** then vlan events are not passed thru, breaking ** operation, but with HW FILTER off it works. If ** using vlans directly on the ixl driver you can ** enable this and get full hardware tag filtering. */ ifp->if_capabilities |= IFCAP_VLAN_HWFILTER; /* * Specify the media types supported by this adapter and register * callbacks to update media and link information */ ifmedia_init(&vsi->media, IFM_IMASK, ixl_media_change, ixl_media_status); aq_error = i40e_aq_get_phy_capabilities(hw, FALSE, TRUE, &abilities, NULL); /* May need delay to detect fiber correctly */ if (aq_error == I40E_ERR_UNKNOWN_PHY) { i40e_msec_delay(200); aq_error = i40e_aq_get_phy_capabilities(hw, FALSE, TRUE, &abilities, NULL); } if (aq_error) { if (aq_error == I40E_ERR_UNKNOWN_PHY) device_printf(dev, "Unknown PHY type detected!\n"); else device_printf(dev, "Error getting supported media types, err %d," " AQ error %d\n", aq_error, hw->aq.asq_last_status); return (0); } ixl_add_ifmedia(vsi, abilities.phy_type); /* Use autoselect media by default */ ifmedia_add(&vsi->media, IFM_ETHER | IFM_AUTO, 0, NULL); ifmedia_set(&vsi->media, IFM_ETHER | IFM_AUTO); ether_ifattach(ifp, hw->mac.addr); return (0); } /* ** Run when the Admin Queue gets a ** link transition interrupt. */ static void ixl_link_event(struct ixl_pf *pf, struct i40e_arq_event_info *e) { struct i40e_hw *hw = &pf->hw; struct i40e_aqc_get_link_status *status = (struct i40e_aqc_get_link_status *)&e->desc.params.raw; bool check; hw->phy.get_link_info = TRUE; i40e_get_link_status(hw, &check); pf->link_up = check; #ifdef IXL_DEBUG printf("Link is %s\n", check ? "up":"down"); #endif /* Report if Unqualified modules are found */ if ((status->link_info & I40E_AQ_MEDIA_AVAILABLE) && (!(status->an_info & I40E_AQ_QUALIFIED_MODULE)) && (!(status->link_info & I40E_AQ_LINK_UP))) device_printf(pf->dev, "Link failed because " "an unqualified module was detected\n"); return; } /********************************************************************* * * Get Firmware Switch configuration * - this will need to be more robust when more complex * switch configurations are enabled. * **********************************************************************/ static int ixl_switch_config(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; device_t dev = vsi->dev; struct i40e_aqc_get_switch_config_resp *sw_config; u8 aq_buf[I40E_AQ_LARGE_BUF]; int ret; u16 next = 0; memset(&aq_buf, 0, sizeof(aq_buf)); sw_config = (struct i40e_aqc_get_switch_config_resp *)aq_buf; ret = i40e_aq_get_switch_config(hw, sw_config, sizeof(aq_buf), &next, NULL); if (ret) { device_printf(dev,"aq_get_switch_config failed (ret=%d)!!\n", ret); return (ret); } #ifdef IXL_DEBUG device_printf(dev, "Switch config: header reported: %d in structure, %d total\n", sw_config->header.num_reported, sw_config->header.num_total); for (int i = 0; i < sw_config->header.num_reported; i++) { device_printf(dev, "%d: type=%d seid=%d uplink=%d downlink=%d\n", i, sw_config->element[i].element_type, sw_config->element[i].seid, sw_config->element[i].uplink_seid, sw_config->element[i].downlink_seid); } #endif /* Simplified due to a single VSI at the moment */ vsi->uplink_seid = sw_config->element[0].uplink_seid; vsi->downlink_seid = sw_config->element[0].downlink_seid; vsi->seid = sw_config->element[0].seid; return (ret); } /********************************************************************* * * Initialize the VSI: this handles contexts, which means things * like the number of descriptors, buffer size, * plus we init the rings thru this function. * **********************************************************************/ static int ixl_initialize_vsi(struct ixl_vsi *vsi) { struct ixl_pf *pf = vsi->back; struct ixl_queue *que = vsi->queues; device_t dev = vsi->dev; struct i40e_hw *hw = vsi->hw; struct i40e_vsi_context ctxt; int err = 0; memset(&ctxt, 0, sizeof(ctxt)); ctxt.seid = vsi->seid; if (pf->veb_seid != 0) ctxt.uplink_seid = pf->veb_seid; ctxt.pf_num = hw->pf_id; err = i40e_aq_get_vsi_params(hw, &ctxt, NULL); if (err) { device_printf(dev,"get vsi params failed %x!!\n", err); return (err); } #ifdef IXL_DEBUG printf("get_vsi_params: seid: %d, uplinkseid: %d, vsi_number: %d, " "vsis_allocated: %d, vsis_unallocated: %d, flags: 0x%x, " "pfnum: %d, vfnum: %d, stat idx: %d, enabled: %d\n", ctxt.seid, ctxt.uplink_seid, ctxt.vsi_number, ctxt.vsis_allocated, ctxt.vsis_unallocated, ctxt.flags, ctxt.pf_num, ctxt.vf_num, ctxt.info.stat_counter_idx, ctxt.info.up_enable_bits); #endif /* ** Set the queue and traffic class bits ** - when multiple traffic classes are supported ** this will need to be more robust. */ ctxt.info.valid_sections = I40E_AQ_VSI_PROP_QUEUE_MAP_VALID; ctxt.info.mapping_flags |= I40E_AQ_VSI_QUE_MAP_CONTIG; ctxt.info.queue_mapping[0] = 0; ctxt.info.tc_mapping[0] = 0x0800; /* Set VLAN receive stripping mode */ ctxt.info.valid_sections |= I40E_AQ_VSI_PROP_VLAN_VALID; ctxt.info.port_vlan_flags = I40E_AQ_VSI_PVLAN_MODE_ALL; if (vsi->ifp->if_capenable & IFCAP_VLAN_HWTAGGING) ctxt.info.port_vlan_flags |= I40E_AQ_VSI_PVLAN_EMOD_STR_BOTH; else ctxt.info.port_vlan_flags |= I40E_AQ_VSI_PVLAN_EMOD_NOTHING; /* Keep copy of VSI info in VSI for statistic counters */ memcpy(&vsi->info, &ctxt.info, sizeof(ctxt.info)); /* Reset VSI statistics */ ixl_vsi_reset_stats(vsi); vsi->hw_filters_add = 0; vsi->hw_filters_del = 0; ctxt.flags = htole16(I40E_AQ_VSI_TYPE_PF); err = i40e_aq_update_vsi_params(hw, &ctxt, NULL); if (err) { device_printf(dev,"update vsi params failed %x!!\n", hw->aq.asq_last_status); return (err); } for (int i = 0; i < vsi->num_queues; i++, que++) { struct tx_ring *txr = &que->txr; struct rx_ring *rxr = &que->rxr; struct i40e_hmc_obj_txq tctx; struct i40e_hmc_obj_rxq rctx; u32 txctl; u16 size; /* Setup the HMC TX Context */ size = que->num_desc * sizeof(struct i40e_tx_desc); memset(&tctx, 0, sizeof(struct i40e_hmc_obj_txq)); tctx.new_context = 1; tctx.base = (txr->dma.pa/IXL_TX_CTX_BASE_UNITS); tctx.qlen = que->num_desc; tctx.fc_ena = 0; tctx.rdylist = vsi->info.qs_handle[0]; /* index is TC */ /* Enable HEAD writeback */ tctx.head_wb_ena = 1; tctx.head_wb_addr = txr->dma.pa + (que->num_desc * sizeof(struct i40e_tx_desc)); tctx.rdylist_act = 0; err = i40e_clear_lan_tx_queue_context(hw, i); if (err) { device_printf(dev, "Unable to clear TX context\n"); break; } err = i40e_set_lan_tx_queue_context(hw, i, &tctx); if (err) { device_printf(dev, "Unable to set TX context\n"); break; } /* Associate the ring with this PF */ txctl = I40E_QTX_CTL_PF_QUEUE; txctl |= ((hw->pf_id << I40E_QTX_CTL_PF_INDX_SHIFT) & I40E_QTX_CTL_PF_INDX_MASK); wr32(hw, I40E_QTX_CTL(i), txctl); ixl_flush(hw); /* Do ring (re)init */ ixl_init_tx_ring(que); /* Next setup the HMC RX Context */ if (vsi->max_frame_size <= MCLBYTES) rxr->mbuf_sz = MCLBYTES; else rxr->mbuf_sz = MJUMPAGESIZE; u16 max_rxmax = rxr->mbuf_sz * hw->func_caps.rx_buf_chain_len; /* Set up an RX context for the HMC */ memset(&rctx, 0, sizeof(struct i40e_hmc_obj_rxq)); rctx.dbuff = rxr->mbuf_sz >> I40E_RXQ_CTX_DBUFF_SHIFT; /* ignore header split for now */ rctx.hbuff = 0 >> I40E_RXQ_CTX_HBUFF_SHIFT; rctx.rxmax = (vsi->max_frame_size < max_rxmax) ? vsi->max_frame_size : max_rxmax; rctx.dtype = 0; rctx.dsize = 1; /* do 32byte descriptors */ rctx.hsplit_0 = 0; /* no HDR split initially */ rctx.base = (rxr->dma.pa/IXL_RX_CTX_BASE_UNITS); rctx.qlen = que->num_desc; rctx.tphrdesc_ena = 1; rctx.tphwdesc_ena = 1; rctx.tphdata_ena = 0; rctx.tphhead_ena = 0; rctx.lrxqthresh = 2; rctx.crcstrip = 1; rctx.l2tsel = 1; rctx.showiv = 1; rctx.fc_ena = 0; rctx.prefena = 1; err = i40e_clear_lan_rx_queue_context(hw, i); if (err) { device_printf(dev, "Unable to clear RX context %d\n", i); break; } err = i40e_set_lan_rx_queue_context(hw, i, &rctx); if (err) { device_printf(dev, "Unable to set RX context %d\n", i); break; } err = ixl_init_rx_ring(que); if (err) { device_printf(dev, "Fail in init_rx_ring %d\n", i); break; } wr32(vsi->hw, I40E_QRX_TAIL(que->me), 0); #ifdef DEV_NETMAP /* preserve queue */ if (vsi->ifp->if_capenable & IFCAP_NETMAP) { struct netmap_adapter *na = NA(vsi->ifp); struct netmap_kring *kring = &na->rx_rings[i]; int t = na->num_rx_desc - 1 - nm_kr_rxspace(kring); wr32(vsi->hw, I40E_QRX_TAIL(que->me), t); } else #endif /* DEV_NETMAP */ wr32(vsi->hw, I40E_QRX_TAIL(que->me), que->num_desc - 1); } return (err); } /********************************************************************* * * Free all VSI structs. * **********************************************************************/ void ixl_free_vsi(struct ixl_vsi *vsi) { struct ixl_pf *pf = (struct ixl_pf *)vsi->back; struct ixl_queue *que = vsi->queues; /* Free station queues */ for (int i = 0; i < vsi->num_queues; i++, que++) { struct tx_ring *txr = &que->txr; struct rx_ring *rxr = &que->rxr; if (!mtx_initialized(&txr->mtx)) /* uninitialized */ continue; IXL_TX_LOCK(txr); ixl_free_que_tx(que); if (txr->base) i40e_free_dma_mem(&pf->hw, &txr->dma); IXL_TX_UNLOCK(txr); IXL_TX_LOCK_DESTROY(txr); if (!mtx_initialized(&rxr->mtx)) /* uninitialized */ continue; IXL_RX_LOCK(rxr); ixl_free_que_rx(que); if (rxr->base) i40e_free_dma_mem(&pf->hw, &rxr->dma); IXL_RX_UNLOCK(rxr); IXL_RX_LOCK_DESTROY(rxr); } free(vsi->queues, M_DEVBUF); /* Free VSI filter list */ ixl_free_mac_filters(vsi); } static void ixl_free_mac_filters(struct ixl_vsi *vsi) { struct ixl_mac_filter *f; while (!SLIST_EMPTY(&vsi->ftl)) { f = SLIST_FIRST(&vsi->ftl); SLIST_REMOVE_HEAD(&vsi->ftl, next); free(f, M_DEVBUF); } } /********************************************************************* * * Allocate memory for the VSI (virtual station interface) and their * associated queues, rings and the descriptors associated with each, * called only once at attach. * **********************************************************************/ static int ixl_setup_stations(struct ixl_pf *pf) { device_t dev = pf->dev; struct ixl_vsi *vsi; struct ixl_queue *que; struct tx_ring *txr; struct rx_ring *rxr; int rsize, tsize; int error = I40E_SUCCESS; vsi = &pf->vsi; vsi->back = (void *)pf; vsi->hw = &pf->hw; vsi->id = 0; vsi->num_vlans = 0; vsi->back = pf; /* Get memory for the station queues */ if (!(vsi->queues = (struct ixl_queue *) malloc(sizeof(struct ixl_queue) * vsi->num_queues, M_DEVBUF, M_NOWAIT | M_ZERO))) { device_printf(dev, "Unable to allocate queue memory\n"); error = ENOMEM; goto early; } for (int i = 0; i < vsi->num_queues; i++) { que = &vsi->queues[i]; que->num_desc = ixl_ringsz; que->me = i; que->vsi = vsi; /* mark the queue as active */ vsi->active_queues |= (u64)1 << que->me; txr = &que->txr; txr->que = que; txr->tail = I40E_QTX_TAIL(que->me); /* Initialize the TX lock */ snprintf(txr->mtx_name, sizeof(txr->mtx_name), "%s:tx(%d)", device_get_nameunit(dev), que->me); mtx_init(&txr->mtx, txr->mtx_name, NULL, MTX_DEF); /* Create the TX descriptor ring */ tsize = roundup2((que->num_desc * sizeof(struct i40e_tx_desc)) + sizeof(u32), DBA_ALIGN); if (i40e_allocate_dma_mem(&pf->hw, &txr->dma, i40e_mem_reserved, tsize, DBA_ALIGN)) { device_printf(dev, "Unable to allocate TX Descriptor memory\n"); error = ENOMEM; goto fail; } txr->base = (struct i40e_tx_desc *)txr->dma.va; bzero((void *)txr->base, tsize); /* Now allocate transmit soft structs for the ring */ if (ixl_allocate_tx_data(que)) { device_printf(dev, "Critical Failure setting up TX structures\n"); error = ENOMEM; goto fail; } /* Allocate a buf ring */ txr->br = buf_ring_alloc(4096, M_DEVBUF, M_WAITOK, &txr->mtx); if (txr->br == NULL) { device_printf(dev, "Critical Failure setting up TX buf ring\n"); error = ENOMEM; goto fail; } /* * Next the RX queues... */ rsize = roundup2(que->num_desc * sizeof(union i40e_rx_desc), DBA_ALIGN); rxr = &que->rxr; rxr->que = que; rxr->tail = I40E_QRX_TAIL(que->me); /* Initialize the RX side lock */ snprintf(rxr->mtx_name, sizeof(rxr->mtx_name), "%s:rx(%d)", device_get_nameunit(dev), que->me); mtx_init(&rxr->mtx, rxr->mtx_name, NULL, MTX_DEF); if (i40e_allocate_dma_mem(&pf->hw, &rxr->dma, i40e_mem_reserved, rsize, 4096)) { device_printf(dev, "Unable to allocate RX Descriptor memory\n"); error = ENOMEM; goto fail; } rxr->base = (union i40e_rx_desc *)rxr->dma.va; bzero((void *)rxr->base, rsize); /* Allocate receive soft structs for the ring*/ if (ixl_allocate_rx_data(que)) { device_printf(dev, "Critical Failure setting up receive structs\n"); error = ENOMEM; goto fail; } } return (0); fail: for (int i = 0; i < vsi->num_queues; i++) { que = &vsi->queues[i]; rxr = &que->rxr; txr = &que->txr; if (rxr->base) i40e_free_dma_mem(&pf->hw, &rxr->dma); if (txr->base) i40e_free_dma_mem(&pf->hw, &txr->dma); } early: return (error); } /* ** Provide a update to the queue RX ** interrupt moderation value. */ static void ixl_set_queue_rx_itr(struct ixl_queue *que) { struct ixl_vsi *vsi = que->vsi; struct i40e_hw *hw = vsi->hw; struct rx_ring *rxr = &que->rxr; u16 rx_itr; u16 rx_latency = 0; int rx_bytes; /* Idle, do nothing */ if (rxr->bytes == 0) return; if (ixl_dynamic_rx_itr) { rx_bytes = rxr->bytes/rxr->itr; rx_itr = rxr->itr; /* Adjust latency range */ switch (rxr->latency) { case IXL_LOW_LATENCY: if (rx_bytes > 10) { rx_latency = IXL_AVE_LATENCY; rx_itr = IXL_ITR_20K; } break; case IXL_AVE_LATENCY: if (rx_bytes > 20) { rx_latency = IXL_BULK_LATENCY; rx_itr = IXL_ITR_8K; } else if (rx_bytes <= 10) { rx_latency = IXL_LOW_LATENCY; rx_itr = IXL_ITR_100K; } break; case IXL_BULK_LATENCY: if (rx_bytes <= 20) { rx_latency = IXL_AVE_LATENCY; rx_itr = IXL_ITR_20K; } break; } rxr->latency = rx_latency; if (rx_itr != rxr->itr) { /* do an exponential smoothing */ rx_itr = (10 * rx_itr * rxr->itr) / ((9 * rx_itr) + rxr->itr); rxr->itr = rx_itr & IXL_MAX_ITR; wr32(hw, I40E_PFINT_ITRN(IXL_RX_ITR, que->me), rxr->itr); } } else { /* We may have have toggled to non-dynamic */ if (vsi->rx_itr_setting & IXL_ITR_DYNAMIC) vsi->rx_itr_setting = ixl_rx_itr; /* Update the hardware if needed */ if (rxr->itr != vsi->rx_itr_setting) { rxr->itr = vsi->rx_itr_setting; wr32(hw, I40E_PFINT_ITRN(IXL_RX_ITR, que->me), rxr->itr); } } rxr->bytes = 0; rxr->packets = 0; return; } /* ** Provide a update to the queue TX ** interrupt moderation value. */ static void ixl_set_queue_tx_itr(struct ixl_queue *que) { struct ixl_vsi *vsi = que->vsi; struct i40e_hw *hw = vsi->hw; struct tx_ring *txr = &que->txr; u16 tx_itr; u16 tx_latency = 0; int tx_bytes; /* Idle, do nothing */ if (txr->bytes == 0) return; if (ixl_dynamic_tx_itr) { tx_bytes = txr->bytes/txr->itr; tx_itr = txr->itr; switch (txr->latency) { case IXL_LOW_LATENCY: if (tx_bytes > 10) { tx_latency = IXL_AVE_LATENCY; tx_itr = IXL_ITR_20K; } break; case IXL_AVE_LATENCY: if (tx_bytes > 20) { tx_latency = IXL_BULK_LATENCY; tx_itr = IXL_ITR_8K; } else if (tx_bytes <= 10) { tx_latency = IXL_LOW_LATENCY; tx_itr = IXL_ITR_100K; } break; case IXL_BULK_LATENCY: if (tx_bytes <= 20) { tx_latency = IXL_AVE_LATENCY; tx_itr = IXL_ITR_20K; } break; } txr->latency = tx_latency; if (tx_itr != txr->itr) { /* do an exponential smoothing */ tx_itr = (10 * tx_itr * txr->itr) / ((9 * tx_itr) + txr->itr); txr->itr = tx_itr & IXL_MAX_ITR; wr32(hw, I40E_PFINT_ITRN(IXL_TX_ITR, que->me), txr->itr); } } else { /* We may have have toggled to non-dynamic */ if (vsi->tx_itr_setting & IXL_ITR_DYNAMIC) vsi->tx_itr_setting = ixl_tx_itr; /* Update the hardware if needed */ if (txr->itr != vsi->tx_itr_setting) { txr->itr = vsi->tx_itr_setting; wr32(hw, I40E_PFINT_ITRN(IXL_TX_ITR, que->me), txr->itr); } } txr->bytes = 0; txr->packets = 0; return; } #define QUEUE_NAME_LEN 32 static void ixl_add_vsi_sysctls(struct ixl_pf *pf, struct ixl_vsi *vsi, struct sysctl_ctx_list *ctx, const char *sysctl_name) { struct sysctl_oid *tree; struct sysctl_oid_list *child; struct sysctl_oid_list *vsi_list; tree = device_get_sysctl_tree(pf->dev); child = SYSCTL_CHILDREN(tree); vsi->vsi_node = SYSCTL_ADD_NODE(ctx, child, OID_AUTO, sysctl_name, CTLFLAG_RD, NULL, "VSI Number"); vsi_list = SYSCTL_CHILDREN(vsi->vsi_node); ixl_add_sysctls_eth_stats(ctx, vsi_list, &vsi->eth_stats); } static void ixl_add_hw_stats(struct ixl_pf *pf) { device_t dev = pf->dev; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *queues = vsi->queues; struct i40e_hw_port_stats *pf_stats = &pf->stats; struct sysctl_ctx_list *ctx = device_get_sysctl_ctx(dev); struct sysctl_oid *tree = device_get_sysctl_tree(dev); struct sysctl_oid_list *child = SYSCTL_CHILDREN(tree); struct sysctl_oid_list *vsi_list; struct sysctl_oid *queue_node; struct sysctl_oid_list *queue_list; struct tx_ring *txr; struct rx_ring *rxr; char queue_namebuf[QUEUE_NAME_LEN]; /* Driver statistics */ SYSCTL_ADD_ULONG(ctx, child, OID_AUTO, "watchdog_events", CTLFLAG_RD, &pf->watchdog_events, "Watchdog timeouts"); SYSCTL_ADD_ULONG(ctx, child, OID_AUTO, "admin_irq", CTLFLAG_RD, &pf->admin_irq, "Admin Queue IRQ Handled"); ixl_add_vsi_sysctls(pf, &pf->vsi, ctx, "pf"); vsi_list = SYSCTL_CHILDREN(pf->vsi.vsi_node); /* Queue statistics */ for (int q = 0; q < vsi->num_queues; q++) { snprintf(queue_namebuf, QUEUE_NAME_LEN, "que%d", q); queue_node = SYSCTL_ADD_NODE(ctx, vsi_list, OID_AUTO, queue_namebuf, CTLFLAG_RD, NULL, "Queue #"); queue_list = SYSCTL_CHILDREN(queue_node); txr = &(queues[q].txr); rxr = &(queues[q].rxr); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "mbuf_defrag_failed", CTLFLAG_RD, &(queues[q].mbuf_defrag_failed), "m_defrag() failed"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "dropped", CTLFLAG_RD, &(queues[q].dropped_pkts), "Driver dropped packets"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "irqs", CTLFLAG_RD, &(queues[q].irqs), "irqs on this queue"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "tso_tx", CTLFLAG_RD, &(queues[q].tso), "TSO"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "tx_dma_setup", CTLFLAG_RD, &(queues[q].tx_dma_setup), "Driver tx dma failure in xmit"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "no_desc_avail", CTLFLAG_RD, &(txr->no_desc), "Queue No Descriptor Available"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "tx_packets", CTLFLAG_RD, &(txr->total_packets), "Queue Packets Transmitted"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "tx_bytes", CTLFLAG_RD, &(txr->tx_bytes), "Queue Bytes Transmitted"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "rx_packets", CTLFLAG_RD, &(rxr->rx_packets), "Queue Packets Received"); SYSCTL_ADD_UQUAD(ctx, queue_list, OID_AUTO, "rx_bytes", CTLFLAG_RD, &(rxr->rx_bytes), "Queue Bytes Received"); } /* MAC stats */ ixl_add_sysctls_mac_stats(ctx, child, pf_stats); } static void ixl_add_sysctls_eth_stats(struct sysctl_ctx_list *ctx, struct sysctl_oid_list *child, struct i40e_eth_stats *eth_stats) { struct ixl_sysctl_info ctls[] = { {ð_stats->rx_bytes, "good_octets_rcvd", "Good Octets Received"}, {ð_stats->rx_unicast, "ucast_pkts_rcvd", "Unicast Packets Received"}, {ð_stats->rx_multicast, "mcast_pkts_rcvd", "Multicast Packets Received"}, {ð_stats->rx_broadcast, "bcast_pkts_rcvd", "Broadcast Packets Received"}, {ð_stats->rx_discards, "rx_discards", "Discarded RX packets"}, {ð_stats->tx_bytes, "good_octets_txd", "Good Octets Transmitted"}, {ð_stats->tx_unicast, "ucast_pkts_txd", "Unicast Packets Transmitted"}, {ð_stats->tx_multicast, "mcast_pkts_txd", "Multicast Packets Transmitted"}, {ð_stats->tx_broadcast, "bcast_pkts_txd", "Broadcast Packets Transmitted"}, // end {0,0,0} }; struct ixl_sysctl_info *entry = ctls; while (entry->stat != 0) { SYSCTL_ADD_UQUAD(ctx, child, OID_AUTO, entry->name, CTLFLAG_RD, entry->stat, entry->description); entry++; } } static void ixl_add_sysctls_mac_stats(struct sysctl_ctx_list *ctx, struct sysctl_oid_list *child, struct i40e_hw_port_stats *stats) { struct sysctl_oid *stat_node = SYSCTL_ADD_NODE(ctx, child, OID_AUTO, "mac", CTLFLAG_RD, NULL, "Mac Statistics"); struct sysctl_oid_list *stat_list = SYSCTL_CHILDREN(stat_node); struct i40e_eth_stats *eth_stats = &stats->eth; ixl_add_sysctls_eth_stats(ctx, stat_list, eth_stats); struct ixl_sysctl_info ctls[] = { {&stats->crc_errors, "crc_errors", "CRC Errors"}, {&stats->illegal_bytes, "illegal_bytes", "Illegal Byte Errors"}, {&stats->mac_local_faults, "local_faults", "MAC Local Faults"}, {&stats->mac_remote_faults, "remote_faults", "MAC Remote Faults"}, {&stats->rx_length_errors, "rx_length_errors", "Receive Length Errors"}, /* Packet Reception Stats */ {&stats->rx_size_64, "rx_frames_64", "64 byte frames received"}, {&stats->rx_size_127, "rx_frames_65_127", "65-127 byte frames received"}, {&stats->rx_size_255, "rx_frames_128_255", "128-255 byte frames received"}, {&stats->rx_size_511, "rx_frames_256_511", "256-511 byte frames received"}, {&stats->rx_size_1023, "rx_frames_512_1023", "512-1023 byte frames received"}, {&stats->rx_size_1522, "rx_frames_1024_1522", "1024-1522 byte frames received"}, {&stats->rx_size_big, "rx_frames_big", "1523-9522 byte frames received"}, {&stats->rx_undersize, "rx_undersize", "Undersized packets received"}, {&stats->rx_fragments, "rx_fragmented", "Fragmented packets received"}, {&stats->rx_oversize, "rx_oversized", "Oversized packets received"}, {&stats->rx_jabber, "rx_jabber", "Received Jabber"}, {&stats->checksum_error, "checksum_errors", "Checksum Errors"}, /* Packet Transmission Stats */ {&stats->tx_size_64, "tx_frames_64", "64 byte frames transmitted"}, {&stats->tx_size_127, "tx_frames_65_127", "65-127 byte frames transmitted"}, {&stats->tx_size_255, "tx_frames_128_255", "128-255 byte frames transmitted"}, {&stats->tx_size_511, "tx_frames_256_511", "256-511 byte frames transmitted"}, {&stats->tx_size_1023, "tx_frames_512_1023", "512-1023 byte frames transmitted"}, {&stats->tx_size_1522, "tx_frames_1024_1522", "1024-1522 byte frames transmitted"}, {&stats->tx_size_big, "tx_frames_big", "1523-9522 byte frames transmitted"}, /* Flow control */ {&stats->link_xon_tx, "xon_txd", "Link XON transmitted"}, {&stats->link_xon_rx, "xon_recvd", "Link XON received"}, {&stats->link_xoff_tx, "xoff_txd", "Link XOFF transmitted"}, {&stats->link_xoff_rx, "xoff_recvd", "Link XOFF received"}, /* End */ {0,0,0} }; struct ixl_sysctl_info *entry = ctls; while (entry->stat != 0) { SYSCTL_ADD_UQUAD(ctx, stat_list, OID_AUTO, entry->name, CTLFLAG_RD, entry->stat, entry->description); entry++; } } /* ** ixl_config_rss - setup RSS ** - note this is done for the single vsi */ static void ixl_config_rss(struct ixl_vsi *vsi) { struct ixl_pf *pf = (struct ixl_pf *)vsi->back; struct i40e_hw *hw = vsi->hw; u32 lut = 0; u64 set_hena = 0, hena; int i, j, que_id; #ifdef RSS u32 rss_hash_config; u32 rss_seed[IXL_KEYSZ]; #else u32 rss_seed[IXL_KEYSZ] = {0x41b01687, 0x183cfd8c, 0xce880440, 0x580cbc3c, 0x35897377, 0x328b25e1, 0x4fa98922, 0xb7d90c14, 0xd5bad70d, 0xcd15a2c1}; #endif #ifdef RSS /* Fetch the configured RSS key */ rss_getkey((uint8_t *) &rss_seed); #endif /* Fill out hash function seed */ for (i = 0; i < IXL_KEYSZ; i++) wr32(hw, I40E_PFQF_HKEY(i), rss_seed[i]); /* Enable PCTYPES for RSS: */ #ifdef RSS rss_hash_config = rss_gethashconfig(); if (rss_hash_config & RSS_HASHTYPE_RSS_IPV4) set_hena |= ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV4_OTHER); if (rss_hash_config & RSS_HASHTYPE_RSS_TCP_IPV4) set_hena |= ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV4_TCP); if (rss_hash_config & RSS_HASHTYPE_RSS_UDP_IPV4) set_hena |= ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV4_UDP); if (rss_hash_config & RSS_HASHTYPE_RSS_IPV6) set_hena |= ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV6_OTHER); if (rss_hash_config & RSS_HASHTYPE_RSS_IPV6_EX) set_hena |= ((u64)1 << I40E_FILTER_PCTYPE_FRAG_IPV6); if (rss_hash_config & RSS_HASHTYPE_RSS_TCP_IPV6) set_hena |= ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV6_TCP); if (rss_hash_config & RSS_HASHTYPE_RSS_UDP_IPV6) set_hena |= ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV6_UDP); #else set_hena = ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV4_UDP) | ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV4_TCP) | ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV4_SCTP) | ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV4_OTHER) | ((u64)1 << I40E_FILTER_PCTYPE_FRAG_IPV4) | ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV6_UDP) | ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV6_TCP) | ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV6_SCTP) | ((u64)1 << I40E_FILTER_PCTYPE_NONF_IPV6_OTHER) | ((u64)1 << I40E_FILTER_PCTYPE_FRAG_IPV6) | ((u64)1 << I40E_FILTER_PCTYPE_L2_PAYLOAD); #endif hena = (u64)rd32(hw, I40E_PFQF_HENA(0)) | ((u64)rd32(hw, I40E_PFQF_HENA(1)) << 32); hena |= set_hena; wr32(hw, I40E_PFQF_HENA(0), (u32)hena); wr32(hw, I40E_PFQF_HENA(1), (u32)(hena >> 32)); /* Populate the LUT with max no. of queues in round robin fashion */ for (i = j = 0; i < pf->hw.func_caps.rss_table_size; i++, j++) { if (j == vsi->num_queues) j = 0; #ifdef RSS /* * Fetch the RSS bucket id for the given indirection entry. * Cap it at the number of configured buckets (which is * num_queues.) */ que_id = rss_get_indirection_to_bucket(i); que_id = que_id % vsi->num_queues; #else que_id = j; #endif /* lut = 4-byte sliding window of 4 lut entries */ lut = (lut << 8) | (que_id & ((0x1 << pf->hw.func_caps.rss_table_entry_width) - 1)); /* On i = 3, we have 4 entries in lut; write to the register */ if ((i & 3) == 3) wr32(hw, I40E_PFQF_HLUT(i >> 2), lut); } ixl_flush(hw); } /* ** This routine is run via an vlan config EVENT, ** it enables us to use the HW Filter table since ** we can get the vlan id. This just creates the ** entry in the soft version of the VFTA, init will ** repopulate the real table. */ static void ixl_register_vlan(void *arg, struct ifnet *ifp, u16 vtag) { struct ixl_vsi *vsi = ifp->if_softc; struct i40e_hw *hw = vsi->hw; struct ixl_pf *pf = (struct ixl_pf *)vsi->back; if (ifp->if_softc != arg) /* Not our event */ return; if ((vtag == 0) || (vtag > 4095)) /* Invalid */ return; IXL_PF_LOCK(pf); ++vsi->num_vlans; ixl_add_filter(vsi, hw->mac.addr, vtag); IXL_PF_UNLOCK(pf); } /* ** This routine is run via an vlan ** unconfig EVENT, remove our entry ** in the soft vfta. */ static void ixl_unregister_vlan(void *arg, struct ifnet *ifp, u16 vtag) { struct ixl_vsi *vsi = ifp->if_softc; struct i40e_hw *hw = vsi->hw; struct ixl_pf *pf = (struct ixl_pf *)vsi->back; if (ifp->if_softc != arg) return; if ((vtag == 0) || (vtag > 4095)) /* Invalid */ return; IXL_PF_LOCK(pf); --vsi->num_vlans; ixl_del_filter(vsi, hw->mac.addr, vtag); IXL_PF_UNLOCK(pf); } /* ** This routine updates vlan filters, called by init ** it scans the filter table and then updates the hw ** after a soft reset. */ static void ixl_setup_vlan_filters(struct ixl_vsi *vsi) { struct ixl_mac_filter *f; int cnt = 0, flags; if (vsi->num_vlans == 0) return; /* ** Scan the filter list for vlan entries, ** mark them for addition and then call ** for the AQ update. */ SLIST_FOREACH(f, &vsi->ftl, next) { if (f->flags & IXL_FILTER_VLAN) { f->flags |= (IXL_FILTER_ADD | IXL_FILTER_USED); cnt++; } } if (cnt == 0) { printf("setup vlan: no filters found!\n"); return; } flags = IXL_FILTER_VLAN; flags |= (IXL_FILTER_ADD | IXL_FILTER_USED); ixl_add_hw_filters(vsi, flags, cnt); return; } /* ** Initialize filter list and add filters that the hardware ** needs to know about. */ static void ixl_init_filters(struct ixl_vsi *vsi) { /* Add broadcast address */ ixl_add_filter(vsi, ixl_bcast_addr, IXL_VLAN_ANY); } /* ** This routine adds mulicast filters */ static void ixl_add_mc_filter(struct ixl_vsi *vsi, u8 *macaddr) { struct ixl_mac_filter *f; /* Does one already exist */ f = ixl_find_filter(vsi, macaddr, IXL_VLAN_ANY); if (f != NULL) return; f = ixl_get_filter(vsi); if (f == NULL) { printf("WARNING: no filter available!!\n"); return; } bcopy(macaddr, f->macaddr, ETHER_ADDR_LEN); f->vlan = IXL_VLAN_ANY; f->flags |= (IXL_FILTER_ADD | IXL_FILTER_USED | IXL_FILTER_MC); return; } static void ixl_reconfigure_filters(struct ixl_vsi *vsi) { ixl_add_hw_filters(vsi, IXL_FILTER_USED, vsi->num_macs); } /* ** This routine adds macvlan filters */ static void ixl_add_filter(struct ixl_vsi *vsi, u8 *macaddr, s16 vlan) { struct ixl_mac_filter *f, *tmp; struct ixl_pf *pf; device_t dev; DEBUGOUT("ixl_add_filter: begin"); pf = vsi->back; dev = pf->dev; /* Does one already exist */ f = ixl_find_filter(vsi, macaddr, vlan); if (f != NULL) return; /* ** Is this the first vlan being registered, if so we ** need to remove the ANY filter that indicates we are ** not in a vlan, and replace that with a 0 filter. */ if ((vlan != IXL_VLAN_ANY) && (vsi->num_vlans == 1)) { tmp = ixl_find_filter(vsi, macaddr, IXL_VLAN_ANY); if (tmp != NULL) { ixl_del_filter(vsi, macaddr, IXL_VLAN_ANY); ixl_add_filter(vsi, macaddr, 0); } } f = ixl_get_filter(vsi); if (f == NULL) { device_printf(dev, "WARNING: no filter available!!\n"); return; } bcopy(macaddr, f->macaddr, ETHER_ADDR_LEN); f->vlan = vlan; f->flags |= (IXL_FILTER_ADD | IXL_FILTER_USED); if (f->vlan != IXL_VLAN_ANY) f->flags |= IXL_FILTER_VLAN; else vsi->num_macs++; ixl_add_hw_filters(vsi, f->flags, 1); return; } static void ixl_del_filter(struct ixl_vsi *vsi, u8 *macaddr, s16 vlan) { struct ixl_mac_filter *f; f = ixl_find_filter(vsi, macaddr, vlan); if (f == NULL) return; f->flags |= IXL_FILTER_DEL; ixl_del_hw_filters(vsi, 1); vsi->num_macs--; /* Check if this is the last vlan removal */ if (vlan != IXL_VLAN_ANY && vsi->num_vlans == 0) { /* Switch back to a non-vlan filter */ ixl_del_filter(vsi, macaddr, 0); ixl_add_filter(vsi, macaddr, IXL_VLAN_ANY); } return; } /* ** Find the filter with both matching mac addr and vlan id */ static struct ixl_mac_filter * ixl_find_filter(struct ixl_vsi *vsi, u8 *macaddr, s16 vlan) { struct ixl_mac_filter *f; bool match = FALSE; SLIST_FOREACH(f, &vsi->ftl, next) { if (!cmp_etheraddr(f->macaddr, macaddr)) continue; if (f->vlan == vlan) { match = TRUE; break; } } if (!match) f = NULL; return (f); } /* ** This routine takes additions to the vsi filter ** table and creates an Admin Queue call to create ** the filters in the hardware. */ static void ixl_add_hw_filters(struct ixl_vsi *vsi, int flags, int cnt) { struct i40e_aqc_add_macvlan_element_data *a, *b; struct ixl_mac_filter *f; struct ixl_pf *pf; struct i40e_hw *hw; device_t dev; int err, j = 0; pf = vsi->back; dev = pf->dev; hw = &pf->hw; IXL_PF_LOCK_ASSERT(pf); a = malloc(sizeof(struct i40e_aqc_add_macvlan_element_data) * cnt, M_DEVBUF, M_NOWAIT | M_ZERO); if (a == NULL) { device_printf(dev, "add_hw_filters failed to get memory\n"); return; } /* ** Scan the filter list, each time we find one ** we add it to the admin queue array and turn off ** the add bit. */ SLIST_FOREACH(f, &vsi->ftl, next) { if (f->flags == flags) { b = &a[j]; // a pox on fvl long names :) bcopy(f->macaddr, b->mac_addr, ETHER_ADDR_LEN); if (f->vlan == IXL_VLAN_ANY) { b->vlan_tag = 0; b->flags = I40E_AQC_MACVLAN_ADD_IGNORE_VLAN; } else { b->vlan_tag = f->vlan; b->flags = 0; } b->flags |= I40E_AQC_MACVLAN_ADD_PERFECT_MATCH; f->flags &= ~IXL_FILTER_ADD; j++; } if (j == cnt) break; } if (j > 0) { err = i40e_aq_add_macvlan(hw, vsi->seid, a, j, NULL); if (err) device_printf(dev, "aq_add_macvlan err %d, " "aq_error %d\n", err, hw->aq.asq_last_status); else vsi->hw_filters_add += j; } free(a, M_DEVBUF); return; } /* ** This routine takes removals in the vsi filter ** table and creates an Admin Queue call to delete ** the filters in the hardware. */ static void ixl_del_hw_filters(struct ixl_vsi *vsi, int cnt) { struct i40e_aqc_remove_macvlan_element_data *d, *e; struct ixl_pf *pf; struct i40e_hw *hw; device_t dev; struct ixl_mac_filter *f, *f_temp; int err, j = 0; DEBUGOUT("ixl_del_hw_filters: begin\n"); pf = vsi->back; hw = &pf->hw; dev = pf->dev; d = malloc(sizeof(struct i40e_aqc_remove_macvlan_element_data) * cnt, M_DEVBUF, M_NOWAIT | M_ZERO); if (d == NULL) { printf("del hw filter failed to get memory\n"); return; } SLIST_FOREACH_SAFE(f, &vsi->ftl, next, f_temp) { if (f->flags & IXL_FILTER_DEL) { e = &d[j]; // a pox on fvl long names :) bcopy(f->macaddr, e->mac_addr, ETHER_ADDR_LEN); e->vlan_tag = (f->vlan == IXL_VLAN_ANY ? 0 : f->vlan); e->flags = I40E_AQC_MACVLAN_DEL_PERFECT_MATCH; /* delete entry from vsi list */ SLIST_REMOVE(&vsi->ftl, f, ixl_mac_filter, next); free(f, M_DEVBUF); j++; } if (j == cnt) break; } if (j > 0) { err = i40e_aq_remove_macvlan(hw, vsi->seid, d, j, NULL); /* NOTE: returns ENOENT every time but seems to work fine, so we'll ignore that specific error. */ // TODO: Does this still occur on current firmwares? if (err && hw->aq.asq_last_status != I40E_AQ_RC_ENOENT) { int sc = 0; for (int i = 0; i < j; i++) sc += (!d[i].error_code); vsi->hw_filters_del += sc; device_printf(dev, "Failed to remove %d/%d filters, aq error %d\n", j - sc, j, hw->aq.asq_last_status); } else vsi->hw_filters_del += j; } free(d, M_DEVBUF); DEBUGOUT("ixl_del_hw_filters: end\n"); return; } static int ixl_enable_rings(struct ixl_vsi *vsi) { struct ixl_pf *pf = vsi->back; struct i40e_hw *hw = &pf->hw; int index, error; u32 reg; error = 0; for (int i = 0; i < vsi->num_queues; i++) { index = vsi->first_queue + i; i40e_pre_tx_queue_cfg(hw, index, TRUE); reg = rd32(hw, I40E_QTX_ENA(index)); reg |= I40E_QTX_ENA_QENA_REQ_MASK | I40E_QTX_ENA_QENA_STAT_MASK; wr32(hw, I40E_QTX_ENA(index), reg); /* Verify the enable took */ for (int j = 0; j < 10; j++) { reg = rd32(hw, I40E_QTX_ENA(index)); if (reg & I40E_QTX_ENA_QENA_STAT_MASK) break; i40e_msec_delay(10); } if ((reg & I40E_QTX_ENA_QENA_STAT_MASK) == 0) { device_printf(pf->dev, "TX queue %d disabled!\n", index); error = ETIMEDOUT; } reg = rd32(hw, I40E_QRX_ENA(index)); reg |= I40E_QRX_ENA_QENA_REQ_MASK | I40E_QRX_ENA_QENA_STAT_MASK; wr32(hw, I40E_QRX_ENA(index), reg); /* Verify the enable took */ for (int j = 0; j < 10; j++) { reg = rd32(hw, I40E_QRX_ENA(index)); if (reg & I40E_QRX_ENA_QENA_STAT_MASK) break; i40e_msec_delay(10); } if ((reg & I40E_QRX_ENA_QENA_STAT_MASK) == 0) { device_printf(pf->dev, "RX queue %d disabled!\n", index); error = ETIMEDOUT; } } return (error); } static int ixl_disable_rings(struct ixl_vsi *vsi) { struct ixl_pf *pf = vsi->back; struct i40e_hw *hw = &pf->hw; int index, error; u32 reg; error = 0; for (int i = 0; i < vsi->num_queues; i++) { index = vsi->first_queue + i; i40e_pre_tx_queue_cfg(hw, index, FALSE); i40e_usec_delay(500); reg = rd32(hw, I40E_QTX_ENA(index)); reg &= ~I40E_QTX_ENA_QENA_REQ_MASK; wr32(hw, I40E_QTX_ENA(index), reg); /* Verify the disable took */ for (int j = 0; j < 10; j++) { reg = rd32(hw, I40E_QTX_ENA(index)); if (!(reg & I40E_QTX_ENA_QENA_STAT_MASK)) break; i40e_msec_delay(10); } if (reg & I40E_QTX_ENA_QENA_STAT_MASK) { device_printf(pf->dev, "TX queue %d still enabled!\n", index); error = ETIMEDOUT; } reg = rd32(hw, I40E_QRX_ENA(index)); reg &= ~I40E_QRX_ENA_QENA_REQ_MASK; wr32(hw, I40E_QRX_ENA(index), reg); /* Verify the disable took */ for (int j = 0; j < 10; j++) { reg = rd32(hw, I40E_QRX_ENA(index)); if (!(reg & I40E_QRX_ENA_QENA_STAT_MASK)) break; i40e_msec_delay(10); } if (reg & I40E_QRX_ENA_QENA_STAT_MASK) { device_printf(pf->dev, "RX queue %d still enabled!\n", index); error = ETIMEDOUT; } } return (error); } /** * ixl_handle_mdd_event * * Called from interrupt handler to identify possibly malicious vfs * (But also detects events from the PF, as well) **/ static void ixl_handle_mdd_event(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; bool mdd_detected = false; bool pf_mdd_detected = false; u32 reg; /* find what triggered the MDD event */ reg = rd32(hw, I40E_GL_MDET_TX); if (reg & I40E_GL_MDET_TX_VALID_MASK) { u8 pf_num = (reg & I40E_GL_MDET_TX_PF_NUM_MASK) >> I40E_GL_MDET_TX_PF_NUM_SHIFT; u8 event = (reg & I40E_GL_MDET_TX_EVENT_MASK) >> I40E_GL_MDET_TX_EVENT_SHIFT; u8 queue = (reg & I40E_GL_MDET_TX_QUEUE_MASK) >> I40E_GL_MDET_TX_QUEUE_SHIFT; device_printf(dev, "Malicious Driver Detection event 0x%02x" " on TX queue %d pf number 0x%02x\n", event, queue, pf_num); wr32(hw, I40E_GL_MDET_TX, 0xffffffff); mdd_detected = true; } reg = rd32(hw, I40E_GL_MDET_RX); if (reg & I40E_GL_MDET_RX_VALID_MASK) { u8 func = (reg & I40E_GL_MDET_RX_FUNCTION_MASK) >> I40E_GL_MDET_RX_FUNCTION_SHIFT; u8 event = (reg & I40E_GL_MDET_RX_EVENT_MASK) >> I40E_GL_MDET_RX_EVENT_SHIFT; u8 queue = (reg & I40E_GL_MDET_RX_QUEUE_MASK) >> I40E_GL_MDET_RX_QUEUE_SHIFT; device_printf(dev, "Malicious Driver Detection event 0x%02x" " on RX queue %d of function 0x%02x\n", event, queue, func); wr32(hw, I40E_GL_MDET_RX, 0xffffffff); mdd_detected = true; } if (mdd_detected) { reg = rd32(hw, I40E_PF_MDET_TX); if (reg & I40E_PF_MDET_TX_VALID_MASK) { wr32(hw, I40E_PF_MDET_TX, 0xFFFF); device_printf(dev, "MDD TX event is for this function 0x%08x", reg); pf_mdd_detected = true; } reg = rd32(hw, I40E_PF_MDET_RX); if (reg & I40E_PF_MDET_RX_VALID_MASK) { wr32(hw, I40E_PF_MDET_RX, 0xFFFF); device_printf(dev, "MDD RX event is for this function 0x%08x", reg); pf_mdd_detected = true; } } /* re-enable mdd interrupt cause */ reg = rd32(hw, I40E_PFINT_ICR0_ENA); reg |= I40E_PFINT_ICR0_ENA_MAL_DETECT_MASK; wr32(hw, I40E_PFINT_ICR0_ENA, reg); ixl_flush(hw); } static void ixl_enable_intr(struct ixl_vsi *vsi) { struct i40e_hw *hw = vsi->hw; struct ixl_queue *que = vsi->queues; if (ixl_enable_msix) { ixl_enable_adminq(hw); for (int i = 0; i < vsi->num_queues; i++, que++) ixl_enable_queue(hw, que->me); } else ixl_enable_legacy(hw); } static void ixl_disable_rings_intr(struct ixl_vsi *vsi) { struct i40e_hw *hw = vsi->hw; struct ixl_queue *que = vsi->queues; for (int i = 0; i < vsi->num_queues; i++, que++) ixl_disable_queue(hw, que->me); } static void ixl_disable_intr(struct ixl_vsi *vsi) { struct i40e_hw *hw = vsi->hw; if (ixl_enable_msix) ixl_disable_adminq(hw); else ixl_disable_legacy(hw); } static void ixl_enable_adminq(struct i40e_hw *hw) { u32 reg; reg = I40E_PFINT_DYN_CTL0_INTENA_MASK | I40E_PFINT_DYN_CTL0_CLEARPBA_MASK | (IXL_ITR_NONE << I40E_PFINT_DYN_CTL0_ITR_INDX_SHIFT); wr32(hw, I40E_PFINT_DYN_CTL0, reg); ixl_flush(hw); return; } static void ixl_disable_adminq(struct i40e_hw *hw) { u32 reg; reg = IXL_ITR_NONE << I40E_PFINT_DYN_CTL0_ITR_INDX_SHIFT; wr32(hw, I40E_PFINT_DYN_CTL0, reg); return; } static void ixl_enable_queue(struct i40e_hw *hw, int id) { u32 reg; reg = I40E_PFINT_DYN_CTLN_INTENA_MASK | I40E_PFINT_DYN_CTLN_CLEARPBA_MASK | (IXL_ITR_NONE << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT); wr32(hw, I40E_PFINT_DYN_CTLN(id), reg); } static void ixl_disable_queue(struct i40e_hw *hw, int id) { u32 reg; reg = IXL_ITR_NONE << I40E_PFINT_DYN_CTLN_ITR_INDX_SHIFT; wr32(hw, I40E_PFINT_DYN_CTLN(id), reg); return; } static void ixl_enable_legacy(struct i40e_hw *hw) { u32 reg; reg = I40E_PFINT_DYN_CTL0_INTENA_MASK | I40E_PFINT_DYN_CTL0_CLEARPBA_MASK | (IXL_ITR_NONE << I40E_PFINT_DYN_CTL0_ITR_INDX_SHIFT); wr32(hw, I40E_PFINT_DYN_CTL0, reg); } static void ixl_disable_legacy(struct i40e_hw *hw) { u32 reg; reg = IXL_ITR_NONE << I40E_PFINT_DYN_CTL0_ITR_INDX_SHIFT; wr32(hw, I40E_PFINT_DYN_CTL0, reg); return; } static void ixl_update_stats_counters(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct ixl_vf *vf; struct i40e_hw_port_stats *nsd = &pf->stats; struct i40e_hw_port_stats *osd = &pf->stats_offsets; /* Update hw stats */ ixl_stat_update32(hw, I40E_GLPRT_CRCERRS(hw->port), pf->stat_offsets_loaded, &osd->crc_errors, &nsd->crc_errors); ixl_stat_update32(hw, I40E_GLPRT_ILLERRC(hw->port), pf->stat_offsets_loaded, &osd->illegal_bytes, &nsd->illegal_bytes); ixl_stat_update48(hw, I40E_GLPRT_GORCH(hw->port), I40E_GLPRT_GORCL(hw->port), pf->stat_offsets_loaded, &osd->eth.rx_bytes, &nsd->eth.rx_bytes); ixl_stat_update48(hw, I40E_GLPRT_GOTCH(hw->port), I40E_GLPRT_GOTCL(hw->port), pf->stat_offsets_loaded, &osd->eth.tx_bytes, &nsd->eth.tx_bytes); ixl_stat_update32(hw, I40E_GLPRT_RDPC(hw->port), pf->stat_offsets_loaded, &osd->eth.rx_discards, &nsd->eth.rx_discards); ixl_stat_update48(hw, I40E_GLPRT_UPRCH(hw->port), I40E_GLPRT_UPRCL(hw->port), pf->stat_offsets_loaded, &osd->eth.rx_unicast, &nsd->eth.rx_unicast); ixl_stat_update48(hw, I40E_GLPRT_UPTCH(hw->port), I40E_GLPRT_UPTCL(hw->port), pf->stat_offsets_loaded, &osd->eth.tx_unicast, &nsd->eth.tx_unicast); ixl_stat_update48(hw, I40E_GLPRT_MPRCH(hw->port), I40E_GLPRT_MPRCL(hw->port), pf->stat_offsets_loaded, &osd->eth.rx_multicast, &nsd->eth.rx_multicast); ixl_stat_update48(hw, I40E_GLPRT_MPTCH(hw->port), I40E_GLPRT_MPTCL(hw->port), pf->stat_offsets_loaded, &osd->eth.tx_multicast, &nsd->eth.tx_multicast); ixl_stat_update48(hw, I40E_GLPRT_BPRCH(hw->port), I40E_GLPRT_BPRCL(hw->port), pf->stat_offsets_loaded, &osd->eth.rx_broadcast, &nsd->eth.rx_broadcast); ixl_stat_update48(hw, I40E_GLPRT_BPTCH(hw->port), I40E_GLPRT_BPTCL(hw->port), pf->stat_offsets_loaded, &osd->eth.tx_broadcast, &nsd->eth.tx_broadcast); ixl_stat_update32(hw, I40E_GLPRT_TDOLD(hw->port), pf->stat_offsets_loaded, &osd->tx_dropped_link_down, &nsd->tx_dropped_link_down); ixl_stat_update32(hw, I40E_GLPRT_MLFC(hw->port), pf->stat_offsets_loaded, &osd->mac_local_faults, &nsd->mac_local_faults); ixl_stat_update32(hw, I40E_GLPRT_MRFC(hw->port), pf->stat_offsets_loaded, &osd->mac_remote_faults, &nsd->mac_remote_faults); ixl_stat_update32(hw, I40E_GLPRT_RLEC(hw->port), pf->stat_offsets_loaded, &osd->rx_length_errors, &nsd->rx_length_errors); /* Flow control (LFC) stats */ ixl_stat_update32(hw, I40E_GLPRT_LXONRXC(hw->port), pf->stat_offsets_loaded, &osd->link_xon_rx, &nsd->link_xon_rx); ixl_stat_update32(hw, I40E_GLPRT_LXONTXC(hw->port), pf->stat_offsets_loaded, &osd->link_xon_tx, &nsd->link_xon_tx); ixl_stat_update32(hw, I40E_GLPRT_LXOFFRXC(hw->port), pf->stat_offsets_loaded, &osd->link_xoff_rx, &nsd->link_xoff_rx); ixl_stat_update32(hw, I40E_GLPRT_LXOFFTXC(hw->port), pf->stat_offsets_loaded, &osd->link_xoff_tx, &nsd->link_xoff_tx); /* Packet size stats rx */ ixl_stat_update48(hw, I40E_GLPRT_PRC64H(hw->port), I40E_GLPRT_PRC64L(hw->port), pf->stat_offsets_loaded, &osd->rx_size_64, &nsd->rx_size_64); ixl_stat_update48(hw, I40E_GLPRT_PRC127H(hw->port), I40E_GLPRT_PRC127L(hw->port), pf->stat_offsets_loaded, &osd->rx_size_127, &nsd->rx_size_127); ixl_stat_update48(hw, I40E_GLPRT_PRC255H(hw->port), I40E_GLPRT_PRC255L(hw->port), pf->stat_offsets_loaded, &osd->rx_size_255, &nsd->rx_size_255); ixl_stat_update48(hw, I40E_GLPRT_PRC511H(hw->port), I40E_GLPRT_PRC511L(hw->port), pf->stat_offsets_loaded, &osd->rx_size_511, &nsd->rx_size_511); ixl_stat_update48(hw, I40E_GLPRT_PRC1023H(hw->port), I40E_GLPRT_PRC1023L(hw->port), pf->stat_offsets_loaded, &osd->rx_size_1023, &nsd->rx_size_1023); ixl_stat_update48(hw, I40E_GLPRT_PRC1522H(hw->port), I40E_GLPRT_PRC1522L(hw->port), pf->stat_offsets_loaded, &osd->rx_size_1522, &nsd->rx_size_1522); ixl_stat_update48(hw, I40E_GLPRT_PRC9522H(hw->port), I40E_GLPRT_PRC9522L(hw->port), pf->stat_offsets_loaded, &osd->rx_size_big, &nsd->rx_size_big); /* Packet size stats tx */ ixl_stat_update48(hw, I40E_GLPRT_PTC64H(hw->port), I40E_GLPRT_PTC64L(hw->port), pf->stat_offsets_loaded, &osd->tx_size_64, &nsd->tx_size_64); ixl_stat_update48(hw, I40E_GLPRT_PTC127H(hw->port), I40E_GLPRT_PTC127L(hw->port), pf->stat_offsets_loaded, &osd->tx_size_127, &nsd->tx_size_127); ixl_stat_update48(hw, I40E_GLPRT_PTC255H(hw->port), I40E_GLPRT_PTC255L(hw->port), pf->stat_offsets_loaded, &osd->tx_size_255, &nsd->tx_size_255); ixl_stat_update48(hw, I40E_GLPRT_PTC511H(hw->port), I40E_GLPRT_PTC511L(hw->port), pf->stat_offsets_loaded, &osd->tx_size_511, &nsd->tx_size_511); ixl_stat_update48(hw, I40E_GLPRT_PTC1023H(hw->port), I40E_GLPRT_PTC1023L(hw->port), pf->stat_offsets_loaded, &osd->tx_size_1023, &nsd->tx_size_1023); ixl_stat_update48(hw, I40E_GLPRT_PTC1522H(hw->port), I40E_GLPRT_PTC1522L(hw->port), pf->stat_offsets_loaded, &osd->tx_size_1522, &nsd->tx_size_1522); ixl_stat_update48(hw, I40E_GLPRT_PTC9522H(hw->port), I40E_GLPRT_PTC9522L(hw->port), pf->stat_offsets_loaded, &osd->tx_size_big, &nsd->tx_size_big); ixl_stat_update32(hw, I40E_GLPRT_RUC(hw->port), pf->stat_offsets_loaded, &osd->rx_undersize, &nsd->rx_undersize); ixl_stat_update32(hw, I40E_GLPRT_RFC(hw->port), pf->stat_offsets_loaded, &osd->rx_fragments, &nsd->rx_fragments); ixl_stat_update32(hw, I40E_GLPRT_ROC(hw->port), pf->stat_offsets_loaded, &osd->rx_oversize, &nsd->rx_oversize); ixl_stat_update32(hw, I40E_GLPRT_RJC(hw->port), pf->stat_offsets_loaded, &osd->rx_jabber, &nsd->rx_jabber); pf->stat_offsets_loaded = true; /* End hw stats */ /* Update vsi stats */ ixl_update_vsi_stats(vsi); for (int i = 0; i < pf->num_vfs; i++) { vf = &pf->vfs[i]; if (vf->vf_flags & VF_FLAG_ENABLED) ixl_update_eth_stats(&pf->vfs[i].vsi); } } /* ** Tasklet handler for MSIX Adminq interrupts ** - do outside interrupt since it might sleep */ static void ixl_do_adminq(void *context, int pending) { struct ixl_pf *pf = context; struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct i40e_arq_event_info event; i40e_status ret; u32 reg, loop = 0; u16 opcode, result; event.buf_len = IXL_AQ_BUF_SZ; event.msg_buf = malloc(event.buf_len, M_DEVBUF, M_NOWAIT | M_ZERO); if (!event.msg_buf) { printf("Unable to allocate adminq memory\n"); return; } IXL_PF_LOCK(pf); /* clean and process any events */ do { ret = i40e_clean_arq_element(hw, &event, &result); if (ret) break; opcode = LE16_TO_CPU(event.desc.opcode); switch (opcode) { case i40e_aqc_opc_get_link_status: ixl_link_event(pf, &event); ixl_update_link_status(pf); break; case i40e_aqc_opc_send_msg_to_pf: #ifdef PCI_IOV ixl_handle_vf_msg(pf, &event); #endif break; case i40e_aqc_opc_event_lan_overflow: break; default: #ifdef IXL_DEBUG printf("AdminQ unknown event %x\n", opcode); #endif break; } } while (result && (loop++ < IXL_ADM_LIMIT)); reg = rd32(hw, I40E_PFINT_ICR0_ENA); reg |= I40E_PFINT_ICR0_ENA_ADMINQ_MASK; wr32(hw, I40E_PFINT_ICR0_ENA, reg); free(event.msg_buf, M_DEVBUF); /* * If there are still messages to process, reschedule ourselves. * Otherwise, re-enable our interrupt and go to sleep. */ if (result > 0) taskqueue_enqueue(pf->tq, &pf->adminq); else ixl_enable_intr(vsi); IXL_PF_UNLOCK(pf); } #ifdef IXL_DEBUG_SYSCTL static int ixl_debug_info(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf; int error, input = 0; error = sysctl_handle_int(oidp, &input, 0, req); if (error || !req->newptr) return (error); if (input == 1) { pf = (struct ixl_pf *)arg1; ixl_print_debug_info(pf); } return (error); } static void ixl_print_debug_info(struct ixl_pf *pf) { struct i40e_hw *hw = &pf->hw; struct ixl_vsi *vsi = &pf->vsi; struct ixl_queue *que = vsi->queues; struct rx_ring *rxr = &que->rxr; struct tx_ring *txr = &que->txr; u32 reg; printf("Queue irqs = %jx\n", (uintmax_t)que->irqs); printf("AdminQ irqs = %jx\n", (uintmax_t)pf->admin_irq); printf("RX next check = %x\n", rxr->next_check); printf("RX not ready = %jx\n", (uintmax_t)rxr->not_done); printf("RX packets = %jx\n", (uintmax_t)rxr->rx_packets); printf("TX desc avail = %x\n", txr->avail); reg = rd32(hw, I40E_GLV_GORCL(0xc)); printf("RX Bytes = %x\n", reg); reg = rd32(hw, I40E_GLPRT_GORCL(hw->port)); printf("Port RX Bytes = %x\n", reg); reg = rd32(hw, I40E_GLV_RDPC(0xc)); printf("RX discard = %x\n", reg); reg = rd32(hw, I40E_GLPRT_RDPC(hw->port)); printf("Port RX discard = %x\n", reg); reg = rd32(hw, I40E_GLV_TEPC(0xc)); printf("TX errors = %x\n", reg); reg = rd32(hw, I40E_GLV_GOTCL(0xc)); printf("TX Bytes = %x\n", reg); reg = rd32(hw, I40E_GLPRT_RUC(hw->port)); printf("RX undersize = %x\n", reg); reg = rd32(hw, I40E_GLPRT_RFC(hw->port)); printf("RX fragments = %x\n", reg); reg = rd32(hw, I40E_GLPRT_ROC(hw->port)); printf("RX oversize = %x\n", reg); reg = rd32(hw, I40E_GLPRT_RLEC(hw->port)); printf("RX length error = %x\n", reg); reg = rd32(hw, I40E_GLPRT_MRFC(hw->port)); printf("mac remote fault = %x\n", reg); reg = rd32(hw, I40E_GLPRT_MLFC(hw->port)); printf("mac local fault = %x\n", reg); } #endif /** * Update VSI-specific ethernet statistics counters. **/ void ixl_update_eth_stats(struct ixl_vsi *vsi) { struct ixl_pf *pf = (struct ixl_pf *)vsi->back; struct i40e_hw *hw = &pf->hw; struct i40e_eth_stats *es; struct i40e_eth_stats *oes; struct i40e_hw_port_stats *nsd; u16 stat_idx = vsi->info.stat_counter_idx; es = &vsi->eth_stats; oes = &vsi->eth_stats_offsets; nsd = &pf->stats; /* Gather up the stats that the hw collects */ ixl_stat_update32(hw, I40E_GLV_TEPC(stat_idx), vsi->stat_offsets_loaded, &oes->tx_errors, &es->tx_errors); ixl_stat_update32(hw, I40E_GLV_RDPC(stat_idx), vsi->stat_offsets_loaded, &oes->rx_discards, &es->rx_discards); ixl_stat_update48(hw, I40E_GLV_GORCH(stat_idx), I40E_GLV_GORCL(stat_idx), vsi->stat_offsets_loaded, &oes->rx_bytes, &es->rx_bytes); ixl_stat_update48(hw, I40E_GLV_UPRCH(stat_idx), I40E_GLV_UPRCL(stat_idx), vsi->stat_offsets_loaded, &oes->rx_unicast, &es->rx_unicast); ixl_stat_update48(hw, I40E_GLV_MPRCH(stat_idx), I40E_GLV_MPRCL(stat_idx), vsi->stat_offsets_loaded, &oes->rx_multicast, &es->rx_multicast); ixl_stat_update48(hw, I40E_GLV_BPRCH(stat_idx), I40E_GLV_BPRCL(stat_idx), vsi->stat_offsets_loaded, &oes->rx_broadcast, &es->rx_broadcast); ixl_stat_update48(hw, I40E_GLV_GOTCH(stat_idx), I40E_GLV_GOTCL(stat_idx), vsi->stat_offsets_loaded, &oes->tx_bytes, &es->tx_bytes); ixl_stat_update48(hw, I40E_GLV_UPTCH(stat_idx), I40E_GLV_UPTCL(stat_idx), vsi->stat_offsets_loaded, &oes->tx_unicast, &es->tx_unicast); ixl_stat_update48(hw, I40E_GLV_MPTCH(stat_idx), I40E_GLV_MPTCL(stat_idx), vsi->stat_offsets_loaded, &oes->tx_multicast, &es->tx_multicast); ixl_stat_update48(hw, I40E_GLV_BPTCH(stat_idx), I40E_GLV_BPTCL(stat_idx), vsi->stat_offsets_loaded, &oes->tx_broadcast, &es->tx_broadcast); vsi->stat_offsets_loaded = true; } static void ixl_update_vsi_stats(struct ixl_vsi *vsi) { struct ixl_pf *pf; struct ifnet *ifp; struct i40e_eth_stats *es; u64 tx_discards; struct i40e_hw_port_stats *nsd; pf = vsi->back; ifp = vsi->ifp; es = &vsi->eth_stats; nsd = &pf->stats; ixl_update_eth_stats(vsi); tx_discards = es->tx_discards + nsd->tx_dropped_link_down; for (int i = 0; i < vsi->num_queues; i++) tx_discards += vsi->queues[i].txr.br->br_drops; /* Update ifnet stats */ IXL_SET_IPACKETS(vsi, es->rx_unicast + es->rx_multicast + es->rx_broadcast); IXL_SET_OPACKETS(vsi, es->tx_unicast + es->tx_multicast + es->tx_broadcast); IXL_SET_IBYTES(vsi, es->rx_bytes); IXL_SET_OBYTES(vsi, es->tx_bytes); IXL_SET_IMCASTS(vsi, es->rx_multicast); IXL_SET_OMCASTS(vsi, es->tx_multicast); IXL_SET_IERRORS(vsi, nsd->crc_errors + nsd->illegal_bytes + nsd->rx_undersize + nsd->rx_oversize + nsd->rx_fragments + nsd->rx_jabber); IXL_SET_OERRORS(vsi, es->tx_errors); IXL_SET_IQDROPS(vsi, es->rx_discards + nsd->eth.rx_discards); IXL_SET_OQDROPS(vsi, tx_discards); IXL_SET_NOPROTO(vsi, es->rx_unknown_protocol); IXL_SET_COLLISIONS(vsi, 0); } /** * Reset all of the stats for the given pf **/ void ixl_pf_reset_stats(struct ixl_pf *pf) { bzero(&pf->stats, sizeof(struct i40e_hw_port_stats)); bzero(&pf->stats_offsets, sizeof(struct i40e_hw_port_stats)); pf->stat_offsets_loaded = false; } /** * Resets all stats of the given vsi **/ void ixl_vsi_reset_stats(struct ixl_vsi *vsi) { bzero(&vsi->eth_stats, sizeof(struct i40e_eth_stats)); bzero(&vsi->eth_stats_offsets, sizeof(struct i40e_eth_stats)); vsi->stat_offsets_loaded = false; } /** * Read and update a 48 bit stat from the hw * * Since the device stats are not reset at PFReset, they likely will not * be zeroed when the driver starts. We'll save the first values read * and use them as offsets to be subtracted from the raw values in order * to report stats that count from zero. **/ static void ixl_stat_update48(struct i40e_hw *hw, u32 hireg, u32 loreg, bool offset_loaded, u64 *offset, u64 *stat) { u64 new_data; #if defined(__FreeBSD__) && (__FreeBSD_version >= 1000000) && defined(__amd64__) new_data = rd64(hw, loreg); #else /* * Use two rd32's instead of one rd64; FreeBSD versions before * 10 don't support 8 byte bus reads/writes. */ new_data = rd32(hw, loreg); new_data |= ((u64)(rd32(hw, hireg) & 0xFFFF)) << 32; #endif if (!offset_loaded) *offset = new_data; if (new_data >= *offset) *stat = new_data - *offset; else *stat = (new_data + ((u64)1 << 48)) - *offset; *stat &= 0xFFFFFFFFFFFFULL; } /** * Read and update a 32 bit stat from the hw **/ static void ixl_stat_update32(struct i40e_hw *hw, u32 reg, bool offset_loaded, u64 *offset, u64 *stat) { u32 new_data; new_data = rd32(hw, reg); if (!offset_loaded) *offset = new_data; if (new_data >= *offset) *stat = (u32)(new_data - *offset); else *stat = (u32)((new_data + ((u64)1 << 32)) - *offset); } /* ** Set flow control using sysctl: ** 0 - off ** 1 - rx pause ** 2 - tx pause ** 3 - full */ static int ixl_set_flowcntl(SYSCTL_HANDLER_ARGS) { /* * TODO: ensure flow control is disabled if * priority flow control is enabled * * TODO: ensure tx CRC by hardware should be enabled * if tx flow control is enabled. */ struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; int error = 0; enum i40e_status_code aq_error = 0; u8 fc_aq_err = 0; /* Get request */ error = sysctl_handle_int(oidp, &pf->fc, 0, req); if ((error) || (req->newptr == NULL)) return (error); if (pf->fc < 0 || pf->fc > 3) { device_printf(dev, "Invalid fc mode; valid modes are 0 through 3\n"); return (EINVAL); } /* ** Changing flow control mode currently does not work on ** 40GBASE-CR4 PHYs */ if (hw->phy.link_info.phy_type == I40E_PHY_TYPE_40GBASE_CR4 || hw->phy.link_info.phy_type == I40E_PHY_TYPE_40GBASE_CR4_CU) { device_printf(dev, "Changing flow control mode unsupported" " on 40GBase-CR4 media.\n"); return (ENODEV); } /* Set fc ability for port */ hw->fc.requested_mode = pf->fc; aq_error = i40e_set_fc(hw, &fc_aq_err, TRUE); if (aq_error) { device_printf(dev, "%s: Error setting new fc mode %d; fc_err %#x\n", __func__, aq_error, fc_aq_err); return (EAGAIN); } return (0); } static int ixl_current_speed(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; int error = 0, index = 0; char *speeds[] = { "Unknown", "100M", "1G", "10G", "40G", "20G" }; ixl_update_link_status(pf); switch (hw->phy.link_info.link_speed) { case I40E_LINK_SPEED_100MB: index = 1; break; case I40E_LINK_SPEED_1GB: index = 2; break; case I40E_LINK_SPEED_10GB: index = 3; break; case I40E_LINK_SPEED_40GB: index = 4; break; case I40E_LINK_SPEED_20GB: index = 5; break; case I40E_LINK_SPEED_UNKNOWN: default: index = 0; break; } error = sysctl_handle_string(oidp, speeds[index], strlen(speeds[index]), req); return (error); } static int ixl_set_advertised_speeds(struct ixl_pf *pf, int speeds) { struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; struct i40e_aq_get_phy_abilities_resp abilities; struct i40e_aq_set_phy_config config; enum i40e_status_code aq_error = 0; /* Get current capability information */ aq_error = i40e_aq_get_phy_capabilities(hw, FALSE, FALSE, &abilities, NULL); if (aq_error) { device_printf(dev, "%s: Error getting phy capabilities %d," " aq error: %d\n", __func__, aq_error, hw->aq.asq_last_status); return (EAGAIN); } /* Prepare new config */ bzero(&config, sizeof(config)); config.phy_type = abilities.phy_type; config.abilities = abilities.abilities | I40E_AQ_PHY_ENABLE_ATOMIC_LINK; config.eee_capability = abilities.eee_capability; config.eeer = abilities.eeer_val; config.low_power_ctrl = abilities.d3_lpan; /* Translate into aq cmd link_speed */ if (speeds & 0x8) config.link_speed |= I40E_LINK_SPEED_20GB; if (speeds & 0x4) config.link_speed |= I40E_LINK_SPEED_10GB; if (speeds & 0x2) config.link_speed |= I40E_LINK_SPEED_1GB; if (speeds & 0x1) config.link_speed |= I40E_LINK_SPEED_100MB; /* Do aq command & restart link */ aq_error = i40e_aq_set_phy_config(hw, &config, NULL); if (aq_error) { device_printf(dev, "%s: Error setting new phy config %d," " aq error: %d\n", __func__, aq_error, hw->aq.asq_last_status); return (EAGAIN); } /* ** This seems a bit heavy handed, but we ** need to get a reinit on some devices */ IXL_PF_LOCK(pf); ixl_stop(pf); ixl_init_locked(pf); IXL_PF_UNLOCK(pf); return (0); } /* ** Control link advertise speed: ** Flags: ** 0x1 - advertise 100 Mb ** 0x2 - advertise 1G ** 0x4 - advertise 10G ** 0x8 - advertise 20G ** ** Does not work on 40G devices. */ static int ixl_set_advertise(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; int requested_ls = 0; int error = 0; /* ** FW doesn't support changing advertised speed ** for 40G devices; speed is always 40G. */ if (i40e_is_40G_device(hw->device_id)) return (ENODEV); /* Read in new mode */ requested_ls = pf->advertised_speed; error = sysctl_handle_int(oidp, &requested_ls, 0, req); if ((error) || (req->newptr == NULL)) return (error); /* Check for sane value */ if (requested_ls < 0x1 || requested_ls > 0xE) { device_printf(dev, "Invalid advertised speed; " "valid modes are 0x1 through 0xE\n"); return (EINVAL); } /* Then check for validity based on adapter type */ switch (hw->device_id) { case I40E_DEV_ID_10G_BASE_T: if (requested_ls & 0x8) { device_printf(dev, "20Gbs speed not supported on this device.\n"); return (EINVAL); } break; case I40E_DEV_ID_20G_KR2: if (requested_ls & 0x1) { device_printf(dev, "100Mbs speed not supported on this device.\n"); return (EINVAL); } break; default: if (requested_ls & ~0x6) { device_printf(dev, "Only 1/10Gbs speeds are supported on this device.\n"); return (EINVAL); } break; } /* Exit if no change */ if (pf->advertised_speed == requested_ls) return (0); error = ixl_set_advertised_speeds(pf, requested_ls); if (error) return (error); pf->advertised_speed = requested_ls; ixl_update_link_status(pf); return (0); } /* ** Get the width and transaction speed of ** the bus this adapter is plugged into. */ static u16 ixl_get_bus_info(struct i40e_hw *hw, device_t dev) { u16 link; u32 offset; /* Get the PCI Express Capabilities offset */ pci_find_cap(dev, PCIY_EXPRESS, &offset); /* ...and read the Link Status Register */ link = pci_read_config(dev, offset + PCIER_LINK_STA, 2); switch (link & I40E_PCI_LINK_WIDTH) { case I40E_PCI_LINK_WIDTH_1: hw->bus.width = i40e_bus_width_pcie_x1; break; case I40E_PCI_LINK_WIDTH_2: hw->bus.width = i40e_bus_width_pcie_x2; break; case I40E_PCI_LINK_WIDTH_4: hw->bus.width = i40e_bus_width_pcie_x4; break; case I40E_PCI_LINK_WIDTH_8: hw->bus.width = i40e_bus_width_pcie_x8; break; default: hw->bus.width = i40e_bus_width_unknown; break; } switch (link & I40E_PCI_LINK_SPEED) { case I40E_PCI_LINK_SPEED_2500: hw->bus.speed = i40e_bus_speed_2500; break; case I40E_PCI_LINK_SPEED_5000: hw->bus.speed = i40e_bus_speed_5000; break; case I40E_PCI_LINK_SPEED_8000: hw->bus.speed = i40e_bus_speed_8000; break; default: hw->bus.speed = i40e_bus_speed_unknown; break; } device_printf(dev,"PCI Express Bus: Speed %s %s\n", ((hw->bus.speed == i40e_bus_speed_8000) ? "8.0GT/s": (hw->bus.speed == i40e_bus_speed_5000) ? "5.0GT/s": (hw->bus.speed == i40e_bus_speed_2500) ? "2.5GT/s":"Unknown"), (hw->bus.width == i40e_bus_width_pcie_x8) ? "Width x8" : (hw->bus.width == i40e_bus_width_pcie_x4) ? "Width x4" : (hw->bus.width == i40e_bus_width_pcie_x1) ? "Width x1" : ("Unknown")); if ((hw->bus.width <= i40e_bus_width_pcie_x8) && (hw->bus.speed < i40e_bus_speed_8000)) { device_printf(dev, "PCI-Express bandwidth available" " for this device\n may be insufficient for" " optimal performance.\n"); device_printf(dev, "For expected performance a x8 " "PCIE Gen3 slot is required.\n"); } return (link); } static int ixl_sysctl_show_fw(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; char buf[32]; snprintf(buf, sizeof(buf), "f%d.%d a%d.%d n%02x.%02x e%08x", hw->aq.fw_maj_ver, hw->aq.fw_min_ver, hw->aq.api_maj_ver, hw->aq.api_min_ver, (hw->nvm.version & IXL_NVM_VERSION_HI_MASK) >> IXL_NVM_VERSION_HI_SHIFT, (hw->nvm.version & IXL_NVM_VERSION_LO_MASK) >> IXL_NVM_VERSION_LO_SHIFT, hw->nvm.eetrack); return (sysctl_handle_string(oidp, buf, strlen(buf), req)); } #ifdef IXL_DEBUG_SYSCTL static int ixl_sysctl_link_status(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; struct i40e_link_status link_status; char buf[512]; enum i40e_status_code aq_error = 0; aq_error = i40e_aq_get_link_info(hw, TRUE, &link_status, NULL); if (aq_error) { printf("i40e_aq_get_link_info() error %d\n", aq_error); return (EPERM); } sprintf(buf, "\n" "PHY Type : %#04x\n" "Speed : %#04x\n" "Link info: %#04x\n" "AN info : %#04x\n" "Ext info : %#04x", link_status.phy_type, link_status.link_speed, link_status.link_info, link_status.an_info, link_status.ext_info); return (sysctl_handle_string(oidp, buf, strlen(buf), req)); } static int ixl_sysctl_phy_abilities(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; char buf[512]; enum i40e_status_code aq_error = 0; struct i40e_aq_get_phy_abilities_resp abilities; aq_error = i40e_aq_get_phy_capabilities(hw, TRUE, FALSE, &abilities, NULL); if (aq_error) { printf("i40e_aq_get_phy_capabilities() error %d\n", aq_error); return (EPERM); } sprintf(buf, "\n" "PHY Type : %#010x\n" "Speed : %#04x\n" "Abilities: %#04x\n" "EEE cap : %#06x\n" "EEER reg : %#010x\n" "D3 Lpan : %#04x", abilities.phy_type, abilities.link_speed, abilities.abilities, abilities.eee_capability, abilities.eeer_val, abilities.d3_lpan); return (sysctl_handle_string(oidp, buf, strlen(buf), req)); } static int ixl_sysctl_sw_filter_list(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct ixl_vsi *vsi = &pf->vsi; struct ixl_mac_filter *f; char *buf, *buf_i; int error = 0; int ftl_len = 0; int ftl_counter = 0; int buf_len = 0; int entry_len = 42; SLIST_FOREACH(f, &vsi->ftl, next) { ftl_len++; } if (ftl_len < 1) { sysctl_handle_string(oidp, "(none)", 6, req); return (0); } buf_len = sizeof(char) * (entry_len + 1) * ftl_len + 2; buf = buf_i = malloc(buf_len, M_DEVBUF, M_NOWAIT); sprintf(buf_i++, "\n"); SLIST_FOREACH(f, &vsi->ftl, next) { sprintf(buf_i, MAC_FORMAT ", vlan %4d, flags %#06x", MAC_FORMAT_ARGS(f->macaddr), f->vlan, f->flags); buf_i += entry_len; /* don't print '\n' for last entry */ if (++ftl_counter != ftl_len) { sprintf(buf_i, "\n"); buf_i++; } } error = sysctl_handle_string(oidp, buf, strlen(buf), req); if (error) printf("sysctl error: %d\n", error); free(buf, M_DEVBUF); return error; } #define IXL_SW_RES_SIZE 0x14 static int ixl_res_alloc_cmp(const void *a, const void *b) { const struct i40e_aqc_switch_resource_alloc_element_resp *one, *two; one = (const struct i40e_aqc_switch_resource_alloc_element_resp *)a; two = (const struct i40e_aqc_switch_resource_alloc_element_resp *)b; return ((int)one->resource_type - (int)two->resource_type); } static int ixl_sysctl_hw_res_alloc(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; struct sbuf *buf; int error = 0; u8 num_entries; struct i40e_aqc_switch_resource_alloc_element_resp resp[IXL_SW_RES_SIZE]; buf = sbuf_new_for_sysctl(NULL, NULL, 0, req); if (!buf) { device_printf(dev, "Could not allocate sbuf for output.\n"); return (ENOMEM); } bzero(resp, sizeof(resp)); error = i40e_aq_get_switch_resource_alloc(hw, &num_entries, resp, IXL_SW_RES_SIZE, NULL); if (error) { device_printf(dev, "%s: get_switch_resource_alloc() error %d, aq error %d\n", __func__, error, hw->aq.asq_last_status); sbuf_delete(buf); return error; } /* Sort entries by type for display */ qsort(resp, num_entries, sizeof(struct i40e_aqc_switch_resource_alloc_element_resp), &ixl_res_alloc_cmp); sbuf_cat(buf, "\n"); sbuf_printf(buf, "# of entries: %d\n", num_entries); sbuf_printf(buf, "Type | Guaranteed | Total | Used | Un-allocated\n" " | (this) | (all) | (this) | (all) \n"); for (int i = 0; i < num_entries; i++) { sbuf_printf(buf, "%#4x | %10d %5d %6d %12d", resp[i].resource_type, resp[i].guaranteed, resp[i].total, resp[i].used, resp[i].total_unalloced); if (i < num_entries - 1) sbuf_cat(buf, "\n"); } error = sbuf_finish(buf); sbuf_delete(buf); return (error); } /* ** Caller must init and delete sbuf; this function will clear and ** finish it for caller. */ static char * ixl_switch_element_string(struct sbuf *s, u16 seid, bool uplink) { sbuf_clear(s); if (seid == 0 && uplink) sbuf_cat(s, "Network"); else if (seid == 0) sbuf_cat(s, "Host"); else if (seid == 1) sbuf_cat(s, "EMP"); else if (seid <= 5) sbuf_printf(s, "MAC %d", seid - 2); else if (seid <= 15) sbuf_cat(s, "Reserved"); else if (seid <= 31) sbuf_printf(s, "PF %d", seid - 16); else if (seid <= 159) sbuf_printf(s, "VF %d", seid - 32); else if (seid <= 287) sbuf_cat(s, "Reserved"); else if (seid <= 511) sbuf_cat(s, "Other"); // for other structures else if (seid <= 895) sbuf_printf(s, "VSI %d", seid - 512); else if (seid <= 1023) sbuf_printf(s, "Reserved"); else sbuf_cat(s, "Invalid"); sbuf_finish(s); return sbuf_data(s); } static int ixl_sysctl_switch_config(SYSCTL_HANDLER_ARGS) { struct ixl_pf *pf = (struct ixl_pf *)arg1; struct i40e_hw *hw = &pf->hw; device_t dev = pf->dev; struct sbuf *buf; struct sbuf *nmbuf; int error = 0; u8 aq_buf[I40E_AQ_LARGE_BUF]; u16 next = 0; struct i40e_aqc_get_switch_config_resp *sw_config; sw_config = (struct i40e_aqc_get_switch_config_resp *)aq_buf; buf = sbuf_new_for_sysctl(NULL, NULL, 0, req); if (!buf) { device_printf(dev, "Could not allocate sbuf for sysctl output.\n"); return (ENOMEM); } error = i40e_aq_get_switch_config(hw, sw_config, sizeof(aq_buf), &next, NULL); if (error) { device_printf(dev, "%s: aq_get_switch_config() error %d, aq error %d\n", __func__, error, hw->aq.asq_last_status); sbuf_delete(buf); return error; } nmbuf = sbuf_new_auto(); if (!nmbuf) { device_printf(dev, "Could not allocate sbuf for name output.\n"); return (ENOMEM); } sbuf_cat(buf, "\n"); // Assuming <= 255 elements in switch sbuf_printf(buf, "# of elements: %d\n", sw_config->header.num_reported); /* Exclude: ** Revision -- all elements are revision 1 for now */ sbuf_printf(buf, "SEID ( Name ) | Uplink | Downlink | Conn Type\n" " | | | (uplink)\n"); for (int i = 0; i < sw_config->header.num_reported; i++) { // "%4d (%8s) | %8s %8s %#8x", sbuf_printf(buf, "%4d", sw_config->element[i].seid); sbuf_cat(buf, " "); sbuf_printf(buf, "(%8s)", ixl_switch_element_string(nmbuf, sw_config->element[i].seid, false)); sbuf_cat(buf, " | "); sbuf_printf(buf, "%8s", ixl_switch_element_string(nmbuf, sw_config->element[i].uplink_seid, true)); sbuf_cat(buf, " "); sbuf_printf(buf, "%8s", ixl_switch_element_string(nmbuf, sw_config->element[i].downlink_seid, false)); sbuf_cat(buf, " "); sbuf_printf(buf, "%#8x", sw_config->element[i].connection_type); if (i < sw_config->header.num_reported - 1) sbuf_cat(buf, "\n"); } sbuf_delete(nmbuf); error = sbuf_finish(buf); sbuf_delete(buf); return (error); } #endif /* IXL_DEBUG_SYSCTL */ #ifdef PCI_IOV static int ixl_vf_alloc_vsi(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_hw *hw; struct ixl_vsi *vsi; struct i40e_vsi_context vsi_ctx; int i; uint16_t first_queue; enum i40e_status_code code; hw = &pf->hw; vsi = &pf->vsi; vsi_ctx.pf_num = hw->pf_id; vsi_ctx.uplink_seid = pf->veb_seid; vsi_ctx.connection_type = IXL_VSI_DATA_PORT; vsi_ctx.vf_num = hw->func_caps.vf_base_id + vf->vf_num; vsi_ctx.flags = I40E_AQ_VSI_TYPE_VF; bzero(&vsi_ctx.info, sizeof(vsi_ctx.info)); vsi_ctx.info.valid_sections = htole16(I40E_AQ_VSI_PROP_SWITCH_VALID); vsi_ctx.info.switch_id = htole16(0); vsi_ctx.info.valid_sections |= htole16(I40E_AQ_VSI_PROP_SECURITY_VALID); vsi_ctx.info.sec_flags = 0; if (vf->vf_flags & VF_FLAG_MAC_ANTI_SPOOF) vsi_ctx.info.sec_flags |= I40E_AQ_VSI_SEC_FLAG_ENABLE_MAC_CHK; vsi_ctx.info.valid_sections |= htole16(I40E_AQ_VSI_PROP_VLAN_VALID); vsi_ctx.info.port_vlan_flags = I40E_AQ_VSI_PVLAN_MODE_ALL | I40E_AQ_VSI_PVLAN_EMOD_NOTHING; vsi_ctx.info.valid_sections |= htole16(I40E_AQ_VSI_PROP_QUEUE_MAP_VALID); vsi_ctx.info.mapping_flags = htole16(I40E_AQ_VSI_QUE_MAP_NONCONTIG); first_queue = vsi->num_queues + vf->vf_num * IXLV_MAX_QUEUES; for (i = 0; i < IXLV_MAX_QUEUES; i++) vsi_ctx.info.queue_mapping[i] = htole16(first_queue + i); for (; i < nitems(vsi_ctx.info.queue_mapping); i++) vsi_ctx.info.queue_mapping[i] = htole16(I40E_AQ_VSI_QUEUE_MASK); vsi_ctx.info.tc_mapping[0] = htole16( (0 << I40E_AQ_VSI_TC_QUE_OFFSET_SHIFT) | (1 << I40E_AQ_VSI_TC_QUE_NUMBER_SHIFT)); code = i40e_aq_add_vsi(hw, &vsi_ctx, NULL); if (code != I40E_SUCCESS) return (ixl_adminq_err_to_errno(hw->aq.asq_last_status)); vf->vsi.seid = vsi_ctx.seid; vf->vsi.vsi_num = vsi_ctx.vsi_number; vf->vsi.first_queue = first_queue; vf->vsi.num_queues = IXLV_MAX_QUEUES; code = i40e_aq_get_vsi_params(hw, &vsi_ctx, NULL); if (code != I40E_SUCCESS) return (ixl_adminq_err_to_errno(hw->aq.asq_last_status)); code = i40e_aq_config_vsi_bw_limit(hw, vf->vsi.seid, 0, 0, NULL); if (code != I40E_SUCCESS) { device_printf(pf->dev, "Failed to disable BW limit: %d\n", ixl_adminq_err_to_errno(hw->aq.asq_last_status)); return (ixl_adminq_err_to_errno(hw->aq.asq_last_status)); } memcpy(&vf->vsi.info, &vsi_ctx.info, sizeof(vf->vsi.info)); return (0); } static int ixl_vf_setup_vsi(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_hw *hw; int error; hw = &pf->hw; error = ixl_vf_alloc_vsi(pf, vf); if (error != 0) return (error); vf->vsi.hw_filters_add = 0; vf->vsi.hw_filters_del = 0; ixl_add_filter(&vf->vsi, ixl_bcast_addr, IXL_VLAN_ANY); ixl_reconfigure_filters(&vf->vsi); return (0); } static void ixl_vf_map_vsi_queue(struct i40e_hw *hw, struct ixl_vf *vf, int qnum, uint32_t val) { uint32_t qtable; int index, shift; /* * Two queues are mapped in a single register, so we have to do some * gymnastics to convert the queue number into a register index and * shift. */ index = qnum / 2; shift = (qnum % 2) * I40E_VSILAN_QTABLE_QINDEX_1_SHIFT; qtable = rd32(hw, I40E_VSILAN_QTABLE(index, vf->vsi.vsi_num)); qtable &= ~(I40E_VSILAN_QTABLE_QINDEX_0_MASK << shift); qtable |= val << shift; wr32(hw, I40E_VSILAN_QTABLE(index, vf->vsi.vsi_num), qtable); } static void ixl_vf_map_queues(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_hw *hw; uint32_t qtable; int i; hw = &pf->hw; /* * Contiguous mappings aren't actually supported by the hardware, * so we have to use non-contiguous mappings. */ wr32(hw, I40E_VSILAN_QBASE(vf->vsi.vsi_num), I40E_VSILAN_QBASE_VSIQTABLE_ENA_MASK); wr32(hw, I40E_VPLAN_MAPENA(vf->vf_num), I40E_VPLAN_MAPENA_TXRX_ENA_MASK); for (i = 0; i < vf->vsi.num_queues; i++) { qtable = (vf->vsi.first_queue + i) << I40E_VPLAN_QTABLE_QINDEX_SHIFT; wr32(hw, I40E_VPLAN_QTABLE(i, vf->vf_num), qtable); } /* Map queues allocated to VF to its VSI. */ for (i = 0; i < vf->vsi.num_queues; i++) ixl_vf_map_vsi_queue(hw, vf, i, vf->vsi.first_queue + i); /* Set rest of VSI queues as unused. */ for (; i < IXL_MAX_VSI_QUEUES; i++) ixl_vf_map_vsi_queue(hw, vf, i, I40E_VSILAN_QTABLE_QINDEX_0_MASK); ixl_flush(hw); } static void ixl_vf_vsi_release(struct ixl_pf *pf, struct ixl_vsi *vsi) { struct i40e_hw *hw; hw = &pf->hw; if (vsi->seid == 0) return; i40e_aq_delete_element(hw, vsi->seid, NULL); } static void ixl_vf_disable_queue_intr(struct i40e_hw *hw, uint32_t vfint_reg) { wr32(hw, vfint_reg, I40E_VFINT_DYN_CTLN_CLEARPBA_MASK); ixl_flush(hw); } static void ixl_vf_unregister_intr(struct i40e_hw *hw, uint32_t vpint_reg) { wr32(hw, vpint_reg, I40E_VPINT_LNKLSTN_FIRSTQ_TYPE_MASK | I40E_VPINT_LNKLSTN_FIRSTQ_INDX_MASK); ixl_flush(hw); } static void ixl_vf_release_resources(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_hw *hw; uint32_t vfint_reg, vpint_reg; int i; hw = &pf->hw; ixl_vf_vsi_release(pf, &vf->vsi); /* Index 0 has a special register. */ ixl_vf_disable_queue_intr(hw, I40E_VFINT_DYN_CTL0(vf->vf_num)); for (i = 1; i < hw->func_caps.num_msix_vectors_vf; i++) { vfint_reg = IXL_VFINT_DYN_CTLN_REG(hw, i , vf->vf_num); ixl_vf_disable_queue_intr(hw, vfint_reg); } /* Index 0 has a special register. */ ixl_vf_unregister_intr(hw, I40E_VPINT_LNKLST0(vf->vf_num)); for (i = 1; i < hw->func_caps.num_msix_vectors_vf; i++) { vpint_reg = IXL_VPINT_LNKLSTN_REG(hw, i, vf->vf_num); ixl_vf_unregister_intr(hw, vpint_reg); } vf->vsi.num_queues = 0; } static int ixl_flush_pcie(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_hw *hw; int i; uint16_t global_vf_num; uint32_t ciad; hw = &pf->hw; global_vf_num = hw->func_caps.vf_base_id + vf->vf_num; wr32(hw, I40E_PF_PCI_CIAA, IXL_PF_PCI_CIAA_VF_DEVICE_STATUS | (global_vf_num << I40E_PF_PCI_CIAA_VF_NUM_SHIFT)); for (i = 0; i < IXL_VF_RESET_TIMEOUT; i++) { ciad = rd32(hw, I40E_PF_PCI_CIAD); if ((ciad & IXL_PF_PCI_CIAD_VF_TRANS_PENDING_MASK) == 0) return (0); DELAY(1); } return (ETIMEDOUT); } static void ixl_reset_vf(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_hw *hw; uint32_t vfrtrig; hw = &pf->hw; vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); vfrtrig |= I40E_VPGEN_VFRTRIG_VFSWR_MASK; wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); ixl_flush(hw); ixl_reinit_vf(pf, vf); } static void ixl_reinit_vf(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_hw *hw; uint32_t vfrstat, vfrtrig; int i, error; hw = &pf->hw; error = ixl_flush_pcie(pf, vf); if (error != 0) device_printf(pf->dev, "Timed out waiting for PCIe activity to stop on VF-%d\n", vf->vf_num); for (i = 0; i < IXL_VF_RESET_TIMEOUT; i++) { DELAY(10); vfrstat = rd32(hw, I40E_VPGEN_VFRSTAT(vf->vf_num)); if (vfrstat & I40E_VPGEN_VFRSTAT_VFRD_MASK) break; } if (i == IXL_VF_RESET_TIMEOUT) device_printf(pf->dev, "VF %d failed to reset\n", vf->vf_num); wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_num), I40E_VFR_COMPLETED); vfrtrig = rd32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num)); vfrtrig &= ~I40E_VPGEN_VFRTRIG_VFSWR_MASK; wr32(hw, I40E_VPGEN_VFRTRIG(vf->vf_num), vfrtrig); if (vf->vsi.seid != 0) ixl_disable_rings(&vf->vsi); ixl_vf_release_resources(pf, vf); ixl_vf_setup_vsi(pf, vf); ixl_vf_map_queues(pf, vf); wr32(hw, I40E_VFGEN_RSTAT1(vf->vf_num), I40E_VFR_VFACTIVE); ixl_flush(hw); } static const char * ixl_vc_opcode_str(uint16_t op) { switch (op) { case I40E_VIRTCHNL_OP_VERSION: return ("VERSION"); case I40E_VIRTCHNL_OP_RESET_VF: return ("RESET_VF"); case I40E_VIRTCHNL_OP_GET_VF_RESOURCES: return ("GET_VF_RESOURCES"); case I40E_VIRTCHNL_OP_CONFIG_TX_QUEUE: return ("CONFIG_TX_QUEUE"); case I40E_VIRTCHNL_OP_CONFIG_RX_QUEUE: return ("CONFIG_RX_QUEUE"); case I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES: return ("CONFIG_VSI_QUEUES"); case I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP: return ("CONFIG_IRQ_MAP"); case I40E_VIRTCHNL_OP_ENABLE_QUEUES: return ("ENABLE_QUEUES"); case I40E_VIRTCHNL_OP_DISABLE_QUEUES: return ("DISABLE_QUEUES"); case I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS: return ("ADD_ETHER_ADDRESS"); case I40E_VIRTCHNL_OP_DEL_ETHER_ADDRESS: return ("DEL_ETHER_ADDRESS"); case I40E_VIRTCHNL_OP_ADD_VLAN: return ("ADD_VLAN"); case I40E_VIRTCHNL_OP_DEL_VLAN: return ("DEL_VLAN"); case I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE: return ("CONFIG_PROMISCUOUS_MODE"); case I40E_VIRTCHNL_OP_GET_STATS: return ("GET_STATS"); case I40E_VIRTCHNL_OP_FCOE: return ("FCOE"); case I40E_VIRTCHNL_OP_EVENT: return ("EVENT"); default: return ("UNKNOWN"); } } static int ixl_vc_opcode_level(uint16_t opcode) { switch (opcode) { case I40E_VIRTCHNL_OP_GET_STATS: return (10); default: return (5); } } static void ixl_send_vf_msg(struct ixl_pf *pf, struct ixl_vf *vf, uint16_t op, enum i40e_status_code status, void *msg, uint16_t len) { struct i40e_hw *hw; int global_vf_id; hw = &pf->hw; global_vf_id = hw->func_caps.vf_base_id + vf->vf_num; I40E_VC_DEBUG(pf, ixl_vc_opcode_level(op), "Sending msg (op=%s[%d], status=%d) to VF-%d\n", ixl_vc_opcode_str(op), op, status, vf->vf_num); i40e_aq_send_msg_to_vf(hw, global_vf_id, op, status, msg, len, NULL); } static void ixl_send_vf_ack(struct ixl_pf *pf, struct ixl_vf *vf, uint16_t op) { ixl_send_vf_msg(pf, vf, op, I40E_SUCCESS, NULL, 0); } static void ixl_send_vf_nack_msg(struct ixl_pf *pf, struct ixl_vf *vf, uint16_t op, enum i40e_status_code status, const char *file, int line) { I40E_VC_DEBUG(pf, 1, "Sending NACK (op=%s[%d], err=%d) to VF-%d from %s:%d\n", ixl_vc_opcode_str(op), op, status, vf->vf_num, file, line); ixl_send_vf_msg(pf, vf, op, status, NULL, 0); } static void ixl_vf_version_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_version_info reply; if (msg_size != sizeof(struct i40e_virtchnl_version_info)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_VERSION, I40E_ERR_PARAM); return; } reply.major = I40E_VIRTCHNL_VERSION_MAJOR; reply.minor = I40E_VIRTCHNL_VERSION_MINOR; ixl_send_vf_msg(pf, vf, I40E_VIRTCHNL_OP_VERSION, I40E_SUCCESS, &reply, sizeof(reply)); } static void ixl_vf_reset_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { if (msg_size != 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_RESET_VF, I40E_ERR_PARAM); return; } ixl_reset_vf(pf, vf); /* No response to a reset message. */ } static void ixl_vf_get_resources_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_vf_resource reply; if (msg_size != 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_GET_VF_RESOURCES, I40E_ERR_PARAM); return; } bzero(&reply, sizeof(reply)); reply.vf_offload_flags = I40E_VIRTCHNL_VF_OFFLOAD_L2; reply.num_vsis = 1; reply.num_queue_pairs = vf->vsi.num_queues; reply.max_vectors = pf->hw.func_caps.num_msix_vectors_vf; reply.vsi_res[0].vsi_id = vf->vsi.vsi_num; reply.vsi_res[0].vsi_type = I40E_VSI_SRIOV; reply.vsi_res[0].num_queue_pairs = vf->vsi.num_queues; memcpy(reply.vsi_res[0].default_mac_addr, vf->mac, ETHER_ADDR_LEN); ixl_send_vf_msg(pf, vf, I40E_VIRTCHNL_OP_GET_VF_RESOURCES, I40E_SUCCESS, &reply, sizeof(reply)); } static int ixl_vf_config_tx_queue(struct ixl_pf *pf, struct ixl_vf *vf, struct i40e_virtchnl_txq_info *info) { struct i40e_hw *hw; struct i40e_hmc_obj_txq txq; uint16_t global_queue_num, global_vf_num; enum i40e_status_code status; uint32_t qtx_ctl; hw = &pf->hw; global_queue_num = vf->vsi.first_queue + info->queue_id; global_vf_num = hw->func_caps.vf_base_id + vf->vf_num; bzero(&txq, sizeof(txq)); status = i40e_clear_lan_tx_queue_context(hw, global_queue_num); if (status != I40E_SUCCESS) return (EINVAL); txq.base = info->dma_ring_addr / IXL_TX_CTX_BASE_UNITS; txq.head_wb_ena = info->headwb_enabled; txq.head_wb_addr = info->dma_headwb_addr; txq.qlen = info->ring_len; txq.rdylist = le16_to_cpu(vf->vsi.info.qs_handle[0]); txq.rdylist_act = 0; status = i40e_set_lan_tx_queue_context(hw, global_queue_num, &txq); if (status != I40E_SUCCESS) return (EINVAL); qtx_ctl = I40E_QTX_CTL_VF_QUEUE | (hw->pf_id << I40E_QTX_CTL_PF_INDX_SHIFT) | (global_vf_num << I40E_QTX_CTL_VFVM_INDX_SHIFT); wr32(hw, I40E_QTX_CTL(global_queue_num), qtx_ctl); ixl_flush(hw); return (0); } static int ixl_vf_config_rx_queue(struct ixl_pf *pf, struct ixl_vf *vf, struct i40e_virtchnl_rxq_info *info) { struct i40e_hw *hw; struct i40e_hmc_obj_rxq rxq; uint16_t global_queue_num; enum i40e_status_code status; hw = &pf->hw; global_queue_num = vf->vsi.first_queue + info->queue_id; bzero(&rxq, sizeof(rxq)); if (info->databuffer_size > IXL_VF_MAX_BUFFER) return (EINVAL); if (info->max_pkt_size > IXL_VF_MAX_FRAME || info->max_pkt_size < ETHER_MIN_LEN) return (EINVAL); if (info->splithdr_enabled) { if (info->hdr_size > IXL_VF_MAX_HDR_BUFFER) return (EINVAL); rxq.hsplit_0 = info->rx_split_pos & (I40E_HMC_OBJ_RX_HSPLIT_0_SPLIT_L2 | I40E_HMC_OBJ_RX_HSPLIT_0_SPLIT_IP | I40E_HMC_OBJ_RX_HSPLIT_0_SPLIT_TCP_UDP | I40E_HMC_OBJ_RX_HSPLIT_0_SPLIT_SCTP); rxq.hbuff = info->hdr_size >> I40E_RXQ_CTX_HBUFF_SHIFT; rxq.dtype = 2; } status = i40e_clear_lan_rx_queue_context(hw, global_queue_num); if (status != I40E_SUCCESS) return (EINVAL); rxq.base = info->dma_ring_addr / IXL_RX_CTX_BASE_UNITS; rxq.qlen = info->ring_len; rxq.dbuff = info->databuffer_size >> I40E_RXQ_CTX_DBUFF_SHIFT; rxq.dsize = 1; rxq.crcstrip = 1; rxq.l2tsel = 1; rxq.rxmax = info->max_pkt_size; rxq.tphrdesc_ena = 1; rxq.tphwdesc_ena = 1; rxq.tphdata_ena = 1; rxq.tphhead_ena = 1; rxq.lrxqthresh = 2; rxq.prefena = 1; status = i40e_set_lan_rx_queue_context(hw, global_queue_num, &rxq); if (status != I40E_SUCCESS) return (EINVAL); return (0); } static void ixl_vf_config_vsi_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_vsi_queue_config_info *info; struct i40e_virtchnl_queue_pair_info *pair; int i; if (msg_size < sizeof(*info)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, I40E_ERR_PARAM); return; } info = msg; if (info->num_queue_pairs == 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, I40E_ERR_PARAM); return; } if (msg_size != sizeof(*info) + info->num_queue_pairs * sizeof(*pair)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, I40E_ERR_PARAM); return; } if (info->vsi_id != vf->vsi.vsi_num) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, I40E_ERR_PARAM); return; } for (i = 0; i < info->num_queue_pairs; i++) { pair = &info->qpair[i]; if (pair->txq.vsi_id != vf->vsi.vsi_num || pair->rxq.vsi_id != vf->vsi.vsi_num || pair->txq.queue_id != pair->rxq.queue_id || pair->txq.queue_id >= vf->vsi.num_queues) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, I40E_ERR_PARAM); return; } if (ixl_vf_config_tx_queue(pf, vf, &pair->txq) != 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, I40E_ERR_PARAM); return; } if (ixl_vf_config_rx_queue(pf, vf, &pair->rxq) != 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, I40E_ERR_PARAM); return; } } ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES); } static void ixl_vf_set_qctl(struct ixl_pf *pf, const struct i40e_virtchnl_vector_map *vector, enum i40e_queue_type cur_type, uint16_t cur_queue, enum i40e_queue_type *last_type, uint16_t *last_queue) { uint32_t offset, qctl; uint16_t itr_indx; if (cur_type == I40E_QUEUE_TYPE_RX) { offset = I40E_QINT_RQCTL(cur_queue); itr_indx = vector->rxitr_idx; } else { offset = I40E_QINT_TQCTL(cur_queue); itr_indx = vector->txitr_idx; } qctl = htole32((vector->vector_id << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) | (*last_type << I40E_QINT_RQCTL_NEXTQ_TYPE_SHIFT) | (*last_queue << I40E_QINT_RQCTL_NEXTQ_INDX_SHIFT) | I40E_QINT_RQCTL_CAUSE_ENA_MASK | (itr_indx << I40E_QINT_RQCTL_ITR_INDX_SHIFT)); wr32(&pf->hw, offset, qctl); *last_type = cur_type; *last_queue = cur_queue; } static void ixl_vf_config_vector(struct ixl_pf *pf, struct ixl_vf *vf, const struct i40e_virtchnl_vector_map *vector) { struct i40e_hw *hw; u_int qindex; enum i40e_queue_type type, last_type; uint32_t lnklst_reg; uint16_t rxq_map, txq_map, cur_queue, last_queue; hw = &pf->hw; rxq_map = vector->rxq_map; txq_map = vector->txq_map; last_queue = IXL_END_OF_INTR_LNKLST; last_type = I40E_QUEUE_TYPE_RX; /* * The datasheet says to optimize performance, RX queues and TX queues * should be interleaved in the interrupt linked list, so we process * both at once here. */ while ((rxq_map != 0) || (txq_map != 0)) { if (txq_map != 0) { qindex = ffs(txq_map) - 1; type = I40E_QUEUE_TYPE_TX; cur_queue = vf->vsi.first_queue + qindex; ixl_vf_set_qctl(pf, vector, type, cur_queue, &last_type, &last_queue); txq_map &= ~(1 << qindex); } if (rxq_map != 0) { qindex = ffs(rxq_map) - 1; type = I40E_QUEUE_TYPE_RX; cur_queue = vf->vsi.first_queue + qindex; ixl_vf_set_qctl(pf, vector, type, cur_queue, &last_type, &last_queue); rxq_map &= ~(1 << qindex); } } if (vector->vector_id == 0) lnklst_reg = I40E_VPINT_LNKLST0(vf->vf_num); else lnklst_reg = IXL_VPINT_LNKLSTN_REG(hw, vector->vector_id, vf->vf_num); wr32(hw, lnklst_reg, (last_queue << I40E_VPINT_LNKLST0_FIRSTQ_INDX_SHIFT) | (last_type << I40E_VPINT_LNKLST0_FIRSTQ_TYPE_SHIFT)); ixl_flush(hw); } static void ixl_vf_config_irq_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_irq_map_info *map; struct i40e_virtchnl_vector_map *vector; struct i40e_hw *hw; int i, largest_txq, largest_rxq; hw = &pf->hw; if (msg_size < sizeof(*map)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, I40E_ERR_PARAM); return; } map = msg; if (map->num_vectors == 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, I40E_ERR_PARAM); return; } if (msg_size != sizeof(*map) + map->num_vectors * sizeof(*vector)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, I40E_ERR_PARAM); return; } for (i = 0; i < map->num_vectors; i++) { vector = &map->vecmap[i]; if ((vector->vector_id >= hw->func_caps.num_msix_vectors_vf) || vector->vsi_id != vf->vsi.vsi_num) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, I40E_ERR_PARAM); return; } if (vector->rxq_map != 0) { largest_rxq = fls(vector->rxq_map) - 1; if (largest_rxq >= vf->vsi.num_queues) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, I40E_ERR_PARAM); return; } } if (vector->txq_map != 0) { largest_txq = fls(vector->txq_map) - 1; if (largest_txq >= vf->vsi.num_queues) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, I40E_ERR_PARAM); return; } } if (vector->rxitr_idx > IXL_MAX_ITR_IDX || vector->txitr_idx > IXL_MAX_ITR_IDX) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, I40E_ERR_PARAM); return; } ixl_vf_config_vector(pf, vf, vector); } ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP); } static void ixl_vf_enable_queues_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_queue_select *select; int error; if (msg_size != sizeof(*select)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ENABLE_QUEUES, I40E_ERR_PARAM); return; } select = msg; if (select->vsi_id != vf->vsi.vsi_num || select->rx_queues == 0 || select->tx_queues == 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ENABLE_QUEUES, I40E_ERR_PARAM); return; } error = ixl_enable_rings(&vf->vsi); if (error) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ENABLE_QUEUES, I40E_ERR_TIMEOUT); return; } ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_ENABLE_QUEUES); } static void ixl_vf_disable_queues_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_queue_select *select; int error; if (msg_size != sizeof(*select)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_DISABLE_QUEUES, I40E_ERR_PARAM); return; } select = msg; if (select->vsi_id != vf->vsi.vsi_num || select->rx_queues == 0 || select->tx_queues == 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_DISABLE_QUEUES, I40E_ERR_PARAM); return; } error = ixl_disable_rings(&vf->vsi); if (error) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_DISABLE_QUEUES, I40E_ERR_TIMEOUT); return; } ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_DISABLE_QUEUES); } static boolean_t ixl_zero_mac(const uint8_t *addr) { uint8_t zero[ETHER_ADDR_LEN] = {0, 0, 0, 0, 0, 0}; return (cmp_etheraddr(addr, zero)); } static boolean_t ixl_bcast_mac(const uint8_t *addr) { return (cmp_etheraddr(addr, ixl_bcast_addr)); } static int ixl_vf_mac_valid(struct ixl_vf *vf, const uint8_t *addr) { if (ixl_zero_mac(addr) || ixl_bcast_mac(addr)) return (EINVAL); /* * If the VF is not allowed to change its MAC address, don't let it * set a MAC filter for an address that is not a multicast address and * is not its assigned MAC. */ if (!(vf->vf_flags & VF_FLAG_SET_MAC_CAP) && !(ETHER_IS_MULTICAST(addr) || cmp_etheraddr(addr, vf->mac))) return (EPERM); return (0); } static void ixl_vf_add_mac_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_ether_addr_list *addr_list; struct i40e_virtchnl_ether_addr *addr; struct ixl_vsi *vsi; int i; size_t expected_size; vsi = &vf->vsi; if (msg_size < sizeof(*addr_list)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS, I40E_ERR_PARAM); return; } addr_list = msg; expected_size = sizeof(*addr_list) + addr_list->num_elements * sizeof(*addr); if (addr_list->num_elements == 0 || addr_list->vsi_id != vsi->vsi_num || msg_size != expected_size) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS, I40E_ERR_PARAM); return; } for (i = 0; i < addr_list->num_elements; i++) { if (ixl_vf_mac_valid(vf, addr_list->list[i].addr) != 0) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS, I40E_ERR_PARAM); return; } } for (i = 0; i < addr_list->num_elements; i++) { addr = &addr_list->list[i]; ixl_add_filter(vsi, addr->addr, IXL_VLAN_ANY); } ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS); } static void ixl_vf_del_mac_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_ether_addr_list *addr_list; struct i40e_virtchnl_ether_addr *addr; size_t expected_size; int i; if (msg_size < sizeof(*addr_list)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS, I40E_ERR_PARAM); return; } addr_list = msg; expected_size = sizeof(*addr_list) + addr_list->num_elements * sizeof(*addr); if (addr_list->num_elements == 0 || addr_list->vsi_id != vf->vsi.vsi_num || msg_size != expected_size) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS, I40E_ERR_PARAM); return; } for (i = 0; i < addr_list->num_elements; i++) { addr = &addr_list->list[i]; if (ixl_zero_mac(addr->addr) || ixl_bcast_mac(addr->addr)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS, I40E_ERR_PARAM); return; } } for (i = 0; i < addr_list->num_elements; i++) { addr = &addr_list->list[i]; ixl_del_filter(&vf->vsi, addr->addr, IXL_VLAN_ANY); } ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_DEL_ETHER_ADDRESS); } static enum i40e_status_code ixl_vf_enable_vlan_strip(struct ixl_pf *pf, struct ixl_vf *vf) { struct i40e_vsi_context vsi_ctx; vsi_ctx.seid = vf->vsi.seid; bzero(&vsi_ctx.info, sizeof(vsi_ctx.info)); vsi_ctx.info.valid_sections = htole16(I40E_AQ_VSI_PROP_VLAN_VALID); vsi_ctx.info.port_vlan_flags = I40E_AQ_VSI_PVLAN_MODE_ALL | I40E_AQ_VSI_PVLAN_EMOD_STR_BOTH; return (i40e_aq_update_vsi_params(&pf->hw, &vsi_ctx, NULL)); } static void ixl_vf_add_vlan_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_vlan_filter_list *filter_list; enum i40e_status_code code; size_t expected_size; int i; if (msg_size < sizeof(*filter_list)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN, I40E_ERR_PARAM); return; } filter_list = msg; expected_size = sizeof(*filter_list) + filter_list->num_elements * sizeof(uint16_t); if (filter_list->num_elements == 0 || filter_list->vsi_id != vf->vsi.vsi_num || msg_size != expected_size) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN, I40E_ERR_PARAM); return; } if (!(vf->vf_flags & VF_FLAG_VLAN_CAP)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN, I40E_ERR_PARAM); return; } for (i = 0; i < filter_list->num_elements; i++) { if (filter_list->vlan_id[i] > EVL_VLID_MASK) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN, I40E_ERR_PARAM); return; } } code = ixl_vf_enable_vlan_strip(pf, vf); if (code != I40E_SUCCESS) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN, I40E_ERR_PARAM); } for (i = 0; i < filter_list->num_elements; i++) ixl_add_filter(&vf->vsi, vf->mac, filter_list->vlan_id[i]); ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN); } static void ixl_vf_del_vlan_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_vlan_filter_list *filter_list; int i; size_t expected_size; if (msg_size < sizeof(*filter_list)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_DEL_VLAN, I40E_ERR_PARAM); return; } filter_list = msg; expected_size = sizeof(*filter_list) + filter_list->num_elements * sizeof(uint16_t); if (filter_list->num_elements == 0 || filter_list->vsi_id != vf->vsi.vsi_num || msg_size != expected_size) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_DEL_VLAN, I40E_ERR_PARAM); return; } for (i = 0; i < filter_list->num_elements; i++) { if (filter_list->vlan_id[i] > EVL_VLID_MASK) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN, I40E_ERR_PARAM); return; } } if (!(vf->vf_flags & VF_FLAG_VLAN_CAP)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_ADD_VLAN, I40E_ERR_PARAM); return; } for (i = 0; i < filter_list->num_elements; i++) ixl_del_filter(&vf->vsi, vf->mac, filter_list->vlan_id[i]); ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_DEL_VLAN); } static void ixl_vf_config_promisc_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_promisc_info *info; enum i40e_status_code code; if (msg_size != sizeof(*info)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE, I40E_ERR_PARAM); return; } if (!vf->vf_flags & VF_FLAG_PROMISC_CAP) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE, I40E_ERR_PARAM); return; } info = msg; if (info->vsi_id != vf->vsi.vsi_num) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE, I40E_ERR_PARAM); return; } code = i40e_aq_set_vsi_unicast_promiscuous(&pf->hw, info->vsi_id, info->flags & I40E_FLAG_VF_UNICAST_PROMISC, NULL); if (code != I40E_SUCCESS) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE, code); return; } code = i40e_aq_set_vsi_multicast_promiscuous(&pf->hw, info->vsi_id, info->flags & I40E_FLAG_VF_MULTICAST_PROMISC, NULL); if (code != I40E_SUCCESS) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE, code); return; } ixl_send_vf_ack(pf, vf, I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE); } static void ixl_vf_get_stats_msg(struct ixl_pf *pf, struct ixl_vf *vf, void *msg, uint16_t msg_size) { struct i40e_virtchnl_queue_select *queue; if (msg_size != sizeof(*queue)) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_GET_STATS, I40E_ERR_PARAM); return; } queue = msg; if (queue->vsi_id != vf->vsi.vsi_num) { i40e_send_vf_nack(pf, vf, I40E_VIRTCHNL_OP_GET_STATS, I40E_ERR_PARAM); return; } ixl_update_eth_stats(&vf->vsi); ixl_send_vf_msg(pf, vf, I40E_VIRTCHNL_OP_GET_STATS, I40E_SUCCESS, &vf->vsi.eth_stats, sizeof(vf->vsi.eth_stats)); } static void ixl_handle_vf_msg(struct ixl_pf *pf, struct i40e_arq_event_info *event) { struct ixl_vf *vf; void *msg; uint16_t vf_num, msg_size; uint32_t opcode; vf_num = le16toh(event->desc.retval) - pf->hw.func_caps.vf_base_id; opcode = le32toh(event->desc.cookie_high); if (vf_num >= pf->num_vfs) { device_printf(pf->dev, "Got msg from illegal VF: %d\n", vf_num); return; } vf = &pf->vfs[vf_num]; msg = event->msg_buf; msg_size = event->msg_len; I40E_VC_DEBUG(pf, ixl_vc_opcode_level(opcode), "Got msg %s(%d) from VF-%d of size %d\n", ixl_vc_opcode_str(opcode), opcode, vf_num, msg_size); switch (opcode) { case I40E_VIRTCHNL_OP_VERSION: ixl_vf_version_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_RESET_VF: ixl_vf_reset_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_GET_VF_RESOURCES: ixl_vf_get_resources_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES: ixl_vf_config_vsi_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP: ixl_vf_config_irq_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_ENABLE_QUEUES: ixl_vf_enable_queues_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_DISABLE_QUEUES: ixl_vf_disable_queues_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_ADD_ETHER_ADDRESS: ixl_vf_add_mac_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_DEL_ETHER_ADDRESS: ixl_vf_del_mac_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_ADD_VLAN: ixl_vf_add_vlan_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_DEL_VLAN: ixl_vf_del_vlan_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_CONFIG_PROMISCUOUS_MODE: ixl_vf_config_promisc_msg(pf, vf, msg, msg_size); break; case I40E_VIRTCHNL_OP_GET_STATS: ixl_vf_get_stats_msg(pf, vf, msg, msg_size); break; /* These two opcodes have been superseded by CONFIG_VSI_QUEUES. */ case I40E_VIRTCHNL_OP_CONFIG_TX_QUEUE: case I40E_VIRTCHNL_OP_CONFIG_RX_QUEUE: default: i40e_send_vf_nack(pf, vf, opcode, I40E_ERR_NOT_IMPLEMENTED); break; } } /* Handle any VFs that have reset themselves via a Function Level Reset(FLR). */ static void ixl_handle_vflr(void *arg, int pending) { struct ixl_pf *pf; struct i40e_hw *hw; uint16_t global_vf_num; uint32_t vflrstat_index, vflrstat_mask, vflrstat, icr0; int i; pf = arg; hw = &pf->hw; IXL_PF_LOCK(pf); for (i = 0; i < pf->num_vfs; i++) { global_vf_num = hw->func_caps.vf_base_id + i; vflrstat_index = IXL_GLGEN_VFLRSTAT_INDEX(global_vf_num); vflrstat_mask = IXL_GLGEN_VFLRSTAT_MASK(global_vf_num); vflrstat = rd32(hw, I40E_GLGEN_VFLRSTAT(vflrstat_index)); if (vflrstat & vflrstat_mask) { wr32(hw, I40E_GLGEN_VFLRSTAT(vflrstat_index), vflrstat_mask); ixl_reinit_vf(pf, &pf->vfs[i]); } } icr0 = rd32(hw, I40E_PFINT_ICR0_ENA); icr0 |= I40E_PFINT_ICR0_ENA_VFLR_MASK; wr32(hw, I40E_PFINT_ICR0_ENA, icr0); ixl_flush(hw); IXL_PF_UNLOCK(pf); } static int ixl_adminq_err_to_errno(enum i40e_admin_queue_err err) { switch (err) { case I40E_AQ_RC_EPERM: return (EPERM); case I40E_AQ_RC_ENOENT: return (ENOENT); case I40E_AQ_RC_ESRCH: return (ESRCH); case I40E_AQ_RC_EINTR: return (EINTR); case I40E_AQ_RC_EIO: return (EIO); case I40E_AQ_RC_ENXIO: return (ENXIO); case I40E_AQ_RC_E2BIG: return (E2BIG); case I40E_AQ_RC_EAGAIN: return (EAGAIN); case I40E_AQ_RC_ENOMEM: return (ENOMEM); case I40E_AQ_RC_EACCES: return (EACCES); case I40E_AQ_RC_EFAULT: return (EFAULT); case I40E_AQ_RC_EBUSY: return (EBUSY); case I40E_AQ_RC_EEXIST: return (EEXIST); case I40E_AQ_RC_EINVAL: return (EINVAL); case I40E_AQ_RC_ENOTTY: return (ENOTTY); case I40E_AQ_RC_ENOSPC: return (ENOSPC); case I40E_AQ_RC_ENOSYS: return (ENOSYS); case I40E_AQ_RC_ERANGE: return (ERANGE); case I40E_AQ_RC_EFLUSHED: return (EINVAL); /* No exact equivalent in errno.h */ case I40E_AQ_RC_BAD_ADDR: return (EFAULT); case I40E_AQ_RC_EMODE: return (EPERM); case I40E_AQ_RC_EFBIG: return (EFBIG); default: return (EINVAL); } } static int ixl_iov_init(device_t dev, uint16_t num_vfs, const nvlist_t *params) { struct ixl_pf *pf; struct i40e_hw *hw; struct ixl_vsi *pf_vsi; enum i40e_status_code ret; int i, error; pf = device_get_softc(dev); hw = &pf->hw; pf_vsi = &pf->vsi; IXL_PF_LOCK(pf); pf->vfs = malloc(sizeof(struct ixl_vf) * num_vfs, M_IXL, M_NOWAIT | M_ZERO); if (pf->vfs == NULL) { error = ENOMEM; goto fail; } for (i = 0; i < num_vfs; i++) sysctl_ctx_init(&pf->vfs[i].ctx); ret = i40e_aq_add_veb(hw, pf_vsi->uplink_seid, pf_vsi->seid, 1, FALSE, FALSE, &pf->veb_seid, NULL); if (ret != I40E_SUCCESS) { error = ixl_adminq_err_to_errno(hw->aq.asq_last_status); device_printf(dev, "add_veb failed; code=%d error=%d", ret, error); goto fail; } ixl_configure_msix(pf); ixl_enable_adminq(hw); pf->num_vfs = num_vfs; IXL_PF_UNLOCK(pf); return (0); fail: free(pf->vfs, M_IXL); pf->vfs = NULL; IXL_PF_UNLOCK(pf); return (error); } static void ixl_iov_uninit(device_t dev) { struct ixl_pf *pf; struct i40e_hw *hw; struct ixl_vsi *vsi; struct ifnet *ifp; struct ixl_vf *vfs; int i, num_vfs; pf = device_get_softc(dev); hw = &pf->hw; vsi = &pf->vsi; ifp = vsi->ifp; IXL_PF_LOCK(pf); for (i = 0; i < pf->num_vfs; i++) { if (pf->vfs[i].vsi.seid != 0) i40e_aq_delete_element(hw, pf->vfs[i].vsi.seid, NULL); } if (pf->veb_seid != 0) { i40e_aq_delete_element(hw, pf->veb_seid, NULL); pf->veb_seid = 0; } #if __FreeBSD_version > 1100022 if ((if_getdrvflags(ifp) & IFF_DRV_RUNNING) == 0) #else if ((ifp->if_drv_flags & IFF_DRV_RUNNING) == 0) #endif ixl_disable_intr(vsi); vfs = pf->vfs; num_vfs = pf->num_vfs; pf->vfs = NULL; pf->num_vfs = 0; IXL_PF_UNLOCK(pf); /* Do this after the unlock as sysctl_ctx_free might sleep. */ for (i = 0; i < num_vfs; i++) sysctl_ctx_free(&vfs[i].ctx); free(vfs, M_IXL); } static int ixl_add_vf(device_t dev, uint16_t vfnum, const nvlist_t *params) { char sysctl_name[QUEUE_NAME_LEN]; struct ixl_pf *pf; struct ixl_vf *vf; const void *mac; size_t size; int error; pf = device_get_softc(dev); vf = &pf->vfs[vfnum]; IXL_PF_LOCK(pf); vf->vf_num = vfnum; vf->vsi.back = pf; vf->vf_flags = VF_FLAG_ENABLED; SLIST_INIT(&vf->vsi.ftl); error = ixl_vf_setup_vsi(pf, vf); if (error != 0) goto out; if (nvlist_exists_binary(params, "mac-addr")) { mac = nvlist_get_binary(params, "mac-addr", &size); bcopy(mac, vf->mac, ETHER_ADDR_LEN); if (nvlist_get_bool(params, "allow-set-mac")) vf->vf_flags |= VF_FLAG_SET_MAC_CAP; } else /* * If the administrator has not specified a MAC address then * we must allow the VF to choose one. */ vf->vf_flags |= VF_FLAG_SET_MAC_CAP; if (nvlist_get_bool(params, "mac-anti-spoof")) vf->vf_flags |= VF_FLAG_MAC_ANTI_SPOOF; if (nvlist_get_bool(params, "allow-promisc")) vf->vf_flags |= VF_FLAG_PROMISC_CAP; vf->vf_flags |= VF_FLAG_VLAN_CAP; ixl_reset_vf(pf, vf); out: IXL_PF_UNLOCK(pf); if (error == 0) { snprintf(sysctl_name, sizeof(sysctl_name), "vf%d", vfnum); ixl_add_vsi_sysctls(pf, &vf->vsi, &vf->ctx, sysctl_name); } return (error); } #endif /* PCI_IOV */ Index: user/ngie/make_check/sys/dev/usb/controller/ehcireg.h =================================================================== --- user/ngie/make_check/sys/dev/usb/controller/ehcireg.h (revision 295077) +++ user/ngie/make_check/sys/dev/usb/controller/ehcireg.h (revision 295078) @@ -1,193 +1,193 @@ /* $FreeBSD$ */ /*- * Copyright (c) 2001 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (lennart@augustsson.net). * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #ifndef _EHCIREG_H_ #define _EHCIREG_H_ /* PCI config registers */ #define PCI_CBMEM 0x10 /* configuration base MEM */ #define PCI_INTERFACE_EHCI 0x20 #define PCI_USBREV 0x60 /* RO USB protocol revision */ #define PCI_USB_REV_MASK 0xff #define PCI_USB_REV_PRE_1_0 0x00 #define PCI_USB_REV_1_0 0x10 #define PCI_USB_REV_1_1 0x11 #define PCI_USB_REV_2_0 0x20 #define PCI_EHCI_FLADJ 0x61 /* RW Frame len adj, SOF=59488+6*fladj */ #define PCI_EHCI_PORTWAKECAP 0x62 /* RW Port wake caps (opt) */ /* EHCI Extended Capabilities */ #define EHCI_EC_LEGSUP 0x01 #define EHCI_EECP_NEXT(x) (((x) >> 8) & 0xff) #define EHCI_EECP_ID(x) ((x) & 0xff) /* Legacy support extended capability */ #define EHCI_LEGSUP_BIOS_SEM 0x02 #define EHCI_LEGSUP_OS_SEM 0x03 #define EHCI_LEGSUP_USBLEGCTLSTS 0x04 /* EHCI capability registers */ #define EHCI_CAPLEN_HCIVERSION 0x00 /* RO Capability register length * (least-significant byte) and * interface version number (two * most significant) */ #define EHCI_CAPLENGTH(x) ((x) & 0xff) #define EHCI_HCIVERSION(x) (((x) >> 16) & 0xffff) #define EHCI_HCSPARAMS 0x04 /* RO Structural parameters */ #define EHCI_HCS_DEBUGPORT(x) (((x) >> 20) & 0xf) #define EHCI_HCS_P_INDICATOR(x) ((x) & 0x10000) #define EHCI_HCS_N_CC(x) (((x) >> 12) & 0xf) /* # of companion ctlrs */ #define EHCI_HCS_N_PCC(x) (((x) >> 8) & 0xf) /* # of ports per comp. */ #define EHCI_HCS_PPC(x) ((x) & 0x10) /* port power control */ #define EHCI_HCS_N_PORTS(x) ((x) & 0xf) /* # of ports */ #define EHCI_HCCPARAMS 0x08 /* RO Capability parameters */ #define EHCI_HCC_EECP(x) (((x) >> 8) & 0xff) /* extended ports caps */ #define EHCI_HCC_IST(x) (((x) >> 4) & 0xf) /* isoc sched threshold */ #define EHCI_HCC_ASPC(x) ((x) & 0x4) /* async sched park cap */ #define EHCI_HCC_PFLF(x) ((x) & 0x2) /* prog frame list flag */ #define EHCI_HCC_64BIT(x) ((x) & 0x1) /* 64 bit address cap */ #define EHCI_HCSP_PORTROUTE 0x0c /* RO Companion port route description */ /* EHCI operational registers. Offset given by EHCI_CAPLENGTH register */ #define EHCI_USBCMD 0x00 /* RO, RW, WO Command register */ #define EHCI_CMD_ITC_M 0x00ff0000 /* RW interrupt threshold ctrl */ #define EHCI_CMD_ITC_1 0x00010000 #define EHCI_CMD_ITC_2 0x00020000 #define EHCI_CMD_ITC_4 0x00040000 #define EHCI_CMD_ITC_8 0x00080000 #define EHCI_CMD_ITC_16 0x00100000 #define EHCI_CMD_ITC_32 0x00200000 #define EHCI_CMD_ITC_64 0x00400000 #define EHCI_CMD_ASPME 0x00000800 /* RW/RO async park enable */ #define EHCI_CMD_ASPMC 0x00000300 /* RW/RO async park count */ #define EHCI_CMD_LHCR 0x00000080 /* RW light host ctrl reset */ #define EHCI_CMD_IAAD 0x00000040 /* RW intr on async adv door * bell */ #define EHCI_CMD_ASE 0x00000020 /* RW async sched enable */ #define EHCI_CMD_PSE 0x00000010 /* RW periodic sched enable */ #define EHCI_CMD_FLS_M 0x0000000c /* RW/RO frame list size */ #define EHCI_CMD_FLS(x) (((x) >> 2) & 3) /* RW/RO frame list size */ #define EHCI_CMD_HCRESET 0x00000002 /* RW reset */ #define EHCI_CMD_RS 0x00000001 /* RW run/stop */ #define EHCI_USBSTS 0x04 /* RO, RW, RWC Status register */ #define EHCI_STS_ASS 0x00008000 /* RO async sched status */ #define EHCI_STS_PSS 0x00004000 /* RO periodic sched status */ #define EHCI_STS_REC 0x00002000 /* RO reclamation */ #define EHCI_STS_HCH 0x00001000 /* RO host controller halted */ #define EHCI_STS_IAA 0x00000020 /* RWC interrupt on async adv */ #define EHCI_STS_HSE 0x00000010 /* RWC host system error */ #define EHCI_STS_FLR 0x00000008 /* RWC frame list rollover */ #define EHCI_STS_PCD 0x00000004 /* RWC port change detect */ #define EHCI_STS_ERRINT 0x00000002 /* RWC error interrupt */ #define EHCI_STS_INT 0x00000001 /* RWC interrupt */ #define EHCI_STS_INTRS(x) ((x) & 0x3f) /* * NOTE: the doorbell interrupt is enabled, but the doorbell is never * used! SiS chipsets require this. */ #define EHCI_NORMAL_INTRS (EHCI_STS_IAA | EHCI_STS_HSE | \ EHCI_STS_PCD | EHCI_STS_ERRINT | EHCI_STS_INT) #define EHCI_USBINTR 0x08 /* RW Interrupt register */ #define EHCI_INTR_IAAE 0x00000020 /* interrupt on async advance * ena */ #define EHCI_INTR_HSEE 0x00000010 /* host system error ena */ #define EHCI_INTR_FLRE 0x00000008 /* frame list rollover ena */ #define EHCI_INTR_PCIE 0x00000004 /* port change ena */ #define EHCI_INTR_UEIE 0x00000002 /* USB error intr ena */ #define EHCI_INTR_UIE 0x00000001 /* USB intr ena */ #define EHCI_FRINDEX 0x0c /* RW Frame Index register */ #define EHCI_CTRLDSSEGMENT 0x10 /* RW Control Data Structure Segment */ #define EHCI_PERIODICLISTBASE 0x14 /* RW Periodic List Base */ #define EHCI_ASYNCLISTADDR 0x18 /* RW Async List Base */ #define EHCI_CONFIGFLAG 0x40 /* RW Configure Flag register */ #define EHCI_CONF_CF 0x00000001 /* RW configure flag */ #define EHCI_PORTSC(n) (0x40+(4*(n))) /* RO, RW, RWC Port Status reg */ #define EHCI_PS_WKOC_E 0x00400000 /* RW wake on over current ena */ #define EHCI_PS_WKDSCNNT_E 0x00200000 /* RW wake on disconnect ena */ #define EHCI_PS_WKCNNT_E 0x00100000 /* RW wake on connect ena */ #define EHCI_PS_PTC 0x000f0000 /* RW port test control */ #define EHCI_PS_PIC 0x0000c000 /* RW port indicator control */ #define EHCI_PS_PO 0x00002000 /* RW port owner */ #define EHCI_PS_PP 0x00001000 /* RW,RO port power */ #define EHCI_PS_LS 0x00000c00 /* RO line status */ #define EHCI_PS_IS_LOWSPEED(x) (((x) & EHCI_PS_LS) == 0x00000400) #define EHCI_PS_PR 0x00000100 /* RW port reset */ #define EHCI_PS_SUSP 0x00000080 /* RW suspend */ #define EHCI_PS_FPR 0x00000040 /* RW force port resume */ #define EHCI_PS_OCC 0x00000020 /* RWC over current change */ #define EHCI_PS_OCA 0x00000010 /* RO over current active */ #define EHCI_PS_PEC 0x00000008 /* RWC port enable change */ #define EHCI_PS_PE 0x00000004 /* RW port enable */ #define EHCI_PS_CSC 0x00000002 /* RWC connect status change */ #define EHCI_PS_CS 0x00000001 /* RO connect status */ #define EHCI_PS_CLEAR (EHCI_PS_OCC | EHCI_PS_PEC | EHCI_PS_CSC) #define EHCI_PORT_RESET_COMPLETE 2 /* ms */ /* * Registers not covered by EHCI specification * * * EHCI_USBMODE register offset is different for cores with LPM support, * bits are equal */ #define EHCI_USBMODE_NOLPM 0x68 /* RW USB Device mode reg (no LPM) */ -#define EHCI_USBMODE_LPM 0xA8 /* RW USB Device mode reg (LPM) */ +#define EHCI_USBMODE_LPM 0xC8 /* RW USB Device mode reg (LPM) */ #define EHCI_UM_CM 0x00000003 /* R/WO Controller Mode */ #define EHCI_UM_CM_IDLE 0x0 /* Idle */ #define EHCI_UM_CM_HOST 0x3 /* Host Controller */ #define EHCI_UM_ES 0x00000004 /* R/WO Endian Select */ #define EHCI_UM_ES_LE 0x0 /* Little-endian byte alignment */ #define EHCI_UM_ES_BE 0x4 /* Big-endian byte alignment */ #define EHCI_UM_SDIS 0x00000010 /* R/WO Stream Disable Mode */ /* * Actual port speed bits depends on EHCI_HOSTC(n) registers presence, * speed encoding is equal */ #define EHCI_HOSTC(n) (0x80+(4*(n))) /* RO, RW Host mode control reg */ #define EHCI_HOSTC_PSPD_SHIFT 25 #define EHCI_HOSTC_PSPD_MASK 0x3 #define EHCI_PORTSC_PSPD_SHIFT 26 #define EHCI_PORTSC_PSPD_MASK 0x3 #define EHCI_PORT_SPEED_FULL 0 #define EHCI_PORT_SPEED_LOW 1 #define EHCI_PORT_SPEED_HIGH 2 #endif /* _EHCIREG_H_ */ Index: user/ngie/make_check/sys/netinet/sctp_input.c =================================================================== --- user/ngie/make_check/sys/netinet/sctp_input.c (revision 295077) +++ user/ngie/make_check/sys/netinet/sctp_input.c (revision 295078) @@ -1,6196 +1,6223 @@ /*- * Copyright (c) 2001-2008, by Cisco Systems, Inc. All rights reserved. * Copyright (c) 2008-2012, by Randall Stewart. All rights reserved. * Copyright (c) 2008-2012, by Michael Tuexen. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * a) Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * b) Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the distribution. * * c) Neither the name of Cisco Systems, Inc. nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF * THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #if defined(INET) || defined(INET6) #include #endif #include static void sctp_stop_all_cookie_timers(struct sctp_tcb *stcb) { struct sctp_nets *net; /* * This now not only stops all cookie timers it also stops any INIT * timers as well. This will make sure that the timers are stopped * in all collision cases. */ SCTP_TCB_LOCK_ASSERT(stcb); TAILQ_FOREACH(net, &stcb->asoc.nets, sctp_next) { if (net->rxt_timer.type == SCTP_TIMER_TYPE_COOKIE) { sctp_timer_stop(SCTP_TIMER_TYPE_COOKIE, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_1); } else if (net->rxt_timer.type == SCTP_TIMER_TYPE_INIT) { sctp_timer_stop(SCTP_TIMER_TYPE_INIT, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_2); } } } /* INIT handler */ static void sctp_handle_init(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_init_chunk *cp, struct sctp_inpcb *inp, - struct sctp_tcb *stcb, int *abort_no_unlock, + struct sctp_tcb *stcb, struct sctp_nets *net, int *abort_no_unlock, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_init *init; struct mbuf *op_err; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_init: handling INIT tcb:%p\n", (void *)stcb); if (stcb == NULL) { SCTP_INP_RLOCK(inp); } /* validate length */ if (ntohs(cp->ch.chunk_length) < sizeof(struct sctp_init_chunk)) { op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } /* validate parameters */ init = &cp->init; if (init->initiate_tag == 0) { /* protocol error... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (ntohl(init->a_rwnd) < SCTP_MIN_RWND) { /* invalid parameter... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (init->num_inbound_streams == 0) { /* protocol error... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (init->num_outbound_streams == 0) { /* protocol error... send abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } if (sctp_validate_init_auth_params(m, offset + sizeof(*cp), offset + ntohs(cp->ch.chunk_length))) { /* auth parameter(s) error... send abort */ op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "Problem with AUTH parameters"); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); if (stcb) *abort_no_unlock = 1; goto outnow; } /* * We are only accepting if we have a socket with positive * so_qlimit. */ if ((stcb == NULL) && ((inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) || (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_ALLGONE) || (inp->sctp_socket == NULL) || (inp->sctp_socket->so_qlimit == 0))) { /* * FIX ME ?? What about TCP model and we have a * match/restart case? Actually no fix is needed. the lookup * will always find the existing assoc so stcb would not be * NULL. It may be questionable to do this since we COULD * just send back the INIT-ACK and hope that the app did * accept()'s by the time the COOKIE was sent. But there is * a price to pay for COOKIE generation and I don't want to * pay it on the chance that the app will actually do some * accepts(). The App just looses and should NOT be in this * state :-) */ if (SCTP_BASE_SYSCTL(sctp_blackhole) == 0) { op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "No listener"); sctp_send_abort(m, iphlen, src, dst, sh, 0, op_err, mflowtype, mflowid, inp->fibnum, vrf_id, port); } goto outnow; } if ((stcb != NULL) && (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT)) { SCTPDBG(SCTP_DEBUG_INPUT3, "sctp_handle_init: sending SHUTDOWN-ACK\n"); sctp_send_shutdown_ack(stcb, NULL); sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_CONTROL_PROC, SCTP_SO_NOT_LOCKED); } else { SCTPDBG(SCTP_DEBUG_INPUT3, "sctp_handle_init: sending INIT-ACK\n"); - sctp_send_initiate_ack(inp, stcb, m, iphlen, offset, src, dst, - sh, cp, + sctp_send_initiate_ack(inp, stcb, net, m, iphlen, offset, + src, dst, sh, cp, mflowtype, mflowid, vrf_id, port, ((stcb == NULL) ? SCTP_HOLDS_LOCK : SCTP_NOT_LOCKED)); } outnow: if (stcb == NULL) { SCTP_INP_RUNLOCK(inp); } } /* * process peer "INIT/INIT-ACK" chunk returns value < 0 on error */ int sctp_is_there_unsent_data(struct sctp_tcb *stcb, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { int unsent_data = 0; unsigned int i; struct sctp_stream_queue_pending *sp; struct sctp_association *asoc; /* * This function returns the number of streams that have true unsent * data on them. Note that as it looks through it will clean up any * places that have old data that has been sent but left at top of * stream queue. */ asoc = &stcb->asoc; SCTP_TCB_SEND_LOCK(stcb); if (!stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { /* Check to see if some data queued */ for (i = 0; i < stcb->asoc.streamoutcnt; i++) { /* sa_ignore FREED_MEMORY */ sp = TAILQ_FIRST(&stcb->asoc.strmout[i].outqueue); if (sp == NULL) { continue; } if ((sp->msg_is_complete) && (sp->length == 0) && (sp->sender_all_done)) { /* * We are doing differed cleanup. Last time * through when we took all the data the * sender_all_done was not set. */ if (sp->put_last_out == 0) { SCTP_PRINTF("Gak, put out entire msg with NO end!-1\n"); SCTP_PRINTF("sender_done:%d len:%d msg_comp:%d put_last_out:%d\n", sp->sender_all_done, sp->length, sp->msg_is_complete, sp->put_last_out); } atomic_subtract_int(&stcb->asoc.stream_queue_cnt, 1); TAILQ_REMOVE(&stcb->asoc.strmout[i].outqueue, sp, next); if (sp->net) { sctp_free_remote_addr(sp->net); sp->net = NULL; } if (sp->data) { sctp_m_freem(sp->data); sp->data = NULL; } sctp_free_a_strmoq(stcb, sp, so_locked); } else { unsent_data++; break; } } } SCTP_TCB_SEND_UNLOCK(stcb); return (unsent_data); } static int sctp_process_init(struct sctp_init_chunk *cp, struct sctp_tcb *stcb) { struct sctp_init *init; struct sctp_association *asoc; struct sctp_nets *lnet; unsigned int i; init = &cp->init; asoc = &stcb->asoc; /* save off parameters */ asoc->peer_vtag = ntohl(init->initiate_tag); asoc->peers_rwnd = ntohl(init->a_rwnd); /* init tsn's */ asoc->highest_tsn_inside_map = asoc->asconf_seq_in = ntohl(init->initial_tsn) - 1; if (!TAILQ_EMPTY(&asoc->nets)) { /* update any ssthresh's that may have a default */ TAILQ_FOREACH(lnet, &asoc->nets, sctp_next) { lnet->ssthresh = asoc->peers_rwnd; if (SCTP_BASE_SYSCTL(sctp_logging_level) & (SCTP_CWND_MONITOR_ENABLE | SCTP_CWND_LOGGING_ENABLE)) { sctp_log_cwnd(stcb, lnet, 0, SCTP_CWND_INITIALIZATION); } } } SCTP_TCB_SEND_LOCK(stcb); if (asoc->pre_open_streams > ntohs(init->num_inbound_streams)) { unsigned int newcnt; struct sctp_stream_out *outs; struct sctp_stream_queue_pending *sp, *nsp; struct sctp_tmit_chunk *chk, *nchk; /* abandon the upper streams */ newcnt = ntohs(init->num_inbound_streams); TAILQ_FOREACH_SAFE(chk, &asoc->send_queue, sctp_next, nchk) { if (chk->rec.data.stream_number >= newcnt) { TAILQ_REMOVE(&asoc->send_queue, chk, sctp_next); asoc->send_queue_cnt--; if (asoc->strmout[chk->rec.data.stream_number].chunks_on_queues > 0) { asoc->strmout[chk->rec.data.stream_number].chunks_on_queues--; #ifdef INVARIANTS } else { panic("No chunks on the queues for sid %u.", chk->rec.data.stream_number); #endif } if (chk->data != NULL) { sctp_free_bufspace(stcb, asoc, chk, 1); sctp_ulp_notify(SCTP_NOTIFY_UNSENT_DG_FAIL, stcb, 0, chk, SCTP_SO_NOT_LOCKED); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } } sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); /* sa_ignore FREED_MEMORY */ } } if (asoc->strmout) { for (i = newcnt; i < asoc->pre_open_streams; i++) { outs = &asoc->strmout[i]; TAILQ_FOREACH_SAFE(sp, &outs->outqueue, next, nsp) { TAILQ_REMOVE(&outs->outqueue, sp, next); asoc->stream_queue_cnt--; sctp_ulp_notify(SCTP_NOTIFY_SPECIAL_SP_FAIL, stcb, 0, sp, SCTP_SO_NOT_LOCKED); if (sp->data) { sctp_m_freem(sp->data); sp->data = NULL; } if (sp->net) { sctp_free_remote_addr(sp->net); sp->net = NULL; } /* Free the chunk */ sctp_free_a_strmoq(stcb, sp, SCTP_SO_NOT_LOCKED); /* sa_ignore FREED_MEMORY */ } outs->state = SCTP_STREAM_CLOSED; } } /* cut back the count */ asoc->pre_open_streams = newcnt; } SCTP_TCB_SEND_UNLOCK(stcb); asoc->streamoutcnt = asoc->pre_open_streams; for (i = 0; i < asoc->streamoutcnt; i++) { asoc->strmout[i].state = SCTP_STREAM_OPEN; } /* EY - nr_sack: initialize highest tsn in nr_mapping_array */ asoc->highest_tsn_inside_nr_map = asoc->highest_tsn_inside_map; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MAP_LOGGING_ENABLE) { sctp_log_map(0, 5, asoc->highest_tsn_inside_map, SCTP_MAP_SLIDE_RESULT); } /* This is the next one we expect */ asoc->str_reset_seq_in = asoc->asconf_seq_in + 1; asoc->mapping_array_base_tsn = ntohl(init->initial_tsn); asoc->tsn_last_delivered = asoc->cumulative_tsn = asoc->asconf_seq_in; asoc->advanced_peer_ack_point = asoc->last_acked_seq; /* open the requested streams */ if (asoc->strmin != NULL) { /* Free the old ones */ struct sctp_queued_to_read *ctl, *nctl; for (i = 0; i < asoc->streamincnt; i++) { TAILQ_FOREACH_SAFE(ctl, &asoc->strmin[i].inqueue, next, nctl) { TAILQ_REMOVE(&asoc->strmin[i].inqueue, ctl, next); sctp_free_remote_addr(ctl->whoFrom); ctl->whoFrom = NULL; sctp_m_freem(ctl->data); ctl->data = NULL; sctp_free_a_readq(stcb, ctl); } } SCTP_FREE(asoc->strmin, SCTP_M_STRMI); } if (asoc->max_inbound_streams > ntohs(init->num_outbound_streams)) { asoc->streamincnt = ntohs(init->num_outbound_streams); } else { asoc->streamincnt = asoc->max_inbound_streams; } SCTP_MALLOC(asoc->strmin, struct sctp_stream_in *, asoc->streamincnt * sizeof(struct sctp_stream_in), SCTP_M_STRMI); if (asoc->strmin == NULL) { /* we didn't get memory for the streams! */ SCTPDBG(SCTP_DEBUG_INPUT2, "process_init: couldn't get memory for the streams!\n"); return (-1); } for (i = 0; i < asoc->streamincnt; i++) { asoc->strmin[i].stream_no = i; asoc->strmin[i].last_sequence_delivered = 0xffff; TAILQ_INIT(&asoc->strmin[i].inqueue); asoc->strmin[i].delivery_started = 0; } /* * load_address_from_init will put the addresses into the * association when the COOKIE is processed or the INIT-ACK is * processed. Both types of COOKIE's existing and new call this * routine. It will remove addresses that are no longer in the * association (for the restarting case where addresses are * removed). Up front when the INIT arrives we will discard it if it * is a restart and new addresses have been added. */ /* sa_ignore MEMLEAK */ return (0); } /* * INIT-ACK message processing/consumption returns value < 0 on error */ static int sctp_process_init_ack(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_init_ack_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, int *abort_no_unlock, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id) { struct sctp_association *asoc; struct mbuf *op_err; int retval, abort_flag; uint32_t initack_limit; int nat_friendly = 0; /* First verify that we have no illegal param's */ abort_flag = 0; op_err = sctp_arethere_unrecognized_parameters(m, (offset + sizeof(struct sctp_init_chunk)), &abort_flag, (struct sctp_chunkhdr *)cp, &nat_friendly); if (abort_flag) { /* Send an abort and notify peer */ sctp_abort_an_association(stcb->sctp_ep, stcb, op_err, SCTP_SO_NOT_LOCKED); *abort_no_unlock = 1; return (-1); } asoc = &stcb->asoc; asoc->peer_supports_nat = (uint8_t) nat_friendly; /* process the peer's parameters in the INIT-ACK */ retval = sctp_process_init((struct sctp_init_chunk *)cp, stcb); if (retval < 0) { return (retval); } initack_limit = offset + ntohs(cp->ch.chunk_length); /* load all addresses */ if ((retval = sctp_load_addresses_from_init(stcb, m, (offset + sizeof(struct sctp_init_chunk)), initack_limit, src, dst, NULL))) { op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "Problem with address parameters"); SCTPDBG(SCTP_DEBUG_INPUT1, "Load addresses from INIT causes an abort %d\n", retval); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } /* if the peer doesn't support asconf, flush the asconf queue */ if (asoc->asconf_supported == 0) { struct sctp_asconf_addr *param, *nparam; TAILQ_FOREACH_SAFE(param, &asoc->asconf_queue, next, nparam) { TAILQ_REMOVE(&asoc->asconf_queue, param, next); SCTP_FREE(param, SCTP_M_ASC_ADDR); } } stcb->asoc.peer_hmac_id = sctp_negotiate_hmacid(stcb->asoc.peer_hmacs, stcb->asoc.local_hmacs); if (op_err) { sctp_queue_op_err(stcb, op_err); /* queuing will steal away the mbuf chain to the out queue */ op_err = NULL; } /* extract the cookie and queue it to "echo" it back... */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; net->error_count = 0; /* * Cancel the INIT timer, We do this first before queueing the * cookie. We always cancel at the primary to assue that we are * canceling the timer started by the INIT which always goes to the * primary. */ sctp_timer_stop(SCTP_TIMER_TYPE_INIT, stcb->sctp_ep, stcb, asoc->primary_destination, SCTP_FROM_SCTP_INPUT + SCTP_LOC_3); /* calculate the RTO */ net->RTO = sctp_calculate_rto(stcb, asoc, net, &asoc->time_entered, sctp_align_safe_nocopy, SCTP_RTT_FROM_NON_DATA); retval = sctp_send_cookie_echo(m, offset, stcb, net); if (retval < 0) { /* * No cookie, we probably should send a op error. But in any * case if there is no cookie in the INIT-ACK, we can * abandon the peer, its broke. */ if (retval == -3) { uint16_t len; len = (uint16_t) (sizeof(struct sctp_error_missing_param) + sizeof(uint16_t)); /* We abort with an error of missing mandatory param */ op_err = sctp_get_mbuf_for_msg(len, 0, M_NOWAIT, 1, MT_DATA); if (op_err != NULL) { struct sctp_error_missing_param *cause; SCTP_BUF_LEN(op_err) = len; cause = mtod(op_err, struct sctp_error_missing_param *); /* Subtract the reserved param */ cause->cause.code = htons(SCTP_CAUSE_MISSING_PARAM); cause->cause.length = htons(len); cause->num_missing_params = htonl(1); cause->type[0] = htons(SCTP_STATE_COOKIE); } sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; } return (retval); } return (0); } static void sctp_handle_heartbeat_ack(struct sctp_heartbeat_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net) { union sctp_sockstore store; struct sctp_nets *r_net, *f_net; struct timeval tv; int req_prim = 0; uint16_t old_error_counter; if (ntohs(cp->ch.chunk_length) != sizeof(struct sctp_heartbeat_chunk)) { /* Invalid length */ return; } memset(&store, 0, sizeof(store)); switch (cp->heartbeat.hb_info.addr_family) { #ifdef INET case AF_INET: if (cp->heartbeat.hb_info.addr_len == sizeof(struct sockaddr_in)) { store.sin.sin_family = cp->heartbeat.hb_info.addr_family; store.sin.sin_len = cp->heartbeat.hb_info.addr_len; store.sin.sin_port = stcb->rport; memcpy(&store.sin.sin_addr, cp->heartbeat.hb_info.address, sizeof(store.sin.sin_addr)); } else { return; } break; #endif #ifdef INET6 case AF_INET6: if (cp->heartbeat.hb_info.addr_len == sizeof(struct sockaddr_in6)) { store.sin6.sin6_family = cp->heartbeat.hb_info.addr_family; store.sin6.sin6_len = cp->heartbeat.hb_info.addr_len; store.sin6.sin6_port = stcb->rport; memcpy(&store.sin6.sin6_addr, cp->heartbeat.hb_info.address, sizeof(struct in6_addr)); } else { return; } break; #endif default: return; } r_net = sctp_findnet(stcb, &store.sa); if (r_net == NULL) { SCTPDBG(SCTP_DEBUG_INPUT1, "Huh? I can't find the address I sent it to, discard\n"); return; } if ((r_net && (r_net->dest_state & SCTP_ADDR_UNCONFIRMED)) && (r_net->heartbeat_random1 == cp->heartbeat.hb_info.random_value1) && (r_net->heartbeat_random2 == cp->heartbeat.hb_info.random_value2)) { /* * If the its a HB and it's random value is correct when can * confirm the destination. */ r_net->dest_state &= ~SCTP_ADDR_UNCONFIRMED; if (r_net->dest_state & SCTP_ADDR_REQ_PRIMARY) { stcb->asoc.primary_destination = r_net; r_net->dest_state &= ~SCTP_ADDR_REQ_PRIMARY; f_net = TAILQ_FIRST(&stcb->asoc.nets); if (f_net != r_net) { /* * first one on the list is NOT the primary * sctp_cmpaddr() is much more efficent if * the primary is the first on the list, * make it so. */ TAILQ_REMOVE(&stcb->asoc.nets, r_net, sctp_next); TAILQ_INSERT_HEAD(&stcb->asoc.nets, r_net, sctp_next); } req_prim = 1; } sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, stcb, 0, (void *)r_net, SCTP_SO_NOT_LOCKED); sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_4); sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net); } old_error_counter = r_net->error_count; r_net->error_count = 0; r_net->hb_responded = 1; tv.tv_sec = cp->heartbeat.hb_info.time_value_1; tv.tv_usec = cp->heartbeat.hb_info.time_value_2; /* Now lets do a RTO with this */ r_net->RTO = sctp_calculate_rto(stcb, &stcb->asoc, r_net, &tv, sctp_align_safe_nocopy, SCTP_RTT_FROM_NON_DATA); if (!(r_net->dest_state & SCTP_ADDR_REACHABLE)) { r_net->dest_state |= SCTP_ADDR_REACHABLE; sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_UP, stcb, 0, (void *)r_net, SCTP_SO_NOT_LOCKED); } if (r_net->dest_state & SCTP_ADDR_PF) { r_net->dest_state &= ~SCTP_ADDR_PF; stcb->asoc.cc_functions.sctp_cwnd_update_exit_pf(stcb, net); } if (old_error_counter > 0) { sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_5); sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, r_net); } if (r_net == stcb->asoc.primary_destination) { if (stcb->asoc.alternate) { /* release the alternate, primary is good */ sctp_free_remote_addr(stcb->asoc.alternate); stcb->asoc.alternate = NULL; } } /* Mobility adaptation */ if (req_prim) { if ((sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_BASE) || sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_FASTHANDOFF)) && sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_PRIM_DELETED)) { sctp_timer_stop(SCTP_TIMER_TYPE_PRIM_DELETED, stcb->sctp_ep, stcb, NULL, SCTP_FROM_SCTP_INPUT + SCTP_LOC_6); if (sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_FASTHANDOFF)) { sctp_assoc_immediate_retrans(stcb, stcb->asoc.primary_destination); } if (sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_BASE)) { sctp_move_chunks_from_net(stcb, stcb->asoc.deleted_primary); } sctp_delete_prim_timer(stcb->sctp_ep, stcb, stcb->asoc.deleted_primary); } } } static int sctp_handle_nat_colliding_state(struct sctp_tcb *stcb) { /* * return 0 means we want you to proceed with the abort non-zero * means no abort processing */ struct sctpasochead *head; if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_COOKIE_WAIT) { /* generate a new vtag and send init */ LIST_REMOVE(stcb, sctp_asocs); stcb->asoc.my_vtag = sctp_select_a_tag(stcb->sctp_ep, stcb->sctp_ep->sctp_lport, stcb->rport, 1); head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(stcb->asoc.my_vtag, SCTP_BASE_INFO(hashasocmark))]; /* * put it in the bucket in the vtag hash of assoc's for the * system */ LIST_INSERT_HEAD(head, stcb, sctp_asocs); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); return (1); } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_COOKIE_ECHOED) { /* * treat like a case where the cookie expired i.e.: - dump * current cookie. - generate a new vtag. - resend init. */ /* generate a new vtag and send init */ LIST_REMOVE(stcb, sctp_asocs); stcb->asoc.state &= ~SCTP_STATE_COOKIE_ECHOED; stcb->asoc.state |= SCTP_STATE_COOKIE_WAIT; sctp_stop_all_cookie_timers(stcb); sctp_toss_old_cookies(stcb, &stcb->asoc); stcb->asoc.my_vtag = sctp_select_a_tag(stcb->sctp_ep, stcb->sctp_ep->sctp_lport, stcb->rport, 1); head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(stcb->asoc.my_vtag, SCTP_BASE_INFO(hashasocmark))]; /* * put it in the bucket in the vtag hash of assoc's for the * system */ LIST_INSERT_HEAD(head, stcb, sctp_asocs); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); return (1); } return (0); } static int sctp_handle_nat_missing_state(struct sctp_tcb *stcb, struct sctp_nets *net) { /* * return 0 means we want you to proceed with the abort non-zero * means no abort processing */ if (stcb->asoc.auth_supported == 0) { SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_nat_missing_state: Peer does not support AUTH, cannot send an asconf\n"); return (0); } sctp_asconf_send_nat_state_update(stcb, net); return (1); } static void sctp_handle_abort(struct sctp_abort_chunk *abort, struct sctp_tcb *stcb, struct sctp_nets *net) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif uint16_t len; uint16_t error; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_abort: handling ABORT\n"); if (stcb == NULL) return; len = ntohs(abort->ch.chunk_length); if (len > sizeof(struct sctp_chunkhdr)) { /* * Need to check the cause codes for our two magic nat * aborts which don't kill the assoc necessarily. */ struct sctp_gen_error_cause *cause; cause = (struct sctp_gen_error_cause *)(abort + 1); error = ntohs(cause->code); if (error == SCTP_CAUSE_NAT_COLLIDING_STATE) { SCTPDBG(SCTP_DEBUG_INPUT2, "Received Colliding state abort flags:%x\n", abort->ch.chunk_flags); if (sctp_handle_nat_colliding_state(stcb)) { return; } } else if (error == SCTP_CAUSE_NAT_MISSING_STATE) { SCTPDBG(SCTP_DEBUG_INPUT2, "Received missing state abort flags:%x\n", abort->ch.chunk_flags); if (sctp_handle_nat_missing_state(stcb, net)) { return; } } } else { error = 0; } /* stop any receive timers */ sctp_timer_stop(SCTP_TIMER_TYPE_RECV, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_7); /* notify user of the abort and clean up... */ sctp_abort_notification(stcb, 1, error, abort, SCTP_SO_NOT_LOCKED); /* free the tcb */ SCTP_STAT_INCR_COUNTER32(sctps_aborted); if ((SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_OPEN) || (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_RECEIVED)) { SCTP_STAT_DECR_GAUGE32(sctps_currestab); } #ifdef SCTP_ASOCLOG_OF_TSNS sctp_print_out_track_log(stcb); #endif #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif stcb->asoc.state |= SCTP_STATE_WAS_ABORTED; (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_8); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_abort: finished\n"); } static void sctp_start_net_timers(struct sctp_tcb *stcb) { uint32_t cnt_hb_sent; struct sctp_nets *net; cnt_hb_sent = 0; TAILQ_FOREACH(net, &stcb->asoc.nets, sctp_next) { /* * For each network start: 1) A pmtu timer. 2) A HB timer 3) * If the dest in unconfirmed send a hb as well if under * max_hb_burst have been sent. */ sctp_timer_start(SCTP_TIMER_TYPE_PATHMTURAISE, stcb->sctp_ep, stcb, net); sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, net); if ((net->dest_state & SCTP_ADDR_UNCONFIRMED) && (cnt_hb_sent < SCTP_BASE_SYSCTL(sctp_hb_maxburst))) { sctp_send_hb(stcb, net, SCTP_SO_NOT_LOCKED); cnt_hb_sent++; } } if (cnt_hb_sent) { sctp_chunk_output(stcb->sctp_ep, stcb, SCTP_OUTPUT_FROM_COOKIE_ACK, SCTP_SO_NOT_LOCKED); } } static void sctp_handle_shutdown(struct sctp_shutdown_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, int *abort_flag) { struct sctp_association *asoc; int some_on_streamwheel; int old_state; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown: handling SHUTDOWN\n"); if (stcb == NULL) return; asoc = &stcb->asoc; if ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED)) { return; } if (ntohs(cp->ch.chunk_length) != sizeof(struct sctp_shutdown_chunk)) { /* Shutdown NOT the expected size */ return; } old_state = SCTP_GET_STATE(asoc); sctp_update_acked(stcb, cp, abort_flag); if (*abort_flag) { return; } if (asoc->control_pdapi) { /* * With a normal shutdown we assume the end of last record. */ SCTP_INP_READ_LOCK(stcb->sctp_ep); asoc->control_pdapi->end_added = 1; asoc->control_pdapi->pdapi_aborted = 1; asoc->control_pdapi = NULL; SCTP_INP_READ_UNLOCK(stcb->sctp_ep); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { /* assoc was freed while we were unlocked */ SCTP_SOCKET_UNLOCK(so, 1); return; } #endif sctp_sorwakeup(stcb->sctp_ep, stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* goto SHUTDOWN_RECEIVED state to block new requests */ if (stcb->sctp_socket) { if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT)) { SCTP_SET_STATE(asoc, SCTP_STATE_SHUTDOWN_RECEIVED); SCTP_CLEAR_SUBSTATE(asoc, SCTP_STATE_SHUTDOWN_PENDING); /* * notify upper layer that peer has initiated a * shutdown */ sctp_ulp_notify(SCTP_NOTIFY_PEER_SHUTDOWN, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); /* reset time */ (void)SCTP_GETTIME_TIMEVAL(&asoc->time_entered); } } if (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_SENT) { /* * stop the shutdown timer, since we WILL move to * SHUTDOWN-ACK-SENT. */ sctp_timer_stop(SCTP_TIMER_TYPE_SHUTDOWN, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_9); } /* Now is there unsent data on a stream somewhere? */ some_on_streamwheel = sctp_is_there_unsent_data(stcb, SCTP_SO_NOT_LOCKED); if (!TAILQ_EMPTY(&asoc->send_queue) || !TAILQ_EMPTY(&asoc->sent_queue) || some_on_streamwheel) { /* By returning we will push more data out */ return; } else { /* no outstanding data to send, so move on... */ /* send SHUTDOWN-ACK */ /* move to SHUTDOWN-ACK-SENT state */ if ((SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) || (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_RECEIVED)) { SCTP_STAT_DECR_GAUGE32(sctps_currestab); } SCTP_CLEAR_SUBSTATE(asoc, SCTP_STATE_SHUTDOWN_PENDING); if (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT) { SCTP_SET_STATE(asoc, SCTP_STATE_SHUTDOWN_ACK_SENT); sctp_stop_timers_for_shutdown(stcb); sctp_send_shutdown_ack(stcb, net); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNACK, stcb->sctp_ep, stcb, net); } else if (old_state == SCTP_STATE_SHUTDOWN_ACK_SENT) { sctp_send_shutdown_ack(stcb, net); } } } static void sctp_handle_shutdown_ack(struct sctp_shutdown_ack_chunk *cp SCTP_UNUSED, struct sctp_tcb *stcb, struct sctp_nets *net) { struct sctp_association *asoc; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; so = SCTP_INP_SO(stcb->sctp_ep); #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_ack: handling SHUTDOWN ACK\n"); if (stcb == NULL) return; asoc = &stcb->asoc; /* process according to association state */ if ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED)) { /* unexpected SHUTDOWN-ACK... do OOTB handling... */ sctp_send_shutdown_complete(stcb, net, 1); SCTP_TCB_UNLOCK(stcb); return; } if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT)) { /* unexpected SHUTDOWN-ACK... so ignore... */ SCTP_TCB_UNLOCK(stcb); return; } if (asoc->control_pdapi) { /* * With a normal shutdown we assume the end of last record. */ SCTP_INP_READ_LOCK(stcb->sctp_ep); asoc->control_pdapi->end_added = 1; asoc->control_pdapi->pdapi_aborted = 1; asoc->control_pdapi = NULL; SCTP_INP_READ_UNLOCK(stcb->sctp_ep); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { /* assoc was freed while we were unlocked */ SCTP_SOCKET_UNLOCK(so, 1); return; } #endif sctp_sorwakeup(stcb->sctp_ep, stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } #ifdef INVARIANTS if (!TAILQ_EMPTY(&asoc->send_queue) || !TAILQ_EMPTY(&asoc->sent_queue) || !stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { panic("Queues are not empty when handling SHUTDOWN-ACK"); } #endif /* stop the timer */ sctp_timer_stop(SCTP_TIMER_TYPE_SHUTDOWN, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_10); /* send SHUTDOWN-COMPLETE */ sctp_send_shutdown_complete(stcb, net, 0); /* notify upper layer protocol */ if (stcb->sctp_socket) { if ((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) { stcb->sctp_socket->so_snd.sb_cc = 0; } sctp_ulp_notify(SCTP_NOTIFY_ASSOC_DOWN, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } SCTP_STAT_INCR_COUNTER32(sctps_shutdown); /* free the TCB but first save off the ep */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_11); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* * Skip past the param header and then we will find the chunk that caused the * problem. There are two possiblities ASCONF or FWD-TSN other than that and * our peer must be broken. */ static void sctp_process_unrecog_chunk(struct sctp_tcb *stcb, struct sctp_paramhdr *phdr, struct sctp_nets *net) { struct sctp_chunkhdr *chk; chk = (struct sctp_chunkhdr *)((caddr_t)phdr + sizeof(*phdr)); switch (chk->chunk_type) { case SCTP_ASCONF_ACK: case SCTP_ASCONF: sctp_asconf_cleanup(stcb, net); break; case SCTP_FORWARD_CUM_TSN: stcb->asoc.prsctp_supported = 0; break; default: SCTPDBG(SCTP_DEBUG_INPUT2, "Peer does not support chunk type %d(%x)??\n", chk->chunk_type, (uint32_t) chk->chunk_type); break; } } /* * Skip past the param header and then we will find the param that caused the * problem. There are a number of param's in a ASCONF OR the prsctp param * these will turn of specific features. * XXX: Is this the right thing to do? */ static void sctp_process_unrecog_param(struct sctp_tcb *stcb, struct sctp_paramhdr *phdr) { struct sctp_paramhdr *pbad; pbad = phdr + 1; switch (ntohs(pbad->param_type)) { /* pr-sctp draft */ case SCTP_PRSCTP_SUPPORTED: stcb->asoc.prsctp_supported = 0; break; case SCTP_SUPPORTED_CHUNK_EXT: break; /* draft-ietf-tsvwg-addip-sctp */ case SCTP_HAS_NAT_SUPPORT: stcb->asoc.peer_supports_nat = 0; break; case SCTP_ADD_IP_ADDRESS: case SCTP_DEL_IP_ADDRESS: case SCTP_SET_PRIM_ADDR: stcb->asoc.asconf_supported = 0; break; case SCTP_SUCCESS_REPORT: case SCTP_ERROR_CAUSE_IND: SCTPDBG(SCTP_DEBUG_INPUT2, "Huh, the peer does not support success? or error cause?\n"); SCTPDBG(SCTP_DEBUG_INPUT2, "Turning off ASCONF to this strange peer\n"); stcb->asoc.asconf_supported = 0; break; default: SCTPDBG(SCTP_DEBUG_INPUT2, "Peer does not support param type %d(%x)??\n", pbad->param_type, (uint32_t) pbad->param_type); break; } } static int sctp_handle_error(struct sctp_chunkhdr *ch, struct sctp_tcb *stcb, struct sctp_nets *net) { int chklen; struct sctp_paramhdr *phdr; uint16_t error, error_type; uint16_t error_len; struct sctp_association *asoc; int adjust; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif /* parse through all of the errors and process */ asoc = &stcb->asoc; phdr = (struct sctp_paramhdr *)((caddr_t)ch + sizeof(struct sctp_chunkhdr)); chklen = ntohs(ch->chunk_length) - sizeof(struct sctp_chunkhdr); error = 0; while ((size_t)chklen >= sizeof(struct sctp_paramhdr)) { /* Process an Error Cause */ error_type = ntohs(phdr->param_type); error_len = ntohs(phdr->param_length); if ((error_len > chklen) || (error_len == 0)) { /* invalid param length for this param */ SCTPDBG(SCTP_DEBUG_INPUT1, "Bogus length in error param- chunk left:%d errorlen:%d\n", chklen, error_len); return (0); } if (error == 0) { /* report the first error cause */ error = error_type; } switch (error_type) { case SCTP_CAUSE_INVALID_STREAM: case SCTP_CAUSE_MISSING_PARAM: case SCTP_CAUSE_INVALID_PARAM: case SCTP_CAUSE_NO_USER_DATA: SCTPDBG(SCTP_DEBUG_INPUT1, "Software error we got a %d back? We have a bug :/ (or do they?)\n", error_type); break; case SCTP_CAUSE_NAT_COLLIDING_STATE: SCTPDBG(SCTP_DEBUG_INPUT2, "Received Colliding state abort flags:%x\n", ch->chunk_flags); if (sctp_handle_nat_colliding_state(stcb)) { return (0); } break; case SCTP_CAUSE_NAT_MISSING_STATE: SCTPDBG(SCTP_DEBUG_INPUT2, "Received missing state abort flags:%x\n", ch->chunk_flags); if (sctp_handle_nat_missing_state(stcb, net)) { return (0); } break; case SCTP_CAUSE_STALE_COOKIE: /* * We only act if we have echoed a cookie and are * waiting. */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) { int *p; p = (int *)((caddr_t)phdr + sizeof(*phdr)); /* Save the time doubled */ asoc->cookie_preserve_req = ntohl(*p) << 1; asoc->stale_cookie_count++; if (asoc->stale_cookie_count > asoc->max_init_times) { sctp_abort_notification(stcb, 0, 0, NULL, SCTP_SO_NOT_LOCKED); /* now free the asoc */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_12); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif return (-1); } /* blast back to INIT state */ sctp_toss_old_cookies(stcb, &stcb->asoc); asoc->state &= ~SCTP_STATE_COOKIE_ECHOED; asoc->state |= SCTP_STATE_COOKIE_WAIT; sctp_stop_all_cookie_timers(stcb); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); } break; case SCTP_CAUSE_UNRESOLVABLE_ADDR: /* * Nothing we can do here, we don't do hostname * addresses so if the peer does not like my IPv6 * (or IPv4 for that matter) it does not matter. If * they don't support that type of address, they can * NOT possibly get that packet type... i.e. with no * IPv6 you can't recieve a IPv6 packet. so we can * safely ignore this one. If we ever added support * for HOSTNAME Addresses, then we would need to do * something here. */ break; case SCTP_CAUSE_UNRECOG_CHUNK: sctp_process_unrecog_chunk(stcb, phdr, net); break; case SCTP_CAUSE_UNRECOG_PARAM: sctp_process_unrecog_param(stcb, phdr); break; case SCTP_CAUSE_COOKIE_IN_SHUTDOWN: /* * We ignore this since the timer will drive out a * new cookie anyway and there timer will drive us * to send a SHUTDOWN_COMPLETE. We can't send one * here since we don't have their tag. */ break; case SCTP_CAUSE_DELETING_LAST_ADDR: case SCTP_CAUSE_RESOURCE_SHORTAGE: case SCTP_CAUSE_DELETING_SRC_ADDR: /* * We should NOT get these here, but in a * ASCONF-ACK. */ SCTPDBG(SCTP_DEBUG_INPUT2, "Peer sends ASCONF errors in a Operational Error?<%d>?\n", error_type); break; case SCTP_CAUSE_OUT_OF_RESC: /* * And what, pray tell do we do with the fact that * the peer is out of resources? Not really sure we * could do anything but abort. I suspect this * should have came WITH an abort instead of in a * OP-ERROR. */ break; default: SCTPDBG(SCTP_DEBUG_INPUT1, "sctp_handle_error: unknown error type = 0x%xh\n", error_type); break; } adjust = SCTP_SIZE32(error_len); chklen -= adjust; phdr = (struct sctp_paramhdr *)((caddr_t)phdr + adjust); } sctp_ulp_notify(SCTP_NOTIFY_REMOTE_ERROR, stcb, error, ch, SCTP_SO_NOT_LOCKED); return (0); } static int sctp_handle_init_ack(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_init_ack_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, int *abort_no_unlock, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id) { struct sctp_init_ack *init_ack; struct mbuf *op_err; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_init_ack: handling INIT-ACK\n"); if (stcb == NULL) { SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_init_ack: TCB is null\n"); return (-1); } if (ntohs(cp->ch.chunk_length) < sizeof(struct sctp_init_ack_chunk)) { /* Invalid length */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } init_ack = &cp->init; /* validate parameters */ if (init_ack->initiate_tag == 0) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } if (ntohl(init_ack->a_rwnd) < SCTP_MIN_RWND) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } if (init_ack->num_inbound_streams == 0) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } if (init_ack->num_outbound_streams == 0) { /* protocol error... send an abort */ op_err = sctp_generate_cause(SCTP_CAUSE_INVALID_PARAM, ""); sctp_abort_association(stcb->sctp_ep, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, net->port); *abort_no_unlock = 1; return (-1); } /* process according to association state... */ switch (stcb->asoc.state & SCTP_STATE_MASK) { case SCTP_STATE_COOKIE_WAIT: /* this is the expected state for this chunk */ /* process the INIT-ACK parameters */ if (stcb->asoc.primary_destination->dest_state & SCTP_ADDR_UNCONFIRMED) { /* * The primary is where we sent the INIT, we can * always consider it confirmed when the INIT-ACK is * returned. Do this before we load addresses * though. */ stcb->asoc.primary_destination->dest_state &= ~SCTP_ADDR_UNCONFIRMED; sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, stcb, 0, (void *)stcb->asoc.primary_destination, SCTP_SO_NOT_LOCKED); } if (sctp_process_init_ack(m, iphlen, offset, src, dst, sh, cp, stcb, net, abort_no_unlock, mflowtype, mflowid, vrf_id) < 0) { /* error in parsing parameters */ return (-1); } /* update our state */ SCTPDBG(SCTP_DEBUG_INPUT2, "moving to COOKIE-ECHOED state\n"); SCTP_SET_STATE(&stcb->asoc, SCTP_STATE_COOKIE_ECHOED); /* reset the RTO calc */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; (void)SCTP_GETTIME_TIMEVAL(&stcb->asoc.time_entered); /* * collapse the init timer back in case of a exponential * backoff */ sctp_timer_start(SCTP_TIMER_TYPE_COOKIE, stcb->sctp_ep, stcb, net); /* * the send at the end of the inbound data processing will * cause the cookie to be sent */ break; case SCTP_STATE_SHUTDOWN_SENT: /* incorrect state... discard */ break; case SCTP_STATE_COOKIE_ECHOED: /* incorrect state... discard */ break; case SCTP_STATE_OPEN: /* incorrect state... discard */ break; case SCTP_STATE_EMPTY: case SCTP_STATE_INUSE: default: /* incorrect state... discard */ return (-1); break; } SCTPDBG(SCTP_DEBUG_INPUT1, "Leaving handle-init-ack end\n"); return (0); } static struct sctp_tcb * sctp_process_cookie_new(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_state_cookie *cookie, int cookie_len, struct sctp_inpcb *inp, struct sctp_nets **netp, struct sockaddr *init_src, int *notification, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port); /* * handle a state cookie for an existing association m: input packet mbuf * chain-- assumes a pullup on IP/SCTP/COOKIE-ECHO chunk note: this is a * "split" mbuf and the cookie signature does not exist offset: offset into * mbuf to the cookie-echo chunk */ static struct sctp_tcb * sctp_process_cookie_existing(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_state_cookie *cookie, int cookie_len, struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_nets **netp, struct sockaddr *init_src, int *notification, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_association *asoc; struct sctp_init_chunk *init_cp, init_buf; struct sctp_init_ack_chunk *initack_cp, initack_buf; struct sctp_nets *net; struct mbuf *op_err; int init_offset, initack_offset, i; int retval; int spec_flag = 0; uint32_t how_indx; #if defined(SCTP_DETAILED_STR_STATS) int j; #endif net = *netp; /* I know that the TCB is non-NULL from the caller */ asoc = &stcb->asoc; for (how_indx = 0; how_indx < sizeof(asoc->cookie_how); how_indx++) { if (asoc->cookie_how[how_indx] == 0) break; } if (how_indx < sizeof(asoc->cookie_how)) { asoc->cookie_how[how_indx] = 1; } if (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) { /* SHUTDOWN came in after sending INIT-ACK */ sctp_send_shutdown_ack(stcb, stcb->asoc.primary_destination); op_err = sctp_generate_cause(SCTP_CAUSE_COOKIE_IN_SHUTDOWN, ""); sctp_send_operr_to(src, dst, sh, cookie->peers_vtag, op_err, mflowtype, mflowid, inp->fibnum, vrf_id, net->port); if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 2; return (NULL); } /* * find and validate the INIT chunk in the cookie (peer's info) the * INIT should start after the cookie-echo header struct (chunk * header, state cookie header struct) */ init_offset = offset += sizeof(struct sctp_cookie_echo_chunk); init_cp = (struct sctp_init_chunk *) sctp_m_getptr(m, init_offset, sizeof(struct sctp_init_chunk), (uint8_t *) & init_buf); if (init_cp == NULL) { /* could not pull a INIT chunk in cookie */ return (NULL); } if (init_cp->ch.chunk_type != SCTP_INITIATION) { return (NULL); } /* * find and validate the INIT-ACK chunk in the cookie (my info) the * INIT-ACK follows the INIT chunk */ initack_offset = init_offset + SCTP_SIZE32(ntohs(init_cp->ch.chunk_length)); initack_cp = (struct sctp_init_ack_chunk *) sctp_m_getptr(m, initack_offset, sizeof(struct sctp_init_ack_chunk), (uint8_t *) & initack_buf); if (initack_cp == NULL) { /* could not pull INIT-ACK chunk in cookie */ return (NULL); } if (initack_cp->ch.chunk_type != SCTP_INITIATION_ACK) { return (NULL); } if ((ntohl(initack_cp->init.initiate_tag) == asoc->my_vtag) && (ntohl(init_cp->init.initiate_tag) == asoc->peer_vtag)) { /* * case D in Section 5.2.4 Table 2: MMAA process accordingly * to get into the OPEN state */ if (ntohl(initack_cp->init.initial_tsn) != asoc->init_seq_number) { /*- * Opps, this means that we somehow generated two vtag's * the same. I.e. we did: * Us Peer * <---INIT(tag=a)------ * ----INIT-ACK(tag=t)--> * ----INIT(tag=t)------> *1 * <---INIT-ACK(tag=a)--- * <----CE(tag=t)------------- *2 * * At point *1 we should be generating a different * tag t'. Which means we would throw away the CE and send * ours instead. Basically this is case C (throw away side). */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 17; return (NULL); } switch (SCTP_GET_STATE(asoc)) { case SCTP_STATE_COOKIE_WAIT: case SCTP_STATE_COOKIE_ECHOED: /* * INIT was sent but got a COOKIE_ECHO with the * correct tags... just accept it...but we must * process the init so that we can make sure we have * the right seq no's. */ /* First we must process the INIT !! */ retval = sctp_process_init(init_cp, stcb); if (retval < 0) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 3; return (NULL); } /* we have already processed the INIT so no problem */ sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_13); sctp_timer_stop(SCTP_TIMER_TYPE_INIT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_14); /* update current state */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) SCTP_STAT_INCR_COUNTER32(sctps_activeestab); else SCTP_STAT_INCR_COUNTER32(sctps_collisionestab); SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } SCTP_STAT_INCR_GAUGE32(sctps_currestab); sctp_stop_all_cookie_timers(stcb); if (((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) && (inp->sctp_socket->so_qlimit == 0) ) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif /* * Here is where collision would go if we * did a connect() and instead got a * init/init-ack/cookie done before the * init-ack came back.. */ stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_add_int(&stcb->asoc.refcnt, -1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { SCTP_SOCKET_UNLOCK(so, 1); return (NULL); } #endif soisconnected(stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* notify upper layer */ *notification = SCTP_NOTIFY_ASSOC_UP; /* * since we did not send a HB make sure we don't * double things */ net->hb_responded = 1; net->RTO = sctp_calculate_rto(stcb, asoc, net, &cookie->time_entered, sctp_align_unsafe_makecopy, SCTP_RTT_FROM_NON_DATA); if (stcb->asoc.sctp_autoclose_ticks && (sctp_is_feature_on(inp, SCTP_PCB_FLAGS_AUTOCLOSE))) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, inp, stcb, NULL); } break; default: /* * we're in the OPEN state (or beyond), so peer must * have simply lost the COOKIE-ACK */ break; } /* end switch */ sctp_stop_all_cookie_timers(stcb); /* * We ignore the return code here.. not sure if we should * somehow abort.. but we do have an existing asoc. This * really should not fail. */ if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 4; return (NULL); } /* respond with a COOKIE-ACK */ sctp_toss_old_cookies(stcb, asoc); sctp_send_cookie_ack(stcb); if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 5; return (stcb); } if (ntohl(initack_cp->init.initiate_tag) != asoc->my_vtag && ntohl(init_cp->init.initiate_tag) == asoc->peer_vtag && cookie->tie_tag_my_vtag == 0 && cookie->tie_tag_peer_vtag == 0) { /* * case C in Section 5.2.4 Table 2: XMOO silently discard */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 6; return (NULL); } /* * If nat support, and the below and stcb is established, send back * a ABORT(colliding state) if we are established. */ if ((SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) && (asoc->peer_supports_nat) && ((ntohl(initack_cp->init.initiate_tag) == asoc->my_vtag) && ((ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) || (asoc->peer_vtag == 0)))) { /* * Special case - Peer's support nat. We may have two init's * that we gave out the same tag on since one was not * established.. i.e. we get INIT from host-1 behind the nat * and we respond tag-a, we get a INIT from host-2 behind * the nat and we get tag-a again. Then we bring up host-1 * (or 2's) assoc, Then comes the cookie from hsot-2 (or 1). * Now we have colliding state. We must send an abort here * with colliding state indication. */ op_err = sctp_generate_cause(SCTP_CAUSE_NAT_COLLIDING_STATE, ""); sctp_send_abort(m, iphlen, src, dst, sh, 0, op_err, mflowtype, mflowid, inp->fibnum, vrf_id, port); return (NULL); } if ((ntohl(initack_cp->init.initiate_tag) == asoc->my_vtag) && ((ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) || (asoc->peer_vtag == 0))) { /* * case B in Section 5.2.4 Table 2: MXAA or MOAA my info * should be ok, re-accept peer info */ if (ntohl(initack_cp->init.initial_tsn) != asoc->init_seq_number) { /* * Extension of case C. If we hit this, then the * random number generator returned the same vtag * when we first sent our INIT-ACK and when we later * sent our INIT. The side with the seq numbers that * are different will be the one that normnally * would have hit case C. This in effect "extends" * our vtags in this collision case to be 64 bits. * The same collision could occur aka you get both * vtag and seq number the same twice in a row.. but * is much less likely. If it did happen then we * would proceed through and bring up the assoc.. we * may end up with the wrong stream setup however.. * which would be bad.. but there is no way to * tell.. until we send on a stream that does not * exist :-) */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 7; return (NULL); } if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 8; sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_15); sctp_stop_all_cookie_timers(stcb); /* * since we did not send a HB make sure we don't double * things */ net->hb_responded = 1; if (stcb->asoc.sctp_autoclose_ticks && sctp_is_feature_on(inp, SCTP_PCB_FLAGS_AUTOCLOSE)) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, inp, stcb, NULL); } asoc->my_rwnd = ntohl(initack_cp->init.a_rwnd); asoc->pre_open_streams = ntohs(initack_cp->init.num_outbound_streams); if (ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) { /* * Ok the peer probably discarded our data (if we * echoed a cookie+data). So anything on the * sent_queue should be marked for retransmit, we * may not get something to kick us so it COULD * still take a timeout to move these.. but it can't * hurt to mark them. */ struct sctp_tmit_chunk *chk; TAILQ_FOREACH(chk, &stcb->asoc.sent_queue, sctp_next) { if (chk->sent < SCTP_DATAGRAM_RESEND) { chk->sent = SCTP_DATAGRAM_RESEND; sctp_flight_size_decrease(chk); sctp_total_flight_decrease(stcb, chk); sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); spec_flag++; } } } /* process the INIT info (peer's info) */ retval = sctp_process_init(init_cp, stcb); if (retval < 0) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 9; return (NULL); } if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 10; return (NULL); } if ((asoc->state & SCTP_STATE_COOKIE_WAIT) || (asoc->state & SCTP_STATE_COOKIE_ECHOED)) { *notification = SCTP_NOTIFY_ASSOC_UP; if (((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) && (inp->sctp_socket->so_qlimit == 0)) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_add_int(&stcb->asoc.refcnt, -1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { SCTP_SOCKET_UNLOCK(so, 1); return (NULL); } #endif soisconnected(stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) SCTP_STAT_INCR_COUNTER32(sctps_activeestab); else SCTP_STAT_INCR_COUNTER32(sctps_collisionestab); SCTP_STAT_INCR_GAUGE32(sctps_currestab); } else if (SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) { SCTP_STAT_INCR_COUNTER32(sctps_restartestab); } else { SCTP_STAT_INCR_COUNTER32(sctps_collisionestab); } SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } sctp_stop_all_cookie_timers(stcb); sctp_toss_old_cookies(stcb, asoc); sctp_send_cookie_ack(stcb); if (spec_flag) { /* * only if we have retrans set do we do this. What * this call does is get only the COOKIE-ACK out and * then when we return the normal call to * sctp_chunk_output will get the retrans out behind * this. */ sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_COOKIE_ACK, SCTP_SO_NOT_LOCKED); } if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 11; return (stcb); } if ((ntohl(initack_cp->init.initiate_tag) != asoc->my_vtag && ntohl(init_cp->init.initiate_tag) != asoc->peer_vtag) && cookie->tie_tag_my_vtag == asoc->my_vtag_nonce && cookie->tie_tag_peer_vtag == asoc->peer_vtag_nonce && cookie->tie_tag_peer_vtag != 0) { struct sctpasochead *head; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif if (asoc->peer_supports_nat) { /* * This is a gross gross hack. Just call the * cookie_new code since we are allowing a duplicate * association. I hope this works... */ return (sctp_process_cookie_new(m, iphlen, offset, src, dst, sh, cookie, cookie_len, inp, netp, init_src, notification, auth_skipped, auth_offset, auth_len, mflowtype, mflowid, vrf_id, port)); } /* * case A in Section 5.2.4 Table 2: XXMM (peer restarted) */ /* temp code */ if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 12; sctp_timer_stop(SCTP_TIMER_TYPE_INIT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_16); sctp_timer_stop(SCTP_TIMER_TYPE_HEARTBEAT, inp, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_17); /* notify upper layer */ *notification = SCTP_NOTIFY_ASSOC_RESTART; atomic_add_int(&stcb->asoc.refcnt, 1); if ((SCTP_GET_STATE(asoc) != SCTP_STATE_OPEN) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT)) { SCTP_STAT_INCR_GAUGE32(sctps_currestab); } if (SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) { SCTP_STAT_INCR_GAUGE32(sctps_restartestab); } else if (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) { SCTP_STAT_INCR_GAUGE32(sctps_collisionestab); } if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } else if (!(asoc->state & SCTP_STATE_SHUTDOWN_SENT)) { /* move to OPEN state, if not in SHUTDOWN_SENT */ SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); } asoc->pre_open_streams = ntohs(initack_cp->init.num_outbound_streams); asoc->init_seq_number = ntohl(initack_cp->init.initial_tsn); asoc->sending_seq = asoc->asconf_seq_out = asoc->str_reset_seq_out = asoc->init_seq_number; asoc->asconf_seq_out_acked = asoc->asconf_seq_out - 1; asoc->asconf_seq_in = asoc->last_acked_seq = asoc->init_seq_number - 1; asoc->str_reset_seq_in = asoc->init_seq_number; asoc->advanced_peer_ack_point = asoc->last_acked_seq; if (asoc->mapping_array) { memset(asoc->mapping_array, 0, asoc->mapping_array_size); } if (asoc->nr_mapping_array) { memset(asoc->nr_mapping_array, 0, asoc->mapping_array_size); } SCTP_TCB_UNLOCK(stcb); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); SCTP_SOCKET_LOCK(so, 1); #endif SCTP_INP_INFO_WLOCK(); SCTP_INP_WLOCK(stcb->sctp_ep); SCTP_TCB_LOCK(stcb); atomic_add_int(&stcb->asoc.refcnt, -1); /* send up all the data */ SCTP_TCB_SEND_LOCK(stcb); sctp_report_all_outbound(stcb, 0, 1, SCTP_SO_LOCKED); for (i = 0; i < stcb->asoc.streamoutcnt; i++) { stcb->asoc.strmout[i].chunks_on_queues = 0; #if defined(SCTP_DETAILED_STR_STATS) for (j = 0; j < SCTP_PR_SCTP_MAX + 1; j++) { asoc->strmout[i].abandoned_sent[j] = 0; asoc->strmout[i].abandoned_unsent[j] = 0; } #else asoc->strmout[i].abandoned_sent[0] = 0; asoc->strmout[i].abandoned_unsent[0] = 0; #endif stcb->asoc.strmout[i].stream_no = i; stcb->asoc.strmout[i].next_sequence_send = 0; stcb->asoc.strmout[i].last_msg_incomplete = 0; } /* process the INIT-ACK info (my info) */ asoc->my_vtag = ntohl(initack_cp->init.initiate_tag); asoc->my_rwnd = ntohl(initack_cp->init.a_rwnd); /* pull from vtag hash */ LIST_REMOVE(stcb, sctp_asocs); /* re-insert to new vtag position */ head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(stcb->asoc.my_vtag, SCTP_BASE_INFO(hashasocmark))]; /* * put it in the bucket in the vtag hash of assoc's for the * system */ LIST_INSERT_HEAD(head, stcb, sctp_asocs); SCTP_TCB_SEND_UNLOCK(stcb); SCTP_INP_WUNLOCK(stcb->sctp_ep); SCTP_INP_INFO_WUNLOCK(); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif asoc->total_flight = 0; asoc->total_flight_count = 0; /* process the INIT info (peer's info) */ retval = sctp_process_init(init_cp, stcb); if (retval < 0) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 13; return (NULL); } /* * since we did not send a HB make sure we don't double * things */ net->hb_responded = 1; if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 14; return (NULL); } /* respond with a COOKIE-ACK */ sctp_stop_all_cookie_timers(stcb); sctp_toss_old_cookies(stcb, asoc); sctp_send_cookie_ack(stcb); if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 15; return (stcb); } if (how_indx < sizeof(asoc->cookie_how)) asoc->cookie_how[how_indx] = 16; /* all other cases... */ return (NULL); } /* * handle a state cookie for a new association m: input packet mbuf chain-- * assumes a pullup on IP/SCTP/COOKIE-ECHO chunk note: this is a "split" mbuf * and the cookie signature does not exist offset: offset into mbuf to the * cookie-echo chunk length: length of the cookie chunk to: where the init * was from returns a new TCB */ static struct sctp_tcb * sctp_process_cookie_new(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_state_cookie *cookie, int cookie_len, struct sctp_inpcb *inp, struct sctp_nets **netp, struct sockaddr *init_src, int *notification, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_tcb *stcb; struct sctp_init_chunk *init_cp, init_buf; struct sctp_init_ack_chunk *initack_cp, initack_buf; union sctp_sockstore store; struct sctp_association *asoc; int init_offset, initack_offset, initack_limit; int retval; int error = 0; uint8_t auth_chunk_buf[SCTP_PARAM_BUFFER_SIZE]; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; so = SCTP_INP_SO(inp); #endif /* * find and validate the INIT chunk in the cookie (peer's info) the * INIT should start after the cookie-echo header struct (chunk * header, state cookie header struct) */ init_offset = offset + sizeof(struct sctp_cookie_echo_chunk); init_cp = (struct sctp_init_chunk *) sctp_m_getptr(m, init_offset, sizeof(struct sctp_init_chunk), (uint8_t *) & init_buf); if (init_cp == NULL) { /* could not pull a INIT chunk in cookie */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: could not pull INIT chunk hdr\n"); return (NULL); } if (init_cp->ch.chunk_type != SCTP_INITIATION) { SCTPDBG(SCTP_DEBUG_INPUT1, "HUH? process_cookie_new: could not find INIT chunk!\n"); return (NULL); } initack_offset = init_offset + SCTP_SIZE32(ntohs(init_cp->ch.chunk_length)); /* * find and validate the INIT-ACK chunk in the cookie (my info) the * INIT-ACK follows the INIT chunk */ initack_cp = (struct sctp_init_ack_chunk *) sctp_m_getptr(m, initack_offset, sizeof(struct sctp_init_ack_chunk), (uint8_t *) & initack_buf); if (initack_cp == NULL) { /* could not pull INIT-ACK chunk in cookie */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: could not pull INIT-ACK chunk hdr\n"); return (NULL); } if (initack_cp->ch.chunk_type != SCTP_INITIATION_ACK) { return (NULL); } /* * NOTE: We can't use the INIT_ACK's chk_length to determine the * "initack_limit" value. This is because the chk_length field * includes the length of the cookie, but the cookie is omitted when * the INIT and INIT_ACK are tacked onto the cookie... */ initack_limit = offset + cookie_len; /* * now that we know the INIT/INIT-ACK are in place, create a new TCB * and popluate */ /* * Here we do a trick, we set in NULL for the proc/thread argument. * We do this since in effect we only use the p argument when the * socket is unbound and we must do an implicit bind. Since we are * getting a cookie, we cannot be unbound. */ stcb = sctp_aloc_assoc(inp, init_src, &error, ntohl(initack_cp->init.initiate_tag), vrf_id, ntohs(initack_cp->init.num_outbound_streams), (struct thread *)NULL ); if (stcb == NULL) { struct mbuf *op_err; /* memory problem? */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: no room for another TCB!\n"); op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, (struct sctp_tcb *)NULL, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); return (NULL); } /* get the correct sctp_nets */ if (netp) *netp = sctp_findnet(stcb, init_src); asoc = &stcb->asoc; /* get scope variables out of cookie */ asoc->scope.ipv4_local_scope = cookie->ipv4_scope; asoc->scope.site_scope = cookie->site_scope; asoc->scope.local_scope = cookie->local_scope; asoc->scope.loopback_scope = cookie->loopback_scope; if ((asoc->scope.ipv4_addr_legal != cookie->ipv4_addr_legal) || (asoc->scope.ipv6_addr_legal != cookie->ipv6_addr_legal)) { struct mbuf *op_err; /* * Houston we have a problem. The EP changed while the * cookie was in flight. Only recourse is to abort the * association. */ atomic_add_int(&stcb->asoc.refcnt, 1); op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, (struct sctp_tcb *)NULL, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_18); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* process the INIT-ACK info (my info) */ asoc->my_vtag = ntohl(initack_cp->init.initiate_tag); asoc->my_rwnd = ntohl(initack_cp->init.a_rwnd); asoc->pre_open_streams = ntohs(initack_cp->init.num_outbound_streams); asoc->init_seq_number = ntohl(initack_cp->init.initial_tsn); asoc->sending_seq = asoc->asconf_seq_out = asoc->str_reset_seq_out = asoc->init_seq_number; asoc->asconf_seq_out_acked = asoc->asconf_seq_out - 1; asoc->asconf_seq_in = asoc->last_acked_seq = asoc->init_seq_number - 1; asoc->str_reset_seq_in = asoc->init_seq_number; asoc->advanced_peer_ack_point = asoc->last_acked_seq; /* process the INIT info (peer's info) */ if (netp) retval = sctp_process_init(init_cp, stcb); else retval = 0; if (retval < 0) { atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_19); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* load all addresses */ if (sctp_load_addresses_from_init(stcb, m, init_offset + sizeof(struct sctp_init_chunk), initack_offset, src, dst, init_src)) { atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_20); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* * verify any preceding AUTH chunk that was skipped */ /* pull the local authentication parameters from the cookie/init-ack */ sctp_auth_get_cookie_params(stcb, m, initack_offset + sizeof(struct sctp_init_ack_chunk), initack_limit - (initack_offset + sizeof(struct sctp_init_ack_chunk))); if (auth_skipped) { struct sctp_auth_chunk *auth; auth = (struct sctp_auth_chunk *) sctp_m_getptr(m, auth_offset, auth_len, auth_chunk_buf); if ((auth == NULL) || sctp_handle_auth(stcb, auth, m, auth_offset)) { /* auth HMAC failed, dump the assoc and packet */ SCTPDBG(SCTP_DEBUG_AUTH1, "COOKIE-ECHO: AUTH failed\n"); atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_21); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } else { /* remaining chunks checked... good to go */ stcb->asoc.authenticated = 1; } } /* update current state */ SCTPDBG(SCTP_DEBUG_INPUT2, "moving to OPEN state\n"); SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } sctp_stop_all_cookie_timers(stcb); SCTP_STAT_INCR_COUNTER32(sctps_passiveestab); SCTP_STAT_INCR_GAUGE32(sctps_currestab); /* * if we're doing ASCONFs, check to see if we have any new local * addresses that need to get added to the peer (eg. addresses * changed while cookie echo in flight). This needs to be done * after we go to the OPEN state to do the correct asconf * processing. else, make sure we have the correct addresses in our * lists */ /* warning, we re-use sin, sin6, sa_store here! */ /* pull in local_address (our "from" address) */ switch (cookie->laddr_type) { #ifdef INET case SCTP_IPV4_ADDRESS: /* source addr is IPv4 */ memset(&store.sin, 0, sizeof(struct sockaddr_in)); store.sin.sin_family = AF_INET; store.sin.sin_len = sizeof(struct sockaddr_in); store.sin.sin_addr.s_addr = cookie->laddress[0]; break; #endif #ifdef INET6 case SCTP_IPV6_ADDRESS: /* source addr is IPv6 */ memset(&store.sin6, 0, sizeof(struct sockaddr_in6)); store.sin6.sin6_family = AF_INET6; store.sin6.sin6_len = sizeof(struct sockaddr_in6); store.sin6.sin6_scope_id = cookie->scope_id; memcpy(&store.sin6.sin6_addr, cookie->laddress, sizeof(struct in6_addr)); break; #endif default: atomic_add_int(&stcb->asoc.refcnt, 1); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_22); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif atomic_subtract_int(&stcb->asoc.refcnt, 1); return (NULL); } /* set up to notify upper layer */ *notification = SCTP_NOTIFY_ASSOC_UP; if (((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) && (inp->sctp_socket->so_qlimit == 0)) { /* * This is an endpoint that called connect() how it got a * cookie that is NEW is a bit of a mystery. It must be that * the INIT was sent, but before it got there.. a complete * INIT/INIT-ACK/COOKIE arrived. But of course then it * should have went to the other code.. not here.. oh well.. * a bit of protection is worth having.. */ stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { SCTP_SOCKET_UNLOCK(so, 1); return (NULL); } #endif soisconnected(stcb->sctp_socket); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } else if ((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) && (inp->sctp_socket->so_qlimit)) { /* * We don't want to do anything with this one. Since it is * the listening guy. The timer will get started for * accepted connections in the caller. */ ; } /* since we did not send a HB make sure we don't double things */ if ((netp) && (*netp)) (*netp)->hb_responded = 1; if (stcb->asoc.sctp_autoclose_ticks && sctp_is_feature_on(inp, SCTP_PCB_FLAGS_AUTOCLOSE)) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, inp, stcb, NULL); } (void)SCTP_GETTIME_TIMEVAL(&stcb->asoc.time_entered); if ((netp != NULL) && (*netp != NULL)) { /* calculate the RTT and set the encaps port */ (*netp)->RTO = sctp_calculate_rto(stcb, asoc, *netp, &cookie->time_entered, sctp_align_unsafe_makecopy, SCTP_RTT_FROM_NON_DATA); #if defined(INET) || defined(INET6) if (((*netp)->port == 0) && (port != 0)) { sctp_pathmtu_adjustment(stcb, (*netp)->mtu - sizeof(struct udphdr)); } (*netp)->port = port; #endif } /* respond with a COOKIE-ACK */ sctp_send_cookie_ack(stcb); /* * check the address lists for any ASCONFs that need to be sent * AFTER the cookie-ack is sent */ sctp_check_address_list(stcb, m, initack_offset + sizeof(struct sctp_init_ack_chunk), initack_limit - (initack_offset + sizeof(struct sctp_init_ack_chunk)), &store.sa, cookie->local_scope, cookie->site_scope, cookie->ipv4_scope, cookie->loopback_scope); return (stcb); } /* * CODE LIKE THIS NEEDS TO RUN IF the peer supports the NAT extension, i.e * we NEED to make sure we are not already using the vtag. If so we * need to send back an ABORT-TRY-AGAIN-WITH-NEW-TAG No middle box bit! head = &SCTP_BASE_INFO(sctp_asochash)[SCTP_PCBHASH_ASOC(tag, SCTP_BASE_INFO(hashasocmark))]; LIST_FOREACH(stcb, head, sctp_asocs) { if ((stcb->asoc.my_vtag == tag) && (stcb->rport == rport) && (inp == stcb->sctp_ep)) { -- SEND ABORT - TRY AGAIN -- } } */ /* * handles a COOKIE-ECHO message stcb: modified to either a new or left as * existing (non-NULL) TCB */ static struct mbuf * sctp_handle_cookie_echo(struct mbuf *m, int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_cookie_echo_chunk *cp, struct sctp_inpcb **inp_p, struct sctp_tcb **stcb, struct sctp_nets **netp, int auth_skipped, uint32_t auth_offset, uint32_t auth_len, struct sctp_tcb **locked_tcb, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port) { struct sctp_state_cookie *cookie; struct sctp_tcb *l_stcb = *stcb; struct sctp_inpcb *l_inp; struct sockaddr *to; struct sctp_pcb *ep; struct mbuf *m_sig; uint8_t calc_sig[SCTP_SIGNATURE_SIZE], tmp_sig[SCTP_SIGNATURE_SIZE]; uint8_t *sig; uint8_t cookie_ok = 0; unsigned int sig_offset, cookie_offset; unsigned int cookie_len; struct timeval now; struct timeval time_expires; int notification = 0; struct sctp_nets *netl; int had_a_existing_tcb = 0; int send_int_conf = 0; #ifdef INET struct sockaddr_in sin; #endif #ifdef INET6 struct sockaddr_in6 sin6; #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_cookie: handling COOKIE-ECHO\n"); if (inp_p == NULL) { return (NULL); } cookie = &cp->cookie; cookie_offset = offset + sizeof(struct sctp_chunkhdr); cookie_len = ntohs(cp->ch.chunk_length); if ((cookie->peerport != sh->src_port) || (cookie->myport != sh->dest_port) || (cookie->my_vtag != sh->v_tag)) { /* * invalid ports or bad tag. Note that we always leave the * v_tag in the header in network order and when we stored * it in the my_vtag slot we also left it in network order. * This maintains the match even though it may be in the * opposite byte order of the machine :-> */ return (NULL); } if (cookie_len < sizeof(struct sctp_cookie_echo_chunk) + sizeof(struct sctp_init_chunk) + sizeof(struct sctp_init_ack_chunk) + SCTP_SIGNATURE_SIZE) { /* cookie too small */ return (NULL); } /* * split off the signature into its own mbuf (since it should not be * calculated in the sctp_hmac_m() call). */ sig_offset = offset + cookie_len - SCTP_SIGNATURE_SIZE; m_sig = m_split(m, sig_offset, M_NOWAIT); if (m_sig == NULL) { /* out of memory or ?? */ return (NULL); } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(m_sig, SCTP_MBUF_SPLIT); } #endif /* * compute the signature/digest for the cookie */ ep = &(*inp_p)->sctp_ep; l_inp = *inp_p; if (l_stcb) { SCTP_TCB_UNLOCK(l_stcb); } SCTP_INP_RLOCK(l_inp); if (l_stcb) { SCTP_TCB_LOCK(l_stcb); } /* which cookie is it? */ if ((cookie->time_entered.tv_sec < (long)ep->time_of_secret_change) && (ep->current_secret_number != ep->last_secret_number)) { /* it's the old cookie */ (void)sctp_hmac_m(SCTP_HMAC, (uint8_t *) ep->secret_key[(int)ep->last_secret_number], SCTP_SECRET_SIZE, m, cookie_offset, calc_sig, 0); } else { /* it's the current cookie */ (void)sctp_hmac_m(SCTP_HMAC, (uint8_t *) ep->secret_key[(int)ep->current_secret_number], SCTP_SECRET_SIZE, m, cookie_offset, calc_sig, 0); } /* get the signature */ SCTP_INP_RUNLOCK(l_inp); sig = (uint8_t *) sctp_m_getptr(m_sig, 0, SCTP_SIGNATURE_SIZE, (uint8_t *) & tmp_sig); if (sig == NULL) { /* couldn't find signature */ sctp_m_freem(m_sig); return (NULL); } /* compare the received digest with the computed digest */ if (memcmp(calc_sig, sig, SCTP_SIGNATURE_SIZE) != 0) { /* try the old cookie? */ if ((cookie->time_entered.tv_sec == (long)ep->time_of_secret_change) && (ep->current_secret_number != ep->last_secret_number)) { /* compute digest with old */ (void)sctp_hmac_m(SCTP_HMAC, (uint8_t *) ep->secret_key[(int)ep->last_secret_number], SCTP_SECRET_SIZE, m, cookie_offset, calc_sig, 0); /* compare */ if (memcmp(calc_sig, sig, SCTP_SIGNATURE_SIZE) == 0) cookie_ok = 1; } } else { cookie_ok = 1; } /* * Now before we continue we must reconstruct our mbuf so that * normal processing of any other chunks will work. */ { struct mbuf *m_at; m_at = m; while (SCTP_BUF_NEXT(m_at) != NULL) { m_at = SCTP_BUF_NEXT(m_at); } SCTP_BUF_NEXT(m_at) = m_sig; } if (cookie_ok == 0) { SCTPDBG(SCTP_DEBUG_INPUT2, "handle_cookie_echo: cookie signature validation failed!\n"); SCTPDBG(SCTP_DEBUG_INPUT2, "offset = %u, cookie_offset = %u, sig_offset = %u\n", (uint32_t) offset, cookie_offset, sig_offset); return (NULL); } /* * check the cookie timestamps to be sure it's not stale */ (void)SCTP_GETTIME_TIMEVAL(&now); /* Expire time is in Ticks, so we convert to seconds */ time_expires.tv_sec = cookie->time_entered.tv_sec + TICKS_TO_SEC(cookie->cookie_life); time_expires.tv_usec = cookie->time_entered.tv_usec; /* * TODO sctp_constants.h needs alternative time macros when _KERNEL * is undefined. */ if (timevalcmp(&now, &time_expires, >)) { /* cookie is stale! */ struct mbuf *op_err; struct sctp_error_stale_cookie *cause; uint32_t tim; op_err = sctp_get_mbuf_for_msg(sizeof(struct sctp_error_stale_cookie), 0, M_NOWAIT, 1, MT_DATA); if (op_err == NULL) { /* FOOBAR */ return (NULL); } /* Set the len */ SCTP_BUF_LEN(op_err) = sizeof(struct sctp_error_stale_cookie); cause = mtod(op_err, struct sctp_error_stale_cookie *); cause->cause.code = htons(SCTP_CAUSE_STALE_COOKIE); cause->cause.length = htons((sizeof(struct sctp_paramhdr) + (sizeof(uint32_t)))); /* seconds to usec */ tim = (now.tv_sec - time_expires.tv_sec) * 1000000; /* add in usec */ if (tim == 0) tim = now.tv_usec - cookie->time_entered.tv_usec; cause->stale_time = htonl(tim); sctp_send_operr_to(src, dst, sh, cookie->peers_vtag, op_err, mflowtype, mflowid, l_inp->fibnum, vrf_id, port); return (NULL); } /* * Now we must see with the lookup address if we have an existing * asoc. This will only happen if we were in the COOKIE-WAIT state * and a INIT collided with us and somewhere the peer sent the * cookie on another address besides the single address our assoc * had for him. In this case we will have one of the tie-tags set at * least AND the address field in the cookie can be used to look it * up. */ to = NULL; switch (cookie->addr_type) { #ifdef INET6 case SCTP_IPV6_ADDRESS: memset(&sin6, 0, sizeof(sin6)); sin6.sin6_family = AF_INET6; sin6.sin6_len = sizeof(sin6); sin6.sin6_port = sh->src_port; sin6.sin6_scope_id = cookie->scope_id; memcpy(&sin6.sin6_addr.s6_addr, cookie->address, sizeof(sin6.sin6_addr.s6_addr)); to = (struct sockaddr *)&sin6; break; #endif #ifdef INET case SCTP_IPV4_ADDRESS: memset(&sin, 0, sizeof(sin)); sin.sin_family = AF_INET; sin.sin_len = sizeof(sin); sin.sin_port = sh->src_port; sin.sin_addr.s_addr = cookie->address[0]; to = (struct sockaddr *)&sin; break; #endif default: /* This should not happen */ return (NULL); } if (*stcb == NULL) { /* Yep, lets check */ *stcb = sctp_findassociation_ep_addr(inp_p, to, netp, dst, NULL); if (*stcb == NULL) { /* * We should have only got back the same inp. If we * got back a different ep we have a problem. The * original findep got back l_inp and now */ if (l_inp != *inp_p) { SCTP_PRINTF("Bad problem find_ep got a diff inp then special_locate?\n"); } } else { if (*locked_tcb == NULL) { /* * In this case we found the assoc only * after we locked the create lock. This * means we are in a colliding case and we * must make sure that we unlock the tcb if * its one of the cases where we throw away * the incoming packets. */ *locked_tcb = *stcb; /* * We must also increment the inp ref count * since the ref_count flags was set when we * did not find the TCB, now we found it * which reduces the refcount.. we must * raise it back out to balance it all :-) */ SCTP_INP_INCR_REF((*stcb)->sctp_ep); if ((*stcb)->sctp_ep != l_inp) { SCTP_PRINTF("Huh? ep:%p diff then l_inp:%p?\n", (void *)(*stcb)->sctp_ep, (void *)l_inp); } } } } cookie_len -= SCTP_SIGNATURE_SIZE; if (*stcb == NULL) { /* this is the "normal" case... get a new TCB */ *stcb = sctp_process_cookie_new(m, iphlen, offset, src, dst, sh, cookie, cookie_len, *inp_p, netp, to, ¬ification, auth_skipped, auth_offset, auth_len, mflowtype, mflowid, vrf_id, port); } else { /* this is abnormal... cookie-echo on existing TCB */ had_a_existing_tcb = 1; *stcb = sctp_process_cookie_existing(m, iphlen, offset, src, dst, sh, cookie, cookie_len, *inp_p, *stcb, netp, to, ¬ification, auth_skipped, auth_offset, auth_len, mflowtype, mflowid, vrf_id, port); } if (*stcb == NULL) { /* still no TCB... must be bad cookie-echo */ return (NULL); } if (*netp != NULL) { (*netp)->flowtype = mflowtype; (*netp)->flowid = mflowid; } /* * Ok, we built an association so confirm the address we sent the * INIT-ACK to. */ netl = sctp_findnet(*stcb, to); /* * This code should in theory NOT run but */ if (netl == NULL) { /* TSNH! Huh, why do I need to add this address here? */ if (sctp_add_remote_addr(*stcb, to, NULL, SCTP_DONOT_SETSCOPE, SCTP_IN_COOKIE_PROC)) { return (NULL); } netl = sctp_findnet(*stcb, to); } if (netl) { if (netl->dest_state & SCTP_ADDR_UNCONFIRMED) { netl->dest_state &= ~SCTP_ADDR_UNCONFIRMED; (void)sctp_set_primary_addr((*stcb), (struct sockaddr *)NULL, netl); send_int_conf = 1; } } sctp_start_net_timers(*stcb); if ((*inp_p)->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) { if (!had_a_existing_tcb || (((*inp_p)->sctp_flags & SCTP_PCB_FLAGS_CONNECTED) == 0)) { /* * If we have a NEW cookie or the connect never * reached the connected state during collision we * must do the TCP accept thing. */ struct socket *so, *oso; struct sctp_inpcb *inp; if (notification == SCTP_NOTIFY_ASSOC_RESTART) { /* * For a restart we will keep the same * socket, no need to do anything. I THINK!! */ sctp_ulp_notify(notification, *stcb, 0, NULL, SCTP_SO_NOT_LOCKED); if (send_int_conf) { sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, (*stcb), 0, (void *)netl, SCTP_SO_NOT_LOCKED); } return (m); } oso = (*inp_p)->sctp_socket; atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); CURVNET_SET(oso->so_vnet); so = sonewconn(oso, 0 ); CURVNET_RESTORE(); SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); if (so == NULL) { struct mbuf *op_err; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *pcb_so; #endif /* Too many sockets */ SCTPDBG(SCTP_DEBUG_INPUT1, "process_cookie_new: no room for another socket!\n"); op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(*inp_p, NULL, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) pcb_so = SCTP_INP_SO(*inp_p); atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); SCTP_SOCKET_LOCK(pcb_so, 1); SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); #endif (void)sctp_free_assoc(*inp_p, *stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_23); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(pcb_so, 1); #endif return (NULL); } inp = (struct sctp_inpcb *)so->so_pcb; SCTP_INP_INCR_REF(inp); /* * We add the unbound flag here so that if we get an * soabort() before we get the move_pcb done, we * will properly cleanup. */ inp->sctp_flags = (SCTP_PCB_FLAGS_TCPTYPE | SCTP_PCB_FLAGS_CONNECTED | SCTP_PCB_FLAGS_IN_TCPPOOL | SCTP_PCB_FLAGS_UNBOUND | (SCTP_PCB_COPY_FLAGS & (*inp_p)->sctp_flags) | SCTP_PCB_FLAGS_DONT_WAKE); inp->sctp_features = (*inp_p)->sctp_features; inp->sctp_mobility_features = (*inp_p)->sctp_mobility_features; inp->sctp_socket = so; inp->sctp_frag_point = (*inp_p)->sctp_frag_point; inp->max_cwnd = (*inp_p)->max_cwnd; inp->sctp_cmt_on_off = (*inp_p)->sctp_cmt_on_off; inp->ecn_supported = (*inp_p)->ecn_supported; inp->prsctp_supported = (*inp_p)->prsctp_supported; inp->auth_supported = (*inp_p)->auth_supported; inp->asconf_supported = (*inp_p)->asconf_supported; inp->reconfig_supported = (*inp_p)->reconfig_supported; inp->nrsack_supported = (*inp_p)->nrsack_supported; inp->pktdrop_supported = (*inp_p)->pktdrop_supported; inp->partial_delivery_point = (*inp_p)->partial_delivery_point; inp->sctp_context = (*inp_p)->sctp_context; inp->local_strreset_support = (*inp_p)->local_strreset_support; inp->fibnum = (*inp_p)->fibnum; inp->inp_starting_point_for_iterator = NULL; /* * copy in the authentication parameters from the * original endpoint */ if (inp->sctp_ep.local_hmacs) sctp_free_hmaclist(inp->sctp_ep.local_hmacs); inp->sctp_ep.local_hmacs = sctp_copy_hmaclist((*inp_p)->sctp_ep.local_hmacs); if (inp->sctp_ep.local_auth_chunks) sctp_free_chunklist(inp->sctp_ep.local_auth_chunks); inp->sctp_ep.local_auth_chunks = sctp_copy_chunklist((*inp_p)->sctp_ep.local_auth_chunks); /* * Now we must move it from one hash table to * another and get the tcb in the right place. */ /* * This is where the one-2-one socket is put into * the accept state waiting for the accept! */ if (*stcb) { (*stcb)->asoc.state |= SCTP_STATE_IN_ACCEPT_QUEUE; } sctp_move_pcb_and_assoc(*inp_p, inp, *stcb); atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); sctp_pull_off_control_to_new_inp((*inp_p), inp, *stcb, 0); SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); /* * now we must check to see if we were aborted while * the move was going on and the lock/unlock * happened. */ if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* * yep it was, we leave the assoc attached * to the socket since the sctp_inpcb_free() * call will send an abort for us. */ SCTP_INP_DECR_REF(inp); return (NULL); } SCTP_INP_DECR_REF(inp); /* Switch over to the new guy */ *inp_p = inp; sctp_ulp_notify(notification, *stcb, 0, NULL, SCTP_SO_NOT_LOCKED); if (send_int_conf) { sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, (*stcb), 0, (void *)netl, SCTP_SO_NOT_LOCKED); } /* * Pull it from the incomplete queue and wake the * guy */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) atomic_add_int(&(*stcb)->asoc.refcnt, 1); SCTP_TCB_UNLOCK((*stcb)); SCTP_SOCKET_LOCK(so, 1); #endif soisconnected(so); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_TCB_LOCK((*stcb)); atomic_subtract_int(&(*stcb)->asoc.refcnt, 1); SCTP_SOCKET_UNLOCK(so, 1); #endif return (m); } } if (notification) { sctp_ulp_notify(notification, *stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } if (send_int_conf) { sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_CONFIRMED, (*stcb), 0, (void *)netl, SCTP_SO_NOT_LOCKED); } return (m); } static void sctp_handle_cookie_ack(struct sctp_cookie_ack_chunk *cp SCTP_UNUSED, struct sctp_tcb *stcb, struct sctp_nets *net) { /* cp must not be used, others call this without a c-ack :-) */ struct sctp_association *asoc; SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_cookie_ack: handling COOKIE-ACK\n"); if ((stcb == NULL) || (net == NULL)) { return; } asoc = &stcb->asoc; sctp_stop_all_cookie_timers(stcb); /* process according to association state */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) { /* state change only needed when I am in right state */ SCTPDBG(SCTP_DEBUG_INPUT2, "moving to OPEN state\n"); SCTP_SET_STATE(asoc, SCTP_STATE_OPEN); sctp_start_net_timers(stcb); if (asoc->state & SCTP_STATE_SHUTDOWN_PENDING) { sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } /* update RTO */ SCTP_STAT_INCR_COUNTER32(sctps_activeestab); SCTP_STAT_INCR_GAUGE32(sctps_currestab); if (asoc->overall_error_count == 0) { net->RTO = sctp_calculate_rto(stcb, asoc, net, &asoc->time_entered, sctp_align_safe_nocopy, SCTP_RTT_FROM_NON_DATA); } (void)SCTP_GETTIME_TIMEVAL(&asoc->time_entered); sctp_ulp_notify(SCTP_NOTIFY_ASSOC_UP, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); if ((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif if ((stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) == 0) { soisconnected(stcb->sctp_socket); } #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } /* * since we did not send a HB make sure we don't double * things */ net->hb_responded = 1; if (stcb->asoc.state & SCTP_STATE_CLOSED_SOCKET) { /* * We don't need to do the asconf thing, nor hb or * autoclose if the socket is closed. */ goto closed_socket; } sctp_timer_start(SCTP_TIMER_TYPE_HEARTBEAT, stcb->sctp_ep, stcb, net); if (stcb->asoc.sctp_autoclose_ticks && sctp_is_feature_on(stcb->sctp_ep, SCTP_PCB_FLAGS_AUTOCLOSE)) { sctp_timer_start(SCTP_TIMER_TYPE_AUTOCLOSE, stcb->sctp_ep, stcb, NULL); } /* * send ASCONF if parameters are pending and ASCONFs are * allowed (eg. addresses changed when init/cookie echo were * in flight) */ if ((sctp_is_feature_on(stcb->sctp_ep, SCTP_PCB_FLAGS_DO_ASCONF)) && (stcb->asoc.asconf_supported == 1) && (!TAILQ_EMPTY(&stcb->asoc.asconf_queue))) { #ifdef SCTP_TIMER_BASED_ASCONF sctp_timer_start(SCTP_TIMER_TYPE_ASCONF, stcb->sctp_ep, stcb, stcb->asoc.primary_destination); #else sctp_send_asconf(stcb, stcb->asoc.primary_destination, SCTP_ADDR_NOT_LOCKED); #endif } } closed_socket: /* Toss the cookie if I can */ sctp_toss_old_cookies(stcb, asoc); if (!TAILQ_EMPTY(&asoc->sent_queue)) { /* Restart the timer if we have pending data */ struct sctp_tmit_chunk *chk; chk = TAILQ_FIRST(&asoc->sent_queue); sctp_timer_start(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, chk->whoTo); } } static void sctp_handle_ecn_echo(struct sctp_ecne_chunk *cp, struct sctp_tcb *stcb) { struct sctp_nets *net; struct sctp_tmit_chunk *lchk; struct sctp_ecne_chunk bkup; uint8_t override_bit; uint32_t tsn, window_data_tsn; int len; unsigned int pkt_cnt; len = ntohs(cp->ch.chunk_length); if ((len != sizeof(struct sctp_ecne_chunk)) && (len != sizeof(struct old_sctp_ecne_chunk))) { return; } if (len == sizeof(struct old_sctp_ecne_chunk)) { /* Its the old format */ memcpy(&bkup, cp, sizeof(struct old_sctp_ecne_chunk)); bkup.num_pkts_since_cwr = htonl(1); cp = &bkup; } SCTP_STAT_INCR(sctps_recvecne); tsn = ntohl(cp->tsn); pkt_cnt = ntohl(cp->num_pkts_since_cwr); lchk = TAILQ_LAST(&stcb->asoc.send_queue, sctpchunk_listhead); if (lchk == NULL) { window_data_tsn = stcb->asoc.sending_seq - 1; } else { window_data_tsn = lchk->rec.data.TSN_seq; } /* Find where it was sent to if possible. */ net = NULL; TAILQ_FOREACH(lchk, &stcb->asoc.sent_queue, sctp_next) { if (lchk->rec.data.TSN_seq == tsn) { net = lchk->whoTo; net->ecn_prev_cwnd = lchk->rec.data.cwnd_at_send; break; } if (SCTP_TSN_GT(lchk->rec.data.TSN_seq, tsn)) { break; } } if (net == NULL) { /* * What to do. A previous send of a CWR was possibly lost. * See how old it is, we may have it marked on the actual * net. */ TAILQ_FOREACH(net, &stcb->asoc.nets, sctp_next) { if (tsn == net->last_cwr_tsn) { /* Found him, send it off */ break; } } if (net == NULL) { /* * If we reach here, we need to send a special CWR * that says hey, we did this a long time ago and * you lost the response. */ net = TAILQ_FIRST(&stcb->asoc.nets); if (net == NULL) { /* TSNH */ return; } override_bit = SCTP_CWR_REDUCE_OVERRIDE; } else { override_bit = 0; } } else { override_bit = 0; } if (SCTP_TSN_GT(tsn, net->cwr_window_tsn) && ((override_bit & SCTP_CWR_REDUCE_OVERRIDE) == 0)) { /* * JRS - Use the congestion control given in the pluggable * CC module */ stcb->asoc.cc_functions.sctp_cwnd_update_after_ecn_echo(stcb, net, 0, pkt_cnt); /* * We reduce once every RTT. So we will only lower cwnd at * the next sending seq i.e. the window_data_tsn */ net->cwr_window_tsn = window_data_tsn; net->ecn_ce_pkt_cnt += pkt_cnt; net->lost_cnt = pkt_cnt; net->last_cwr_tsn = tsn; } else { override_bit |= SCTP_CWR_IN_SAME_WINDOW; if (SCTP_TSN_GT(tsn, net->last_cwr_tsn) && ((override_bit & SCTP_CWR_REDUCE_OVERRIDE) == 0)) { /* * Another loss in the same window update how many * marks/packets lost we have had. */ int cnt = 1; if (pkt_cnt > net->lost_cnt) { /* Should be the case */ cnt = (pkt_cnt - net->lost_cnt); net->ecn_ce_pkt_cnt += cnt; } net->lost_cnt = pkt_cnt; net->last_cwr_tsn = tsn; /* * Most CC functions will ignore this call, since we * are in-window yet of the initial CE the peer saw. */ stcb->asoc.cc_functions.sctp_cwnd_update_after_ecn_echo(stcb, net, 1, cnt); } } /* * We always send a CWR this way if our previous one was lost our * peer will get an update, or if it is not time again to reduce we * still get the cwr to the peer. Note we set the override when we * could not find the TSN on the chunk or the destination network. */ sctp_send_cwr(stcb, net, net->last_cwr_tsn, override_bit); } static void sctp_handle_ecn_cwr(struct sctp_cwr_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net) { /* * Here we get a CWR from the peer. We must look in the outqueue and * make sure that we have a covered ECNE in the control chunk part. * If so remove it. */ struct sctp_tmit_chunk *chk; struct sctp_ecne_chunk *ecne; int override; uint32_t cwr_tsn; cwr_tsn = ntohl(cp->tsn); override = cp->ch.chunk_flags & SCTP_CWR_REDUCE_OVERRIDE; TAILQ_FOREACH(chk, &stcb->asoc.control_send_queue, sctp_next) { if (chk->rec.chunk_id.id != SCTP_ECN_ECHO) { continue; } if ((override == 0) && (chk->whoTo != net)) { /* Must be from the right src unless override is set */ continue; } ecne = mtod(chk->data, struct sctp_ecne_chunk *); if (SCTP_TSN_GE(cwr_tsn, ntohl(ecne->tsn))) { /* this covers this ECNE, we can remove it */ stcb->asoc.ecn_echo_cnt_onq--; TAILQ_REMOVE(&stcb->asoc.control_send_queue, chk, sctp_next); sctp_m_freem(chk->data); chk->data = NULL; stcb->asoc.ctrl_queue_cnt--; sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); if (override == 0) { break; } } } } static void sctp_handle_shutdown_complete(struct sctp_shutdown_complete_chunk *cp SCTP_UNUSED, struct sctp_tcb *stcb, struct sctp_nets *net) { struct sctp_association *asoc; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_complete: handling SHUTDOWN-COMPLETE\n"); if (stcb == NULL) return; asoc = &stcb->asoc; /* process according to association state */ if (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT) { /* unexpected SHUTDOWN-COMPLETE... so ignore... */ SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_complete: not in SCTP_STATE_SHUTDOWN_ACK_SENT --- ignore\n"); SCTP_TCB_UNLOCK(stcb); return; } /* notify upper layer protocol */ if (stcb->sctp_socket) { sctp_ulp_notify(SCTP_NOTIFY_ASSOC_DOWN, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } #ifdef INVARIANTS if (!TAILQ_EMPTY(&asoc->send_queue) || !TAILQ_EMPTY(&asoc->sent_queue) || !stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { panic("Queues are not empty when handling SHUTDOWN-COMPLETE"); } #endif /* stop the timer */ sctp_timer_stop(SCTP_TIMER_TYPE_SHUTDOWNACK, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_24); SCTP_STAT_INCR_COUNTER32(sctps_shutdown); /* free the TCB */ SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_handle_shutdown_complete: calls free-asoc\n"); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(stcb->sctp_ep); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(stcb->sctp_ep, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_25); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif return; } static int process_chunk_drop(struct sctp_tcb *stcb, struct sctp_chunk_desc *desc, struct sctp_nets *net, uint8_t flg) { switch (desc->chunk_type) { case SCTP_DATA: /* find the tsn to resend (possibly */ { uint32_t tsn; struct sctp_tmit_chunk *tp1; tsn = ntohl(desc->tsn_ifany); TAILQ_FOREACH(tp1, &stcb->asoc.sent_queue, sctp_next) { if (tp1->rec.data.TSN_seq == tsn) { /* found it */ break; } if (SCTP_TSN_GT(tp1->rec.data.TSN_seq, tsn)) { /* not found */ tp1 = NULL; break; } } if (tp1 == NULL) { /* * Do it the other way , aka without paying * attention to queue seq order. */ SCTP_STAT_INCR(sctps_pdrpdnfnd); TAILQ_FOREACH(tp1, &stcb->asoc.sent_queue, sctp_next) { if (tp1->rec.data.TSN_seq == tsn) { /* found it */ break; } } } if (tp1 == NULL) { SCTP_STAT_INCR(sctps_pdrptsnnf); } if ((tp1) && (tp1->sent < SCTP_DATAGRAM_ACKED)) { uint8_t *ddp; if (((flg & SCTP_BADCRC) == 0) && ((flg & SCTP_FROM_MIDDLE_BOX) == 0)) { return (0); } if ((stcb->asoc.peers_rwnd == 0) && ((flg & SCTP_FROM_MIDDLE_BOX) == 0)) { SCTP_STAT_INCR(sctps_pdrpdiwnp); return (0); } if (stcb->asoc.peers_rwnd == 0 && (flg & SCTP_FROM_MIDDLE_BOX)) { SCTP_STAT_INCR(sctps_pdrpdizrw); return (0); } ddp = (uint8_t *) (mtod(tp1->data, caddr_t)+ sizeof(struct sctp_data_chunk)); { unsigned int iii; for (iii = 0; iii < sizeof(desc->data_bytes); iii++) { if (ddp[iii] != desc->data_bytes[iii]) { SCTP_STAT_INCR(sctps_pdrpbadd); return (-1); } } } if (tp1->do_rtt) { /* * this guy had a RTO calculation * pending on it, cancel it */ if (tp1->whoTo->rto_needed == 0) { tp1->whoTo->rto_needed = 1; } tp1->do_rtt = 0; } SCTP_STAT_INCR(sctps_pdrpmark); if (tp1->sent != SCTP_DATAGRAM_RESEND) sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); /* * mark it as if we were doing a FR, since * we will be getting gap ack reports behind * the info from the router. */ tp1->rec.data.doing_fast_retransmit = 1; /* * mark the tsn with what sequences can * cause a new FR. */ if (TAILQ_EMPTY(&stcb->asoc.send_queue)) { tp1->rec.data.fast_retran_tsn = stcb->asoc.sending_seq; } else { tp1->rec.data.fast_retran_tsn = (TAILQ_FIRST(&stcb->asoc.send_queue))->rec.data.TSN_seq; } /* restart the timer */ sctp_timer_stop(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, tp1->whoTo, SCTP_FROM_SCTP_INPUT + SCTP_LOC_26); sctp_timer_start(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, tp1->whoTo); /* fix counts and things */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_FLIGHT_LOGGING_ENABLE) { sctp_misc_ints(SCTP_FLIGHT_LOG_DOWN_PDRP, tp1->whoTo->flight_size, tp1->book_size, (uintptr_t) stcb, tp1->rec.data.TSN_seq); } if (tp1->sent < SCTP_DATAGRAM_RESEND) { sctp_flight_size_decrease(tp1); sctp_total_flight_decrease(stcb, tp1); } tp1->sent = SCTP_DATAGRAM_RESEND; } { /* audit code */ unsigned int audit; audit = 0; TAILQ_FOREACH(tp1, &stcb->asoc.sent_queue, sctp_next) { if (tp1->sent == SCTP_DATAGRAM_RESEND) audit++; } TAILQ_FOREACH(tp1, &stcb->asoc.control_send_queue, sctp_next) { if (tp1->sent == SCTP_DATAGRAM_RESEND) audit++; } if (audit != stcb->asoc.sent_queue_retran_cnt) { SCTP_PRINTF("**Local Audit finds cnt:%d asoc cnt:%d\n", audit, stcb->asoc.sent_queue_retran_cnt); #ifndef SCTP_AUDITING_ENABLED stcb->asoc.sent_queue_retran_cnt = audit; #endif } } } break; case SCTP_ASCONF: { struct sctp_tmit_chunk *asconf; TAILQ_FOREACH(asconf, &stcb->asoc.control_send_queue, sctp_next) { if (asconf->rec.chunk_id.id == SCTP_ASCONF) { break; } } if (asconf) { if (asconf->sent != SCTP_DATAGRAM_RESEND) sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); asconf->sent = SCTP_DATAGRAM_RESEND; asconf->snd_count--; } } break; case SCTP_INITIATION: /* resend the INIT */ stcb->asoc.dropped_special_cnt++; if (stcb->asoc.dropped_special_cnt < SCTP_RETRY_DROPPED_THRESH) { /* * If we can get it in, in a few attempts we do * this, otherwise we let the timer fire. */ sctp_timer_stop(SCTP_TIMER_TYPE_INIT, stcb->sctp_ep, stcb, net, SCTP_FROM_SCTP_INPUT + SCTP_LOC_27); sctp_send_initiate(stcb->sctp_ep, stcb, SCTP_SO_NOT_LOCKED); } break; case SCTP_SELECTIVE_ACK: case SCTP_NR_SELECTIVE_ACK: /* resend the sack */ sctp_send_sack(stcb, SCTP_SO_NOT_LOCKED); break; case SCTP_HEARTBEAT_REQUEST: /* resend a demand HB */ if ((stcb->asoc.overall_error_count + 3) < stcb->asoc.max_send_times) { /* * Only retransmit if we KNOW we wont destroy the * tcb */ sctp_send_hb(stcb, net, SCTP_SO_NOT_LOCKED); } break; case SCTP_SHUTDOWN: sctp_send_shutdown(stcb, net); break; case SCTP_SHUTDOWN_ACK: sctp_send_shutdown_ack(stcb, net); break; case SCTP_COOKIE_ECHO: { struct sctp_tmit_chunk *cookie; cookie = NULL; TAILQ_FOREACH(cookie, &stcb->asoc.control_send_queue, sctp_next) { if (cookie->rec.chunk_id.id == SCTP_COOKIE_ECHO) { break; } } if (cookie) { if (cookie->sent != SCTP_DATAGRAM_RESEND) sctp_ucount_incr(stcb->asoc.sent_queue_retran_cnt); cookie->sent = SCTP_DATAGRAM_RESEND; sctp_stop_all_cookie_timers(stcb); } } break; case SCTP_COOKIE_ACK: sctp_send_cookie_ack(stcb); break; case SCTP_ASCONF_ACK: /* resend last asconf ack */ sctp_send_asconf_ack(stcb); break; case SCTP_FORWARD_CUM_TSN: send_forward_tsn(stcb, &stcb->asoc); break; /* can't do anything with these */ case SCTP_PACKET_DROPPED: case SCTP_INITIATION_ACK: /* this should not happen */ case SCTP_HEARTBEAT_ACK: case SCTP_ABORT_ASSOCIATION: case SCTP_OPERATION_ERROR: case SCTP_SHUTDOWN_COMPLETE: case SCTP_ECN_ECHO: case SCTP_ECN_CWR: default: break; } return (0); } void sctp_reset_in_stream(struct sctp_tcb *stcb, uint32_t number_entries, uint16_t * list) { uint32_t i; uint16_t temp; /* * We set things to 0xffff since this is the last delivered sequence * and we will be sending in 0 after the reset. */ if (number_entries) { for (i = 0; i < number_entries; i++) { temp = ntohs(list[i]); if (temp >= stcb->asoc.streamincnt) { continue; } stcb->asoc.strmin[temp].last_sequence_delivered = 0xffff; } } else { list = NULL; for (i = 0; i < stcb->asoc.streamincnt; i++) { stcb->asoc.strmin[i].last_sequence_delivered = 0xffff; } } sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_RECV, stcb, number_entries, (void *)list, SCTP_SO_NOT_LOCKED); } static void sctp_reset_out_streams(struct sctp_tcb *stcb, uint32_t number_entries, uint16_t * list) { uint32_t i; uint16_t temp; if (number_entries > 0) { for (i = 0; i < number_entries; i++) { temp = ntohs(list[i]); if (temp >= stcb->asoc.streamoutcnt) { /* no such stream */ continue; } stcb->asoc.strmout[temp].next_sequence_send = 0; } } else { for (i = 0; i < stcb->asoc.streamoutcnt; i++) { stcb->asoc.strmout[i].next_sequence_send = 0; } } sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_SEND, stcb, number_entries, (void *)list, SCTP_SO_NOT_LOCKED); } static void sctp_reset_clear_pending(struct sctp_tcb *stcb, uint32_t number_entries, uint16_t * list) { uint32_t i; uint16_t temp; if (number_entries > 0) { for (i = 0; i < number_entries; i++) { temp = ntohs(list[i]); if (temp >= stcb->asoc.streamoutcnt) { /* no such stream */ continue; } stcb->asoc.strmout[temp].state = SCTP_STREAM_OPEN; } } else { for (i = 0; i < stcb->asoc.streamoutcnt; i++) { stcb->asoc.strmout[i].state = SCTP_STREAM_OPEN; } } } struct sctp_stream_reset_request * sctp_find_stream_reset(struct sctp_tcb *stcb, uint32_t seq, struct sctp_tmit_chunk **bchk) { struct sctp_association *asoc; struct sctp_chunkhdr *ch; struct sctp_stream_reset_request *r; struct sctp_tmit_chunk *chk; int len, clen; asoc = &stcb->asoc; if (TAILQ_EMPTY(&stcb->asoc.control_send_queue)) { asoc->stream_reset_outstanding = 0; return (NULL); } if (stcb->asoc.str_reset == NULL) { asoc->stream_reset_outstanding = 0; return (NULL); } chk = stcb->asoc.str_reset; if (chk->data == NULL) { return (NULL); } if (bchk) { /* he wants a copy of the chk pointer */ *bchk = chk; } clen = chk->send_size; ch = mtod(chk->data, struct sctp_chunkhdr *); r = (struct sctp_stream_reset_request *)(ch + 1); if (ntohl(r->request_seq) == seq) { /* found it */ return (r); } len = SCTP_SIZE32(ntohs(r->ph.param_length)); if (clen > (len + (int)sizeof(struct sctp_chunkhdr))) { /* move to the next one, there can only be a max of two */ r = (struct sctp_stream_reset_request *)((caddr_t)r + len); if (ntohl(r->request_seq) == seq) { return (r); } } /* that seq is not here */ return (NULL); } static void sctp_clean_up_stream_reset(struct sctp_tcb *stcb) { struct sctp_association *asoc; struct sctp_tmit_chunk *chk = stcb->asoc.str_reset; if (stcb->asoc.str_reset == NULL) { return; } asoc = &stcb->asoc; sctp_timer_stop(SCTP_TIMER_TYPE_STRRESET, stcb->sctp_ep, stcb, chk->whoTo, SCTP_FROM_SCTP_INPUT + SCTP_LOC_28); TAILQ_REMOVE(&asoc->control_send_queue, chk, sctp_next); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } asoc->ctrl_queue_cnt--; sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); /* sa_ignore NO_NULL_CHK */ stcb->asoc.str_reset = NULL; } static int sctp_handle_stream_reset_response(struct sctp_tcb *stcb, uint32_t seq, uint32_t action, struct sctp_stream_reset_response *respin) { uint16_t type; int lparm_len; struct sctp_association *asoc = &stcb->asoc; struct sctp_tmit_chunk *chk; struct sctp_stream_reset_request *req_param; struct sctp_stream_reset_out_request *req_out_param; struct sctp_stream_reset_in_request *req_in_param; uint32_t number_entries; if (asoc->stream_reset_outstanding == 0) { /* duplicate */ return (0); } if (seq == stcb->asoc.str_reset_seq_out) { req_param = sctp_find_stream_reset(stcb, seq, &chk); if (req_param != NULL) { stcb->asoc.str_reset_seq_out++; type = ntohs(req_param->ph.param_type); lparm_len = ntohs(req_param->ph.param_length); if (type == SCTP_STR_RESET_OUT_REQUEST) { int no_clear = 0; req_out_param = (struct sctp_stream_reset_out_request *)req_param; number_entries = (lparm_len - sizeof(struct sctp_stream_reset_out_request)) / sizeof(uint16_t); asoc->stream_reset_out_is_outstanding = 0; if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_PERFORMED) { /* do it */ sctp_reset_out_streams(stcb, number_entries, req_out_param->list_of_streams); } else if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_DENIED_OUT, stcb, number_entries, req_out_param->list_of_streams, SCTP_SO_NOT_LOCKED); } else if (action == SCTP_STREAM_RESET_RESULT_IN_PROGRESS) { /* * Set it up so we don't stop * retransmitting */ asoc->stream_reset_outstanding++; stcb->asoc.str_reset_seq_out--; asoc->stream_reset_out_is_outstanding = 1; no_clear = 1; } else { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_FAILED_OUT, stcb, number_entries, req_out_param->list_of_streams, SCTP_SO_NOT_LOCKED); } if (no_clear == 0) { sctp_reset_clear_pending(stcb, number_entries, req_out_param->list_of_streams); } } else if (type == SCTP_STR_RESET_IN_REQUEST) { req_in_param = (struct sctp_stream_reset_in_request *)req_param; number_entries = (lparm_len - sizeof(struct sctp_stream_reset_in_request)) / sizeof(uint16_t); if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_DENIED_IN, stcb, number_entries, req_in_param->list_of_streams, SCTP_SO_NOT_LOCKED); } else if (action != SCTP_STREAM_RESET_RESULT_PERFORMED) { sctp_ulp_notify(SCTP_NOTIFY_STR_RESET_FAILED_IN, stcb, number_entries, req_in_param->list_of_streams, SCTP_SO_NOT_LOCKED); } } else if (type == SCTP_STR_RESET_ADD_OUT_STREAMS) { /* Ok we now may have more streams */ int num_stream; num_stream = stcb->asoc.strm_pending_add_size; if (num_stream > (stcb->asoc.strm_realoutsize - stcb->asoc.streamoutcnt)) { /* TSNH */ num_stream = stcb->asoc.strm_realoutsize - stcb->asoc.streamoutcnt; } stcb->asoc.strm_pending_add_size = 0; if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_PERFORMED) { /* Put the new streams into effect */ int i; for (i = asoc->streamoutcnt; i < (asoc->streamoutcnt + num_stream); i++) { asoc->strmout[i].state = SCTP_STREAM_OPEN; } asoc->streamoutcnt += num_stream; sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, 0); } else if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_DENIED); } else { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_FAILED); } } else if (type == SCTP_STR_RESET_ADD_IN_STREAMS) { if (asoc->stream_reset_outstanding) asoc->stream_reset_outstanding--; if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_DENIED); } else if (action != SCTP_STREAM_RESET_RESULT_PERFORMED) { sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, SCTP_STREAM_CHANGE_FAILED); } } else if (type == SCTP_STR_RESET_TSN_REQUEST) { /** * a) Adopt the new in tsn. * b) reset the map * c) Adopt the new out-tsn */ struct sctp_stream_reset_response_tsn *resp; struct sctp_forward_tsn_chunk fwdtsn; int abort_flag = 0; if (respin == NULL) { /* huh ? */ return (0); } if (ntohs(respin->ph.param_length) < sizeof(struct sctp_stream_reset_response_tsn)) { return (0); } if (action == SCTP_STREAM_RESET_RESULT_PERFORMED) { resp = (struct sctp_stream_reset_response_tsn *)respin; asoc->stream_reset_outstanding--; fwdtsn.ch.chunk_length = htons(sizeof(struct sctp_forward_tsn_chunk)); fwdtsn.ch.chunk_type = SCTP_FORWARD_CUM_TSN; fwdtsn.new_cumulative_tsn = htonl(ntohl(resp->senders_next_tsn) - 1); sctp_handle_forward_tsn(stcb, &fwdtsn, &abort_flag, NULL, 0); if (abort_flag) { return (1); } stcb->asoc.highest_tsn_inside_map = (ntohl(resp->senders_next_tsn) - 1); if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MAP_LOGGING_ENABLE) { sctp_log_map(0, 7, asoc->highest_tsn_inside_map, SCTP_MAP_SLIDE_RESULT); } stcb->asoc.tsn_last_delivered = stcb->asoc.cumulative_tsn = stcb->asoc.highest_tsn_inside_map; stcb->asoc.mapping_array_base_tsn = ntohl(resp->senders_next_tsn); memset(stcb->asoc.mapping_array, 0, stcb->asoc.mapping_array_size); stcb->asoc.highest_tsn_inside_nr_map = stcb->asoc.highest_tsn_inside_map; memset(stcb->asoc.nr_mapping_array, 0, stcb->asoc.mapping_array_size); stcb->asoc.sending_seq = ntohl(resp->receivers_next_tsn); stcb->asoc.last_acked_seq = stcb->asoc.cumulative_tsn; sctp_reset_out_streams(stcb, 0, (uint16_t *) NULL); sctp_reset_in_stream(stcb, 0, (uint16_t *) NULL); sctp_notify_stream_reset_tsn(stcb, stcb->asoc.sending_seq, (stcb->asoc.mapping_array_base_tsn + 1), 0); } else if (action == SCTP_STREAM_RESET_RESULT_DENIED) { sctp_notify_stream_reset_tsn(stcb, stcb->asoc.sending_seq, (stcb->asoc.mapping_array_base_tsn + 1), SCTP_ASSOC_RESET_DENIED); } else { sctp_notify_stream_reset_tsn(stcb, stcb->asoc.sending_seq, (stcb->asoc.mapping_array_base_tsn + 1), SCTP_ASSOC_RESET_FAILED); } } /* get rid of the request and get the request flags */ if (asoc->stream_reset_outstanding == 0) { sctp_clean_up_stream_reset(stcb); } } } if (asoc->stream_reset_outstanding == 0) { sctp_send_stream_reset_out_if_possible(stcb, SCTP_SO_NOT_LOCKED); } return (0); } static void sctp_handle_str_reset_request_in(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_in_request *req, int trunc) { uint32_t seq; int len, i; int number_entries; uint16_t temp; /* * peer wants me to send a str-reset to him for my outgoing seq's if * seq_in is right. */ struct sctp_association *asoc = &stcb->asoc; seq = ntohl(req->request_seq); if (asoc->str_reset_seq_in == seq) { asoc->last_reset_action[1] = asoc->last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_RESET_STREAM_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (trunc) { /* Can't do it, since they exceeded our buffer size */ asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (stcb->asoc.stream_reset_out_is_outstanding == 0) { len = ntohs(req->ph.param_length); number_entries = ((len - sizeof(struct sctp_stream_reset_in_request)) / sizeof(uint16_t)); if (number_entries) { for (i = 0; i < number_entries; i++) { temp = ntohs(req->list_of_streams[i]); if (temp >= stcb->asoc.streamoutcnt) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; goto bad_boy; } req->list_of_streams[i] = temp; } for (i = 0; i < number_entries; i++) { if (stcb->asoc.strmout[req->list_of_streams[i]].state == SCTP_STREAM_OPEN) { stcb->asoc.strmout[req->list_of_streams[i]].state = SCTP_STREAM_RESET_PENDING; } } } else { /* Its all */ for (i = 0; i < stcb->asoc.streamoutcnt; i++) { if (stcb->asoc.strmout[i].state == SCTP_STREAM_OPEN) stcb->asoc.strmout[i].state = SCTP_STREAM_RESET_PENDING; } } asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; } else { /* Can't do it, since we have sent one out */ asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_ERR_IN_PROGRESS; } bad_boy: sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); asoc->str_reset_seq_in++; } else if (asoc->str_reset_seq_in - 1 == seq) { sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if (asoc->str_reset_seq_in - 2 == seq) { sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } sctp_send_stream_reset_out_if_possible(stcb, SCTP_SO_NOT_LOCKED); } static int sctp_handle_str_reset_request_tsn(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_tsn_request *req) { /* reset all in and out and update the tsn */ /* * A) reset my str-seq's on in and out. B) Select a receive next, * and set cum-ack to it. Also process this selected number as a * fwd-tsn as well. C) set in the response my next sending seq. */ struct sctp_forward_tsn_chunk fwdtsn; struct sctp_association *asoc = &stcb->asoc; int abort_flag = 0; uint32_t seq; seq = ntohl(req->request_seq); if (asoc->str_reset_seq_in == seq) { asoc->last_reset_action[1] = stcb->asoc.last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_CHANGE_ASSOC_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else { fwdtsn.ch.chunk_length = htons(sizeof(struct sctp_forward_tsn_chunk)); fwdtsn.ch.chunk_type = SCTP_FORWARD_CUM_TSN; fwdtsn.ch.chunk_flags = 0; fwdtsn.new_cumulative_tsn = htonl(stcb->asoc.highest_tsn_inside_map + 1); sctp_handle_forward_tsn(stcb, &fwdtsn, &abort_flag, NULL, 0); if (abort_flag) { return (1); } asoc->highest_tsn_inside_map += SCTP_STREAM_RESET_TSN_DELTA; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MAP_LOGGING_ENABLE) { sctp_log_map(0, 10, asoc->highest_tsn_inside_map, SCTP_MAP_SLIDE_RESULT); } asoc->tsn_last_delivered = asoc->cumulative_tsn = asoc->highest_tsn_inside_map; asoc->mapping_array_base_tsn = asoc->highest_tsn_inside_map + 1; memset(asoc->mapping_array, 0, asoc->mapping_array_size); asoc->highest_tsn_inside_nr_map = asoc->highest_tsn_inside_map; memset(asoc->nr_mapping_array, 0, asoc->mapping_array_size); atomic_add_int(&asoc->sending_seq, 1); /* save off historical data for retrans */ asoc->last_sending_seq[1] = asoc->last_sending_seq[0]; asoc->last_sending_seq[0] = asoc->sending_seq; asoc->last_base_tsnsent[1] = asoc->last_base_tsnsent[0]; asoc->last_base_tsnsent[0] = asoc->mapping_array_base_tsn; sctp_reset_out_streams(stcb, 0, (uint16_t *) NULL); sctp_reset_in_stream(stcb, 0, (uint16_t *) NULL); asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; sctp_notify_stream_reset_tsn(stcb, asoc->sending_seq, (asoc->mapping_array_base_tsn + 1), 0); } sctp_add_stream_reset_result_tsn(chk, seq, asoc->last_reset_action[0], asoc->last_sending_seq[0], asoc->last_base_tsnsent[0]); asoc->str_reset_seq_in++; } else if (asoc->str_reset_seq_in - 1 == seq) { sctp_add_stream_reset_result_tsn(chk, seq, asoc->last_reset_action[0], asoc->last_sending_seq[0], asoc->last_base_tsnsent[0]); } else if (asoc->str_reset_seq_in - 2 == seq) { sctp_add_stream_reset_result_tsn(chk, seq, asoc->last_reset_action[1], asoc->last_sending_seq[1], asoc->last_base_tsnsent[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } return (0); } static void sctp_handle_str_reset_request_out(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_out_request *req, int trunc) { uint32_t seq, tsn; int number_entries, len; struct sctp_association *asoc = &stcb->asoc; seq = ntohl(req->request_seq); /* now if its not a duplicate we process it */ if (asoc->str_reset_seq_in == seq) { len = ntohs(req->ph.param_length); number_entries = ((len - sizeof(struct sctp_stream_reset_out_request)) / sizeof(uint16_t)); /* * the sender is resetting, handle the list issue.. we must * a) verify if we can do the reset, if so no problem b) If * we can't do the reset we must copy the request. c) queue * it, and setup the data in processor to trigger it off * when needed and dequeue all the queued data. */ tsn = ntohl(req->send_reset_at_tsn); /* move the reset action back one */ asoc->last_reset_action[1] = asoc->last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_RESET_STREAM_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (trunc) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (SCTP_TSN_GE(asoc->cumulative_tsn, tsn)) { /* we can do it now */ sctp_reset_in_stream(stcb, number_entries, req->list_of_streams); asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; } else { /* * we must queue it up and thus wait for the TSN's * to arrive that are at or before tsn */ struct sctp_stream_reset_list *liste; int siz; siz = sizeof(struct sctp_stream_reset_list) + (number_entries * sizeof(uint16_t)); SCTP_MALLOC(liste, struct sctp_stream_reset_list *, siz, SCTP_M_STRESET); if (liste == NULL) { /* gak out of memory */ asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); return; } liste->seq = seq; liste->tsn = tsn; liste->number_entries = number_entries; memcpy(&liste->list_of_streams, req->list_of_streams, number_entries * sizeof(uint16_t)); TAILQ_INSERT_TAIL(&asoc->resetHead, liste, next_resp); asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_IN_PROGRESS; } sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); asoc->str_reset_seq_in++; } else if ((asoc->str_reset_seq_in - 1) == seq) { /* * one seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if ((asoc->str_reset_seq_in - 2) == seq) { /* * two seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } } static void sctp_handle_str_reset_add_strm(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_add_strm *str_add) { /* * Peer is requesting to add more streams. If its within our * max-streams we will allow it. */ uint32_t num_stream, i; uint32_t seq; struct sctp_association *asoc = &stcb->asoc; struct sctp_queued_to_read *ctl, *nctl; /* Get the number. */ seq = ntohl(str_add->request_seq); num_stream = ntohs(str_add->number_of_streams); /* Now what would be the new total? */ if (asoc->str_reset_seq_in == seq) { num_stream += stcb->asoc.streamincnt; stcb->asoc.last_reset_action[1] = stcb->asoc.last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_CHANGE_ASSOC_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if ((num_stream > stcb->asoc.max_inbound_streams) || (num_stream > 0xffff)) { /* We must reject it they ask for to many */ denied: stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else { /* Ok, we can do that :-) */ struct sctp_stream_in *oldstrm; /* save off the old */ oldstrm = stcb->asoc.strmin; SCTP_MALLOC(stcb->asoc.strmin, struct sctp_stream_in *, (num_stream * sizeof(struct sctp_stream_in)), SCTP_M_STRMI); if (stcb->asoc.strmin == NULL) { stcb->asoc.strmin = oldstrm; goto denied; } /* copy off the old data */ for (i = 0; i < stcb->asoc.streamincnt; i++) { TAILQ_INIT(&stcb->asoc.strmin[i].inqueue); stcb->asoc.strmin[i].stream_no = i; stcb->asoc.strmin[i].last_sequence_delivered = oldstrm[i].last_sequence_delivered; stcb->asoc.strmin[i].delivery_started = oldstrm[i].delivery_started; /* now anything on those queues? */ TAILQ_FOREACH_SAFE(ctl, &oldstrm[i].inqueue, next, nctl) { TAILQ_REMOVE(&oldstrm[i].inqueue, ctl, next); TAILQ_INSERT_TAIL(&stcb->asoc.strmin[i].inqueue, ctl, next); } } /* Init the new streams */ for (i = stcb->asoc.streamincnt; i < num_stream; i++) { TAILQ_INIT(&stcb->asoc.strmin[i].inqueue); stcb->asoc.strmin[i].stream_no = i; stcb->asoc.strmin[i].last_sequence_delivered = 0xffff; stcb->asoc.strmin[i].delivery_started = 0; } SCTP_FREE(oldstrm, SCTP_M_STRMI); /* update the size */ stcb->asoc.streamincnt = num_stream; stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; sctp_notify_stream_reset_add(stcb, stcb->asoc.streamincnt, stcb->asoc.streamoutcnt, 0); } sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); asoc->str_reset_seq_in++; } else if ((asoc->str_reset_seq_in - 1) == seq) { /* * one seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if ((asoc->str_reset_seq_in - 2) == seq) { /* * two seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } } static void sctp_handle_str_reset_add_out_strm(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, struct sctp_stream_reset_add_strm *str_add) { /* * Peer is requesting to add more streams. If its within our * max-streams we will allow it. */ uint16_t num_stream; uint32_t seq; struct sctp_association *asoc = &stcb->asoc; /* Get the number. */ seq = ntohl(str_add->request_seq); num_stream = ntohs(str_add->number_of_streams); /* Now what would be the new total? */ if (asoc->str_reset_seq_in == seq) { stcb->asoc.last_reset_action[1] = stcb->asoc.last_reset_action[0]; if (!(asoc->local_strreset_support & SCTP_ENABLE_CHANGE_ASSOC_REQ)) { asoc->last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } else if (stcb->asoc.stream_reset_outstanding) { /* We must reject it we have something pending */ stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_ERR_IN_PROGRESS; } else { /* Ok, we can do that :-) */ int mychk; mychk = stcb->asoc.streamoutcnt; mychk += num_stream; if (mychk < 0x10000) { stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_PERFORMED; if (sctp_send_str_reset_req(stcb, 0, NULL, 0, 0, 1, num_stream, 0, 1)) { stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } } else { stcb->asoc.last_reset_action[0] = SCTP_STREAM_RESET_RESULT_DENIED; } } sctp_add_stream_reset_result(chk, seq, stcb->asoc.last_reset_action[0]); asoc->str_reset_seq_in++; } else if ((asoc->str_reset_seq_in - 1) == seq) { /* * one seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[0]); } else if ((asoc->str_reset_seq_in - 2) == seq) { /* * two seq back, just echo back last action since my * response was lost. */ sctp_add_stream_reset_result(chk, seq, asoc->last_reset_action[1]); } else { sctp_add_stream_reset_result(chk, seq, SCTP_STREAM_RESET_RESULT_ERR_BAD_SEQNO); } } #ifdef __GNUC__ __attribute__((noinline)) #endif static int sctp_handle_stream_reset(struct sctp_tcb *stcb, struct mbuf *m, int offset, struct sctp_chunkhdr *ch_req) { uint16_t remaining_length, param_len, ptype; struct sctp_paramhdr pstore; uint8_t cstore[SCTP_CHUNK_BUFFER_SIZE]; uint32_t seq = 0; int num_req = 0; int trunc = 0; struct sctp_tmit_chunk *chk; struct sctp_chunkhdr *ch; struct sctp_paramhdr *ph; int ret_code = 0; int num_param = 0; /* now it may be a reset or a reset-response */ remaining_length = ntohs(ch_req->chunk_length) - sizeof(struct sctp_chunkhdr); /* setup for adding the response */ sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { return (ret_code); } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_STREAM_RESET; chk->rec.chunk_id.can_take_data = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->no_fr_allowed = 0; chk->book_size = chk->send_size = sizeof(struct sctp_chunkhdr); chk->book_size_scale = 0; chk->data = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (chk->data == NULL) { strres_nochunk: if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); return (ret_code); } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); /* setup chunk parameters */ chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->whoTo = NULL; ch = mtod(chk->data, struct sctp_chunkhdr *); ch->chunk_type = SCTP_STREAM_RESET; ch->chunk_flags = 0; ch->chunk_length = htons(chk->send_size); SCTP_BUF_LEN(chk->data) = SCTP_SIZE32(chk->send_size); offset += sizeof(struct sctp_chunkhdr); while (remaining_length >= sizeof(struct sctp_paramhdr)) { ph = (struct sctp_paramhdr *)sctp_m_getptr(m, offset, sizeof(pstore), (uint8_t *) & pstore); if (ph == NULL) { /* TSNH */ break; } param_len = ntohs(ph->param_length); if ((param_len > remaining_length) || (param_len < (sizeof(struct sctp_paramhdr) + sizeof(uint32_t)))) { /* bad parameter length */ break; } ph = (struct sctp_paramhdr *)sctp_m_getptr(m, offset, min(param_len, sizeof(cstore)), (uint8_t *) & cstore); if (ph == NULL) { /* TSNH */ break; } ptype = ntohs(ph->param_type); num_param++; if (param_len > sizeof(cstore)) { trunc = 1; } else { trunc = 0; } if (num_param > SCTP_MAX_RESET_PARAMS) { /* hit the max of parameters already sorry.. */ break; } if (ptype == SCTP_STR_RESET_OUT_REQUEST) { struct sctp_stream_reset_out_request *req_out; if (param_len < sizeof(struct sctp_stream_reset_out_request)) { break; } req_out = (struct sctp_stream_reset_out_request *)ph; num_req++; if (stcb->asoc.stream_reset_outstanding) { seq = ntohl(req_out->response_seq); if (seq == stcb->asoc.str_reset_seq_out) { /* implicit ack */ (void)sctp_handle_stream_reset_response(stcb, seq, SCTP_STREAM_RESET_RESULT_PERFORMED, NULL); } } sctp_handle_str_reset_request_out(stcb, chk, req_out, trunc); } else if (ptype == SCTP_STR_RESET_ADD_OUT_STREAMS) { struct sctp_stream_reset_add_strm *str_add; if (param_len < sizeof(struct sctp_stream_reset_add_strm)) { break; } str_add = (struct sctp_stream_reset_add_strm *)ph; num_req++; sctp_handle_str_reset_add_strm(stcb, chk, str_add); } else if (ptype == SCTP_STR_RESET_ADD_IN_STREAMS) { struct sctp_stream_reset_add_strm *str_add; if (param_len < sizeof(struct sctp_stream_reset_add_strm)) { break; } str_add = (struct sctp_stream_reset_add_strm *)ph; num_req++; sctp_handle_str_reset_add_out_strm(stcb, chk, str_add); } else if (ptype == SCTP_STR_RESET_IN_REQUEST) { struct sctp_stream_reset_in_request *req_in; num_req++; req_in = (struct sctp_stream_reset_in_request *)ph; sctp_handle_str_reset_request_in(stcb, chk, req_in, trunc); } else if (ptype == SCTP_STR_RESET_TSN_REQUEST) { struct sctp_stream_reset_tsn_request *req_tsn; num_req++; req_tsn = (struct sctp_stream_reset_tsn_request *)ph; if (sctp_handle_str_reset_request_tsn(stcb, chk, req_tsn)) { ret_code = 1; goto strres_nochunk; } /* no more */ break; } else if (ptype == SCTP_STR_RESET_RESPONSE) { struct sctp_stream_reset_response *resp; uint32_t result; if (param_len < sizeof(struct sctp_stream_reset_response)) { break; } resp = (struct sctp_stream_reset_response *)ph; seq = ntohl(resp->response_seq); result = ntohl(resp->result); if (sctp_handle_stream_reset_response(stcb, seq, result, resp)) { ret_code = 1; goto strres_nochunk; } } else { break; } offset += SCTP_SIZE32(param_len); if (remaining_length >= SCTP_SIZE32(param_len)) { remaining_length -= SCTP_SIZE32(param_len); } else { remaining_length = 0; } } if (num_req == 0) { /* we have no response free the stuff */ goto strres_nochunk; } /* ok we have a chunk to link in */ TAILQ_INSERT_TAIL(&stcb->asoc.control_send_queue, chk, sctp_next); stcb->asoc.ctrl_queue_cnt++; return (ret_code); } /* * Handle a router or endpoints report of a packet loss, there are two ways * to handle this, either we get the whole packet and must disect it * ourselves (possibly with truncation and or corruption) or it is a summary * from a middle box that did the disectting for us. */ static void sctp_handle_packet_dropped(struct sctp_pktdrop_chunk *cp, struct sctp_tcb *stcb, struct sctp_nets *net, uint32_t limit) { uint32_t bottle_bw, on_queue; uint16_t trunc_len; unsigned int chlen; unsigned int at; struct sctp_chunk_desc desc; struct sctp_chunkhdr *ch; chlen = ntohs(cp->ch.chunk_length); chlen -= sizeof(struct sctp_pktdrop_chunk); /* XXX possible chlen underflow */ if (chlen == 0) { ch = NULL; if (cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX) SCTP_STAT_INCR(sctps_pdrpbwrpt); } else { ch = (struct sctp_chunkhdr *)(cp->data + sizeof(struct sctphdr)); chlen -= sizeof(struct sctphdr); /* XXX possible chlen underflow */ memset(&desc, 0, sizeof(desc)); } trunc_len = (uint16_t) ntohs(cp->trunc_len); if (trunc_len > limit) { trunc_len = limit; } /* now the chunks themselves */ while ((ch != NULL) && (chlen >= sizeof(struct sctp_chunkhdr))) { desc.chunk_type = ch->chunk_type; /* get amount we need to move */ at = ntohs(ch->chunk_length); if (at < sizeof(struct sctp_chunkhdr)) { /* corrupt chunk, maybe at the end? */ SCTP_STAT_INCR(sctps_pdrpcrupt); break; } if (trunc_len == 0) { /* we are supposed to have all of it */ if (at > chlen) { /* corrupt skip it */ SCTP_STAT_INCR(sctps_pdrpcrupt); break; } } else { /* is there enough of it left ? */ if (desc.chunk_type == SCTP_DATA) { if (chlen < (sizeof(struct sctp_data_chunk) + sizeof(desc.data_bytes))) { break; } } else { if (chlen < sizeof(struct sctp_chunkhdr)) { break; } } } if (desc.chunk_type == SCTP_DATA) { /* can we get out the tsn? */ if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX)) SCTP_STAT_INCR(sctps_pdrpmbda); if (chlen >= (sizeof(struct sctp_data_chunk) + sizeof(uint32_t))) { /* yep */ struct sctp_data_chunk *dcp; uint8_t *ddp; unsigned int iii; dcp = (struct sctp_data_chunk *)ch; ddp = (uint8_t *) (dcp + 1); for (iii = 0; iii < sizeof(desc.data_bytes); iii++) { desc.data_bytes[iii] = ddp[iii]; } desc.tsn_ifany = dcp->dp.tsn; } else { /* nope we are done. */ SCTP_STAT_INCR(sctps_pdrpnedat); break; } } else { if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX)) SCTP_STAT_INCR(sctps_pdrpmbct); } if (process_chunk_drop(stcb, &desc, net, cp->ch.chunk_flags)) { SCTP_STAT_INCR(sctps_pdrppdbrk); break; } if (SCTP_SIZE32(at) > chlen) { break; } chlen -= SCTP_SIZE32(at); if (chlen < sizeof(struct sctp_chunkhdr)) { /* done, none left */ break; } ch = (struct sctp_chunkhdr *)((caddr_t)ch + SCTP_SIZE32(at)); } /* Now update any rwnd --- possibly */ if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX) == 0) { /* From a peer, we get a rwnd report */ uint32_t a_rwnd; SCTP_STAT_INCR(sctps_pdrpfehos); bottle_bw = ntohl(cp->bottle_bw); on_queue = ntohl(cp->current_onq); if (bottle_bw && on_queue) { /* a rwnd report is in here */ if (bottle_bw > on_queue) a_rwnd = bottle_bw - on_queue; else a_rwnd = 0; if (a_rwnd == 0) stcb->asoc.peers_rwnd = 0; else { if (a_rwnd > stcb->asoc.total_flight) { stcb->asoc.peers_rwnd = a_rwnd - stcb->asoc.total_flight; } else { stcb->asoc.peers_rwnd = 0; } if (stcb->asoc.peers_rwnd < stcb->sctp_ep->sctp_ep.sctp_sws_sender) { /* SWS sender side engages */ stcb->asoc.peers_rwnd = 0; } } } } else { SCTP_STAT_INCR(sctps_pdrpfmbox); } /* now middle boxes in sat networks get a cwnd bump */ if ((cp->ch.chunk_flags & SCTP_FROM_MIDDLE_BOX) && (stcb->asoc.sat_t3_loss_recovery == 0) && (stcb->asoc.sat_network)) { /* * This is debateable but for sat networks it makes sense * Note if a T3 timer has went off, we will prohibit any * changes to cwnd until we exit the t3 loss recovery. */ stcb->asoc.cc_functions.sctp_cwnd_update_after_packet_dropped(stcb, net, cp, &bottle_bw, &on_queue); } } /* * handles all control chunks in a packet inputs: - m: mbuf chain, assumed to * still contain IP/SCTP header - stcb: is the tcb found for this packet - * offset: offset into the mbuf chain to first chunkhdr - length: is the * length of the complete packet outputs: - length: modified to remaining * length after control processing - netp: modified to new sctp_nets after * cookie-echo processing - return NULL to discard the packet (ie. no asoc, * bad packet,...) otherwise return the tcb for this packet */ #ifdef __GNUC__ __attribute__((noinline)) #endif static struct sctp_tcb * sctp_process_control(struct mbuf *m, int iphlen, int *offset, int length, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_chunkhdr *ch, struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_nets **netp, int *fwd_tsn_seen, uint8_t mflowtype, uint32_t mflowid, uint16_t fibnum, uint32_t vrf_id, uint16_t port) { struct sctp_association *asoc; struct mbuf *op_err; char msg[SCTP_DIAG_INFO_LEN]; uint32_t vtag_in; int num_chunks = 0; /* number of control chunks processed */ uint32_t chk_length; int ret; int abort_no_unlock = 0; int ecne_seen = 0; /* * How big should this be, and should it be alloc'd? Lets try the * d-mtu-ceiling for now (2k) and that should hopefully work ... * until we get into jumbo grams and such.. */ uint8_t chunk_buf[SCTP_CHUNK_BUFFER_SIZE]; struct sctp_tcb *locked_tcb = stcb; int got_auth = 0; uint32_t auth_offset = 0, auth_len = 0; int auth_skipped = 0; int asconf_cnt = 0; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so; #endif SCTPDBG(SCTP_DEBUG_INPUT1, "sctp_process_control: iphlen=%u, offset=%u, length=%u stcb:%p\n", iphlen, *offset, length, (void *)stcb); /* validate chunk header length... */ if (ntohs(ch->chunk_length) < sizeof(*ch)) { SCTPDBG(SCTP_DEBUG_INPUT1, "Invalid header length %d\n", ntohs(ch->chunk_length)); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } /* * validate the verification tag */ vtag_in = ntohl(sh->v_tag); if (locked_tcb) { SCTP_TCB_LOCK_ASSERT(locked_tcb); } if (ch->chunk_type == SCTP_INITIATION) { SCTPDBG(SCTP_DEBUG_INPUT1, "Its an INIT of len:%d vtag:%x\n", ntohs(ch->chunk_length), vtag_in); if (vtag_in != 0) { /* protocol error- silently discard... */ SCTP_STAT_INCR(sctps_badvtag); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else if (ch->chunk_type != SCTP_COOKIE_ECHO) { /* * If there is no stcb, skip the AUTH chunk and process * later after a stcb is found (to validate the lookup was * valid. */ if ((ch->chunk_type == SCTP_AUTHENTICATION) && (stcb == NULL) && (inp->auth_supported == 1)) { /* save this chunk for later processing */ auth_skipped = 1; auth_offset = *offset; auth_len = ntohs(ch->chunk_length); /* (temporarily) move past this chunk */ *offset += SCTP_SIZE32(auth_len); if (*offset >= length) { /* no more data left in the mbuf chain */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, sizeof(struct sctp_chunkhdr), chunk_buf); } if (ch == NULL) { /* Help */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if (ch->chunk_type == SCTP_COOKIE_ECHO) { goto process_control_chunks; } /* * first check if it's an ASCONF with an unknown src addr we * need to look inside to find the association */ if (ch->chunk_type == SCTP_ASCONF && stcb == NULL) { struct sctp_chunkhdr *asconf_ch = ch; uint32_t asconf_offset = 0, asconf_len = 0; /* inp's refcount may be reduced */ SCTP_INP_INCR_REF(inp); asconf_offset = *offset; do { asconf_len = ntohs(asconf_ch->chunk_length); if (asconf_len < sizeof(struct sctp_asconf_paramhdr)) break; stcb = sctp_findassociation_ep_asconf(m, *offset, dst, sh, &inp, netp, vrf_id); if (stcb != NULL) break; asconf_offset += SCTP_SIZE32(asconf_len); asconf_ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, asconf_offset, sizeof(struct sctp_chunkhdr), chunk_buf); } while (asconf_ch != NULL && asconf_ch->chunk_type == SCTP_ASCONF); if (stcb == NULL) { /* * reduce inp's refcount if not reduced in * sctp_findassociation_ep_asconf(). */ SCTP_INP_DECR_REF(inp); } else { locked_tcb = stcb; } /* now go back and verify any auth chunk to be sure */ if (auth_skipped && (stcb != NULL)) { struct sctp_auth_chunk *auth; auth = (struct sctp_auth_chunk *) sctp_m_getptr(m, auth_offset, auth_len, chunk_buf); got_auth = 1; auth_skipped = 0; if ((auth == NULL) || sctp_handle_auth(stcb, auth, m, auth_offset)) { /* auth HMAC failed so dump it */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } else { /* remaining chunks are HMAC checked */ stcb->asoc.authenticated = 1; } } } if (stcb == NULL) { snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); /* no association, so it's out of the blue... */ sctp_handle_ootb(m, iphlen, *offset, src, dst, sh, inp, op_err, mflowtype, mflowid, inp->fibnum, vrf_id, port); *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } asoc = &stcb->asoc; /* ABORT and SHUTDOWN can use either v_tag... */ if ((ch->chunk_type == SCTP_ABORT_ASSOCIATION) || (ch->chunk_type == SCTP_SHUTDOWN_COMPLETE) || (ch->chunk_type == SCTP_PACKET_DROPPED)) { /* Take the T-bit always into account. */ if ((((ch->chunk_flags & SCTP_HAD_NO_TCB) == 0) && (vtag_in == asoc->my_vtag)) || (((ch->chunk_flags & SCTP_HAD_NO_TCB) == SCTP_HAD_NO_TCB) && (vtag_in == asoc->peer_vtag))) { /* this is valid */ } else { /* drop this packet... */ SCTP_STAT_INCR(sctps_badvtag); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else if (ch->chunk_type == SCTP_SHUTDOWN_ACK) { if (vtag_in != asoc->my_vtag) { /* * this could be a stale SHUTDOWN-ACK or the * peer never got the SHUTDOWN-COMPLETE and * is still hung; we have started a new asoc * but it won't complete until the shutdown * is completed */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, *offset, src, dst, sh, inp, op_err, mflowtype, mflowid, fibnum, vrf_id, port); return (NULL); } } else { /* for all other chunks, vtag must match */ if (vtag_in != asoc->my_vtag) { /* invalid vtag... */ SCTPDBG(SCTP_DEBUG_INPUT3, "invalid vtag: %xh, expect %xh\n", vtag_in, asoc->my_vtag); SCTP_STAT_INCR(sctps_badvtag); if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } } } /* end if !SCTP_COOKIE_ECHO */ /* * process all control chunks... */ if (((ch->chunk_type == SCTP_SELECTIVE_ACK) || (ch->chunk_type == SCTP_NR_SELECTIVE_ACK) || (ch->chunk_type == SCTP_HEARTBEAT_REQUEST)) && (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_COOKIE_ECHOED)) { /* implied cookie-ack.. we must have lost the ack */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_cookie_ack((struct sctp_cookie_ack_chunk *)ch, stcb, *netp); } process_control_chunks: while (IS_SCTP_CONTROL(ch)) { /* validate chunk length */ chk_length = ntohs(ch->chunk_length); SCTPDBG(SCTP_DEBUG_INPUT2, "sctp_process_control: processing a chunk type=%u, len=%u\n", ch->chunk_type, chk_length); SCTP_LTRACE_CHK(inp, stcb, ch->chunk_type, chk_length); if (chk_length < sizeof(*ch) || (*offset + (int)chk_length) > length) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } SCTP_STAT_INCR_COUNTER64(sctps_incontrolchunks); /* * INIT-ACK only gets the init ack "header" portion only * because we don't have to process the peer's COOKIE. All * others get a complete chunk. */ if ((ch->chunk_type == SCTP_INITIATION_ACK) || (ch->chunk_type == SCTP_INITIATION)) { /* get an init-ack chunk */ ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, sizeof(struct sctp_init_ack_chunk), chunk_buf); if (ch == NULL) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else { /* For cookies and all other chunks. */ if (chk_length > sizeof(chunk_buf)) { /* * use just the size of the chunk buffer so * the front part of our chunks fit in * contiguous space up to the chunk buffer * size (508 bytes). For chunks that need to * get more than that they must use the * sctp_m_getptr() function or other means * (e.g. know how to parse mbuf chains). * Cookies do this already. */ ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, (sizeof(chunk_buf) - 4), chunk_buf); if (ch == NULL) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } else { /* We can fit it all */ ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, chk_length, chunk_buf); if (ch == NULL) { SCTP_PRINTF("sctp_process_control: Can't get the all data....\n"); *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } } } num_chunks++; /* Save off the last place we got a control from */ if (stcb != NULL) { if (((netp != NULL) && (*netp != NULL)) || (ch->chunk_type == SCTP_ASCONF)) { /* * allow last_control to be NULL if * ASCONF... ASCONF processing will find the * right net later */ if ((netp != NULL) && (*netp != NULL)) stcb->asoc.last_control_chunk_from = *netp; } } #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xB0, ch->chunk_type); #endif /* check to see if this chunk required auth, but isn't */ if ((stcb != NULL) && (stcb->asoc.auth_supported == 1) && sctp_auth_is_required_chunk(ch->chunk_type, stcb->asoc.local_auth_chunks) && !stcb->asoc.authenticated) { /* "silently" ignore */ SCTP_STAT_INCR(sctps_recvauthmissing); goto next_chunk; } switch (ch->chunk_type) { case SCTP_INITIATION: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_INIT\n"); /* The INIT chunk must be the only chunk. */ if ((num_chunks > 1) || (length - *offset > (int)SCTP_SIZE32(chk_length))) { /* RFC 4960 requires that no ABORT is sent */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } /* Honor our resource limit. */ if (chk_length > SCTP_LARGEST_INIT_ACCEPTED) { op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); *offset = length; return (NULL); } sctp_handle_init(m, iphlen, *offset, src, dst, sh, (struct sctp_init_chunk *)ch, inp, - stcb, &abort_no_unlock, + stcb, *netp, &abort_no_unlock, mflowtype, mflowid, vrf_id, port); *offset = length; if ((!abort_no_unlock) && (locked_tcb)) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); break; case SCTP_PAD_CHUNK: break; case SCTP_INITIATION_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_INIT-ACK\n"); if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ if ((stcb) && (stcb->asoc.total_output_queue_size)) { ; } else { if (locked_tcb != stcb) { /* Very unlikely */ SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; if (stcb) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(inp); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_29); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif } return (NULL); } } /* The INIT-ACK chunk must be the only chunk. */ if ((num_chunks > 1) || (length - *offset > (int)SCTP_SIZE32(chk_length))) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if ((netp) && (*netp)) { ret = sctp_handle_init_ack(m, iphlen, *offset, src, dst, sh, (struct sctp_init_ack_chunk *)ch, stcb, *netp, &abort_no_unlock, mflowtype, mflowid, vrf_id); } else { ret = -1; } *offset = length; if (abort_no_unlock) { return (NULL); } /* * Special case, I must call the output routine to * get the cookie echoed */ if ((stcb != NULL) && (ret == 0)) { sctp_chunk_output(stcb->sctp_ep, stcb, SCTP_OUTPUT_FROM_CONTROL_PROC, SCTP_SO_NOT_LOCKED); } if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); break; case SCTP_SELECTIVE_ACK: { struct sctp_sack_chunk *sack; int abort_now = 0; uint32_t a_rwnd, cum_ack; uint16_t num_seg, num_dup; uint8_t flags; int offset_seg, offset_dup; SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SACK\n"); SCTP_STAT_INCR(sctps_recvsacks); if (stcb == NULL) { SCTPDBG(SCTP_DEBUG_INDATA1, "No stcb when processing SACK chunk\n"); break; } if (chk_length < sizeof(struct sctp_sack_chunk)) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size on SACK chunk, too small\n"); break; } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) { /*- * If we have sent a shutdown-ack, we will pay no * attention to a sack sent in to us since * we don't care anymore. */ break; } sack = (struct sctp_sack_chunk *)ch; flags = ch->chunk_flags; cum_ack = ntohl(sack->sack.cum_tsn_ack); num_seg = ntohs(sack->sack.num_gap_ack_blks); num_dup = ntohs(sack->sack.num_dup_tsns); a_rwnd = (uint32_t) ntohl(sack->sack.a_rwnd); if (sizeof(struct sctp_sack_chunk) + num_seg * sizeof(struct sctp_gap_ack_block) + num_dup * sizeof(uint32_t) != chk_length) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size of SACK chunk\n"); break; } offset_seg = *offset + sizeof(struct sctp_sack_chunk); offset_dup = offset_seg + num_seg * sizeof(struct sctp_gap_ack_block); SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SACK process cum_ack:%x num_seg:%d a_rwnd:%d\n", cum_ack, num_seg, a_rwnd); stcb->asoc.seen_a_sack_this_pkt = 1; if ((stcb->asoc.pr_sctp_cnt == 0) && (num_seg == 0) && SCTP_TSN_GE(cum_ack, stcb->asoc.last_acked_seq) && (stcb->asoc.saw_sack_with_frags == 0) && (stcb->asoc.saw_sack_with_nr_frags == 0) && (!TAILQ_EMPTY(&stcb->asoc.sent_queue)) ) { /* * We have a SIMPLE sack having no * prior segments and data on sent * queue to be acked.. Use the * faster path sack processing. We * also allow window update sacks * with no missing segments to go * this way too. */ sctp_express_handle_sack(stcb, cum_ack, a_rwnd, &abort_now, ecne_seen); } else { if (netp && *netp) sctp_handle_sack(m, offset_seg, offset_dup, stcb, num_seg, 0, num_dup, &abort_now, flags, cum_ack, a_rwnd, ecne_seen); } if (abort_now) { /* ABORT signal from sack processing */ *offset = length; return (NULL); } if (TAILQ_EMPTY(&stcb->asoc.send_queue) && TAILQ_EMPTY(&stcb->asoc.sent_queue) && (stcb->asoc.stream_queue_cnt == 0)) { sctp_ulp_notify(SCTP_NOTIFY_SENDER_DRY, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } } break; /* * EY - nr_sack: If the received chunk is an * nr_sack chunk */ case SCTP_NR_SELECTIVE_ACK: { struct sctp_nr_sack_chunk *nr_sack; int abort_now = 0; uint32_t a_rwnd, cum_ack; uint16_t num_seg, num_nr_seg, num_dup; uint8_t flags; int offset_seg, offset_dup; SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_NR_SACK\n"); SCTP_STAT_INCR(sctps_recvsacks); if (stcb == NULL) { SCTPDBG(SCTP_DEBUG_INDATA1, "No stcb when processing NR-SACK chunk\n"); break; } if (stcb->asoc.nrsack_supported == 0) { goto unknown_chunk; } if (chk_length < sizeof(struct sctp_nr_sack_chunk)) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size on NR-SACK chunk, too small\n"); break; } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) { /*- * If we have sent a shutdown-ack, we will pay no * attention to a sack sent in to us since * we don't care anymore. */ break; } nr_sack = (struct sctp_nr_sack_chunk *)ch; flags = ch->chunk_flags; cum_ack = ntohl(nr_sack->nr_sack.cum_tsn_ack); num_seg = ntohs(nr_sack->nr_sack.num_gap_ack_blks); num_nr_seg = ntohs(nr_sack->nr_sack.num_nr_gap_ack_blks); num_dup = ntohs(nr_sack->nr_sack.num_dup_tsns); a_rwnd = (uint32_t) ntohl(nr_sack->nr_sack.a_rwnd); if (sizeof(struct sctp_nr_sack_chunk) + (num_seg + num_nr_seg) * sizeof(struct sctp_gap_ack_block) + num_dup * sizeof(uint32_t) != chk_length) { SCTPDBG(SCTP_DEBUG_INDATA1, "Bad size of NR_SACK chunk\n"); break; } offset_seg = *offset + sizeof(struct sctp_nr_sack_chunk); offset_dup = offset_seg + num_seg * sizeof(struct sctp_gap_ack_block); SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_NR_SACK process cum_ack:%x num_seg:%d a_rwnd:%d\n", cum_ack, num_seg, a_rwnd); stcb->asoc.seen_a_sack_this_pkt = 1; if ((stcb->asoc.pr_sctp_cnt == 0) && (num_seg == 0) && (num_nr_seg == 0) && SCTP_TSN_GE(cum_ack, stcb->asoc.last_acked_seq) && (stcb->asoc.saw_sack_with_frags == 0) && (stcb->asoc.saw_sack_with_nr_frags == 0) && (!TAILQ_EMPTY(&stcb->asoc.sent_queue))) { /* * We have a SIMPLE sack having no * prior segments and data on sent * queue to be acked. Use the faster * path sack processing. We also * allow window update sacks with no * missing segments to go this way * too. */ sctp_express_handle_sack(stcb, cum_ack, a_rwnd, &abort_now, ecne_seen); } else { if (netp && *netp) sctp_handle_sack(m, offset_seg, offset_dup, stcb, num_seg, num_nr_seg, num_dup, &abort_now, flags, cum_ack, a_rwnd, ecne_seen); } if (abort_now) { /* ABORT signal from sack processing */ *offset = length; return (NULL); } if (TAILQ_EMPTY(&stcb->asoc.send_queue) && TAILQ_EMPTY(&stcb->asoc.sent_queue) && (stcb->asoc.stream_queue_cnt == 0)) { sctp_ulp_notify(SCTP_NOTIFY_SENDER_DRY, stcb, 0, NULL, SCTP_SO_NOT_LOCKED); } } break; case SCTP_HEARTBEAT_REQUEST: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_HEARTBEAT\n"); if ((stcb) && netp && *netp) { SCTP_STAT_INCR(sctps_recvheartbeat); sctp_send_heartbeat_ack(stcb, m, *offset, chk_length, *netp); /* He's alive so give him credit */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; } break; case SCTP_HEARTBEAT_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_HEARTBEAT-ACK\n"); if ((stcb == NULL) || (chk_length != sizeof(struct sctp_heartbeat_chunk))) { /* Its not ours */ *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } /* He's alive so give him credit */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; SCTP_STAT_INCR(sctps_recvheartbeatack); if (netp && *netp) sctp_handle_heartbeat_ack((struct sctp_heartbeat_chunk *)ch, stcb, *netp); break; case SCTP_ABORT_ASSOCIATION: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ABORT, stcb %p\n", (void *)stcb); if ((stcb) && netp && *netp) sctp_handle_abort((struct sctp_abort_chunk *)ch, stcb, *netp); *offset = length; return (NULL); break; case SCTP_SHUTDOWN: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SHUTDOWN, stcb %p\n", (void *)stcb); if ((stcb == NULL) || (chk_length != sizeof(struct sctp_shutdown_chunk))) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if (netp && *netp) { int abort_flag = 0; sctp_handle_shutdown((struct sctp_shutdown_chunk *)ch, stcb, *netp, &abort_flag); if (abort_flag) { *offset = length; return (NULL); } } break; case SCTP_SHUTDOWN_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SHUTDOWN-ACK, stcb %p\n", (void *)stcb); if ((stcb) && (netp) && (*netp)) sctp_handle_shutdown_ack((struct sctp_shutdown_ack_chunk *)ch, stcb, *netp); *offset = length; return (NULL); break; case SCTP_OPERATION_ERROR: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_OP-ERR\n"); if ((stcb) && netp && *netp && sctp_handle_error(ch, stcb, *netp) < 0) { *offset = length; return (NULL); } break; case SCTP_COOKIE_ECHO: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_COOKIE-ECHO, stcb %p\n", (void *)stcb); if ((stcb) && (stcb->asoc.total_output_queue_size)) { ; } else { if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ abend: if (stcb) { SCTP_TCB_UNLOCK(stcb); } *offset = length; return (NULL); } } /* * First are we accepting? We do this again here * since it is possible that a previous endpoint WAS * listening responded to a INIT-ACK and then * closed. We opened and bound.. and are now no * longer listening. */ if ((stcb == NULL) && (inp->sctp_socket->so_qlen >= inp->sctp_socket->so_qlimit)) { if ((inp->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) && (SCTP_BASE_SYSCTL(sctp_abort_if_one_2_one_hits_limit))) { op_err = sctp_generate_cause(SCTP_CAUSE_OUT_OF_RESC, ""); sctp_abort_association(inp, stcb, m, iphlen, src, dst, sh, op_err, mflowtype, mflowid, vrf_id, port); } *offset = length; return (NULL); } else { struct mbuf *ret_buf; struct sctp_inpcb *linp; if (stcb) { linp = NULL; } else { linp = inp; } if (linp) { SCTP_ASOC_CREATE_LOCK(linp); if ((inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) || (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_ALLGONE)) { SCTP_ASOC_CREATE_UNLOCK(linp); goto abend; } } if (netp) { ret_buf = sctp_handle_cookie_echo(m, iphlen, *offset, src, dst, sh, (struct sctp_cookie_echo_chunk *)ch, &inp, &stcb, netp, auth_skipped, auth_offset, auth_len, &locked_tcb, mflowtype, mflowid, vrf_id, port); } else { ret_buf = NULL; } if (linp) { SCTP_ASOC_CREATE_UNLOCK(linp); } if (ret_buf == NULL) { if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } SCTPDBG(SCTP_DEBUG_INPUT3, "GAK, null buffer\n"); *offset = length; return (NULL); } /* if AUTH skipped, see if it verified... */ if (auth_skipped) { got_auth = 1; auth_skipped = 0; } if (!TAILQ_EMPTY(&stcb->asoc.sent_queue)) { /* * Restart the timer if we have * pending data */ struct sctp_tmit_chunk *chk; chk = TAILQ_FIRST(&stcb->asoc.sent_queue); sctp_timer_start(SCTP_TIMER_TYPE_SEND, stcb->sctp_ep, stcb, chk->whoTo); } } break; case SCTP_COOKIE_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_COOKIE-ACK, stcb %p\n", (void *)stcb); if ((stcb == NULL) || chk_length != sizeof(struct sctp_cookie_ack_chunk)) { if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ if ((stcb) && (stcb->asoc.total_output_queue_size)) { ; } else if (stcb) { #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(inp); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_30); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif *offset = length; return (NULL); } } /* He's alive so give him credit */ if ((stcb) && netp && *netp) { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_cookie_ack((struct sctp_cookie_ack_chunk *)ch, stcb, *netp); } break; case SCTP_ECN_ECHO: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ECN-ECHO\n"); /* He's alive so give him credit */ if ((stcb == NULL) || (chk_length != sizeof(struct sctp_ecne_chunk))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (stcb) { if (stcb->asoc.ecn_supported == 0) { goto unknown_chunk; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_ecn_echo((struct sctp_ecne_chunk *)ch, stcb); ecne_seen = 1; } break; case SCTP_ECN_CWR: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ECN-CWR\n"); /* He's alive so give him credit */ if ((stcb == NULL) || (chk_length != sizeof(struct sctp_cwr_chunk))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (stcb) { if (stcb->asoc.ecn_supported == 0) { goto unknown_chunk; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_ecn_cwr((struct sctp_cwr_chunk *)ch, stcb, *netp); } break; case SCTP_SHUTDOWN_COMPLETE: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_SHUTDOWN-COMPLETE, stcb %p\n", (void *)stcb); /* must be first and only chunk */ if ((num_chunks > 1) || (length - *offset > (int)SCTP_SIZE32(chk_length))) { *offset = length; if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } return (NULL); } if ((stcb) && netp && *netp) { sctp_handle_shutdown_complete((struct sctp_shutdown_complete_chunk *)ch, stcb, *netp); } *offset = length; return (NULL); break; case SCTP_ASCONF: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ASCONF\n"); /* He's alive so give him credit */ if (stcb) { if (stcb->asoc.asconf_supported == 0) { goto unknown_chunk; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_asconf(m, *offset, src, (struct sctp_asconf_chunk *)ch, stcb, asconf_cnt == 0); asconf_cnt++; } break; case SCTP_ASCONF_ACK: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_ASCONF-ACK\n"); if (chk_length < sizeof(struct sctp_asconf_ack_chunk)) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if ((stcb) && netp && *netp) { if (stcb->asoc.asconf_supported == 0) { goto unknown_chunk; } /* He's alive so give him credit */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_asconf_ack(m, *offset, (struct sctp_asconf_ack_chunk *)ch, stcb, *netp, &abort_no_unlock); if (abort_no_unlock) return (NULL); } break; case SCTP_FORWARD_CUM_TSN: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_FWD-TSN\n"); if (chk_length < sizeof(struct sctp_forward_tsn_chunk)) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } /* He's alive so give him credit */ if (stcb) { int abort_flag = 0; if (stcb->asoc.prsctp_supported == 0) { goto unknown_chunk; } stcb->asoc.overall_error_count = 0; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } *fwd_tsn_seen = 1; if (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) { /* We are not interested anymore */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) so = SCTP_INP_SO(inp); atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 1); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); #endif (void)sctp_free_assoc(inp, stcb, SCTP_NORMAL_PROC, SCTP_FROM_SCTP_INPUT + SCTP_LOC_31); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) SCTP_SOCKET_UNLOCK(so, 1); #endif *offset = length; return (NULL); } sctp_handle_forward_tsn(stcb, (struct sctp_forward_tsn_chunk *)ch, &abort_flag, m, *offset); if (abort_flag) { *offset = length; return (NULL); } else { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; } } break; case SCTP_STREAM_RESET: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_STREAM_RESET\n"); if (((stcb == NULL) || (ch == NULL) || (chk_length < sizeof(struct sctp_stream_reset_tsn_req)))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (stcb->asoc.reconfig_supported == 0) { goto unknown_chunk; } if (sctp_handle_stream_reset(stcb, m, *offset, ch)) { /* stop processing */ *offset = length; return (NULL); } break; case SCTP_PACKET_DROPPED: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_PACKET_DROPPED\n"); /* re-get it all please */ if (chk_length < sizeof(struct sctp_pktdrop_chunk)) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (ch && (stcb) && netp && (*netp)) { if (stcb->asoc.pktdrop_supported == 0) { goto unknown_chunk; } sctp_handle_packet_dropped((struct sctp_pktdrop_chunk *)ch, stcb, *netp, min(chk_length, (sizeof(chunk_buf) - 4))); } break; case SCTP_AUTHENTICATION: SCTPDBG(SCTP_DEBUG_INPUT3, "SCTP_AUTHENTICATION\n"); if (stcb == NULL) { /* save the first AUTH for later processing */ if (auth_skipped == 0) { auth_offset = *offset; auth_len = chk_length; auth_skipped = 1; } /* skip this chunk (temporarily) */ goto next_chunk; } if (stcb->asoc.auth_supported == 0) { goto unknown_chunk; } if ((chk_length < (sizeof(struct sctp_auth_chunk))) || (chk_length > (sizeof(struct sctp_auth_chunk) + SCTP_AUTH_DIGEST_LEN_MAX))) { /* Its not ours */ if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } if (got_auth == 1) { /* skip this chunk... it's already auth'd */ goto next_chunk; } got_auth = 1; if ((ch == NULL) || sctp_handle_auth(stcb, (struct sctp_auth_chunk *)ch, m, *offset)) { /* auth HMAC failed so dump the packet */ *offset = length; return (stcb); } else { /* remaining chunks are HMAC checked */ stcb->asoc.authenticated = 1; } break; default: unknown_chunk: /* it's an unknown chunk! */ if ((ch->chunk_type & 0x40) && (stcb != NULL)) { struct sctp_gen_error_cause *cause; int len; op_err = sctp_get_mbuf_for_msg(sizeof(struct sctp_gen_error_cause), 0, M_NOWAIT, 1, MT_DATA); if (op_err != NULL) { len = min(SCTP_SIZE32(chk_length), (uint32_t) (length - *offset)); cause = mtod(op_err, struct sctp_gen_error_cause *); cause->code = htons(SCTP_CAUSE_UNRECOG_CHUNK); cause->length = htons(len + sizeof(struct sctp_gen_error_cause)); SCTP_BUF_LEN(op_err) = sizeof(struct sctp_gen_error_cause); SCTP_BUF_NEXT(op_err) = SCTP_M_COPYM(m, *offset, len, M_NOWAIT); if (SCTP_BUF_NEXT(op_err) != NULL) { #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(SCTP_BUF_NEXT(op_err), SCTP_MBUF_ICOPY); } #endif sctp_queue_op_err(stcb, op_err); } else { sctp_m_freem(op_err); } } } if ((ch->chunk_type & 0x80) == 0) { /* discard this packet */ *offset = length; return (stcb); } /* else skip this bad chunk and continue... */ break; } /* switch (ch->chunk_type) */ next_chunk: /* get the next chunk */ *offset += SCTP_SIZE32(chk_length); if (*offset >= length) { /* no more data left in the mbuf chain */ break; } ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, *offset, sizeof(struct sctp_chunkhdr), chunk_buf); if (ch == NULL) { if (locked_tcb) { SCTP_TCB_UNLOCK(locked_tcb); } *offset = length; return (NULL); } } /* while */ if (asconf_cnt > 0 && stcb != NULL) { sctp_send_asconf_ack(stcb); } return (stcb); } /* * common input chunk processing (v4 and v6) */ void sctp_common_input_processing(struct mbuf **mm, int iphlen, int offset, int length, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_chunkhdr *ch, #if !defined(SCTP_WITH_NO_CSUM) uint8_t compute_crc, #endif uint8_t ecn_bits, uint8_t mflowtype, uint32_t mflowid, uint16_t fibnum, uint32_t vrf_id, uint16_t port) { uint32_t high_tsn; int fwd_tsn_seen = 0, data_processed = 0; struct mbuf *m = *mm, *op_err; char msg[SCTP_DIAG_INFO_LEN]; int un_sent; int cnt_ctrl_ready = 0; struct sctp_inpcb *inp = NULL, *inp_decr = NULL; struct sctp_tcb *stcb = NULL; struct sctp_nets *net = NULL; SCTP_STAT_INCR(sctps_recvdatagrams); #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xE0, 1); sctp_auditing(0, inp, stcb, net); #endif #if !defined(SCTP_WITH_NO_CSUM) if (compute_crc != 0) { uint32_t check, calc_check; check = sh->checksum; sh->checksum = 0; calc_check = sctp_calculate_cksum(m, iphlen); sh->checksum = check; if (calc_check != check) { SCTPDBG(SCTP_DEBUG_INPUT1, "Bad CSUM on SCTP packet calc_check:%x check:%x m:%p mlen:%d iphlen:%d\n", calc_check, check, (void *)m, length, iphlen); stcb = sctp_findassociation_addr(m, offset, src, dst, sh, ch, &inp, &net, vrf_id); #if defined(INET) || defined(INET6) - if ((net != NULL) && (port != 0)) { + if ((ch->chunk_type != SCTP_INITIATION) && + (net != NULL) && (net->port != port)) { if (net->port == 0) { - sctp_pathmtu_adjustment(stcb, net->mtu - sizeof(struct udphdr)); + /* UDP encapsulation turned on. */ + net->mtu -= sizeof(struct udphdr); + if (stcb->asoc.smallest_mtu > net->mtu) { + sctp_pathmtu_adjustment(stcb, net->mtu); + } + } else if (port == 0) { + /* UDP encapsulation turned off. */ + net->mtu += sizeof(struct udphdr); + /* XXX Update smallest_mtu */ } net->port = port; } #endif if (net != NULL) { net->flowtype = mflowtype; net->flowid = mflowid; } if ((inp != NULL) && (stcb != NULL)) { sctp_send_packet_dropped(stcb, net, m, length, iphlen, 1); sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_INPUT_ERROR, SCTP_SO_NOT_LOCKED); } else if ((inp != NULL) && (stcb == NULL)) { inp_decr = inp; } SCTP_STAT_INCR(sctps_badsum); SCTP_STAT_INCR_COUNTER32(sctps_checksumerrors); goto out; } } #endif /* Destination port of 0 is illegal, based on RFC4960. */ if (sh->dest_port == 0) { SCTP_STAT_INCR(sctps_hdrops); goto out; } stcb = sctp_findassociation_addr(m, offset, src, dst, sh, ch, &inp, &net, vrf_id); #if defined(INET) || defined(INET6) - if ((net != NULL) && (port != 0)) { + if ((ch->chunk_type != SCTP_INITIATION) && + (net != NULL) && (net->port != port)) { if (net->port == 0) { - sctp_pathmtu_adjustment(stcb, net->mtu - sizeof(struct udphdr)); + /* UDP encapsulation turned on. */ + net->mtu -= sizeof(struct udphdr); + if (stcb->asoc.smallest_mtu > net->mtu) { + sctp_pathmtu_adjustment(stcb, net->mtu); + } + } else if (port == 0) { + /* UDP encapsulation turned off. */ + net->mtu += sizeof(struct udphdr); + /* XXX Update smallest_mtu */ } net->port = port; } #endif if (net != NULL) { net->flowtype = mflowtype; net->flowid = mflowid; } if (inp == NULL) { SCTP_STAT_INCR(sctps_noport); if (badport_bandlim(BANDLIM_SCTP_OOTB) < 0) { goto out; } if (ch->chunk_type == SCTP_SHUTDOWN_ACK) { sctp_send_shutdown_complete2(src, dst, sh, mflowtype, mflowid, fibnum, vrf_id, port); goto out; } if (ch->chunk_type == SCTP_SHUTDOWN_COMPLETE) { goto out; } if (ch->chunk_type != SCTP_ABORT_ASSOCIATION) { if ((SCTP_BASE_SYSCTL(sctp_blackhole) == 0) || ((SCTP_BASE_SYSCTL(sctp_blackhole) == 1) && (ch->chunk_type != SCTP_INIT))) { op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), "Out of the blue"); sctp_send_abort(m, iphlen, src, dst, sh, 0, op_err, mflowtype, mflowid, fibnum, vrf_id, port); } } goto out; } else if (stcb == NULL) { inp_decr = inp; } #ifdef IPSEC /*- * I very much doubt any of the IPSEC stuff will work but I have no * idea, so I will leave it in place. */ if (inp != NULL) { switch (dst->sa_family) { #ifdef INET case AF_INET: if (ipsec4_in_reject(m, &inp->ip_inp.inp)) { SCTP_STAT_INCR(sctps_hdrops); goto out; } break; #endif #ifdef INET6 case AF_INET6: if (ipsec6_in_reject(m, &inp->ip_inp.inp)) { SCTP_STAT_INCR(sctps_hdrops); goto out; } break; #endif default: break; } } #endif SCTPDBG(SCTP_DEBUG_INPUT1, "Ok, Common input processing called, m:%p iphlen:%d offset:%d length:%d stcb:%p\n", (void *)m, iphlen, offset, length, (void *)stcb); if (stcb) { /* always clear this before beginning a packet */ stcb->asoc.authenticated = 0; stcb->asoc.seen_a_sack_this_pkt = 0; SCTPDBG(SCTP_DEBUG_INPUT1, "stcb:%p state:%x\n", (void *)stcb, stcb->asoc.state); if ((stcb->asoc.state & SCTP_STATE_WAS_ABORTED) || (stcb->asoc.state & SCTP_STATE_ABOUT_TO_BE_FREED)) { /*- * If we hit here, we had a ref count * up when the assoc was aborted and the * timer is clearing out the assoc, we should * NOT respond to any packet.. its OOTB. */ SCTP_TCB_UNLOCK(stcb); stcb = NULL; snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, offset, src, dst, sh, inp, op_err, mflowtype, mflowid, inp->fibnum, vrf_id, port); goto out; } } if (IS_SCTP_CONTROL(ch)) { /* process the control portion of the SCTP packet */ /* sa_ignore NO_NULL_CHK */ stcb = sctp_process_control(m, iphlen, &offset, length, src, dst, sh, ch, inp, stcb, &net, &fwd_tsn_seen, mflowtype, mflowid, fibnum, vrf_id, port); if (stcb) { /* * This covers us if the cookie-echo was there and * it changes our INP. */ inp = stcb->sctp_ep; #if defined(INET) || defined(INET6) - if ((net != NULL) && (port != 0)) { + if ((ch->chunk_type != SCTP_INITIATION) && + (net != NULL) && (net->port != port)) { if (net->port == 0) { - sctp_pathmtu_adjustment(stcb, net->mtu - sizeof(struct udphdr)); + /* UDP encapsulation turned on. */ + net->mtu -= sizeof(struct udphdr); + if (stcb->asoc.smallest_mtu > net->mtu) { + sctp_pathmtu_adjustment(stcb, net->mtu); + } + } else if (port == 0) { + /* UDP encapsulation turned off. */ + net->mtu += sizeof(struct udphdr); + /* XXX Update smallest_mtu */ } net->port = port; } #endif } } else { /* * no control chunks, so pre-process DATA chunks (these * checks are taken care of by control processing) */ /* * if DATA only packet, and auth is required, then punt... * can't have authenticated without any AUTH (control) * chunks */ if ((stcb != NULL) && (stcb->asoc.auth_supported == 1) && sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.local_auth_chunks)) { /* "silently" ignore */ SCTP_STAT_INCR(sctps_recvauthmissing); goto out; } if (stcb == NULL) { /* out of the blue DATA chunk */ snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, offset, src, dst, sh, inp, op_err, mflowtype, mflowid, fibnum, vrf_id, port); goto out; } if (stcb->asoc.my_vtag != ntohl(sh->v_tag)) { /* v_tag mismatch! */ SCTP_STAT_INCR(sctps_badvtag); goto out; } } if (stcb == NULL) { /* * no valid TCB for this packet, or we found it's a bad * packet while processing control, or we're done with this * packet (done or skip rest of data), so we drop it... */ goto out; } /* * DATA chunk processing */ /* plow through the data chunks while length > offset */ /* * Rest should be DATA only. Check authentication state if AUTH for * DATA is required. */ if ((length > offset) && (stcb != NULL) && (stcb->asoc.auth_supported == 1) && sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.local_auth_chunks) && !stcb->asoc.authenticated) { /* "silently" ignore */ SCTP_STAT_INCR(sctps_recvauthmissing); SCTPDBG(SCTP_DEBUG_AUTH1, "Data chunk requires AUTH, skipped\n"); goto trigger_send; } if (length > offset) { int retval; /* * First check to make sure our state is correct. We would * not get here unless we really did have a tag, so we don't * abort if this happens, just dump the chunk silently. */ switch (SCTP_GET_STATE(&stcb->asoc)) { case SCTP_STATE_COOKIE_ECHOED: /* * we consider data with valid tags in this state * shows us the cookie-ack was lost. Imply it was * there. */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_THRESHOLD_LOGGING) { sctp_misc_ints(SCTP_THRESHOLD_CLEAR, stcb->asoc.overall_error_count, 0, SCTP_FROM_SCTP_INPUT, __LINE__); } stcb->asoc.overall_error_count = 0; sctp_handle_cookie_ack((struct sctp_cookie_ack_chunk *)ch, stcb, net); break; case SCTP_STATE_COOKIE_WAIT: /* * We consider OOTB any data sent during asoc setup. */ snprintf(msg, sizeof(msg), "OOTB, %s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_handle_ootb(m, iphlen, offset, src, dst, sh, inp, op_err, mflowtype, mflowid, inp->fibnum, vrf_id, port); goto out; /* sa_ignore NOTREACHED */ break; case SCTP_STATE_EMPTY: /* should not happen */ case SCTP_STATE_INUSE: /* should not happen */ case SCTP_STATE_SHUTDOWN_RECEIVED: /* This is a peer error */ case SCTP_STATE_SHUTDOWN_ACK_SENT: default: goto out; /* sa_ignore NOTREACHED */ break; case SCTP_STATE_OPEN: case SCTP_STATE_SHUTDOWN_SENT: break; } /* plow through the data chunks while length > offset */ retval = sctp_process_data(mm, iphlen, &offset, length, inp, stcb, net, &high_tsn); if (retval == 2) { /* * The association aborted, NO UNLOCK needed since * the association is destroyed. */ stcb = NULL; goto out; } data_processed = 1; /* * Anything important needs to have been m_copy'ed in * process_data */ } /* take care of ecn */ if ((data_processed == 1) && (stcb->asoc.ecn_supported == 1) && ((ecn_bits & SCTP_CE_BITS) == SCTP_CE_BITS)) { /* Yep, we need to add a ECNE */ sctp_send_ecn_echo(stcb, net, high_tsn); } if ((data_processed == 0) && (fwd_tsn_seen)) { int was_a_gap; uint32_t highest_tsn; if (SCTP_TSN_GT(stcb->asoc.highest_tsn_inside_nr_map, stcb->asoc.highest_tsn_inside_map)) { highest_tsn = stcb->asoc.highest_tsn_inside_nr_map; } else { highest_tsn = stcb->asoc.highest_tsn_inside_map; } was_a_gap = SCTP_TSN_GT(highest_tsn, stcb->asoc.cumulative_tsn); stcb->asoc.send_sack = 1; sctp_sack_check(stcb, was_a_gap); } else if (fwd_tsn_seen) { stcb->asoc.send_sack = 1; } /* trigger send of any chunks in queue... */ trigger_send: #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xE0, 2); sctp_auditing(1, inp, stcb, net); #endif SCTPDBG(SCTP_DEBUG_INPUT1, "Check for chunk output prw:%d tqe:%d tf=%d\n", stcb->asoc.peers_rwnd, TAILQ_EMPTY(&stcb->asoc.control_send_queue), stcb->asoc.total_flight); un_sent = (stcb->asoc.total_output_queue_size - stcb->asoc.total_flight); if (!TAILQ_EMPTY(&stcb->asoc.control_send_queue)) { cnt_ctrl_ready = stcb->asoc.ctrl_queue_cnt - stcb->asoc.ecn_echo_cnt_onq; } if (cnt_ctrl_ready || stcb->asoc.trigger_reset || ((un_sent) && (stcb->asoc.peers_rwnd > 0 || (stcb->asoc.peers_rwnd <= 0 && stcb->asoc.total_flight == 0)))) { SCTPDBG(SCTP_DEBUG_INPUT3, "Calling chunk OUTPUT\n"); sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_CONTROL_PROC, SCTP_SO_NOT_LOCKED); SCTPDBG(SCTP_DEBUG_INPUT3, "chunk OUTPUT returns\n"); } #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xE0, 3); sctp_auditing(2, inp, stcb, net); #endif out: if (stcb != NULL) { SCTP_TCB_UNLOCK(stcb); } if (inp_decr != NULL) { /* reduce ref-count */ SCTP_INP_WLOCK(inp_decr); SCTP_INP_DECR_REF(inp_decr); SCTP_INP_WUNLOCK(inp_decr); } return; } #ifdef INET void sctp_input_with_port(struct mbuf *i_pak, int off, uint16_t port) { struct mbuf *m; int iphlen; uint32_t vrf_id = 0; uint8_t ecn_bits; struct sockaddr_in src, dst; struct ip *ip; struct sctphdr *sh; struct sctp_chunkhdr *ch; int length, offset; #if !defined(SCTP_WITH_NO_CSUM) uint8_t compute_crc; #endif uint32_t mflowid; uint8_t mflowtype; uint16_t fibnum; iphlen = off; if (SCTP_GET_PKT_VRFID(i_pak, vrf_id)) { SCTP_RELEASE_PKT(i_pak); return; } m = SCTP_HEADER_TO_CHAIN(i_pak); #ifdef SCTP_MBUF_LOGGING /* Log in any input mbufs */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(m, SCTP_MBUF_INPUT); } #endif #ifdef SCTP_PACKET_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LAST_PACKET_TRACING) { sctp_packet_log(m); } #endif SCTPDBG(SCTP_DEBUG_CRCOFFLOAD, "sctp_input(): Packet of length %d received on %s with csum_flags 0x%b.\n", m->m_pkthdr.len, if_name(m->m_pkthdr.rcvif), (int)m->m_pkthdr.csum_flags, CSUM_BITS); mflowid = m->m_pkthdr.flowid; mflowtype = M_HASHTYPE_GET(m); fibnum = M_GETFIB(m); SCTP_STAT_INCR(sctps_recvpackets); SCTP_STAT_INCR_COUNTER64(sctps_inpackets); /* Get IP, SCTP, and first chunk header together in the first mbuf. */ offset = iphlen + sizeof(struct sctphdr) + sizeof(struct sctp_chunkhdr); if (SCTP_BUF_LEN(m) < offset) { if ((m = m_pullup(m, offset)) == NULL) { SCTP_STAT_INCR(sctps_hdrops); return; } } ip = mtod(m, struct ip *); sh = (struct sctphdr *)((caddr_t)ip + iphlen); ch = (struct sctp_chunkhdr *)((caddr_t)sh + sizeof(struct sctphdr)); offset -= sizeof(struct sctp_chunkhdr); memset(&src, 0, sizeof(struct sockaddr_in)); src.sin_family = AF_INET; src.sin_len = sizeof(struct sockaddr_in); src.sin_port = sh->src_port; src.sin_addr = ip->ip_src; memset(&dst, 0, sizeof(struct sockaddr_in)); dst.sin_family = AF_INET; dst.sin_len = sizeof(struct sockaddr_in); dst.sin_port = sh->dest_port; dst.sin_addr = ip->ip_dst; length = ntohs(ip->ip_len); /* Validate mbuf chain length with IP payload length. */ if (SCTP_HEADER_LEN(m) != length) { SCTPDBG(SCTP_DEBUG_INPUT1, "sctp_input() length:%d reported length:%d\n", length, SCTP_HEADER_LEN(m)); SCTP_STAT_INCR(sctps_hdrops); goto out; } /* SCTP does not allow broadcasts or multicasts */ if (IN_MULTICAST(ntohl(dst.sin_addr.s_addr))) { goto out; } if (SCTP_IS_IT_BROADCAST(dst.sin_addr, m)) { goto out; } ecn_bits = ip->ip_tos; #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_recvnocrc); #else if (m->m_pkthdr.csum_flags & CSUM_SCTP_VALID) { SCTP_STAT_INCR(sctps_recvhwcrc); compute_crc = 0; } else { SCTP_STAT_INCR(sctps_recvswcrc); compute_crc = 1; } #endif sctp_common_input_processing(&m, iphlen, offset, length, (struct sockaddr *)&src, (struct sockaddr *)&dst, sh, ch, #if !defined(SCTP_WITH_NO_CSUM) compute_crc, #endif ecn_bits, mflowtype, mflowid, fibnum, vrf_id, port); out: if (m) { sctp_m_freem(m); } return; } #if defined(__FreeBSD__) && defined(SCTP_MCORE_INPUT) && defined(SMP) extern int *sctp_cpuarry; #endif int sctp_input(struct mbuf **mp, int *offp, int proto SCTP_UNUSED) { struct mbuf *m; int off; m = *mp; off = *offp; #if defined(__FreeBSD__) && defined(SCTP_MCORE_INPUT) && defined(SMP) if (mp_ncpus > 1) { struct ip *ip; struct sctphdr *sh; int offset; int cpu_to_use; uint32_t flowid, tag; if (M_HASHTYPE_GET(m) != M_HASHTYPE_NONE) { flowid = m->m_pkthdr.flowid; } else { /* * No flow id built by lower layers fix it so we * create one. */ offset = off + sizeof(struct sctphdr); if (SCTP_BUF_LEN(m) < offset) { if ((m = m_pullup(m, offset)) == NULL) { SCTP_STAT_INCR(sctps_hdrops); return (IPPROTO_DONE); } } ip = mtod(m, struct ip *); sh = (struct sctphdr *)((caddr_t)ip + off); tag = htonl(sh->v_tag); flowid = tag ^ ntohs(sh->dest_port) ^ ntohs(sh->src_port); m->m_pkthdr.flowid = flowid; M_HASHTYPE_SET(m, M_HASHTYPE_OPAQUE); } cpu_to_use = sctp_cpuarry[flowid % mp_ncpus]; sctp_queue_to_mcore(m, off, cpu_to_use); return (IPPROTO_DONE); } #endif sctp_input_with_port(m, off, 0); return (IPPROTO_DONE); } #endif Index: user/ngie/make_check/sys/netinet/sctp_output.c =================================================================== --- user/ngie/make_check/sys/netinet/sctp_output.c (revision 295077) +++ user/ngie/make_check/sys/netinet/sctp_output.c (revision 295078) @@ -1,13687 +1,13734 @@ /*- * Copyright (c) 2001-2008, by Cisco Systems, Inc. All rights reserved. * Copyright (c) 2008-2012, by Randall Stewart. All rights reserved. * Copyright (c) 2008-2012, by Michael Tuexen. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * a) Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * b) Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the distribution. * * c) Neither the name of Cisco Systems, Inc. nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF * THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #if defined(INET) || defined(INET6) #include #endif #include #include #define SCTP_MAX_GAPS_INARRAY 4 struct sack_track { uint8_t right_edge; /* mergable on the right edge */ uint8_t left_edge; /* mergable on the left edge */ uint8_t num_entries; uint8_t spare; struct sctp_gap_ack_block gaps[SCTP_MAX_GAPS_INARRAY]; }; struct sack_track sack_array[256] = { {0, 0, 0, 0, /* 0x00 */ {{0, 0}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 1, 0, /* 0x01 */ {{0, 0}, {0, 0}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x02 */ {{1, 1}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 1, 0, /* 0x03 */ {{0, 1}, {0, 0}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x04 */ {{2, 2}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x05 */ {{0, 0}, {2, 2}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x06 */ {{1, 2}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 1, 0, /* 0x07 */ {{0, 2}, {0, 0}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x08 */ {{3, 3}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x09 */ {{0, 0}, {3, 3}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x0a */ {{1, 1}, {3, 3}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x0b */ {{0, 1}, {3, 3}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x0c */ {{2, 3}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x0d */ {{0, 0}, {2, 3}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x0e */ {{1, 3}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 1, 0, /* 0x0f */ {{0, 3}, {0, 0}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x10 */ {{4, 4}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x11 */ {{0, 0}, {4, 4}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x12 */ {{1, 1}, {4, 4}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x13 */ {{0, 1}, {4, 4}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x14 */ {{2, 2}, {4, 4}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x15 */ {{0, 0}, {2, 2}, {4, 4}, {0, 0} } }, {0, 0, 2, 0, /* 0x16 */ {{1, 2}, {4, 4}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x17 */ {{0, 2}, {4, 4}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x18 */ {{3, 4}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x19 */ {{0, 0}, {3, 4}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x1a */ {{1, 1}, {3, 4}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x1b */ {{0, 1}, {3, 4}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x1c */ {{2, 4}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x1d */ {{0, 0}, {2, 4}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x1e */ {{1, 4}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 1, 0, /* 0x1f */ {{0, 4}, {0, 0}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x20 */ {{5, 5}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x21 */ {{0, 0}, {5, 5}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x22 */ {{1, 1}, {5, 5}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x23 */ {{0, 1}, {5, 5}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x24 */ {{2, 2}, {5, 5}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x25 */ {{0, 0}, {2, 2}, {5, 5}, {0, 0} } }, {0, 0, 2, 0, /* 0x26 */ {{1, 2}, {5, 5}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x27 */ {{0, 2}, {5, 5}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x28 */ {{3, 3}, {5, 5}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x29 */ {{0, 0}, {3, 3}, {5, 5}, {0, 0} } }, {0, 0, 3, 0, /* 0x2a */ {{1, 1}, {3, 3}, {5, 5}, {0, 0} } }, {1, 0, 3, 0, /* 0x2b */ {{0, 1}, {3, 3}, {5, 5}, {0, 0} } }, {0, 0, 2, 0, /* 0x2c */ {{2, 3}, {5, 5}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x2d */ {{0, 0}, {2, 3}, {5, 5}, {0, 0} } }, {0, 0, 2, 0, /* 0x2e */ {{1, 3}, {5, 5}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x2f */ {{0, 3}, {5, 5}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x30 */ {{4, 5}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x31 */ {{0, 0}, {4, 5}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x32 */ {{1, 1}, {4, 5}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x33 */ {{0, 1}, {4, 5}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x34 */ {{2, 2}, {4, 5}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x35 */ {{0, 0}, {2, 2}, {4, 5}, {0, 0} } }, {0, 0, 2, 0, /* 0x36 */ {{1, 2}, {4, 5}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x37 */ {{0, 2}, {4, 5}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x38 */ {{3, 5}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x39 */ {{0, 0}, {3, 5}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x3a */ {{1, 1}, {3, 5}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x3b */ {{0, 1}, {3, 5}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x3c */ {{2, 5}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x3d */ {{0, 0}, {2, 5}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x3e */ {{1, 5}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 1, 0, /* 0x3f */ {{0, 5}, {0, 0}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x40 */ {{6, 6}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x41 */ {{0, 0}, {6, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x42 */ {{1, 1}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x43 */ {{0, 1}, {6, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x44 */ {{2, 2}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x45 */ {{0, 0}, {2, 2}, {6, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x46 */ {{1, 2}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x47 */ {{0, 2}, {6, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x48 */ {{3, 3}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x49 */ {{0, 0}, {3, 3}, {6, 6}, {0, 0} } }, {0, 0, 3, 0, /* 0x4a */ {{1, 1}, {3, 3}, {6, 6}, {0, 0} } }, {1, 0, 3, 0, /* 0x4b */ {{0, 1}, {3, 3}, {6, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x4c */ {{2, 3}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x4d */ {{0, 0}, {2, 3}, {6, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x4e */ {{1, 3}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x4f */ {{0, 3}, {6, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x50 */ {{4, 4}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x51 */ {{0, 0}, {4, 4}, {6, 6}, {0, 0} } }, {0, 0, 3, 0, /* 0x52 */ {{1, 1}, {4, 4}, {6, 6}, {0, 0} } }, {1, 0, 3, 0, /* 0x53 */ {{0, 1}, {4, 4}, {6, 6}, {0, 0} } }, {0, 0, 3, 0, /* 0x54 */ {{2, 2}, {4, 4}, {6, 6}, {0, 0} } }, {1, 0, 4, 0, /* 0x55 */ {{0, 0}, {2, 2}, {4, 4}, {6, 6} } }, {0, 0, 3, 0, /* 0x56 */ {{1, 2}, {4, 4}, {6, 6}, {0, 0} } }, {1, 0, 3, 0, /* 0x57 */ {{0, 2}, {4, 4}, {6, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x58 */ {{3, 4}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x59 */ {{0, 0}, {3, 4}, {6, 6}, {0, 0} } }, {0, 0, 3, 0, /* 0x5a */ {{1, 1}, {3, 4}, {6, 6}, {0, 0} } }, {1, 0, 3, 0, /* 0x5b */ {{0, 1}, {3, 4}, {6, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x5c */ {{2, 4}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x5d */ {{0, 0}, {2, 4}, {6, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x5e */ {{1, 4}, {6, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x5f */ {{0, 4}, {6, 6}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x60 */ {{5, 6}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x61 */ {{0, 0}, {5, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x62 */ {{1, 1}, {5, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x63 */ {{0, 1}, {5, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x64 */ {{2, 2}, {5, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x65 */ {{0, 0}, {2, 2}, {5, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x66 */ {{1, 2}, {5, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x67 */ {{0, 2}, {5, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x68 */ {{3, 3}, {5, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x69 */ {{0, 0}, {3, 3}, {5, 6}, {0, 0} } }, {0, 0, 3, 0, /* 0x6a */ {{1, 1}, {3, 3}, {5, 6}, {0, 0} } }, {1, 0, 3, 0, /* 0x6b */ {{0, 1}, {3, 3}, {5, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x6c */ {{2, 3}, {5, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x6d */ {{0, 0}, {2, 3}, {5, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x6e */ {{1, 3}, {5, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x6f */ {{0, 3}, {5, 6}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x70 */ {{4, 6}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x71 */ {{0, 0}, {4, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x72 */ {{1, 1}, {4, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x73 */ {{0, 1}, {4, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x74 */ {{2, 2}, {4, 6}, {0, 0}, {0, 0} } }, {1, 0, 3, 0, /* 0x75 */ {{0, 0}, {2, 2}, {4, 6}, {0, 0} } }, {0, 0, 2, 0, /* 0x76 */ {{1, 2}, {4, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x77 */ {{0, 2}, {4, 6}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x78 */ {{3, 6}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x79 */ {{0, 0}, {3, 6}, {0, 0}, {0, 0} } }, {0, 0, 2, 0, /* 0x7a */ {{1, 1}, {3, 6}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x7b */ {{0, 1}, {3, 6}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x7c */ {{2, 6}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 2, 0, /* 0x7d */ {{0, 0}, {2, 6}, {0, 0}, {0, 0} } }, {0, 0, 1, 0, /* 0x7e */ {{1, 6}, {0, 0}, {0, 0}, {0, 0} } }, {1, 0, 1, 0, /* 0x7f */ {{0, 6}, {0, 0}, {0, 0}, {0, 0} } }, {0, 1, 1, 0, /* 0x80 */ {{7, 7}, {0, 0}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0x81 */ {{0, 0}, {7, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0x82 */ {{1, 1}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0x83 */ {{0, 1}, {7, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0x84 */ {{2, 2}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0x85 */ {{0, 0}, {2, 2}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0x86 */ {{1, 2}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0x87 */ {{0, 2}, {7, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0x88 */ {{3, 3}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0x89 */ {{0, 0}, {3, 3}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0x8a */ {{1, 1}, {3, 3}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0x8b */ {{0, 1}, {3, 3}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0x8c */ {{2, 3}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0x8d */ {{0, 0}, {2, 3}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0x8e */ {{1, 3}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0x8f */ {{0, 3}, {7, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0x90 */ {{4, 4}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0x91 */ {{0, 0}, {4, 4}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0x92 */ {{1, 1}, {4, 4}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0x93 */ {{0, 1}, {4, 4}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0x94 */ {{2, 2}, {4, 4}, {7, 7}, {0, 0} } }, {1, 1, 4, 0, /* 0x95 */ {{0, 0}, {2, 2}, {4, 4}, {7, 7} } }, {0, 1, 3, 0, /* 0x96 */ {{1, 2}, {4, 4}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0x97 */ {{0, 2}, {4, 4}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0x98 */ {{3, 4}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0x99 */ {{0, 0}, {3, 4}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0x9a */ {{1, 1}, {3, 4}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0x9b */ {{0, 1}, {3, 4}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0x9c */ {{2, 4}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0x9d */ {{0, 0}, {2, 4}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0x9e */ {{1, 4}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0x9f */ {{0, 4}, {7, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xa0 */ {{5, 5}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xa1 */ {{0, 0}, {5, 5}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xa2 */ {{1, 1}, {5, 5}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xa3 */ {{0, 1}, {5, 5}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xa4 */ {{2, 2}, {5, 5}, {7, 7}, {0, 0} } }, {1, 1, 4, 0, /* 0xa5 */ {{0, 0}, {2, 2}, {5, 5}, {7, 7} } }, {0, 1, 3, 0, /* 0xa6 */ {{1, 2}, {5, 5}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xa7 */ {{0, 2}, {5, 5}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xa8 */ {{3, 3}, {5, 5}, {7, 7}, {0, 0} } }, {1, 1, 4, 0, /* 0xa9 */ {{0, 0}, {3, 3}, {5, 5}, {7, 7} } }, {0, 1, 4, 0, /* 0xaa */ {{1, 1}, {3, 3}, {5, 5}, {7, 7} } }, {1, 1, 4, 0, /* 0xab */ {{0, 1}, {3, 3}, {5, 5}, {7, 7} } }, {0, 1, 3, 0, /* 0xac */ {{2, 3}, {5, 5}, {7, 7}, {0, 0} } }, {1, 1, 4, 0, /* 0xad */ {{0, 0}, {2, 3}, {5, 5}, {7, 7} } }, {0, 1, 3, 0, /* 0xae */ {{1, 3}, {5, 5}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xaf */ {{0, 3}, {5, 5}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xb0 */ {{4, 5}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xb1 */ {{0, 0}, {4, 5}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xb2 */ {{1, 1}, {4, 5}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xb3 */ {{0, 1}, {4, 5}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xb4 */ {{2, 2}, {4, 5}, {7, 7}, {0, 0} } }, {1, 1, 4, 0, /* 0xb5 */ {{0, 0}, {2, 2}, {4, 5}, {7, 7} } }, {0, 1, 3, 0, /* 0xb6 */ {{1, 2}, {4, 5}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xb7 */ {{0, 2}, {4, 5}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xb8 */ {{3, 5}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xb9 */ {{0, 0}, {3, 5}, {7, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xba */ {{1, 1}, {3, 5}, {7, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xbb */ {{0, 1}, {3, 5}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xbc */ {{2, 5}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xbd */ {{0, 0}, {2, 5}, {7, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xbe */ {{1, 5}, {7, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xbf */ {{0, 5}, {7, 7}, {0, 0}, {0, 0} } }, {0, 1, 1, 0, /* 0xc0 */ {{6, 7}, {0, 0}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xc1 */ {{0, 0}, {6, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xc2 */ {{1, 1}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xc3 */ {{0, 1}, {6, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xc4 */ {{2, 2}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xc5 */ {{0, 0}, {2, 2}, {6, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xc6 */ {{1, 2}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xc7 */ {{0, 2}, {6, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xc8 */ {{3, 3}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xc9 */ {{0, 0}, {3, 3}, {6, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xca */ {{1, 1}, {3, 3}, {6, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xcb */ {{0, 1}, {3, 3}, {6, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xcc */ {{2, 3}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xcd */ {{0, 0}, {2, 3}, {6, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xce */ {{1, 3}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xcf */ {{0, 3}, {6, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xd0 */ {{4, 4}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xd1 */ {{0, 0}, {4, 4}, {6, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xd2 */ {{1, 1}, {4, 4}, {6, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xd3 */ {{0, 1}, {4, 4}, {6, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xd4 */ {{2, 2}, {4, 4}, {6, 7}, {0, 0} } }, {1, 1, 4, 0, /* 0xd5 */ {{0, 0}, {2, 2}, {4, 4}, {6, 7} } }, {0, 1, 3, 0, /* 0xd6 */ {{1, 2}, {4, 4}, {6, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xd7 */ {{0, 2}, {4, 4}, {6, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xd8 */ {{3, 4}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xd9 */ {{0, 0}, {3, 4}, {6, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xda */ {{1, 1}, {3, 4}, {6, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xdb */ {{0, 1}, {3, 4}, {6, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xdc */ {{2, 4}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xdd */ {{0, 0}, {2, 4}, {6, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xde */ {{1, 4}, {6, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xdf */ {{0, 4}, {6, 7}, {0, 0}, {0, 0} } }, {0, 1, 1, 0, /* 0xe0 */ {{5, 7}, {0, 0}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xe1 */ {{0, 0}, {5, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xe2 */ {{1, 1}, {5, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xe3 */ {{0, 1}, {5, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xe4 */ {{2, 2}, {5, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xe5 */ {{0, 0}, {2, 2}, {5, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xe6 */ {{1, 2}, {5, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xe7 */ {{0, 2}, {5, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xe8 */ {{3, 3}, {5, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xe9 */ {{0, 0}, {3, 3}, {5, 7}, {0, 0} } }, {0, 1, 3, 0, /* 0xea */ {{1, 1}, {3, 3}, {5, 7}, {0, 0} } }, {1, 1, 3, 0, /* 0xeb */ {{0, 1}, {3, 3}, {5, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xec */ {{2, 3}, {5, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xed */ {{0, 0}, {2, 3}, {5, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xee */ {{1, 3}, {5, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xef */ {{0, 3}, {5, 7}, {0, 0}, {0, 0} } }, {0, 1, 1, 0, /* 0xf0 */ {{4, 7}, {0, 0}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xf1 */ {{0, 0}, {4, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xf2 */ {{1, 1}, {4, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xf3 */ {{0, 1}, {4, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xf4 */ {{2, 2}, {4, 7}, {0, 0}, {0, 0} } }, {1, 1, 3, 0, /* 0xf5 */ {{0, 0}, {2, 2}, {4, 7}, {0, 0} } }, {0, 1, 2, 0, /* 0xf6 */ {{1, 2}, {4, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xf7 */ {{0, 2}, {4, 7}, {0, 0}, {0, 0} } }, {0, 1, 1, 0, /* 0xf8 */ {{3, 7}, {0, 0}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xf9 */ {{0, 0}, {3, 7}, {0, 0}, {0, 0} } }, {0, 1, 2, 0, /* 0xfa */ {{1, 1}, {3, 7}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xfb */ {{0, 1}, {3, 7}, {0, 0}, {0, 0} } }, {0, 1, 1, 0, /* 0xfc */ {{2, 7}, {0, 0}, {0, 0}, {0, 0} } }, {1, 1, 2, 0, /* 0xfd */ {{0, 0}, {2, 7}, {0, 0}, {0, 0} } }, {0, 1, 1, 0, /* 0xfe */ {{1, 7}, {0, 0}, {0, 0}, {0, 0} } }, {1, 1, 1, 0, /* 0xff */ {{0, 7}, {0, 0}, {0, 0}, {0, 0} } } }; int sctp_is_address_in_scope(struct sctp_ifa *ifa, struct sctp_scoping *scope, int do_update) { if ((scope->loopback_scope == 0) && (ifa->ifn_p) && SCTP_IFN_IS_IFT_LOOP(ifa->ifn_p)) { /* * skip loopback if not in scope * */ return (0); } switch (ifa->address.sa.sa_family) { #ifdef INET case AF_INET: if (scope->ipv4_addr_legal) { struct sockaddr_in *sin; sin = &ifa->address.sin; if (sin->sin_addr.s_addr == 0) { /* not in scope , unspecified */ return (0); } if ((scope->ipv4_local_scope == 0) && (IN4_ISPRIVATE_ADDRESS(&sin->sin_addr))) { /* private address not in scope */ return (0); } } else { return (0); } break; #endif #ifdef INET6 case AF_INET6: if (scope->ipv6_addr_legal) { struct sockaddr_in6 *sin6; /* * Must update the flags, bummer, which means any * IFA locks must now be applied HERE <-> */ if (do_update) { sctp_gather_internal_ifa_flags(ifa); } if (ifa->localifa_flags & SCTP_ADDR_IFA_UNUSEABLE) { return (0); } /* ok to use deprecated addresses? */ sin6 = &ifa->address.sin6; if (IN6_IS_ADDR_UNSPECIFIED(&sin6->sin6_addr)) { /* skip unspecifed addresses */ return (0); } if ( /* (local_scope == 0) && */ (IN6_IS_ADDR_LINKLOCAL(&sin6->sin6_addr))) { return (0); } if ((scope->site_scope == 0) && (IN6_IS_ADDR_SITELOCAL(&sin6->sin6_addr))) { return (0); } } else { return (0); } break; #endif default: return (0); } return (1); } static struct mbuf * sctp_add_addr_to_mbuf(struct mbuf *m, struct sctp_ifa *ifa, uint16_t * len) { #if defined(INET) || defined(INET6) struct sctp_paramhdr *parmh; struct mbuf *mret; uint16_t plen; #endif switch (ifa->address.sa.sa_family) { #ifdef INET case AF_INET: plen = (uint16_t) sizeof(struct sctp_ipv4addr_param); break; #endif #ifdef INET6 case AF_INET6: plen = (uint16_t) sizeof(struct sctp_ipv6addr_param); break; #endif default: return (m); } #if defined(INET) || defined(INET6) if (M_TRAILINGSPACE(m) >= plen) { /* easy side we just drop it on the end */ parmh = (struct sctp_paramhdr *)(SCTP_BUF_AT(m, SCTP_BUF_LEN(m))); mret = m; } else { /* Need more space */ mret = m; while (SCTP_BUF_NEXT(mret) != NULL) { mret = SCTP_BUF_NEXT(mret); } SCTP_BUF_NEXT(mret) = sctp_get_mbuf_for_msg(plen, 0, M_NOWAIT, 1, MT_DATA); if (SCTP_BUF_NEXT(mret) == NULL) { /* We are hosed, can't add more addresses */ return (m); } mret = SCTP_BUF_NEXT(mret); parmh = mtod(mret, struct sctp_paramhdr *); } /* now add the parameter */ switch (ifa->address.sa.sa_family) { #ifdef INET case AF_INET: { struct sctp_ipv4addr_param *ipv4p; struct sockaddr_in *sin; sin = &ifa->address.sin; ipv4p = (struct sctp_ipv4addr_param *)parmh; parmh->param_type = htons(SCTP_IPV4_ADDRESS); parmh->param_length = htons(plen); ipv4p->addr = sin->sin_addr.s_addr; SCTP_BUF_LEN(mret) += plen; break; } #endif #ifdef INET6 case AF_INET6: { struct sctp_ipv6addr_param *ipv6p; struct sockaddr_in6 *sin6; sin6 = &ifa->address.sin6; ipv6p = (struct sctp_ipv6addr_param *)parmh; parmh->param_type = htons(SCTP_IPV6_ADDRESS); parmh->param_length = htons(plen); memcpy(ipv6p->addr, &sin6->sin6_addr, sizeof(ipv6p->addr)); /* clear embedded scope in the address */ in6_clearscope((struct in6_addr *)ipv6p->addr); SCTP_BUF_LEN(mret) += plen; break; } #endif default: return (m); } if (len != NULL) { *len += plen; } return (mret); #endif } struct mbuf * sctp_add_addresses_to_i_ia(struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_scoping *scope, struct mbuf *m_at, int cnt_inits_to, uint16_t * padding_len, uint16_t * chunk_len) { struct sctp_vrf *vrf = NULL; int cnt, limit_out = 0, total_count; uint32_t vrf_id; vrf_id = inp->def_vrf_id; SCTP_IPI_ADDR_RLOCK(); vrf = sctp_find_vrf(vrf_id); if (vrf == NULL) { SCTP_IPI_ADDR_RUNLOCK(); return (m_at); } if (inp->sctp_flags & SCTP_PCB_FLAGS_BOUNDALL) { struct sctp_ifa *sctp_ifap; struct sctp_ifn *sctp_ifnp; cnt = cnt_inits_to; if (vrf->total_ifa_count > SCTP_COUNT_LIMIT) { limit_out = 1; cnt = SCTP_ADDRESS_LIMIT; goto skip_count; } LIST_FOREACH(sctp_ifnp, &vrf->ifnlist, next_ifn) { if ((scope->loopback_scope == 0) && SCTP_IFN_IS_IFT_LOOP(sctp_ifnp)) { /* * Skip loopback devices if loopback_scope * not set */ continue; } LIST_FOREACH(sctp_ifap, &sctp_ifnp->ifalist, next_ifa) { #ifdef INET if ((sctp_ifap->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifap->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((sctp_ifap->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifap->address.sin6.sin6_addr) != 0)) { continue; } #endif if (sctp_is_addr_restricted(stcb, sctp_ifap)) { continue; } if (sctp_is_address_in_scope(sctp_ifap, scope, 1) == 0) { continue; } cnt++; if (cnt > SCTP_ADDRESS_LIMIT) { break; } } if (cnt > SCTP_ADDRESS_LIMIT) { break; } } skip_count: if (cnt > 1) { total_count = 0; LIST_FOREACH(sctp_ifnp, &vrf->ifnlist, next_ifn) { cnt = 0; if ((scope->loopback_scope == 0) && SCTP_IFN_IS_IFT_LOOP(sctp_ifnp)) { /* * Skip loopback devices if * loopback_scope not set */ continue; } LIST_FOREACH(sctp_ifap, &sctp_ifnp->ifalist, next_ifa) { #ifdef INET if ((sctp_ifap->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifap->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((sctp_ifap->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifap->address.sin6.sin6_addr) != 0)) { continue; } #endif if (sctp_is_addr_restricted(stcb, sctp_ifap)) { continue; } if (sctp_is_address_in_scope(sctp_ifap, scope, 0) == 0) { continue; } if ((chunk_len != NULL) && (padding_len != NULL) && (*padding_len > 0)) { memset(mtod(m_at, caddr_t)+*chunk_len, 0, *padding_len); SCTP_BUF_LEN(m_at) += *padding_len; *chunk_len += *padding_len; *padding_len = 0; } m_at = sctp_add_addr_to_mbuf(m_at, sctp_ifap, chunk_len); if (limit_out) { cnt++; total_count++; if (cnt >= 2) { /* * two from each * address */ break; } if (total_count > SCTP_ADDRESS_LIMIT) { /* No more addresses */ break; } } } } } } else { struct sctp_laddr *laddr; cnt = cnt_inits_to; /* First, how many ? */ LIST_FOREACH(laddr, &inp->sctp_addr_list, sctp_nxt_addr) { if (laddr->ifa == NULL) { continue; } if (laddr->ifa->localifa_flags & SCTP_BEING_DELETED) /* * Address being deleted by the system, dont * list. */ continue; if (laddr->action == SCTP_DEL_IP_ADDRESS) { /* * Address being deleted on this ep don't * list. */ continue; } if (sctp_is_address_in_scope(laddr->ifa, scope, 1) == 0) { continue; } cnt++; } /* * To get through a NAT we only list addresses if we have * more than one. That way if you just bind a single address * we let the source of the init dictate our address. */ if (cnt > 1) { cnt = cnt_inits_to; LIST_FOREACH(laddr, &inp->sctp_addr_list, sctp_nxt_addr) { if (laddr->ifa == NULL) { continue; } if (laddr->ifa->localifa_flags & SCTP_BEING_DELETED) { continue; } if (sctp_is_address_in_scope(laddr->ifa, scope, 0) == 0) { continue; } if ((chunk_len != NULL) && (padding_len != NULL) && (*padding_len > 0)) { memset(mtod(m_at, caddr_t)+*chunk_len, 0, *padding_len); SCTP_BUF_LEN(m_at) += *padding_len; *chunk_len += *padding_len; *padding_len = 0; } m_at = sctp_add_addr_to_mbuf(m_at, laddr->ifa, chunk_len); cnt++; if (cnt >= SCTP_ADDRESS_LIMIT) { break; } } } } SCTP_IPI_ADDR_RUNLOCK(); return (m_at); } static struct sctp_ifa * sctp_is_ifa_addr_preferred(struct sctp_ifa *ifa, uint8_t dest_is_loop, uint8_t dest_is_priv, sa_family_t fam) { uint8_t dest_is_global = 0; /* dest_is_priv is true if destination is a private address */ /* dest_is_loop is true if destination is a loopback addresses */ /** * Here we determine if its a preferred address. A preferred address * means it is the same scope or higher scope then the destination. * L = loopback, P = private, G = global * ----------------------------------------- * src | dest | result * ---------------------------------------- * L | L | yes * ----------------------------------------- * P | L | yes-v4 no-v6 * ----------------------------------------- * G | L | yes-v4 no-v6 * ----------------------------------------- * L | P | no * ----------------------------------------- * P | P | yes * ----------------------------------------- * G | P | no * ----------------------------------------- * L | G | no * ----------------------------------------- * P | G | no * ----------------------------------------- * G | G | yes * ----------------------------------------- */ if (ifa->address.sa.sa_family != fam) { /* forget mis-matched family */ return (NULL); } if ((dest_is_priv == 0) && (dest_is_loop == 0)) { dest_is_global = 1; } SCTPDBG(SCTP_DEBUG_OUTPUT2, "Is destination preferred:"); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, &ifa->address.sa); /* Ok the address may be ok */ #ifdef INET6 if (fam == AF_INET6) { /* ok to use deprecated addresses? no lets not! */ if (ifa->localifa_flags & SCTP_ADDR_IFA_UNUSEABLE) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "NO:1\n"); return (NULL); } if (ifa->src_is_priv && !ifa->src_is_loop) { if (dest_is_loop) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "NO:2\n"); return (NULL); } } if (ifa->src_is_glob) { if (dest_is_loop) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "NO:3\n"); return (NULL); } } } #endif /* * Now that we know what is what, implement or table this could in * theory be done slicker (it used to be), but this is * straightforward and easier to validate :-) */ SCTPDBG(SCTP_DEBUG_OUTPUT3, "src_loop:%d src_priv:%d src_glob:%d\n", ifa->src_is_loop, ifa->src_is_priv, ifa->src_is_glob); SCTPDBG(SCTP_DEBUG_OUTPUT3, "dest_loop:%d dest_priv:%d dest_glob:%d\n", dest_is_loop, dest_is_priv, dest_is_global); if ((ifa->src_is_loop) && (dest_is_priv)) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "NO:4\n"); return (NULL); } if ((ifa->src_is_glob) && (dest_is_priv)) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "NO:5\n"); return (NULL); } if ((ifa->src_is_loop) && (dest_is_global)) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "NO:6\n"); return (NULL); } if ((ifa->src_is_priv) && (dest_is_global)) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "NO:7\n"); return (NULL); } SCTPDBG(SCTP_DEBUG_OUTPUT3, "YES\n"); /* its a preferred address */ return (ifa); } static struct sctp_ifa * sctp_is_ifa_addr_acceptable(struct sctp_ifa *ifa, uint8_t dest_is_loop, uint8_t dest_is_priv, sa_family_t fam) { uint8_t dest_is_global = 0; /** * Here we determine if its a acceptable address. A acceptable * address means it is the same scope or higher scope but we can * allow for NAT which means its ok to have a global dest and a * private src. * * L = loopback, P = private, G = global * ----------------------------------------- * src | dest | result * ----------------------------------------- * L | L | yes * ----------------------------------------- * P | L | yes-v4 no-v6 * ----------------------------------------- * G | L | yes * ----------------------------------------- * L | P | no * ----------------------------------------- * P | P | yes * ----------------------------------------- * G | P | yes - May not work * ----------------------------------------- * L | G | no * ----------------------------------------- * P | G | yes - May not work * ----------------------------------------- * G | G | yes * ----------------------------------------- */ if (ifa->address.sa.sa_family != fam) { /* forget non matching family */ SCTPDBG(SCTP_DEBUG_OUTPUT3, "ifa_fam:%d fam:%d\n", ifa->address.sa.sa_family, fam); return (NULL); } /* Ok the address may be ok */ SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT3, &ifa->address.sa); SCTPDBG(SCTP_DEBUG_OUTPUT3, "dst_is_loop:%d dest_is_priv:%d\n", dest_is_loop, dest_is_priv); if ((dest_is_loop == 0) && (dest_is_priv == 0)) { dest_is_global = 1; } #ifdef INET6 if (fam == AF_INET6) { /* ok to use deprecated addresses? */ if (ifa->localifa_flags & SCTP_ADDR_IFA_UNUSEABLE) { return (NULL); } if (ifa->src_is_priv) { /* Special case, linklocal to loop */ if (dest_is_loop) return (NULL); } } #endif /* * Now that we know what is what, implement our table. This could in * theory be done slicker (it used to be), but this is * straightforward and easier to validate :-) */ SCTPDBG(SCTP_DEBUG_OUTPUT3, "ifa->src_is_loop:%d dest_is_priv:%d\n", ifa->src_is_loop, dest_is_priv); if ((ifa->src_is_loop == 1) && (dest_is_priv)) { return (NULL); } SCTPDBG(SCTP_DEBUG_OUTPUT3, "ifa->src_is_loop:%d dest_is_glob:%d\n", ifa->src_is_loop, dest_is_global); if ((ifa->src_is_loop == 1) && (dest_is_global)) { return (NULL); } SCTPDBG(SCTP_DEBUG_OUTPUT3, "address is acceptable\n"); /* its an acceptable address */ return (ifa); } int sctp_is_addr_restricted(struct sctp_tcb *stcb, struct sctp_ifa *ifa) { struct sctp_laddr *laddr; if (stcb == NULL) { /* There are no restrictions, no TCB :-) */ return (0); } LIST_FOREACH(laddr, &stcb->asoc.sctp_restricted_addrs, sctp_nxt_addr) { if (laddr->ifa == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "%s: NULL ifa\n", __func__); continue; } if (laddr->ifa == ifa) { /* Yes it is on the list */ return (1); } } return (0); } int sctp_is_addr_in_ep(struct sctp_inpcb *inp, struct sctp_ifa *ifa) { struct sctp_laddr *laddr; if (ifa == NULL) return (0); LIST_FOREACH(laddr, &inp->sctp_addr_list, sctp_nxt_addr) { if (laddr->ifa == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "%s: NULL ifa\n", __func__); continue; } if ((laddr->ifa == ifa) && laddr->action == 0) /* same pointer */ return (1); } return (0); } static struct sctp_ifa * sctp_choose_boundspecific_inp(struct sctp_inpcb *inp, sctp_route_t * ro, uint32_t vrf_id, int non_asoc_addr_ok, uint8_t dest_is_priv, uint8_t dest_is_loop, sa_family_t fam) { struct sctp_laddr *laddr, *starting_point; void *ifn; int resettotop = 0; struct sctp_ifn *sctp_ifn; struct sctp_ifa *sctp_ifa, *sifa; struct sctp_vrf *vrf; uint32_t ifn_index; vrf = sctp_find_vrf(vrf_id); if (vrf == NULL) return (NULL); ifn = SCTP_GET_IFN_VOID_FROM_ROUTE(ro); ifn_index = SCTP_GET_IF_INDEX_FROM_ROUTE(ro); sctp_ifn = sctp_find_ifn(ifn, ifn_index); /* * first question, is the ifn we will emit on in our list, if so, we * want such an address. Note that we first looked for a preferred * address. */ if (sctp_ifn) { /* is a preferred one on the interface we route out? */ LIST_FOREACH(sctp_ifa, &sctp_ifn->ifalist, next_ifa) { #ifdef INET if ((sctp_ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((sctp_ifa->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin6.sin6_addr) != 0)) { continue; } #endif if ((sctp_ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) continue; sifa = sctp_is_ifa_addr_preferred(sctp_ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; if (sctp_is_addr_in_ep(inp, sifa)) { atomic_add_int(&sifa->refcount, 1); return (sifa); } } } /* * ok, now we now need to find one on the list of the addresses. We * can't get one on the emitting interface so let's find first a * preferred one. If not that an acceptable one otherwise... we * return NULL. */ starting_point = inp->next_addr_touse; once_again: if (inp->next_addr_touse == NULL) { inp->next_addr_touse = LIST_FIRST(&inp->sctp_addr_list); resettotop = 1; } for (laddr = inp->next_addr_touse; laddr; laddr = LIST_NEXT(laddr, sctp_nxt_addr)) { if (laddr->ifa == NULL) { /* address has been removed */ continue; } if (laddr->action == SCTP_DEL_IP_ADDRESS) { /* address is being deleted */ continue; } sifa = sctp_is_ifa_addr_preferred(laddr->ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; atomic_add_int(&sifa->refcount, 1); return (sifa); } if (resettotop == 0) { inp->next_addr_touse = NULL; goto once_again; } inp->next_addr_touse = starting_point; resettotop = 0; once_again_too: if (inp->next_addr_touse == NULL) { inp->next_addr_touse = LIST_FIRST(&inp->sctp_addr_list); resettotop = 1; } /* ok, what about an acceptable address in the inp */ for (laddr = inp->next_addr_touse; laddr; laddr = LIST_NEXT(laddr, sctp_nxt_addr)) { if (laddr->ifa == NULL) { /* address has been removed */ continue; } if (laddr->action == SCTP_DEL_IP_ADDRESS) { /* address is being deleted */ continue; } sifa = sctp_is_ifa_addr_acceptable(laddr->ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; atomic_add_int(&sifa->refcount, 1); return (sifa); } if (resettotop == 0) { inp->next_addr_touse = NULL; goto once_again_too; } /* * no address bound can be a source for the destination we are in * trouble */ return (NULL); } static struct sctp_ifa * sctp_choose_boundspecific_stcb(struct sctp_inpcb *inp, struct sctp_tcb *stcb, sctp_route_t * ro, uint32_t vrf_id, uint8_t dest_is_priv, uint8_t dest_is_loop, int non_asoc_addr_ok, sa_family_t fam) { struct sctp_laddr *laddr, *starting_point; void *ifn; struct sctp_ifn *sctp_ifn; struct sctp_ifa *sctp_ifa, *sifa; uint8_t start_at_beginning = 0; struct sctp_vrf *vrf; uint32_t ifn_index; /* * first question, is the ifn we will emit on in our list, if so, we * want that one. */ vrf = sctp_find_vrf(vrf_id); if (vrf == NULL) return (NULL); ifn = SCTP_GET_IFN_VOID_FROM_ROUTE(ro); ifn_index = SCTP_GET_IF_INDEX_FROM_ROUTE(ro); sctp_ifn = sctp_find_ifn(ifn, ifn_index); /* * first question, is the ifn we will emit on in our list? If so, * we want that one. First we look for a preferred. Second, we go * for an acceptable. */ if (sctp_ifn) { /* first try for a preferred address on the ep */ LIST_FOREACH(sctp_ifa, &sctp_ifn->ifalist, next_ifa) { #ifdef INET if ((sctp_ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((sctp_ifa->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin6.sin6_addr) != 0)) { continue; } #endif if ((sctp_ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) continue; if (sctp_is_addr_in_ep(inp, sctp_ifa)) { sifa = sctp_is_ifa_addr_preferred(sctp_ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* on the no-no list */ continue; } atomic_add_int(&sifa->refcount, 1); return (sifa); } } /* next try for an acceptable address on the ep */ LIST_FOREACH(sctp_ifa, &sctp_ifn->ifalist, next_ifa) { #ifdef INET if ((sctp_ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((sctp_ifa->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin6.sin6_addr) != 0)) { continue; } #endif if ((sctp_ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) continue; if (sctp_is_addr_in_ep(inp, sctp_ifa)) { sifa = sctp_is_ifa_addr_acceptable(sctp_ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* on the no-no list */ continue; } atomic_add_int(&sifa->refcount, 1); return (sifa); } } } /* * if we can't find one like that then we must look at all addresses * bound to pick one at first preferable then secondly acceptable. */ starting_point = stcb->asoc.last_used_address; sctp_from_the_top: if (stcb->asoc.last_used_address == NULL) { start_at_beginning = 1; stcb->asoc.last_used_address = LIST_FIRST(&inp->sctp_addr_list); } /* search beginning with the last used address */ for (laddr = stcb->asoc.last_used_address; laddr; laddr = LIST_NEXT(laddr, sctp_nxt_addr)) { if (laddr->ifa == NULL) { /* address has been removed */ continue; } if (laddr->action == SCTP_DEL_IP_ADDRESS) { /* address is being deleted */ continue; } sifa = sctp_is_ifa_addr_preferred(laddr->ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* on the no-no list */ continue; } stcb->asoc.last_used_address = laddr; atomic_add_int(&sifa->refcount, 1); return (sifa); } if (start_at_beginning == 0) { stcb->asoc.last_used_address = NULL; goto sctp_from_the_top; } /* now try for any higher scope than the destination */ stcb->asoc.last_used_address = starting_point; start_at_beginning = 0; sctp_from_the_top2: if (stcb->asoc.last_used_address == NULL) { start_at_beginning = 1; stcb->asoc.last_used_address = LIST_FIRST(&inp->sctp_addr_list); } /* search beginning with the last used address */ for (laddr = stcb->asoc.last_used_address; laddr; laddr = LIST_NEXT(laddr, sctp_nxt_addr)) { if (laddr->ifa == NULL) { /* address has been removed */ continue; } if (laddr->action == SCTP_DEL_IP_ADDRESS) { /* address is being deleted */ continue; } sifa = sctp_is_ifa_addr_acceptable(laddr->ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* on the no-no list */ continue; } stcb->asoc.last_used_address = laddr; atomic_add_int(&sifa->refcount, 1); return (sifa); } if (start_at_beginning == 0) { stcb->asoc.last_used_address = NULL; goto sctp_from_the_top2; } return (NULL); } static struct sctp_ifa * sctp_select_nth_preferred_addr_from_ifn_boundall(struct sctp_ifn *ifn, struct sctp_inpcb *inp, struct sctp_tcb *stcb, int non_asoc_addr_ok, uint8_t dest_is_loop, uint8_t dest_is_priv, int addr_wanted, sa_family_t fam, sctp_route_t * ro ) { struct sctp_ifa *ifa, *sifa; int num_eligible_addr = 0; #ifdef INET6 struct sockaddr_in6 sin6, lsa6; if (fam == AF_INET6) { memcpy(&sin6, &ro->ro_dst, sizeof(struct sockaddr_in6)); (void)sa6_recoverscope(&sin6); } #endif /* INET6 */ LIST_FOREACH(ifa, &ifn->ifalist, next_ifa) { #ifdef INET if ((ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &ifa->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((ifa->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &ifa->address.sin6.sin6_addr) != 0)) { continue; } #endif if ((ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) continue; sifa = sctp_is_ifa_addr_preferred(ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; #ifdef INET6 if (fam == AF_INET6 && dest_is_loop && sifa->src_is_loop && sifa->src_is_priv) { /* * don't allow fe80::1 to be a src on loop ::1, we * don't list it to the peer so we will get an * abort. */ continue; } if (fam == AF_INET6 && IN6_IS_ADDR_LINKLOCAL(&sifa->address.sin6.sin6_addr) && IN6_IS_ADDR_LINKLOCAL(&sin6.sin6_addr)) { /* * link-local <-> link-local must belong to the same * scope. */ memcpy(&lsa6, &sifa->address.sin6, sizeof(struct sockaddr_in6)); (void)sa6_recoverscope(&lsa6); if (sin6.sin6_scope_id != lsa6.sin6_scope_id) { continue; } } #endif /* INET6 */ /* * Check if the IPv6 address matches to next-hop. In the * mobile case, old IPv6 address may be not deleted from the * interface. Then, the interface has previous and new * addresses. We should use one corresponding to the * next-hop. (by micchie) */ #ifdef INET6 if (stcb && fam == AF_INET6 && sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_BASE)) { if (sctp_v6src_match_nexthop(&sifa->address.sin6, ro) == 0) { continue; } } #endif #ifdef INET /* Avoid topologically incorrect IPv4 address */ if (stcb && fam == AF_INET && sctp_is_mobility_feature_on(stcb->sctp_ep, SCTP_MOBILITY_BASE)) { if (sctp_v4src_match_nexthop(sifa, ro) == 0) { continue; } } #endif if (stcb) { if (sctp_is_address_in_scope(ifa, &stcb->asoc.scope, 0) == 0) { continue; } if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* * It is restricted for some reason.. * probably not yet added. */ continue; } } if (num_eligible_addr >= addr_wanted) { return (sifa); } num_eligible_addr++; } return (NULL); } static int sctp_count_num_preferred_boundall(struct sctp_ifn *ifn, struct sctp_inpcb *inp, struct sctp_tcb *stcb, int non_asoc_addr_ok, uint8_t dest_is_loop, uint8_t dest_is_priv, sa_family_t fam) { struct sctp_ifa *ifa, *sifa; int num_eligible_addr = 0; LIST_FOREACH(ifa, &ifn->ifalist, next_ifa) { #ifdef INET if ((ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &ifa->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((ifa->address.sa.sa_family == AF_INET6) && (stcb != NULL) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &ifa->address.sin6.sin6_addr) != 0)) { continue; } #endif if ((ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) { continue; } sifa = sctp_is_ifa_addr_preferred(ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) { continue; } if (stcb) { if (sctp_is_address_in_scope(ifa, &stcb->asoc.scope, 0) == 0) { continue; } if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* * It is restricted for some reason.. * probably not yet added. */ continue; } } num_eligible_addr++; } return (num_eligible_addr); } static struct sctp_ifa * sctp_choose_boundall(struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_nets *net, sctp_route_t * ro, uint32_t vrf_id, uint8_t dest_is_priv, uint8_t dest_is_loop, int non_asoc_addr_ok, sa_family_t fam) { int cur_addr_num = 0, num_preferred = 0; void *ifn; struct sctp_ifn *sctp_ifn, *looked_at = NULL, *emit_ifn; struct sctp_ifa *sctp_ifa, *sifa; uint32_t ifn_index; struct sctp_vrf *vrf; #ifdef INET int retried = 0; #endif /*- * For boundall we can use any address in the association. * If non_asoc_addr_ok is set we can use any address (at least in * theory). So we look for preferred addresses first. If we find one, * we use it. Otherwise we next try to get an address on the * interface, which we should be able to do (unless non_asoc_addr_ok * is false and we are routed out that way). In these cases where we * can't use the address of the interface we go through all the * ifn's looking for an address we can use and fill that in. Punting * means we send back address 0, which will probably cause problems * actually since then IP will fill in the address of the route ifn, * which means we probably already rejected it.. i.e. here comes an * abort :-<. */ vrf = sctp_find_vrf(vrf_id); if (vrf == NULL) return (NULL); ifn = SCTP_GET_IFN_VOID_FROM_ROUTE(ro); ifn_index = SCTP_GET_IF_INDEX_FROM_ROUTE(ro); SCTPDBG(SCTP_DEBUG_OUTPUT2, "ifn from route:%p ifn_index:%d\n", ifn, ifn_index); emit_ifn = looked_at = sctp_ifn = sctp_find_ifn(ifn, ifn_index); if (sctp_ifn == NULL) { /* ?? We don't have this guy ?? */ SCTPDBG(SCTP_DEBUG_OUTPUT2, "No ifn emit interface?\n"); goto bound_all_plan_b; } SCTPDBG(SCTP_DEBUG_OUTPUT2, "ifn_index:%d name:%s is emit interface\n", ifn_index, sctp_ifn->ifn_name); if (net) { cur_addr_num = net->indx_of_eligible_next_to_use; } num_preferred = sctp_count_num_preferred_boundall(sctp_ifn, inp, stcb, non_asoc_addr_ok, dest_is_loop, dest_is_priv, fam); SCTPDBG(SCTP_DEBUG_OUTPUT2, "Found %d preferred source addresses for intf:%s\n", num_preferred, sctp_ifn->ifn_name); if (num_preferred == 0) { /* * no eligible addresses, we must use some other interface * address if we can find one. */ goto bound_all_plan_b; } /* * Ok we have num_eligible_addr set with how many we can use, this * may vary from call to call due to addresses being deprecated * etc.. */ if (cur_addr_num >= num_preferred) { cur_addr_num = 0; } /* * select the nth address from the list (where cur_addr_num is the * nth) and 0 is the first one, 1 is the second one etc... */ SCTPDBG(SCTP_DEBUG_OUTPUT2, "cur_addr_num:%d\n", cur_addr_num); sctp_ifa = sctp_select_nth_preferred_addr_from_ifn_boundall(sctp_ifn, inp, stcb, non_asoc_addr_ok, dest_is_loop, dest_is_priv, cur_addr_num, fam, ro); /* if sctp_ifa is NULL something changed??, fall to plan b. */ if (sctp_ifa) { atomic_add_int(&sctp_ifa->refcount, 1); if (net) { /* save off where the next one we will want */ net->indx_of_eligible_next_to_use = cur_addr_num + 1; } return (sctp_ifa); } /* * plan_b: Look at all interfaces and find a preferred address. If * no preferred fall through to plan_c. */ bound_all_plan_b: SCTPDBG(SCTP_DEBUG_OUTPUT2, "Trying Plan B\n"); LIST_FOREACH(sctp_ifn, &vrf->ifnlist, next_ifn) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "Examine interface %s\n", sctp_ifn->ifn_name); if (dest_is_loop == 0 && SCTP_IFN_IS_IFT_LOOP(sctp_ifn)) { /* wrong base scope */ SCTPDBG(SCTP_DEBUG_OUTPUT2, "skip\n"); continue; } if ((sctp_ifn == looked_at) && looked_at) { /* already looked at this guy */ SCTPDBG(SCTP_DEBUG_OUTPUT2, "already seen\n"); continue; } num_preferred = sctp_count_num_preferred_boundall(sctp_ifn, inp, stcb, non_asoc_addr_ok, dest_is_loop, dest_is_priv, fam); SCTPDBG(SCTP_DEBUG_OUTPUT2, "Found ifn:%p %d preferred source addresses\n", ifn, num_preferred); if (num_preferred == 0) { /* None on this interface. */ SCTPDBG(SCTP_DEBUG_OUTPUT2, "No prefered -- skipping to next\n"); continue; } SCTPDBG(SCTP_DEBUG_OUTPUT2, "num preferred:%d on interface:%p cur_addr_num:%d\n", num_preferred, (void *)sctp_ifn, cur_addr_num); /* * Ok we have num_eligible_addr set with how many we can * use, this may vary from call to call due to addresses * being deprecated etc.. */ if (cur_addr_num >= num_preferred) { cur_addr_num = 0; } sifa = sctp_select_nth_preferred_addr_from_ifn_boundall(sctp_ifn, inp, stcb, non_asoc_addr_ok, dest_is_loop, dest_is_priv, cur_addr_num, fam, ro); if (sifa == NULL) continue; if (net) { net->indx_of_eligible_next_to_use = cur_addr_num + 1; SCTPDBG(SCTP_DEBUG_OUTPUT2, "we selected %d\n", cur_addr_num); SCTPDBG(SCTP_DEBUG_OUTPUT2, "Source:"); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, &sifa->address.sa); SCTPDBG(SCTP_DEBUG_OUTPUT2, "Dest:"); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, &net->ro._l_addr.sa); } atomic_add_int(&sifa->refcount, 1); return (sifa); } #ifdef INET again_with_private_addresses_allowed: #endif /* plan_c: do we have an acceptable address on the emit interface */ sifa = NULL; SCTPDBG(SCTP_DEBUG_OUTPUT2, "Trying Plan C: find acceptable on interface\n"); if (emit_ifn == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "Jump to Plan D - no emit_ifn\n"); goto plan_d; } LIST_FOREACH(sctp_ifa, &emit_ifn->ifalist, next_ifa) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "ifa:%p\n", (void *)sctp_ifa); #ifdef INET if ((sctp_ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin.sin_addr) != 0)) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "Jailed\n"); continue; } #endif #ifdef INET6 if ((sctp_ifa->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin6.sin6_addr) != 0)) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "Jailed\n"); continue; } #endif if ((sctp_ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "Defer\n"); continue; } sifa = sctp_is_ifa_addr_acceptable(sctp_ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "IFA not acceptable\n"); continue; } if (stcb) { if (sctp_is_address_in_scope(sifa, &stcb->asoc.scope, 0) == 0) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "NOT in scope\n"); sifa = NULL; continue; } if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* * It is restricted for some reason.. * probably not yet added. */ SCTPDBG(SCTP_DEBUG_OUTPUT2, "Its resticted\n"); sifa = NULL; continue; } } atomic_add_int(&sifa->refcount, 1); goto out; } plan_d: /* * plan_d: We are in trouble. No preferred address on the emit * interface. And not even a preferred address on all interfaces. Go * out and see if we can find an acceptable address somewhere * amongst all interfaces. */ SCTPDBG(SCTP_DEBUG_OUTPUT2, "Trying Plan D looked_at is %p\n", (void *)looked_at); LIST_FOREACH(sctp_ifn, &vrf->ifnlist, next_ifn) { if (dest_is_loop == 0 && SCTP_IFN_IS_IFT_LOOP(sctp_ifn)) { /* wrong base scope */ continue; } LIST_FOREACH(sctp_ifa, &sctp_ifn->ifalist, next_ifa) { #ifdef INET if ((sctp_ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((sctp_ifa->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin6.sin6_addr) != 0)) { continue; } #endif if ((sctp_ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) continue; sifa = sctp_is_ifa_addr_acceptable(sctp_ifa, dest_is_loop, dest_is_priv, fam); if (sifa == NULL) continue; if (stcb) { if (sctp_is_address_in_scope(sifa, &stcb->asoc.scope, 0) == 0) { sifa = NULL; continue; } if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, sifa)) && (!sctp_is_addr_pending(stcb, sifa)))) { /* * It is restricted for some * reason.. probably not yet added. */ sifa = NULL; continue; } } goto out; } } #ifdef INET if ((retried == 0) && (stcb->asoc.scope.ipv4_local_scope == 0)) { stcb->asoc.scope.ipv4_local_scope = 1; retried = 1; goto again_with_private_addresses_allowed; } else if (retried == 1) { stcb->asoc.scope.ipv4_local_scope = 0; } #endif out: #ifdef INET if (sifa) { if (retried == 1) { LIST_FOREACH(sctp_ifn, &vrf->ifnlist, next_ifn) { if (dest_is_loop == 0 && SCTP_IFN_IS_IFT_LOOP(sctp_ifn)) { /* wrong base scope */ continue; } LIST_FOREACH(sctp_ifa, &sctp_ifn->ifalist, next_ifa) { struct sctp_ifa *tmp_sifa; #ifdef INET if ((sctp_ifa->address.sa.sa_family == AF_INET) && (prison_check_ip4(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin.sin_addr) != 0)) { continue; } #endif #ifdef INET6 if ((sctp_ifa->address.sa.sa_family == AF_INET6) && (prison_check_ip6(inp->ip_inp.inp.inp_cred, &sctp_ifa->address.sin6.sin6_addr) != 0)) { continue; } #endif if ((sctp_ifa->localifa_flags & SCTP_ADDR_DEFER_USE) && (non_asoc_addr_ok == 0)) continue; tmp_sifa = sctp_is_ifa_addr_acceptable(sctp_ifa, dest_is_loop, dest_is_priv, fam); if (tmp_sifa == NULL) { continue; } if (tmp_sifa == sifa) { continue; } if (stcb) { if (sctp_is_address_in_scope(tmp_sifa, &stcb->asoc.scope, 0) == 0) { continue; } if (((non_asoc_addr_ok == 0) && (sctp_is_addr_restricted(stcb, tmp_sifa))) || (non_asoc_addr_ok && (sctp_is_addr_restricted(stcb, tmp_sifa)) && (!sctp_is_addr_pending(stcb, tmp_sifa)))) { /* * It is restricted * for some reason.. * probably not yet * added. */ continue; } } if ((tmp_sifa->address.sin.sin_family == AF_INET) && (IN4_ISPRIVATE_ADDRESS(&(tmp_sifa->address.sin.sin_addr)))) { sctp_add_local_addr_restricted(stcb, tmp_sifa); } } } } atomic_add_int(&sifa->refcount, 1); } #endif return (sifa); } /* tcb may be NULL */ struct sctp_ifa * sctp_source_address_selection(struct sctp_inpcb *inp, struct sctp_tcb *stcb, sctp_route_t * ro, struct sctp_nets *net, int non_asoc_addr_ok, uint32_t vrf_id) { struct sctp_ifa *answer; uint8_t dest_is_priv, dest_is_loop; sa_family_t fam; #ifdef INET struct sockaddr_in *to = (struct sockaddr_in *)&ro->ro_dst; #endif #ifdef INET6 struct sockaddr_in6 *to6 = (struct sockaddr_in6 *)&ro->ro_dst; #endif /** * Rules: - Find the route if needed, cache if I can. - Look at * interface address in route, Is it in the bound list. If so we * have the best source. - If not we must rotate amongst the * addresses. * * Cavets and issues * * Do we need to pay attention to scope. We can have a private address * or a global address we are sourcing or sending to. So if we draw * it out * zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz * For V4 * ------------------------------------------ * source * dest * result * ----------------------------------------- * Private * Global * NAT * ----------------------------------------- * Private * Private * No problem * ----------------------------------------- * Global * Private * Huh, How will this work? * ----------------------------------------- * Global * Global * No Problem *------------------------------------------ * zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz * For V6 *------------------------------------------ * source * dest * result * ----------------------------------------- * Linklocal * Global * * ----------------------------------------- * Linklocal * Linklocal * No problem * ----------------------------------------- * Global * Linklocal * Huh, How will this work? * ----------------------------------------- * Global * Global * No Problem *------------------------------------------ * zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz * * And then we add to that what happens if there are multiple addresses * assigned to an interface. Remember the ifa on a ifn is a linked * list of addresses. So one interface can have more than one IP * address. What happens if we have both a private and a global * address? Do we then use context of destination to sort out which * one is best? And what about NAT's sending P->G may get you a NAT * translation, or should you select the G thats on the interface in * preference. * * Decisions: * * - count the number of addresses on the interface. * - if it is one, no problem except case . * For we will assume a NAT out there. * - if there are more than one, then we need to worry about scope P * or G. We should prefer G -> G and P -> P if possible. * Then as a secondary fall back to mixed types G->P being a last * ditch one. * - The above all works for bound all, but bound specific we need to * use the same concept but instead only consider the bound * addresses. If the bound set is NOT assigned to the interface then * we must use rotation amongst the bound addresses.. */ if (ro->ro_rt == NULL) { /* * Need a route to cache. */ SCTP_RTALLOC(ro, vrf_id, inp->fibnum); } if (ro->ro_rt == NULL) { return (NULL); } fam = ro->ro_dst.sa_family; dest_is_priv = dest_is_loop = 0; /* Setup our scopes for the destination */ switch (fam) { #ifdef INET case AF_INET: /* Scope based on outbound address */ if (IN4_ISLOOPBACK_ADDRESS(&to->sin_addr)) { dest_is_loop = 1; if (net != NULL) { /* mark it as local */ net->addr_is_local = 1; } } else if ((IN4_ISPRIVATE_ADDRESS(&to->sin_addr))) { dest_is_priv = 1; } break; #endif #ifdef INET6 case AF_INET6: /* Scope based on outbound address */ if (IN6_IS_ADDR_LOOPBACK(&to6->sin6_addr) || SCTP_ROUTE_IS_REAL_LOOP(ro)) { /* * If the address is a loopback address, which * consists of "::1" OR "fe80::1%lo0", we are * loopback scope. But we don't use dest_is_priv * (link local addresses). */ dest_is_loop = 1; if (net != NULL) { /* mark it as local */ net->addr_is_local = 1; } } else if (IN6_IS_ADDR_LINKLOCAL(&to6->sin6_addr)) { dest_is_priv = 1; } break; #endif } SCTPDBG(SCTP_DEBUG_OUTPUT2, "Select source addr for:"); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, (struct sockaddr *)&ro->ro_dst); SCTP_IPI_ADDR_RLOCK(); if (inp->sctp_flags & SCTP_PCB_FLAGS_BOUNDALL) { /* * Bound all case */ answer = sctp_choose_boundall(inp, stcb, net, ro, vrf_id, dest_is_priv, dest_is_loop, non_asoc_addr_ok, fam); SCTP_IPI_ADDR_RUNLOCK(); return (answer); } /* * Subset bound case */ if (stcb) { answer = sctp_choose_boundspecific_stcb(inp, stcb, ro, vrf_id, dest_is_priv, dest_is_loop, non_asoc_addr_ok, fam); } else { answer = sctp_choose_boundspecific_inp(inp, ro, vrf_id, non_asoc_addr_ok, dest_is_priv, dest_is_loop, fam); } SCTP_IPI_ADDR_RUNLOCK(); return (answer); } static int sctp_find_cmsg(int c_type, void *data, struct mbuf *control, size_t cpsize) { struct cmsghdr cmh; int tlen, at, found; struct sctp_sndinfo sndinfo; struct sctp_prinfo prinfo; struct sctp_authinfo authinfo; tlen = SCTP_BUF_LEN(control); at = 0; found = 0; /* * Independent of how many mbufs, find the c_type inside the control * structure and copy out the data. */ while (at < tlen) { if ((tlen - at) < (int)CMSG_ALIGN(sizeof(cmh))) { /* There is not enough room for one more. */ return (found); } m_copydata(control, at, sizeof(cmh), (caddr_t)&cmh); if (cmh.cmsg_len < CMSG_ALIGN(sizeof(cmh))) { /* We dont't have a complete CMSG header. */ return (found); } if (((int)cmh.cmsg_len + at) > tlen) { /* We don't have the complete CMSG. */ return (found); } if ((cmh.cmsg_level == IPPROTO_SCTP) && ((c_type == cmh.cmsg_type) || ((c_type == SCTP_SNDRCV) && ((cmh.cmsg_type == SCTP_SNDINFO) || (cmh.cmsg_type == SCTP_PRINFO) || (cmh.cmsg_type == SCTP_AUTHINFO))))) { if (c_type == cmh.cmsg_type) { if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < cpsize) { return (found); } /* It is exactly what we want. Copy it out. */ m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), cpsize, (caddr_t)data); return (1); } else { struct sctp_sndrcvinfo *sndrcvinfo; sndrcvinfo = (struct sctp_sndrcvinfo *)data; if (found == 0) { if (cpsize < sizeof(struct sctp_sndrcvinfo)) { return (found); } memset(sndrcvinfo, 0, sizeof(struct sctp_sndrcvinfo)); } switch (cmh.cmsg_type) { case SCTP_SNDINFO: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct sctp_sndinfo)) { return (found); } m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct sctp_sndinfo), (caddr_t)&sndinfo); sndrcvinfo->sinfo_stream = sndinfo.snd_sid; sndrcvinfo->sinfo_flags = sndinfo.snd_flags; sndrcvinfo->sinfo_ppid = sndinfo.snd_ppid; sndrcvinfo->sinfo_context = sndinfo.snd_context; sndrcvinfo->sinfo_assoc_id = sndinfo.snd_assoc_id; break; case SCTP_PRINFO: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct sctp_prinfo)) { return (found); } m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct sctp_prinfo), (caddr_t)&prinfo); if (prinfo.pr_policy != SCTP_PR_SCTP_NONE) { sndrcvinfo->sinfo_timetolive = prinfo.pr_value; } else { sndrcvinfo->sinfo_timetolive = 0; } sndrcvinfo->sinfo_flags |= prinfo.pr_policy; break; case SCTP_AUTHINFO: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct sctp_authinfo)) { return (found); } m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct sctp_authinfo), (caddr_t)&authinfo); sndrcvinfo->sinfo_keynumber_valid = 1; sndrcvinfo->sinfo_keynumber = authinfo.auth_keynumber; break; default: return (found); } found = 1; } } at += CMSG_ALIGN(cmh.cmsg_len); } return (found); } static int sctp_process_cmsgs_for_init(struct sctp_tcb *stcb, struct mbuf *control, int *error) { struct cmsghdr cmh; int tlen, at; struct sctp_initmsg initmsg; #ifdef INET struct sockaddr_in sin; #endif #ifdef INET6 struct sockaddr_in6 sin6; #endif tlen = SCTP_BUF_LEN(control); at = 0; while (at < tlen) { if ((tlen - at) < (int)CMSG_ALIGN(sizeof(cmh))) { /* There is not enough room for one more. */ *error = EINVAL; return (1); } m_copydata(control, at, sizeof(cmh), (caddr_t)&cmh); if (cmh.cmsg_len < CMSG_ALIGN(sizeof(cmh))) { /* We dont't have a complete CMSG header. */ *error = EINVAL; return (1); } if (((int)cmh.cmsg_len + at) > tlen) { /* We don't have the complete CMSG. */ *error = EINVAL; return (1); } if (cmh.cmsg_level == IPPROTO_SCTP) { switch (cmh.cmsg_type) { case SCTP_INIT: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct sctp_initmsg)) { *error = EINVAL; return (1); } m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct sctp_initmsg), (caddr_t)&initmsg); if (initmsg.sinit_max_attempts) stcb->asoc.max_init_times = initmsg.sinit_max_attempts; if (initmsg.sinit_num_ostreams) stcb->asoc.pre_open_streams = initmsg.sinit_num_ostreams; if (initmsg.sinit_max_instreams) stcb->asoc.max_inbound_streams = initmsg.sinit_max_instreams; if (initmsg.sinit_max_init_timeo) stcb->asoc.initial_init_rto_max = initmsg.sinit_max_init_timeo; if (stcb->asoc.streamoutcnt < stcb->asoc.pre_open_streams) { struct sctp_stream_out *tmp_str; unsigned int i; #if defined(SCTP_DETAILED_STR_STATS) int j; #endif /* Default is NOT correct */ SCTPDBG(SCTP_DEBUG_OUTPUT1, "Ok, default:%d pre_open:%d\n", stcb->asoc.streamoutcnt, stcb->asoc.pre_open_streams); SCTP_TCB_UNLOCK(stcb); SCTP_MALLOC(tmp_str, struct sctp_stream_out *, (stcb->asoc.pre_open_streams * sizeof(struct sctp_stream_out)), SCTP_M_STRMO); SCTP_TCB_LOCK(stcb); if (tmp_str != NULL) { SCTP_FREE(stcb->asoc.strmout, SCTP_M_STRMO); stcb->asoc.strmout = tmp_str; stcb->asoc.strm_realoutsize = stcb->asoc.streamoutcnt = stcb->asoc.pre_open_streams; } else { stcb->asoc.pre_open_streams = stcb->asoc.streamoutcnt; } for (i = 0; i < stcb->asoc.streamoutcnt; i++) { TAILQ_INIT(&stcb->asoc.strmout[i].outqueue); stcb->asoc.strmout[i].chunks_on_queues = 0; stcb->asoc.strmout[i].next_sequence_send = 0; #if defined(SCTP_DETAILED_STR_STATS) for (j = 0; j < SCTP_PR_SCTP_MAX + 1; j++) { stcb->asoc.strmout[i].abandoned_sent[j] = 0; stcb->asoc.strmout[i].abandoned_unsent[j] = 0; } #else stcb->asoc.strmout[i].abandoned_sent[0] = 0; stcb->asoc.strmout[i].abandoned_unsent[0] = 0; #endif stcb->asoc.strmout[i].stream_no = i; stcb->asoc.strmout[i].last_msg_incomplete = 0; stcb->asoc.strmout[i].state = SCTP_STREAM_OPENING; stcb->asoc.ss_functions.sctp_ss_init_stream(&stcb->asoc.strmout[i], NULL); } } break; #ifdef INET case SCTP_DSTADDRV4: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct in_addr)) { *error = EINVAL; return (1); } memset(&sin, 0, sizeof(struct sockaddr_in)); sin.sin_family = AF_INET; sin.sin_len = sizeof(struct sockaddr_in); sin.sin_port = stcb->rport; m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct in_addr), (caddr_t)&sin.sin_addr); if ((sin.sin_addr.s_addr == INADDR_ANY) || (sin.sin_addr.s_addr == INADDR_BROADCAST) || IN_MULTICAST(ntohl(sin.sin_addr.s_addr))) { *error = EINVAL; return (1); } if (sctp_add_remote_addr(stcb, (struct sockaddr *)&sin, NULL, SCTP_DONOT_SETSCOPE, SCTP_ADDR_IS_CONFIRMED)) { *error = ENOBUFS; return (1); } break; #endif #ifdef INET6 case SCTP_DSTADDRV6: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct in6_addr)) { *error = EINVAL; return (1); } memset(&sin6, 0, sizeof(struct sockaddr_in6)); sin6.sin6_family = AF_INET6; sin6.sin6_len = sizeof(struct sockaddr_in6); sin6.sin6_port = stcb->rport; m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct in6_addr), (caddr_t)&sin6.sin6_addr); if (IN6_IS_ADDR_UNSPECIFIED(&sin6.sin6_addr) || IN6_IS_ADDR_MULTICAST(&sin6.sin6_addr)) { *error = EINVAL; return (1); } #ifdef INET if (IN6_IS_ADDR_V4MAPPED(&sin6.sin6_addr)) { in6_sin6_2_sin(&sin, &sin6); if ((sin.sin_addr.s_addr == INADDR_ANY) || (sin.sin_addr.s_addr == INADDR_BROADCAST) || IN_MULTICAST(ntohl(sin.sin_addr.s_addr))) { *error = EINVAL; return (1); } if (sctp_add_remote_addr(stcb, (struct sockaddr *)&sin, NULL, SCTP_DONOT_SETSCOPE, SCTP_ADDR_IS_CONFIRMED)) { *error = ENOBUFS; return (1); } } else #endif if (sctp_add_remote_addr(stcb, (struct sockaddr *)&sin6, NULL, SCTP_DONOT_SETSCOPE, SCTP_ADDR_IS_CONFIRMED)) { *error = ENOBUFS; return (1); } break; #endif default: break; } } at += CMSG_ALIGN(cmh.cmsg_len); } return (0); } static struct sctp_tcb * sctp_findassociation_cmsgs(struct sctp_inpcb **inp_p, uint16_t port, struct mbuf *control, struct sctp_nets **net_p, int *error) { struct cmsghdr cmh; int tlen, at; struct sctp_tcb *stcb; struct sockaddr *addr; #ifdef INET struct sockaddr_in sin; #endif #ifdef INET6 struct sockaddr_in6 sin6; #endif tlen = SCTP_BUF_LEN(control); at = 0; while (at < tlen) { if ((tlen - at) < (int)CMSG_ALIGN(sizeof(cmh))) { /* There is not enough room for one more. */ *error = EINVAL; return (NULL); } m_copydata(control, at, sizeof(cmh), (caddr_t)&cmh); if (cmh.cmsg_len < CMSG_ALIGN(sizeof(cmh))) { /* We dont't have a complete CMSG header. */ *error = EINVAL; return (NULL); } if (((int)cmh.cmsg_len + at) > tlen) { /* We don't have the complete CMSG. */ *error = EINVAL; return (NULL); } if (cmh.cmsg_level == IPPROTO_SCTP) { switch (cmh.cmsg_type) { #ifdef INET case SCTP_DSTADDRV4: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct in_addr)) { *error = EINVAL; return (NULL); } memset(&sin, 0, sizeof(struct sockaddr_in)); sin.sin_family = AF_INET; sin.sin_len = sizeof(struct sockaddr_in); sin.sin_port = port; m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct in_addr), (caddr_t)&sin.sin_addr); addr = (struct sockaddr *)&sin; break; #endif #ifdef INET6 case SCTP_DSTADDRV6: if ((size_t)(cmh.cmsg_len - CMSG_ALIGN(sizeof(cmh))) < sizeof(struct in6_addr)) { *error = EINVAL; return (NULL); } memset(&sin6, 0, sizeof(struct sockaddr_in6)); sin6.sin6_family = AF_INET6; sin6.sin6_len = sizeof(struct sockaddr_in6); sin6.sin6_port = port; m_copydata(control, at + CMSG_ALIGN(sizeof(cmh)), sizeof(struct in6_addr), (caddr_t)&sin6.sin6_addr); #ifdef INET if (IN6_IS_ADDR_V4MAPPED(&sin6.sin6_addr)) { in6_sin6_2_sin(&sin, &sin6); addr = (struct sockaddr *)&sin; } else #endif addr = (struct sockaddr *)&sin6; break; #endif default: addr = NULL; break; } if (addr) { stcb = sctp_findassociation_ep_addr(inp_p, addr, net_p, NULL, NULL); if (stcb != NULL) { return (stcb); } } } at += CMSG_ALIGN(cmh.cmsg_len); } return (NULL); } static struct mbuf * sctp_add_cookie(struct mbuf *init, int init_offset, struct mbuf *initack, int initack_offset, struct sctp_state_cookie *stc_in, uint8_t ** signature) { struct mbuf *copy_init, *copy_initack, *m_at, *sig, *mret; struct sctp_state_cookie *stc; struct sctp_paramhdr *ph; uint8_t *foo; int sig_offset; uint16_t cookie_sz; mret = sctp_get_mbuf_for_msg((sizeof(struct sctp_state_cookie) + sizeof(struct sctp_paramhdr)), 0, M_NOWAIT, 1, MT_DATA); if (mret == NULL) { return (NULL); } copy_init = SCTP_M_COPYM(init, init_offset, M_COPYALL, M_NOWAIT); if (copy_init == NULL) { sctp_m_freem(mret); return (NULL); } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(copy_init, SCTP_MBUF_ICOPY); } #endif copy_initack = SCTP_M_COPYM(initack, initack_offset, M_COPYALL, M_NOWAIT); if (copy_initack == NULL) { sctp_m_freem(mret); sctp_m_freem(copy_init); return (NULL); } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(copy_initack, SCTP_MBUF_ICOPY); } #endif /* easy side we just drop it on the end */ ph = mtod(mret, struct sctp_paramhdr *); SCTP_BUF_LEN(mret) = sizeof(struct sctp_state_cookie) + sizeof(struct sctp_paramhdr); stc = (struct sctp_state_cookie *)((caddr_t)ph + sizeof(struct sctp_paramhdr)); ph->param_type = htons(SCTP_STATE_COOKIE); ph->param_length = 0; /* fill in at the end */ /* Fill in the stc cookie data */ memcpy(stc, stc_in, sizeof(struct sctp_state_cookie)); /* tack the INIT and then the INIT-ACK onto the chain */ cookie_sz = 0; for (m_at = mret; m_at; m_at = SCTP_BUF_NEXT(m_at)) { cookie_sz += SCTP_BUF_LEN(m_at); if (SCTP_BUF_NEXT(m_at) == NULL) { SCTP_BUF_NEXT(m_at) = copy_init; break; } } for (m_at = copy_init; m_at; m_at = SCTP_BUF_NEXT(m_at)) { cookie_sz += SCTP_BUF_LEN(m_at); if (SCTP_BUF_NEXT(m_at) == NULL) { SCTP_BUF_NEXT(m_at) = copy_initack; break; } } for (m_at = copy_initack; m_at; m_at = SCTP_BUF_NEXT(m_at)) { cookie_sz += SCTP_BUF_LEN(m_at); if (SCTP_BUF_NEXT(m_at) == NULL) { break; } } sig = sctp_get_mbuf_for_msg(SCTP_SECRET_SIZE, 0, M_NOWAIT, 1, MT_DATA); if (sig == NULL) { /* no space, so free the entire chain */ sctp_m_freem(mret); return (NULL); } SCTP_BUF_LEN(sig) = 0; SCTP_BUF_NEXT(m_at) = sig; sig_offset = 0; foo = (uint8_t *) (mtod(sig, caddr_t)+sig_offset); memset(foo, 0, SCTP_SIGNATURE_SIZE); *signature = foo; SCTP_BUF_LEN(sig) += SCTP_SIGNATURE_SIZE; cookie_sz += SCTP_SIGNATURE_SIZE; ph->param_length = htons(cookie_sz); return (mret); } static uint8_t sctp_get_ect(struct sctp_tcb *stcb) { if ((stcb != NULL) && (stcb->asoc.ecn_supported == 1)) { return (SCTP_ECT0_BIT); } else { return (0); } } #if defined(INET) || defined(INET6) static void sctp_handle_no_route(struct sctp_tcb *stcb, struct sctp_nets *net, int so_locked) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "dropped packet - no valid source addr\n"); if (net) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Destination was "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT1, &net->ro._l_addr.sa); if (net->dest_state & SCTP_ADDR_CONFIRMED) { if ((net->dest_state & SCTP_ADDR_REACHABLE) && stcb) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "no route takes interface %p down\n", (void *)net); sctp_ulp_notify(SCTP_NOTIFY_INTERFACE_DOWN, stcb, 0, (void *)net, so_locked); net->dest_state &= ~SCTP_ADDR_REACHABLE; net->dest_state &= ~SCTP_ADDR_PF; } } if (stcb) { if (net == stcb->asoc.primary_destination) { /* need a new primary */ struct sctp_nets *alt; alt = sctp_find_alternate_net(stcb, net, 0); if (alt != net) { if (stcb->asoc.alternate) { sctp_free_remote_addr(stcb->asoc.alternate); } stcb->asoc.alternate = alt; atomic_add_int(&stcb->asoc.alternate->ref_count, 1); if (net->ro._s_addr) { sctp_free_ifa(net->ro._s_addr); net->ro._s_addr = NULL; } net->src_addr_selected = 0; } } } } } #endif static int sctp_lowlevel_chunk_output(struct sctp_inpcb *inp, struct sctp_tcb *stcb, /* may be NULL */ struct sctp_nets *net, struct sockaddr *to, struct mbuf *m, uint32_t auth_offset, struct sctp_auth_chunk *auth, uint16_t auth_keyid, int nofragment_flag, int ecn_ok, int out_of_asoc_ok, uint16_t src_port, uint16_t dest_port, uint32_t v_tag, uint16_t port, union sctp_sockstore *over_addr, uint8_t mflowtype, uint32_t mflowid, #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) int so_locked SCTP_UNUSED #else int so_locked #endif ) /* nofragment_flag to tell if IP_DF should be set (IPv4 only) */ { /** * Given a mbuf chain (via SCTP_BUF_NEXT()) that holds a packet header * WITH an SCTPHDR but no IP header, endpoint inp and sa structure: * - fill in the HMAC digest of any AUTH chunk in the packet. * - calculate and fill in the SCTP checksum. * - prepend an IP address header. * - if boundall use INADDR_ANY. * - if boundspecific do source address selection. * - set fragmentation option for ipV4. * - On return from IP output, check/adjust mtu size of output * interface and smallest_mtu size as well. */ /* Will need ifdefs around this */ struct mbuf *newm; struct sctphdr *sctphdr; int packet_length; int ret; #if defined(INET) || defined(INET6) uint32_t vrf_id; #endif #if defined(INET) || defined(INET6) struct mbuf *o_pak; sctp_route_t *ro = NULL; struct udphdr *udp = NULL; #endif uint8_t tos_value; #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) struct socket *so = NULL; #endif if ((net) && (net->dest_state & SCTP_ADDR_OUT_OF_SCOPE)) { SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EFAULT); sctp_m_freem(m); return (EFAULT); } #if defined(INET) || defined(INET6) if (stcb) { vrf_id = stcb->asoc.vrf_id; } else { vrf_id = inp->def_vrf_id; } #endif /* fill in the HMAC digest for any AUTH chunk in the packet */ if ((auth != NULL) && (stcb != NULL)) { sctp_fill_hmac_digest_m(m, auth_offset, auth, stcb, auth_keyid); } if (net) { tos_value = net->dscp; } else if (stcb) { tos_value = stcb->asoc.default_dscp; } else { tos_value = inp->sctp_ep.default_dscp; } switch (to->sa_family) { #ifdef INET case AF_INET: { struct ip *ip = NULL; sctp_route_t iproute; int len; len = SCTP_MIN_V4_OVERHEAD; if (port) { len += sizeof(struct udphdr); } newm = sctp_get_mbuf_for_msg(len, 1, M_NOWAIT, 1, MT_DATA); if (newm == NULL) { sctp_m_freem(m); SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } SCTP_ALIGN_TO_END(newm, len); SCTP_BUF_LEN(newm) = len; SCTP_BUF_NEXT(newm) = m; m = newm; if (net != NULL) { m->m_pkthdr.flowid = net->flowid; M_HASHTYPE_SET(m, net->flowtype); } else { m->m_pkthdr.flowid = mflowid; M_HASHTYPE_SET(m, mflowtype); } packet_length = sctp_calculate_len(m); ip = mtod(m, struct ip *); ip->ip_v = IPVERSION; ip->ip_hl = (sizeof(struct ip) >> 2); if (tos_value == 0) { /* * This means especially, that it is not set * at the SCTP layer. So use the value from * the IP layer. */ tos_value = inp->ip_inp.inp.inp_ip_tos; } tos_value &= 0xfc; if (ecn_ok) { tos_value |= sctp_get_ect(stcb); } if ((nofragment_flag) && (port == 0)) { ip->ip_off = htons(IP_DF); } else { ip->ip_off = htons(0); } /* FreeBSD has a function for ip_id's */ ip_fillid(ip); ip->ip_ttl = inp->ip_inp.inp.inp_ip_ttl; ip->ip_len = htons(packet_length); ip->ip_tos = tos_value; if (port) { ip->ip_p = IPPROTO_UDP; } else { ip->ip_p = IPPROTO_SCTP; } ip->ip_sum = 0; if (net == NULL) { ro = &iproute; memset(&iproute, 0, sizeof(iproute)); memcpy(&ro->ro_dst, to, to->sa_len); } else { ro = (sctp_route_t *) & net->ro; } /* Now the address selection part */ ip->ip_dst.s_addr = ((struct sockaddr_in *)to)->sin_addr.s_addr; /* call the routine to select the src address */ if (net && out_of_asoc_ok == 0) { if (net->ro._s_addr && (net->ro._s_addr->localifa_flags & (SCTP_BEING_DELETED | SCTP_ADDR_IFA_UNUSEABLE))) { sctp_free_ifa(net->ro._s_addr); net->ro._s_addr = NULL; net->src_addr_selected = 0; if (ro->ro_rt) { RTFREE(ro->ro_rt); ro->ro_rt = NULL; } } if (net->src_addr_selected == 0) { /* Cache the source address */ net->ro._s_addr = sctp_source_address_selection(inp, stcb, ro, net, 0, vrf_id); net->src_addr_selected = 1; } if (net->ro._s_addr == NULL) { /* No route to host */ net->src_addr_selected = 0; sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } ip->ip_src = net->ro._s_addr->address.sin.sin_addr; } else { if (over_addr == NULL) { struct sctp_ifa *_lsrc; _lsrc = sctp_source_address_selection(inp, stcb, ro, net, out_of_asoc_ok, vrf_id); if (_lsrc == NULL) { sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } ip->ip_src = _lsrc->address.sin.sin_addr; sctp_free_ifa(_lsrc); } else { ip->ip_src = over_addr->sin.sin_addr; SCTP_RTALLOC(ro, vrf_id, inp->fibnum); } } if (port) { if (htons(SCTP_BASE_SYSCTL(sctp_udp_tunneling_port)) == 0) { sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } udp = (struct udphdr *)((caddr_t)ip + sizeof(struct ip)); udp->uh_sport = htons(SCTP_BASE_SYSCTL(sctp_udp_tunneling_port)); udp->uh_dport = port; udp->uh_ulen = htons(packet_length - sizeof(struct ip)); if (V_udp_cksum) { udp->uh_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, udp->uh_ulen + htons(IPPROTO_UDP)); } else { udp->uh_sum = 0; } sctphdr = (struct sctphdr *)((caddr_t)udp + sizeof(struct udphdr)); } else { sctphdr = (struct sctphdr *)((caddr_t)ip + sizeof(struct ip)); } sctphdr->src_port = src_port; sctphdr->dest_port = dest_port; sctphdr->v_tag = v_tag; sctphdr->checksum = 0; /* * If source address selection fails and we find no * route then the ip_output should fail as well with * a NO_ROUTE_TO_HOST type error. We probably should * catch that somewhere and abort the association * right away (assuming this is an INIT being sent). */ if (ro->ro_rt == NULL) { /* * src addr selection failed to find a route * (or valid source addr), so we can't get * there from here (yet)! */ sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } if (ro != &iproute) { memcpy(&iproute, ro, sizeof(*ro)); } SCTPDBG(SCTP_DEBUG_OUTPUT3, "Calling ipv4 output routine from low level src addr:%x\n", (uint32_t) (ntohl(ip->ip_src.s_addr))); SCTPDBG(SCTP_DEBUG_OUTPUT3, "Destination is %x\n", (uint32_t) (ntohl(ip->ip_dst.s_addr))); SCTPDBG(SCTP_DEBUG_OUTPUT3, "RTP route is %p through\n", (void *)ro->ro_rt); if (SCTP_GET_HEADER_FOR_OUTPUT(o_pak)) { /* failed to prepend data, give up */ SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); sctp_m_freem(m); return (ENOMEM); } SCTP_ATTACH_CHAIN(o_pak, m, packet_length); if (port) { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else sctphdr->checksum = sctp_calculate_cksum(m, sizeof(struct ip) + sizeof(struct udphdr)); SCTP_STAT_INCR(sctps_sendswcrc); #endif if (V_udp_cksum) { SCTP_ENABLE_UDP_CSUM(o_pak); } } else { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else m->m_pkthdr.csum_flags = CSUM_SCTP; m->m_pkthdr.csum_data = offsetof(struct sctphdr, checksum); SCTP_STAT_INCR(sctps_sendhwcrc); #endif } #ifdef SCTP_PACKET_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LAST_PACKET_TRACING) sctp_packet_log(o_pak); #endif /* send it out. table id is taken from stcb */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) if ((SCTP_BASE_SYSCTL(sctp_output_unlocked)) && (so_locked)) { so = SCTP_INP_SO(inp); SCTP_SOCKET_UNLOCK(so, 0); } #endif SCTP_IP_OUTPUT(ret, o_pak, ro, stcb, vrf_id); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) if ((SCTP_BASE_SYSCTL(sctp_output_unlocked)) && (so_locked)) { atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 0); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); } #endif SCTP_STAT_INCR(sctps_sendpackets); SCTP_STAT_INCR_COUNTER64(sctps_outpackets); if (ret) SCTP_STAT_INCR(sctps_senderrors); SCTPDBG(SCTP_DEBUG_OUTPUT3, "IP output returns %d\n", ret); if (net == NULL) { /* free tempy routes */ RO_RTFREE(ro); } else { /* * PMTU check versus smallest asoc MTU goes * here */ if ((ro->ro_rt != NULL) && (net->ro._s_addr)) { uint32_t mtu; mtu = SCTP_GATHER_MTU_FROM_ROUTE(net->ro._s_addr, &net->ro._l_addr.sa, ro->ro_rt); if (net->port) { mtu -= sizeof(struct udphdr); } if (mtu && (stcb->asoc.smallest_mtu > mtu)) { sctp_mtu_size_reset(inp, &stcb->asoc, mtu); net->mtu = mtu; } } else if (ro->ro_rt == NULL) { /* route was freed */ if (net->ro._s_addr && net->src_addr_selected) { sctp_free_ifa(net->ro._s_addr); net->ro._s_addr = NULL; } net->src_addr_selected = 0; } } return (ret); } #endif #ifdef INET6 case AF_INET6: { uint32_t flowlabel, flowinfo; struct ip6_hdr *ip6h; struct route_in6 ip6route; struct ifnet *ifp; struct sockaddr_in6 *sin6, tmp, *lsa6, lsa6_tmp; int prev_scope = 0; struct sockaddr_in6 lsa6_storage; int error; u_short prev_port = 0; int len; if (net) { flowlabel = net->flowlabel; } else if (stcb) { flowlabel = stcb->asoc.default_flowlabel; } else { flowlabel = inp->sctp_ep.default_flowlabel; } if (flowlabel == 0) { /* * This means especially, that it is not set * at the SCTP layer. So use the value from * the IP layer. */ flowlabel = ntohl(((struct in6pcb *)inp)->in6p_flowinfo); } flowlabel &= 0x000fffff; len = SCTP_MIN_OVERHEAD; if (port) { len += sizeof(struct udphdr); } newm = sctp_get_mbuf_for_msg(len, 1, M_NOWAIT, 1, MT_DATA); if (newm == NULL) { sctp_m_freem(m); SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } SCTP_ALIGN_TO_END(newm, len); SCTP_BUF_LEN(newm) = len; SCTP_BUF_NEXT(newm) = m; m = newm; if (net != NULL) { m->m_pkthdr.flowid = net->flowid; M_HASHTYPE_SET(m, net->flowtype); } else { m->m_pkthdr.flowid = mflowid; M_HASHTYPE_SET(m, mflowtype); } packet_length = sctp_calculate_len(m); ip6h = mtod(m, struct ip6_hdr *); /* protect *sin6 from overwrite */ sin6 = (struct sockaddr_in6 *)to; tmp = *sin6; sin6 = &tmp; /* KAME hack: embed scopeid */ if (sa6_embedscope(sin6, MODULE_GLOBAL(ip6_use_defzone)) != 0) { SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } if (net == NULL) { memset(&ip6route, 0, sizeof(ip6route)); ro = (sctp_route_t *) & ip6route; memcpy(&ro->ro_dst, sin6, sin6->sin6_len); } else { ro = (sctp_route_t *) & net->ro; } /* * We assume here that inp_flow is in host byte * order within the TCB! */ if (tos_value == 0) { /* * This means especially, that it is not set * at the SCTP layer. So use the value from * the IP layer. */ tos_value = (ntohl(((struct in6pcb *)inp)->in6p_flowinfo) >> 20) & 0xff; } tos_value &= 0xfc; if (ecn_ok) { tos_value |= sctp_get_ect(stcb); } flowinfo = 0x06; flowinfo <<= 8; flowinfo |= tos_value; flowinfo <<= 20; flowinfo |= flowlabel; ip6h->ip6_flow = htonl(flowinfo); if (port) { ip6h->ip6_nxt = IPPROTO_UDP; } else { ip6h->ip6_nxt = IPPROTO_SCTP; } ip6h->ip6_plen = (packet_length - sizeof(struct ip6_hdr)); ip6h->ip6_dst = sin6->sin6_addr; /* * Add SRC address selection here: we can only reuse * to a limited degree the kame src-addr-sel, since * we can try their selection but it may not be * bound. */ bzero(&lsa6_tmp, sizeof(lsa6_tmp)); lsa6_tmp.sin6_family = AF_INET6; lsa6_tmp.sin6_len = sizeof(lsa6_tmp); lsa6 = &lsa6_tmp; if (net && out_of_asoc_ok == 0) { if (net->ro._s_addr && (net->ro._s_addr->localifa_flags & (SCTP_BEING_DELETED | SCTP_ADDR_IFA_UNUSEABLE))) { sctp_free_ifa(net->ro._s_addr); net->ro._s_addr = NULL; net->src_addr_selected = 0; if (ro->ro_rt) { RTFREE(ro->ro_rt); ro->ro_rt = NULL; } } if (net->src_addr_selected == 0) { sin6 = (struct sockaddr_in6 *)&net->ro._l_addr; /* KAME hack: embed scopeid */ if (sa6_embedscope(sin6, MODULE_GLOBAL(ip6_use_defzone)) != 0) { SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } /* Cache the source address */ net->ro._s_addr = sctp_source_address_selection(inp, stcb, ro, net, 0, vrf_id); (void)sa6_recoverscope(sin6); net->src_addr_selected = 1; } if (net->ro._s_addr == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "V6:No route to host\n"); net->src_addr_selected = 0; sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } lsa6->sin6_addr = net->ro._s_addr->address.sin6.sin6_addr; } else { sin6 = (struct sockaddr_in6 *)&ro->ro_dst; /* KAME hack: embed scopeid */ if (sa6_embedscope(sin6, MODULE_GLOBAL(ip6_use_defzone)) != 0) { SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } if (over_addr == NULL) { struct sctp_ifa *_lsrc; _lsrc = sctp_source_address_selection(inp, stcb, ro, net, out_of_asoc_ok, vrf_id); if (_lsrc == NULL) { sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } lsa6->sin6_addr = _lsrc->address.sin6.sin6_addr; sctp_free_ifa(_lsrc); } else { lsa6->sin6_addr = over_addr->sin6.sin6_addr; SCTP_RTALLOC(ro, vrf_id, inp->fibnum); } (void)sa6_recoverscope(sin6); } lsa6->sin6_port = inp->sctp_lport; if (ro->ro_rt == NULL) { /* * src addr selection failed to find a route * (or valid source addr), so we can't get * there from here! */ sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } /* * XXX: sa6 may not have a valid sin6_scope_id in * the non-SCOPEDROUTING case. */ bzero(&lsa6_storage, sizeof(lsa6_storage)); lsa6_storage.sin6_family = AF_INET6; lsa6_storage.sin6_len = sizeof(lsa6_storage); lsa6_storage.sin6_addr = lsa6->sin6_addr; if ((error = sa6_recoverscope(&lsa6_storage)) != 0) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "recover scope fails error %d\n", error); sctp_m_freem(m); return (error); } /* XXX */ lsa6_storage.sin6_addr = lsa6->sin6_addr; lsa6_storage.sin6_port = inp->sctp_lport; lsa6 = &lsa6_storage; ip6h->ip6_src = lsa6->sin6_addr; if (port) { if (htons(SCTP_BASE_SYSCTL(sctp_udp_tunneling_port)) == 0) { sctp_handle_no_route(stcb, net, so_locked); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EHOSTUNREACH); sctp_m_freem(m); return (EHOSTUNREACH); } udp = (struct udphdr *)((caddr_t)ip6h + sizeof(struct ip6_hdr)); udp->uh_sport = htons(SCTP_BASE_SYSCTL(sctp_udp_tunneling_port)); udp->uh_dport = port; udp->uh_ulen = htons(packet_length - sizeof(struct ip6_hdr)); udp->uh_sum = 0; sctphdr = (struct sctphdr *)((caddr_t)udp + sizeof(struct udphdr)); } else { sctphdr = (struct sctphdr *)((caddr_t)ip6h + sizeof(struct ip6_hdr)); } sctphdr->src_port = src_port; sctphdr->dest_port = dest_port; sctphdr->v_tag = v_tag; sctphdr->checksum = 0; /* * We set the hop limit now since there is a good * chance that our ro pointer is now filled */ ip6h->ip6_hlim = SCTP_GET_HLIM(inp, ro); ifp = SCTP_GET_IFN_VOID_FROM_ROUTE(ro); #ifdef SCTP_DEBUG /* Copy to be sure something bad is not happening */ sin6->sin6_addr = ip6h->ip6_dst; lsa6->sin6_addr = ip6h->ip6_src; #endif SCTPDBG(SCTP_DEBUG_OUTPUT3, "Calling ipv6 output routine from low level\n"); SCTPDBG(SCTP_DEBUG_OUTPUT3, "src: "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT3, (struct sockaddr *)lsa6); SCTPDBG(SCTP_DEBUG_OUTPUT3, "dst: "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT3, (struct sockaddr *)sin6); if (net) { sin6 = (struct sockaddr_in6 *)&net->ro._l_addr; /* * preserve the port and scope for link * local send */ prev_scope = sin6->sin6_scope_id; prev_port = sin6->sin6_port; } if (SCTP_GET_HEADER_FOR_OUTPUT(o_pak)) { /* failed to prepend data, give up */ sctp_m_freem(m); SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } SCTP_ATTACH_CHAIN(o_pak, m, packet_length); if (port) { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else sctphdr->checksum = sctp_calculate_cksum(m, sizeof(struct ip6_hdr) + sizeof(struct udphdr)); SCTP_STAT_INCR(sctps_sendswcrc); #endif if ((udp->uh_sum = in6_cksum(o_pak, IPPROTO_UDP, sizeof(struct ip6_hdr), packet_length - sizeof(struct ip6_hdr))) == 0) { udp->uh_sum = 0xffff; } } else { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else m->m_pkthdr.csum_flags = CSUM_SCTP_IPV6; m->m_pkthdr.csum_data = offsetof(struct sctphdr, checksum); SCTP_STAT_INCR(sctps_sendhwcrc); #endif } /* send it out. table id is taken from stcb */ #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) if ((SCTP_BASE_SYSCTL(sctp_output_unlocked)) && (so_locked)) { so = SCTP_INP_SO(inp); SCTP_SOCKET_UNLOCK(so, 0); } #endif #ifdef SCTP_PACKET_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LAST_PACKET_TRACING) sctp_packet_log(o_pak); #endif SCTP_IP6_OUTPUT(ret, o_pak, (struct route_in6 *)ro, &ifp, stcb, vrf_id); #if defined(__APPLE__) || defined(SCTP_SO_LOCK_TESTING) if ((SCTP_BASE_SYSCTL(sctp_output_unlocked)) && (so_locked)) { atomic_add_int(&stcb->asoc.refcnt, 1); SCTP_TCB_UNLOCK(stcb); SCTP_SOCKET_LOCK(so, 0); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); } #endif if (net) { /* for link local this must be done */ sin6->sin6_scope_id = prev_scope; sin6->sin6_port = prev_port; } SCTPDBG(SCTP_DEBUG_OUTPUT3, "return from send is %d\n", ret); SCTP_STAT_INCR(sctps_sendpackets); SCTP_STAT_INCR_COUNTER64(sctps_outpackets); if (ret) { SCTP_STAT_INCR(sctps_senderrors); } if (net == NULL) { /* Now if we had a temp route free it */ RO_RTFREE(ro); } else { /* * PMTU check versus smallest asoc MTU goes * here */ if (ro->ro_rt == NULL) { /* Route was freed */ if (net->ro._s_addr && net->src_addr_selected) { sctp_free_ifa(net->ro._s_addr); net->ro._s_addr = NULL; } net->src_addr_selected = 0; } if ((ro->ro_rt != NULL) && (net->ro._s_addr)) { uint32_t mtu; mtu = SCTP_GATHER_MTU_FROM_ROUTE(net->ro._s_addr, &net->ro._l_addr.sa, ro->ro_rt); if (mtu && (stcb->asoc.smallest_mtu > mtu)) { sctp_mtu_size_reset(inp, &stcb->asoc, mtu); net->mtu = mtu; if (net->port) { net->mtu -= sizeof(struct udphdr); } } } else if (ifp) { if (ND_IFINFO(ifp)->linkmtu && (stcb->asoc.smallest_mtu > ND_IFINFO(ifp)->linkmtu)) { sctp_mtu_size_reset(inp, &stcb->asoc, ND_IFINFO(ifp)->linkmtu); } } } return (ret); } #endif default: SCTPDBG(SCTP_DEBUG_OUTPUT1, "Unknown protocol (TSNH) type %d\n", ((struct sockaddr *)to)->sa_family); sctp_m_freem(m); SCTP_LTRACE_ERR_RET_PKT(m, inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EFAULT); return (EFAULT); } } void sctp_send_initiate(struct sctp_inpcb *inp, struct sctp_tcb *stcb, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { struct mbuf *m, *m_last; struct sctp_nets *net; struct sctp_init_chunk *init; struct sctp_supported_addr_param *sup_addr; struct sctp_adaptation_layer_indication *ali; struct sctp_supported_chunk_types_param *pr_supported; struct sctp_paramhdr *ph; int cnt_inits_to = 0; int ret; uint16_t num_ext, chunk_len, padding_len, parameter_len; /* INIT's always go to the primary (and usually ONLY address) */ net = stcb->asoc.primary_destination; if (net == NULL) { net = TAILQ_FIRST(&stcb->asoc.nets); if (net == NULL) { /* TSNH */ return; } /* we confirm any address we send an INIT to */ net->dest_state &= ~SCTP_ADDR_UNCONFIRMED; (void)sctp_set_primary_addr(stcb, NULL, net); } else { /* we confirm any address we send an INIT to */ net->dest_state &= ~SCTP_ADDR_UNCONFIRMED; } SCTPDBG(SCTP_DEBUG_OUTPUT4, "Sending INIT\n"); #ifdef INET6 if (net->ro._l_addr.sa.sa_family == AF_INET6) { /* * special hook, if we are sending to link local it will not * show up in our private address count. */ if (IN6_IS_ADDR_LINKLOCAL(&net->ro._l_addr.sin6.sin6_addr)) cnt_inits_to = 1; } #endif if (SCTP_OS_TIMER_PENDING(&net->rxt_timer.timer)) { /* This case should not happen */ SCTPDBG(SCTP_DEBUG_OUTPUT4, "Sending INIT - failed timer?\n"); return; } /* start the INIT timer */ sctp_timer_start(SCTP_TIMER_TYPE_INIT, inp, stcb, net); m = sctp_get_mbuf_for_msg(MCLBYTES, 1, M_NOWAIT, 1, MT_DATA); if (m == NULL) { /* No memory, INIT timer will re-attempt. */ SCTPDBG(SCTP_DEBUG_OUTPUT4, "Sending INIT - mbuf?\n"); return; } chunk_len = (uint16_t) sizeof(struct sctp_init_chunk); padding_len = 0; /* Now lets put the chunk header in place */ init = mtod(m, struct sctp_init_chunk *); /* now the chunk header */ init->ch.chunk_type = SCTP_INITIATION; init->ch.chunk_flags = 0; /* fill in later from mbuf we build */ init->ch.chunk_length = 0; /* place in my tag */ init->init.initiate_tag = htonl(stcb->asoc.my_vtag); /* set up some of the credits. */ init->init.a_rwnd = htonl(max(inp->sctp_socket ? SCTP_SB_LIMIT_RCV(inp->sctp_socket) : 0, SCTP_MINIMAL_RWND)); init->init.num_outbound_streams = htons(stcb->asoc.pre_open_streams); init->init.num_inbound_streams = htons(stcb->asoc.max_inbound_streams); init->init.initial_tsn = htonl(stcb->asoc.init_seq_number); /* Adaptation layer indication parameter */ if (inp->sctp_ep.adaptation_layer_indicator_provided) { parameter_len = (uint16_t) sizeof(struct sctp_adaptation_layer_indication); ali = (struct sctp_adaptation_layer_indication *)(mtod(m, caddr_t)+chunk_len); ali->ph.param_type = htons(SCTP_ULP_ADAPTATION); ali->ph.param_length = htons(parameter_len); ali->indication = htonl(inp->sctp_ep.adaptation_layer_indicator); chunk_len += parameter_len; } /* ECN parameter */ if (stcb->asoc.ecn_supported == 1) { parameter_len = (uint16_t) sizeof(struct sctp_paramhdr); ph = (struct sctp_paramhdr *)(mtod(m, caddr_t)+chunk_len); ph->param_type = htons(SCTP_ECN_CAPABLE); ph->param_length = htons(parameter_len); chunk_len += parameter_len; } /* PR-SCTP supported parameter */ if (stcb->asoc.prsctp_supported == 1) { parameter_len = (uint16_t) sizeof(struct sctp_paramhdr); ph = (struct sctp_paramhdr *)(mtod(m, caddr_t)+chunk_len); ph->param_type = htons(SCTP_PRSCTP_SUPPORTED); ph->param_length = htons(parameter_len); chunk_len += parameter_len; } /* Add NAT friendly parameter. */ if (SCTP_BASE_SYSCTL(sctp_inits_include_nat_friendly)) { parameter_len = (uint16_t) sizeof(struct sctp_paramhdr); ph = (struct sctp_paramhdr *)(mtod(m, caddr_t)+chunk_len); ph->param_type = htons(SCTP_HAS_NAT_SUPPORT); ph->param_length = htons(parameter_len); chunk_len += parameter_len; } /* And now tell the peer which extensions we support */ num_ext = 0; pr_supported = (struct sctp_supported_chunk_types_param *)(mtod(m, caddr_t)+chunk_len); if (stcb->asoc.prsctp_supported == 1) { pr_supported->chunk_types[num_ext++] = SCTP_FORWARD_CUM_TSN; } if (stcb->asoc.auth_supported == 1) { pr_supported->chunk_types[num_ext++] = SCTP_AUTHENTICATION; } if (stcb->asoc.asconf_supported == 1) { pr_supported->chunk_types[num_ext++] = SCTP_ASCONF; pr_supported->chunk_types[num_ext++] = SCTP_ASCONF_ACK; } if (stcb->asoc.reconfig_supported == 1) { pr_supported->chunk_types[num_ext++] = SCTP_STREAM_RESET; } if (stcb->asoc.nrsack_supported == 1) { pr_supported->chunk_types[num_ext++] = SCTP_NR_SELECTIVE_ACK; } if (stcb->asoc.pktdrop_supported == 1) { pr_supported->chunk_types[num_ext++] = SCTP_PACKET_DROPPED; } if (num_ext > 0) { parameter_len = (uint16_t) sizeof(struct sctp_supported_chunk_types_param) + num_ext; pr_supported->ph.param_type = htons(SCTP_SUPPORTED_CHUNK_EXT); pr_supported->ph.param_length = htons(parameter_len); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; } /* add authentication parameters */ if (stcb->asoc.auth_supported) { /* attach RANDOM parameter, if available */ if (stcb->asoc.authinfo.random != NULL) { struct sctp_auth_random *randp; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } randp = (struct sctp_auth_random *)(mtod(m, caddr_t)+chunk_len); parameter_len = (uint16_t) sizeof(struct sctp_auth_random) + stcb->asoc.authinfo.random_len; /* random key already contains the header */ memcpy(randp, stcb->asoc.authinfo.random->key, parameter_len); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; } /* add HMAC_ALGO parameter */ if (stcb->asoc.local_hmacs != NULL) { struct sctp_auth_hmac_algo *hmacs; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } hmacs = (struct sctp_auth_hmac_algo *)(mtod(m, caddr_t)+chunk_len); parameter_len = (uint16_t) (sizeof(struct sctp_auth_hmac_algo) + stcb->asoc.local_hmacs->num_algo * sizeof(uint16_t)); hmacs->ph.param_type = htons(SCTP_HMAC_LIST); hmacs->ph.param_length = htons(parameter_len); sctp_serialize_hmaclist(stcb->asoc.local_hmacs, (uint8_t *) hmacs->hmac_ids); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; } /* add CHUNKS parameter */ if (stcb->asoc.local_auth_chunks != NULL) { struct sctp_auth_chunk_list *chunks; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } chunks = (struct sctp_auth_chunk_list *)(mtod(m, caddr_t)+chunk_len); parameter_len = (uint16_t) (sizeof(struct sctp_auth_chunk_list) + sctp_auth_get_chklist_size(stcb->asoc.local_auth_chunks)); chunks->ph.param_type = htons(SCTP_CHUNK_LIST); chunks->ph.param_length = htons(parameter_len); sctp_serialize_auth_chunks(stcb->asoc.local_auth_chunks, chunks->chunk_types); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; } } /* now any cookie time extensions */ if (stcb->asoc.cookie_preserve_req) { struct sctp_cookie_perserve_param *cookie_preserve; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } parameter_len = (uint16_t) sizeof(struct sctp_cookie_perserve_param); cookie_preserve = (struct sctp_cookie_perserve_param *)(mtod(m, caddr_t)+chunk_len); cookie_preserve->ph.param_type = htons(SCTP_COOKIE_PRESERVE); cookie_preserve->ph.param_length = htons(parameter_len); cookie_preserve->time = htonl(stcb->asoc.cookie_preserve_req); stcb->asoc.cookie_preserve_req = 0; chunk_len += parameter_len; } if (stcb->asoc.scope.ipv4_addr_legal || stcb->asoc.scope.ipv6_addr_legal) { uint8_t i; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } parameter_len = (uint16_t) sizeof(struct sctp_paramhdr); if (stcb->asoc.scope.ipv4_addr_legal) { parameter_len += (uint16_t) sizeof(uint16_t); } if (stcb->asoc.scope.ipv6_addr_legal) { parameter_len += (uint16_t) sizeof(uint16_t); } sup_addr = (struct sctp_supported_addr_param *)(mtod(m, caddr_t)+chunk_len); sup_addr->ph.param_type = htons(SCTP_SUPPORTED_ADDRTYPE); sup_addr->ph.param_length = htons(parameter_len); i = 0; if (stcb->asoc.scope.ipv4_addr_legal) { sup_addr->addr_type[i++] = htons(SCTP_IPV4_ADDRESS); } if (stcb->asoc.scope.ipv6_addr_legal) { sup_addr->addr_type[i++] = htons(SCTP_IPV6_ADDRESS); } padding_len = 4 - 2 * i; chunk_len += parameter_len; } SCTP_BUF_LEN(m) = chunk_len; /* now the addresses */ /* * To optimize this we could put the scoping stuff into a structure * and remove the individual uint8's from the assoc structure. Then * we could just sifa in the address within the stcb. But for now * this is a quick hack to get the address stuff teased apart. */ m_last = sctp_add_addresses_to_i_ia(inp, stcb, &stcb->asoc.scope, m, cnt_inits_to, &padding_len, &chunk_len); init->ch.chunk_length = htons(chunk_len); if (padding_len > 0) { if (sctp_add_pad_tombuf(m_last, padding_len) == NULL) { sctp_m_freem(m); return; } } SCTPDBG(SCTP_DEBUG_OUTPUT4, "Sending INIT - calls lowlevel_output\n"); ret = sctp_lowlevel_chunk_output(inp, stcb, net, (struct sockaddr *)&net->ro._l_addr, m, 0, NULL, 0, 0, 0, 0, inp->sctp_lport, stcb->rport, htonl(0), net->port, NULL, 0, 0, so_locked); SCTPDBG(SCTP_DEBUG_OUTPUT4, "lowlevel_output - %d\n", ret); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); (void)SCTP_GETTIME_TIMEVAL(&net->last_sent_time); } struct mbuf * sctp_arethere_unrecognized_parameters(struct mbuf *in_initpkt, int param_offset, int *abort_processing, struct sctp_chunkhdr *cp, int *nat_friendly) { /* * Given a mbuf containing an INIT or INIT-ACK with the param_offset * being equal to the beginning of the params i.e. (iphlen + * sizeof(struct sctp_init_msg) parse through the parameters to the * end of the mbuf verifying that all parameters are known. * * For unknown parameters build and return a mbuf with * UNRECOGNIZED_PARAMETER errors. If the flags indicate to stop * processing this chunk stop, and set *abort_processing to 1. * * By having param_offset be pre-set to where parameters begin it is * hoped that this routine may be reused in the future by new * features. */ struct sctp_paramhdr *phdr, params; struct mbuf *mat, *op_err; char tempbuf[SCTP_PARAM_BUFFER_SIZE]; int at, limit, pad_needed; uint16_t ptype, plen, padded_size; int err_at; *abort_processing = 0; mat = in_initpkt; err_at = 0; limit = ntohs(cp->chunk_length) - sizeof(struct sctp_init_chunk); at = param_offset; op_err = NULL; SCTPDBG(SCTP_DEBUG_OUTPUT1, "Check for unrecognized param's\n"); phdr = sctp_get_next_param(mat, at, ¶ms, sizeof(params)); while ((phdr != NULL) && ((size_t)limit >= sizeof(struct sctp_paramhdr))) { ptype = ntohs(phdr->param_type); plen = ntohs(phdr->param_length); if ((plen > limit) || (plen < sizeof(struct sctp_paramhdr))) { /* wacked parameter */ SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error %d\n", plen); goto invalid_size; } limit -= SCTP_SIZE32(plen); /*- * All parameters for all chunks that we know/understand are * listed here. We process them other places and make * appropriate stop actions per the upper bits. However this * is the generic routine processor's can call to get back * an operr.. to either incorporate (init-ack) or send. */ padded_size = SCTP_SIZE32(plen); switch (ptype) { /* Param's with variable size */ case SCTP_HEARTBEAT_INFO: case SCTP_STATE_COOKIE: case SCTP_UNRECOG_PARAM: case SCTP_ERROR_CAUSE_IND: /* ok skip fwd */ at += padded_size; break; /* Param's with variable size within a range */ case SCTP_CHUNK_LIST: case SCTP_SUPPORTED_CHUNK_EXT: if (padded_size > (sizeof(struct sctp_supported_chunk_types_param) + (sizeof(uint8_t) * SCTP_MAX_SUPPORTED_EXT))) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error chklist %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_SUPPORTED_ADDRTYPE: if (padded_size > SCTP_MAX_ADDR_PARAMS_SIZE) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error supaddrtype %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_RANDOM: if (padded_size > (sizeof(struct sctp_auth_random) + SCTP_RANDOM_MAX_SIZE)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error random %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_SET_PRIM_ADDR: case SCTP_DEL_IP_ADDRESS: case SCTP_ADD_IP_ADDRESS: if ((padded_size != sizeof(struct sctp_asconf_addrv4_param)) && (padded_size != sizeof(struct sctp_asconf_addr_param))) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error setprim %d\n", plen); goto invalid_size; } at += padded_size; break; /* Param's with a fixed size */ case SCTP_IPV4_ADDRESS: if (padded_size != sizeof(struct sctp_ipv4addr_param)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error ipv4 addr %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_IPV6_ADDRESS: if (padded_size != sizeof(struct sctp_ipv6addr_param)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error ipv6 addr %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_COOKIE_PRESERVE: if (padded_size != sizeof(struct sctp_cookie_perserve_param)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error cookie-preserve %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_HAS_NAT_SUPPORT: *nat_friendly = 1; /* fall through */ case SCTP_PRSCTP_SUPPORTED: if (padded_size != sizeof(struct sctp_paramhdr)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error prsctp/nat support %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_ECN_CAPABLE: if (padded_size != sizeof(struct sctp_paramhdr)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error ecn %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_ULP_ADAPTATION: if (padded_size != sizeof(struct sctp_adaptation_layer_indication)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error adapatation %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_SUCCESS_REPORT: if (padded_size != sizeof(struct sctp_asconf_paramhdr)) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Invalid size - error success %d\n", plen); goto invalid_size; } at += padded_size; break; case SCTP_HOSTNAME_ADDRESS: { /* We can NOT handle HOST NAME addresses!! */ int l_len; SCTPDBG(SCTP_DEBUG_OUTPUT1, "Can't handle hostname addresses.. abort processing\n"); *abort_processing = 1; if (op_err == NULL) { /* Ok need to try to get a mbuf */ #ifdef INET6 l_len = SCTP_MIN_OVERHEAD; #else l_len = SCTP_MIN_V4_OVERHEAD; #endif l_len += sizeof(struct sctp_chunkhdr); l_len += plen; l_len += sizeof(struct sctp_paramhdr); op_err = sctp_get_mbuf_for_msg(l_len, 0, M_NOWAIT, 1, MT_DATA); if (op_err) { SCTP_BUF_LEN(op_err) = 0; /* * pre-reserve space for ip * and sctp header and * chunk hdr */ #ifdef INET6 SCTP_BUF_RESV_UF(op_err, sizeof(struct ip6_hdr)); #else SCTP_BUF_RESV_UF(op_err, sizeof(struct ip)); #endif SCTP_BUF_RESV_UF(op_err, sizeof(struct sctphdr)); SCTP_BUF_RESV_UF(op_err, sizeof(struct sctp_chunkhdr)); } } if (op_err) { /* If we have space */ struct sctp_paramhdr s; if (err_at % 4) { uint32_t cpthis = 0; pad_needed = 4 - (err_at % 4); m_copyback(op_err, err_at, pad_needed, (caddr_t)&cpthis); err_at += pad_needed; } s.param_type = htons(SCTP_CAUSE_UNRESOLVABLE_ADDR); s.param_length = htons(sizeof(s) + plen); m_copyback(op_err, err_at, sizeof(s), (caddr_t)&s); err_at += sizeof(s); phdr = sctp_get_next_param(mat, at, (struct sctp_paramhdr *)tempbuf, min(sizeof(tempbuf), plen)); if (phdr == NULL) { sctp_m_freem(op_err); /* * we are out of memory but * we still need to have a * look at what to do (the * system is in trouble * though). */ return (NULL); } m_copyback(op_err, err_at, plen, (caddr_t)phdr); } return (op_err); break; } default: /* * we do not recognize the parameter figure out what * we do. */ SCTPDBG(SCTP_DEBUG_OUTPUT1, "Hit default param %x\n", ptype); if ((ptype & 0x4000) == 0x4000) { /* Report bit is set?? */ SCTPDBG(SCTP_DEBUG_OUTPUT1, "report op err\n"); if (op_err == NULL) { int l_len; /* Ok need to try to get an mbuf */ #ifdef INET6 l_len = SCTP_MIN_OVERHEAD; #else l_len = SCTP_MIN_V4_OVERHEAD; #endif l_len += sizeof(struct sctp_chunkhdr); l_len += plen; l_len += sizeof(struct sctp_paramhdr); op_err = sctp_get_mbuf_for_msg(l_len, 0, M_NOWAIT, 1, MT_DATA); if (op_err) { SCTP_BUF_LEN(op_err) = 0; #ifdef INET6 SCTP_BUF_RESV_UF(op_err, sizeof(struct ip6_hdr)); #else SCTP_BUF_RESV_UF(op_err, sizeof(struct ip)); #endif SCTP_BUF_RESV_UF(op_err, sizeof(struct sctphdr)); SCTP_BUF_RESV_UF(op_err, sizeof(struct sctp_chunkhdr)); } } if (op_err) { /* If we have space */ struct sctp_paramhdr s; if (err_at % 4) { uint32_t cpthis = 0; pad_needed = 4 - (err_at % 4); m_copyback(op_err, err_at, pad_needed, (caddr_t)&cpthis); err_at += pad_needed; } s.param_type = htons(SCTP_UNRECOG_PARAM); s.param_length = htons(sizeof(s) + plen); m_copyback(op_err, err_at, sizeof(s), (caddr_t)&s); err_at += sizeof(s); if (plen > sizeof(tempbuf)) { plen = sizeof(tempbuf); } phdr = sctp_get_next_param(mat, at, (struct sctp_paramhdr *)tempbuf, min(sizeof(tempbuf), plen)); if (phdr == NULL) { sctp_m_freem(op_err); /* * we are out of memory but * we still need to have a * look at what to do (the * system is in trouble * though). */ op_err = NULL; goto more_processing; } m_copyback(op_err, err_at, plen, (caddr_t)phdr); err_at += plen; } } more_processing: if ((ptype & 0x8000) == 0x0000) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "stop proc\n"); return (op_err); } else { /* skip this chunk and continue processing */ SCTPDBG(SCTP_DEBUG_OUTPUT1, "move on\n"); at += SCTP_SIZE32(plen); } break; } phdr = sctp_get_next_param(mat, at, ¶ms, sizeof(params)); } return (op_err); invalid_size: SCTPDBG(SCTP_DEBUG_OUTPUT1, "abort flag set\n"); *abort_processing = 1; if ((op_err == NULL) && phdr) { int l_len; #ifdef INET6 l_len = SCTP_MIN_OVERHEAD; #else l_len = SCTP_MIN_V4_OVERHEAD; #endif l_len += sizeof(struct sctp_chunkhdr); l_len += (2 * sizeof(struct sctp_paramhdr)); op_err = sctp_get_mbuf_for_msg(l_len, 0, M_NOWAIT, 1, MT_DATA); if (op_err) { SCTP_BUF_LEN(op_err) = 0; #ifdef INET6 SCTP_BUF_RESV_UF(op_err, sizeof(struct ip6_hdr)); #else SCTP_BUF_RESV_UF(op_err, sizeof(struct ip)); #endif SCTP_BUF_RESV_UF(op_err, sizeof(struct sctphdr)); SCTP_BUF_RESV_UF(op_err, sizeof(struct sctp_chunkhdr)); } } if ((op_err) && phdr) { struct sctp_paramhdr s; if (err_at % 4) { uint32_t cpthis = 0; pad_needed = 4 - (err_at % 4); m_copyback(op_err, err_at, pad_needed, (caddr_t)&cpthis); err_at += pad_needed; } s.param_type = htons(SCTP_CAUSE_PROTOCOL_VIOLATION); s.param_length = htons(sizeof(s) + sizeof(struct sctp_paramhdr)); m_copyback(op_err, err_at, sizeof(s), (caddr_t)&s); err_at += sizeof(s); /* Only copy back the p-hdr that caused the issue */ m_copyback(op_err, err_at, sizeof(struct sctp_paramhdr), (caddr_t)phdr); } return (op_err); } static int sctp_are_there_new_addresses(struct sctp_association *asoc, struct mbuf *in_initpkt, int offset, struct sockaddr *src) { /* * Given a INIT packet, look through the packet to verify that there * are NO new addresses. As we go through the parameters add reports * of any un-understood parameters that require an error. Also we * must return (1) to drop the packet if we see a un-understood * parameter that tells us to drop the chunk. */ struct sockaddr *sa_touse; struct sockaddr *sa; struct sctp_paramhdr *phdr, params; uint16_t ptype, plen; uint8_t fnd; struct sctp_nets *net; + int check_src; #ifdef INET struct sockaddr_in sin4, *sa4; #endif #ifdef INET6 struct sockaddr_in6 sin6, *sa6; #endif #ifdef INET memset(&sin4, 0, sizeof(sin4)); sin4.sin_family = AF_INET; sin4.sin_len = sizeof(sin4); #endif #ifdef INET6 memset(&sin6, 0, sizeof(sin6)); sin6.sin6_family = AF_INET6; sin6.sin6_len = sizeof(sin6); #endif /* First what about the src address of the pkt ? */ - fnd = 0; - TAILQ_FOREACH(net, &asoc->nets, sctp_next) { - sa = (struct sockaddr *)&net->ro._l_addr; - if (sa->sa_family == src->sa_family) { + check_src = 0; + switch (src->sa_family) { #ifdef INET - if (sa->sa_family == AF_INET) { - struct sockaddr_in *src4; + case AF_INET: + if (asoc->scope.ipv4_addr_legal) { + check_src = 1; + } + break; +#endif +#ifdef INET6 + case AF_INET6: + if (asoc->scope.ipv6_addr_legal) { + check_src = 1; + } + break; +#endif + default: + /* TSNH */ + break; + } + if (check_src) { + fnd = 0; + TAILQ_FOREACH(net, &asoc->nets, sctp_next) { + sa = (struct sockaddr *)&net->ro._l_addr; + if (sa->sa_family == src->sa_family) { +#ifdef INET + if (sa->sa_family == AF_INET) { + struct sockaddr_in *src4; - sa4 = (struct sockaddr_in *)sa; - src4 = (struct sockaddr_in *)src; - if (sa4->sin_addr.s_addr == src4->sin_addr.s_addr) { - fnd = 1; - break; + sa4 = (struct sockaddr_in *)sa; + src4 = (struct sockaddr_in *)src; + if (sa4->sin_addr.s_addr == src4->sin_addr.s_addr) { + fnd = 1; + break; + } } - } #endif #ifdef INET6 - if (sa->sa_family == AF_INET6) { - struct sockaddr_in6 *src6; + if (sa->sa_family == AF_INET6) { + struct sockaddr_in6 *src6; - sa6 = (struct sockaddr_in6 *)sa; - src6 = (struct sockaddr_in6 *)src; - if (SCTP6_ARE_ADDR_EQUAL(sa6, src6)) { - fnd = 1; - break; + sa6 = (struct sockaddr_in6 *)sa; + src6 = (struct sockaddr_in6 *)src; + if (SCTP6_ARE_ADDR_EQUAL(sa6, src6)) { + fnd = 1; + break; + } } - } #endif + } } + if (fnd == 0) { + /* New address added! no need to look futher. */ + return (1); + } } - if (fnd == 0) { - /* New address added! no need to look futher. */ - return (1); - } /* Ok so far lets munge through the rest of the packet */ offset += sizeof(struct sctp_init_chunk); phdr = sctp_get_next_param(in_initpkt, offset, ¶ms, sizeof(params)); while (phdr) { sa_touse = NULL; ptype = ntohs(phdr->param_type); plen = ntohs(phdr->param_length); switch (ptype) { #ifdef INET case SCTP_IPV4_ADDRESS: { struct sctp_ipv4addr_param *p4, p4_buf; phdr = sctp_get_next_param(in_initpkt, offset, (struct sctp_paramhdr *)&p4_buf, sizeof(p4_buf)); if (plen != sizeof(struct sctp_ipv4addr_param) || phdr == NULL) { return (1); } - p4 = (struct sctp_ipv4addr_param *)phdr; - sin4.sin_addr.s_addr = p4->addr; - sa_touse = (struct sockaddr *)&sin4; + if (asoc->scope.ipv4_addr_legal) { + p4 = (struct sctp_ipv4addr_param *)phdr; + sin4.sin_addr.s_addr = p4->addr; + sa_touse = (struct sockaddr *)&sin4; + } break; } #endif #ifdef INET6 case SCTP_IPV6_ADDRESS: { struct sctp_ipv6addr_param *p6, p6_buf; phdr = sctp_get_next_param(in_initpkt, offset, (struct sctp_paramhdr *)&p6_buf, sizeof(p6_buf)); if (plen != sizeof(struct sctp_ipv6addr_param) || phdr == NULL) { return (1); } - p6 = (struct sctp_ipv6addr_param *)phdr; - memcpy((caddr_t)&sin6.sin6_addr, p6->addr, - sizeof(p6->addr)); - sa_touse = (struct sockaddr *)&sin6; + if (asoc->scope.ipv6_addr_legal) { + p6 = (struct sctp_ipv6addr_param *)phdr; + memcpy((caddr_t)&sin6.sin6_addr, p6->addr, + sizeof(p6->addr)); + sa_touse = (struct sockaddr *)&sin6; + } break; } #endif default: sa_touse = NULL; break; } if (sa_touse) { /* ok, sa_touse points to one to check */ fnd = 0; TAILQ_FOREACH(net, &asoc->nets, sctp_next) { sa = (struct sockaddr *)&net->ro._l_addr; if (sa->sa_family != sa_touse->sa_family) { continue; } #ifdef INET if (sa->sa_family == AF_INET) { sa4 = (struct sockaddr_in *)sa; if (sa4->sin_addr.s_addr == sin4.sin_addr.s_addr) { fnd = 1; break; } } #endif #ifdef INET6 if (sa->sa_family == AF_INET6) { sa6 = (struct sockaddr_in6 *)sa; if (SCTP6_ARE_ADDR_EQUAL( sa6, &sin6)) { fnd = 1; break; } } #endif } if (!fnd) { /* New addr added! no need to look further */ return (1); } } offset += SCTP_SIZE32(plen); phdr = sctp_get_next_param(in_initpkt, offset, ¶ms, sizeof(params)); } return (0); } /* * Given a MBUF chain that was sent into us containing an INIT. Build a * INIT-ACK with COOKIE and send back. We assume that the in_initpkt has done * a pullup to include IPv6/4header, SCTP header and initial part of INIT * message (i.e. the struct sctp_init_msg). */ void sctp_send_initiate_ack(struct sctp_inpcb *inp, struct sctp_tcb *stcb, - struct mbuf *init_pkt, int iphlen, int offset, + struct sctp_nets *src_net, struct mbuf *init_pkt, + int iphlen, int offset, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, struct sctp_init_chunk *init_chk, uint8_t mflowtype, uint32_t mflowid, uint32_t vrf_id, uint16_t port, int hold_inp_lock) { struct sctp_association *asoc; struct mbuf *m, *m_tmp, *m_last, *m_cookie, *op_err; struct sctp_init_ack_chunk *initack; struct sctp_adaptation_layer_indication *ali; struct sctp_supported_chunk_types_param *pr_supported; struct sctp_paramhdr *ph; union sctp_sockstore *over_addr; struct sctp_scoping scp; #ifdef INET struct sockaddr_in *dst4 = (struct sockaddr_in *)dst; struct sockaddr_in *src4 = (struct sockaddr_in *)src; struct sockaddr_in *sin; #endif #ifdef INET6 struct sockaddr_in6 *dst6 = (struct sockaddr_in6 *)dst; struct sockaddr_in6 *src6 = (struct sockaddr_in6 *)src; struct sockaddr_in6 *sin6; #endif struct sockaddr *to; struct sctp_state_cookie stc; struct sctp_nets *net = NULL; uint8_t *signature = NULL; int cnt_inits_to = 0; uint16_t his_limit, i_want; int abort_flag; int nat_friendly = 0; struct socket *so; uint16_t num_ext, chunk_len, padding_len, parameter_len; if (stcb) { asoc = &stcb->asoc; } else { asoc = NULL; } if ((asoc != NULL) && - (SCTP_GET_STATE(asoc) != SCTP_STATE_COOKIE_WAIT) && - (sctp_are_there_new_addresses(asoc, init_pkt, offset, src))) { - /* new addresses, out of here in non-cookie-wait states */ - /* - * Send a ABORT, we don't add the new address error clause - * though we even set the T bit and copy in the 0 tag.. this - * looks no different than if no listener was present. - */ - op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), - "Address added"); - sctp_send_abort(init_pkt, iphlen, src, dst, sh, 0, op_err, - mflowtype, mflowid, inp->fibnum, - vrf_id, port); - return; + (SCTP_GET_STATE(asoc) != SCTP_STATE_COOKIE_WAIT)) { + if (sctp_are_there_new_addresses(asoc, init_pkt, offset, src)) { + /* + * new addresses, out of here in non-cookie-wait + * states + * + * Send an ABORT, without the new address error cause. + * This looks no different than if no listener was + * present. + */ + op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), + "Address added"); + sctp_send_abort(init_pkt, iphlen, src, dst, sh, 0, op_err, + mflowtype, mflowid, inp->fibnum, + vrf_id, port); + return; + } + if (src_net != NULL && (src_net->port != port)) { + /* + * change of remote encapsulation port, out of here + * in non-cookie-wait states + * + * Send an ABORT, without an specific error cause. This + * looks no different than if no listener was + * present. + */ + op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), + "Remote encapsulation port changed"); + sctp_send_abort(init_pkt, iphlen, src, dst, sh, 0, op_err, + mflowtype, mflowid, inp->fibnum, + vrf_id, port); + return; + } } abort_flag = 0; op_err = sctp_arethere_unrecognized_parameters(init_pkt, (offset + sizeof(struct sctp_init_chunk)), &abort_flag, (struct sctp_chunkhdr *)init_chk, &nat_friendly); if (abort_flag) { do_a_abort: if (op_err == NULL) { char msg[SCTP_DIAG_INFO_LEN]; snprintf(msg, sizeof(msg), "%s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); } sctp_send_abort(init_pkt, iphlen, src, dst, sh, init_chk->init.initiate_tag, op_err, mflowtype, mflowid, inp->fibnum, vrf_id, port); return; } m = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (m == NULL) { /* No memory, INIT timer will re-attempt. */ if (op_err) sctp_m_freem(op_err); return; } chunk_len = (uint16_t) sizeof(struct sctp_init_ack_chunk); padding_len = 0; /* * We might not overwrite the identification[] completely and on * some platforms time_entered will contain some padding. Therefore * zero out the cookie to avoid putting uninitialized memory on the * wire. */ memset(&stc, 0, sizeof(struct sctp_state_cookie)); /* the time I built cookie */ (void)SCTP_GETTIME_TIMEVAL(&stc.time_entered); /* populate any tie tags */ if (asoc != NULL) { /* unlock before tag selections */ stc.tie_tag_my_vtag = asoc->my_vtag_nonce; stc.tie_tag_peer_vtag = asoc->peer_vtag_nonce; stc.cookie_life = asoc->cookie_life; net = asoc->primary_destination; } else { stc.tie_tag_my_vtag = 0; stc.tie_tag_peer_vtag = 0; /* life I will award this cookie */ stc.cookie_life = inp->sctp_ep.def_cookie_life; } /* copy in the ports for later check */ stc.myport = sh->dest_port; stc.peerport = sh->src_port; /* * If we wanted to honor cookie life extentions, we would add to * stc.cookie_life. For now we should NOT honor any extension */ stc.site_scope = stc.local_scope = stc.loopback_scope = 0; if (inp->sctp_flags & SCTP_PCB_FLAGS_BOUND_V6) { stc.ipv6_addr_legal = 1; if (SCTP_IPV6_V6ONLY(inp)) { stc.ipv4_addr_legal = 0; } else { stc.ipv4_addr_legal = 1; } } else { stc.ipv6_addr_legal = 0; stc.ipv4_addr_legal = 1; } stc.ipv4_scope = 0; if (net == NULL) { to = src; switch (dst->sa_family) { #ifdef INET case AF_INET: { /* lookup address */ stc.address[0] = src4->sin_addr.s_addr; stc.address[1] = 0; stc.address[2] = 0; stc.address[3] = 0; stc.addr_type = SCTP_IPV4_ADDRESS; /* local from address */ stc.laddress[0] = dst4->sin_addr.s_addr; stc.laddress[1] = 0; stc.laddress[2] = 0; stc.laddress[3] = 0; stc.laddr_type = SCTP_IPV4_ADDRESS; /* scope_id is only for v6 */ stc.scope_id = 0; if ((IN4_ISPRIVATE_ADDRESS(&src4->sin_addr)) || (IN4_ISPRIVATE_ADDRESS(&dst4->sin_addr))) { stc.ipv4_scope = 1; } /* Must use the address in this case */ if (sctp_is_address_on_local_host(src, vrf_id)) { stc.loopback_scope = 1; stc.ipv4_scope = 1; stc.site_scope = 1; stc.local_scope = 0; } break; } #endif #ifdef INET6 case AF_INET6: { stc.addr_type = SCTP_IPV6_ADDRESS; memcpy(&stc.address, &src6->sin6_addr, sizeof(struct in6_addr)); stc.scope_id = in6_getscope(&src6->sin6_addr); if (sctp_is_address_on_local_host(src, vrf_id)) { stc.loopback_scope = 1; stc.local_scope = 0; stc.site_scope = 1; stc.ipv4_scope = 1; } else if (IN6_IS_ADDR_LINKLOCAL(&src6->sin6_addr) || IN6_IS_ADDR_LINKLOCAL(&dst6->sin6_addr)) { /* * If the new destination or source * is a LINK_LOCAL we must have * common both site and local scope. * Don't set local scope though * since we must depend on the * source to be added implicitly. We * cannot assure just because we * share one link that all links are * common. */ stc.local_scope = 0; stc.site_scope = 1; stc.ipv4_scope = 1; /* * we start counting for the private * address stuff at 1. since the * link local we source from won't * show up in our scoped count. */ cnt_inits_to = 1; /* * pull out the scope_id from * incoming pkt */ } else if (IN6_IS_ADDR_SITELOCAL(&src6->sin6_addr) || IN6_IS_ADDR_SITELOCAL(&dst6->sin6_addr)) { /* * If the new destination or source * is SITE_LOCAL then we must have * site scope in common. */ stc.site_scope = 1; } memcpy(&stc.laddress, &dst6->sin6_addr, sizeof(struct in6_addr)); stc.laddr_type = SCTP_IPV6_ADDRESS; break; } #endif default: /* TSNH */ goto do_a_abort; break; } } else { /* set the scope per the existing tcb */ #ifdef INET6 struct sctp_nets *lnet; #endif stc.loopback_scope = asoc->scope.loopback_scope; stc.ipv4_scope = asoc->scope.ipv4_local_scope; stc.site_scope = asoc->scope.site_scope; stc.local_scope = asoc->scope.local_scope; #ifdef INET6 /* Why do we not consider IPv4 LL addresses? */ TAILQ_FOREACH(lnet, &asoc->nets, sctp_next) { if (lnet->ro._l_addr.sin6.sin6_family == AF_INET6) { if (IN6_IS_ADDR_LINKLOCAL(&lnet->ro._l_addr.sin6.sin6_addr)) { /* * if we have a LL address, start * counting at 1. */ cnt_inits_to = 1; } } } #endif /* use the net pointer */ to = (struct sockaddr *)&net->ro._l_addr; switch (to->sa_family) { #ifdef INET case AF_INET: sin = (struct sockaddr_in *)to; stc.address[0] = sin->sin_addr.s_addr; stc.address[1] = 0; stc.address[2] = 0; stc.address[3] = 0; stc.addr_type = SCTP_IPV4_ADDRESS; if (net->src_addr_selected == 0) { /* * strange case here, the INIT should have * did the selection. */ net->ro._s_addr = sctp_source_address_selection(inp, stcb, (sctp_route_t *) & net->ro, net, 0, vrf_id); if (net->ro._s_addr == NULL) return; net->src_addr_selected = 1; } stc.laddress[0] = net->ro._s_addr->address.sin.sin_addr.s_addr; stc.laddress[1] = 0; stc.laddress[2] = 0; stc.laddress[3] = 0; stc.laddr_type = SCTP_IPV4_ADDRESS; /* scope_id is only for v6 */ stc.scope_id = 0; break; #endif #ifdef INET6 case AF_INET6: sin6 = (struct sockaddr_in6 *)to; memcpy(&stc.address, &sin6->sin6_addr, sizeof(struct in6_addr)); stc.addr_type = SCTP_IPV6_ADDRESS; stc.scope_id = sin6->sin6_scope_id; if (net->src_addr_selected == 0) { /* * strange case here, the INIT should have * done the selection. */ net->ro._s_addr = sctp_source_address_selection(inp, stcb, (sctp_route_t *) & net->ro, net, 0, vrf_id); if (net->ro._s_addr == NULL) return; net->src_addr_selected = 1; } memcpy(&stc.laddress, &net->ro._s_addr->address.sin6.sin6_addr, sizeof(struct in6_addr)); stc.laddr_type = SCTP_IPV6_ADDRESS; break; #endif } } /* Now lets put the SCTP header in place */ initack = mtod(m, struct sctp_init_ack_chunk *); /* Save it off for quick ref */ stc.peers_vtag = init_chk->init.initiate_tag; /* who are we */ memcpy(stc.identification, SCTP_VERSION_STRING, min(strlen(SCTP_VERSION_STRING), sizeof(stc.identification))); memset(stc.reserved, 0, SCTP_RESERVE_SPACE); /* now the chunk header */ initack->ch.chunk_type = SCTP_INITIATION_ACK; initack->ch.chunk_flags = 0; /* fill in later from mbuf we build */ initack->ch.chunk_length = 0; /* place in my tag */ if ((asoc != NULL) && ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_INUSE) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED))) { /* re-use the v-tags and init-seq here */ initack->init.initiate_tag = htonl(asoc->my_vtag); initack->init.initial_tsn = htonl(asoc->init_seq_number); } else { uint32_t vtag, itsn; if (hold_inp_lock) { SCTP_INP_INCR_REF(inp); SCTP_INP_RUNLOCK(inp); } if (asoc) { atomic_add_int(&asoc->refcnt, 1); SCTP_TCB_UNLOCK(stcb); new_tag: vtag = sctp_select_a_tag(inp, inp->sctp_lport, sh->src_port, 1); if ((asoc->peer_supports_nat) && (vtag == asoc->my_vtag)) { /* * Got a duplicate vtag on some guy behind a * nat make sure we don't use it. */ goto new_tag; } initack->init.initiate_tag = htonl(vtag); /* get a TSN to use too */ itsn = sctp_select_initial_TSN(&inp->sctp_ep); initack->init.initial_tsn = htonl(itsn); SCTP_TCB_LOCK(stcb); atomic_add_int(&asoc->refcnt, -1); } else { vtag = sctp_select_a_tag(inp, inp->sctp_lport, sh->src_port, 1); initack->init.initiate_tag = htonl(vtag); /* get a TSN to use too */ initack->init.initial_tsn = htonl(sctp_select_initial_TSN(&inp->sctp_ep)); } if (hold_inp_lock) { SCTP_INP_RLOCK(inp); SCTP_INP_DECR_REF(inp); } } /* save away my tag to */ stc.my_vtag = initack->init.initiate_tag; /* set up some of the credits. */ so = inp->sctp_socket; if (so == NULL) { /* memory problem */ sctp_m_freem(m); return; } else { initack->init.a_rwnd = htonl(max(SCTP_SB_LIMIT_RCV(so), SCTP_MINIMAL_RWND)); } /* set what I want */ his_limit = ntohs(init_chk->init.num_inbound_streams); /* choose what I want */ if (asoc != NULL) { if (asoc->streamoutcnt > asoc->pre_open_streams) { i_want = asoc->streamoutcnt; } else { i_want = asoc->pre_open_streams; } } else { i_want = inp->sctp_ep.pre_open_stream_count; } if (his_limit < i_want) { /* I Want more :< */ initack->init.num_outbound_streams = init_chk->init.num_inbound_streams; } else { /* I can have what I want :> */ initack->init.num_outbound_streams = htons(i_want); } /* tell him his limit. */ initack->init.num_inbound_streams = htons(inp->sctp_ep.max_open_streams_intome); /* adaptation layer indication parameter */ if (inp->sctp_ep.adaptation_layer_indicator_provided) { parameter_len = (uint16_t) sizeof(struct sctp_adaptation_layer_indication); ali = (struct sctp_adaptation_layer_indication *)(mtod(m, caddr_t)+chunk_len); ali->ph.param_type = htons(SCTP_ULP_ADAPTATION); ali->ph.param_length = htons(parameter_len); ali->indication = htonl(inp->sctp_ep.adaptation_layer_indicator); chunk_len += parameter_len; } /* ECN parameter */ if (((asoc != NULL) && (asoc->ecn_supported == 1)) || ((asoc == NULL) && (inp->ecn_supported == 1))) { parameter_len = (uint16_t) sizeof(struct sctp_paramhdr); ph = (struct sctp_paramhdr *)(mtod(m, caddr_t)+chunk_len); ph->param_type = htons(SCTP_ECN_CAPABLE); ph->param_length = htons(parameter_len); chunk_len += parameter_len; } /* PR-SCTP supported parameter */ if (((asoc != NULL) && (asoc->prsctp_supported == 1)) || ((asoc == NULL) && (inp->prsctp_supported == 1))) { parameter_len = (uint16_t) sizeof(struct sctp_paramhdr); ph = (struct sctp_paramhdr *)(mtod(m, caddr_t)+chunk_len); ph->param_type = htons(SCTP_PRSCTP_SUPPORTED); ph->param_length = htons(parameter_len); chunk_len += parameter_len; } /* Add NAT friendly parameter */ if (nat_friendly) { parameter_len = (uint16_t) sizeof(struct sctp_paramhdr); ph = (struct sctp_paramhdr *)(mtod(m, caddr_t)+chunk_len); ph->param_type = htons(SCTP_HAS_NAT_SUPPORT); ph->param_length = htons(parameter_len); chunk_len += parameter_len; } /* And now tell the peer which extensions we support */ num_ext = 0; pr_supported = (struct sctp_supported_chunk_types_param *)(mtod(m, caddr_t)+chunk_len); if (((asoc != NULL) && (asoc->prsctp_supported == 1)) || ((asoc == NULL) && (inp->prsctp_supported == 1))) { pr_supported->chunk_types[num_ext++] = SCTP_FORWARD_CUM_TSN; } if (((asoc != NULL) && (asoc->auth_supported == 1)) || ((asoc == NULL) && (inp->auth_supported == 1))) { pr_supported->chunk_types[num_ext++] = SCTP_AUTHENTICATION; } if (((asoc != NULL) && (asoc->asconf_supported == 1)) || ((asoc == NULL) && (inp->asconf_supported == 1))) { pr_supported->chunk_types[num_ext++] = SCTP_ASCONF; pr_supported->chunk_types[num_ext++] = SCTP_ASCONF_ACK; } if (((asoc != NULL) && (asoc->reconfig_supported == 1)) || ((asoc == NULL) && (inp->reconfig_supported == 1))) { pr_supported->chunk_types[num_ext++] = SCTP_STREAM_RESET; } if (((asoc != NULL) && (asoc->nrsack_supported == 1)) || ((asoc == NULL) && (inp->nrsack_supported == 1))) { pr_supported->chunk_types[num_ext++] = SCTP_NR_SELECTIVE_ACK; } if (((asoc != NULL) && (asoc->pktdrop_supported == 1)) || ((asoc == NULL) && (inp->pktdrop_supported == 1))) { pr_supported->chunk_types[num_ext++] = SCTP_PACKET_DROPPED; } if (num_ext > 0) { parameter_len = (uint16_t) sizeof(struct sctp_supported_chunk_types_param) + num_ext; pr_supported->ph.param_type = htons(SCTP_SUPPORTED_CHUNK_EXT); pr_supported->ph.param_length = htons(parameter_len); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; } /* add authentication parameters */ if (((asoc != NULL) && (asoc->auth_supported == 1)) || ((asoc == NULL) && (inp->auth_supported == 1))) { struct sctp_auth_random *randp; struct sctp_auth_hmac_algo *hmacs; struct sctp_auth_chunk_list *chunks; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } /* generate and add RANDOM parameter */ randp = (struct sctp_auth_random *)(mtod(m, caddr_t)+chunk_len); parameter_len = (uint16_t) sizeof(struct sctp_auth_random) + SCTP_AUTH_RANDOM_SIZE_DEFAULT; randp->ph.param_type = htons(SCTP_RANDOM); randp->ph.param_length = htons(parameter_len); SCTP_READ_RANDOM(randp->random_data, SCTP_AUTH_RANDOM_SIZE_DEFAULT); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } /* add HMAC_ALGO parameter */ hmacs = (struct sctp_auth_hmac_algo *)(mtod(m, caddr_t)+chunk_len); parameter_len = (uint16_t) sizeof(struct sctp_auth_hmac_algo) + sctp_serialize_hmaclist(inp->sctp_ep.local_hmacs, (uint8_t *) hmacs->hmac_ids); hmacs->ph.param_type = htons(SCTP_HMAC_LIST); hmacs->ph.param_length = htons(parameter_len); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; padding_len = 0; } /* add CHUNKS parameter */ chunks = (struct sctp_auth_chunk_list *)(mtod(m, caddr_t)+chunk_len); parameter_len = (uint16_t) sizeof(struct sctp_auth_chunk_list) + sctp_serialize_auth_chunks(inp->sctp_ep.local_auth_chunks, chunks->chunk_types); chunks->ph.param_type = htons(SCTP_CHUNK_LIST); chunks->ph.param_length = htons(parameter_len); padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; } SCTP_BUF_LEN(m) = chunk_len; m_last = m; /* now the addresses */ /* * To optimize this we could put the scoping stuff into a structure * and remove the individual uint8's from the stc structure. Then we * could just sifa in the address within the stc.. but for now this * is a quick hack to get the address stuff teased apart. */ scp.ipv4_addr_legal = stc.ipv4_addr_legal; scp.ipv6_addr_legal = stc.ipv6_addr_legal; scp.loopback_scope = stc.loopback_scope; scp.ipv4_local_scope = stc.ipv4_scope; scp.local_scope = stc.local_scope; scp.site_scope = stc.site_scope; m_last = sctp_add_addresses_to_i_ia(inp, stcb, &scp, m_last, cnt_inits_to, &padding_len, &chunk_len); /* padding_len can only be positive, if no addresses have been added */ if (padding_len > 0) { memset(mtod(m, caddr_t)+chunk_len, 0, padding_len); chunk_len += padding_len; SCTP_BUF_LEN(m) += padding_len; padding_len = 0; } /* tack on the operational error if present */ if (op_err) { parameter_len = 0; for (m_tmp = op_err; m_tmp != NULL; m_tmp = SCTP_BUF_NEXT(m_tmp)) { parameter_len += SCTP_BUF_LEN(m_tmp); } padding_len = SCTP_SIZE32(parameter_len) - parameter_len; SCTP_BUF_NEXT(m_last) = op_err; while (SCTP_BUF_NEXT(m_last) != NULL) { m_last = SCTP_BUF_NEXT(m_last); } chunk_len += parameter_len; } if (padding_len > 0) { m_last = sctp_add_pad_tombuf(m_last, padding_len); if (m_last == NULL) { /* Houston we have a problem, no space */ sctp_m_freem(m); return; } chunk_len += padding_len; padding_len = 0; } /* Now we must build a cookie */ m_cookie = sctp_add_cookie(init_pkt, offset, m, 0, &stc, &signature); if (m_cookie == NULL) { /* memory problem */ sctp_m_freem(m); return; } /* Now append the cookie to the end and update the space/size */ SCTP_BUF_NEXT(m_last) = m_cookie; parameter_len = 0; for (m_tmp = m_cookie; m_tmp != NULL; m_tmp = SCTP_BUF_NEXT(m_tmp)) { parameter_len += SCTP_BUF_LEN(m_tmp); if (SCTP_BUF_NEXT(m_tmp) == NULL) { m_last = m_tmp; } } padding_len = SCTP_SIZE32(parameter_len) - parameter_len; chunk_len += parameter_len; /* * Place in the size, but we don't include the last pad (if any) in * the INIT-ACK. */ initack->ch.chunk_length = htons(chunk_len); /* * Time to sign the cookie, we don't sign over the cookie signature * though thus we set trailer. */ (void)sctp_hmac_m(SCTP_HMAC, (uint8_t *) inp->sctp_ep.secret_key[(int)(inp->sctp_ep.current_secret_number)], SCTP_SECRET_SIZE, m_cookie, sizeof(struct sctp_paramhdr), (uint8_t *) signature, SCTP_SIGNATURE_SIZE); /* * We sifa 0 here to NOT set IP_DF if its IPv4, we ignore the return * here since the timer will drive a retranmission. */ if (padding_len > 0) { if (sctp_add_pad_tombuf(m_last, padding_len) == NULL) { sctp_m_freem(m); return; } } if (stc.loopback_scope) { over_addr = (union sctp_sockstore *)dst; } else { over_addr = NULL; } (void)sctp_lowlevel_chunk_output(inp, NULL, NULL, to, m, 0, NULL, 0, 0, 0, 0, inp->sctp_lport, sh->src_port, init_chk->init.initiate_tag, port, over_addr, mflowtype, mflowid, SCTP_SO_NOT_LOCKED); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } static void sctp_prune_prsctp(struct sctp_tcb *stcb, struct sctp_association *asoc, struct sctp_sndrcvinfo *srcv, int dataout) { int freed_spc = 0; struct sctp_tmit_chunk *chk, *nchk; SCTP_TCB_LOCK_ASSERT(stcb); if ((asoc->prsctp_supported) && (asoc->sent_queue_cnt_removeable > 0)) { TAILQ_FOREACH(chk, &asoc->sent_queue, sctp_next) { /* * Look for chunks marked with the PR_SCTP flag AND * the buffer space flag. If the one being sent is * equal or greater priority then purge the old one * and free some space. */ if (PR_SCTP_BUF_ENABLED(chk->flags)) { /* * This one is PR-SCTP AND buffer space * limited type */ if (chk->rec.data.timetodrop.tv_sec >= (long)srcv->sinfo_timetolive) { /* * Lower numbers equates to higher * priority so if the one we are * looking at has a larger or equal * priority we want to drop the data * and NOT retransmit it. */ if (chk->data) { /* * We release the book_size * if the mbuf is here */ int ret_spc; uint8_t sent; if (chk->sent > SCTP_DATAGRAM_UNSENT) sent = 1; else sent = 0; ret_spc = sctp_release_pr_sctp_chunk(stcb, chk, sent, SCTP_SO_LOCKED); freed_spc += ret_spc; if (freed_spc >= dataout) { return; } } /* if chunk was present */ } /* if of sufficent priority */ } /* if chunk has enabled */ } /* tailqforeach */ TAILQ_FOREACH_SAFE(chk, &asoc->send_queue, sctp_next, nchk) { /* Here we must move to the sent queue and mark */ if (PR_SCTP_BUF_ENABLED(chk->flags)) { if (chk->rec.data.timetodrop.tv_sec >= (long)srcv->sinfo_timetolive) { if (chk->data) { /* * We release the book_size * if the mbuf is here */ int ret_spc; ret_spc = sctp_release_pr_sctp_chunk(stcb, chk, 0, SCTP_SO_LOCKED); freed_spc += ret_spc; if (freed_spc >= dataout) { return; } } /* end if chk->data */ } /* end if right class */ } /* end if chk pr-sctp */ } /* tailqforeachsafe (chk) */ } /* if enabled in asoc */ } int sctp_get_frag_point(struct sctp_tcb *stcb, struct sctp_association *asoc) { int siz, ovh; /* * For endpoints that have both v6 and v4 addresses we must reserve * room for the ipv6 header, for those that are only dealing with V4 * we use a larger frag point. */ if (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_BOUND_V6) { ovh = SCTP_MED_OVERHEAD; } else { ovh = SCTP_MED_V4_OVERHEAD; } if (stcb->asoc.sctp_frag_point > asoc->smallest_mtu) siz = asoc->smallest_mtu - ovh; else siz = (stcb->asoc.sctp_frag_point - ovh); /* * if (siz > (MCLBYTES-sizeof(struct sctp_data_chunk))) { */ /* A data chunk MUST fit in a cluster */ /* siz = (MCLBYTES - sizeof(struct sctp_data_chunk)); */ /* } */ /* adjust for an AUTH chunk if DATA requires auth */ if (sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.peer_auth_chunks)) siz -= sctp_get_auth_chunk_len(stcb->asoc.peer_hmac_id); if (siz % 4) { /* make it an even word boundary please */ siz -= (siz % 4); } return (siz); } static void sctp_set_prsctp_policy(struct sctp_stream_queue_pending *sp) { /* * We assume that the user wants PR_SCTP_TTL if the user provides a * positive lifetime but does not specify any PR_SCTP policy. */ if (PR_SCTP_ENABLED(sp->sinfo_flags)) { sp->act_flags |= PR_SCTP_POLICY(sp->sinfo_flags); } else if (sp->timetolive > 0) { sp->sinfo_flags |= SCTP_PR_SCTP_TTL; sp->act_flags |= PR_SCTP_POLICY(sp->sinfo_flags); } else { return; } switch (PR_SCTP_POLICY(sp->sinfo_flags)) { case CHUNK_FLAGS_PR_SCTP_BUF: /* * Time to live is a priority stored in tv_sec when doing * the buffer drop thing. */ sp->ts.tv_sec = sp->timetolive; sp->ts.tv_usec = 0; break; case CHUNK_FLAGS_PR_SCTP_TTL: { struct timeval tv; (void)SCTP_GETTIME_TIMEVAL(&sp->ts); tv.tv_sec = sp->timetolive / 1000; tv.tv_usec = (sp->timetolive * 1000) % 1000000; /* * TODO sctp_constants.h needs alternative time * macros when _KERNEL is undefined. */ timevaladd(&sp->ts, &tv); } break; case CHUNK_FLAGS_PR_SCTP_RTX: /* * Time to live is a the number or retransmissions stored in * tv_sec. */ sp->ts.tv_sec = sp->timetolive; sp->ts.tv_usec = 0; break; default: SCTPDBG(SCTP_DEBUG_USRREQ1, "Unknown PR_SCTP policy %u.\n", PR_SCTP_POLICY(sp->sinfo_flags)); break; } } static int sctp_msg_append(struct sctp_tcb *stcb, struct sctp_nets *net, struct mbuf *m, struct sctp_sndrcvinfo *srcv, int hold_stcb_lock) { int error = 0; struct mbuf *at; struct sctp_stream_queue_pending *sp = NULL; struct sctp_stream_out *strm; /* * Given an mbuf chain, put it into the association send queue and * place it on the wheel */ if (srcv->sinfo_stream >= stcb->asoc.streamoutcnt) { /* Invalid stream number */ SCTP_LTRACE_ERR_RET_PKT(m, NULL, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_now; } if ((stcb->asoc.stream_locked) && (stcb->asoc.stream_locked_on != srcv->sinfo_stream)) { SCTP_LTRACE_ERR_RET_PKT(m, NULL, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_now; } strm = &stcb->asoc.strmout[srcv->sinfo_stream]; /* Now can we send this? */ if ((SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_SENT) || (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) || (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_SHUTDOWN_RECEIVED) || (stcb->asoc.state & SCTP_STATE_SHUTDOWN_PENDING)) { /* got data while shutting down */ SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ECONNRESET); error = ECONNRESET; goto out_now; } sctp_alloc_a_strmoq(stcb, sp); if (sp == NULL) { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); error = ENOMEM; goto out_now; } sp->sinfo_flags = srcv->sinfo_flags; sp->timetolive = srcv->sinfo_timetolive; sp->ppid = srcv->sinfo_ppid; sp->context = srcv->sinfo_context; if (sp->sinfo_flags & SCTP_ADDR_OVER) { sp->net = net; atomic_add_int(&sp->net->ref_count, 1); } else { sp->net = NULL; } (void)SCTP_GETTIME_TIMEVAL(&sp->ts); sp->stream = srcv->sinfo_stream; sp->msg_is_complete = 1; sp->sender_all_done = 1; sp->some_taken = 0; sp->data = m; sp->tail_mbuf = NULL; sctp_set_prsctp_policy(sp); /* * We could in theory (for sendall) sifa the length in, but we would * still have to hunt through the chain since we need to setup the * tail_mbuf */ sp->length = 0; for (at = m; at; at = SCTP_BUF_NEXT(at)) { if (SCTP_BUF_NEXT(at) == NULL) sp->tail_mbuf = at; sp->length += SCTP_BUF_LEN(at); } if (srcv->sinfo_keynumber_valid) { sp->auth_keyid = srcv->sinfo_keynumber; } else { sp->auth_keyid = stcb->asoc.authinfo.active_keyid; } if (sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.peer_auth_chunks)) { sctp_auth_key_acquire(stcb, sp->auth_keyid); sp->holds_key_ref = 1; } if (hold_stcb_lock == 0) { SCTP_TCB_SEND_LOCK(stcb); } sctp_snd_sb_alloc(stcb, sp->length); atomic_add_int(&stcb->asoc.stream_queue_cnt, 1); TAILQ_INSERT_TAIL(&strm->outqueue, sp, next); stcb->asoc.ss_functions.sctp_ss_add_to_stream(stcb, &stcb->asoc, strm, sp, 1); m = NULL; if (hold_stcb_lock == 0) { SCTP_TCB_SEND_UNLOCK(stcb); } out_now: if (m) { sctp_m_freem(m); } return (error); } static struct mbuf * sctp_copy_mbufchain(struct mbuf *clonechain, struct mbuf *outchain, struct mbuf **endofchain, int can_take_mbuf, int sizeofcpy, uint8_t copy_by_ref) { struct mbuf *m; struct mbuf *appendchain; caddr_t cp; int len; if (endofchain == NULL) { /* error */ error_out: if (outchain) sctp_m_freem(outchain); return (NULL); } if (can_take_mbuf) { appendchain = clonechain; } else { if (!copy_by_ref && (sizeofcpy <= (int)((((SCTP_BASE_SYSCTL(sctp_mbuf_threshold_count) - 1) * MLEN) + MHLEN))) ) { /* Its not in a cluster */ if (*endofchain == NULL) { /* lets get a mbuf cluster */ if (outchain == NULL) { /* This is the general case */ new_mbuf: outchain = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_HEADER); if (outchain == NULL) { goto error_out; } SCTP_BUF_LEN(outchain) = 0; *endofchain = outchain; /* get the prepend space */ SCTP_BUF_RESV_UF(outchain, (SCTP_FIRST_MBUF_RESV + 4)); } else { /* * We really should not get a NULL * in endofchain */ /* find end */ m = outchain; while (m) { if (SCTP_BUF_NEXT(m) == NULL) { *endofchain = m; break; } m = SCTP_BUF_NEXT(m); } /* sanity */ if (*endofchain == NULL) { /* * huh, TSNH XXX maybe we * should panic */ sctp_m_freem(outchain); goto new_mbuf; } } /* get the new end of length */ len = M_TRAILINGSPACE(*endofchain); } else { /* how much is left at the end? */ len = M_TRAILINGSPACE(*endofchain); } /* Find the end of the data, for appending */ cp = (mtod((*endofchain), caddr_t)+SCTP_BUF_LEN((*endofchain))); /* Now lets copy it out */ if (len >= sizeofcpy) { /* It all fits, copy it in */ m_copydata(clonechain, 0, sizeofcpy, cp); SCTP_BUF_LEN((*endofchain)) += sizeofcpy; } else { /* fill up the end of the chain */ if (len > 0) { m_copydata(clonechain, 0, len, cp); SCTP_BUF_LEN((*endofchain)) += len; /* now we need another one */ sizeofcpy -= len; } m = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_HEADER); if (m == NULL) { /* We failed */ goto error_out; } SCTP_BUF_NEXT((*endofchain)) = m; *endofchain = m; cp = mtod((*endofchain), caddr_t); m_copydata(clonechain, len, sizeofcpy, cp); SCTP_BUF_LEN((*endofchain)) += sizeofcpy; } return (outchain); } else { /* copy the old fashion way */ appendchain = SCTP_M_COPYM(clonechain, 0, M_COPYALL, M_NOWAIT); #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(appendchain, SCTP_MBUF_ICOPY); } #endif } } if (appendchain == NULL) { /* error */ if (outchain) sctp_m_freem(outchain); return (NULL); } if (outchain) { /* tack on to the end */ if (*endofchain != NULL) { SCTP_BUF_NEXT(((*endofchain))) = appendchain; } else { m = outchain; while (m) { if (SCTP_BUF_NEXT(m) == NULL) { SCTP_BUF_NEXT(m) = appendchain; break; } m = SCTP_BUF_NEXT(m); } } /* * save off the end and update the end-chain postion */ m = appendchain; while (m) { if (SCTP_BUF_NEXT(m) == NULL) { *endofchain = m; break; } m = SCTP_BUF_NEXT(m); } return (outchain); } else { /* save off the end and update the end-chain postion */ m = appendchain; while (m) { if (SCTP_BUF_NEXT(m) == NULL) { *endofchain = m; break; } m = SCTP_BUF_NEXT(m); } return (appendchain); } } static int sctp_med_chunk_output(struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_association *asoc, int *num_out, int *reason_code, int control_only, int from_where, struct timeval *now, int *now_filled, int frag_point, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ); static void sctp_sendall_iterator(struct sctp_inpcb *inp, struct sctp_tcb *stcb, void *ptr, uint32_t val SCTP_UNUSED) { struct sctp_copy_all *ca; struct mbuf *m; int ret = 0; int added_control = 0; int un_sent, do_chunk_output = 1; struct sctp_association *asoc; struct sctp_nets *net; ca = (struct sctp_copy_all *)ptr; if (ca->m == NULL) { return; } if (ca->inp != inp) { /* TSNH */ return; } if (ca->sndlen > 0) { m = SCTP_M_COPYM(ca->m, 0, M_COPYALL, M_NOWAIT); if (m == NULL) { /* can't copy so we are done */ ca->cnt_failed++; return; } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(m, SCTP_MBUF_ICOPY); } #endif } else { m = NULL; } SCTP_TCB_LOCK_ASSERT(stcb); if (stcb->asoc.alternate) { net = stcb->asoc.alternate; } else { net = stcb->asoc.primary_destination; } if (ca->sndrcv.sinfo_flags & SCTP_ABORT) { /* Abort this assoc with m as the user defined reason */ if (m != NULL) { SCTP_BUF_PREPEND(m, sizeof(struct sctp_paramhdr), M_NOWAIT); } else { m = sctp_get_mbuf_for_msg(sizeof(struct sctp_paramhdr), 0, M_NOWAIT, 1, MT_DATA); SCTP_BUF_LEN(m) = sizeof(struct sctp_paramhdr); } if (m != NULL) { struct sctp_paramhdr *ph; ph = mtod(m, struct sctp_paramhdr *); ph->param_type = htons(SCTP_CAUSE_USER_INITIATED_ABT); ph->param_length = htons(sizeof(struct sctp_paramhdr) + ca->sndlen); } /* * We add one here to keep the assoc from dis-appearing on * us. */ atomic_add_int(&stcb->asoc.refcnt, 1); sctp_abort_an_association(inp, stcb, m, SCTP_SO_NOT_LOCKED); /* * sctp_abort_an_association calls sctp_free_asoc() free * association will NOT free it since we incremented the * refcnt .. we do this to prevent it being freed and things * getting tricky since we could end up (from free_asoc) * calling inpcb_free which would get a recursive lock call * to the iterator lock.. But as a consequence of that the * stcb will return to us un-locked.. since free_asoc * returns with either no TCB or the TCB unlocked, we must * relock.. to unlock in the iterator timer :-0 */ SCTP_TCB_LOCK(stcb); atomic_add_int(&stcb->asoc.refcnt, -1); goto no_chunk_output; } else { if (m) { ret = sctp_msg_append(stcb, net, m, &ca->sndrcv, 1); } asoc = &stcb->asoc; if (ca->sndrcv.sinfo_flags & SCTP_EOF) { /* shutdown this assoc */ int cnt; cnt = sctp_is_there_unsent_data(stcb, SCTP_SO_NOT_LOCKED); if (TAILQ_EMPTY(&asoc->send_queue) && TAILQ_EMPTY(&asoc->sent_queue) && (cnt == 0)) { if (asoc->locked_on_sending) { goto abort_anyway; } /* * there is nothing queued to send, so I'm * done... */ if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT)) { /* * only send SHUTDOWN the first time * through */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) { SCTP_STAT_DECR_GAUGE32(sctps_currestab); } SCTP_SET_STATE(asoc, SCTP_STATE_SHUTDOWN_SENT); SCTP_CLEAR_SUBSTATE(asoc, SCTP_STATE_SHUTDOWN_PENDING); sctp_stop_timers_for_shutdown(stcb); sctp_send_shutdown(stcb, net); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWN, stcb->sctp_ep, stcb, net); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); added_control = 1; do_chunk_output = 0; } } else { /* * we still got (or just got) data to send, * so set SHUTDOWN_PENDING */ /* * XXX sockets draft says that SCTP_EOF * should be sent with no data. currently, * we will allow user data to be sent first * and move to SHUTDOWN-PENDING */ if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT)) { if (asoc->locked_on_sending) { /* * Locked to send out the * data */ struct sctp_stream_queue_pending *sp; sp = TAILQ_LAST(&asoc->locked_on_sending->outqueue, sctp_streamhead); if (sp) { if ((sp->length == 0) && (sp->msg_is_complete == 0)) asoc->state |= SCTP_STATE_PARTIAL_MSG_LEFT; } } asoc->state |= SCTP_STATE_SHUTDOWN_PENDING; if (TAILQ_EMPTY(&asoc->send_queue) && TAILQ_EMPTY(&asoc->sent_queue) && (asoc->state & SCTP_STATE_PARTIAL_MSG_LEFT)) { struct mbuf *op_err; char msg[SCTP_DIAG_INFO_LEN]; abort_anyway: snprintf(msg, sizeof(msg), "%s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); atomic_add_int(&stcb->asoc.refcnt, 1); sctp_abort_an_association(stcb->sctp_ep, stcb, op_err, SCTP_SO_NOT_LOCKED); atomic_add_int(&stcb->asoc.refcnt, -1); goto no_chunk_output; } sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } } } } un_sent = ((stcb->asoc.total_output_queue_size - stcb->asoc.total_flight) + (stcb->asoc.stream_queue_cnt * sizeof(struct sctp_data_chunk))); if ((sctp_is_feature_off(inp, SCTP_PCB_FLAGS_NODELAY)) && (stcb->asoc.total_flight > 0) && (un_sent < (int)(stcb->asoc.smallest_mtu - SCTP_MIN_OVERHEAD))) { do_chunk_output = 0; } if (do_chunk_output) sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_USR_SEND, SCTP_SO_NOT_LOCKED); else if (added_control) { int num_out, reason, now_filled = 0; struct timeval now; int frag_point; frag_point = sctp_get_frag_point(stcb, &stcb->asoc); (void)sctp_med_chunk_output(inp, stcb, &stcb->asoc, &num_out, &reason, 1, 1, &now, &now_filled, frag_point, SCTP_SO_NOT_LOCKED); } no_chunk_output: if (ret) { ca->cnt_failed++; } else { ca->cnt_sent++; } } static void sctp_sendall_completes(void *ptr, uint32_t val SCTP_UNUSED) { struct sctp_copy_all *ca; ca = (struct sctp_copy_all *)ptr; /* * Do a notify here? Kacheong suggests that the notify be done at * the send time.. so you would push up a notification if any send * failed. Don't know if this is feasable since the only failures we * have is "memory" related and if you cannot get an mbuf to send * the data you surely can't get an mbuf to send up to notify the * user you can't send the data :-> */ /* now free everything */ sctp_m_freem(ca->m); SCTP_FREE(ca, SCTP_M_COPYAL); } static struct mbuf * sctp_copy_out_all(struct uio *uio, int len) { struct mbuf *ret, *at; int left, willcpy, cancpy, error; ret = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_WAITOK, 1, MT_DATA); if (ret == NULL) { /* TSNH */ return (NULL); } left = len; SCTP_BUF_LEN(ret) = 0; /* save space for the data chunk header */ cancpy = M_TRAILINGSPACE(ret); willcpy = min(cancpy, left); at = ret; while (left > 0) { /* Align data to the end */ error = uiomove(mtod(at, caddr_t), willcpy, uio); if (error) { err_out_now: sctp_m_freem(at); return (NULL); } SCTP_BUF_LEN(at) = willcpy; SCTP_BUF_NEXT_PKT(at) = SCTP_BUF_NEXT(at) = 0; left -= willcpy; if (left > 0) { SCTP_BUF_NEXT(at) = sctp_get_mbuf_for_msg(left, 0, M_WAITOK, 1, MT_DATA); if (SCTP_BUF_NEXT(at) == NULL) { goto err_out_now; } at = SCTP_BUF_NEXT(at); SCTP_BUF_LEN(at) = 0; cancpy = M_TRAILINGSPACE(at); willcpy = min(cancpy, left); } } return (ret); } static int sctp_sendall(struct sctp_inpcb *inp, struct uio *uio, struct mbuf *m, struct sctp_sndrcvinfo *srcv) { int ret; struct sctp_copy_all *ca; SCTP_MALLOC(ca, struct sctp_copy_all *, sizeof(struct sctp_copy_all), SCTP_M_COPYAL); if (ca == NULL) { sctp_m_freem(m); SCTP_LTRACE_ERR_RET(inp, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } memset(ca, 0, sizeof(struct sctp_copy_all)); ca->inp = inp; if (srcv) { memcpy(&ca->sndrcv, srcv, sizeof(struct sctp_nonpad_sndrcvinfo)); } /* * take off the sendall flag, it would be bad if we failed to do * this :-0 */ ca->sndrcv.sinfo_flags &= ~SCTP_SENDALL; /* get length and mbuf chain */ if (uio) { ca->sndlen = uio->uio_resid; ca->m = sctp_copy_out_all(uio, ca->sndlen); if (ca->m == NULL) { SCTP_FREE(ca, SCTP_M_COPYAL); SCTP_LTRACE_ERR_RET(inp, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } } else { /* Gather the length of the send */ struct mbuf *mat; ca->sndlen = 0; for (mat = m; mat; mat = SCTP_BUF_NEXT(mat)) { ca->sndlen += SCTP_BUF_LEN(mat); } } ret = sctp_initiate_iterator(NULL, sctp_sendall_iterator, NULL, SCTP_PCB_ANY_FLAGS, SCTP_PCB_ANY_FEATURES, SCTP_ASOC_ANY_STATE, (void *)ca, 0, sctp_sendall_completes, inp, 1); if (ret) { SCTP_PRINTF("Failed to initiate iterator for sendall\n"); SCTP_FREE(ca, SCTP_M_COPYAL); SCTP_LTRACE_ERR_RET_PKT(m, inp, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, EFAULT); return (EFAULT); } return (0); } void sctp_toss_old_cookies(struct sctp_tcb *stcb, struct sctp_association *asoc) { struct sctp_tmit_chunk *chk, *nchk; TAILQ_FOREACH_SAFE(chk, &asoc->control_send_queue, sctp_next, nchk) { if (chk->rec.chunk_id.id == SCTP_COOKIE_ECHO) { TAILQ_REMOVE(&asoc->control_send_queue, chk, sctp_next); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } asoc->ctrl_queue_cnt--; sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); } } } void sctp_toss_old_asconf(struct sctp_tcb *stcb) { struct sctp_association *asoc; struct sctp_tmit_chunk *chk, *nchk; struct sctp_asconf_chunk *acp; asoc = &stcb->asoc; TAILQ_FOREACH_SAFE(chk, &asoc->asconf_send_queue, sctp_next, nchk) { /* find SCTP_ASCONF chunk in queue */ if (chk->rec.chunk_id.id == SCTP_ASCONF) { if (chk->data) { acp = mtod(chk->data, struct sctp_asconf_chunk *); if (SCTP_TSN_GT(ntohl(acp->serial_number), asoc->asconf_seq_out_acked)) { /* Not Acked yet */ break; } } TAILQ_REMOVE(&asoc->asconf_send_queue, chk, sctp_next); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } asoc->ctrl_queue_cnt--; sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); } } } static void sctp_clean_up_datalist(struct sctp_tcb *stcb, struct sctp_association *asoc, struct sctp_tmit_chunk **data_list, int bundle_at, struct sctp_nets *net) { int i; struct sctp_tmit_chunk *tp1; for (i = 0; i < bundle_at; i++) { /* off of the send queue */ TAILQ_REMOVE(&asoc->send_queue, data_list[i], sctp_next); asoc->send_queue_cnt--; if (i > 0) { /* * Any chunk NOT 0 you zap the time chunk 0 gets * zapped or set based on if a RTO measurment is * needed. */ data_list[i]->do_rtt = 0; } /* record time */ data_list[i]->sent_rcv_time = net->last_sent_time; data_list[i]->rec.data.cwnd_at_send = net->cwnd; data_list[i]->rec.data.fast_retran_tsn = data_list[i]->rec.data.TSN_seq; if (data_list[i]->whoTo == NULL) { data_list[i]->whoTo = net; atomic_add_int(&net->ref_count, 1); } /* on to the sent queue */ tp1 = TAILQ_LAST(&asoc->sent_queue, sctpchunk_listhead); if ((tp1) && SCTP_TSN_GT(tp1->rec.data.TSN_seq, data_list[i]->rec.data.TSN_seq)) { struct sctp_tmit_chunk *tpp; /* need to move back */ back_up_more: tpp = TAILQ_PREV(tp1, sctpchunk_listhead, sctp_next); if (tpp == NULL) { TAILQ_INSERT_BEFORE(tp1, data_list[i], sctp_next); goto all_done; } tp1 = tpp; if (SCTP_TSN_GT(tp1->rec.data.TSN_seq, data_list[i]->rec.data.TSN_seq)) { goto back_up_more; } TAILQ_INSERT_AFTER(&asoc->sent_queue, tp1, data_list[i], sctp_next); } else { TAILQ_INSERT_TAIL(&asoc->sent_queue, data_list[i], sctp_next); } all_done: /* This does not lower until the cum-ack passes it */ asoc->sent_queue_cnt++; if ((asoc->peers_rwnd <= 0) && (asoc->total_flight == 0) && (bundle_at == 1)) { /* Mark the chunk as being a window probe */ SCTP_STAT_INCR(sctps_windowprobed); } #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xC2, 3); #endif data_list[i]->sent = SCTP_DATAGRAM_SENT; data_list[i]->snd_count = 1; data_list[i]->rec.data.chunk_was_revoked = 0; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_FLIGHT_LOGGING_ENABLE) { sctp_misc_ints(SCTP_FLIGHT_LOG_UP, data_list[i]->whoTo->flight_size, data_list[i]->book_size, (uintptr_t) data_list[i]->whoTo, data_list[i]->rec.data.TSN_seq); } sctp_flight_size_increase(data_list[i]); sctp_total_flight_increase(stcb, data_list[i]); if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_RWND_ENABLE) { sctp_log_rwnd(SCTP_DECREASE_PEER_RWND, asoc->peers_rwnd, data_list[i]->send_size, SCTP_BASE_SYSCTL(sctp_peer_chunk_oh)); } asoc->peers_rwnd = sctp_sbspace_sub(asoc->peers_rwnd, (uint32_t) (data_list[i]->send_size + SCTP_BASE_SYSCTL(sctp_peer_chunk_oh))); if (asoc->peers_rwnd < stcb->sctp_ep->sctp_ep.sctp_sws_sender) { /* SWS sender side engages */ asoc->peers_rwnd = 0; } } if (asoc->cc_functions.sctp_cwnd_update_packet_transmitted) { (*asoc->cc_functions.sctp_cwnd_update_packet_transmitted) (stcb, net); } } static void sctp_clean_up_ctl(struct sctp_tcb *stcb, struct sctp_association *asoc, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { struct sctp_tmit_chunk *chk, *nchk; TAILQ_FOREACH_SAFE(chk, &asoc->control_send_queue, sctp_next, nchk) { if ((chk->rec.chunk_id.id == SCTP_SELECTIVE_ACK) || (chk->rec.chunk_id.id == SCTP_NR_SELECTIVE_ACK) || /* EY */ (chk->rec.chunk_id.id == SCTP_HEARTBEAT_REQUEST) || (chk->rec.chunk_id.id == SCTP_HEARTBEAT_ACK) || (chk->rec.chunk_id.id == SCTP_FORWARD_CUM_TSN) || (chk->rec.chunk_id.id == SCTP_SHUTDOWN) || (chk->rec.chunk_id.id == SCTP_SHUTDOWN_ACK) || (chk->rec.chunk_id.id == SCTP_OPERATION_ERROR) || (chk->rec.chunk_id.id == SCTP_PACKET_DROPPED) || (chk->rec.chunk_id.id == SCTP_COOKIE_ACK) || (chk->rec.chunk_id.id == SCTP_ECN_CWR) || (chk->rec.chunk_id.id == SCTP_ASCONF_ACK)) { /* Stray chunks must be cleaned up */ clean_up_anyway: TAILQ_REMOVE(&asoc->control_send_queue, chk, sctp_next); if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } asoc->ctrl_queue_cnt--; if (chk->rec.chunk_id.id == SCTP_FORWARD_CUM_TSN) asoc->fwd_tsn_cnt--; sctp_free_a_chunk(stcb, chk, so_locked); } else if (chk->rec.chunk_id.id == SCTP_STREAM_RESET) { /* special handling, we must look into the param */ if (chk != asoc->str_reset) { goto clean_up_anyway; } } } } static int sctp_can_we_split_this(struct sctp_tcb *stcb, uint32_t length, uint32_t goal_mtu, uint32_t frag_point, int eeor_on) { /* * Make a decision on if I should split a msg into multiple parts. * This is only asked of incomplete messages. */ if (eeor_on) { /* * If we are doing EEOR we need to always send it if its the * entire thing, since it might be all the guy is putting in * the hopper. */ if (goal_mtu >= length) { /*- * If we have data outstanding, * we get another chance when the sack * arrives to transmit - wait for more data */ if (stcb->asoc.total_flight == 0) { /* * If nothing is in flight, we zero the * packet counter. */ return (length); } return (0); } else { /* You can fill the rest */ return (goal_mtu); } } /*- * For those strange folk that make the send buffer * smaller than our fragmentation point, we can't * get a full msg in so we have to allow splitting. */ if (SCTP_SB_LIMIT_SND(stcb->sctp_socket) < frag_point) { return (length); } if ((length <= goal_mtu) || ((length - goal_mtu) < SCTP_BASE_SYSCTL(sctp_min_residual))) { /* Sub-optimial residual don't split in non-eeor mode. */ return (0); } /* * If we reach here length is larger than the goal_mtu. Do we wish * to split it for the sake of packet putting together? */ if (goal_mtu >= min(SCTP_BASE_SYSCTL(sctp_min_split_point), frag_point)) { /* Its ok to split it */ return (min(goal_mtu, frag_point)); } /* Nope, can't split */ return (0); } static uint32_t sctp_move_to_outqueue(struct sctp_tcb *stcb, struct sctp_stream_out *strq, uint32_t goal_mtu, uint32_t frag_point, int *locked, int *giveup, int eeor_mode, int *bail, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { /* Move from the stream to the send_queue keeping track of the total */ struct sctp_association *asoc; struct sctp_stream_queue_pending *sp; struct sctp_tmit_chunk *chk; struct sctp_data_chunk *dchkh; uint32_t to_move, length; uint8_t rcv_flags = 0; uint8_t some_taken; uint8_t send_lock_up = 0; SCTP_TCB_LOCK_ASSERT(stcb); asoc = &stcb->asoc; one_more_time: /* sa_ignore FREED_MEMORY */ sp = TAILQ_FIRST(&strq->outqueue); if (sp == NULL) { *locked = 0; if (send_lock_up == 0) { SCTP_TCB_SEND_LOCK(stcb); send_lock_up = 1; } sp = TAILQ_FIRST(&strq->outqueue); if (sp) { goto one_more_time; } if (strq->last_msg_incomplete) { SCTP_PRINTF("Huh? Stream:%d lm_in_c=%d but queue is NULL\n", strq->stream_no, strq->last_msg_incomplete); strq->last_msg_incomplete = 0; } to_move = 0; if (send_lock_up) { SCTP_TCB_SEND_UNLOCK(stcb); send_lock_up = 0; } goto out_of; } if ((sp->msg_is_complete) && (sp->length == 0)) { if (sp->sender_all_done) { /* * We are doing differed cleanup. Last time through * when we took all the data the sender_all_done was * not set. */ if ((sp->put_last_out == 0) && (sp->discard_rest == 0)) { SCTP_PRINTF("Gak, put out entire msg with NO end!-1\n"); SCTP_PRINTF("sender_done:%d len:%d msg_comp:%d put_last_out:%d send_lock:%d\n", sp->sender_all_done, sp->length, sp->msg_is_complete, sp->put_last_out, send_lock_up); } if ((TAILQ_NEXT(sp, next) == NULL) && (send_lock_up == 0)) { SCTP_TCB_SEND_LOCK(stcb); send_lock_up = 1; } atomic_subtract_int(&asoc->stream_queue_cnt, 1); TAILQ_REMOVE(&strq->outqueue, sp, next); if ((strq->state == SCTP_STREAM_RESET_PENDING) && (strq->chunks_on_queues == 0) && TAILQ_EMPTY(&strq->outqueue)) { stcb->asoc.trigger_reset = 1; } stcb->asoc.ss_functions.sctp_ss_remove_from_stream(stcb, asoc, strq, sp, send_lock_up); if (sp->net) { sctp_free_remote_addr(sp->net); sp->net = NULL; } if (sp->data) { sctp_m_freem(sp->data); sp->data = NULL; } sctp_free_a_strmoq(stcb, sp, so_locked); /* we can't be locked to it */ *locked = 0; stcb->asoc.locked_on_sending = NULL; if (send_lock_up) { SCTP_TCB_SEND_UNLOCK(stcb); send_lock_up = 0; } /* back to get the next msg */ goto one_more_time; } else { /* * sender just finished this but still holds a * reference */ *locked = 1; *giveup = 1; to_move = 0; goto out_of; } } else { /* is there some to get */ if (sp->length == 0) { /* no */ *locked = 1; *giveup = 1; to_move = 0; goto out_of; } else if (sp->discard_rest) { if (send_lock_up == 0) { SCTP_TCB_SEND_LOCK(stcb); send_lock_up = 1; } /* Whack down the size */ atomic_subtract_int(&stcb->asoc.total_output_queue_size, sp->length); if ((stcb->sctp_socket != NULL) && ((stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) || (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL))) { atomic_subtract_int(&stcb->sctp_socket->so_snd.sb_cc, sp->length); } if (sp->data) { sctp_m_freem(sp->data); sp->data = NULL; sp->tail_mbuf = NULL; } sp->length = 0; sp->some_taken = 1; *locked = 1; *giveup = 1; to_move = 0; goto out_of; } } some_taken = sp->some_taken; re_look: length = sp->length; if (sp->msg_is_complete) { /* The message is complete */ to_move = min(length, frag_point); if (to_move == length) { /* All of it fits in the MTU */ if (sp->some_taken) { rcv_flags |= SCTP_DATA_LAST_FRAG; sp->put_last_out = 1; } else { rcv_flags |= SCTP_DATA_NOT_FRAG; sp->put_last_out = 1; } } else { /* Not all of it fits, we fragment */ if (sp->some_taken == 0) { rcv_flags |= SCTP_DATA_FIRST_FRAG; } sp->some_taken = 1; } } else { to_move = sctp_can_we_split_this(stcb, length, goal_mtu, frag_point, eeor_mode); if (to_move) { /*- * We use a snapshot of length in case it * is expanding during the compare. */ uint32_t llen; llen = length; if (to_move >= llen) { to_move = llen; if (send_lock_up == 0) { /*- * We are taking all of an incomplete msg * thus we need a send lock. */ SCTP_TCB_SEND_LOCK(stcb); send_lock_up = 1; if (sp->msg_is_complete) { /* * the sender finished the * msg */ goto re_look; } } } if (sp->some_taken == 0) { rcv_flags |= SCTP_DATA_FIRST_FRAG; sp->some_taken = 1; } } else { /* Nothing to take. */ if (sp->some_taken) { *locked = 1; } *giveup = 1; to_move = 0; goto out_of; } } /* If we reach here, we can copy out a chunk */ sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* No chunk memory */ *giveup = 1; to_move = 0; goto out_of; } /* * Setup for unordered if needed by looking at the user sent info * flags. */ if (sp->sinfo_flags & SCTP_UNORDERED) { rcv_flags |= SCTP_DATA_UNORDERED; } if ((SCTP_BASE_SYSCTL(sctp_enable_sack_immediately) && ((sp->sinfo_flags & SCTP_EOF) == SCTP_EOF)) || ((sp->sinfo_flags & SCTP_SACK_IMMEDIATELY) == SCTP_SACK_IMMEDIATELY)) { rcv_flags |= SCTP_DATA_SACK_IMMEDIATELY; } /* clear out the chunk before setting up */ memset(chk, 0, sizeof(*chk)); chk->rec.data.rcv_flags = rcv_flags; if (to_move >= length) { /* we think we can steal the whole thing */ if ((sp->sender_all_done == 0) && (send_lock_up == 0)) { SCTP_TCB_SEND_LOCK(stcb); send_lock_up = 1; } if (to_move < sp->length) { /* bail, it changed */ goto dont_do_it; } chk->data = sp->data; chk->last_mbuf = sp->tail_mbuf; /* register the stealing */ sp->data = sp->tail_mbuf = NULL; } else { struct mbuf *m; dont_do_it: chk->data = SCTP_M_COPYM(sp->data, 0, to_move, M_NOWAIT); chk->last_mbuf = NULL; if (chk->data == NULL) { sp->some_taken = some_taken; sctp_free_a_chunk(stcb, chk, so_locked); *bail = 1; to_move = 0; goto out_of; } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(chk->data, SCTP_MBUF_ICOPY); } #endif /* Pull off the data */ m_adj(sp->data, to_move); /* Now lets work our way down and compact it */ m = sp->data; while (m && (SCTP_BUF_LEN(m) == 0)) { sp->data = SCTP_BUF_NEXT(m); SCTP_BUF_NEXT(m) = NULL; if (sp->tail_mbuf == m) { /*- * Freeing tail? TSNH since * we supposedly were taking less * than the sp->length. */ #ifdef INVARIANTS panic("Huh, freing tail? - TSNH"); #else SCTP_PRINTF("Huh, freeing tail? - TSNH\n"); sp->tail_mbuf = sp->data = NULL; sp->length = 0; #endif } sctp_m_free(m); m = sp->data; } } if (SCTP_BUF_IS_EXTENDED(chk->data)) { chk->copy_by_ref = 1; } else { chk->copy_by_ref = 0; } /* * get last_mbuf and counts of mb useage This is ugly but hopefully * its only one mbuf. */ if (chk->last_mbuf == NULL) { chk->last_mbuf = chk->data; while (SCTP_BUF_NEXT(chk->last_mbuf) != NULL) { chk->last_mbuf = SCTP_BUF_NEXT(chk->last_mbuf); } } if (to_move > length) { /*- This should not happen either * since we always lower to_move to the size * of sp->length if its larger. */ #ifdef INVARIANTS panic("Huh, how can to_move be larger?"); #else SCTP_PRINTF("Huh, how can to_move be larger?\n"); sp->length = 0; #endif } else { atomic_subtract_int(&sp->length, to_move); } if (M_LEADINGSPACE(chk->data) < (int)sizeof(struct sctp_data_chunk)) { /* Not enough room for a chunk header, get some */ struct mbuf *m; m = sctp_get_mbuf_for_msg(1, 0, M_NOWAIT, 0, MT_DATA); if (m == NULL) { /* * we're in trouble here. _PREPEND below will free * all the data if there is no leading space, so we * must put the data back and restore. */ if (send_lock_up == 0) { SCTP_TCB_SEND_LOCK(stcb); send_lock_up = 1; } if (sp->data == NULL) { /* unsteal the data */ sp->data = chk->data; sp->tail_mbuf = chk->last_mbuf; } else { struct mbuf *m_tmp; /* reassemble the data */ m_tmp = sp->data; sp->data = chk->data; SCTP_BUF_NEXT(chk->last_mbuf) = m_tmp; } sp->some_taken = some_taken; atomic_add_int(&sp->length, to_move); chk->data = NULL; *bail = 1; sctp_free_a_chunk(stcb, chk, so_locked); to_move = 0; goto out_of; } else { SCTP_BUF_LEN(m) = 0; SCTP_BUF_NEXT(m) = chk->data; chk->data = m; M_ALIGN(chk->data, 4); } } SCTP_BUF_PREPEND(chk->data, sizeof(struct sctp_data_chunk), M_NOWAIT); if (chk->data == NULL) { /* HELP, TSNH since we assured it would not above? */ #ifdef INVARIANTS panic("prepend failes HELP?"); #else SCTP_PRINTF("prepend fails HELP?\n"); sctp_free_a_chunk(stcb, chk, so_locked); #endif *bail = 1; to_move = 0; goto out_of; } sctp_snd_sb_alloc(stcb, sizeof(struct sctp_data_chunk)); chk->book_size = chk->send_size = (to_move + sizeof(struct sctp_data_chunk)); chk->book_size_scale = 0; chk->sent = SCTP_DATAGRAM_UNSENT; chk->flags = 0; chk->asoc = &stcb->asoc; chk->pad_inplace = 0; chk->no_fr_allowed = 0; chk->rec.data.stream_seq = strq->next_sequence_send; if ((rcv_flags & SCTP_DATA_LAST_FRAG) && !(rcv_flags & SCTP_DATA_UNORDERED)) { strq->next_sequence_send++; } chk->rec.data.stream_number = sp->stream; chk->rec.data.payloadtype = sp->ppid; chk->rec.data.context = sp->context; chk->rec.data.doing_fast_retransmit = 0; chk->rec.data.timetodrop = sp->ts; chk->flags = sp->act_flags; if (sp->net) { chk->whoTo = sp->net; atomic_add_int(&chk->whoTo->ref_count, 1); } else chk->whoTo = NULL; if (sp->holds_key_ref) { chk->auth_keyid = sp->auth_keyid; sctp_auth_key_acquire(stcb, chk->auth_keyid); chk->holds_key_ref = 1; } chk->rec.data.TSN_seq = atomic_fetchadd_int(&asoc->sending_seq, 1); if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_AT_SEND_2_OUTQ) { sctp_misc_ints(SCTP_STRMOUT_LOG_SEND, (uintptr_t) stcb, sp->length, (uint32_t) ((chk->rec.data.stream_number << 16) | chk->rec.data.stream_seq), chk->rec.data.TSN_seq); } dchkh = mtod(chk->data, struct sctp_data_chunk *); /* * Put the rest of the things in place now. Size was done earlier in * previous loop prior to padding. */ #ifdef SCTP_ASOCLOG_OF_TSNS SCTP_TCB_LOCK_ASSERT(stcb); if (asoc->tsn_out_at >= SCTP_TSN_LOG_SIZE) { asoc->tsn_out_at = 0; asoc->tsn_out_wrapped = 1; } asoc->out_tsnlog[asoc->tsn_out_at].tsn = chk->rec.data.TSN_seq; asoc->out_tsnlog[asoc->tsn_out_at].strm = chk->rec.data.stream_number; asoc->out_tsnlog[asoc->tsn_out_at].seq = chk->rec.data.stream_seq; asoc->out_tsnlog[asoc->tsn_out_at].sz = chk->send_size; asoc->out_tsnlog[asoc->tsn_out_at].flgs = chk->rec.data.rcv_flags; asoc->out_tsnlog[asoc->tsn_out_at].stcb = (void *)stcb; asoc->out_tsnlog[asoc->tsn_out_at].in_pos = asoc->tsn_out_at; asoc->out_tsnlog[asoc->tsn_out_at].in_out = 2; asoc->tsn_out_at++; #endif dchkh->ch.chunk_type = SCTP_DATA; dchkh->ch.chunk_flags = chk->rec.data.rcv_flags; dchkh->dp.tsn = htonl(chk->rec.data.TSN_seq); dchkh->dp.stream_id = htons(strq->stream_no); dchkh->dp.stream_sequence = htons(chk->rec.data.stream_seq); dchkh->dp.protocol_id = chk->rec.data.payloadtype; dchkh->ch.chunk_length = htons(chk->send_size); /* Now advance the chk->send_size by the actual pad needed. */ if (chk->send_size < SCTP_SIZE32(chk->book_size)) { /* need a pad */ struct mbuf *lm; int pads; pads = SCTP_SIZE32(chk->book_size) - chk->send_size; lm = sctp_pad_lastmbuf(chk->data, pads, chk->last_mbuf); if (lm != NULL) { chk->last_mbuf = lm; chk->pad_inplace = 1; } chk->send_size += pads; } if (PR_SCTP_ENABLED(chk->flags)) { asoc->pr_sctp_cnt++; } if (sp->msg_is_complete && (sp->length == 0) && (sp->sender_all_done)) { /* All done pull and kill the message */ atomic_subtract_int(&asoc->stream_queue_cnt, 1); if (sp->put_last_out == 0) { SCTP_PRINTF("Gak, put out entire msg with NO end!-2\n"); SCTP_PRINTF("sender_done:%d len:%d msg_comp:%d put_last_out:%d send_lock:%d\n", sp->sender_all_done, sp->length, sp->msg_is_complete, sp->put_last_out, send_lock_up); } if ((send_lock_up == 0) && (TAILQ_NEXT(sp, next) == NULL)) { SCTP_TCB_SEND_LOCK(stcb); send_lock_up = 1; } TAILQ_REMOVE(&strq->outqueue, sp, next); if ((strq->state == SCTP_STREAM_RESET_PENDING) && (strq->chunks_on_queues == 0) && TAILQ_EMPTY(&strq->outqueue)) { stcb->asoc.trigger_reset = 1; } stcb->asoc.ss_functions.sctp_ss_remove_from_stream(stcb, asoc, strq, sp, send_lock_up); if (sp->net) { sctp_free_remote_addr(sp->net); sp->net = NULL; } if (sp->data) { sctp_m_freem(sp->data); sp->data = NULL; } sctp_free_a_strmoq(stcb, sp, so_locked); /* we can't be locked to it */ *locked = 0; stcb->asoc.locked_on_sending = NULL; } else { /* more to go, we are locked */ *locked = 1; } asoc->chunks_on_out_queue++; strq->chunks_on_queues++; TAILQ_INSERT_TAIL(&asoc->send_queue, chk, sctp_next); asoc->send_queue_cnt++; out_of: if (send_lock_up) { SCTP_TCB_SEND_UNLOCK(stcb); } return (to_move); } static void sctp_fill_outqueue(struct sctp_tcb *stcb, struct sctp_nets *net, int frag_point, int eeor_mode, int *quit_now, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { struct sctp_association *asoc; struct sctp_stream_out *strq; int goal_mtu, moved_how_much, total_moved = 0, bail = 0; int locked, giveup; SCTP_TCB_LOCK_ASSERT(stcb); asoc = &stcb->asoc; switch (net->ro._l_addr.sa.sa_family) { #ifdef INET case AF_INET: goal_mtu = net->mtu - SCTP_MIN_V4_OVERHEAD; break; #endif #ifdef INET6 case AF_INET6: goal_mtu = net->mtu - SCTP_MIN_OVERHEAD; break; #endif default: /* TSNH */ goal_mtu = net->mtu; break; } /* Need an allowance for the data chunk header too */ goal_mtu -= sizeof(struct sctp_data_chunk); /* must make even word boundary */ goal_mtu &= 0xfffffffc; if (asoc->locked_on_sending) { /* We are stuck on one stream until the message completes. */ strq = asoc->locked_on_sending; locked = 1; } else { strq = stcb->asoc.ss_functions.sctp_ss_select_stream(stcb, net, asoc); locked = 0; } while ((goal_mtu > 0) && strq) { giveup = 0; bail = 0; moved_how_much = sctp_move_to_outqueue(stcb, strq, goal_mtu, frag_point, &locked, &giveup, eeor_mode, &bail, so_locked); if (moved_how_much) stcb->asoc.ss_functions.sctp_ss_scheduled(stcb, net, asoc, strq, moved_how_much); if (locked) { asoc->locked_on_sending = strq; if ((moved_how_much == 0) || (giveup) || bail) /* no more to move for now */ break; } else { asoc->locked_on_sending = NULL; if ((giveup) || bail) { break; } strq = stcb->asoc.ss_functions.sctp_ss_select_stream(stcb, net, asoc); if (strq == NULL) { break; } } total_moved += moved_how_much; goal_mtu -= (moved_how_much + sizeof(struct sctp_data_chunk)); goal_mtu &= 0xfffffffc; } if (bail) *quit_now = 1; stcb->asoc.ss_functions.sctp_ss_packet_done(stcb, net, asoc); if (total_moved == 0) { if ((stcb->asoc.sctp_cmt_on_off == 0) && (net == stcb->asoc.primary_destination)) { /* ran dry for primary network net */ SCTP_STAT_INCR(sctps_primary_randry); } else if (stcb->asoc.sctp_cmt_on_off > 0) { /* ran dry with CMT on */ SCTP_STAT_INCR(sctps_cmt_randry); } } } void sctp_fix_ecn_echo(struct sctp_association *asoc) { struct sctp_tmit_chunk *chk; TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if (chk->rec.chunk_id.id == SCTP_ECN_ECHO) { chk->sent = SCTP_DATAGRAM_UNSENT; } } } void sctp_move_chunks_from_net(struct sctp_tcb *stcb, struct sctp_nets *net) { struct sctp_association *asoc; struct sctp_tmit_chunk *chk; struct sctp_stream_queue_pending *sp; unsigned int i; if (net == NULL) { return; } asoc = &stcb->asoc; for (i = 0; i < stcb->asoc.streamoutcnt; i++) { TAILQ_FOREACH(sp, &stcb->asoc.strmout[i].outqueue, next) { if (sp->net == net) { sctp_free_remote_addr(sp->net); sp->net = NULL; } } } TAILQ_FOREACH(chk, &asoc->send_queue, sctp_next) { if (chk->whoTo == net) { sctp_free_remote_addr(chk->whoTo); chk->whoTo = NULL; } } } int sctp_med_chunk_output(struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_association *asoc, int *num_out, int *reason_code, int control_only, int from_where, struct timeval *now, int *now_filled, int frag_point, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { /** * Ok this is the generic chunk service queue. we must do the * following: - Service the stream queue that is next, moving any * message (note I must get a complete message i.e. FIRST/MIDDLE and * LAST to the out queue in one pass) and assigning TSN's - Check to * see if the cwnd/rwnd allows any output, if so we go ahead and * fomulate and send the low level chunks. Making sure to combine * any control in the control chunk queue also. */ struct sctp_nets *net, *start_at, *sack_goes_to = NULL, *old_start_at = NULL; struct mbuf *outchain, *endoutchain; struct sctp_tmit_chunk *chk, *nchk; /* temp arrays for unlinking */ struct sctp_tmit_chunk *data_list[SCTP_MAX_DATA_BUNDLING]; int no_fragmentflg, error; unsigned int max_rwnd_per_dest, max_send_per_dest; int one_chunk, hbflag, skip_data_for_this_net; int asconf, cookie, no_out_cnt; int bundle_at, ctl_cnt, no_data_chunks, eeor_mode; unsigned int mtu, r_mtu, omtu, mx_mtu, to_out; int tsns_sent = 0; uint32_t auth_offset = 0; struct sctp_auth_chunk *auth = NULL; uint16_t auth_keyid; int override_ok = 1; int skip_fill_up = 0; int data_auth_reqd = 0; /* * JRS 5/14/07 - Add flag for whether a heartbeat is sent to the * destination. */ int quit_now = 0; *num_out = 0; *reason_code = 0; auth_keyid = stcb->asoc.authinfo.active_keyid; if ((asoc->state & SCTP_STATE_SHUTDOWN_PENDING) || (asoc->state & SCTP_STATE_SHUTDOWN_RECEIVED) || (sctp_is_feature_on(inp, SCTP_PCB_FLAGS_EXPLICIT_EOR))) { eeor_mode = 1; } else { eeor_mode = 0; } ctl_cnt = no_out_cnt = asconf = cookie = 0; /* * First lets prime the pump. For each destination, if there is room * in the flight size, attempt to pull an MTU's worth out of the * stream queues into the general send_queue */ #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xC2, 2); #endif SCTP_TCB_LOCK_ASSERT(stcb); hbflag = 0; if (control_only) no_data_chunks = 1; else no_data_chunks = 0; /* Nothing to possible to send? */ if ((TAILQ_EMPTY(&asoc->control_send_queue) || (asoc->ctrl_queue_cnt == stcb->asoc.ecn_echo_cnt_onq)) && TAILQ_EMPTY(&asoc->asconf_send_queue) && TAILQ_EMPTY(&asoc->send_queue) && stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { nothing_to_send: *reason_code = 9; return (0); } if (asoc->peers_rwnd == 0) { /* No room in peers rwnd */ *reason_code = 1; if (asoc->total_flight > 0) { /* we are allowed one chunk in flight */ no_data_chunks = 1; } } if (stcb->asoc.ecn_echo_cnt_onq) { /* Record where a sack goes, if any */ if (no_data_chunks && (asoc->ctrl_queue_cnt == stcb->asoc.ecn_echo_cnt_onq)) { /* Nothing but ECNe to send - we don't do that */ goto nothing_to_send; } TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if ((chk->rec.chunk_id.id == SCTP_SELECTIVE_ACK) || (chk->rec.chunk_id.id == SCTP_NR_SELECTIVE_ACK)) { sack_goes_to = chk->whoTo; break; } } } max_rwnd_per_dest = ((asoc->peers_rwnd + asoc->total_flight) / asoc->numnets); if (stcb->sctp_socket) max_send_per_dest = SCTP_SB_LIMIT_SND(stcb->sctp_socket) / asoc->numnets; else max_send_per_dest = 0; if (no_data_chunks == 0) { /* How many non-directed chunks are there? */ TAILQ_FOREACH(chk, &asoc->send_queue, sctp_next) { if (chk->whoTo == NULL) { /* * We already have non-directed chunks on * the queue, no need to do a fill-up. */ skip_fill_up = 1; break; } } } if ((no_data_chunks == 0) && (skip_fill_up == 0) && (!stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc))) { TAILQ_FOREACH(net, &asoc->nets, sctp_next) { /* * This for loop we are in takes in each net, if * its's got space in cwnd and has data sent to it * (when CMT is off) then it calls * sctp_fill_outqueue for the net. This gets data on * the send queue for that network. * * In sctp_fill_outqueue TSN's are assigned and data is * copied out of the stream buffers. Note mostly * copy by reference (we hope). */ net->window_probe = 0; if ((net != stcb->asoc.alternate) && ((net->dest_state & SCTP_ADDR_PF) || (!(net->dest_state & SCTP_ADDR_REACHABLE)) || (net->dest_state & SCTP_ADDR_UNCONFIRMED))) { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, net, 1, SCTP_CWND_LOG_FILL_OUTQ_CALLED); } continue; } if ((stcb->asoc.cc_functions.sctp_cwnd_new_transmission_begins) && (net->flight_size == 0)) { (*stcb->asoc.cc_functions.sctp_cwnd_new_transmission_begins) (stcb, net); } if (net->flight_size >= net->cwnd) { /* skip this network, no room - can't fill */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, net, 3, SCTP_CWND_LOG_FILL_OUTQ_CALLED); } continue; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, net, 4, SCTP_CWND_LOG_FILL_OUTQ_CALLED); } sctp_fill_outqueue(stcb, net, frag_point, eeor_mode, &quit_now, so_locked); if (quit_now) { /* memory alloc failure */ no_data_chunks = 1; break; } } } /* now service each destination and send out what we can for it */ /* Nothing to send? */ if (TAILQ_EMPTY(&asoc->control_send_queue) && TAILQ_EMPTY(&asoc->asconf_send_queue) && TAILQ_EMPTY(&asoc->send_queue)) { *reason_code = 8; return (0); } if (asoc->sctp_cmt_on_off > 0) { /* get the last start point */ start_at = asoc->last_net_cmt_send_started; if (start_at == NULL) { /* null so to beginning */ start_at = TAILQ_FIRST(&asoc->nets); } else { start_at = TAILQ_NEXT(asoc->last_net_cmt_send_started, sctp_next); if (start_at == NULL) { start_at = TAILQ_FIRST(&asoc->nets); } } asoc->last_net_cmt_send_started = start_at; } else { start_at = TAILQ_FIRST(&asoc->nets); } TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if (chk->whoTo == NULL) { if (asoc->alternate) { chk->whoTo = asoc->alternate; } else { chk->whoTo = asoc->primary_destination; } atomic_add_int(&chk->whoTo->ref_count, 1); } } old_start_at = NULL; again_one_more_time: for (net = start_at; net != NULL; net = TAILQ_NEXT(net, sctp_next)) { /* how much can we send? */ /* SCTPDBG("Examine for sending net:%x\n", (uint32_t)net); */ if (old_start_at && (old_start_at == net)) { /* through list ocmpletely. */ break; } tsns_sent = 0xa; if (TAILQ_EMPTY(&asoc->control_send_queue) && TAILQ_EMPTY(&asoc->asconf_send_queue) && (net->flight_size >= net->cwnd)) { /* * Nothing on control or asconf and flight is full, * we can skip even in the CMT case. */ continue; } bundle_at = 0; endoutchain = outchain = NULL; no_fragmentflg = 1; one_chunk = 0; if (net->dest_state & SCTP_ADDR_UNCONFIRMED) { skip_data_for_this_net = 1; } else { skip_data_for_this_net = 0; } switch (((struct sockaddr *)&net->ro._l_addr)->sa_family) { #ifdef INET case AF_INET: mtu = net->mtu - SCTP_MIN_V4_OVERHEAD; break; #endif #ifdef INET6 case AF_INET6: mtu = net->mtu - SCTP_MIN_OVERHEAD; break; #endif default: /* TSNH */ mtu = net->mtu; break; } mx_mtu = mtu; to_out = 0; if (mtu > asoc->peers_rwnd) { if (asoc->total_flight > 0) { /* We have a packet in flight somewhere */ r_mtu = asoc->peers_rwnd; } else { /* We are always allowed to send one MTU out */ one_chunk = 1; r_mtu = mtu; } } else { r_mtu = mtu; } error = 0; /************************/ /* ASCONF transmission */ /************************/ /* Now first lets go through the asconf queue */ TAILQ_FOREACH_SAFE(chk, &asoc->asconf_send_queue, sctp_next, nchk) { if (chk->rec.chunk_id.id != SCTP_ASCONF) { continue; } if (chk->whoTo == NULL) { if (asoc->alternate == NULL) { if (asoc->primary_destination != net) { break; } } else { if (asoc->alternate != net) { break; } } } else { if (chk->whoTo != net) { break; } } if (chk->data == NULL) { break; } if (chk->sent != SCTP_DATAGRAM_UNSENT && chk->sent != SCTP_DATAGRAM_RESEND) { break; } /* * if no AUTH is yet included and this chunk * requires it, make sure to account for it. We * don't apply the size until the AUTH chunk is * actually added below in case there is no room for * this chunk. NOTE: we overload the use of "omtu" * here */ if ((auth == NULL) && sctp_auth_is_required_chunk(chk->rec.chunk_id.id, stcb->asoc.peer_auth_chunks)) { omtu = sctp_get_auth_chunk_len(stcb->asoc.peer_hmac_id); } else omtu = 0; /* Here we do NOT factor the r_mtu */ if ((chk->send_size < (int)(mtu - omtu)) || (chk->flags & CHUNK_FLAGS_FRAGMENT_OK)) { /* * We probably should glom the mbuf chain * from the chk->data for control but the * problem is it becomes yet one more level * of tracking to do if for some reason * output fails. Then I have got to * reconstruct the merged control chain.. el * yucko.. for now we take the easy way and * do the copy */ /* * Add an AUTH chunk, if chunk requires it * save the offset into the chain for AUTH */ if ((auth == NULL) && (sctp_auth_is_required_chunk(chk->rec.chunk_id.id, stcb->asoc.peer_auth_chunks))) { outchain = sctp_add_auth_chunk(outchain, &endoutchain, &auth, &auth_offset, stcb, chk->rec.chunk_id.id); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } outchain = sctp_copy_mbufchain(chk->data, outchain, &endoutchain, (int)chk->rec.chunk_id.can_take_data, chk->send_size, chk->copy_by_ref); if (outchain == NULL) { *reason_code = 8; SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); /* update our MTU size */ if (mtu > (chk->send_size + omtu)) mtu -= (chk->send_size + omtu); else mtu = 0; to_out += (chk->send_size + omtu); /* Do clear IP_DF ? */ if (chk->flags & CHUNK_FLAGS_FRAGMENT_OK) { no_fragmentflg = 0; } if (chk->rec.chunk_id.can_take_data) chk->data = NULL; /* * set hb flag since we can use these for * RTO */ hbflag = 1; asconf = 1; /* * should sysctl this: don't bundle data * with ASCONF since it requires AUTH */ no_data_chunks = 1; chk->sent = SCTP_DATAGRAM_SENT; if (chk->whoTo == NULL) { chk->whoTo = net; atomic_add_int(&net->ref_count, 1); } chk->snd_count++; if (mtu == 0) { /* * Ok we are out of room but we can * output without effecting the * flight size since this little guy * is a control only packet. */ sctp_timer_start(SCTP_TIMER_TYPE_ASCONF, inp, stcb, net); /* * do NOT clear the asconf flag as * it is used to do appropriate * source address selection. */ if (*now_filled == 0) { (void)SCTP_GETTIME_TIMEVAL(now); *now_filled = 1; } net->last_sent_time = *now; hbflag = 0; if ((error = sctp_lowlevel_chunk_output(inp, stcb, net, (struct sockaddr *)&net->ro._l_addr, outchain, auth_offset, auth, stcb->asoc.authinfo.active_keyid, no_fragmentflg, 0, asconf, inp->sctp_lport, stcb->rport, htonl(stcb->asoc.peer_vtag), net->port, NULL, 0, 0, so_locked))) { /* * error, we could not * output */ SCTPDBG(SCTP_DEBUG_OUTPUT3, "Gak send error %d\n", error); if (from_where == 0) { SCTP_STAT_INCR(sctps_lowlevelerrusr); } if (error == ENOBUFS) { asoc->ifp_had_enobuf = 1; SCTP_STAT_INCR(sctps_lowlevelerr); } /* error, could not output */ if (error == EHOSTUNREACH) { /* * Destination went * unreachable * during this send */ sctp_move_chunks_from_net(stcb, net); } *reason_code = 7; break; } else { asoc->ifp_had_enobuf = 0; } /* * increase the number we sent, if a * cookie is sent we don't tell them * any was sent out. */ outchain = endoutchain = NULL; auth = NULL; auth_offset = 0; if (!no_out_cnt) *num_out += ctl_cnt; /* recalc a clean slate and setup */ switch (net->ro._l_addr.sa.sa_family) { #ifdef INET case AF_INET: mtu = net->mtu - SCTP_MIN_V4_OVERHEAD; break; #endif #ifdef INET6 case AF_INET6: mtu = net->mtu - SCTP_MIN_OVERHEAD; break; #endif default: /* TSNH */ mtu = net->mtu; break; } to_out = 0; no_fragmentflg = 1; } } } if (error != 0) { /* try next net */ continue; } /************************/ /* Control transmission */ /************************/ /* Now first lets go through the control queue */ TAILQ_FOREACH_SAFE(chk, &asoc->control_send_queue, sctp_next, nchk) { if ((sack_goes_to) && (chk->rec.chunk_id.id == SCTP_ECN_ECHO) && (chk->whoTo != sack_goes_to)) { /* * if we have a sack in queue, and we are * looking at an ecn echo that is NOT queued * to where the sack is going.. */ if (chk->whoTo == net) { /* * Don't transmit it to where its * going (current net) */ continue; } else if (sack_goes_to == net) { /* * But do transmit it to this * address */ goto skip_net_check; } } if (chk->whoTo == NULL) { if (asoc->alternate == NULL) { if (asoc->primary_destination != net) { continue; } } else { if (asoc->alternate != net) { continue; } } } else { if (chk->whoTo != net) { continue; } } skip_net_check: if (chk->data == NULL) { continue; } if (chk->sent != SCTP_DATAGRAM_UNSENT) { /* * It must be unsent. Cookies and ASCONF's * hang around but there timers will force * when marked for resend. */ continue; } /* * if no AUTH is yet included and this chunk * requires it, make sure to account for it. We * don't apply the size until the AUTH chunk is * actually added below in case there is no room for * this chunk. NOTE: we overload the use of "omtu" * here */ if ((auth == NULL) && sctp_auth_is_required_chunk(chk->rec.chunk_id.id, stcb->asoc.peer_auth_chunks)) { omtu = sctp_get_auth_chunk_len(stcb->asoc.peer_hmac_id); } else omtu = 0; /* Here we do NOT factor the r_mtu */ if ((chk->send_size <= (int)(mtu - omtu)) || (chk->flags & CHUNK_FLAGS_FRAGMENT_OK)) { /* * We probably should glom the mbuf chain * from the chk->data for control but the * problem is it becomes yet one more level * of tracking to do if for some reason * output fails. Then I have got to * reconstruct the merged control chain.. el * yucko.. for now we take the easy way and * do the copy */ /* * Add an AUTH chunk, if chunk requires it * save the offset into the chain for AUTH */ if ((auth == NULL) && (sctp_auth_is_required_chunk(chk->rec.chunk_id.id, stcb->asoc.peer_auth_chunks))) { outchain = sctp_add_auth_chunk(outchain, &endoutchain, &auth, &auth_offset, stcb, chk->rec.chunk_id.id); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } outchain = sctp_copy_mbufchain(chk->data, outchain, &endoutchain, (int)chk->rec.chunk_id.can_take_data, chk->send_size, chk->copy_by_ref); if (outchain == NULL) { *reason_code = 8; SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); /* update our MTU size */ if (mtu > (chk->send_size + omtu)) mtu -= (chk->send_size + omtu); else mtu = 0; to_out += (chk->send_size + omtu); /* Do clear IP_DF ? */ if (chk->flags & CHUNK_FLAGS_FRAGMENT_OK) { no_fragmentflg = 0; } if (chk->rec.chunk_id.can_take_data) chk->data = NULL; /* Mark things to be removed, if needed */ if ((chk->rec.chunk_id.id == SCTP_SELECTIVE_ACK) || (chk->rec.chunk_id.id == SCTP_NR_SELECTIVE_ACK) || /* EY */ (chk->rec.chunk_id.id == SCTP_HEARTBEAT_REQUEST) || (chk->rec.chunk_id.id == SCTP_HEARTBEAT_ACK) || (chk->rec.chunk_id.id == SCTP_SHUTDOWN) || (chk->rec.chunk_id.id == SCTP_SHUTDOWN_ACK) || (chk->rec.chunk_id.id == SCTP_OPERATION_ERROR) || (chk->rec.chunk_id.id == SCTP_COOKIE_ACK) || (chk->rec.chunk_id.id == SCTP_ECN_CWR) || (chk->rec.chunk_id.id == SCTP_PACKET_DROPPED) || (chk->rec.chunk_id.id == SCTP_ASCONF_ACK)) { if (chk->rec.chunk_id.id == SCTP_HEARTBEAT_REQUEST) { hbflag = 1; } /* remove these chunks at the end */ if ((chk->rec.chunk_id.id == SCTP_SELECTIVE_ACK) || (chk->rec.chunk_id.id == SCTP_NR_SELECTIVE_ACK)) { /* turn off the timer */ if (SCTP_OS_TIMER_PENDING(&stcb->asoc.dack_timer.timer)) { sctp_timer_stop(SCTP_TIMER_TYPE_RECV, inp, stcb, net, SCTP_FROM_SCTP_OUTPUT + SCTP_LOC_1); } } ctl_cnt++; } else { /* * Other chunks, since they have * timers running (i.e. COOKIE) we * just "trust" that it gets sent or * retransmitted. */ ctl_cnt++; if (chk->rec.chunk_id.id == SCTP_COOKIE_ECHO) { cookie = 1; no_out_cnt = 1; } else if (chk->rec.chunk_id.id == SCTP_ECN_ECHO) { /* * Increment ecne send count * here this means we may be * over-zealous in our * counting if the send * fails, but its the best * place to do it (we used * to do it in the queue of * the chunk, but that did * not tell how many times * it was sent. */ SCTP_STAT_INCR(sctps_sendecne); } chk->sent = SCTP_DATAGRAM_SENT; if (chk->whoTo == NULL) { chk->whoTo = net; atomic_add_int(&net->ref_count, 1); } chk->snd_count++; } if (mtu == 0) { /* * Ok we are out of room but we can * output without effecting the * flight size since this little guy * is a control only packet. */ if (asconf) { sctp_timer_start(SCTP_TIMER_TYPE_ASCONF, inp, stcb, net); /* * do NOT clear the asconf * flag as it is used to do * appropriate source * address selection. */ } if (cookie) { sctp_timer_start(SCTP_TIMER_TYPE_COOKIE, inp, stcb, net); cookie = 0; } /* Only HB or ASCONF advances time */ if (hbflag) { if (*now_filled == 0) { (void)SCTP_GETTIME_TIMEVAL(now); *now_filled = 1; } net->last_sent_time = *now; hbflag = 0; } if ((error = sctp_lowlevel_chunk_output(inp, stcb, net, (struct sockaddr *)&net->ro._l_addr, outchain, auth_offset, auth, stcb->asoc.authinfo.active_keyid, no_fragmentflg, 0, asconf, inp->sctp_lport, stcb->rport, htonl(stcb->asoc.peer_vtag), net->port, NULL, 0, 0, so_locked))) { /* * error, we could not * output */ SCTPDBG(SCTP_DEBUG_OUTPUT3, "Gak send error %d\n", error); if (from_where == 0) { SCTP_STAT_INCR(sctps_lowlevelerrusr); } if (error == ENOBUFS) { asoc->ifp_had_enobuf = 1; SCTP_STAT_INCR(sctps_lowlevelerr); } if (error == EHOSTUNREACH) { /* * Destination went * unreachable * during this send */ sctp_move_chunks_from_net(stcb, net); } *reason_code = 7; break; } else { asoc->ifp_had_enobuf = 0; } /* * increase the number we sent, if a * cookie is sent we don't tell them * any was sent out. */ outchain = endoutchain = NULL; auth = NULL; auth_offset = 0; if (!no_out_cnt) *num_out += ctl_cnt; /* recalc a clean slate and setup */ switch (net->ro._l_addr.sa.sa_family) { #ifdef INET case AF_INET: mtu = net->mtu - SCTP_MIN_V4_OVERHEAD; break; #endif #ifdef INET6 case AF_INET6: mtu = net->mtu - SCTP_MIN_OVERHEAD; break; #endif default: /* TSNH */ mtu = net->mtu; break; } to_out = 0; no_fragmentflg = 1; } } } if (error != 0) { /* try next net */ continue; } /* JRI: if dest is in PF state, do not send data to it */ if ((asoc->sctp_cmt_on_off > 0) && (net != stcb->asoc.alternate) && (net->dest_state & SCTP_ADDR_PF)) { goto no_data_fill; } if (net->flight_size >= net->cwnd) { goto no_data_fill; } if ((asoc->sctp_cmt_on_off > 0) && (SCTP_BASE_SYSCTL(sctp_buffer_splitting) & SCTP_RECV_BUFFER_SPLITTING) && (net->flight_size > max_rwnd_per_dest)) { goto no_data_fill; } /* * We need a specific accounting for the usage of the send * buffer. We also need to check the number of messages per * net. For now, this is better than nothing and it disabled * by default... */ if ((asoc->sctp_cmt_on_off > 0) && (SCTP_BASE_SYSCTL(sctp_buffer_splitting) & SCTP_SEND_BUFFER_SPLITTING) && (max_send_per_dest > 0) && (net->flight_size > max_send_per_dest)) { goto no_data_fill; } /*********************/ /* Data transmission */ /*********************/ /* * if AUTH for DATA is required and no AUTH has been added * yet, account for this in the mtu now... if no data can be * bundled, this adjustment won't matter anyways since the * packet will be going out... */ data_auth_reqd = sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.peer_auth_chunks); if (data_auth_reqd && (auth == NULL)) { mtu -= sctp_get_auth_chunk_len(stcb->asoc.peer_hmac_id); } /* now lets add any data within the MTU constraints */ switch (((struct sockaddr *)&net->ro._l_addr)->sa_family) { #ifdef INET case AF_INET: if (net->mtu > SCTP_MIN_V4_OVERHEAD) omtu = net->mtu - SCTP_MIN_V4_OVERHEAD; else omtu = 0; break; #endif #ifdef INET6 case AF_INET6: if (net->mtu > SCTP_MIN_OVERHEAD) omtu = net->mtu - SCTP_MIN_OVERHEAD; else omtu = 0; break; #endif default: /* TSNH */ omtu = 0; break; } if ((((SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) || (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_RECEIVED)) && (skip_data_for_this_net == 0)) || (cookie)) { TAILQ_FOREACH_SAFE(chk, &asoc->send_queue, sctp_next, nchk) { if (no_data_chunks) { /* let only control go out */ *reason_code = 1; break; } if (net->flight_size >= net->cwnd) { /* skip this net, no room for data */ *reason_code = 2; break; } if ((chk->whoTo != NULL) && (chk->whoTo != net)) { /* Don't send the chunk on this net */ continue; } if (asoc->sctp_cmt_on_off == 0) { if ((asoc->alternate) && (asoc->alternate != net) && (chk->whoTo == NULL)) { continue; } else if ((net != asoc->primary_destination) && (asoc->alternate == NULL) && (chk->whoTo == NULL)) { continue; } } if ((chk->send_size > omtu) && ((chk->flags & CHUNK_FLAGS_FRAGMENT_OK) == 0)) { /*- * strange, we have a chunk that is * to big for its destination and * yet no fragment ok flag. * Something went wrong when the * PMTU changed...we did not mark * this chunk for some reason?? I * will fix it here by letting IP * fragment it for now and printing * a warning. This really should not * happen ... */ SCTP_PRINTF("Warning chunk of %d bytes > mtu:%d and yet PMTU disc missed\n", chk->send_size, mtu); chk->flags |= CHUNK_FLAGS_FRAGMENT_OK; } if (SCTP_BASE_SYSCTL(sctp_enable_sack_immediately) && ((asoc->state & SCTP_STATE_SHUTDOWN_PENDING) == SCTP_STATE_SHUTDOWN_PENDING)) { struct sctp_data_chunk *dchkh; dchkh = mtod(chk->data, struct sctp_data_chunk *); dchkh->ch.chunk_flags |= SCTP_DATA_SACK_IMMEDIATELY; } if (((chk->send_size <= mtu) && (chk->send_size <= r_mtu)) || ((chk->flags & CHUNK_FLAGS_FRAGMENT_OK) && (chk->send_size <= asoc->peers_rwnd))) { /* ok we will add this one */ /* * Add an AUTH chunk, if chunk * requires it, save the offset into * the chain for AUTH */ if (data_auth_reqd) { if (auth == NULL) { outchain = sctp_add_auth_chunk(outchain, &endoutchain, &auth, &auth_offset, stcb, SCTP_DATA); auth_keyid = chk->auth_keyid; override_ok = 0; SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } else if (override_ok) { /* * use this data's * keyid */ auth_keyid = chk->auth_keyid; override_ok = 0; } else if (auth_keyid != chk->auth_keyid) { /* * different keyid, * so done bundling */ break; } } outchain = sctp_copy_mbufchain(chk->data, outchain, &endoutchain, 0, chk->send_size, chk->copy_by_ref); if (outchain == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT3, "No memory?\n"); if (!SCTP_OS_TIMER_PENDING(&net->rxt_timer.timer)) { sctp_timer_start(SCTP_TIMER_TYPE_SEND, inp, stcb, net); } *reason_code = 3; SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } /* upate our MTU size */ /* Do clear IP_DF ? */ if (chk->flags & CHUNK_FLAGS_FRAGMENT_OK) { no_fragmentflg = 0; } /* unsigned subtraction of mtu */ if (mtu > chk->send_size) mtu -= chk->send_size; else mtu = 0; /* unsigned subtraction of r_mtu */ if (r_mtu > chk->send_size) r_mtu -= chk->send_size; else r_mtu = 0; to_out += chk->send_size; if ((to_out > mx_mtu) && no_fragmentflg) { #ifdef INVARIANTS panic("Exceeding mtu of %d out size is %d", mx_mtu, to_out); #else SCTP_PRINTF("Exceeding mtu of %d out size is %d\n", mx_mtu, to_out); #endif } chk->window_probe = 0; data_list[bundle_at++] = chk; if (bundle_at >= SCTP_MAX_DATA_BUNDLING) { break; } if (chk->sent == SCTP_DATAGRAM_UNSENT) { if ((chk->rec.data.rcv_flags & SCTP_DATA_UNORDERED) == 0) { SCTP_STAT_INCR_COUNTER64(sctps_outorderchunks); } else { SCTP_STAT_INCR_COUNTER64(sctps_outunorderchunks); } if (((chk->rec.data.rcv_flags & SCTP_DATA_LAST_FRAG) == SCTP_DATA_LAST_FRAG) && ((chk->rec.data.rcv_flags & SCTP_DATA_FIRST_FRAG) == 0)) /* * Count number of * user msg's that * were fragmented * we do this by * counting when we * see a LAST * fragment only. */ SCTP_STAT_INCR_COUNTER64(sctps_fragusrmsgs); } if ((mtu == 0) || (r_mtu == 0) || (one_chunk)) { if ((one_chunk) && (stcb->asoc.total_flight == 0)) { data_list[0]->window_probe = 1; net->window_probe = 1; } break; } } else { /* * Must be sent in order of the * TSN's (on a network) */ break; } } /* for (chunk gather loop for this net) */ } /* if asoc.state OPEN */ no_data_fill: /* Is there something to send for this destination? */ if (outchain) { /* We may need to start a control timer or two */ if (asconf) { sctp_timer_start(SCTP_TIMER_TYPE_ASCONF, inp, stcb, net); /* * do NOT clear the asconf flag as it is * used to do appropriate source address * selection. */ } if (cookie) { sctp_timer_start(SCTP_TIMER_TYPE_COOKIE, inp, stcb, net); cookie = 0; } /* must start a send timer if data is being sent */ if (bundle_at && (!SCTP_OS_TIMER_PENDING(&net->rxt_timer.timer))) { /* * no timer running on this destination * restart it. */ sctp_timer_start(SCTP_TIMER_TYPE_SEND, inp, stcb, net); } if (bundle_at || hbflag) { /* For data/asconf and hb set time */ if (*now_filled == 0) { (void)SCTP_GETTIME_TIMEVAL(now); *now_filled = 1; } net->last_sent_time = *now; } /* Now send it, if there is anything to send :> */ if ((error = sctp_lowlevel_chunk_output(inp, stcb, net, (struct sockaddr *)&net->ro._l_addr, outchain, auth_offset, auth, auth_keyid, no_fragmentflg, bundle_at, asconf, inp->sctp_lport, stcb->rport, htonl(stcb->asoc.peer_vtag), net->port, NULL, 0, 0, so_locked))) { /* error, we could not output */ SCTPDBG(SCTP_DEBUG_OUTPUT3, "Gak send error %d\n", error); if (from_where == 0) { SCTP_STAT_INCR(sctps_lowlevelerrusr); } if (error == ENOBUFS) { SCTP_STAT_INCR(sctps_lowlevelerr); asoc->ifp_had_enobuf = 1; } if (error == EHOSTUNREACH) { /* * Destination went unreachable * during this send */ sctp_move_chunks_from_net(stcb, net); } *reason_code = 6; /*- * I add this line to be paranoid. As far as * I can tell the continue, takes us back to * the top of the for, but just to make sure * I will reset these again here. */ ctl_cnt = bundle_at = 0; continue; /* This takes us back to the * for() for the nets. */ } else { asoc->ifp_had_enobuf = 0; } endoutchain = NULL; auth = NULL; auth_offset = 0; if (!no_out_cnt) { *num_out += (ctl_cnt + bundle_at); } if (bundle_at) { /* setup for a RTO measurement */ tsns_sent = data_list[0]->rec.data.TSN_seq; /* fill time if not already filled */ if (*now_filled == 0) { (void)SCTP_GETTIME_TIMEVAL(&asoc->time_last_sent); *now_filled = 1; *now = asoc->time_last_sent; } else { asoc->time_last_sent = *now; } if (net->rto_needed) { data_list[0]->do_rtt = 1; net->rto_needed = 0; } SCTP_STAT_INCR_BY(sctps_senddata, bundle_at); sctp_clean_up_datalist(stcb, asoc, data_list, bundle_at, net); } if (one_chunk) { break; } } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, net, tsns_sent, SCTP_CWND_LOG_FROM_SEND); } } if (old_start_at == NULL) { old_start_at = start_at; start_at = TAILQ_FIRST(&asoc->nets); if (old_start_at) goto again_one_more_time; } /* * At the end there should be no NON timed chunks hanging on this * queue. */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, net, *num_out, SCTP_CWND_LOG_FROM_SEND); } if ((*num_out == 0) && (*reason_code == 0)) { *reason_code = 4; } else { *reason_code = 5; } sctp_clean_up_ctl(stcb, asoc, so_locked); return (0); } void sctp_queue_op_err(struct sctp_tcb *stcb, struct mbuf *op_err) { /*- * Prepend a OPERATIONAL_ERROR chunk header and put on the end of * the control chunk queue. */ struct sctp_chunkhdr *hdr; struct sctp_tmit_chunk *chk; struct mbuf *mat, *last_mbuf; uint32_t chunk_length; uint16_t padding_length; SCTP_TCB_LOCK_ASSERT(stcb); SCTP_BUF_PREPEND(op_err, sizeof(struct sctp_chunkhdr), M_NOWAIT); if (op_err == NULL) { return; } last_mbuf = NULL; chunk_length = 0; for (mat = op_err; mat != NULL; mat = SCTP_BUF_NEXT(mat)) { chunk_length += SCTP_BUF_LEN(mat); if (SCTP_BUF_NEXT(mat) == NULL) { last_mbuf = mat; } } if (chunk_length > SCTP_MAX_CHUNK_LENGTH) { sctp_m_freem(op_err); return; } padding_length = chunk_length % 4; if (padding_length != 0) { padding_length = 4 - padding_length; } if (padding_length != 0) { if (sctp_add_pad_tombuf(last_mbuf, padding_length) == NULL) { sctp_m_freem(op_err); return; } } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ sctp_m_freem(op_err); return; } chk->copy_by_ref = 0; chk->send_size = (uint16_t) chunk_length; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->asoc = &stcb->asoc; chk->data = op_err; chk->whoTo = NULL; hdr = mtod(op_err, struct sctp_chunkhdr *); hdr->chunk_type = SCTP_OPERATION_ERROR; hdr->chunk_flags = 0; hdr->chunk_length = htons(chk->send_size); TAILQ_INSERT_TAIL(&chk->asoc->control_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; } int sctp_send_cookie_echo(struct mbuf *m, int offset, struct sctp_tcb *stcb, struct sctp_nets *net) { /*- * pull out the cookie and put it at the front of the control chunk * queue. */ int at; struct mbuf *cookie; struct sctp_paramhdr parm, *phdr; struct sctp_chunkhdr *hdr; struct sctp_tmit_chunk *chk; uint16_t ptype, plen; SCTP_TCB_LOCK_ASSERT(stcb); /* First find the cookie in the param area */ cookie = NULL; at = offset + sizeof(struct sctp_init_chunk); for (;;) { phdr = sctp_get_next_param(m, at, &parm, sizeof(parm)); if (phdr == NULL) { return (-3); } ptype = ntohs(phdr->param_type); plen = ntohs(phdr->param_length); if (ptype == SCTP_STATE_COOKIE) { int pad; /* found the cookie */ if ((pad = (plen % 4))) { plen += 4 - pad; } cookie = SCTP_M_COPYM(m, at, plen, M_NOWAIT); if (cookie == NULL) { /* No memory */ return (-2); } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(cookie, SCTP_MBUF_ICOPY); } #endif break; } at += SCTP_SIZE32(plen); } /* ok, we got the cookie lets change it into a cookie echo chunk */ /* first the change from param to cookie */ hdr = mtod(cookie, struct sctp_chunkhdr *); hdr->chunk_type = SCTP_COOKIE_ECHO; hdr->chunk_flags = 0; /* get the chunk stuff now and place it in the FRONT of the queue */ sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ sctp_m_freem(cookie); return (-5); } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_COOKIE_ECHO; chk->rec.chunk_id.can_take_data = 0; chk->flags = CHUNK_FLAGS_FRAGMENT_OK; chk->send_size = plen; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->asoc = &stcb->asoc; chk->data = cookie; chk->whoTo = net; atomic_add_int(&chk->whoTo->ref_count, 1); TAILQ_INSERT_HEAD(&chk->asoc->control_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; return (0); } void sctp_send_heartbeat_ack(struct sctp_tcb *stcb, struct mbuf *m, int offset, int chk_length, struct sctp_nets *net) { /* * take a HB request and make it into a HB ack and send it. */ struct mbuf *outchain; struct sctp_chunkhdr *chdr; struct sctp_tmit_chunk *chk; if (net == NULL) /* must have a net pointer */ return; outchain = SCTP_M_COPYM(m, offset, chk_length, M_NOWAIT); if (outchain == NULL) { /* gak out of memory */ return; } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(outchain, SCTP_MBUF_ICOPY); } #endif chdr = mtod(outchain, struct sctp_chunkhdr *); chdr->chunk_type = SCTP_HEARTBEAT_ACK; chdr->chunk_flags = 0; if (chk_length % 4) { /* need pad */ uint32_t cpthis = 0; int padlen; padlen = 4 - (chk_length % 4); m_copyback(outchain, chk_length, padlen, (caddr_t)&cpthis); } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ sctp_m_freem(outchain); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_HEARTBEAT_ACK; chk->rec.chunk_id.can_take_data = 1; chk->flags = 0; chk->send_size = chk_length; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->asoc = &stcb->asoc; chk->data = outchain; chk->whoTo = net; atomic_add_int(&chk->whoTo->ref_count, 1); TAILQ_INSERT_TAIL(&chk->asoc->control_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; } void sctp_send_cookie_ack(struct sctp_tcb *stcb) { /* formulate and queue a cookie-ack back to sender */ struct mbuf *cookie_ack; struct sctp_chunkhdr *hdr; struct sctp_tmit_chunk *chk; SCTP_TCB_LOCK_ASSERT(stcb); cookie_ack = sctp_get_mbuf_for_msg(sizeof(struct sctp_chunkhdr), 0, M_NOWAIT, 1, MT_HEADER); if (cookie_ack == NULL) { /* no mbuf's */ return; } SCTP_BUF_RESV_UF(cookie_ack, SCTP_MIN_OVERHEAD); sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ sctp_m_freem(cookie_ack); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_COOKIE_ACK; chk->rec.chunk_id.can_take_data = 1; chk->flags = 0; chk->send_size = sizeof(struct sctp_chunkhdr); chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->asoc = &stcb->asoc; chk->data = cookie_ack; if (chk->asoc->last_control_chunk_from != NULL) { chk->whoTo = chk->asoc->last_control_chunk_from; atomic_add_int(&chk->whoTo->ref_count, 1); } else { chk->whoTo = NULL; } hdr = mtod(cookie_ack, struct sctp_chunkhdr *); hdr->chunk_type = SCTP_COOKIE_ACK; hdr->chunk_flags = 0; hdr->chunk_length = htons(chk->send_size); SCTP_BUF_LEN(cookie_ack) = chk->send_size; TAILQ_INSERT_TAIL(&chk->asoc->control_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; return; } void sctp_send_shutdown_ack(struct sctp_tcb *stcb, struct sctp_nets *net) { /* formulate and queue a SHUTDOWN-ACK back to the sender */ struct mbuf *m_shutdown_ack; struct sctp_shutdown_ack_chunk *ack_cp; struct sctp_tmit_chunk *chk; m_shutdown_ack = sctp_get_mbuf_for_msg(sizeof(struct sctp_shutdown_ack_chunk), 0, M_NOWAIT, 1, MT_HEADER); if (m_shutdown_ack == NULL) { /* no mbuf's */ return; } SCTP_BUF_RESV_UF(m_shutdown_ack, SCTP_MIN_OVERHEAD); sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ sctp_m_freem(m_shutdown_ack); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_SHUTDOWN_ACK; chk->rec.chunk_id.can_take_data = 1; chk->flags = 0; chk->send_size = sizeof(struct sctp_chunkhdr); chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->data = m_shutdown_ack; chk->whoTo = net; if (chk->whoTo) { atomic_add_int(&chk->whoTo->ref_count, 1); } ack_cp = mtod(m_shutdown_ack, struct sctp_shutdown_ack_chunk *); ack_cp->ch.chunk_type = SCTP_SHUTDOWN_ACK; ack_cp->ch.chunk_flags = 0; ack_cp->ch.chunk_length = htons(chk->send_size); SCTP_BUF_LEN(m_shutdown_ack) = chk->send_size; TAILQ_INSERT_TAIL(&chk->asoc->control_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; return; } void sctp_send_shutdown(struct sctp_tcb *stcb, struct sctp_nets *net) { /* formulate and queue a SHUTDOWN to the sender */ struct mbuf *m_shutdown; struct sctp_shutdown_chunk *shutdown_cp; struct sctp_tmit_chunk *chk; m_shutdown = sctp_get_mbuf_for_msg(sizeof(struct sctp_shutdown_chunk), 0, M_NOWAIT, 1, MT_HEADER); if (m_shutdown == NULL) { /* no mbuf's */ return; } SCTP_BUF_RESV_UF(m_shutdown, SCTP_MIN_OVERHEAD); sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ sctp_m_freem(m_shutdown); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_SHUTDOWN; chk->rec.chunk_id.can_take_data = 1; chk->flags = 0; chk->send_size = sizeof(struct sctp_shutdown_chunk); chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->data = m_shutdown; chk->whoTo = net; if (chk->whoTo) { atomic_add_int(&chk->whoTo->ref_count, 1); } shutdown_cp = mtod(m_shutdown, struct sctp_shutdown_chunk *); shutdown_cp->ch.chunk_type = SCTP_SHUTDOWN; shutdown_cp->ch.chunk_flags = 0; shutdown_cp->ch.chunk_length = htons(chk->send_size); shutdown_cp->cumulative_tsn_ack = htonl(stcb->asoc.cumulative_tsn); SCTP_BUF_LEN(m_shutdown) = chk->send_size; TAILQ_INSERT_TAIL(&chk->asoc->control_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; return; } void sctp_send_asconf(struct sctp_tcb *stcb, struct sctp_nets *net, int addr_locked) { /* * formulate and queue an ASCONF to the peer. ASCONF parameters * should be queued on the assoc queue. */ struct sctp_tmit_chunk *chk; struct mbuf *m_asconf; int len; SCTP_TCB_LOCK_ASSERT(stcb); if ((!TAILQ_EMPTY(&stcb->asoc.asconf_send_queue)) && (!sctp_is_feature_on(stcb->sctp_ep, SCTP_PCB_FLAGS_MULTIPLE_ASCONFS))) { /* can't send a new one if there is one in flight already */ return; } /* compose an ASCONF chunk, maximum length is PMTU */ m_asconf = sctp_compose_asconf(stcb, &len, addr_locked); if (m_asconf == NULL) { return; } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ sctp_m_freem(m_asconf); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_ASCONF; chk->rec.chunk_id.can_take_data = 0; chk->flags = CHUNK_FLAGS_FRAGMENT_OK; chk->data = m_asconf; chk->send_size = len; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->asoc = &stcb->asoc; chk->whoTo = net; if (chk->whoTo) { atomic_add_int(&chk->whoTo->ref_count, 1); } TAILQ_INSERT_TAIL(&chk->asoc->asconf_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; return; } void sctp_send_asconf_ack(struct sctp_tcb *stcb) { /* * formulate and queue a asconf-ack back to sender. the asconf-ack * must be stored in the tcb. */ struct sctp_tmit_chunk *chk; struct sctp_asconf_ack *ack, *latest_ack; struct mbuf *m_ack; struct sctp_nets *net = NULL; SCTP_TCB_LOCK_ASSERT(stcb); /* Get the latest ASCONF-ACK */ latest_ack = TAILQ_LAST(&stcb->asoc.asconf_ack_sent, sctp_asconf_ackhead); if (latest_ack == NULL) { return; } if (latest_ack->last_sent_to != NULL && latest_ack->last_sent_to == stcb->asoc.last_control_chunk_from) { /* we're doing a retransmission */ net = sctp_find_alternate_net(stcb, stcb->asoc.last_control_chunk_from, 0); if (net == NULL) { /* no alternate */ if (stcb->asoc.last_control_chunk_from == NULL) { if (stcb->asoc.alternate) { net = stcb->asoc.alternate; } else { net = stcb->asoc.primary_destination; } } else { net = stcb->asoc.last_control_chunk_from; } } } else { /* normal case */ if (stcb->asoc.last_control_chunk_from == NULL) { if (stcb->asoc.alternate) { net = stcb->asoc.alternate; } else { net = stcb->asoc.primary_destination; } } else { net = stcb->asoc.last_control_chunk_from; } } latest_ack->last_sent_to = net; TAILQ_FOREACH(ack, &stcb->asoc.asconf_ack_sent, next) { if (ack->data == NULL) { continue; } /* copy the asconf_ack */ m_ack = SCTP_M_COPYM(ack->data, 0, M_COPYALL, M_NOWAIT); if (m_ack == NULL) { /* couldn't copy it */ return; } #ifdef SCTP_MBUF_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_MBUF_LOGGING_ENABLE) { sctp_log_mbc(m_ack, SCTP_MBUF_ICOPY); } #endif sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { /* no memory */ if (m_ack) sctp_m_freem(m_ack); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_ASCONF_ACK; chk->rec.chunk_id.can_take_data = 1; chk->flags = CHUNK_FLAGS_FRAGMENT_OK; chk->whoTo = net; if (chk->whoTo) { atomic_add_int(&chk->whoTo->ref_count, 1); } chk->data = m_ack; chk->send_size = ack->len; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->asoc = &stcb->asoc; TAILQ_INSERT_TAIL(&chk->asoc->control_send_queue, chk, sctp_next); chk->asoc->ctrl_queue_cnt++; } return; } static int sctp_chunk_retransmission(struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_association *asoc, int *cnt_out, struct timeval *now, int *now_filled, int *fr_done, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { /*- * send out one MTU of retransmission. If fast_retransmit is * happening we ignore the cwnd. Otherwise we obey the cwnd and * rwnd. For a Cookie or Asconf in the control chunk queue we * retransmit them by themselves. * * For data chunks we will pick out the lowest TSN's in the sent_queue * marked for resend and bundle them all together (up to a MTU of * destination). The address to send to should have been * selected/changed where the retransmission was marked (i.e. in FR * or t3-timeout routines). */ struct sctp_tmit_chunk *data_list[SCTP_MAX_DATA_BUNDLING]; struct sctp_tmit_chunk *chk, *fwd; struct mbuf *m, *endofchain; struct sctp_nets *net = NULL; uint32_t tsns_sent = 0; int no_fragmentflg, bundle_at, cnt_thru; unsigned int mtu; int error, i, one_chunk, fwd_tsn, ctl_cnt, tmr_started; struct sctp_auth_chunk *auth = NULL; uint32_t auth_offset = 0; uint16_t auth_keyid; int override_ok = 1; int data_auth_reqd = 0; uint32_t dmtu = 0; SCTP_TCB_LOCK_ASSERT(stcb); tmr_started = ctl_cnt = bundle_at = error = 0; no_fragmentflg = 1; fwd_tsn = 0; *cnt_out = 0; fwd = NULL; endofchain = m = NULL; auth_keyid = stcb->asoc.authinfo.active_keyid; #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xC3, 1); #endif if ((TAILQ_EMPTY(&asoc->sent_queue)) && (TAILQ_EMPTY(&asoc->control_send_queue))) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "SCTP hits empty queue with cnt set to %d?\n", asoc->sent_queue_retran_cnt); asoc->sent_queue_cnt = 0; asoc->sent_queue_cnt_removeable = 0; /* send back 0/0 so we enter normal transmission */ *cnt_out = 0; return (0); } TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if ((chk->rec.chunk_id.id == SCTP_COOKIE_ECHO) || (chk->rec.chunk_id.id == SCTP_STREAM_RESET) || (chk->rec.chunk_id.id == SCTP_FORWARD_CUM_TSN)) { if (chk->sent != SCTP_DATAGRAM_RESEND) { continue; } if (chk->rec.chunk_id.id == SCTP_STREAM_RESET) { if (chk != asoc->str_reset) { /* * not eligible for retran if its * not ours */ continue; } } ctl_cnt++; if (chk->rec.chunk_id.id == SCTP_FORWARD_CUM_TSN) { fwd_tsn = 1; } /* * Add an AUTH chunk, if chunk requires it save the * offset into the chain for AUTH */ if ((auth == NULL) && (sctp_auth_is_required_chunk(chk->rec.chunk_id.id, stcb->asoc.peer_auth_chunks))) { m = sctp_add_auth_chunk(m, &endofchain, &auth, &auth_offset, stcb, chk->rec.chunk_id.id); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } m = sctp_copy_mbufchain(chk->data, m, &endofchain, 0, chk->send_size, chk->copy_by_ref); break; } } one_chunk = 0; cnt_thru = 0; /* do we have control chunks to retransmit? */ if (m != NULL) { /* Start a timer no matter if we suceed or fail */ if (chk->rec.chunk_id.id == SCTP_COOKIE_ECHO) { sctp_timer_start(SCTP_TIMER_TYPE_COOKIE, inp, stcb, chk->whoTo); } else if (chk->rec.chunk_id.id == SCTP_ASCONF) sctp_timer_start(SCTP_TIMER_TYPE_ASCONF, inp, stcb, chk->whoTo); chk->snd_count++; /* update our count */ if ((error = sctp_lowlevel_chunk_output(inp, stcb, chk->whoTo, (struct sockaddr *)&chk->whoTo->ro._l_addr, m, auth_offset, auth, stcb->asoc.authinfo.active_keyid, no_fragmentflg, 0, 0, inp->sctp_lport, stcb->rport, htonl(stcb->asoc.peer_vtag), chk->whoTo->port, NULL, 0, 0, so_locked))) { SCTP_STAT_INCR(sctps_lowlevelerr); return (error); } endofchain = NULL; auth = NULL; auth_offset = 0; /* * We don't want to mark the net->sent time here since this * we use this for HB and retrans cannot measure RTT */ /* (void)SCTP_GETTIME_TIMEVAL(&chk->whoTo->last_sent_time); */ *cnt_out += 1; chk->sent = SCTP_DATAGRAM_SENT; sctp_ucount_decr(stcb->asoc.sent_queue_retran_cnt); if (fwd_tsn == 0) { return (0); } else { /* Clean up the fwd-tsn list */ sctp_clean_up_ctl(stcb, asoc, so_locked); return (0); } } /* * Ok, it is just data retransmission we need to do or that and a * fwd-tsn with it all. */ if (TAILQ_EMPTY(&asoc->sent_queue)) { return (SCTP_RETRAN_DONE); } if ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT)) { /* not yet open, resend the cookie and that is it */ return (1); } #ifdef SCTP_AUDITING_ENABLED sctp_auditing(20, inp, stcb, NULL); #endif data_auth_reqd = sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.peer_auth_chunks); TAILQ_FOREACH(chk, &asoc->sent_queue, sctp_next) { if (chk->sent != SCTP_DATAGRAM_RESEND) { /* No, not sent to this net or not ready for rtx */ continue; } if (chk->data == NULL) { SCTP_PRINTF("TSN:%x chk->snd_count:%d chk->sent:%d can't retran - no data\n", chk->rec.data.TSN_seq, chk->snd_count, chk->sent); continue; } if ((SCTP_BASE_SYSCTL(sctp_max_retran_chunk)) && (chk->snd_count >= SCTP_BASE_SYSCTL(sctp_max_retran_chunk))) { struct mbuf *op_err; char msg[SCTP_DIAG_INFO_LEN]; snprintf(msg, sizeof(msg), "TSN %8.8x retransmitted %d times, giving up", chk->rec.data.TSN_seq, chk->snd_count); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); atomic_add_int(&stcb->asoc.refcnt, 1); sctp_abort_an_association(stcb->sctp_ep, stcb, op_err, so_locked); SCTP_TCB_LOCK(stcb); atomic_subtract_int(&stcb->asoc.refcnt, 1); return (SCTP_RETRAN_EXIT); } /* pick up the net */ net = chk->whoTo; switch (net->ro._l_addr.sa.sa_family) { #ifdef INET case AF_INET: mtu = net->mtu - SCTP_MIN_V4_OVERHEAD; break; #endif #ifdef INET6 case AF_INET6: mtu = net->mtu - SCTP_MIN_OVERHEAD; break; #endif default: /* TSNH */ mtu = net->mtu; break; } if ((asoc->peers_rwnd < mtu) && (asoc->total_flight > 0)) { /* No room in peers rwnd */ uint32_t tsn; tsn = asoc->last_acked_seq + 1; if (tsn == chk->rec.data.TSN_seq) { /* * we make a special exception for this * case. The peer has no rwnd but is missing * the lowest chunk.. which is probably what * is holding up the rwnd. */ goto one_chunk_around; } return (1); } one_chunk_around: if (asoc->peers_rwnd < mtu) { one_chunk = 1; if ((asoc->peers_rwnd == 0) && (asoc->total_flight == 0)) { chk->window_probe = 1; chk->whoTo->window_probe = 1; } } #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xC3, 2); #endif bundle_at = 0; m = NULL; net->fast_retran_ip = 0; if (chk->rec.data.doing_fast_retransmit == 0) { /* * if no FR in progress skip destination that have * flight_size > cwnd. */ if (net->flight_size >= net->cwnd) { continue; } } else { /* * Mark the destination net to have FR recovery * limits put on it. */ *fr_done = 1; net->fast_retran_ip = 1; } /* * if no AUTH is yet included and this chunk requires it, * make sure to account for it. We don't apply the size * until the AUTH chunk is actually added below in case * there is no room for this chunk. */ if (data_auth_reqd && (auth == NULL)) { dmtu = sctp_get_auth_chunk_len(stcb->asoc.peer_hmac_id); } else dmtu = 0; if ((chk->send_size <= (mtu - dmtu)) || (chk->flags & CHUNK_FLAGS_FRAGMENT_OK)) { /* ok we will add this one */ if (data_auth_reqd) { if (auth == NULL) { m = sctp_add_auth_chunk(m, &endofchain, &auth, &auth_offset, stcb, SCTP_DATA); auth_keyid = chk->auth_keyid; override_ok = 0; SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } else if (override_ok) { auth_keyid = chk->auth_keyid; override_ok = 0; } else if (chk->auth_keyid != auth_keyid) { /* different keyid, so done bundling */ break; } } m = sctp_copy_mbufchain(chk->data, m, &endofchain, 0, chk->send_size, chk->copy_by_ref); if (m == NULL) { SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } /* Do clear IP_DF ? */ if (chk->flags & CHUNK_FLAGS_FRAGMENT_OK) { no_fragmentflg = 0; } /* upate our MTU size */ if (mtu > (chk->send_size + dmtu)) mtu -= (chk->send_size + dmtu); else mtu = 0; data_list[bundle_at++] = chk; if (one_chunk && (asoc->total_flight <= 0)) { SCTP_STAT_INCR(sctps_windowprobed); } } if (one_chunk == 0) { /* * now are there anymore forward from chk to pick * up? */ for (fwd = TAILQ_NEXT(chk, sctp_next); fwd != NULL; fwd = TAILQ_NEXT(fwd, sctp_next)) { if (fwd->sent != SCTP_DATAGRAM_RESEND) { /* Nope, not for retran */ continue; } if (fwd->whoTo != net) { /* Nope, not the net in question */ continue; } if (data_auth_reqd && (auth == NULL)) { dmtu = sctp_get_auth_chunk_len(stcb->asoc.peer_hmac_id); } else dmtu = 0; if (fwd->send_size <= (mtu - dmtu)) { if (data_auth_reqd) { if (auth == NULL) { m = sctp_add_auth_chunk(m, &endofchain, &auth, &auth_offset, stcb, SCTP_DATA); auth_keyid = fwd->auth_keyid; override_ok = 0; SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } else if (override_ok) { auth_keyid = fwd->auth_keyid; override_ok = 0; } else if (fwd->auth_keyid != auth_keyid) { /* * different keyid, * so done bundling */ break; } } m = sctp_copy_mbufchain(fwd->data, m, &endofchain, 0, fwd->send_size, fwd->copy_by_ref); if (m == NULL) { SCTP_LTRACE_ERR_RET(inp, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } /* Do clear IP_DF ? */ if (fwd->flags & CHUNK_FLAGS_FRAGMENT_OK) { no_fragmentflg = 0; } /* upate our MTU size */ if (mtu > (fwd->send_size + dmtu)) mtu -= (fwd->send_size + dmtu); else mtu = 0; data_list[bundle_at++] = fwd; if (bundle_at >= SCTP_MAX_DATA_BUNDLING) { break; } } else { /* can't fit so we are done */ break; } } } /* Is there something to send for this destination? */ if (m) { /* * No matter if we fail/or suceed we should start a * timer. A failure is like a lost IP packet :-) */ if (!SCTP_OS_TIMER_PENDING(&net->rxt_timer.timer)) { /* * no timer running on this destination * restart it. */ sctp_timer_start(SCTP_TIMER_TYPE_SEND, inp, stcb, net); tmr_started = 1; } /* Now lets send it, if there is anything to send :> */ if ((error = sctp_lowlevel_chunk_output(inp, stcb, net, (struct sockaddr *)&net->ro._l_addr, m, auth_offset, auth, auth_keyid, no_fragmentflg, 0, 0, inp->sctp_lport, stcb->rport, htonl(stcb->asoc.peer_vtag), net->port, NULL, 0, 0, so_locked))) { /* error, we could not output */ SCTP_STAT_INCR(sctps_lowlevelerr); return (error); } endofchain = NULL; auth = NULL; auth_offset = 0; /* For HB's */ /* * We don't want to mark the net->sent time here * since this we use this for HB and retrans cannot * measure RTT */ /* (void)SCTP_GETTIME_TIMEVAL(&net->last_sent_time); */ /* For auto-close */ cnt_thru++; if (*now_filled == 0) { (void)SCTP_GETTIME_TIMEVAL(&asoc->time_last_sent); *now = asoc->time_last_sent; *now_filled = 1; } else { asoc->time_last_sent = *now; } *cnt_out += bundle_at; #ifdef SCTP_AUDITING_ENABLED sctp_audit_log(0xC4, bundle_at); #endif if (bundle_at) { tsns_sent = data_list[0]->rec.data.TSN_seq; } for (i = 0; i < bundle_at; i++) { SCTP_STAT_INCR(sctps_sendretransdata); data_list[i]->sent = SCTP_DATAGRAM_SENT; /* * When we have a revoked data, and we * retransmit it, then we clear the revoked * flag since this flag dictates if we * subtracted from the fs */ if (data_list[i]->rec.data.chunk_was_revoked) { /* Deflate the cwnd */ data_list[i]->whoTo->cwnd -= data_list[i]->book_size; data_list[i]->rec.data.chunk_was_revoked = 0; } data_list[i]->snd_count++; sctp_ucount_decr(asoc->sent_queue_retran_cnt); /* record the time */ data_list[i]->sent_rcv_time = asoc->time_last_sent; if (data_list[i]->book_size_scale) { /* * need to double the book size on * this one */ data_list[i]->book_size_scale = 0; /* * Since we double the booksize, we * must also double the output queue * size, since this get shrunk when * we free by this amount. */ atomic_add_int(&((asoc)->total_output_queue_size), data_list[i]->book_size); data_list[i]->book_size *= 2; } else { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_RWND_ENABLE) { sctp_log_rwnd(SCTP_DECREASE_PEER_RWND, asoc->peers_rwnd, data_list[i]->send_size, SCTP_BASE_SYSCTL(sctp_peer_chunk_oh)); } asoc->peers_rwnd = sctp_sbspace_sub(asoc->peers_rwnd, (uint32_t) (data_list[i]->send_size + SCTP_BASE_SYSCTL(sctp_peer_chunk_oh))); } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_FLIGHT_LOGGING_ENABLE) { sctp_misc_ints(SCTP_FLIGHT_LOG_UP_RSND, data_list[i]->whoTo->flight_size, data_list[i]->book_size, (uintptr_t) data_list[i]->whoTo, data_list[i]->rec.data.TSN_seq); } sctp_flight_size_increase(data_list[i]); sctp_total_flight_increase(stcb, data_list[i]); if (asoc->peers_rwnd < stcb->sctp_ep->sctp_ep.sctp_sws_sender) { /* SWS sender side engages */ asoc->peers_rwnd = 0; } if ((i == 0) && (data_list[i]->rec.data.doing_fast_retransmit)) { SCTP_STAT_INCR(sctps_sendfastretrans); if ((data_list[i] == TAILQ_FIRST(&asoc->sent_queue)) && (tmr_started == 0)) { /*- * ok we just fast-retrans'd * the lowest TSN, i.e the * first on the list. In * this case we want to give * some more time to get a * SACK back without a * t3-expiring. */ sctp_timer_stop(SCTP_TIMER_TYPE_SEND, inp, stcb, net, SCTP_FROM_SCTP_OUTPUT + SCTP_LOC_2); sctp_timer_start(SCTP_TIMER_TYPE_SEND, inp, stcb, net); } } } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, net, tsns_sent, SCTP_CWND_LOG_FROM_RESEND); } #ifdef SCTP_AUDITING_ENABLED sctp_auditing(21, inp, stcb, NULL); #endif } else { /* None will fit */ return (1); } if (asoc->sent_queue_retran_cnt <= 0) { /* all done we have no more to retran */ asoc->sent_queue_retran_cnt = 0; break; } if (one_chunk) { /* No more room in rwnd */ return (1); } /* stop the for loop here. we sent out a packet */ break; } return (0); } static void sctp_timer_validation(struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_association *asoc) { struct sctp_nets *net; /* Validate that a timer is running somewhere */ TAILQ_FOREACH(net, &asoc->nets, sctp_next) { if (SCTP_OS_TIMER_PENDING(&net->rxt_timer.timer)) { /* Here is a timer */ return; } } SCTP_TCB_LOCK_ASSERT(stcb); /* Gak, we did not have a timer somewhere */ SCTPDBG(SCTP_DEBUG_OUTPUT3, "Deadlock avoided starting timer on a dest at retran\n"); if (asoc->alternate) { sctp_timer_start(SCTP_TIMER_TYPE_SEND, inp, stcb, asoc->alternate); } else { sctp_timer_start(SCTP_TIMER_TYPE_SEND, inp, stcb, asoc->primary_destination); } return; } void sctp_chunk_output(struct sctp_inpcb *inp, struct sctp_tcb *stcb, int from_where, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { /*- * Ok this is the generic chunk service queue. we must do the * following: * - See if there are retransmits pending, if so we must * do these first. * - Service the stream queue that is next, moving any * message (note I must get a complete message i.e. * FIRST/MIDDLE and LAST to the out queue in one pass) and assigning * TSN's * - Check to see if the cwnd/rwnd allows any output, if so we * go ahead and fomulate and send the low level chunks. Making sure * to combine any control in the control chunk queue also. */ struct sctp_association *asoc; struct sctp_nets *net; int error = 0, num_out, tot_out = 0, ret = 0, reason_code; unsigned int burst_cnt = 0; struct timeval now; int now_filled = 0; int nagle_on; int frag_point = sctp_get_frag_point(stcb, &stcb->asoc); int un_sent = 0; int fr_done; unsigned int tot_frs = 0; asoc = &stcb->asoc; do_it_again: /* The Nagle algorithm is only applied when handling a send call. */ if (from_where == SCTP_OUTPUT_FROM_USR_SEND) { if (sctp_is_feature_on(inp, SCTP_PCB_FLAGS_NODELAY)) { nagle_on = 0; } else { nagle_on = 1; } } else { nagle_on = 0; } SCTP_TCB_LOCK_ASSERT(stcb); un_sent = (stcb->asoc.total_output_queue_size - stcb->asoc.total_flight); if ((un_sent <= 0) && (TAILQ_EMPTY(&asoc->control_send_queue)) && (TAILQ_EMPTY(&asoc->asconf_send_queue)) && (asoc->sent_queue_retran_cnt == 0) && (asoc->trigger_reset == 0)) { /* Nothing to do unless there is something to be sent left */ return; } /* * Do we have something to send, data or control AND a sack timer * running, if so piggy-back the sack. */ if (SCTP_OS_TIMER_PENDING(&stcb->asoc.dack_timer.timer)) { sctp_send_sack(stcb, so_locked); (void)SCTP_OS_TIMER_STOP(&stcb->asoc.dack_timer.timer); } while (asoc->sent_queue_retran_cnt) { /*- * Ok, it is retransmission time only, we send out only ONE * packet with a single call off to the retran code. */ if (from_where == SCTP_OUTPUT_FROM_COOKIE_ACK) { /*- * Special hook for handling cookiess discarded * by peer that carried data. Send cookie-ack only * and then the next call with get the retran's. */ (void)sctp_med_chunk_output(inp, stcb, asoc, &num_out, &reason_code, 1, from_where, &now, &now_filled, frag_point, so_locked); return; } else if (from_where != SCTP_OUTPUT_FROM_HB_TMR) { /* if its not from a HB then do it */ fr_done = 0; ret = sctp_chunk_retransmission(inp, stcb, asoc, &num_out, &now, &now_filled, &fr_done, so_locked); if (fr_done) { tot_frs++; } } else { /* * its from any other place, we don't allow retran * output (only control) */ ret = 1; } if (ret > 0) { /* Can't send anymore */ /*- * now lets push out control by calling med-level * output once. this assures that we WILL send HB's * if queued too. */ (void)sctp_med_chunk_output(inp, stcb, asoc, &num_out, &reason_code, 1, from_where, &now, &now_filled, frag_point, so_locked); #ifdef SCTP_AUDITING_ENABLED sctp_auditing(8, inp, stcb, NULL); #endif sctp_timer_validation(inp, stcb, asoc); return; } if (ret < 0) { /*- * The count was off.. retran is not happening so do * the normal retransmission. */ #ifdef SCTP_AUDITING_ENABLED sctp_auditing(9, inp, stcb, NULL); #endif if (ret == SCTP_RETRAN_EXIT) { return; } break; } if (from_where == SCTP_OUTPUT_FROM_T3) { /* Only one transmission allowed out of a timeout */ #ifdef SCTP_AUDITING_ENABLED sctp_auditing(10, inp, stcb, NULL); #endif /* Push out any control */ (void)sctp_med_chunk_output(inp, stcb, asoc, &num_out, &reason_code, 1, from_where, &now, &now_filled, frag_point, so_locked); return; } if ((asoc->fr_max_burst > 0) && (tot_frs >= asoc->fr_max_burst)) { /* Hit FR burst limit */ return; } if ((num_out == 0) && (ret == 0)) { /* No more retrans to send */ break; } } #ifdef SCTP_AUDITING_ENABLED sctp_auditing(12, inp, stcb, NULL); #endif /* Check for bad destinations, if they exist move chunks around. */ TAILQ_FOREACH(net, &asoc->nets, sctp_next) { if (!(net->dest_state & SCTP_ADDR_REACHABLE)) { /*- * if possible move things off of this address we * still may send below due to the dormant state but * we try to find an alternate address to send to * and if we have one we move all queued data on the * out wheel to this alternate address. */ if (net->ref_count > 1) sctp_move_chunks_from_net(stcb, net); } else { /*- * if ((asoc->sat_network) || (net->addr_is_local)) * { burst_limit = asoc->max_burst * * SCTP_SAT_NETWORK_BURST_INCR; } */ if (asoc->max_burst > 0) { if (SCTP_BASE_SYSCTL(sctp_use_cwnd_based_maxburst)) { if ((net->flight_size + (asoc->max_burst * net->mtu)) < net->cwnd) { /* * JRS - Use the congestion * control given in the * congestion control module */ asoc->cc_functions.sctp_cwnd_update_after_output(stcb, net, asoc->max_burst); if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_MAXBURST_ENABLE) { sctp_log_maxburst(stcb, net, 0, asoc->max_burst, SCTP_MAX_BURST_APPLIED); } SCTP_STAT_INCR(sctps_maxburstqueued); } net->fast_retran_ip = 0; } else { if (net->flight_size == 0) { /* * Should be decaying the * cwnd here */ ; } } } } } burst_cnt = 0; do { error = sctp_med_chunk_output(inp, stcb, asoc, &num_out, &reason_code, 0, from_where, &now, &now_filled, frag_point, so_locked); if (error) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "Error %d was returned from med-c-op\n", error); if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_MAXBURST_ENABLE) { sctp_log_maxburst(stcb, asoc->primary_destination, error, burst_cnt, SCTP_MAX_BURST_ERROR_STOP); } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, NULL, error, SCTP_SEND_NOW_COMPLETES); sctp_log_cwnd(stcb, NULL, 0xdeadbeef, SCTP_SEND_NOW_COMPLETES); } break; } SCTPDBG(SCTP_DEBUG_OUTPUT3, "m-c-o put out %d\n", num_out); tot_out += num_out; burst_cnt++; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, NULL, num_out, SCTP_SEND_NOW_COMPLETES); if (num_out == 0) { sctp_log_cwnd(stcb, NULL, reason_code, SCTP_SEND_NOW_COMPLETES); } } if (nagle_on) { /* * When the Nagle algorithm is used, look at how * much is unsent, then if its smaller than an MTU * and we have data in flight we stop, except if we * are handling a fragmented user message. */ un_sent = ((stcb->asoc.total_output_queue_size - stcb->asoc.total_flight) + (stcb->asoc.stream_queue_cnt * sizeof(struct sctp_data_chunk))); if ((un_sent < (int)(stcb->asoc.smallest_mtu - SCTP_MIN_OVERHEAD)) && (stcb->asoc.total_flight > 0) && ((stcb->asoc.locked_on_sending == NULL) || sctp_is_feature_on(inp, SCTP_PCB_FLAGS_EXPLICIT_EOR))) { break; } } if (TAILQ_EMPTY(&asoc->control_send_queue) && TAILQ_EMPTY(&asoc->send_queue) && stcb->asoc.ss_functions.sctp_ss_is_empty(stcb, asoc)) { /* Nothing left to send */ break; } if ((stcb->asoc.total_output_queue_size - stcb->asoc.total_flight) <= 0) { /* Nothing left to send */ break; } } while (num_out && ((asoc->max_burst == 0) || SCTP_BASE_SYSCTL(sctp_use_cwnd_based_maxburst) || (burst_cnt < asoc->max_burst))); if (SCTP_BASE_SYSCTL(sctp_use_cwnd_based_maxburst) == 0) { if ((asoc->max_burst > 0) && (burst_cnt >= asoc->max_burst)) { SCTP_STAT_INCR(sctps_maxburstqueued); asoc->burst_limit_applied = 1; if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_MAXBURST_ENABLE) { sctp_log_maxburst(stcb, asoc->primary_destination, 0, burst_cnt, SCTP_MAX_BURST_APPLIED); } } else { asoc->burst_limit_applied = 0; } } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_CWND_LOGGING_ENABLE) { sctp_log_cwnd(stcb, NULL, tot_out, SCTP_SEND_NOW_COMPLETES); } SCTPDBG(SCTP_DEBUG_OUTPUT1, "Ok, we have put out %d chunks\n", tot_out); /*- * Now we need to clean up the control chunk chain if a ECNE is on * it. It must be marked as UNSENT again so next call will continue * to send it until such time that we get a CWR, to remove it. */ if (stcb->asoc.ecn_echo_cnt_onq) sctp_fix_ecn_echo(asoc); if (stcb->asoc.trigger_reset) { if (sctp_send_stream_reset_out_if_possible(stcb, so_locked) == 0) { goto do_it_again; } } return; } int sctp_output( struct sctp_inpcb *inp, struct mbuf *m, struct sockaddr *addr, struct mbuf *control, struct thread *p, int flags) { if (inp == NULL) { SCTP_LTRACE_ERR_RET_PKT(m, inp, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } if (inp->sctp_socket == NULL) { SCTP_LTRACE_ERR_RET_PKT(m, inp, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } return (sctp_sosend(inp->sctp_socket, addr, (struct uio *)NULL, m, control, flags, p )); } void send_forward_tsn(struct sctp_tcb *stcb, struct sctp_association *asoc) { struct sctp_tmit_chunk *chk; struct sctp_forward_tsn_chunk *fwdtsn; uint32_t advance_peer_ack_point; SCTP_TCB_LOCK_ASSERT(stcb); TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if (chk->rec.chunk_id.id == SCTP_FORWARD_CUM_TSN) { /* mark it to unsent */ chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; /* Do we correct its output location? */ if (chk->whoTo) { sctp_free_remote_addr(chk->whoTo); chk->whoTo = NULL; } goto sctp_fill_in_rest; } } /* Ok if we reach here we must build one */ sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { return; } asoc->fwd_tsn_cnt++; chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_FORWARD_CUM_TSN; chk->rec.chunk_id.can_take_data = 0; chk->flags = 0; chk->asoc = asoc; chk->whoTo = NULL; chk->data = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (chk->data == NULL) { sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); return; } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; TAILQ_INSERT_TAIL(&asoc->control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt++; sctp_fill_in_rest: /*- * Here we go through and fill out the part that deals with * stream/seq of the ones we skip. */ SCTP_BUF_LEN(chk->data) = 0; { struct sctp_tmit_chunk *at, *tp1, *last; struct sctp_strseq *strseq; unsigned int cnt_of_space, i, ovh; unsigned int space_needed; unsigned int cnt_of_skipped = 0; TAILQ_FOREACH(at, &asoc->sent_queue, sctp_next) { if ((at->sent != SCTP_FORWARD_TSN_SKIP) && (at->sent != SCTP_DATAGRAM_NR_ACKED)) { /* no more to look at */ break; } if (at->rec.data.rcv_flags & SCTP_DATA_UNORDERED) { /* We don't report these */ continue; } cnt_of_skipped++; } space_needed = (sizeof(struct sctp_forward_tsn_chunk) + (cnt_of_skipped * sizeof(struct sctp_strseq))); cnt_of_space = M_TRAILINGSPACE(chk->data); if (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_BOUND_V6) { ovh = SCTP_MIN_OVERHEAD; } else { ovh = SCTP_MIN_V4_OVERHEAD; } if (cnt_of_space > (asoc->smallest_mtu - ovh)) { /* trim to a mtu size */ cnt_of_space = asoc->smallest_mtu - ovh; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_TRY_ADVANCE) { sctp_misc_ints(SCTP_FWD_TSN_CHECK, 0xff, 0, cnt_of_skipped, asoc->advanced_peer_ack_point); } advance_peer_ack_point = asoc->advanced_peer_ack_point; if (cnt_of_space < space_needed) { /*- * ok we must trim down the chunk by lowering the * advance peer ack point. */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_TRY_ADVANCE) { sctp_misc_ints(SCTP_FWD_TSN_CHECK, 0xff, 0xff, cnt_of_space, space_needed); } cnt_of_skipped = cnt_of_space - sizeof(struct sctp_forward_tsn_chunk); cnt_of_skipped /= sizeof(struct sctp_strseq); /*- * Go through and find the TSN that will be the one * we report. */ at = TAILQ_FIRST(&asoc->sent_queue); if (at != NULL) { for (i = 0; i < cnt_of_skipped; i++) { tp1 = TAILQ_NEXT(at, sctp_next); if (tp1 == NULL) { break; } at = tp1; } } if (at && SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LOG_TRY_ADVANCE) { sctp_misc_ints(SCTP_FWD_TSN_CHECK, 0xff, cnt_of_skipped, at->rec.data.TSN_seq, asoc->advanced_peer_ack_point); } last = at; /*- * last now points to last one I can report, update * peer ack point */ if (last) advance_peer_ack_point = last->rec.data.TSN_seq; space_needed = sizeof(struct sctp_forward_tsn_chunk) + cnt_of_skipped * sizeof(struct sctp_strseq); } chk->send_size = space_needed; /* Setup the chunk */ fwdtsn = mtod(chk->data, struct sctp_forward_tsn_chunk *); fwdtsn->ch.chunk_length = htons(chk->send_size); fwdtsn->ch.chunk_flags = 0; fwdtsn->ch.chunk_type = SCTP_FORWARD_CUM_TSN; fwdtsn->new_cumulative_tsn = htonl(advance_peer_ack_point); SCTP_BUF_LEN(chk->data) = chk->send_size; fwdtsn++; /*- * Move pointer to after the fwdtsn and transfer to the * strseq pointer. */ strseq = (struct sctp_strseq *)fwdtsn; /*- * Now populate the strseq list. This is done blindly * without pulling out duplicate stream info. This is * inefficent but won't harm the process since the peer will * look at these in sequence and will thus release anything. * It could mean we exceed the PMTU and chop off some that * we could have included.. but this is unlikely (aka 1432/4 * would mean 300+ stream seq's would have to be reported in * one FWD-TSN. With a bit of work we can later FIX this to * optimize and pull out duplcates.. but it does add more * overhead. So for now... not! */ at = TAILQ_FIRST(&asoc->sent_queue); for (i = 0; i < cnt_of_skipped; i++) { tp1 = TAILQ_NEXT(at, sctp_next); if (tp1 == NULL) break; if (at->rec.data.rcv_flags & SCTP_DATA_UNORDERED) { /* We don't report these */ i--; at = tp1; continue; } if (at->rec.data.TSN_seq == advance_peer_ack_point) { at->rec.data.fwd_tsn_cnt = 0; } strseq->stream = ntohs(at->rec.data.stream_number); strseq->sequence = ntohs(at->rec.data.stream_seq); strseq++; at = tp1; } } return; } void sctp_send_sack(struct sctp_tcb *stcb, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { /*- * Queue up a SACK or NR-SACK in the control queue. * We must first check to see if a SACK or NR-SACK is * somehow on the control queue. * If so, we will take and and remove the old one. */ struct sctp_association *asoc; struct sctp_tmit_chunk *chk, *a_chk; struct sctp_sack_chunk *sack; struct sctp_nr_sack_chunk *nr_sack; struct sctp_gap_ack_block *gap_descriptor; struct sack_track *selector; int mergeable = 0; int offset; caddr_t limit; uint32_t *dup; int limit_reached = 0; unsigned int i, siz, j; unsigned int num_gap_blocks = 0, num_nr_gap_blocks = 0, space; int num_dups = 0; int space_req; uint32_t highest_tsn; uint8_t flags; uint8_t type; uint8_t tsn_map; if (stcb->asoc.nrsack_supported == 1) { type = SCTP_NR_SELECTIVE_ACK; } else { type = SCTP_SELECTIVE_ACK; } a_chk = NULL; asoc = &stcb->asoc; SCTP_TCB_LOCK_ASSERT(stcb); if (asoc->last_data_chunk_from == NULL) { /* Hmm we never received anything */ return; } sctp_slide_mapping_arrays(stcb); sctp_set_rwnd(stcb, asoc); TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if (chk->rec.chunk_id.id == type) { /* Hmm, found a sack already on queue, remove it */ TAILQ_REMOVE(&asoc->control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt--; a_chk = chk; if (a_chk->data) { sctp_m_freem(a_chk->data); a_chk->data = NULL; } if (a_chk->whoTo) { sctp_free_remote_addr(a_chk->whoTo); a_chk->whoTo = NULL; } break; } } if (a_chk == NULL) { sctp_alloc_a_chunk(stcb, a_chk); if (a_chk == NULL) { /* No memory so we drop the idea, and set a timer */ if (stcb->asoc.delayed_ack) { sctp_timer_stop(SCTP_TIMER_TYPE_RECV, stcb->sctp_ep, stcb, NULL, SCTP_FROM_SCTP_OUTPUT + SCTP_LOC_3); sctp_timer_start(SCTP_TIMER_TYPE_RECV, stcb->sctp_ep, stcb, NULL); } else { stcb->asoc.send_sack = 1; } return; } a_chk->copy_by_ref = 0; a_chk->rec.chunk_id.id = type; a_chk->rec.chunk_id.can_take_data = 1; } /* Clear our pkt counts */ asoc->data_pkts_seen = 0; a_chk->flags = 0; a_chk->asoc = asoc; a_chk->snd_count = 0; a_chk->send_size = 0; /* fill in later */ a_chk->sent = SCTP_DATAGRAM_UNSENT; a_chk->whoTo = NULL; if (!(asoc->last_data_chunk_from->dest_state & SCTP_ADDR_REACHABLE)) { /*- * Ok, the destination for the SACK is unreachable, lets see if * we can select an alternate to asoc->last_data_chunk_from */ a_chk->whoTo = sctp_find_alternate_net(stcb, asoc->last_data_chunk_from, 0); if (a_chk->whoTo == NULL) { /* Nope, no alternate */ a_chk->whoTo = asoc->last_data_chunk_from; } } else { a_chk->whoTo = asoc->last_data_chunk_from; } if (a_chk->whoTo) { atomic_add_int(&a_chk->whoTo->ref_count, 1); } if (SCTP_TSN_GT(asoc->highest_tsn_inside_map, asoc->highest_tsn_inside_nr_map)) { highest_tsn = asoc->highest_tsn_inside_map; } else { highest_tsn = asoc->highest_tsn_inside_nr_map; } if (highest_tsn == asoc->cumulative_tsn) { /* no gaps */ if (type == SCTP_SELECTIVE_ACK) { space_req = sizeof(struct sctp_sack_chunk); } else { space_req = sizeof(struct sctp_nr_sack_chunk); } } else { /* gaps get a cluster */ space_req = MCLBYTES; } /* Ok now lets formulate a MBUF with our sack */ a_chk->data = sctp_get_mbuf_for_msg(space_req, 0, M_NOWAIT, 1, MT_DATA); if ((a_chk->data == NULL) || (a_chk->whoTo == NULL)) { /* rats, no mbuf memory */ if (a_chk->data) { /* was a problem with the destination */ sctp_m_freem(a_chk->data); a_chk->data = NULL; } sctp_free_a_chunk(stcb, a_chk, so_locked); /* sa_ignore NO_NULL_CHK */ if (stcb->asoc.delayed_ack) { sctp_timer_stop(SCTP_TIMER_TYPE_RECV, stcb->sctp_ep, stcb, NULL, SCTP_FROM_SCTP_OUTPUT + SCTP_LOC_4); sctp_timer_start(SCTP_TIMER_TYPE_RECV, stcb->sctp_ep, stcb, NULL); } else { stcb->asoc.send_sack = 1; } return; } /* ok, lets go through and fill it in */ SCTP_BUF_RESV_UF(a_chk->data, SCTP_MIN_OVERHEAD); space = M_TRAILINGSPACE(a_chk->data); if (space > (a_chk->whoTo->mtu - SCTP_MIN_OVERHEAD)) { space = (a_chk->whoTo->mtu - SCTP_MIN_OVERHEAD); } limit = mtod(a_chk->data, caddr_t); limit += space; flags = 0; if ((asoc->sctp_cmt_on_off > 0) && SCTP_BASE_SYSCTL(sctp_cmt_use_dac)) { /*- * CMT DAC algorithm: If 2 (i.e., 0x10) packets have been * received, then set high bit to 1, else 0. Reset * pkts_rcvd. */ flags |= (asoc->cmt_dac_pkts_rcvd << 6); asoc->cmt_dac_pkts_rcvd = 0; } #ifdef SCTP_ASOCLOG_OF_TSNS stcb->asoc.cumack_logsnt[stcb->asoc.cumack_log_atsnt] = asoc->cumulative_tsn; stcb->asoc.cumack_log_atsnt++; if (stcb->asoc.cumack_log_atsnt >= SCTP_TSN_LOG_SIZE) { stcb->asoc.cumack_log_atsnt = 0; } #endif /* reset the readers interpretation */ stcb->freed_by_sorcv_sincelast = 0; if (type == SCTP_SELECTIVE_ACK) { sack = mtod(a_chk->data, struct sctp_sack_chunk *); nr_sack = NULL; gap_descriptor = (struct sctp_gap_ack_block *)((caddr_t)sack + sizeof(struct sctp_sack_chunk)); if (highest_tsn > asoc->mapping_array_base_tsn) { siz = (((highest_tsn - asoc->mapping_array_base_tsn) + 1) + 7) / 8; } else { siz = (((MAX_TSN - highest_tsn) + 1) + highest_tsn + 7) / 8; } } else { sack = NULL; nr_sack = mtod(a_chk->data, struct sctp_nr_sack_chunk *); gap_descriptor = (struct sctp_gap_ack_block *)((caddr_t)nr_sack + sizeof(struct sctp_nr_sack_chunk)); if (asoc->highest_tsn_inside_map > asoc->mapping_array_base_tsn) { siz = (((asoc->highest_tsn_inside_map - asoc->mapping_array_base_tsn) + 1) + 7) / 8; } else { siz = (((MAX_TSN - asoc->mapping_array_base_tsn) + 1) + asoc->highest_tsn_inside_map + 7) / 8; } } if (SCTP_TSN_GT(asoc->mapping_array_base_tsn, asoc->cumulative_tsn)) { offset = 1; } else { offset = asoc->mapping_array_base_tsn - asoc->cumulative_tsn; } if (((type == SCTP_SELECTIVE_ACK) && SCTP_TSN_GT(highest_tsn, asoc->cumulative_tsn)) || ((type == SCTP_NR_SELECTIVE_ACK) && SCTP_TSN_GT(asoc->highest_tsn_inside_map, asoc->cumulative_tsn))) { /* we have a gap .. maybe */ for (i = 0; i < siz; i++) { tsn_map = asoc->mapping_array[i]; if (type == SCTP_SELECTIVE_ACK) { tsn_map |= asoc->nr_mapping_array[i]; } if (i == 0) { /* * Clear all bits corresponding to TSNs * smaller or equal to the cumulative TSN. */ tsn_map &= (~0 << (1 - offset)); } selector = &sack_array[tsn_map]; if (mergeable && selector->right_edge) { /* * Backup, left and right edges were ok to * merge. */ num_gap_blocks--; gap_descriptor--; } if (selector->num_entries == 0) mergeable = 0; else { for (j = 0; j < selector->num_entries; j++) { if (mergeable && selector->right_edge) { /* * do a merge by NOT setting * the left side */ mergeable = 0; } else { /* * no merge, set the left * side */ mergeable = 0; gap_descriptor->start = htons((selector->gaps[j].start + offset)); } gap_descriptor->end = htons((selector->gaps[j].end + offset)); num_gap_blocks++; gap_descriptor++; if (((caddr_t)gap_descriptor + sizeof(struct sctp_gap_ack_block)) > limit) { /* no more room */ limit_reached = 1; break; } } if (selector->left_edge) { mergeable = 1; } } if (limit_reached) { /* Reached the limit stop */ break; } offset += 8; } } if ((type == SCTP_NR_SELECTIVE_ACK) && (limit_reached == 0)) { mergeable = 0; if (asoc->highest_tsn_inside_nr_map > asoc->mapping_array_base_tsn) { siz = (((asoc->highest_tsn_inside_nr_map - asoc->mapping_array_base_tsn) + 1) + 7) / 8; } else { siz = (((MAX_TSN - asoc->mapping_array_base_tsn) + 1) + asoc->highest_tsn_inside_nr_map + 7) / 8; } if (SCTP_TSN_GT(asoc->mapping_array_base_tsn, asoc->cumulative_tsn)) { offset = 1; } else { offset = asoc->mapping_array_base_tsn - asoc->cumulative_tsn; } if (SCTP_TSN_GT(asoc->highest_tsn_inside_nr_map, asoc->cumulative_tsn)) { /* we have a gap .. maybe */ for (i = 0; i < siz; i++) { tsn_map = asoc->nr_mapping_array[i]; if (i == 0) { /* * Clear all bits corresponding to * TSNs smaller or equal to the * cumulative TSN. */ tsn_map &= (~0 << (1 - offset)); } selector = &sack_array[tsn_map]; if (mergeable && selector->right_edge) { /* * Backup, left and right edges were * ok to merge. */ num_nr_gap_blocks--; gap_descriptor--; } if (selector->num_entries == 0) mergeable = 0; else { for (j = 0; j < selector->num_entries; j++) { if (mergeable && selector->right_edge) { /* * do a merge by NOT * setting the left * side */ mergeable = 0; } else { /* * no merge, set the * left side */ mergeable = 0; gap_descriptor->start = htons((selector->gaps[j].start + offset)); } gap_descriptor->end = htons((selector->gaps[j].end + offset)); num_nr_gap_blocks++; gap_descriptor++; if (((caddr_t)gap_descriptor + sizeof(struct sctp_gap_ack_block)) > limit) { /* no more room */ limit_reached = 1; break; } } if (selector->left_edge) { mergeable = 1; } } if (limit_reached) { /* Reached the limit stop */ break; } offset += 8; } } } /* now we must add any dups we are going to report. */ if ((limit_reached == 0) && (asoc->numduptsns)) { dup = (uint32_t *) gap_descriptor; for (i = 0; i < asoc->numduptsns; i++) { *dup = htonl(asoc->dup_tsns[i]); dup++; num_dups++; if (((caddr_t)dup + sizeof(uint32_t)) > limit) { /* no more room */ break; } } asoc->numduptsns = 0; } /* * now that the chunk is prepared queue it to the control chunk * queue. */ if (type == SCTP_SELECTIVE_ACK) { a_chk->send_size = sizeof(struct sctp_sack_chunk) + (num_gap_blocks + num_nr_gap_blocks) * sizeof(struct sctp_gap_ack_block) + num_dups * sizeof(int32_t); SCTP_BUF_LEN(a_chk->data) = a_chk->send_size; sack->sack.cum_tsn_ack = htonl(asoc->cumulative_tsn); sack->sack.a_rwnd = htonl(asoc->my_rwnd); sack->sack.num_gap_ack_blks = htons(num_gap_blocks); sack->sack.num_dup_tsns = htons(num_dups); sack->ch.chunk_type = type; sack->ch.chunk_flags = flags; sack->ch.chunk_length = htons(a_chk->send_size); } else { a_chk->send_size = sizeof(struct sctp_nr_sack_chunk) + (num_gap_blocks + num_nr_gap_blocks) * sizeof(struct sctp_gap_ack_block) + num_dups * sizeof(int32_t); SCTP_BUF_LEN(a_chk->data) = a_chk->send_size; nr_sack->nr_sack.cum_tsn_ack = htonl(asoc->cumulative_tsn); nr_sack->nr_sack.a_rwnd = htonl(asoc->my_rwnd); nr_sack->nr_sack.num_gap_ack_blks = htons(num_gap_blocks); nr_sack->nr_sack.num_nr_gap_ack_blks = htons(num_nr_gap_blocks); nr_sack->nr_sack.num_dup_tsns = htons(num_dups); nr_sack->nr_sack.reserved = 0; nr_sack->ch.chunk_type = type; nr_sack->ch.chunk_flags = flags; nr_sack->ch.chunk_length = htons(a_chk->send_size); } TAILQ_INSERT_TAIL(&asoc->control_send_queue, a_chk, sctp_next); asoc->my_last_reported_rwnd = asoc->my_rwnd; asoc->ctrl_queue_cnt++; asoc->send_sack = 0; SCTP_STAT_INCR(sctps_sendsacks); return; } void sctp_send_abort_tcb(struct sctp_tcb *stcb, struct mbuf *operr, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { struct mbuf *m_abort, *m, *m_last; struct mbuf *m_out, *m_end = NULL; struct sctp_abort_chunk *abort; struct sctp_auth_chunk *auth = NULL; struct sctp_nets *net; uint32_t vtag; uint32_t auth_offset = 0; uint16_t cause_len, chunk_len, padding_len; SCTP_TCB_LOCK_ASSERT(stcb); /*- * Add an AUTH chunk, if chunk requires it and save the offset into * the chain for AUTH */ if (sctp_auth_is_required_chunk(SCTP_ABORT_ASSOCIATION, stcb->asoc.peer_auth_chunks)) { m_out = sctp_add_auth_chunk(NULL, &m_end, &auth, &auth_offset, stcb, SCTP_ABORT_ASSOCIATION); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } else { m_out = NULL; } m_abort = sctp_get_mbuf_for_msg(sizeof(struct sctp_abort_chunk), 0, M_NOWAIT, 1, MT_HEADER); if (m_abort == NULL) { if (m_out) { sctp_m_freem(m_out); } if (operr) { sctp_m_freem(operr); } return; } /* link in any error */ SCTP_BUF_NEXT(m_abort) = operr; cause_len = 0; m_last = NULL; for (m = operr; m; m = SCTP_BUF_NEXT(m)) { cause_len += (uint16_t) SCTP_BUF_LEN(m); if (SCTP_BUF_NEXT(m) == NULL) { m_last = m; } } SCTP_BUF_LEN(m_abort) = sizeof(struct sctp_abort_chunk); chunk_len = (uint16_t) sizeof(struct sctp_abort_chunk) + cause_len; padding_len = SCTP_SIZE32(chunk_len) - chunk_len; if (m_out == NULL) { /* NO Auth chunk prepended, so reserve space in front */ SCTP_BUF_RESV_UF(m_abort, SCTP_MIN_OVERHEAD); m_out = m_abort; } else { /* Put AUTH chunk at the front of the chain */ SCTP_BUF_NEXT(m_end) = m_abort; } if (stcb->asoc.alternate) { net = stcb->asoc.alternate; } else { net = stcb->asoc.primary_destination; } /* Fill in the ABORT chunk header. */ abort = mtod(m_abort, struct sctp_abort_chunk *); abort->ch.chunk_type = SCTP_ABORT_ASSOCIATION; if (stcb->asoc.peer_vtag == 0) { /* This happens iff the assoc is in COOKIE-WAIT state. */ vtag = stcb->asoc.my_vtag; abort->ch.chunk_flags = SCTP_HAD_NO_TCB; } else { vtag = stcb->asoc.peer_vtag; abort->ch.chunk_flags = 0; } abort->ch.chunk_length = htons(chunk_len); /* Add padding, if necessary. */ if (padding_len > 0) { if ((m_last == NULL) || (sctp_add_pad_tombuf(m_last, padding_len) == NULL)) { sctp_m_freem(m_out); return; } } (void)sctp_lowlevel_chunk_output(stcb->sctp_ep, stcb, net, (struct sockaddr *)&net->ro._l_addr, m_out, auth_offset, auth, stcb->asoc.authinfo.active_keyid, 1, 0, 0, stcb->sctp_ep->sctp_lport, stcb->rport, htonl(vtag), stcb->asoc.primary_destination->port, NULL, 0, 0, so_locked); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); } void sctp_send_shutdown_complete(struct sctp_tcb *stcb, struct sctp_nets *net, int reflect_vtag) { /* formulate and SEND a SHUTDOWN-COMPLETE */ struct mbuf *m_shutdown_comp; struct sctp_shutdown_complete_chunk *shutdown_complete; uint32_t vtag; uint8_t flags; m_shutdown_comp = sctp_get_mbuf_for_msg(sizeof(struct sctp_chunkhdr), 0, M_NOWAIT, 1, MT_HEADER); if (m_shutdown_comp == NULL) { /* no mbuf's */ return; } if (reflect_vtag) { flags = SCTP_HAD_NO_TCB; vtag = stcb->asoc.my_vtag; } else { flags = 0; vtag = stcb->asoc.peer_vtag; } shutdown_complete = mtod(m_shutdown_comp, struct sctp_shutdown_complete_chunk *); shutdown_complete->ch.chunk_type = SCTP_SHUTDOWN_COMPLETE; shutdown_complete->ch.chunk_flags = flags; shutdown_complete->ch.chunk_length = htons(sizeof(struct sctp_shutdown_complete_chunk)); SCTP_BUF_LEN(m_shutdown_comp) = sizeof(struct sctp_shutdown_complete_chunk); (void)sctp_lowlevel_chunk_output(stcb->sctp_ep, stcb, net, (struct sockaddr *)&net->ro._l_addr, m_shutdown_comp, 0, NULL, 0, 1, 0, 0, stcb->sctp_ep->sctp_lport, stcb->rport, htonl(vtag), net->port, NULL, 0, 0, SCTP_SO_NOT_LOCKED); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); return; } static void sctp_send_resp_msg(struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, uint32_t vtag, uint8_t type, struct mbuf *cause, uint8_t mflowtype, uint32_t mflowid, uint16_t fibnum, uint32_t vrf_id, uint16_t port) { struct mbuf *o_pak; struct mbuf *mout; struct sctphdr *shout; struct sctp_chunkhdr *ch; #if defined(INET) || defined(INET6) struct udphdr *udp; int ret; #endif int len, cause_len, padding_len; #ifdef INET struct sockaddr_in *src_sin, *dst_sin; struct ip *ip; #endif #ifdef INET6 struct sockaddr_in6 *src_sin6, *dst_sin6; struct ip6_hdr *ip6; #endif /* Compute the length of the cause and add final padding. */ cause_len = 0; if (cause != NULL) { struct mbuf *m_at, *m_last = NULL; for (m_at = cause; m_at; m_at = SCTP_BUF_NEXT(m_at)) { if (SCTP_BUF_NEXT(m_at) == NULL) m_last = m_at; cause_len += SCTP_BUF_LEN(m_at); } padding_len = cause_len % 4; if (padding_len != 0) { padding_len = 4 - padding_len; } if (padding_len != 0) { if (sctp_add_pad_tombuf(m_last, padding_len) == NULL) { sctp_m_freem(cause); return; } } } else { padding_len = 0; } /* Get an mbuf for the header. */ len = sizeof(struct sctphdr) + sizeof(struct sctp_chunkhdr); switch (dst->sa_family) { #ifdef INET case AF_INET: len += sizeof(struct ip); break; #endif #ifdef INET6 case AF_INET6: len += sizeof(struct ip6_hdr); break; #endif default: break; } #if defined(INET) || defined(INET6) if (port) { len += sizeof(struct udphdr); } #endif mout = sctp_get_mbuf_for_msg(len + max_linkhdr, 1, M_NOWAIT, 1, MT_DATA); if (mout == NULL) { if (cause) { sctp_m_freem(cause); } return; } SCTP_BUF_RESV_UF(mout, max_linkhdr); SCTP_BUF_LEN(mout) = len; SCTP_BUF_NEXT(mout) = cause; M_SETFIB(mout, fibnum); mout->m_pkthdr.flowid = mflowid; M_HASHTYPE_SET(mout, mflowtype); #ifdef INET ip = NULL; #endif #ifdef INET6 ip6 = NULL; #endif switch (dst->sa_family) { #ifdef INET case AF_INET: src_sin = (struct sockaddr_in *)src; dst_sin = (struct sockaddr_in *)dst; ip = mtod(mout, struct ip *); ip->ip_v = IPVERSION; ip->ip_hl = (sizeof(struct ip) >> 2); ip->ip_tos = 0; ip->ip_off = 0; ip_fillid(ip); ip->ip_ttl = MODULE_GLOBAL(ip_defttl); if (port) { ip->ip_p = IPPROTO_UDP; } else { ip->ip_p = IPPROTO_SCTP; } ip->ip_src.s_addr = dst_sin->sin_addr.s_addr; ip->ip_dst.s_addr = src_sin->sin_addr.s_addr; ip->ip_sum = 0; len = sizeof(struct ip); shout = (struct sctphdr *)((caddr_t)ip + len); break; #endif #ifdef INET6 case AF_INET6: src_sin6 = (struct sockaddr_in6 *)src; dst_sin6 = (struct sockaddr_in6 *)dst; ip6 = mtod(mout, struct ip6_hdr *); ip6->ip6_flow = htonl(0x60000000); if (V_ip6_auto_flowlabel) { ip6->ip6_flow |= (htonl(ip6_randomflowlabel()) & IPV6_FLOWLABEL_MASK); } ip6->ip6_hlim = MODULE_GLOBAL(ip6_defhlim); if (port) { ip6->ip6_nxt = IPPROTO_UDP; } else { ip6->ip6_nxt = IPPROTO_SCTP; } ip6->ip6_src = dst_sin6->sin6_addr; ip6->ip6_dst = src_sin6->sin6_addr; len = sizeof(struct ip6_hdr); shout = (struct sctphdr *)((caddr_t)ip6 + len); break; #endif default: len = 0; shout = mtod(mout, struct sctphdr *); break; } #if defined(INET) || defined(INET6) if (port) { if (htons(SCTP_BASE_SYSCTL(sctp_udp_tunneling_port)) == 0) { sctp_m_freem(mout); return; } udp = (struct udphdr *)shout; udp->uh_sport = htons(SCTP_BASE_SYSCTL(sctp_udp_tunneling_port)); udp->uh_dport = port; udp->uh_sum = 0; udp->uh_ulen = htons(sizeof(struct udphdr) + sizeof(struct sctphdr) + sizeof(struct sctp_chunkhdr) + cause_len + padding_len); len += sizeof(struct udphdr); shout = (struct sctphdr *)((caddr_t)shout + sizeof(struct udphdr)); } else { udp = NULL; } #endif shout->src_port = sh->dest_port; shout->dest_port = sh->src_port; shout->checksum = 0; if (vtag) { shout->v_tag = htonl(vtag); } else { shout->v_tag = sh->v_tag; } len += sizeof(struct sctphdr); ch = (struct sctp_chunkhdr *)((caddr_t)shout + sizeof(struct sctphdr)); ch->chunk_type = type; if (vtag) { ch->chunk_flags = 0; } else { ch->chunk_flags = SCTP_HAD_NO_TCB; } ch->chunk_length = htons(sizeof(struct sctp_chunkhdr) + cause_len); len += sizeof(struct sctp_chunkhdr); len += cause_len + padding_len; if (SCTP_GET_HEADER_FOR_OUTPUT(o_pak)) { sctp_m_freem(mout); return; } SCTP_ATTACH_CHAIN(o_pak, mout, len); switch (dst->sa_family) { #ifdef INET case AF_INET: if (port) { if (V_udp_cksum) { udp->uh_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, udp->uh_ulen + htons(IPPROTO_UDP)); } else { udp->uh_sum = 0; } } ip->ip_len = htons(len); if (port) { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else shout->checksum = sctp_calculate_cksum(mout, sizeof(struct ip) + sizeof(struct udphdr)); SCTP_STAT_INCR(sctps_sendswcrc); #endif if (V_udp_cksum) { SCTP_ENABLE_UDP_CSUM(o_pak); } } else { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else mout->m_pkthdr.csum_flags = CSUM_SCTP; mout->m_pkthdr.csum_data = offsetof(struct sctphdr, checksum); SCTP_STAT_INCR(sctps_sendhwcrc); #endif } #ifdef SCTP_PACKET_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LAST_PACKET_TRACING) { sctp_packet_log(o_pak); } #endif SCTP_IP_OUTPUT(ret, o_pak, NULL, NULL, vrf_id); break; #endif #ifdef INET6 case AF_INET6: ip6->ip6_plen = len - sizeof(struct ip6_hdr); if (port) { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else shout->checksum = sctp_calculate_cksum(mout, sizeof(struct ip6_hdr) + sizeof(struct udphdr)); SCTP_STAT_INCR(sctps_sendswcrc); #endif if ((udp->uh_sum = in6_cksum(o_pak, IPPROTO_UDP, sizeof(struct ip6_hdr), len - sizeof(struct ip6_hdr))) == 0) { udp->uh_sum = 0xffff; } } else { #if defined(SCTP_WITH_NO_CSUM) SCTP_STAT_INCR(sctps_sendnocrc); #else mout->m_pkthdr.csum_flags = CSUM_SCTP_IPV6; mout->m_pkthdr.csum_data = offsetof(struct sctphdr, checksum); SCTP_STAT_INCR(sctps_sendhwcrc); #endif } #ifdef SCTP_PACKET_LOGGING if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_LAST_PACKET_TRACING) { sctp_packet_log(o_pak); } #endif SCTP_IP6_OUTPUT(ret, o_pak, NULL, NULL, NULL, vrf_id); break; #endif default: SCTPDBG(SCTP_DEBUG_OUTPUT1, "Unknown protocol (TSNH) type %d\n", dst->sa_family); sctp_m_freem(mout); SCTP_LTRACE_ERR_RET_PKT(mout, NULL, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, EFAULT); return; } SCTP_STAT_INCR(sctps_sendpackets); SCTP_STAT_INCR_COUNTER64(sctps_outpackets); SCTP_STAT_INCR_COUNTER64(sctps_outcontrolchunks); return; } void sctp_send_shutdown_complete2(struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, uint8_t mflowtype, uint32_t mflowid, uint16_t fibnum, uint32_t vrf_id, uint16_t port) { sctp_send_resp_msg(src, dst, sh, 0, SCTP_SHUTDOWN_COMPLETE, NULL, mflowtype, mflowid, fibnum, vrf_id, port); } void sctp_send_hb(struct sctp_tcb *stcb, struct sctp_nets *net, int so_locked #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ) { struct sctp_tmit_chunk *chk; struct sctp_heartbeat_chunk *hb; struct timeval now; SCTP_TCB_LOCK_ASSERT(stcb); if (net == NULL) { return; } (void)SCTP_GETTIME_TIMEVAL(&now); switch (net->ro._l_addr.sa.sa_family) { #ifdef INET case AF_INET: break; #endif #ifdef INET6 case AF_INET6: break; #endif default: return; } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT4, "Gak, can't get a chunk for hb\n"); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_HEARTBEAT_REQUEST; chk->rec.chunk_id.can_take_data = 1; chk->flags = 0; chk->asoc = &stcb->asoc; chk->send_size = sizeof(struct sctp_heartbeat_chunk); chk->data = sctp_get_mbuf_for_msg(chk->send_size, 0, M_NOWAIT, 1, MT_HEADER); if (chk->data == NULL) { sctp_free_a_chunk(stcb, chk, so_locked); return; } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); SCTP_BUF_LEN(chk->data) = chk->send_size; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->whoTo = net; atomic_add_int(&chk->whoTo->ref_count, 1); /* Now we have a mbuf that we can fill in with the details */ hb = mtod(chk->data, struct sctp_heartbeat_chunk *); memset(hb, 0, sizeof(struct sctp_heartbeat_chunk)); /* fill out chunk header */ hb->ch.chunk_type = SCTP_HEARTBEAT_REQUEST; hb->ch.chunk_flags = 0; hb->ch.chunk_length = htons(chk->send_size); /* Fill out hb parameter */ hb->heartbeat.hb_info.ph.param_type = htons(SCTP_HEARTBEAT_INFO); hb->heartbeat.hb_info.ph.param_length = htons(sizeof(struct sctp_heartbeat_info_param)); hb->heartbeat.hb_info.time_value_1 = now.tv_sec; hb->heartbeat.hb_info.time_value_2 = now.tv_usec; /* Did our user request this one, put it in */ hb->heartbeat.hb_info.addr_family = (uint8_t) net->ro._l_addr.sa.sa_family; hb->heartbeat.hb_info.addr_len = net->ro._l_addr.sa.sa_len; if (net->dest_state & SCTP_ADDR_UNCONFIRMED) { /* * we only take from the entropy pool if the address is not * confirmed. */ net->heartbeat_random1 = hb->heartbeat.hb_info.random_value1 = sctp_select_initial_TSN(&stcb->sctp_ep->sctp_ep); net->heartbeat_random2 = hb->heartbeat.hb_info.random_value2 = sctp_select_initial_TSN(&stcb->sctp_ep->sctp_ep); } else { net->heartbeat_random1 = hb->heartbeat.hb_info.random_value1 = 0; net->heartbeat_random2 = hb->heartbeat.hb_info.random_value2 = 0; } switch (net->ro._l_addr.sa.sa_family) { #ifdef INET case AF_INET: memcpy(hb->heartbeat.hb_info.address, &net->ro._l_addr.sin.sin_addr, sizeof(net->ro._l_addr.sin.sin_addr)); break; #endif #ifdef INET6 case AF_INET6: memcpy(hb->heartbeat.hb_info.address, &net->ro._l_addr.sin6.sin6_addr, sizeof(net->ro._l_addr.sin6.sin6_addr)); break; #endif default: if (chk->data) { sctp_m_freem(chk->data); chk->data = NULL; } sctp_free_a_chunk(stcb, chk, so_locked); return; break; } net->hb_responded = 0; TAILQ_INSERT_TAIL(&stcb->asoc.control_send_queue, chk, sctp_next); stcb->asoc.ctrl_queue_cnt++; SCTP_STAT_INCR(sctps_sendheartbeat); return; } void sctp_send_ecn_echo(struct sctp_tcb *stcb, struct sctp_nets *net, uint32_t high_tsn) { struct sctp_association *asoc; struct sctp_ecne_chunk *ecne; struct sctp_tmit_chunk *chk; if (net == NULL) { return; } asoc = &stcb->asoc; SCTP_TCB_LOCK_ASSERT(stcb); TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if ((chk->rec.chunk_id.id == SCTP_ECN_ECHO) && (net == chk->whoTo)) { /* found a previous ECN_ECHO update it if needed */ uint32_t cnt, ctsn; ecne = mtod(chk->data, struct sctp_ecne_chunk *); ctsn = ntohl(ecne->tsn); if (SCTP_TSN_GT(high_tsn, ctsn)) { ecne->tsn = htonl(high_tsn); SCTP_STAT_INCR(sctps_queue_upd_ecne); } cnt = ntohl(ecne->num_pkts_since_cwr); cnt++; ecne->num_pkts_since_cwr = htonl(cnt); return; } } /* nope could not find one to update so we must build one */ sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { return; } SCTP_STAT_INCR(sctps_queue_upd_ecne); chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_ECN_ECHO; chk->rec.chunk_id.can_take_data = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->send_size = sizeof(struct sctp_ecne_chunk); chk->data = sctp_get_mbuf_for_msg(chk->send_size, 0, M_NOWAIT, 1, MT_HEADER); if (chk->data == NULL) { sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); return; } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); SCTP_BUF_LEN(chk->data) = chk->send_size; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->whoTo = net; atomic_add_int(&chk->whoTo->ref_count, 1); stcb->asoc.ecn_echo_cnt_onq++; ecne = mtod(chk->data, struct sctp_ecne_chunk *); ecne->ch.chunk_type = SCTP_ECN_ECHO; ecne->ch.chunk_flags = 0; ecne->ch.chunk_length = htons(sizeof(struct sctp_ecne_chunk)); ecne->tsn = htonl(high_tsn); ecne->num_pkts_since_cwr = htonl(1); TAILQ_INSERT_HEAD(&stcb->asoc.control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt++; } void sctp_send_packet_dropped(struct sctp_tcb *stcb, struct sctp_nets *net, struct mbuf *m, int len, int iphlen, int bad_crc) { struct sctp_association *asoc; struct sctp_pktdrop_chunk *drp; struct sctp_tmit_chunk *chk; uint8_t *datap; int was_trunc = 0; int fullsz = 0; long spc; int offset; struct sctp_chunkhdr *ch, chunk_buf; unsigned int chk_length; if (!stcb) { return; } asoc = &stcb->asoc; SCTP_TCB_LOCK_ASSERT(stcb); if (asoc->pktdrop_supported == 0) { /*- * peer must declare support before I send one. */ return; } if (stcb->sctp_socket == NULL) { return; } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_PACKET_DROPPED; chk->rec.chunk_id.can_take_data = 1; chk->flags = 0; len -= iphlen; chk->send_size = len; /* Validate that we do not have an ABORT in here. */ offset = iphlen + sizeof(struct sctphdr); ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, offset, sizeof(*ch), (uint8_t *) & chunk_buf); while (ch != NULL) { chk_length = ntohs(ch->chunk_length); if (chk_length < sizeof(*ch)) { /* break to abort land */ break; } switch (ch->chunk_type) { case SCTP_PACKET_DROPPED: case SCTP_ABORT_ASSOCIATION: case SCTP_INITIATION_ACK: /** * We don't respond with an PKT-DROP to an ABORT * or PKT-DROP. We also do not respond to an * INIT-ACK, because we can't know if the initiation * tag is correct or not. */ sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); return; default: break; } offset += SCTP_SIZE32(chk_length); ch = (struct sctp_chunkhdr *)sctp_m_getptr(m, offset, sizeof(*ch), (uint8_t *) & chunk_buf); } if ((len + SCTP_MAX_OVERHEAD + sizeof(struct sctp_pktdrop_chunk)) > min(stcb->asoc.smallest_mtu, MCLBYTES)) { /* * only send 1 mtu worth, trim off the excess on the end. */ fullsz = len; len = min(stcb->asoc.smallest_mtu, MCLBYTES) - SCTP_MAX_OVERHEAD; was_trunc = 1; } chk->asoc = &stcb->asoc; chk->data = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (chk->data == NULL) { jump_out: sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); return; } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); drp = mtod(chk->data, struct sctp_pktdrop_chunk *); if (drp == NULL) { sctp_m_freem(chk->data); chk->data = NULL; goto jump_out; } chk->book_size = SCTP_SIZE32((chk->send_size + sizeof(struct sctp_pktdrop_chunk) + sizeof(struct sctphdr) + SCTP_MED_OVERHEAD)); chk->book_size_scale = 0; if (was_trunc) { drp->ch.chunk_flags = SCTP_PACKET_TRUNCATED; drp->trunc_len = htons(fullsz); /* * Len is already adjusted to size minus overhead above take * out the pkt_drop chunk itself from it. */ chk->send_size = len - sizeof(struct sctp_pktdrop_chunk); len = chk->send_size; } else { /* no truncation needed */ drp->ch.chunk_flags = 0; drp->trunc_len = htons(0); } if (bad_crc) { drp->ch.chunk_flags |= SCTP_BADCRC; } chk->send_size += sizeof(struct sctp_pktdrop_chunk); SCTP_BUF_LEN(chk->data) = chk->send_size; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; if (net) { /* we should hit here */ chk->whoTo = net; atomic_add_int(&chk->whoTo->ref_count, 1); } else { chk->whoTo = NULL; } drp->ch.chunk_type = SCTP_PACKET_DROPPED; drp->ch.chunk_length = htons(chk->send_size); spc = SCTP_SB_LIMIT_RCV(stcb->sctp_socket); if (spc < 0) { spc = 0; } drp->bottle_bw = htonl(spc); if (asoc->my_rwnd) { drp->current_onq = htonl(asoc->size_on_reasm_queue + asoc->size_on_all_streams + asoc->my_rwnd_control_len + stcb->sctp_socket->so_rcv.sb_cc); } else { /*- * If my rwnd is 0, possibly from mbuf depletion as well as * space used, tell the peer there is NO space aka onq == bw */ drp->current_onq = htonl(spc); } drp->reserved = 0; datap = drp->data; m_copydata(m, iphlen, len, (caddr_t)datap); TAILQ_INSERT_TAIL(&stcb->asoc.control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt++; } void sctp_send_cwr(struct sctp_tcb *stcb, struct sctp_nets *net, uint32_t high_tsn, uint8_t override) { struct sctp_association *asoc; struct sctp_cwr_chunk *cwr; struct sctp_tmit_chunk *chk; SCTP_TCB_LOCK_ASSERT(stcb); if (net == NULL) { return; } asoc = &stcb->asoc; TAILQ_FOREACH(chk, &asoc->control_send_queue, sctp_next) { if ((chk->rec.chunk_id.id == SCTP_ECN_CWR) && (net == chk->whoTo)) { /* * found a previous CWR queued to same destination * update it if needed */ uint32_t ctsn; cwr = mtod(chk->data, struct sctp_cwr_chunk *); ctsn = ntohl(cwr->tsn); if (SCTP_TSN_GT(high_tsn, ctsn)) { cwr->tsn = htonl(high_tsn); } if (override & SCTP_CWR_REDUCE_OVERRIDE) { /* Make sure override is carried */ cwr->ch.chunk_flags |= SCTP_CWR_REDUCE_OVERRIDE; } return; } } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_ECN_CWR; chk->rec.chunk_id.can_take_data = 1; chk->flags = 0; chk->asoc = &stcb->asoc; chk->send_size = sizeof(struct sctp_cwr_chunk); chk->data = sctp_get_mbuf_for_msg(chk->send_size, 0, M_NOWAIT, 1, MT_HEADER); if (chk->data == NULL) { sctp_free_a_chunk(stcb, chk, SCTP_SO_NOT_LOCKED); return; } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); SCTP_BUF_LEN(chk->data) = chk->send_size; chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; chk->whoTo = net; atomic_add_int(&chk->whoTo->ref_count, 1); cwr = mtod(chk->data, struct sctp_cwr_chunk *); cwr->ch.chunk_type = SCTP_ECN_CWR; cwr->ch.chunk_flags = override; cwr->ch.chunk_length = htons(sizeof(struct sctp_cwr_chunk)); cwr->tsn = htonl(high_tsn); TAILQ_INSERT_TAIL(&stcb->asoc.control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt++; } static int sctp_add_stream_reset_out(struct sctp_tcb *stcb, struct sctp_tmit_chunk *chk, uint32_t seq, uint32_t resp_seq, uint32_t last_sent) { uint16_t len, old_len, i; struct sctp_stream_reset_out_request *req_out; struct sctp_chunkhdr *ch; int at; int number_entries = 0; ch = mtod(chk->data, struct sctp_chunkhdr *); old_len = len = SCTP_SIZE32(ntohs(ch->chunk_length)); /* get to new offset for the param. */ req_out = (struct sctp_stream_reset_out_request *)((caddr_t)ch + len); /* now how long will this param be? */ for (i = 0; i < stcb->asoc.streamoutcnt; i++) { if ((stcb->asoc.strmout[i].state == SCTP_STREAM_RESET_PENDING) && (stcb->asoc.strmout[i].chunks_on_queues == 0) && TAILQ_EMPTY(&stcb->asoc.strmout[i].outqueue)) { number_entries++; } } if (number_entries == 0) { return (0); } if (number_entries == stcb->asoc.streamoutcnt) { number_entries = 0; } if (number_entries > SCTP_MAX_STREAMS_AT_ONCE_RESET) { number_entries = SCTP_MAX_STREAMS_AT_ONCE_RESET; } len = (sizeof(struct sctp_stream_reset_out_request) + (sizeof(uint16_t) * number_entries)); req_out->ph.param_type = htons(SCTP_STR_RESET_OUT_REQUEST); req_out->ph.param_length = htons(len); req_out->request_seq = htonl(seq); req_out->response_seq = htonl(resp_seq); req_out->send_reset_at_tsn = htonl(last_sent); at = 0; if (number_entries) { for (i = 0; i < stcb->asoc.streamoutcnt; i++) { if ((stcb->asoc.strmout[i].state == SCTP_STREAM_RESET_PENDING) && (stcb->asoc.strmout[i].chunks_on_queues == 0) && TAILQ_EMPTY(&stcb->asoc.strmout[i].outqueue)) { req_out->list_of_streams[at] = htons(i); at++; stcb->asoc.strmout[i].state = SCTP_STREAM_RESET_IN_FLIGHT; if (at >= number_entries) { break; } } } } else { for (i = 0; i < stcb->asoc.streamoutcnt; i++) { stcb->asoc.strmout[i].state = SCTP_STREAM_RESET_IN_FLIGHT; } } if (SCTP_SIZE32(len) > len) { /*- * Need to worry about the pad we may end up adding to the * end. This is easy since the struct is either aligned to 4 * bytes or 2 bytes off. */ req_out->list_of_streams[number_entries] = 0; } /* now fix the chunk length */ ch->chunk_length = htons(len + old_len); chk->book_size = len + old_len; chk->book_size_scale = 0; chk->send_size = SCTP_SIZE32(chk->book_size); SCTP_BUF_LEN(chk->data) = chk->send_size; return (1); } static void sctp_add_stream_reset_in(struct sctp_tmit_chunk *chk, int number_entries, uint16_t * list, uint32_t seq) { uint16_t len, old_len, i; struct sctp_stream_reset_in_request *req_in; struct sctp_chunkhdr *ch; ch = mtod(chk->data, struct sctp_chunkhdr *); old_len = len = SCTP_SIZE32(ntohs(ch->chunk_length)); /* get to new offset for the param. */ req_in = (struct sctp_stream_reset_in_request *)((caddr_t)ch + len); /* now how long will this param be? */ len = (sizeof(struct sctp_stream_reset_in_request) + (sizeof(uint16_t) * number_entries)); req_in->ph.param_type = htons(SCTP_STR_RESET_IN_REQUEST); req_in->ph.param_length = htons(len); req_in->request_seq = htonl(seq); if (number_entries) { for (i = 0; i < number_entries; i++) { req_in->list_of_streams[i] = htons(list[i]); } } if (SCTP_SIZE32(len) > len) { /*- * Need to worry about the pad we may end up adding to the * end. This is easy since the struct is either aligned to 4 * bytes or 2 bytes off. */ req_in->list_of_streams[number_entries] = 0; } /* now fix the chunk length */ ch->chunk_length = htons(len + old_len); chk->book_size = len + old_len; chk->book_size_scale = 0; chk->send_size = SCTP_SIZE32(chk->book_size); SCTP_BUF_LEN(chk->data) = chk->send_size; return; } static void sctp_add_stream_reset_tsn(struct sctp_tmit_chunk *chk, uint32_t seq) { uint16_t len, old_len; struct sctp_stream_reset_tsn_request *req_tsn; struct sctp_chunkhdr *ch; ch = mtod(chk->data, struct sctp_chunkhdr *); old_len = len = SCTP_SIZE32(ntohs(ch->chunk_length)); /* get to new offset for the param. */ req_tsn = (struct sctp_stream_reset_tsn_request *)((caddr_t)ch + len); /* now how long will this param be? */ len = sizeof(struct sctp_stream_reset_tsn_request); req_tsn->ph.param_type = htons(SCTP_STR_RESET_TSN_REQUEST); req_tsn->ph.param_length = htons(len); req_tsn->request_seq = htonl(seq); /* now fix the chunk length */ ch->chunk_length = htons(len + old_len); chk->send_size = len + old_len; chk->book_size = SCTP_SIZE32(chk->send_size); chk->book_size_scale = 0; SCTP_BUF_LEN(chk->data) = SCTP_SIZE32(chk->send_size); return; } void sctp_add_stream_reset_result(struct sctp_tmit_chunk *chk, uint32_t resp_seq, uint32_t result) { uint16_t len, old_len; struct sctp_stream_reset_response *resp; struct sctp_chunkhdr *ch; ch = mtod(chk->data, struct sctp_chunkhdr *); old_len = len = SCTP_SIZE32(ntohs(ch->chunk_length)); /* get to new offset for the param. */ resp = (struct sctp_stream_reset_response *)((caddr_t)ch + len); /* now how long will this param be? */ len = sizeof(struct sctp_stream_reset_response); resp->ph.param_type = htons(SCTP_STR_RESET_RESPONSE); resp->ph.param_length = htons(len); resp->response_seq = htonl(resp_seq); resp->result = ntohl(result); /* now fix the chunk length */ ch->chunk_length = htons(len + old_len); chk->book_size = len + old_len; chk->book_size_scale = 0; chk->send_size = SCTP_SIZE32(chk->book_size); SCTP_BUF_LEN(chk->data) = chk->send_size; return; } void sctp_send_deferred_reset_response(struct sctp_tcb *stcb, struct sctp_stream_reset_list *ent, int response) { struct sctp_association *asoc; struct sctp_tmit_chunk *chk; struct sctp_chunkhdr *ch; asoc = &stcb->asoc; /* * Reset our last reset action to the new one IP -> response * (PERFORMED probably). This assures that if we fail to send, a * retran from the peer will get the new response. */ asoc->last_reset_action[0] = response; if (asoc->stream_reset_outstanding) { return; } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return; } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_STREAM_RESET; chk->rec.chunk_id.can_take_data = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->book_size = sizeof(struct sctp_chunkhdr); chk->send_size = SCTP_SIZE32(chk->book_size); chk->book_size_scale = 0; chk->data = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (chk->data == NULL) { sctp_free_a_chunk(stcb, chk, SCTP_SO_LOCKED); SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return; } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); sctp_add_stream_reset_result(chk, ent->seq, response); /* setup chunk parameters */ chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; if (stcb->asoc.alternate) { chk->whoTo = stcb->asoc.alternate; } else { chk->whoTo = stcb->asoc.primary_destination; } ch = mtod(chk->data, struct sctp_chunkhdr *); ch->chunk_type = SCTP_STREAM_RESET; ch->chunk_flags = 0; ch->chunk_length = htons(chk->book_size); atomic_add_int(&chk->whoTo->ref_count, 1); SCTP_BUF_LEN(chk->data) = chk->send_size; /* insert the chunk for sending */ TAILQ_INSERT_TAIL(&asoc->control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt++; } void sctp_add_stream_reset_result_tsn(struct sctp_tmit_chunk *chk, uint32_t resp_seq, uint32_t result, uint32_t send_una, uint32_t recv_next) { uint16_t len, old_len; struct sctp_stream_reset_response_tsn *resp; struct sctp_chunkhdr *ch; ch = mtod(chk->data, struct sctp_chunkhdr *); old_len = len = SCTP_SIZE32(ntohs(ch->chunk_length)); /* get to new offset for the param. */ resp = (struct sctp_stream_reset_response_tsn *)((caddr_t)ch + len); /* now how long will this param be? */ len = sizeof(struct sctp_stream_reset_response_tsn); resp->ph.param_type = htons(SCTP_STR_RESET_RESPONSE); resp->ph.param_length = htons(len); resp->response_seq = htonl(resp_seq); resp->result = htonl(result); resp->senders_next_tsn = htonl(send_una); resp->receivers_next_tsn = htonl(recv_next); /* now fix the chunk length */ ch->chunk_length = htons(len + old_len); chk->book_size = len + old_len; chk->send_size = SCTP_SIZE32(chk->book_size); chk->book_size_scale = 0; SCTP_BUF_LEN(chk->data) = chk->send_size; return; } static void sctp_add_an_out_stream(struct sctp_tmit_chunk *chk, uint32_t seq, uint16_t adding) { uint16_t len, old_len; struct sctp_chunkhdr *ch; struct sctp_stream_reset_add_strm *addstr; ch = mtod(chk->data, struct sctp_chunkhdr *); old_len = len = SCTP_SIZE32(ntohs(ch->chunk_length)); /* get to new offset for the param. */ addstr = (struct sctp_stream_reset_add_strm *)((caddr_t)ch + len); /* now how long will this param be? */ len = sizeof(struct sctp_stream_reset_add_strm); /* Fill it out. */ addstr->ph.param_type = htons(SCTP_STR_RESET_ADD_OUT_STREAMS); addstr->ph.param_length = htons(len); addstr->request_seq = htonl(seq); addstr->number_of_streams = htons(adding); addstr->reserved = 0; /* now fix the chunk length */ ch->chunk_length = htons(len + old_len); chk->send_size = len + old_len; chk->book_size = SCTP_SIZE32(chk->send_size); chk->book_size_scale = 0; SCTP_BUF_LEN(chk->data) = SCTP_SIZE32(chk->send_size); return; } static void sctp_add_an_in_stream(struct sctp_tmit_chunk *chk, uint32_t seq, uint16_t adding) { uint16_t len, old_len; struct sctp_chunkhdr *ch; struct sctp_stream_reset_add_strm *addstr; ch = mtod(chk->data, struct sctp_chunkhdr *); old_len = len = SCTP_SIZE32(ntohs(ch->chunk_length)); /* get to new offset for the param. */ addstr = (struct sctp_stream_reset_add_strm *)((caddr_t)ch + len); /* now how long will this param be? */ len = sizeof(struct sctp_stream_reset_add_strm); /* Fill it out. */ addstr->ph.param_type = htons(SCTP_STR_RESET_ADD_IN_STREAMS); addstr->ph.param_length = htons(len); addstr->request_seq = htonl(seq); addstr->number_of_streams = htons(adding); addstr->reserved = 0; /* now fix the chunk length */ ch->chunk_length = htons(len + old_len); chk->send_size = len + old_len; chk->book_size = SCTP_SIZE32(chk->send_size); chk->book_size_scale = 0; SCTP_BUF_LEN(chk->data) = SCTP_SIZE32(chk->send_size); return; } int sctp_send_stream_reset_out_if_possible(struct sctp_tcb *stcb, int so_locked) { struct sctp_association *asoc; struct sctp_tmit_chunk *chk; struct sctp_chunkhdr *ch; uint32_t seq; asoc = &stcb->asoc; asoc->trigger_reset = 0; if (asoc->stream_reset_outstanding) { return (EALREADY); } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_STREAM_RESET; chk->rec.chunk_id.can_take_data = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->book_size = sizeof(struct sctp_chunkhdr); chk->send_size = SCTP_SIZE32(chk->book_size); chk->book_size_scale = 0; chk->data = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (chk->data == NULL) { sctp_free_a_chunk(stcb, chk, so_locked); SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); /* setup chunk parameters */ chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; if (stcb->asoc.alternate) { chk->whoTo = stcb->asoc.alternate; } else { chk->whoTo = stcb->asoc.primary_destination; } ch = mtod(chk->data, struct sctp_chunkhdr *); ch->chunk_type = SCTP_STREAM_RESET; ch->chunk_flags = 0; ch->chunk_length = htons(chk->book_size); atomic_add_int(&chk->whoTo->ref_count, 1); SCTP_BUF_LEN(chk->data) = chk->send_size; seq = stcb->asoc.str_reset_seq_out; if (sctp_add_stream_reset_out(stcb, chk, seq, (stcb->asoc.str_reset_seq_in - 1), (stcb->asoc.sending_seq - 1))) { seq++; asoc->stream_reset_outstanding++; } else { m_freem(chk->data); chk->data = NULL; sctp_free_a_chunk(stcb, chk, so_locked); return (ENOENT); } asoc->str_reset = chk; /* insert the chunk for sending */ TAILQ_INSERT_TAIL(&asoc->control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt++; if (stcb->asoc.send_sack) { sctp_send_sack(stcb, so_locked); } sctp_timer_start(SCTP_TIMER_TYPE_STRRESET, stcb->sctp_ep, stcb, chk->whoTo); return (0); } int sctp_send_str_reset_req(struct sctp_tcb *stcb, uint16_t number_entries, uint16_t * list, uint8_t send_in_req, uint8_t send_tsn_req, uint8_t add_stream, uint16_t adding_o, uint16_t adding_i, uint8_t peer_asked) { struct sctp_association *asoc; struct sctp_tmit_chunk *chk; struct sctp_chunkhdr *ch; int can_send_out_req = 0; uint32_t seq; asoc = &stcb->asoc; if (asoc->stream_reset_outstanding) { /*- * Already one pending, must get ACK back to clear the flag. */ SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EBUSY); return (EBUSY); } if ((send_in_req == 0) && (send_tsn_req == 0) && (add_stream == 0)) { /* nothing to do */ SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } if (send_tsn_req && send_in_req) { /* error, can't do that */ SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } else if (send_in_req) { can_send_out_req = 1; } if (number_entries > (MCLBYTES - SCTP_MIN_OVERHEAD - sizeof(struct sctp_chunkhdr) - sizeof(struct sctp_stream_reset_out_request)) / sizeof(uint16_t)) { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } sctp_alloc_a_chunk(stcb, chk); if (chk == NULL) { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } chk->copy_by_ref = 0; chk->rec.chunk_id.id = SCTP_STREAM_RESET; chk->rec.chunk_id.can_take_data = 0; chk->flags = 0; chk->asoc = &stcb->asoc; chk->book_size = sizeof(struct sctp_chunkhdr); chk->send_size = SCTP_SIZE32(chk->book_size); chk->book_size_scale = 0; chk->data = sctp_get_mbuf_for_msg(MCLBYTES, 0, M_NOWAIT, 1, MT_DATA); if (chk->data == NULL) { sctp_free_a_chunk(stcb, chk, SCTP_SO_LOCKED); SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ENOMEM); return (ENOMEM); } SCTP_BUF_RESV_UF(chk->data, SCTP_MIN_OVERHEAD); /* setup chunk parameters */ chk->sent = SCTP_DATAGRAM_UNSENT; chk->snd_count = 0; if (stcb->asoc.alternate) { chk->whoTo = stcb->asoc.alternate; } else { chk->whoTo = stcb->asoc.primary_destination; } atomic_add_int(&chk->whoTo->ref_count, 1); ch = mtod(chk->data, struct sctp_chunkhdr *); ch->chunk_type = SCTP_STREAM_RESET; ch->chunk_flags = 0; ch->chunk_length = htons(chk->book_size); SCTP_BUF_LEN(chk->data) = chk->send_size; seq = stcb->asoc.str_reset_seq_out; if (can_send_out_req) { int ret; ret = sctp_add_stream_reset_out(stcb, chk, seq, (stcb->asoc.str_reset_seq_in - 1), (stcb->asoc.sending_seq - 1)); if (ret) { seq++; asoc->stream_reset_outstanding++; } } if ((add_stream & 1) && ((stcb->asoc.strm_realoutsize - stcb->asoc.streamoutcnt) < adding_o)) { /* Need to allocate more */ struct sctp_stream_out *oldstream; struct sctp_stream_queue_pending *sp, *nsp; int i; #if defined(SCTP_DETAILED_STR_STATS) int j; #endif oldstream = stcb->asoc.strmout; /* get some more */ SCTP_MALLOC(stcb->asoc.strmout, struct sctp_stream_out *, (stcb->asoc.streamoutcnt + adding_o) * sizeof(struct sctp_stream_out), SCTP_M_STRMO); if (stcb->asoc.strmout == NULL) { uint8_t x; stcb->asoc.strmout = oldstream; /* Turn off the bit */ x = add_stream & 0xfe; add_stream = x; goto skip_stuff; } /* * Ok now we proceed with copying the old out stuff and * initializing the new stuff. */ SCTP_TCB_SEND_LOCK(stcb); stcb->asoc.ss_functions.sctp_ss_clear(stcb, &stcb->asoc, 0, 1); for (i = 0; i < stcb->asoc.streamoutcnt; i++) { TAILQ_INIT(&stcb->asoc.strmout[i].outqueue); stcb->asoc.strmout[i].chunks_on_queues = oldstream[i].chunks_on_queues; stcb->asoc.strmout[i].next_sequence_send = oldstream[i].next_sequence_send; stcb->asoc.strmout[i].last_msg_incomplete = oldstream[i].last_msg_incomplete; stcb->asoc.strmout[i].stream_no = i; stcb->asoc.strmout[i].state = oldstream[i].state; stcb->asoc.ss_functions.sctp_ss_init_stream(&stcb->asoc.strmout[i], &oldstream[i]); /* now anything on those queues? */ TAILQ_FOREACH_SAFE(sp, &oldstream[i].outqueue, next, nsp) { TAILQ_REMOVE(&oldstream[i].outqueue, sp, next); TAILQ_INSERT_TAIL(&stcb->asoc.strmout[i].outqueue, sp, next); } /* Now move assoc pointers too */ if (stcb->asoc.last_out_stream == &oldstream[i]) { stcb->asoc.last_out_stream = &stcb->asoc.strmout[i]; } if (stcb->asoc.locked_on_sending == &oldstream[i]) { stcb->asoc.locked_on_sending = &stcb->asoc.strmout[i]; } } /* now the new streams */ stcb->asoc.ss_functions.sctp_ss_init(stcb, &stcb->asoc, 1); for (i = stcb->asoc.streamoutcnt; i < (stcb->asoc.streamoutcnt + adding_o); i++) { TAILQ_INIT(&stcb->asoc.strmout[i].outqueue); stcb->asoc.strmout[i].chunks_on_queues = 0; #if defined(SCTP_DETAILED_STR_STATS) for (j = 0; j < SCTP_PR_SCTP_MAX + 1; j++) { stcb->asoc.strmout[i].abandoned_sent[j] = 0; stcb->asoc.strmout[i].abandoned_unsent[j] = 0; } #else stcb->asoc.strmout[i].abandoned_sent[0] = 0; stcb->asoc.strmout[i].abandoned_unsent[0] = 0; #endif stcb->asoc.strmout[i].next_sequence_send = 0x0; stcb->asoc.strmout[i].stream_no = i; stcb->asoc.strmout[i].last_msg_incomplete = 0; stcb->asoc.ss_functions.sctp_ss_init_stream(&stcb->asoc.strmout[i], NULL); stcb->asoc.strmout[i].state = SCTP_STREAM_CLOSED; } stcb->asoc.strm_realoutsize = stcb->asoc.streamoutcnt + adding_o; SCTP_FREE(oldstream, SCTP_M_STRMO); SCTP_TCB_SEND_UNLOCK(stcb); } skip_stuff: if ((add_stream & 1) && (adding_o > 0)) { asoc->strm_pending_add_size = adding_o; asoc->peer_req_out = peer_asked; sctp_add_an_out_stream(chk, seq, adding_o); seq++; asoc->stream_reset_outstanding++; } if ((add_stream & 2) && (adding_i > 0)) { sctp_add_an_in_stream(chk, seq, adding_i); seq++; asoc->stream_reset_outstanding++; } if (send_in_req) { sctp_add_stream_reset_in(chk, number_entries, list, seq); seq++; asoc->stream_reset_outstanding++; } if (send_tsn_req) { sctp_add_stream_reset_tsn(chk, seq); asoc->stream_reset_outstanding++; } asoc->str_reset = chk; /* insert the chunk for sending */ TAILQ_INSERT_TAIL(&asoc->control_send_queue, chk, sctp_next); asoc->ctrl_queue_cnt++; if (stcb->asoc.send_sack) { sctp_send_sack(stcb, SCTP_SO_LOCKED); } sctp_timer_start(SCTP_TIMER_TYPE_STRRESET, stcb->sctp_ep, stcb, chk->whoTo); return (0); } void sctp_send_abort(struct mbuf *m, int iphlen, struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, uint32_t vtag, struct mbuf *cause, uint8_t mflowtype, uint32_t mflowid, uint16_t fibnum, uint32_t vrf_id, uint16_t port) { /* Don't respond to an ABORT with an ABORT. */ if (sctp_is_there_an_abort_here(m, iphlen, &vtag)) { if (cause) sctp_m_freem(cause); return; } sctp_send_resp_msg(src, dst, sh, vtag, SCTP_ABORT_ASSOCIATION, cause, mflowtype, mflowid, fibnum, vrf_id, port); return; } void sctp_send_operr_to(struct sockaddr *src, struct sockaddr *dst, struct sctphdr *sh, uint32_t vtag, struct mbuf *cause, uint8_t mflowtype, uint32_t mflowid, uint16_t fibnum, uint32_t vrf_id, uint16_t port) { sctp_send_resp_msg(src, dst, sh, vtag, SCTP_OPERATION_ERROR, cause, mflowtype, mflowid, fibnum, vrf_id, port); return; } static struct mbuf * sctp_copy_resume(struct uio *uio, int max_send_len, int user_marks_eor, int *error, uint32_t * sndout, struct mbuf **new_tail) { struct mbuf *m; m = m_uiotombuf(uio, M_WAITOK, max_send_len, 0, (M_PKTHDR | (user_marks_eor ? M_EOR : 0))); if (m == NULL) { SCTP_LTRACE_ERR_RET(NULL, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, ENOBUFS); *error = ENOBUFS; } else { *sndout = m_length(m, NULL); *new_tail = m_last(m); } return (m); } static int sctp_copy_one(struct sctp_stream_queue_pending *sp, struct uio *uio, int resv_upfront) { sp->data = m_uiotombuf(uio, M_WAITOK, sp->length, resv_upfront, 0); if (sp->data == NULL) { SCTP_LTRACE_ERR_RET(NULL, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, ENOBUFS); return (ENOBUFS); } sp->tail_mbuf = m_last(sp->data); return (0); } static struct sctp_stream_queue_pending * sctp_copy_it_in(struct sctp_tcb *stcb, struct sctp_association *asoc, struct sctp_sndrcvinfo *srcv, struct uio *uio, struct sctp_nets *net, int max_send_len, int user_marks_eor, int *error) { /*- * This routine must be very careful in its work. Protocol * processing is up and running so care must be taken to spl...() * when you need to do something that may effect the stcb/asoc. The * sb is locked however. When data is copied the protocol processing * should be enabled since this is a slower operation... */ struct sctp_stream_queue_pending *sp = NULL; int resv_in_first; *error = 0; /* Now can we send this? */ if ((SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_SENT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_RECEIVED) || (asoc->state & SCTP_STATE_SHUTDOWN_PENDING)) { /* got data while shutting down */ SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ECONNRESET); *error = ECONNRESET; goto out_now; } sctp_alloc_a_strmoq(stcb, sp); if (sp == NULL) { SCTP_LTRACE_ERR_RET(NULL, stcb, net, SCTP_FROM_SCTP_OUTPUT, ENOMEM); *error = ENOMEM; goto out_now; } sp->act_flags = 0; sp->sender_all_done = 0; sp->sinfo_flags = srcv->sinfo_flags; sp->timetolive = srcv->sinfo_timetolive; sp->ppid = srcv->sinfo_ppid; sp->context = srcv->sinfo_context; (void)SCTP_GETTIME_TIMEVAL(&sp->ts); sp->stream = srcv->sinfo_stream; sp->length = min(uio->uio_resid, max_send_len); if ((sp->length == (uint32_t) uio->uio_resid) && ((user_marks_eor == 0) || (srcv->sinfo_flags & SCTP_EOF) || (user_marks_eor && (srcv->sinfo_flags & SCTP_EOR)))) { sp->msg_is_complete = 1; } else { sp->msg_is_complete = 0; } sp->sender_all_done = 0; sp->some_taken = 0; sp->put_last_out = 0; resv_in_first = sizeof(struct sctp_data_chunk); sp->data = sp->tail_mbuf = NULL; if (sp->length == 0) { *error = 0; goto skip_copy; } if (srcv->sinfo_keynumber_valid) { sp->auth_keyid = srcv->sinfo_keynumber; } else { sp->auth_keyid = stcb->asoc.authinfo.active_keyid; } if (sctp_auth_is_required_chunk(SCTP_DATA, stcb->asoc.peer_auth_chunks)) { sctp_auth_key_acquire(stcb, sp->auth_keyid); sp->holds_key_ref = 1; } *error = sctp_copy_one(sp, uio, resv_in_first); skip_copy: if (*error) { sctp_free_a_strmoq(stcb, sp, SCTP_SO_LOCKED); sp = NULL; } else { if (sp->sinfo_flags & SCTP_ADDR_OVER) { sp->net = net; atomic_add_int(&sp->net->ref_count, 1); } else { sp->net = NULL; } sctp_set_prsctp_policy(sp); } out_now: return (sp); } int sctp_sosend(struct socket *so, struct sockaddr *addr, struct uio *uio, struct mbuf *top, struct mbuf *control, int flags, struct thread *p ) { int error, use_sndinfo = 0; struct sctp_sndrcvinfo sndrcvninfo; struct sockaddr *addr_to_use; #if defined(INET) && defined(INET6) struct sockaddr_in sin; #endif if (control) { /* process cmsg snd/rcv info (maybe a assoc-id) */ if (sctp_find_cmsg(SCTP_SNDRCV, (void *)&sndrcvninfo, control, sizeof(sndrcvninfo))) { /* got one */ use_sndinfo = 1; } } addr_to_use = addr; #if defined(INET) && defined(INET6) if ((addr) && (addr->sa_family == AF_INET6)) { struct sockaddr_in6 *sin6; sin6 = (struct sockaddr_in6 *)addr; if (IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) { in6_sin6_2_sin(&sin, sin6); addr_to_use = (struct sockaddr *)&sin; } } #endif error = sctp_lower_sosend(so, addr_to_use, uio, top, control, flags, use_sndinfo ? &sndrcvninfo : NULL ,p ); return (error); } int sctp_lower_sosend(struct socket *so, struct sockaddr *addr, struct uio *uio, struct mbuf *i_pak, struct mbuf *control, int flags, struct sctp_sndrcvinfo *srcv , struct thread *p ) { unsigned int sndlen = 0, max_len; int error, len; struct mbuf *top = NULL; int queue_only = 0, queue_only_for_init = 0; int free_cnt_applied = 0; int un_sent; int now_filled = 0; unsigned int inqueue_bytes = 0; struct sctp_block_entry be; struct sctp_inpcb *inp; struct sctp_tcb *stcb = NULL; struct timeval now; struct sctp_nets *net; struct sctp_association *asoc; struct sctp_inpcb *t_inp; int user_marks_eor; int create_lock_applied = 0; int nagle_applies = 0; int some_on_control = 0; int got_all_of_the_send = 0; int hold_tcblock = 0; int non_blocking = 0; uint32_t local_add_more, local_soresv = 0; uint16_t port; uint16_t sinfo_flags; sctp_assoc_t sinfo_assoc_id; error = 0; net = NULL; stcb = NULL; asoc = NULL; t_inp = inp = (struct sctp_inpcb *)so->so_pcb; if (inp == NULL) { SCTP_LTRACE_ERR_RET(NULL, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; if (i_pak) { SCTP_RELEASE_PKT(i_pak); } return (error); } if ((uio == NULL) && (i_pak == NULL)) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } user_marks_eor = sctp_is_feature_on(inp, SCTP_PCB_FLAGS_EXPLICIT_EOR); atomic_add_int(&inp->total_sends, 1); if (uio) { if (uio->uio_resid < 0) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); return (EINVAL); } sndlen = uio->uio_resid; } else { top = SCTP_HEADER_TO_CHAIN(i_pak); sndlen = SCTP_HEADER_LEN(i_pak); } SCTPDBG(SCTP_DEBUG_OUTPUT1, "Send called addr:%p send length %d\n", (void *)addr, sndlen); if ((inp->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) && (inp->sctp_socket->so_qlimit)) { /* The listener can NOT send */ SCTP_LTRACE_ERR_RET(NULL, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, ENOTCONN); error = ENOTCONN; goto out_unlocked; } /** * Pre-screen address, if one is given the sin-len * must be set correctly! */ if (addr) { union sctp_sockstore *raddr = (union sctp_sockstore *)addr; switch (raddr->sa.sa_family) { #ifdef INET case AF_INET: if (raddr->sin.sin_len != sizeof(struct sockaddr_in)) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } port = raddr->sin.sin_port; break; #endif #ifdef INET6 case AF_INET6: if (raddr->sin6.sin6_len != sizeof(struct sockaddr_in6)) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } port = raddr->sin6.sin6_port; break; #endif default: SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EAFNOSUPPORT); error = EAFNOSUPPORT; goto out_unlocked; } } else port = 0; if (srcv) { sinfo_flags = srcv->sinfo_flags; sinfo_assoc_id = srcv->sinfo_assoc_id; if (INVALID_SINFO_FLAG(sinfo_flags) || PR_SCTP_INVALID_POLICY(sinfo_flags)) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } if (srcv->sinfo_flags) SCTP_STAT_INCR(sctps_sends_with_flags); } else { sinfo_flags = inp->def_send.sinfo_flags; sinfo_assoc_id = inp->def_send.sinfo_assoc_id; } if (sinfo_flags & SCTP_SENDALL) { /* its a sendall */ error = sctp_sendall(inp, uio, top, srcv); top = NULL; goto out_unlocked; } if ((sinfo_flags & SCTP_ADDR_OVER) && (addr == NULL)) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } /* now we must find the assoc */ if ((inp->sctp_flags & SCTP_PCB_FLAGS_CONNECTED) || (inp->sctp_flags & SCTP_PCB_FLAGS_IN_TCPPOOL)) { SCTP_INP_RLOCK(inp); stcb = LIST_FIRST(&inp->sctp_asoc_list); if (stcb) { SCTP_TCB_LOCK(stcb); hold_tcblock = 1; } SCTP_INP_RUNLOCK(inp); } else if (sinfo_assoc_id) { stcb = sctp_findassociation_ep_asocid(inp, sinfo_assoc_id, 0); } else if (addr) { /*- * Since we did not use findep we must * increment it, and if we don't find a tcb * decrement it. */ SCTP_INP_WLOCK(inp); SCTP_INP_INCR_REF(inp); SCTP_INP_WUNLOCK(inp); stcb = sctp_findassociation_ep_addr(&t_inp, addr, &net, NULL, NULL); if (stcb == NULL) { SCTP_INP_WLOCK(inp); SCTP_INP_DECR_REF(inp); SCTP_INP_WUNLOCK(inp); } else { hold_tcblock = 1; } } if ((stcb == NULL) && (addr)) { /* Possible implicit send? */ SCTP_ASOC_CREATE_LOCK(inp); create_lock_applied = 1; if ((inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_GONE) || (inp->sctp_flags & SCTP_PCB_FLAGS_SOCKET_ALLGONE)) { /* Should I really unlock ? */ SCTP_LTRACE_ERR_RET(NULL, NULL, NULL, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } if (((inp->sctp_flags & SCTP_PCB_FLAGS_BOUND_V6) == 0) && (addr->sa_family == AF_INET6)) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } SCTP_INP_WLOCK(inp); SCTP_INP_INCR_REF(inp); SCTP_INP_WUNLOCK(inp); /* With the lock applied look again */ stcb = sctp_findassociation_ep_addr(&t_inp, addr, &net, NULL, NULL); if ((stcb == NULL) && (control != NULL) && (port > 0)) { stcb = sctp_findassociation_cmsgs(&t_inp, port, control, &net, &error); } if (stcb == NULL) { SCTP_INP_WLOCK(inp); SCTP_INP_DECR_REF(inp); SCTP_INP_WUNLOCK(inp); } else { hold_tcblock = 1; } if (error) { goto out_unlocked; } if (t_inp != inp) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, ENOTCONN); error = ENOTCONN; goto out_unlocked; } } if (stcb == NULL) { if (addr == NULL) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, ENOENT); error = ENOENT; goto out_unlocked; } else { /* We must go ahead and start the INIT process */ uint32_t vrf_id; if ((sinfo_flags & SCTP_ABORT) || ((sinfo_flags & SCTP_EOF) && (sndlen == 0))) { /*- * User asks to abort a non-existant assoc, * or EOF a non-existant assoc with no data */ SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, ENOENT); error = ENOENT; goto out_unlocked; } /* get an asoc/stcb struct */ vrf_id = inp->def_vrf_id; #ifdef INVARIANTS if (create_lock_applied == 0) { panic("Error, should hold create lock and I don't?"); } #endif stcb = sctp_aloc_assoc(inp, addr, &error, 0, vrf_id, inp->sctp_ep.pre_open_stream_count, p ); if (stcb == NULL) { /* Error is setup for us in the call */ goto out_unlocked; } if (stcb->sctp_ep->sctp_flags & SCTP_PCB_FLAGS_TCPTYPE) { stcb->sctp_ep->sctp_flags |= SCTP_PCB_FLAGS_CONNECTED; /* * Set the connected flag so we can queue * data */ soisconnecting(so); } hold_tcblock = 1; if (create_lock_applied) { SCTP_ASOC_CREATE_UNLOCK(inp); create_lock_applied = 0; } else { SCTP_PRINTF("Huh-3? create lock should have been on??\n"); } /* * Turn on queue only flag to prevent data from * being sent */ queue_only = 1; asoc = &stcb->asoc; SCTP_SET_STATE(asoc, SCTP_STATE_COOKIE_WAIT); (void)SCTP_GETTIME_TIMEVAL(&asoc->time_entered); /* initialize authentication params for the assoc */ sctp_initialize_auth_params(inp, stcb); if (control) { if (sctp_process_cmsgs_for_init(stcb, control, &error)) { sctp_free_assoc(inp, stcb, SCTP_PCBFREE_FORCE, SCTP_FROM_SCTP_OUTPUT + SCTP_LOC_5); hold_tcblock = 0; stcb = NULL; goto out_unlocked; } } /* out with the INIT */ queue_only_for_init = 1; /*- * we may want to dig in after this call and adjust the MTU * value. It defaulted to 1500 (constant) but the ro * structure may now have an update and thus we may need to * change it BEFORE we append the message. */ } } else asoc = &stcb->asoc; if (srcv == NULL) srcv = (struct sctp_sndrcvinfo *)&asoc->def_send; if (srcv->sinfo_flags & SCTP_ADDR_OVER) { if (addr) net = sctp_findnet(stcb, addr); else net = NULL; if ((net == NULL) || ((port != 0) && (port != stcb->rport))) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } } else { if (stcb->asoc.alternate) { net = stcb->asoc.alternate; } else { net = stcb->asoc.primary_destination; } } atomic_add_int(&stcb->total_sends, 1); /* Keep the stcb from being freed under our feet */ atomic_add_int(&asoc->refcnt, 1); free_cnt_applied = 1; if (sctp_is_feature_on(inp, SCTP_PCB_FLAGS_NO_FRAGMENT)) { if (sndlen > asoc->smallest_mtu) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EMSGSIZE); error = EMSGSIZE; goto out_unlocked; } } if (SCTP_SO_IS_NBIO(so) || (flags & MSG_NBIO) ) { non_blocking = 1; } /* would we block? */ if (non_blocking) { if (hold_tcblock == 0) { SCTP_TCB_LOCK(stcb); hold_tcblock = 1; } inqueue_bytes = stcb->asoc.total_output_queue_size - (stcb->asoc.chunks_on_out_queue * sizeof(struct sctp_data_chunk)); if ((SCTP_SB_LIMIT_SND(so) < (sndlen + inqueue_bytes + stcb->asoc.sb_send_resv)) || (stcb->asoc.chunks_on_out_queue >= SCTP_BASE_SYSCTL(sctp_max_chunks_on_queue))) { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EWOULDBLOCK); if (sndlen > SCTP_SB_LIMIT_SND(so)) error = EMSGSIZE; else error = EWOULDBLOCK; goto out_unlocked; } stcb->asoc.sb_send_resv += sndlen; SCTP_TCB_UNLOCK(stcb); hold_tcblock = 0; } else { atomic_add_int(&stcb->asoc.sb_send_resv, sndlen); } local_soresv = sndlen; if (stcb->asoc.state & SCTP_STATE_ABOUT_TO_BE_FREED) { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ECONNRESET); error = ECONNRESET; goto out_unlocked; } if (create_lock_applied) { SCTP_ASOC_CREATE_UNLOCK(inp); create_lock_applied = 0; } /* Is the stream no. valid? */ if (srcv->sinfo_stream >= asoc->streamoutcnt) { /* Invalid stream number */ SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } if ((asoc->strmout[srcv->sinfo_stream].state != SCTP_STREAM_OPEN) && (asoc->strmout[srcv->sinfo_stream].state != SCTP_STREAM_OPENING)) { /* * Can't queue any data while stream reset is underway. */ if (asoc->strmout[srcv->sinfo_stream].state > SCTP_STREAM_OPEN) { error = EAGAIN; } else { error = EINVAL; } SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, error); goto out_unlocked; } if ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED)) { queue_only = 1; } /* we are now done with all control */ if (control) { sctp_m_freem(control); control = NULL; } if ((SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_SENT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_RECEIVED) || (SCTP_GET_STATE(asoc) == SCTP_STATE_SHUTDOWN_ACK_SENT) || (asoc->state & SCTP_STATE_SHUTDOWN_PENDING)) { if (srcv->sinfo_flags & SCTP_ABORT) { ; } else { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ECONNRESET); error = ECONNRESET; goto out_unlocked; } } /* Ok, we will attempt a msgsnd :> */ if (p) { p->td_ru.ru_msgsnd++; } /* Are we aborting? */ if (srcv->sinfo_flags & SCTP_ABORT) { struct mbuf *mm; int tot_demand, tot_out = 0, max_out; SCTP_STAT_INCR(sctps_sends_with_abort); if ((SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_WAIT) || (SCTP_GET_STATE(asoc) == SCTP_STATE_COOKIE_ECHOED)) { /* It has to be up before we abort */ /* how big is the user initiated abort? */ SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out; } if (hold_tcblock) { SCTP_TCB_UNLOCK(stcb); hold_tcblock = 0; } if (top) { struct mbuf *cntm = NULL; mm = sctp_get_mbuf_for_msg(sizeof(struct sctp_paramhdr), 0, M_WAITOK, 1, MT_DATA); if (sndlen != 0) { for (cntm = top; cntm; cntm = SCTP_BUF_NEXT(cntm)) { tot_out += SCTP_BUF_LEN(cntm); } } } else { /* Must fit in a MTU */ tot_out = sndlen; tot_demand = (tot_out + sizeof(struct sctp_paramhdr)); if (tot_demand > SCTP_DEFAULT_ADD_MORE) { /* To big */ SCTP_LTRACE_ERR_RET(NULL, stcb, net, SCTP_FROM_SCTP_OUTPUT, EMSGSIZE); error = EMSGSIZE; goto out; } mm = sctp_get_mbuf_for_msg(tot_demand, 0, M_WAITOK, 1, MT_DATA); } if (mm == NULL) { SCTP_LTRACE_ERR_RET(NULL, stcb, net, SCTP_FROM_SCTP_OUTPUT, ENOMEM); error = ENOMEM; goto out; } max_out = asoc->smallest_mtu - sizeof(struct sctp_paramhdr); max_out -= sizeof(struct sctp_abort_msg); if (tot_out > max_out) { tot_out = max_out; } if (mm) { struct sctp_paramhdr *ph; /* now move forward the data pointer */ ph = mtod(mm, struct sctp_paramhdr *); ph->param_type = htons(SCTP_CAUSE_USER_INITIATED_ABT); ph->param_length = htons(sizeof(struct sctp_paramhdr) + tot_out); ph++; SCTP_BUF_LEN(mm) = tot_out + sizeof(struct sctp_paramhdr); if (top == NULL) { error = uiomove((caddr_t)ph, (int)tot_out, uio); if (error) { /*- * Here if we can't get his data we * still abort we just don't get to * send the users note :-0 */ sctp_m_freem(mm); mm = NULL; } } else { if (sndlen != 0) { SCTP_BUF_NEXT(mm) = top; } } } if (hold_tcblock == 0) { SCTP_TCB_LOCK(stcb); } atomic_add_int(&stcb->asoc.refcnt, -1); free_cnt_applied = 0; /* release this lock, otherwise we hang on ourselves */ sctp_abort_an_association(stcb->sctp_ep, stcb, mm, SCTP_SO_LOCKED); /* now relock the stcb so everything is sane */ hold_tcblock = 0; stcb = NULL; /* * In this case top is already chained to mm avoid double * free, since we free it below if top != NULL and driver * would free it after sending the packet out */ if (sndlen != 0) { top = NULL; } goto out_unlocked; } /* Calculate the maximum we can send */ inqueue_bytes = stcb->asoc.total_output_queue_size - (stcb->asoc.chunks_on_out_queue * sizeof(struct sctp_data_chunk)); if (SCTP_SB_LIMIT_SND(so) > inqueue_bytes) { if (non_blocking) { /* we already checked for non-blocking above. */ max_len = sndlen; } else { max_len = SCTP_SB_LIMIT_SND(so) - inqueue_bytes; } } else { max_len = 0; } if (hold_tcblock) { SCTP_TCB_UNLOCK(stcb); hold_tcblock = 0; } if (asoc->strmout == NULL) { /* huh? software error */ SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EFAULT); error = EFAULT; goto out_unlocked; } /* Unless E_EOR mode is on, we must make a send FIT in one call. */ if ((user_marks_eor == 0) && (sndlen > SCTP_SB_LIMIT_SND(stcb->sctp_socket))) { /* It will NEVER fit */ SCTP_LTRACE_ERR_RET(NULL, stcb, net, SCTP_FROM_SCTP_OUTPUT, EMSGSIZE); error = EMSGSIZE; goto out_unlocked; } if ((uio == NULL) && user_marks_eor) { /*- * We do not support eeor mode for * sending with mbuf chains (like sendfile). */ SCTP_LTRACE_ERR_RET(NULL, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out_unlocked; } if (user_marks_eor) { local_add_more = min(SCTP_SB_LIMIT_SND(so), SCTP_BASE_SYSCTL(sctp_add_more_threshold)); } else { /*- * For non-eeor the whole message must fit in * the socket send buffer. */ local_add_more = sndlen; } len = 0; if (non_blocking) { goto skip_preblock; } if (((max_len <= local_add_more) && (SCTP_SB_LIMIT_SND(so) >= local_add_more)) || (max_len == 0) || ((stcb->asoc.chunks_on_out_queue + stcb->asoc.stream_queue_cnt) >= SCTP_BASE_SYSCTL(sctp_max_chunks_on_queue))) { /* No room right now ! */ SOCKBUF_LOCK(&so->so_snd); inqueue_bytes = stcb->asoc.total_output_queue_size - (stcb->asoc.chunks_on_out_queue * sizeof(struct sctp_data_chunk)); while ((SCTP_SB_LIMIT_SND(so) < (inqueue_bytes + local_add_more)) || ((stcb->asoc.stream_queue_cnt + stcb->asoc.chunks_on_out_queue) >= SCTP_BASE_SYSCTL(sctp_max_chunks_on_queue))) { SCTPDBG(SCTP_DEBUG_OUTPUT1, "pre_block limit:%u <(inq:%d + %d) || (%d+%d > %d)\n", (unsigned int)SCTP_SB_LIMIT_SND(so), inqueue_bytes, local_add_more, stcb->asoc.stream_queue_cnt, stcb->asoc.chunks_on_out_queue, SCTP_BASE_SYSCTL(sctp_max_chunks_on_queue)); if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_BLK_LOGGING_ENABLE) { sctp_log_block(SCTP_BLOCK_LOG_INTO_BLKA, asoc, sndlen); } be.error = 0; stcb->block_entry = &be; error = sbwait(&so->so_snd); stcb->block_entry = NULL; if (error || so->so_error || be.error) { if (error == 0) { if (so->so_error) error = so->so_error; if (be.error) { error = be.error; } } SOCKBUF_UNLOCK(&so->so_snd); goto out_unlocked; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_BLK_LOGGING_ENABLE) { sctp_log_block(SCTP_BLOCK_LOG_OUTOF_BLK, asoc, stcb->asoc.total_output_queue_size); } if (stcb->asoc.state & SCTP_STATE_ABOUT_TO_BE_FREED) { goto out_unlocked; } inqueue_bytes = stcb->asoc.total_output_queue_size - (stcb->asoc.chunks_on_out_queue * sizeof(struct sctp_data_chunk)); } if (SCTP_SB_LIMIT_SND(so) > inqueue_bytes) { max_len = SCTP_SB_LIMIT_SND(so) - inqueue_bytes; } else { max_len = 0; } SOCKBUF_UNLOCK(&so->so_snd); } skip_preblock: if (stcb->asoc.state & SCTP_STATE_ABOUT_TO_BE_FREED) { goto out_unlocked; } /* * sndlen covers for mbuf case uio_resid covers for the non-mbuf * case NOTE: uio will be null when top/mbuf is passed */ if (sndlen == 0) { if (srcv->sinfo_flags & SCTP_EOF) { got_all_of_the_send = 1; goto dataless_eof; } else { SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out; } } if (top == NULL) { struct sctp_stream_queue_pending *sp; struct sctp_stream_out *strm; uint32_t sndout; SCTP_TCB_SEND_LOCK(stcb); if ((asoc->stream_locked) && (asoc->stream_locked_on != srcv->sinfo_stream)) { SCTP_TCB_SEND_UNLOCK(stcb); SCTP_LTRACE_ERR_RET(inp, stcb, net, SCTP_FROM_SCTP_OUTPUT, EINVAL); error = EINVAL; goto out; } SCTP_TCB_SEND_UNLOCK(stcb); strm = &stcb->asoc.strmout[srcv->sinfo_stream]; if (strm->last_msg_incomplete == 0) { do_a_copy_in: sp = sctp_copy_it_in(stcb, asoc, srcv, uio, net, max_len, user_marks_eor, &error); if ((sp == NULL) || (error)) { goto out; } SCTP_TCB_SEND_LOCK(stcb); if (sp->msg_is_complete) { strm->last_msg_incomplete = 0; asoc->stream_locked = 0; } else { /* * Just got locked to this guy in case of an * interrupt. */ strm->last_msg_incomplete = 1; asoc->stream_locked = 1; asoc->stream_locked_on = srcv->sinfo_stream; sp->sender_all_done = 0; } sctp_snd_sb_alloc(stcb, sp->length); atomic_add_int(&asoc->stream_queue_cnt, 1); if (srcv->sinfo_flags & SCTP_UNORDERED) { SCTP_STAT_INCR(sctps_sends_with_unord); } TAILQ_INSERT_TAIL(&strm->outqueue, sp, next); stcb->asoc.ss_functions.sctp_ss_add_to_stream(stcb, asoc, strm, sp, 1); SCTP_TCB_SEND_UNLOCK(stcb); } else { SCTP_TCB_SEND_LOCK(stcb); sp = TAILQ_LAST(&strm->outqueue, sctp_streamhead); SCTP_TCB_SEND_UNLOCK(stcb); if (sp == NULL) { /* ???? Huh ??? last msg is gone */ #ifdef INVARIANTS panic("Warning: Last msg marked incomplete, yet nothing left?"); #else SCTP_PRINTF("Warning: Last msg marked incomplete, yet nothing left?\n"); strm->last_msg_incomplete = 0; #endif goto do_a_copy_in; } } while (uio->uio_resid > 0) { /* How much room do we have? */ struct mbuf *new_tail, *mm; if (SCTP_SB_LIMIT_SND(so) > stcb->asoc.total_output_queue_size) max_len = SCTP_SB_LIMIT_SND(so) - stcb->asoc.total_output_queue_size; else max_len = 0; if ((max_len > SCTP_BASE_SYSCTL(sctp_add_more_threshold)) || (max_len && (SCTP_SB_LIMIT_SND(so) < SCTP_BASE_SYSCTL(sctp_add_more_threshold))) || (uio->uio_resid && (uio->uio_resid <= (int)max_len))) { sndout = 0; new_tail = NULL; if (hold_tcblock) { SCTP_TCB_UNLOCK(stcb); hold_tcblock = 0; } mm = sctp_copy_resume(uio, max_len, user_marks_eor, &error, &sndout, &new_tail); if ((mm == NULL) || error) { if (mm) { sctp_m_freem(mm); } goto out; } /* Update the mbuf and count */ SCTP_TCB_SEND_LOCK(stcb); if (stcb->asoc.state & SCTP_STATE_ABOUT_TO_BE_FREED) { /* * we need to get out. Peer probably * aborted. */ sctp_m_freem(mm); if (stcb->asoc.state & SCTP_PCB_FLAGS_WAS_ABORTED) { SCTP_LTRACE_ERR_RET(NULL, stcb, NULL, SCTP_FROM_SCTP_OUTPUT, ECONNRESET); error = ECONNRESET; } SCTP_TCB_SEND_UNLOCK(stcb); goto out; } if (sp->tail_mbuf) { /* tack it to the end */ SCTP_BUF_NEXT(sp->tail_mbuf) = mm; sp->tail_mbuf = new_tail; } else { /* A stolen mbuf */ sp->data = mm; sp->tail_mbuf = new_tail; } sctp_snd_sb_alloc(stcb, sndout); atomic_add_int(&sp->length, sndout); len += sndout; /* Did we reach EOR? */ if ((uio->uio_resid == 0) && ((user_marks_eor == 0) || (srcv->sinfo_flags & SCTP_EOF) || (user_marks_eor && (srcv->sinfo_flags & SCTP_EOR)))) { sp->msg_is_complete = 1; } else { sp->msg_is_complete = 0; } SCTP_TCB_SEND_UNLOCK(stcb); } if (uio->uio_resid == 0) { /* got it all? */ continue; } /* PR-SCTP? */ if ((asoc->prsctp_supported) && (asoc->sent_queue_cnt_removeable > 0)) { /* * This is ugly but we must assure locking * order */ if (hold_tcblock == 0) { SCTP_TCB_LOCK(stcb); hold_tcblock = 1; } sctp_prune_prsctp(stcb, asoc, srcv, sndlen); inqueue_bytes = stcb->asoc.total_output_queue_size - (stcb->asoc.chunks_on_out_queue * sizeof(struct sctp_data_chunk)); if (SCTP_SB_LIMIT_SND(so) > stcb->asoc.total_output_queue_size) max_len = SCTP_SB_LIMIT_SND(so) - inqueue_bytes; else max_len = 0; if (max_len > 0) { continue; } SCTP_TCB_UNLOCK(stcb); hold_tcblock = 0; } /* wait for space now */ if (non_blocking) { /* Non-blocking io in place out */ goto skip_out_eof; } /* What about the INIT, send it maybe */ if (queue_only_for_init) { if (hold_tcblock == 0) { SCTP_TCB_LOCK(stcb); hold_tcblock = 1; } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_OPEN) { /* a collision took us forward? */ queue_only = 0; } else { sctp_send_initiate(inp, stcb, SCTP_SO_LOCKED); SCTP_SET_STATE(asoc, SCTP_STATE_COOKIE_WAIT); queue_only = 1; } } if ((net->flight_size > net->cwnd) && (asoc->sctp_cmt_on_off == 0)) { SCTP_STAT_INCR(sctps_send_cwnd_avoid); queue_only = 1; } else if (asoc->ifp_had_enobuf) { SCTP_STAT_INCR(sctps_ifnomemqueued); if (net->flight_size > (2 * net->mtu)) { queue_only = 1; } asoc->ifp_had_enobuf = 0; } un_sent = ((stcb->asoc.total_output_queue_size - stcb->asoc.total_flight) + (stcb->asoc.stream_queue_cnt * sizeof(struct sctp_data_chunk))); if ((sctp_is_feature_off(inp, SCTP_PCB_FLAGS_NODELAY)) && (stcb->asoc.total_flight > 0) && (stcb->asoc.stream_queue_cnt < SCTP_MAX_DATA_BUNDLING) && (un_sent < (int)(stcb->asoc.smallest_mtu - SCTP_MIN_OVERHEAD))) { /*- * Ok, Nagle is set on and we have data outstanding. * Don't send anything and let SACKs drive out the * data unless we have a "full" segment to send. */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_NAGLE_LOGGING_ENABLE) { sctp_log_nagle_event(stcb, SCTP_NAGLE_APPLIED); } SCTP_STAT_INCR(sctps_naglequeued); nagle_applies = 1; } else { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_NAGLE_LOGGING_ENABLE) { if (sctp_is_feature_off(inp, SCTP_PCB_FLAGS_NODELAY)) sctp_log_nagle_event(stcb, SCTP_NAGLE_SKIPPED); } SCTP_STAT_INCR(sctps_naglesent); nagle_applies = 0; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_BLK_LOGGING_ENABLE) { sctp_misc_ints(SCTP_CWNDLOG_PRESEND, queue_only_for_init, queue_only, nagle_applies, un_sent); sctp_misc_ints(SCTP_CWNDLOG_PRESEND, stcb->asoc.total_output_queue_size, stcb->asoc.total_flight, stcb->asoc.chunks_on_out_queue, stcb->asoc.total_flight_count); } if (queue_only_for_init) queue_only_for_init = 0; if ((queue_only == 0) && (nagle_applies == 0)) { /*- * need to start chunk output * before blocking.. note that if * a lock is already applied, then * the input via the net is happening * and I don't need to start output :-D */ if (hold_tcblock == 0) { if (SCTP_TCB_TRYLOCK(stcb)) { hold_tcblock = 1; sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_USR_SEND, SCTP_SO_LOCKED); } } else { sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_USR_SEND, SCTP_SO_LOCKED); } if (hold_tcblock == 1) { SCTP_TCB_UNLOCK(stcb); hold_tcblock = 0; } } SOCKBUF_LOCK(&so->so_snd); /*- * This is a bit strange, but I think it will * work. The total_output_queue_size is locked and * protected by the TCB_LOCK, which we just released. * There is a race that can occur between releasing it * above, and me getting the socket lock, where sacks * come in but we have not put the SB_WAIT on the * so_snd buffer to get the wakeup. After the LOCK * is applied the sack_processing will also need to * LOCK the so->so_snd to do the actual sowwakeup(). So * once we have the socket buffer lock if we recheck the * size we KNOW we will get to sleep safely with the * wakeup flag in place. */ if (SCTP_SB_LIMIT_SND(so) <= (stcb->asoc.total_output_queue_size + min(SCTP_BASE_SYSCTL(sctp_add_more_threshold), SCTP_SB_LIMIT_SND(so)))) { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_BLK_LOGGING_ENABLE) { sctp_log_block(SCTP_BLOCK_LOG_INTO_BLK, asoc, uio->uio_resid); } be.error = 0; stcb->block_entry = &be; error = sbwait(&so->so_snd); stcb->block_entry = NULL; if (error || so->so_error || be.error) { if (error == 0) { if (so->so_error) error = so->so_error; if (be.error) { error = be.error; } } SOCKBUF_UNLOCK(&so->so_snd); goto out_unlocked; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_BLK_LOGGING_ENABLE) { sctp_log_block(SCTP_BLOCK_LOG_OUTOF_BLK, asoc, stcb->asoc.total_output_queue_size); } } SOCKBUF_UNLOCK(&so->so_snd); if (stcb->asoc.state & SCTP_STATE_ABOUT_TO_BE_FREED) { goto out_unlocked; } } SCTP_TCB_SEND_LOCK(stcb); if (sp) { if (sp->msg_is_complete == 0) { strm->last_msg_incomplete = 1; asoc->stream_locked = 1; asoc->stream_locked_on = srcv->sinfo_stream; } else { sp->sender_all_done = 1; strm->last_msg_incomplete = 0; asoc->stream_locked = 0; } } else { SCTP_PRINTF("Huh no sp TSNH?\n"); strm->last_msg_incomplete = 0; asoc->stream_locked = 0; } SCTP_TCB_SEND_UNLOCK(stcb); if (uio->uio_resid == 0) { got_all_of_the_send = 1; } } else { /* We send in a 0, since we do NOT have any locks */ error = sctp_msg_append(stcb, net, top, srcv, 0); top = NULL; if (srcv->sinfo_flags & SCTP_EOF) { /* * This should only happen for Panda for the mbuf * send case, which does NOT yet support EEOR mode. * Thus, we can just set this flag to do the proper * EOF handling. */ got_all_of_the_send = 1; } } if (error) { goto out; } dataless_eof: /* EOF thing ? */ if ((srcv->sinfo_flags & SCTP_EOF) && (got_all_of_the_send == 1)) { int cnt; SCTP_STAT_INCR(sctps_sends_with_eof); error = 0; if (hold_tcblock == 0) { SCTP_TCB_LOCK(stcb); hold_tcblock = 1; } cnt = sctp_is_there_unsent_data(stcb, SCTP_SO_LOCKED); if (TAILQ_EMPTY(&asoc->send_queue) && TAILQ_EMPTY(&asoc->sent_queue) && (cnt == 0)) { if (asoc->locked_on_sending) { goto abort_anyway; } /* there is nothing queued to send, so I'm done... */ if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT)) { struct sctp_nets *netp; /* only send SHUTDOWN the first time through */ if (SCTP_GET_STATE(asoc) == SCTP_STATE_OPEN) { SCTP_STAT_DECR_GAUGE32(sctps_currestab); } SCTP_SET_STATE(asoc, SCTP_STATE_SHUTDOWN_SENT); SCTP_CLEAR_SUBSTATE(asoc, SCTP_STATE_SHUTDOWN_PENDING); sctp_stop_timers_for_shutdown(stcb); if (stcb->asoc.alternate) { netp = stcb->asoc.alternate; } else { netp = stcb->asoc.primary_destination; } sctp_send_shutdown(stcb, netp); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWN, stcb->sctp_ep, stcb, netp); sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); } } else { /*- * we still got (or just got) data to send, so set * SHUTDOWN_PENDING */ /*- * XXX sockets draft says that SCTP_EOF should be * sent with no data. currently, we will allow user * data to be sent first and move to * SHUTDOWN-PENDING */ if ((SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_SENT) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_RECEIVED) && (SCTP_GET_STATE(asoc) != SCTP_STATE_SHUTDOWN_ACK_SENT)) { if (hold_tcblock == 0) { SCTP_TCB_LOCK(stcb); hold_tcblock = 1; } if (asoc->locked_on_sending) { /* Locked to send out the data */ struct sctp_stream_queue_pending *sp; sp = TAILQ_LAST(&asoc->locked_on_sending->outqueue, sctp_streamhead); if (sp) { if ((sp->length == 0) && (sp->msg_is_complete == 0)) asoc->state |= SCTP_STATE_PARTIAL_MSG_LEFT; } } asoc->state |= SCTP_STATE_SHUTDOWN_PENDING; if (TAILQ_EMPTY(&asoc->send_queue) && TAILQ_EMPTY(&asoc->sent_queue) && (asoc->state & SCTP_STATE_PARTIAL_MSG_LEFT)) { struct mbuf *op_err; char msg[SCTP_DIAG_INFO_LEN]; abort_anyway: if (free_cnt_applied) { atomic_add_int(&stcb->asoc.refcnt, -1); free_cnt_applied = 0; } snprintf(msg, sizeof(msg), "%s:%d at %s", __FILE__, __LINE__, __func__); op_err = sctp_generate_cause(SCTP_BASE_SYSCTL(sctp_diag_info_code), msg); sctp_abort_an_association(stcb->sctp_ep, stcb, op_err, SCTP_SO_LOCKED); /* * now relock the stcb so everything * is sane */ hold_tcblock = 0; stcb = NULL; goto out; } sctp_timer_start(SCTP_TIMER_TYPE_SHUTDOWNGUARD, stcb->sctp_ep, stcb, asoc->primary_destination); sctp_feature_off(inp, SCTP_PCB_FLAGS_NODELAY); } } } skip_out_eof: if (!TAILQ_EMPTY(&stcb->asoc.control_send_queue)) { some_on_control = 1; } if (queue_only_for_init) { if (hold_tcblock == 0) { SCTP_TCB_LOCK(stcb); hold_tcblock = 1; } if (SCTP_GET_STATE(&stcb->asoc) == SCTP_STATE_OPEN) { /* a collision took us forward? */ queue_only = 0; } else { sctp_send_initiate(inp, stcb, SCTP_SO_LOCKED); SCTP_SET_STATE(&stcb->asoc, SCTP_STATE_COOKIE_WAIT); queue_only = 1; } } if ((net->flight_size > net->cwnd) && (stcb->asoc.sctp_cmt_on_off == 0)) { SCTP_STAT_INCR(sctps_send_cwnd_avoid); queue_only = 1; } else if (asoc->ifp_had_enobuf) { SCTP_STAT_INCR(sctps_ifnomemqueued); if (net->flight_size > (2 * net->mtu)) { queue_only = 1; } asoc->ifp_had_enobuf = 0; } un_sent = ((stcb->asoc.total_output_queue_size - stcb->asoc.total_flight) + (stcb->asoc.stream_queue_cnt * sizeof(struct sctp_data_chunk))); if ((sctp_is_feature_off(inp, SCTP_PCB_FLAGS_NODELAY)) && (stcb->asoc.total_flight > 0) && (stcb->asoc.stream_queue_cnt < SCTP_MAX_DATA_BUNDLING) && (un_sent < (int)(stcb->asoc.smallest_mtu - SCTP_MIN_OVERHEAD))) { /*- * Ok, Nagle is set on and we have data outstanding. * Don't send anything and let SACKs drive out the * data unless wen have a "full" segment to send. */ if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_NAGLE_LOGGING_ENABLE) { sctp_log_nagle_event(stcb, SCTP_NAGLE_APPLIED); } SCTP_STAT_INCR(sctps_naglequeued); nagle_applies = 1; } else { if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_NAGLE_LOGGING_ENABLE) { if (sctp_is_feature_off(inp, SCTP_PCB_FLAGS_NODELAY)) sctp_log_nagle_event(stcb, SCTP_NAGLE_SKIPPED); } SCTP_STAT_INCR(sctps_naglesent); nagle_applies = 0; } if (SCTP_BASE_SYSCTL(sctp_logging_level) & SCTP_BLK_LOGGING_ENABLE) { sctp_misc_ints(SCTP_CWNDLOG_PRESEND, queue_only_for_init, queue_only, nagle_applies, un_sent); sctp_misc_ints(SCTP_CWNDLOG_PRESEND, stcb->asoc.total_output_queue_size, stcb->asoc.total_flight, stcb->asoc.chunks_on_out_queue, stcb->asoc.total_flight_count); } if ((queue_only == 0) && (nagle_applies == 0) && (stcb->asoc.peers_rwnd && un_sent)) { /* we can attempt to send too. */ if (hold_tcblock == 0) { /* * If there is activity recv'ing sacks no need to * send */ if (SCTP_TCB_TRYLOCK(stcb)) { sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_USR_SEND, SCTP_SO_LOCKED); hold_tcblock = 1; } } else { sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_USR_SEND, SCTP_SO_LOCKED); } } else if ((queue_only == 0) && (stcb->asoc.peers_rwnd == 0) && (stcb->asoc.total_flight == 0)) { /* We get to have a probe outstanding */ if (hold_tcblock == 0) { hold_tcblock = 1; SCTP_TCB_LOCK(stcb); } sctp_chunk_output(inp, stcb, SCTP_OUTPUT_FROM_USR_SEND, SCTP_SO_LOCKED); } else if (some_on_control) { int num_out, reason, frag_point; /* Here we do control only */ if (hold_tcblock == 0) { hold_tcblock = 1; SCTP_TCB_LOCK(stcb); } frag_point = sctp_get_frag_point(stcb, &stcb->asoc); (void)sctp_med_chunk_output(inp, stcb, &stcb->asoc, &num_out, &reason, 1, 1, &now, &now_filled, frag_point, SCTP_SO_LOCKED); } SCTPDBG(SCTP_DEBUG_OUTPUT1, "USR Send complete qo:%d prw:%d unsent:%d tf:%d cooq:%d toqs:%d err:%d\n", queue_only, stcb->asoc.peers_rwnd, un_sent, stcb->asoc.total_flight, stcb->asoc.chunks_on_out_queue, stcb->asoc.total_output_queue_size, error); out: out_unlocked: if (local_soresv && stcb) { atomic_subtract_int(&stcb->asoc.sb_send_resv, sndlen); } if (create_lock_applied) { SCTP_ASOC_CREATE_UNLOCK(inp); } if ((stcb) && hold_tcblock) { SCTP_TCB_UNLOCK(stcb); } if (stcb && free_cnt_applied) { atomic_add_int(&stcb->asoc.refcnt, -1); } #ifdef INVARIANTS if (stcb) { if (mtx_owned(&stcb->tcb_mtx)) { panic("Leaving with tcb mtx owned?"); } if (mtx_owned(&stcb->tcb_send_mtx)) { panic("Leaving with tcb send mtx owned?"); } } #endif if (top) { sctp_m_freem(top); } if (control) { sctp_m_freem(control); } return (error); } /* * generate an AUTHentication chunk, if required */ struct mbuf * sctp_add_auth_chunk(struct mbuf *m, struct mbuf **m_end, struct sctp_auth_chunk **auth_ret, uint32_t * offset, struct sctp_tcb *stcb, uint8_t chunk) { struct mbuf *m_auth; struct sctp_auth_chunk *auth; int chunk_len; struct mbuf *cn; if ((m_end == NULL) || (auth_ret == NULL) || (offset == NULL) || (stcb == NULL)) return (m); if (stcb->asoc.auth_supported == 0) { return (m); } /* does the requested chunk require auth? */ if (!sctp_auth_is_required_chunk(chunk, stcb->asoc.peer_auth_chunks)) { return (m); } m_auth = sctp_get_mbuf_for_msg(sizeof(*auth), 0, M_NOWAIT, 1, MT_HEADER); if (m_auth == NULL) { /* no mbuf's */ return (m); } /* reserve some space if this will be the first mbuf */ if (m == NULL) SCTP_BUF_RESV_UF(m_auth, SCTP_MIN_OVERHEAD); /* fill in the AUTH chunk details */ auth = mtod(m_auth, struct sctp_auth_chunk *); bzero(auth, sizeof(*auth)); auth->ch.chunk_type = SCTP_AUTHENTICATION; auth->ch.chunk_flags = 0; chunk_len = sizeof(*auth) + sctp_get_hmac_digest_len(stcb->asoc.peer_hmac_id); auth->ch.chunk_length = htons(chunk_len); auth->hmac_id = htons(stcb->asoc.peer_hmac_id); /* key id and hmac digest will be computed and filled in upon send */ /* save the offset where the auth was inserted into the chain */ *offset = 0; for (cn = m; cn; cn = SCTP_BUF_NEXT(cn)) { *offset += SCTP_BUF_LEN(cn); } /* update length and return pointer to the auth chunk */ SCTP_BUF_LEN(m_auth) = chunk_len; m = sctp_copy_mbufchain(m_auth, m, m_end, 1, chunk_len, 0); if (auth_ret != NULL) *auth_ret = auth; return (m); } #ifdef INET6 int sctp_v6src_match_nexthop(struct sockaddr_in6 *src6, sctp_route_t * ro) { struct nd_prefix *pfx = NULL; struct nd_pfxrouter *pfxrtr = NULL; struct sockaddr_in6 gw6; if (ro == NULL || ro->ro_rt == NULL || src6->sin6_family != AF_INET6) return (0); /* get prefix entry of address */ LIST_FOREACH(pfx, &MODULE_GLOBAL(nd_prefix), ndpr_entry) { if (pfx->ndpr_stateflags & NDPRF_DETACHED) continue; if (IN6_ARE_MASKED_ADDR_EQUAL(&pfx->ndpr_prefix.sin6_addr, &src6->sin6_addr, &pfx->ndpr_mask)) break; } /* no prefix entry in the prefix list */ if (pfx == NULL) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "No prefix entry for "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, (struct sockaddr *)src6); return (0); } SCTPDBG(SCTP_DEBUG_OUTPUT2, "v6src_match_nexthop(), Prefix entry is "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, (struct sockaddr *)src6); /* search installed gateway from prefix entry */ LIST_FOREACH(pfxrtr, &pfx->ndpr_advrtrs, pfr_entry) { memset(&gw6, 0, sizeof(struct sockaddr_in6)); gw6.sin6_family = AF_INET6; gw6.sin6_len = sizeof(struct sockaddr_in6); memcpy(&gw6.sin6_addr, &pfxrtr->router->rtaddr, sizeof(struct in6_addr)); SCTPDBG(SCTP_DEBUG_OUTPUT2, "prefix router is "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, (struct sockaddr *)&gw6); SCTPDBG(SCTP_DEBUG_OUTPUT2, "installed router is "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, ro->ro_rt->rt_gateway); if (sctp_cmpaddr((struct sockaddr *)&gw6, ro->ro_rt->rt_gateway)) { SCTPDBG(SCTP_DEBUG_OUTPUT2, "pfxrouter is installed\n"); return (1); } } SCTPDBG(SCTP_DEBUG_OUTPUT2, "pfxrouter is not installed\n"); return (0); } #endif int sctp_v4src_match_nexthop(struct sctp_ifa *sifa, sctp_route_t * ro) { #ifdef INET struct sockaddr_in *sin, *mask; struct ifaddr *ifa; struct in_addr srcnetaddr, gwnetaddr; if (ro == NULL || ro->ro_rt == NULL || sifa->address.sa.sa_family != AF_INET) { return (0); } ifa = (struct ifaddr *)sifa->ifa; mask = (struct sockaddr_in *)(ifa->ifa_netmask); sin = &sifa->address.sin; srcnetaddr.s_addr = (sin->sin_addr.s_addr & mask->sin_addr.s_addr); SCTPDBG(SCTP_DEBUG_OUTPUT1, "match_nexthop4: src address is "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, &sifa->address.sa); SCTPDBG(SCTP_DEBUG_OUTPUT1, "network address is %x\n", srcnetaddr.s_addr); sin = (struct sockaddr_in *)ro->ro_rt->rt_gateway; gwnetaddr.s_addr = (sin->sin_addr.s_addr & mask->sin_addr.s_addr); SCTPDBG(SCTP_DEBUG_OUTPUT1, "match_nexthop4: nexthop is "); SCTPDBG_ADDR(SCTP_DEBUG_OUTPUT2, ro->ro_rt->rt_gateway); SCTPDBG(SCTP_DEBUG_OUTPUT1, "network address is %x\n", gwnetaddr.s_addr); if (srcnetaddr.s_addr == gwnetaddr.s_addr) { return (1); } #endif return (0); } Index: user/ngie/make_check/sys/netinet/sctp_output.h =================================================================== --- user/ngie/make_check/sys/netinet/sctp_output.h (revision 295077) +++ user/ngie/make_check/sys/netinet/sctp_output.h (revision 295078) @@ -1,216 +1,217 @@ /*- * Copyright (c) 2001-2007, by Cisco Systems, Inc. All rights reserved. * Copyright (c) 2008-2012, by Randall Stewart. All rights reserved. * Copyright (c) 2008-2012, by Michael Tuexen. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * a) Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * b) Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the distribution. * * c) Neither the name of Cisco Systems, Inc. nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF * THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #ifndef _NETINET_SCTP_OUTPUT_H_ #define _NETINET_SCTP_OUTPUT_H_ #include #if defined(_KERNEL) || defined(__Userspace__) struct mbuf * sctp_add_addresses_to_i_ia(struct sctp_inpcb *inp, struct sctp_tcb *stcb, struct sctp_scoping *scope, struct mbuf *m_at, int cnt_inits_to, uint16_t * padding_len, uint16_t * chunk_len); int sctp_is_addr_restricted(struct sctp_tcb *, struct sctp_ifa *); int sctp_is_address_in_scope(struct sctp_ifa *ifa, struct sctp_scoping *scope, int do_update); int sctp_is_addr_in_ep(struct sctp_inpcb *inp, struct sctp_ifa *ifa); struct sctp_ifa * sctp_source_address_selection(struct sctp_inpcb *inp, struct sctp_tcb *stcb, sctp_route_t * ro, struct sctp_nets *net, int non_asoc_addr_ok, uint32_t vrf_id); int sctp_v6src_match_nexthop(struct sockaddr_in6 *src6, sctp_route_t * ro); int sctp_v4src_match_nexthop(struct sctp_ifa *sifa, sctp_route_t * ro); void sctp_send_initiate(struct sctp_inpcb *, struct sctp_tcb *, int #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ); void -sctp_send_initiate_ack(struct sctp_inpcb *, struct sctp_tcb *, struct mbuf *, +sctp_send_initiate_ack(struct sctp_inpcb *, struct sctp_tcb *, + struct sctp_nets *, struct mbuf *, int, int, struct sockaddr *, struct sockaddr *, struct sctphdr *, struct sctp_init_chunk *, uint8_t, uint32_t, uint32_t, uint16_t, int); struct mbuf * sctp_arethere_unrecognized_parameters(struct mbuf *, int, int *, struct sctp_chunkhdr *, int *); void sctp_queue_op_err(struct sctp_tcb *, struct mbuf *); int sctp_send_cookie_echo(struct mbuf *, int, struct sctp_tcb *, struct sctp_nets *); void sctp_send_cookie_ack(struct sctp_tcb *); void sctp_send_heartbeat_ack(struct sctp_tcb *, struct mbuf *, int, int, struct sctp_nets *); void sctp_remove_from_wheel(struct sctp_tcb *stcb, struct sctp_association *asoc, struct sctp_stream_out *strq, int holds_lock); void sctp_send_shutdown(struct sctp_tcb *, struct sctp_nets *); void sctp_send_shutdown_ack(struct sctp_tcb *, struct sctp_nets *); void sctp_send_shutdown_complete(struct sctp_tcb *, struct sctp_nets *, int); void sctp_send_shutdown_complete2(struct sockaddr *, struct sockaddr *, struct sctphdr *, uint8_t, uint32_t, uint16_t, uint32_t, uint16_t); void sctp_send_asconf(struct sctp_tcb *, struct sctp_nets *, int addr_locked); void sctp_send_asconf_ack(struct sctp_tcb *); int sctp_get_frag_point(struct sctp_tcb *, struct sctp_association *); void sctp_toss_old_cookies(struct sctp_tcb *, struct sctp_association *); void sctp_toss_old_asconf(struct sctp_tcb *); void sctp_fix_ecn_echo(struct sctp_association *); void sctp_move_chunks_from_net(struct sctp_tcb *stcb, struct sctp_nets *net); int sctp_output(struct sctp_inpcb *, struct mbuf *, struct sockaddr *, struct mbuf *, struct thread *, int); void sctp_chunk_output(struct sctp_inpcb *, struct sctp_tcb *, int, int #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ); void sctp_send_abort_tcb(struct sctp_tcb *, struct mbuf *, int #if !defined(__APPLE__) && !defined(SCTP_SO_LOCK_TESTING) SCTP_UNUSED #endif ); void send_forward_tsn(struct sctp_tcb *, struct sctp_association *); void sctp_send_sack(struct sctp_tcb *, int); void sctp_send_hb(struct sctp_tcb *, struct sctp_nets *, int); void sctp_send_ecn_echo(struct sctp_tcb *, struct sctp_nets *, uint32_t); void sctp_send_packet_dropped(struct sctp_tcb *, struct sctp_nets *, struct mbuf *, int, int, int); void sctp_send_cwr(struct sctp_tcb *, struct sctp_nets *, uint32_t, uint8_t); void sctp_add_stream_reset_result(struct sctp_tmit_chunk *, uint32_t, uint32_t); void sctp_send_deferred_reset_response(struct sctp_tcb *, struct sctp_stream_reset_list *, int); void sctp_add_stream_reset_result_tsn(struct sctp_tmit_chunk *, uint32_t, uint32_t, uint32_t, uint32_t); int sctp_send_stream_reset_out_if_possible(struct sctp_tcb *, int); int sctp_send_str_reset_req(struct sctp_tcb *, uint16_t, uint16_t *, uint8_t, uint8_t, uint8_t, uint16_t, uint16_t, uint8_t); void sctp_send_abort(struct mbuf *, int, struct sockaddr *, struct sockaddr *, struct sctphdr *, uint32_t, struct mbuf *, uint8_t, uint32_t, uint16_t, uint32_t, uint16_t); void sctp_send_operr_to(struct sockaddr *, struct sockaddr *, struct sctphdr *, uint32_t, struct mbuf *, uint8_t, uint32_t, uint16_t, uint32_t, uint16_t); #endif /* _KERNEL || __Userspace__ */ #if defined(_KERNEL) || defined(__Userspace__) int sctp_sosend(struct socket *so, struct sockaddr *addr, struct uio *uio, struct mbuf *top, struct mbuf *control, int flags, struct thread *p ); #endif #endif Index: user/ngie/make_check/sys/netinet/sctp_var.h =================================================================== --- user/ngie/make_check/sys/netinet/sctp_var.h (revision 295077) +++ user/ngie/make_check/sys/netinet/sctp_var.h (revision 295078) @@ -1,356 +1,356 @@ /*- * Copyright (c) 2001-2008, by Cisco Systems, Inc. All rights reserved. * Copyright (c) 2008-2012, by Randall Stewart. All rights reserved. * Copyright (c) 2008-2012, by Michael Tuexen. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are met: * * a) Redistributions of source code must retain the above copyright notice, * this list of conditions and the following disclaimer. * * b) Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the distribution. * * c) Neither the name of Cisco Systems, Inc. nor the names of its * contributors may be used to endorse or promote products derived * from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF * THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #ifndef _NETINET_SCTP_VAR_H_ #define _NETINET_SCTP_VAR_H_ #include #if defined(_KERNEL) || defined(__Userspace__) extern struct pr_usrreqs sctp_usrreqs; #define sctp_feature_on(inp, feature) (inp->sctp_features |= feature) #define sctp_feature_off(inp, feature) (inp->sctp_features &= ~feature) #define sctp_is_feature_on(inp, feature) ((inp->sctp_features & feature) == feature) #define sctp_is_feature_off(inp, feature) ((inp->sctp_features & feature) == 0) #define sctp_stcb_feature_on(inp, stcb, feature) {\ if (stcb) { \ stcb->asoc.sctp_features |= feature; \ } else if (inp) { \ inp->sctp_features |= feature; \ } \ } #define sctp_stcb_feature_off(inp, stcb, feature) {\ if (stcb) { \ stcb->asoc.sctp_features &= ~feature; \ } else if (inp) { \ inp->sctp_features &= ~feature; \ } \ } #define sctp_stcb_is_feature_on(inp, stcb, feature) \ (((stcb != NULL) && \ ((stcb->asoc.sctp_features & feature) == feature)) || \ ((stcb == NULL) && (inp != NULL) && \ ((inp->sctp_features & feature) == feature))) #define sctp_stcb_is_feature_off(inp, stcb, feature) \ (((stcb != NULL) && \ ((stcb->asoc.sctp_features & feature) == 0)) || \ ((stcb == NULL) && (inp != NULL) && \ ((inp->sctp_features & feature) == 0)) || \ ((stcb == NULL) && (inp == NULL))) /* managing mobility_feature in inpcb (by micchie) */ #define sctp_mobility_feature_on(inp, feature) (inp->sctp_mobility_features |= feature) #define sctp_mobility_feature_off(inp, feature) (inp->sctp_mobility_features &= ~feature) #define sctp_is_mobility_feature_on(inp, feature) (inp->sctp_mobility_features & feature) #define sctp_is_mobility_feature_off(inp, feature) ((inp->sctp_mobility_features & feature) == 0) #define sctp_maxspace(sb) (max((sb)->sb_hiwat,SCTP_MINIMAL_RWND)) #define sctp_sbspace(asoc, sb) ((long) ((sctp_maxspace(sb) > (asoc)->sb_cc) ? (sctp_maxspace(sb) - (asoc)->sb_cc) : 0)) #define sctp_sbspace_failedmsgs(sb) ((long) ((sctp_maxspace(sb) > (sb)->sb_cc) ? (sctp_maxspace(sb) - (sb)->sb_cc) : 0)) -#define sctp_sbspace_sub(a,b) ((a > b) ? (a - b) : 0) +#define sctp_sbspace_sub(a,b) (((a) > (b)) ? ((a) - (b)) : 0) /* * I tried to cache the readq entries at one point. But the reality * is that it did not add any performance since this meant we had to * lock the STCB on read. And at that point once you have to do an * extra lock, it really does not matter if the lock is in the ZONE * stuff or in our code. Note that this same problem would occur with * an mbuf cache as well so it is not really worth doing, at least * right now :-D */ #define sctp_free_a_readq(_stcb, _readq) { \ SCTP_ZONE_FREE(SCTP_BASE_INFO(ipi_zone_readq), (_readq)); \ SCTP_DECR_READQ_COUNT(); \ } #define sctp_alloc_a_readq(_stcb, _readq) { \ (_readq) = SCTP_ZONE_GET(SCTP_BASE_INFO(ipi_zone_readq), struct sctp_queued_to_read); \ if ((_readq)) { \ SCTP_INCR_READQ_COUNT(); \ } \ } #define sctp_free_a_strmoq(_stcb, _strmoq, _so_locked) { \ if ((_strmoq)->holds_key_ref) { \ sctp_auth_key_release(stcb, sp->auth_keyid, _so_locked); \ (_strmoq)->holds_key_ref = 0; \ } \ SCTP_ZONE_FREE(SCTP_BASE_INFO(ipi_zone_strmoq), (_strmoq)); \ SCTP_DECR_STRMOQ_COUNT(); \ } #define sctp_alloc_a_strmoq(_stcb, _strmoq) { \ (_strmoq) = SCTP_ZONE_GET(SCTP_BASE_INFO(ipi_zone_strmoq), struct sctp_stream_queue_pending); \ if ((_strmoq)) { \ memset(_strmoq, 0, sizeof(struct sctp_stream_queue_pending)); \ SCTP_INCR_STRMOQ_COUNT(); \ (_strmoq)->holds_key_ref = 0; \ } \ } #define sctp_free_a_chunk(_stcb, _chk, _so_locked) { \ if ((_chk)->holds_key_ref) {\ sctp_auth_key_release((_stcb), (_chk)->auth_keyid, _so_locked); \ (_chk)->holds_key_ref = 0; \ } \ if (_stcb) { \ SCTP_TCB_LOCK_ASSERT((_stcb)); \ if ((_chk)->whoTo) { \ sctp_free_remote_addr((_chk)->whoTo); \ (_chk)->whoTo = NULL; \ } \ if (((_stcb)->asoc.free_chunk_cnt > SCTP_BASE_SYSCTL(sctp_asoc_free_resc_limit)) || \ (SCTP_BASE_INFO(ipi_free_chunks) > SCTP_BASE_SYSCTL(sctp_system_free_resc_limit))) { \ SCTP_ZONE_FREE(SCTP_BASE_INFO(ipi_zone_chunk), (_chk)); \ SCTP_DECR_CHK_COUNT(); \ } else { \ TAILQ_INSERT_TAIL(&(_stcb)->asoc.free_chunks, (_chk), sctp_next); \ (_stcb)->asoc.free_chunk_cnt++; \ atomic_add_int(&SCTP_BASE_INFO(ipi_free_chunks), 1); \ } \ } else { \ SCTP_ZONE_FREE(SCTP_BASE_INFO(ipi_zone_chunk), (_chk)); \ SCTP_DECR_CHK_COUNT(); \ } \ } #define sctp_alloc_a_chunk(_stcb, _chk) { \ if (TAILQ_EMPTY(&(_stcb)->asoc.free_chunks)) { \ (_chk) = SCTP_ZONE_GET(SCTP_BASE_INFO(ipi_zone_chunk), struct sctp_tmit_chunk); \ if ((_chk)) { \ SCTP_INCR_CHK_COUNT(); \ (_chk)->whoTo = NULL; \ (_chk)->holds_key_ref = 0; \ } \ } else { \ (_chk) = TAILQ_FIRST(&(_stcb)->asoc.free_chunks); \ TAILQ_REMOVE(&(_stcb)->asoc.free_chunks, (_chk), sctp_next); \ atomic_subtract_int(&SCTP_BASE_INFO(ipi_free_chunks), 1); \ (_chk)->holds_key_ref = 0; \ SCTP_STAT_INCR(sctps_cached_chk); \ (_stcb)->asoc.free_chunk_cnt--; \ } \ } #define sctp_free_remote_addr(__net) { \ if ((__net)) { \ if (SCTP_DECREMENT_AND_CHECK_REFCOUNT(&(__net)->ref_count)) { \ (void)SCTP_OS_TIMER_STOP(&(__net)->rxt_timer.timer); \ (void)SCTP_OS_TIMER_STOP(&(__net)->pmtu_timer.timer); \ (void)SCTP_OS_TIMER_STOP(&(__net)->hb_timer.timer); \ if ((__net)->ro.ro_rt) { \ RTFREE((__net)->ro.ro_rt); \ (__net)->ro.ro_rt = NULL; \ } \ if ((__net)->src_addr_selected) { \ sctp_free_ifa((__net)->ro._s_addr); \ (__net)->ro._s_addr = NULL; \ } \ (__net)->src_addr_selected = 0; \ (__net)->dest_state &= ~SCTP_ADDR_REACHABLE; \ SCTP_ZONE_FREE(SCTP_BASE_INFO(ipi_zone_net), (__net)); \ SCTP_DECR_RADDR_COUNT(); \ } \ } \ } #define sctp_sbfree(ctl, stcb, sb, m) { \ SCTP_SAVE_ATOMIC_DECREMENT(&(sb)->sb_cc, SCTP_BUF_LEN((m))); \ SCTP_SAVE_ATOMIC_DECREMENT(&(sb)->sb_mbcnt, MSIZE); \ if (((ctl)->do_not_ref_stcb == 0) && stcb) {\ SCTP_SAVE_ATOMIC_DECREMENT(&(stcb)->asoc.sb_cc, SCTP_BUF_LEN((m))); \ SCTP_SAVE_ATOMIC_DECREMENT(&(stcb)->asoc.my_rwnd_control_len, MSIZE); \ } \ if (SCTP_BUF_TYPE(m) != MT_DATA && SCTP_BUF_TYPE(m) != MT_HEADER && \ SCTP_BUF_TYPE(m) != MT_OOBDATA) \ atomic_subtract_int(&(sb)->sb_ctl,SCTP_BUF_LEN((m))); \ } #define sctp_sballoc(stcb, sb, m) { \ atomic_add_int(&(sb)->sb_cc,SCTP_BUF_LEN((m))); \ atomic_add_int(&(sb)->sb_mbcnt, MSIZE); \ if (stcb) { \ atomic_add_int(&(stcb)->asoc.sb_cc,SCTP_BUF_LEN((m))); \ atomic_add_int(&(stcb)->asoc.my_rwnd_control_len, MSIZE); \ } \ if (SCTP_BUF_TYPE(m) != MT_DATA && SCTP_BUF_TYPE(m) != MT_HEADER && \ SCTP_BUF_TYPE(m) != MT_OOBDATA) \ atomic_add_int(&(sb)->sb_ctl,SCTP_BUF_LEN((m))); \ } #define sctp_ucount_incr(val) { \ val++; \ } #define sctp_ucount_decr(val) { \ if (val > 0) { \ val--; \ } else { \ val = 0; \ } \ } #define sctp_mbuf_crush(data) do { \ struct mbuf *_m; \ _m = (data); \ while (_m && (SCTP_BUF_LEN(_m) == 0)) { \ (data) = SCTP_BUF_NEXT(_m); \ SCTP_BUF_NEXT(_m) = NULL; \ sctp_m_free(_m); \ _m = (data); \ } \ } while (0) #define sctp_flight_size_decrease(tp1) do { \ if (tp1->whoTo->flight_size >= tp1->book_size) \ tp1->whoTo->flight_size -= tp1->book_size; \ else \ tp1->whoTo->flight_size = 0; \ } while (0) #define sctp_flight_size_increase(tp1) do { \ (tp1)->whoTo->flight_size += (tp1)->book_size; \ } while (0) #ifdef SCTP_FS_SPEC_LOG #define sctp_total_flight_decrease(stcb, tp1) do { \ if (stcb->asoc.fs_index > SCTP_FS_SPEC_LOG_SIZE) \ stcb->asoc.fs_index = 0;\ stcb->asoc.fslog[stcb->asoc.fs_index].total_flight = stcb->asoc.total_flight; \ stcb->asoc.fslog[stcb->asoc.fs_index].tsn = tp1->rec.data.TSN_seq; \ stcb->asoc.fslog[stcb->asoc.fs_index].book = tp1->book_size; \ stcb->asoc.fslog[stcb->asoc.fs_index].sent = tp1->sent; \ stcb->asoc.fslog[stcb->asoc.fs_index].incr = 0; \ stcb->asoc.fslog[stcb->asoc.fs_index].decr = 1; \ stcb->asoc.fs_index++; \ tp1->window_probe = 0; \ if (stcb->asoc.total_flight >= tp1->book_size) { \ stcb->asoc.total_flight -= tp1->book_size; \ if (stcb->asoc.total_flight_count > 0) \ stcb->asoc.total_flight_count--; \ } else { \ stcb->asoc.total_flight = 0; \ stcb->asoc.total_flight_count = 0; \ } \ } while (0) #define sctp_total_flight_increase(stcb, tp1) do { \ if (stcb->asoc.fs_index > SCTP_FS_SPEC_LOG_SIZE) \ stcb->asoc.fs_index = 0;\ stcb->asoc.fslog[stcb->asoc.fs_index].total_flight = stcb->asoc.total_flight; \ stcb->asoc.fslog[stcb->asoc.fs_index].tsn = tp1->rec.data.TSN_seq; \ stcb->asoc.fslog[stcb->asoc.fs_index].book = tp1->book_size; \ stcb->asoc.fslog[stcb->asoc.fs_index].sent = tp1->sent; \ stcb->asoc.fslog[stcb->asoc.fs_index].incr = 1; \ stcb->asoc.fslog[stcb->asoc.fs_index].decr = 0; \ stcb->asoc.fs_index++; \ (stcb)->asoc.total_flight_count++; \ (stcb)->asoc.total_flight += (tp1)->book_size; \ } while (0) #else #define sctp_total_flight_decrease(stcb, tp1) do { \ tp1->window_probe = 0; \ if (stcb->asoc.total_flight >= tp1->book_size) { \ stcb->asoc.total_flight -= tp1->book_size; \ if (stcb->asoc.total_flight_count > 0) \ stcb->asoc.total_flight_count--; \ } else { \ stcb->asoc.total_flight = 0; \ stcb->asoc.total_flight_count = 0; \ } \ } while (0) #define sctp_total_flight_increase(stcb, tp1) do { \ (stcb)->asoc.total_flight_count++; \ (stcb)->asoc.total_flight += (tp1)->book_size; \ } while (0) #endif #define SCTP_PF_ENABLED(_net) (_net->pf_threshold < _net->failure_threshold) #define SCTP_NET_IS_PF(_net) (_net->pf_threshold < _net->error_count) struct sctp_nets; struct sctp_inpcb; struct sctp_tcb; struct sctphdr; void sctp_close(struct socket *so); int sctp_disconnect(struct socket *so); void sctp_ctlinput(int, struct sockaddr *, void *); int sctp_ctloutput(struct socket *, struct sockopt *); #ifdef INET void sctp_input_with_port(struct mbuf *, int, uint16_t); int sctp_input(struct mbuf **, int *, int); #endif void sctp_pathmtu_adjustment(struct sctp_tcb *, uint16_t); void sctp_drain(void); void sctp_init(void); void sctp_finish(void); int sctp_flush(struct socket *, int); int sctp_shutdown(struct socket *); void sctp_notify(struct sctp_inpcb *, struct ip *ip, struct sctphdr *, struct sockaddr *, struct sctp_tcb *, struct sctp_nets *); int sctp_bindx(struct socket *, int, struct sockaddr_storage *, int, int, struct proc *); /* can't use sctp_assoc_t here */ int sctp_peeloff(struct socket *, struct socket *, int, caddr_t, int *); int sctp_ingetaddr(struct socket *, struct sockaddr **); int sctp_peeraddr(struct socket *, struct sockaddr **); int sctp_listen(struct socket *, int, struct thread *); int sctp_accept(struct socket *, struct sockaddr **); #endif /* _KERNEL */ #endif /* !_NETINET_SCTP_VAR_H_ */ Index: user/ngie/make_check/sys/riscv/conf/DEFAULTS =================================================================== --- user/ngie/make_check/sys/riscv/conf/DEFAULTS (nonexistent) +++ user/ngie/make_check/sys/riscv/conf/DEFAULTS (revision 295078) @@ -0,0 +1,13 @@ +# +# DEFAULTS -- Default kernel configuration file for FreeBSD/RISC-V +# +# $FreeBSD$ + +machine riscv riscv64 + +# Pseudo devices. +device mem # Memory and kernel memory devices + +# Default partitioning schemes +options GEOM_PART_BSD +options GEOM_PART_MBR Property changes on: user/ngie/make_check/sys/riscv/conf/DEFAULTS ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/make_check/sys/riscv/conf/GENERIC =================================================================== --- user/ngie/make_check/sys/riscv/conf/GENERIC (nonexistent) +++ user/ngie/make_check/sys/riscv/conf/GENERIC (revision 295078) @@ -0,0 +1,104 @@ +# +# GENERIC -- Generic kernel configuration file for FreeBSD/RISC-V +# +# For more information on this file, please read the config(5) manual page, +# and/or the handbook section on Kernel Configuration Files: +# +# http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html +# +# The handbook is also available locally in /usr/share/doc/handbook +# if you've installed the doc distribution, otherwise always see the +# FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the +# latest information. +# +# An exhaustive list of options and more detailed explanations of the +# device lines is also present in the ../../conf/NOTES and NOTES files. +# If you are in doubt as to the purpose or necessity of a line, check first +# in NOTES. +# +# $FreeBSD$ + +cpu RISCV +ident GENERIC + +makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols +# makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support +makeoptions NO_MODULES=1 # We don't yet support modules on RISC-V + +options SCHED_ULE # ULE scheduler +options PREEMPTION # Enable kernel thread preemption +options INET # InterNETworking +options INET6 # IPv6 communications protocols +options IPSEC # IP (v4/v6) security +options TCP_OFFLOAD # TCP offload +options SCTP # Stream Control Transmission Protocol +options FFS # Berkeley Fast Filesystem +options SOFTUPDATES # Enable FFS soft updates support +options UFS_ACL # Support for access control lists +options UFS_DIRHASH # Improve performance on big directories +options UFS_GJOURNAL # Enable gjournal-based UFS journaling +options QUOTA # Enable disk quotas for UFS +options MD_ROOT # MD is a potential root device +options NFSCL # Network Filesystem Client +options NFSD # Network Filesystem Server +options NFSLOCKD # Network Lock Manager +options NFS_ROOT # NFS usable as /, requires NFSCL +options MSDOSFS # MSDOS Filesystem +options CD9660 # ISO 9660 Filesystem +options PROCFS # Process filesystem (requires PSEUDOFS) +options PSEUDOFS # Pseudo-filesystem framework +options GEOM_PART_GPT # GUID Partition Tables. +# options GEOM_RAID # Soft RAID functionality. +options GEOM_LABEL # Provides labelization +options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI +options KTRACE # ktrace(1) support +# options STACK # stack(9) support +options SYSVSHM # SYSV-style shared memory +options SYSVMSG # SYSV-style message queues +options SYSVSEM # SYSV-style semaphores +options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions +options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. +options KBD_INSTALL_CDEV # install a CDEV entry in /dev +# options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) +options AUDIT # Security event auditing +options CAPABILITY_MODE # Capsicum capability mode +options CAPABILITIES # Capsicum capabilities +options MAC # TrustedBSD MAC Framework +# options KDTRACE_FRAME # Ensure frames are compiled in +# options KDTRACE_HOOKS # Kernel DTrace hooks +# options VFP # Floating-point support +options RACCT # Resource accounting framework +options RACCT_DEFAULT_TO_DISABLED # Set kern.racct.enable=0 by default +options RCTL # Resource limits +# options SMP + +# Debugging support. Always need this: +# options KDB # Enable kernel debugger support. +# options KDB_TRACE # Print a stack trace for a panic. +# For full debugger support use (turn off in stable branch): +# options DDB # Support DDB. +# options GDB # Support remote GDB. +options DEADLKRES # Enable the deadlock resolver +options INVARIANTS # Enable calls of extra sanity checking +options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS +# options WITNESS # Enable checks to detect deadlocks and cycles +# options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed +options MALLOC_DEBUG_MAXZONES=8 # Separate malloc(9) zones + +options ROOTDEVNAME=\"ufs:/dev/htif_blk0\" +# options EARLY_PRINTF + +# Pseudo devices. +device loop # Network loopback +device random # Entropy device +device ether # Ethernet support +device vlan # 802.1Q VLAN support +device tun # Packet tunnel. +device md # Memory "disks" +device gif # IPv6 and IPv4 tunneling +device firmware # firmware assist module + +# RISCVTODO: This needs to be done via loader (when it's available). +options FDT +options FDT_DTB_STATIC +makeoptions FDT_DTS_FILE=spike.dts Property changes on: user/ngie/make_check/sys/riscv/conf/GENERIC ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/make_check/sys/riscv/htif/htif.c =================================================================== --- user/ngie/make_check/sys/riscv/htif/htif.c (nonexistent) +++ user/ngie/make_check/sys/riscv/htif/htif.c (revision 295078) @@ -0,0 +1,284 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "htif.h" + +static struct resource_spec htif_spec[] = { + { SYS_RES_IRQ, 0, RF_ACTIVE }, + { -1, 0 } +}; + +struct intr_entry { + void (*func) (void *, uint64_t); + void *arg; +}; + +struct intr_entry intrs[HTIF_NDEV]; + +uint64_t +htif_command(uint64_t arg) +{ + + return (machine_command(ECALL_HTIF_CMD, arg)); +} + +int +htif_setup_intr(int id, void *func, void *arg) +{ + + if (id >= HTIF_NDEV) + return (-1); + + intrs[id].func = func; + intrs[id].arg = arg; + + return (0); +} + +static void +htif_handle_entry(struct htif_softc *sc) +{ + uint64_t entry; + uint8_t devcmd; + uint8_t devid; + + entry = machine_command(ECALL_HTIF_GET_ENTRY, 0); + while (entry) { + devid = HTIF_DEV_ID(entry); + devcmd = HTIF_DEV_CMD(entry); + + if (devcmd == HTIF_CMD_IDENTIFY) { + /* Enumeration interrupt */ + if (devid == sc->identify_id) + sc->identify_done = 1; + } else { + /* Device interrupt */ + if (intrs[devid].func != NULL) + intrs[devid].func(intrs[devid].arg, entry); + } + + entry = machine_command(ECALL_HTIF_GET_ENTRY, 0); + } +} + +static int +htif_intr(void *arg) +{ + struct htif_softc *sc; + + sc = arg; + + htif_handle_entry(sc); + + csr_clear(sip, SIE_SSIE); + + return (FILTER_HANDLED); +} + +static int +htif_add_device(struct htif_softc *sc, int i, char *id, char *name) +{ + struct htif_dev_ivars *di; + + di = malloc(sizeof(struct htif_dev_ivars), M_DEVBUF, M_WAITOK | M_ZERO); + di->sc = sc; + di->index = i; + di->id = malloc(HTIF_ID_LEN, M_DEVBUF, M_WAITOK | M_ZERO); + memcpy(di->id, id, HTIF_ID_LEN); + + di->dev = device_add_child(sc->dev, name, -1); + device_set_ivars(di->dev, di); + + return (0); +} + +static int +htif_enumerate(struct htif_softc *sc) +{ + char id[HTIF_ID_LEN] __aligned(HTIF_ALIGN); + uint64_t paddr; + uint64_t data; + uint64_t cmd; + int len; + int i; + + device_printf(sc->dev, "Enumerating devices\n"); + + for (i = 0; i < HTIF_NDEV; i++) { + paddr = pmap_kextract((vm_offset_t)&id); + data = (paddr << IDENTIFY_PADDR_SHIFT); + data |= IDENTIFY_IDENT; + + sc->identify_id = i; + sc->identify_done = 0; + + cmd = i; + cmd <<= HTIF_DEV_ID_SHIFT; + cmd |= (HTIF_CMD_IDENTIFY << HTIF_CMD_SHIFT); + cmd |= data; + + htif_command(cmd); + + /* Do poll as interrupts are disabled yet */ + while (sc->identify_done == 0) { + htif_handle_entry(sc); + } + + len = strnlen(id, sizeof(id)); + if (len <= 0) { + continue; + } + + if (bootverbose) + printf(" %d %s\n", i, id); + + if (strncmp(id, "disk", 4) == 0) + htif_add_device(sc, i, id, "htif_blk"); + else if (strncmp(id, "bcd", 3) == 0) + htif_add_device(sc, i, id, "htif_console"); + else if (strncmp(id, "syscall_proxy", 13) == 0) + htif_add_device(sc, i, id, "htif_syscall_proxy"); + } + + return (bus_generic_attach(sc->dev)); +} + +int +htif_read_ivar(device_t dev, device_t child, int which, uintptr_t *result) +{ + struct htif_dev_ivars *ivars; + + ivars = device_get_ivars(child); + + switch (which) { + case HTIF_IVAR_INDEX: + *result = ivars->index; + break; + case HTIF_IVAR_ID: + *result = (uintptr_t)ivars->id; + default: + return (EINVAL); + } + + return (0); +} + +static int +htif_probe(device_t dev) +{ + + if (!ofw_bus_status_okay(dev)) + return (ENXIO); + + if (!ofw_bus_is_compatible(dev, "riscv,htif")) + return (ENXIO); + + device_set_desc(dev, "HTIF bus device"); + return (BUS_PROBE_DEFAULT); +} + +static int +htif_attach(device_t dev) +{ + struct htif_softc *sc; + int error; + + sc = device_get_softc(dev); + sc->dev = dev; + + if (bus_alloc_resources(dev, htif_spec, sc->res)) { + device_printf(dev, "could not allocate resources\n"); + return (ENXIO); + } + + /* Setup IRQs handler */ + error = bus_setup_intr(dev, sc->res[0], INTR_TYPE_CLK, + htif_intr, NULL, sc, &sc->ihl[0]); + if (error) { + device_printf(dev, "Unable to alloc int resource.\n"); + return (ENXIO); + } + + csr_set(sie, SIE_SSIE); + + return (htif_enumerate(sc)); +} + +static device_method_t htif_methods[] = { + DEVMETHOD(device_probe, htif_probe), + DEVMETHOD(device_attach, htif_attach), + + /* Bus interface */ + DEVMETHOD(bus_read_ivar, htif_read_ivar), + + DEVMETHOD_END +}; + +static driver_t htif_driver = { + "htif", + htif_methods, + sizeof(struct htif_softc) +}; + +static devclass_t htif_devclass; + +DRIVER_MODULE(htif, simplebus, htif_driver, + htif_devclass, 0, 0); Property changes on: user/ngie/make_check/sys/riscv/htif/htif.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/htif/htif.h =================================================================== --- user/ngie/make_check/sys/riscv/htif/htif.h (nonexistent) +++ user/ngie/make_check/sys/riscv/htif/htif.h (revision 295078) @@ -0,0 +1,93 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#define HTIF_DEV_ID_SHIFT (56) +#define HTIF_DEV_ID_MASK (0xfful << HTIF_DEV_ID_SHIFT) +#define HTIF_CMD_SHIFT (48) +#define HTIF_CMD_MASK (0xfful << HTIF_CMD_SHIFT) +#define HTIF_DATA_SHIFT (0) +#define HTIF_DATA_MASK (0xffffffff << HTIF_DATA_SHIFT) + +#define HTIF_CMD_READ (0x00ul) +#define HTIF_CMD_WRITE (0x01ul) +#define HTIF_CMD_READ_CONTROL_REG (0x02ul) +#define HTIF_CMD_WRITE_CONTROL_REG (0x03ul) +#define HTIF_CMD_IDENTIFY (0xfful) +#define IDENTIFY_PADDR_SHIFT 8 +#define IDENTIFY_IDENT 0xff + +#define HTIF_NDEV (256) +#define HTIF_ID_LEN (64) +#define HTIF_ALIGN (64) + +#define HTIF_DEV_CMD(entry) ((entry & HTIF_CMD_MASK) >> HTIF_CMD_SHIFT) +#define HTIF_DEV_ID(entry) ((entry & HTIF_DEV_ID_MASK) >> HTIF_DEV_ID_SHIFT) +#define HTIF_DEV_DATA(entry) ((entry & HTIF_DATA_MASK) >> HTIF_DATA_SHIFT) + +/* bus softc */ +struct htif_softc { + struct resource *res[1]; + void *ihl[1]; + device_t dev; + uint64_t identify_id; + uint64_t identify_done; +}; + +/* device private data */ +struct htif_dev_ivars { + char *id; + int index; + device_t dev; + struct htif_softc *sc; +}; + +uint64_t htif_command(uint64_t); +int htif_setup_intr(int id, void *func, void *arg); +int htif_read_ivar(device_t dev, device_t child, int which, uintptr_t *result); + +enum htif_device_ivars { + HTIF_IVAR_INDEX, + HTIF_IVAR_ID, +}; + +/* + * Simplified accessors for HTIF devices + */ +#define HTIF_ACCESSOR(var, ivar, type) \ + __BUS_ACCESSOR(htif, var, HTIF, ivar, type) + +HTIF_ACCESSOR(index, INDEX, int); +HTIF_ACCESSOR(id, ID, char *); Property changes on: user/ngie/make_check/sys/riscv/htif/htif.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/htif/htif_block.c =================================================================== --- user/ngie/make_check/sys/riscv/htif/htif_block.c (nonexistent) +++ user/ngie/make_check/sys/riscv/htif/htif_block.c (revision 295078) @@ -0,0 +1,289 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include + +#include "htif.h" + +#define SECTOR_SIZE_SHIFT (9) +#define SECTOR_SIZE (1 << SECTOR_SIZE_SHIFT) + +#define HTIF_BLK_LOCK(_sc) mtx_lock(&(_sc)->sc_mtx) +#define HTIF_BLK_UNLOCK(_sc) mtx_unlock(&(_sc)->sc_mtx) +#define HTIF_BLK_LOCK_INIT(_sc) \ + mtx_init(&_sc->sc_mtx, device_get_nameunit(_sc->dev), \ + "htif_blk", MTX_DEF) +#define HTIF_BLK_LOCK_DESTROY(_sc) mtx_destroy(&_sc->sc_mtx); +#define HTIF_BLK_ASSERT_LOCKED(_sc) mtx_assert(&_sc->sc_mtx, MA_OWNED); +#define HTIF_BLK_ASSERT_UNLOCKED(_sc) mtx_assert(&_sc->sc_mtx, MA_NOTOWNED); + +static void htif_blk_task(void *arg); + +static disk_open_t htif_blk_open; +static disk_close_t htif_blk_close; +static disk_strategy_t htif_blk_strategy; + +struct htif_blk_softc { + device_t dev; + struct disk *disk; + struct mtx htif_io_mtx; + struct mtx sc_mtx; + struct proc *p; + struct bio_queue_head bio_queue; + int running; + int intr_chan; + int cmd_done; + int index; + uint16_t curtag; +}; + +struct htif_blk_request { + uint64_t addr; + uint64_t offset; /* offset in bytes */ + uint64_t size; /* length in bytes */ + uint64_t tag; +}; + +static void +htif_blk_intr(void *arg, uint64_t entry) +{ + struct htif_blk_softc *sc; + uint64_t devcmd; + uint64_t data; + + sc = arg; + + devcmd = HTIF_DEV_CMD(entry); + data = HTIF_DEV_DATA(entry); + + if (sc->curtag == data) { + sc->cmd_done = 1; + wakeup(&sc->intr_chan); + } +} + +static int +htif_blk_probe(device_t dev) +{ + + return (0); +} + +static int +htif_blk_attach(device_t dev) +{ + struct htif_blk_softc *sc; + char prefix[] = " size="; + char *str; + long size; + + sc = device_get_softc(dev); + sc->dev = dev; + + mtx_init(&sc->htif_io_mtx, device_get_nameunit(dev), "htif_blk", MTX_DEF); + HTIF_BLK_LOCK_INIT(sc); + + str = strstr(htif_get_id(dev), prefix); + + size = strtol((str + 6), NULL, 10); + if (size == 0) { + return (ENXIO); + } + + sc->index = htif_get_index(dev); + if (sc->index < 0) + return (EINVAL); + htif_setup_intr(sc->index, htif_blk_intr, sc); + + sc->disk = disk_alloc(); + sc->disk->d_drv1 = sc; + + sc->disk->d_maxsize = 4096; /* Max transfer */ + sc->disk->d_name = "htif_blk"; + sc->disk->d_open = htif_blk_open; + sc->disk->d_close = htif_blk_close; + sc->disk->d_strategy = htif_blk_strategy; + sc->disk->d_unit = 0; + sc->disk->d_sectorsize = SECTOR_SIZE; + sc->disk->d_mediasize = size; + disk_create(sc->disk, DISK_VERSION); + + bioq_init(&sc->bio_queue); + + sc->running = 1; + + kproc_create(&htif_blk_task, sc, &sc->p, 0, 0, "%s: transfer", + device_get_nameunit(dev)); + + return (0); +} + +static int +htif_blk_open(struct disk *dp) +{ + + return (0); +} + +static int +htif_blk_close(struct disk *dp) +{ + + return (0); +} + +static void +htif_blk_task(void *arg) +{ + struct htif_blk_request req __aligned(HTIF_ALIGN); + struct htif_blk_softc *sc; + struct bio *bp; + uint64_t paddr; + uint64_t cmd; + int i; + + sc = (struct htif_blk_softc *)arg; + + while (1) { + HTIF_BLK_LOCK(sc); + do { + bp = bioq_takefirst(&sc->bio_queue); + if (bp == NULL) + msleep(sc, &sc->sc_mtx, PRIBIO, "jobqueue", 0); + } while (bp == NULL); + HTIF_BLK_UNLOCK(sc); + + if (bp->bio_cmd == BIO_READ || bp->bio_cmd == BIO_WRITE) { + req.offset = (bp->bio_pblkno * sc->disk->d_sectorsize); + req.size = bp->bio_bcount; + paddr = vtophys(bp->bio_data); + KASSERT(paddr != 0, ("paddr is 0")); + req.addr = paddr; + req.tag = sc->curtag; + + cmd = sc->index; + cmd <<= HTIF_DEV_ID_SHIFT; + if (bp->bio_cmd == BIO_READ) + cmd |= (HTIF_CMD_READ << HTIF_CMD_SHIFT); + else + cmd |= (HTIF_CMD_WRITE << HTIF_CMD_SHIFT); + paddr = vtophys(&req); + KASSERT(paddr != 0, ("paddr is 0")); + cmd |= paddr; + + sc->cmd_done = 0; + htif_command(cmd); + + /* Wait for interrupt */ + HTIF_BLK_LOCK(sc); + i = 0; + while (sc->cmd_done == 0) { + msleep(&sc->intr_chan, &sc->sc_mtx, PRIBIO, "intr", hz/2); + + if (i++ > 2) { + /* TODO: try to re-issue operation on timeout ? */ + bp->bio_error = EIO; + bp->bio_flags |= BIO_ERROR; + disk_err(bp, "hard error", -1, 1); + break; + } + } + HTIF_BLK_UNLOCK(sc); + + biodone(bp); + } else { + printf("unknown op %d\n", bp->bio_cmd); + } + } +} + +static void +htif_blk_strategy(struct bio *bp) +{ + struct htif_blk_softc *sc; + + sc = bp->bio_disk->d_drv1; + + HTIF_BLK_LOCK(sc); + if (sc->running > 0) { + bioq_disksort(&sc->bio_queue, bp); + HTIF_BLK_UNLOCK(sc); + wakeup(sc); + } else { + HTIF_BLK_UNLOCK(sc); + biofinish(bp, NULL, ENXIO); + } +} + +static device_method_t htif_blk_methods[] = { + DEVMETHOD(device_probe, htif_blk_probe), + DEVMETHOD(device_attach, htif_blk_attach), +}; + +static driver_t htif_blk_driver = { + "htif_blk", + htif_blk_methods, + sizeof(struct htif_blk_softc) +}; + +static devclass_t htif_blk_devclass; + +DRIVER_MODULE(htif_blk, htif, htif_blk_driver, htif_blk_devclass, 0, 0); Property changes on: user/ngie/make_check/sys/riscv/htif/htif_block.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/htif/htif_console.c =================================================================== --- user/ngie/make_check/sys/riscv/htif/htif_console.c (nonexistent) +++ user/ngie/make_check/sys/riscv/htif/htif_console.c (revision 295078) @@ -0,0 +1,361 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "htif.h" + +#include + +#include + +extern uint64_t console_intr; + +static tsw_outwakeup_t riscvtty_outwakeup; + +static struct ttydevsw riscv_ttydevsw = { + .tsw_flags = TF_NOPREFIX, + .tsw_outwakeup = riscvtty_outwakeup, +}; + +static int polltime; +static struct callout riscv_callout; +static struct tty *tp = NULL; + +#if defined(KDB) +static int alt_break_state; +#endif + +static void riscv_timeout(void *); + +static cn_probe_t riscv_cnprobe; +static cn_init_t riscv_cninit; +static cn_term_t riscv_cnterm; +static cn_getc_t riscv_cngetc; +static cn_putc_t riscv_cnputc; +static cn_grab_t riscv_cngrab; +static cn_ungrab_t riscv_cnungrab; + +CONSOLE_DRIVER(riscv); + +#define MAX_BURST_LEN 1 +#define QUEUE_SIZE 256 +#define CONSOLE_DEFAULT_ID 1ul + +struct queue_entry { + uint64_t data; + uint64_t used; + struct queue_entry *next; +}; + +struct queue_entry cnqueue[QUEUE_SIZE]; +struct queue_entry *entry_last; +struct queue_entry *entry_served; + +static void +htif_putc(int c) +{ + uint64_t cmd; + + cmd = (HTIF_CMD_WRITE << HTIF_CMD_SHIFT); + cmd |= (CONSOLE_DEFAULT_ID << HTIF_DEV_ID_SHIFT); + cmd |= c; + + htif_command(cmd); +} + +static uint8_t +htif_getc(void) +{ + uint64_t cmd; + uint8_t res; + + cmd = (HTIF_CMD_READ << HTIF_CMD_SHIFT); + cmd |= (CONSOLE_DEFAULT_ID << HTIF_DEV_ID_SHIFT); + + res = htif_command(cmd); + + return (res); +} + +static void +riscv_putc(int c) +{ + uint64_t counter; + uint64_t *cc; + uint64_t val; + + val = 0; + counter = 0; + + cc = (uint64_t*)&console_intr; + *cc = 0; + + htif_putc(c); + + /* Wait for an interrupt */ + __asm __volatile( + "li %0, 1\n" /* counter = 1 */ + "slli %0, %0, 12\n" /* counter <<= 12 */ + "1:" + "addi %0, %0, -1\n" /* counter -= 1 */ + "beqz %0, 2f\n" /* counter == 0 ? finish */ + "ld %1, 0(%2)\n" /* val = *cc */ + "beqz %1, 1b\n" /* val == 0 ? repeat */ + "2:" + : "=&r"(counter), "=&r"(val) : "r"(cc) + ); +} + +#ifdef EARLY_PRINTF +early_putc_t *early_putc = riscv_putc; +#endif + +static void +cn_drvinit(void *unused) +{ + + if (riscv_consdev.cn_pri != CN_DEAD && + riscv_consdev.cn_name[0] != '\0') { + tp = tty_alloc(&riscv_ttydevsw, NULL); + tty_init_console(tp, 0); + tty_makedev(tp, NULL, "%s", "rcons"); + + polltime = 1; + + callout_init(&riscv_callout, 1); + callout_reset(&riscv_callout, polltime, riscv_timeout, NULL); + } +} + +SYSINIT(cndev, SI_SUB_CONFIGURE, SI_ORDER_MIDDLE, cn_drvinit, NULL); + +static void +riscvtty_outwakeup(struct tty *tp) +{ + u_char buf[MAX_BURST_LEN]; + int len; + int i; + + for (;;) { + len = ttydisc_getc(tp, buf, sizeof(buf)); + if (len == 0) + break; + + KASSERT(len == 1, ("tty error")); + + for (i = 0; i < len; i++) + riscv_putc(buf[i]); + } +} + +static void +riscv_timeout(void *v) +{ + int c; + + tty_lock(tp); + while ((c = riscv_cngetc(NULL)) != -1) + ttydisc_rint(tp, c, 0); + ttydisc_rint_done(tp); + tty_unlock(tp); + + callout_reset(&riscv_callout, polltime, riscv_timeout, NULL); +} + +static void +riscv_cnprobe(struct consdev *cp) +{ + + cp->cn_pri = CN_NORMAL; +} + +static void +riscv_cninit(struct consdev *cp) +{ + int i; + + strcpy(cp->cn_name, "rcons"); + + for (i = 0; i < QUEUE_SIZE; i++) { + if (i == (QUEUE_SIZE - 1)) + cnqueue[i].next = &cnqueue[0]; + else + cnqueue[i].next = &cnqueue[i+1]; + cnqueue[i].data = 0; + cnqueue[i].used = 0; + } + + entry_last = &cnqueue[0]; + entry_served = &cnqueue[0]; +} + +static void +riscv_cnterm(struct consdev *cp) +{ + +} + +static void +riscv_cngrab(struct consdev *cp) +{ + +} + +static void +riscv_cnungrab(struct consdev *cp) +{ + +} + +static int +riscv_cngetc(struct consdev *cp) +{ + uint8_t data; + int ch; + + ch = htif_getc(); + + if (entry_served->used == 1) { + data = entry_served->data; + entry_served->used = 0; + entry_served = entry_served->next; + ch = (data & 0xff); + if (ch > 0 && ch < 0xff) { +#if defined(KDB) + kdb_alt_break(ch, &alt_break_state); +#endif + return (ch); + } + } + + return (-1); +} + +static void +riscv_cnputc(struct consdev *cp, int c) +{ + + riscv_putc(c); +} + +/* + * Bus interface. + */ + +struct htif_console_softc { + device_t dev; + int running; + int intr_chan; + int cmd_done; + int curtag; + int index; +}; + +static void +htif_console_intr(void *arg, uint64_t entry) +{ + struct htif_console_softc *sc; + uint8_t devcmd; + uint64_t data; + + sc = arg; + + devcmd = HTIF_DEV_CMD(entry); + data = HTIF_DEV_DATA(entry); + + if (devcmd == 0) { + entry_last->data = data; + entry_last->used = 1; + entry_last = entry_last->next; + } +} + +static int +htif_console_probe(device_t dev) +{ + + return (0); +} + +static int +htif_console_attach(device_t dev) +{ + struct htif_console_softc *sc; + + sc = device_get_softc(dev); + sc->dev = dev; + + sc->index = htif_get_index(dev); + if (sc->index < 0) + return (EINVAL); + + htif_setup_intr(sc->index, htif_console_intr, sc); + + return (0); +} + +static device_method_t htif_console_methods[] = { + DEVMETHOD(device_probe, htif_console_probe), + DEVMETHOD(device_attach, htif_console_attach), + DEVMETHOD_END +}; + +static driver_t htif_console_driver = { + "htif_console", + htif_console_methods, + sizeof(struct htif_console_softc) +}; + +static devclass_t htif_console_devclass; + +DRIVER_MODULE(htif_console, htif, htif_console_driver, + htif_console_devclass, 0, 0); Property changes on: user/ngie/make_check/sys/riscv/htif/htif_console.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/autoconf.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/autoconf.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/autoconf.c (revision 295078) @@ -0,0 +1,94 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +/* + * Setup the system to run on the current machine. + * + * Configure() is called at boot time and initializes the vba + * device tables and the memory controller monitoring. Available + * devices are determined (from possibilities mentioned in ioconf.c), + * and the drivers are initialized. + */ + +#include +#include +#include +#include +#include + +static void configure_first(void *); +static void configure(void *); +static void configure_final(void *); + +SYSINIT(configure1, SI_SUB_CONFIGURE, SI_ORDER_FIRST, configure_first, NULL); +/* SI_ORDER_SECOND is hookable */ +SYSINIT(configure2, SI_SUB_CONFIGURE, SI_ORDER_THIRD, configure, NULL); +/* SI_ORDER_MIDDLE is hookable */ +SYSINIT(configure3, SI_SUB_CONFIGURE, SI_ORDER_ANY, configure_final, NULL); + +/* + * Determine i/o configuration for a machine. + */ +static void +configure_first(void *dummy) +{ + + /* nexus0 is the top of the riscv device tree */ + device_add_child(root_bus, "nexus", 0); +} + +static void +configure(void *dummy) +{ + + /* initialize new bus architecture */ + root_bus_configure(); +} + +static void +configure_final(void *dummy) +{ + + intr_enable(); + + cninit_finish(); + + if (bootverbose) + printf("Device configuration finished.\n"); + + cold = 0; +} Property changes on: user/ngie/make_check/sys/riscv/riscv/autoconf.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/bcopy.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/bcopy.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/bcopy.c (revision 295078) @@ -0,0 +1,139 @@ +/*- + * Copyright (c) 1990 The Regents of the University of California. + * All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * Chris Torek. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * From: sys/powerpc/powerpc/bcopy.c + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +/* + * sizeof(word) MUST BE A POWER OF TWO + * SO THAT wmask BELOW IS ALL ONES + */ +typedef long word; /* "word" used for optimal copy speed */ + +#define wsize sizeof(word) +#define wmask (wsize - 1) + +/* + * Copy a block of memory, handling overlap. + * This is the routine that actually implements + * (the portable versions of) bcopy, memcpy, and memmove. + */ +void * +memcpy(void *dst0, const void *src0, size_t length) +{ + char *dst; + const char *src; + size_t t; + + dst = dst0; + src = src0; + + if (length == 0 || dst == src) { /* nothing to do */ + goto done; + } + + /* + * Macros: loop-t-times; and loop-t-times, t>0 + */ +#define TLOOP(s) if (t) TLOOP1(s) +#define TLOOP1(s) do { s; } while (--t) + + if ((unsigned long)dst < (unsigned long)src) { + /* + * Copy forward. + */ + t = (size_t)src; /* only need low bits */ + + if ((t | (uintptr_t)dst) & wmask) { + /* + * Try to align operands. This cannot be done + * unless the low bits match. + */ + if ((t ^ (uintptr_t)dst) & wmask || length < wsize) { + t = length; + } else { + t = wsize - (t & wmask); + } + + length -= t; + TLOOP1(*dst++ = *src++); + } + /* + * Copy whole words, then mop up any trailing bytes. + */ + t = length / wsize; + TLOOP(*(word *)dst = *(const word *)src; src += wsize; + dst += wsize); + t = length & wmask; + TLOOP(*dst++ = *src++); + } else { + /* + * Copy backwards. Otherwise essentially the same. + * Alignment works as before, except that it takes + * (t&wmask) bytes to align, not wsize-(t&wmask). + */ + src += length; + dst += length; + t = (uintptr_t)src; + + if ((t | (uintptr_t)dst) & wmask) { + if ((t ^ (uintptr_t)dst) & wmask || length <= wsize) { + t = length; + } else { + t &= wmask; + } + + length -= t; + TLOOP1(*--dst = *--src); + } + t = length / wsize; + TLOOP(src -= wsize; dst -= wsize; + *(word *)dst = *(const word *)src); + t = length & wmask; + TLOOP(*--dst = *--src); + } +done: + return (dst0); +} + +void +bcopy(const void *src0, void *dst0, size_t length) +{ + + memcpy(dst0, src0, length); +} + Property changes on: user/ngie/make_check/sys/riscv/riscv/bcopy.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/bus_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/bus_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/bus_machdep.c (revision 295078) @@ -0,0 +1,144 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include "opt_platform.h" + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +#include + +struct bus_space memmap_bus = { + /* cookie */ + .bs_cookie = NULL, + + /* mapping/unmapping */ + .bs_map = NULL, + .bs_unmap = NULL, + .bs_subregion = NULL, + + /* allocation/deallocation */ + .bs_alloc = NULL, + .bs_free = NULL, + + /* barrier */ + .bs_barrier = NULL, + + /* read single */ + .bs_r_1 = NULL, + .bs_r_2 = NULL, + .bs_r_4 = NULL, + .bs_r_8 = NULL, + + /* read multiple */ + .bs_rm_1 = NULL, + .bs_rm_2 = NULL, + .bs_rm_4 = NULL, + .bs_rm_8 = NULL, + + /* write single */ + .bs_w_1 = NULL, + .bs_w_2 = NULL, + .bs_w_4 = NULL, + .bs_w_8 = NULL, + + /* write multiple */ + .bs_wm_1 = NULL, + .bs_wm_2 = NULL, + .bs_wm_4 = NULL, + .bs_wm_8 = NULL, + + /* write region */ + .bs_wr_1 = NULL, + .bs_wr_2 = NULL, + .bs_wr_4 = NULL, + .bs_wr_8 = NULL, + + /* set multiple */ + .bs_sm_1 = NULL, + .bs_sm_2 = NULL, + .bs_sm_4 = NULL, + .bs_sm_8 = NULL, + + /* set region */ + .bs_sr_1 = NULL, + .bs_sr_2 = NULL, + .bs_sr_4 = NULL, + .bs_sr_8 = NULL, + + /* copy */ + .bs_c_1 = NULL, + .bs_c_2 = NULL, + .bs_c_4 = NULL, + .bs_c_8 = NULL, + + /* read single stream */ + .bs_r_1_s = NULL, + .bs_r_2_s = NULL, + .bs_r_4_s = NULL, + .bs_r_8_s = NULL, + + /* read multiple stream */ + .bs_rm_1_s = NULL, + .bs_rm_2_s = NULL, + .bs_rm_4_s = NULL, + .bs_rm_8_s = NULL, + + /* read region stream */ + .bs_rr_1_s = NULL, + .bs_rr_2_s = NULL, + .bs_rr_4_s = NULL, + .bs_rr_8_s = NULL, + + /* write single stream */ + .bs_w_1_s = NULL, + .bs_w_2_s = NULL, + .bs_w_4_s = NULL, + .bs_w_8_s = NULL, + + /* write multiple stream */ + .bs_wm_1_s = NULL, + .bs_wm_2_s = NULL, + .bs_wm_4_s = NULL, + .bs_wm_8_s = NULL, + + /* write region stream */ + .bs_wr_1_s = NULL, + .bs_wr_2_s = NULL, + .bs_wr_4_s = NULL, + .bs_wr_8_s = NULL, +}; Property changes on: user/ngie/make_check/sys/riscv/riscv/bus_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/busdma_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/busdma_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/busdma_machdep.c (revision 295078) @@ -0,0 +1,102 @@ +/*- + * Copyright (c) 1997, 1998 Justin T. Gibbs. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions, and the following disclaimer, + * without modification, immediately at the beginning of the file. + * 2. The name of the author may not be used to endorse or promote products + * derived from this software without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR + * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +int +_bus_dmamap_load_phys(bus_dma_tag_t dmat, bus_dmamap_t map, vm_paddr_t buf, + bus_size_t buflen, int flags, bus_dma_segment_t *segs, int *segp) +{ + + panic("busdma"); +} + +int +_bus_dmamap_load_ma(bus_dma_tag_t dmat, bus_dmamap_t map, struct vm_page **ma, + bus_size_t tlen, int ma_offs, int flags, bus_dma_segment_t *segs, + int *segp) +{ + + panic("busdma"); +} + +int +_bus_dmamap_load_buffer(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf, + bus_size_t buflen, pmap_t pmap, int flags, bus_dma_segment_t *segs, + int *segp) +{ + + panic("busdma"); +} + +void +__bus_dmamap_waitok(bus_dma_tag_t dmat, bus_dmamap_t map, + struct memdesc *mem, bus_dmamap_callback_t *callback, void *callback_arg) +{ + + panic("busdma"); +} + +bus_dma_segment_t * +_bus_dmamap_complete(bus_dma_tag_t dmat, bus_dmamap_t map, + bus_dma_segment_t *segs, int nsegs, int error) +{ + + panic("busdma"); +} + +/* + * Release the mapping held by map. + */ +void +_bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map) +{ + + panic("busdma"); +} + +void +_bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op) +{ + + panic("busdma"); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/busdma_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/clock.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/clock.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/clock.c (revision 295078) @@ -0,0 +1,46 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +void +cpu_initclocks(void) +{ + + cpu_initclocks_bsp(); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/clock.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/copyinout.S =================================================================== --- user/ngie/make_check/sys/riscv/riscv/copyinout.S (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/copyinout.S (revision 295078) @@ -0,0 +1,137 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include + +#include "assym.s" + +/* + * Fault handler for the copy{in,out} functions below. + */ +ENTRY(copyio_fault) + SET_FAULT_HANDLER(x0, a1) /* Clear the handler */ +copyio_fault_nopcb: + li a0, EFAULT + ret +END(copyio_fault) + +/* + * Copies from a kernel to user address + * + * int copyout(const void *kaddr, void *udaddr, size_t len) + */ +ENTRY(copyout) + beqz a2, 2f /* If len == 0 then skip loop */ + add a3, a1, a2 + li a4, VM_MAXUSER_ADDRESS + bgt a3, a4, copyio_fault_nopcb + + la a6, copyio_fault /* Get the handler address */ + SET_FAULT_HANDLER(a6, a7) /* Set the handler */ + +1: lb a4, 0(a0) /* Load from kaddr */ + addi a0, a0, 1 + sb a4, 0(a1) /* Store in uaddr */ + addi a1, a1, 1 + addi a2, a2, -1 /* len-- */ + bnez a2, 1b + + SET_FAULT_HANDLER(x0, a7) /* Clear the handler */ + +2: li a0, 0 /* return 0 */ + ret +END(copyout) + +/* + * Copies from a user to kernel address + * + * int copyin(const void *uaddr, void *kdaddr, size_t len) + */ +ENTRY(copyin) + beqz a2, 2f /* If len == 0 then skip loop */ + add a3, a0, a2 + li a4, VM_MAXUSER_ADDRESS + bgt a3, a4, copyio_fault_nopcb + + la a6, copyio_fault /* Get the handler address */ + SET_FAULT_HANDLER(a6, a7) /* Set the handler */ + +1: lb a4, 0(a0) /* Load from uaddr */ + addi a0, a0, 1 + sb a4, 0(a1) /* Store in kaddr */ + addi a1, a1, 1 + addi a2, a2, -1 /* len-- */ + bnez a2, 1b + + SET_FAULT_HANDLER(x0, a7) /* Clear the handler */ + +2: li a0, 0 /* return 0 */ + ret +END(copyin) + +/* + * Copies a string from a user to kernel address + * + * int copyinstr(const void *udaddr, void *kaddr, size_t len, size_t *done) + */ +ENTRY(copyinstr) + mv a5, x0 /* count = 0 */ + beqz a2, 3f /* If len == 0 then skip loop */ + li a7, VM_MAXUSER_ADDRESS + + la a6, copyio_fault /* Get the handler address */ + SET_FAULT_HANDLER(a6, a7) /* Set the handler */ + +1: bgt a7, a0, copyio_fault + lb a4, 0(a0) /* Load from uaddr */ + addi a0, a0, 1 + sb a4, 0(a1) /* Store in kaddr */ + addi a1, a1, 1 + beqz a4, 2f + addi a2, a2, -1 /* len-- */ + addi a5, a5, 1 /* count++ */ + bnez a2, 1b + +2: SET_FAULT_HANDLER(x0, a7) /* Clear the handler */ + +3: beqz a3, 4f /* Check if done != NULL */ + addi a5, a5, 1 /* count++ */ + sd a5, 0(a3) /* done = count */ + +4: mv a0, x0 /* return 0 */ + ret +END(copyinstr) Property changes on: user/ngie/make_check/sys/riscv/riscv/copyinout.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/copystr.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/copystr.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/copystr.c (revision 295078) @@ -0,0 +1,59 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +int +copystr(const void * __restrict kfaddr, void * __restrict kdaddr, size_t len, + size_t * __restrict lencopied) +{ + const char *src; + size_t pos; + char *dst; + int error; + + error = ENAMETOOLONG; + src = kfaddr; + dst = kdaddr; + for (pos = 0; pos < len; pos++) { + dst[pos] = src[pos]; + if (src[pos] == '\0') { + /* Increment pos to hold the number of bytes copied */ + pos++; + error = 0; + break; + } + } + + if (lencopied != NULL) + *lencopied = pos; + + return (error); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/copystr.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/cpufunc_asm.S =================================================================== --- user/ngie/make_check/sys/riscv/riscv/cpufunc_asm.S (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/cpufunc_asm.S (revision 295078) @@ -0,0 +1,101 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +#include +__FBSDID("$FreeBSD$"); + + .text + .align 2 + +.Lpage_mask: + .word PAGE_MASK + +ENTRY(riscv_nullop) + ret +END(riscv_nullop) + +/* + * Generic functions to read/modify/write the internal coprocessor registers + */ + +ENTRY(riscv_tlb_flushID) + sfence.vm + ret +END(riscv_tlb_flushID) + +ENTRY(riscv_tlb_flushID_SE) + sfence.vm + ret +END(riscv_tlb_flushID_SE) + +/* + * void riscv_dcache_wb_range(vm_offset_t, vm_size_t) + */ +ENTRY(riscv_dcache_wb_range) + /* RISCVTODO */ + ret +END(riscv_dcache_wb_range) + +/* + * void riscv_dcache_wbinv_range(vm_offset_t, vm_size_t) + */ +ENTRY(riscv_dcache_wbinv_range) + /* RISCVTODO */ + ret +END(riscv_dcache_wbinv_range) + +/* + * void riscv_dcache_inv_range(vm_offset_t, vm_size_t) + */ +ENTRY(riscv_dcache_inv_range) + /* RISCVTODO */ + ret +END(riscv_dcache_inv_range) + +/* + * void riscv_idcache_wbinv_range(vm_offset_t, vm_size_t) + */ +ENTRY(riscv_idcache_wbinv_range) + /* RISCVTODO */ + ret +END(riscv_idcache_wbinv_range) + +/* + * void riscv_icache_sync_range(vm_offset_t, vm_size_t) + */ +ENTRY(riscv_icache_sync_range) + /* RISCVTODO */ + ret +END(riscv_icache_sync_range) Property changes on: user/ngie/make_check/sys/riscv/riscv/cpufunc_asm.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/devmap.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/devmap.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/devmap.c (revision 295078) @@ -0,0 +1,61 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +/* RISC-V doesn't provide memory-mapped devices yet */ + +#include "opt_ddb.h" + +#include +#include +#include +#include +#include +#include +#include + +void * +pmap_mapdev(vm_offset_t pa, vm_size_t size) +{ + + return (NULL); +} + +void +pmap_unmapdev(vm_offset_t va, vm_size_t size) +{ + +} Property changes on: user/ngie/make_check/sys/riscv/riscv/devmap.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/dump_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/dump_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/dump_machdep.c (revision 295078) @@ -0,0 +1,57 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include "opt_watchdog.h" + +#include +#include +#include +#include +#include +#include +#include + +int do_minidump = 1; +SYSCTL_INT(_debug, OID_AUTO, minidump, CTLFLAG_RWTUN, &do_minidump, 0, + "Enable mini crash dumps"); + +void +dumpsys_map_chunk(vm_paddr_t pa, size_t chunk, void **va) +{ + + printf("dumpsys_map_chunk\n"); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/dump_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/elf_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/elf_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/elf_machdep.c (revision 295078) @@ -0,0 +1,169 @@ +/*- + * Copyright 1996-1998 John D. Polstra. + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#include +#include + +struct sysentvec elf64_freebsd_sysvec = { + .sv_size = SYS_MAXSYSCALL, + .sv_table = sysent, + .sv_mask = 0, + .sv_errsize = 0, + .sv_errtbl = NULL, + .sv_transtrap = NULL, + .sv_fixup = __elfN(freebsd_fixup), + .sv_sendsig = sendsig, + .sv_sigcode = sigcode, + .sv_szsigcode = &szsigcode, + .sv_name = "FreeBSD ELF64", + .sv_coredump = __elfN(coredump), + .sv_imgact_try = NULL, + .sv_minsigstksz = MINSIGSTKSZ, + .sv_pagesize = PAGE_SIZE, + .sv_minuser = VM_MIN_ADDRESS, + .sv_maxuser = VM_MAXUSER_ADDRESS, + .sv_usrstack = USRSTACK, + .sv_psstrings = PS_STRINGS, + .sv_stackprot = VM_PROT_ALL, + .sv_copyout_strings = exec_copyout_strings, + .sv_setregs = exec_setregs, + .sv_fixlimit = NULL, + .sv_maxssiz = NULL, + .sv_flags = SV_ABI_FREEBSD | SV_LP64, + .sv_set_syscall_retval = cpu_set_syscall_retval, + .sv_fetch_syscall_args = cpu_fetch_syscall_args, + .sv_syscallnames = syscallnames, + .sv_schedtail = NULL, + .sv_thread_detach = NULL, + .sv_trap = NULL, +}; +INIT_SYSENTVEC(elf64_sysvec, &elf64_freebsd_sysvec); + +static Elf64_Brandinfo freebsd_brand_info = { + .brand = ELFOSABI_FREEBSD, + .machine = EM_RISCV, + .compat_3_brand = "FreeBSD", + .emul_path = NULL, + .interp_path = "/libexec/ld-elf.so.1", + .sysvec = &elf64_freebsd_sysvec, + .interp_newpath = NULL, + .brand_note = &elf64_freebsd_brandnote, + .flags = BI_CAN_EXEC_DYN | BI_BRAND_NOTE +}; + +SYSINIT(elf64, SI_SUB_EXEC, SI_ORDER_FIRST, + (sysinit_cfunc_t) elf64_insert_brand_entry, + &freebsd_brand_info); + +static Elf64_Brandinfo freebsd_brand_oinfo = { + .brand = ELFOSABI_FREEBSD, + .machine = EM_RISCV, + .compat_3_brand = "FreeBSD", + .emul_path = NULL, + .interp_path = "/usr/libexec/ld-elf.so.1", + .sysvec = &elf64_freebsd_sysvec, + .interp_newpath = NULL, + .brand_note = &elf64_freebsd_brandnote, + .flags = BI_CAN_EXEC_DYN | BI_BRAND_NOTE +}; + +SYSINIT(oelf64, SI_SUB_EXEC, SI_ORDER_ANY, + (sysinit_cfunc_t) elf64_insert_brand_entry, + &freebsd_brand_oinfo); + +void +elf64_dump_thread(struct thread *td, void *dst, size_t *off) +{ + +} + +/* Process one elf relocation with addend. */ +static int +elf_reloc_internal(linker_file_t lf, Elf_Addr relocbase, const void *data, + int type, int local, elf_lookup_fn lookup) +{ + + panic("elf_reloc_internal"); +} + +int +elf_reloc(linker_file_t lf, Elf_Addr relocbase, const void *data, int type, + elf_lookup_fn lookup) +{ + + return (elf_reloc_internal(lf, relocbase, data, type, 0, lookup)); +} + +int +elf_reloc_local(linker_file_t lf, Elf_Addr relocbase, const void *data, + int type, elf_lookup_fn lookup) +{ + + return (elf_reloc_internal(lf, relocbase, data, type, 1, lookup)); +} + +int +elf_cpu_load_file(linker_file_t lf __unused) +{ + + return (0); +} + +int +elf_cpu_unload_file(linker_file_t lf __unused) +{ + + return (0); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/elf_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/exception.S =================================================================== --- user/ngie/make_check/sys/riscv/riscv/exception.S (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/exception.S (revision 295078) @@ -0,0 +1,456 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include "assym.s" + +#include +#include + +.macro save_registers el + addi sp, sp, -280 + + sd ra, (TF_RA)(sp) + sd gp, (TF_GP)(sp) + sd tp, (TF_TP)(sp) + + sd t0, (TF_T + 0 * 8)(sp) + sd t1, (TF_T + 1 * 8)(sp) + sd t2, (TF_T + 2 * 8)(sp) + sd t3, (TF_T + 3 * 8)(sp) + sd t4, (TF_T + 4 * 8)(sp) + sd t5, (TF_T + 5 * 8)(sp) + sd t6, (TF_T + 6 * 8)(sp) + + sd s0, (TF_S + 0 * 8)(sp) + sd s1, (TF_S + 1 * 8)(sp) + sd s2, (TF_S + 2 * 8)(sp) + sd s3, (TF_S + 3 * 8)(sp) + sd s4, (TF_S + 4 * 8)(sp) + sd s5, (TF_S + 5 * 8)(sp) + sd s6, (TF_S + 6 * 8)(sp) + sd s7, (TF_S + 7 * 8)(sp) + sd s8, (TF_S + 8 * 8)(sp) + sd s9, (TF_S + 9 * 8)(sp) + sd s10, (TF_S + 10 * 8)(sp) + sd s11, (TF_S + 11 * 8)(sp) + + sd a0, (TF_A + 0 * 8)(sp) + sd a1, (TF_A + 1 * 8)(sp) + sd a2, (TF_A + 2 * 8)(sp) + sd a3, (TF_A + 3 * 8)(sp) + sd a4, (TF_A + 4 * 8)(sp) + sd a5, (TF_A + 5 * 8)(sp) + sd a6, (TF_A + 6 * 8)(sp) + sd a7, (TF_A + 7 * 8)(sp) + +#if 0 + /* XXX: temporary test: spin if stack is not kernel one */ +.if \el == 1 /* kernel */ + mv t0, sp + srli t0, t0, 63 +1: + beqz t0, 1b +.endif +#endif + +.if \el == 1 + /* Store kernel sp */ + sd sp, (TF_SP)(sp) +.else + /* Store user sp */ + csrr t0, sscratch + sd t0, (TF_SP)(sp) +.endif + li t0, 0 + csrw sscratch, t0 + csrr t0, sepc + sd t0, (TF_SEPC)(sp) + csrr t0, sstatus + sd t0, (TF_SSTATUS)(sp) + csrr t0, sbadaddr + sd t0, (TF_SBADADDR)(sp) + csrr t0, scause + sd t0, (TF_SCAUSE)(sp) +.endm + +.macro load_registers el + ld t0, (TF_SSTATUS)(sp) +.if \el == 0 + /* Ensure user interrupts will be enabled on eret. */ + ori t0, t0, SSTATUS_PIE +.else + /* + * Disable interrupts for supervisor mode exceptions. + * For user mode exceptions we have already done this + * in do_ast. + */ + li t1, ~SSTATUS_IE + and t0, t0, t1 +.endif + csrw sstatus, t0 + + ld t0, (TF_SEPC)(sp) + csrw sepc, t0 + +.if \el == 0 + /* Load user sp */ + ld t0, (TF_SP)(sp) + csrw sscratch, t0 +.endif + + ld ra, (TF_RA)(sp) + ld gp, (TF_GP)(sp) + ld tp, (TF_TP)(sp) + + ld t0, (TF_T + 0 * 8)(sp) + ld t1, (TF_T + 1 * 8)(sp) + ld t2, (TF_T + 2 * 8)(sp) + ld t3, (TF_T + 3 * 8)(sp) + ld t4, (TF_T + 4 * 8)(sp) + ld t5, (TF_T + 5 * 8)(sp) + ld t6, (TF_T + 6 * 8)(sp) + + ld s0, (TF_S + 0 * 8)(sp) + ld s1, (TF_S + 1 * 8)(sp) + ld s2, (TF_S + 2 * 8)(sp) + ld s3, (TF_S + 3 * 8)(sp) + ld s4, (TF_S + 4 * 8)(sp) + ld s5, (TF_S + 5 * 8)(sp) + ld s6, (TF_S + 6 * 8)(sp) + ld s7, (TF_S + 7 * 8)(sp) + ld s8, (TF_S + 8 * 8)(sp) + ld s9, (TF_S + 9 * 8)(sp) + ld s10, (TF_S + 10 * 8)(sp) + ld s11, (TF_S + 11 * 8)(sp) + + ld a0, (TF_A + 0 * 8)(sp) + ld a1, (TF_A + 1 * 8)(sp) + ld a2, (TF_A + 2 * 8)(sp) + ld a3, (TF_A + 3 * 8)(sp) + ld a4, (TF_A + 4 * 8)(sp) + ld a5, (TF_A + 5 * 8)(sp) + ld a6, (TF_A + 6 * 8)(sp) + ld a7, (TF_A + 7 * 8)(sp) + + addi sp, sp, 280 +.endm + +.macro do_ast + /* Disable interrupts */ + csrr a4, sstatus +1: + csrci sstatus, SSTATUS_IE + + la a1, pcpup + ld a1, 0(a1) + ld a1, PC_CURTHREAD(a1) + lw a2, TD_FLAGS(a1) + + li a3, (TDF_ASTPENDING|TDF_NEEDRESCHED) + and a2, a2, a3 + beqz a2, 2f + + /* Restore interrupts */ + andi a4, a4, SSTATUS_IE + csrs sstatus, a4 + + /* Handle the ast */ + mv a0, sp + call _C_LABEL(ast) + + /* Re-check for new ast scheduled */ + j 1b +2: +.endm + +ENTRY(cpu_exception_handler_supervisor) + save_registers 1 + mv a0, sp + call _C_LABEL(do_trap_supervisor) + load_registers 1 + eret +END(cpu_exception_handler_supervisor) + +ENTRY(cpu_exception_handler_user) + csrrw sp, sscratch, sp + save_registers 0 + mv a0, sp + call _C_LABEL(do_trap_user) + do_ast + load_registers 0 + csrrw sp, sscratch, sp + eret +END(cpu_exception_handler_user) + +/* + * Trap handlers + */ + .text +bad_trap: + j bad_trap + +user_trap: + csrrw sp, mscratch, sp + addi sp, sp, -64 + sd t0, (8 * 0)(sp) + sd t1, (8 * 1)(sp) + sd t2, (8 * 2)(sp) + sd t3, (8 * 3)(sp) + sd t4, (8 * 4)(sp) + sd t5, (8 * 5)(sp) + sd a0, (8 * 7)(sp) + + la t2, _C_LABEL(cpu_exception_handler_user) + + csrr t0, mcause + bltz t0, machine_interrupt + j exit_mrts + +supervisor_trap: + /* Save state */ + csrrw sp, mscratch, sp + addi sp, sp, -64 + sd t0, (8 * 0)(sp) + sd t1, (8 * 1)(sp) + sd t2, (8 * 2)(sp) + sd t3, (8 * 3)(sp) + sd t4, (8 * 4)(sp) + sd t5, (8 * 5)(sp) + sd a0, (8 * 7)(sp) + + la t2, _C_LABEL(cpu_exception_handler_supervisor) + + csrr t0, mcause + bltz t0, machine_interrupt + + li t1, EXCP_SMODE_ENV_CALL + beq t0, t1, supervisor_call + j exit_mrts + +machine_interrupt: + /* Type of interrupt ? */ + csrr t0, mcause + andi t0, t0, 3 + li t1, 0 + beq t1, t0, software_interrupt + li t1, 1 + beq t1, t0, timer_interrupt + li t1, 2 + beq t1, t0, htif_interrupt + + /* not reached */ +1: + j 1b + +software_interrupt: + /* Redirect to supervisor */ + j exit_mrts + +timer_interrupt: + /* Disable machine timer interrupts */ + li t0, MIE_MTIE + csrc mie, t0 + + /* Clear machine pending */ + li t0, MIP_MTIP + csrc mip, t0 + + /* Post supervisor software interrupt */ + li t0, MIP_STIP + csrs mip, t0 + + /* If PRV1 is PRV_U (user) then serve a trap */ + csrr t0, mstatus + li t1, (MSTATUS_PRV_M << MSTATUS_PRV1_SHIFT) + and t0, t0, t1 + beqz t0, 1f + + /* If PRV1 is supervisor and interrupts were enabled, then serve a trap */ + csrr t0, mstatus + li t1, (SR_IE1 | (MSTATUS_PRV_M << MSTATUS_PRV1_SHIFT)) + and t0, t0, t1 + li t1, (SR_IE1 | (MSTATUS_PRV_S << MSTATUS_PRV1_SHIFT)) + beq t0, t1, 1f + + j exit + +1: + /* Serve a trap in supervisor mode */ + j exit_mrts + +htif_interrupt: +1: + li t5, 0 + csrrw t5, mfromhost, t5 + beqz t5, 3f + + /* Console PUT intr ? */ + mv t1, t5 + li t0, 0x101 + srli t1, t1, 48 + bne t1, t0, 2f + /* Yes */ + la t0, console_intr + li t1, 1 + sd t1, 0(t0) + j 3f + +2: + /* Save entry */ + la t0, htif_ring_cursor + beqz t0, 3f /* not initialized */ + ld t0, 0(t0) /* load struct */ + sd t5, 0(t0) /* put entry */ + li t4, 1 + sd t4, 8(t0) /* mark used */ + ld t4, 16(t0) /* take next */ + /* Update cursor */ + la t0, htif_ring_cursor + sd t4, 0(t0) + + /* Post supervisor software interrupt */ + li t0, MIP_SSIP + csrs mip, t0 + +3: + j exit + +supervisor_call: + csrr t1, mepc + addi t1, t1, 4 /* Next instruction in t1 */ + li t4, ECALL_HTIF_CMD + beq t5, t4, htif_cmd + li t4, ECALL_HTIF_GET_ENTRY + beq t5, t4, htif_get_entry + li t4, ECALL_MTIMECMP + beq t5, t4, set_mtimecmp + li t4, ECALL_CLEAR_PENDING + beq t5, t4, clear_pending + li t4, ECALL_MCPUID_GET + beq t5, t4, mcpuid_get + li t4, ECALL_MIMPID_GET + beq t5, t4, mimpid_get + j exit_next_instr + +mcpuid_get: + csrr t6, mcpuid + j exit_next_instr + +mimpid_get: + csrr t6, mimpid + j exit_next_instr + +htif_get_entry: + li t6, 0 /* preset return value */ + la t0, htif_ring_last + ld t0, 0(t0) /* load struct */ + ld t4, 8(t0) /* get used */ + beqz t4, 1f + ld t6, 0(t0) /* get entry */ + li t4, 0 + sd t4, 8(t0) /* mark free */ + sd t4, 0(t0) /* free entry, just in case */ + ld t4, 16(t0) /* take next */ + la t0, htif_ring_last + sd t4, 0(t0) +1: + /* Exit. Result is stored in t6 */ + j exit_next_instr + +htif_cmd: + mv t0, t6 +1: + csrrw t0, mtohost, t0 + bnez t0, 1b + j exit_next_instr + +set_mtimecmp: + csrr t2, stime + add t6, t6, t2 + csrw mtimecmp, t6 + + /* Enable interrupts */ + li t0, (MIE_MTIE | MIE_STIE) + csrs mie, t0 + j exit_next_instr + +clear_pending: + li t0, MIP_STIP + csrc mip, t0 + j exit_next_instr + +/* + * Trap exit functions + */ +exit_next_instr: + /* Next instruction is in t1 */ + csrw mepc, t1 +exit: + /* Restore state */ + ld t0, (8 * 0)(sp) + ld t1, (8 * 1)(sp) + ld t2, (8 * 2)(sp) + ld t3, (8 * 3)(sp) + ld t4, (8 * 4)(sp) + ld t5, (8 * 5)(sp) + ld a0, (8 * 7)(sp) + addi sp, sp, 64 + csrrw sp, mscratch, sp + eret + +/* + * Redirect to supervisor + */ +exit_mrts: + /* Setup exception handler */ + li t1, KERNBASE + add t2, t2, t1 + csrw stvec, t2 + + /* Restore state */ + ld t0, (8 * 0)(sp) + ld t1, (8 * 1)(sp) + ld t2, (8 * 2)(sp) + ld t3, (8 * 3)(sp) + ld t4, (8 * 4)(sp) + ld t5, (8 * 5)(sp) + ld a0, (8 * 7)(sp) + addi sp, sp, 64 + csrrw sp, mscratch, sp + + /* Redirect to supervisor */ + mrts Property changes on: user/ngie/make_check/sys/riscv/riscv/exception.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/genassym.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/genassym.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/genassym.c (revision 295078) @@ -0,0 +1,98 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +ASSYM(KERNBASE, KERNBASE); +ASSYM(VM_MAXUSER_ADDRESS, VM_MAXUSER_ADDRESS); +ASSYM(TDF_ASTPENDING, TDF_ASTPENDING); +ASSYM(TDF_NEEDRESCHED, TDF_NEEDRESCHED); + +ASSYM(PCB_ONFAULT, offsetof(struct pcb, pcb_onfault)); +ASSYM(PCB_L1ADDR, offsetof(struct pcb, pcb_l1addr)); +ASSYM(PCB_SIZE, sizeof(struct pcb)); +ASSYM(PCB_RA, offsetof(struct pcb, pcb_ra)); +ASSYM(PCB_SP, offsetof(struct pcb, pcb_sp)); +ASSYM(PCB_GP, offsetof(struct pcb, pcb_gp)); +ASSYM(PCB_TP, offsetof(struct pcb, pcb_tp)); +ASSYM(PCB_T, offsetof(struct pcb, pcb_t)); +ASSYM(PCB_S, offsetof(struct pcb, pcb_s)); +ASSYM(PCB_A, offsetof(struct pcb, pcb_a)); + +ASSYM(SF_UC, offsetof(struct sigframe, sf_uc)); + +ASSYM(PC_CURPCB, offsetof(struct pcpu, pc_curpcb)); +ASSYM(PC_CURTHREAD, offsetof(struct pcpu, pc_curthread)); + +ASSYM(TD_PCB, offsetof(struct thread, td_pcb)); +ASSYM(TD_FLAGS, offsetof(struct thread, td_flags)); +ASSYM(TD_PROC, offsetof(struct thread, td_proc)); +ASSYM(TD_FRAME, offsetof(struct thread, td_frame)); +ASSYM(TD_MD, offsetof(struct thread, td_md)); +ASSYM(TD_LOCK, offsetof(struct thread, td_lock)); + +ASSYM(TF_RA, offsetof(struct trapframe, tf_ra)); +ASSYM(TF_SP, offsetof(struct trapframe, tf_sp)); +ASSYM(TF_GP, offsetof(struct trapframe, tf_gp)); +ASSYM(TF_TP, offsetof(struct trapframe, tf_tp)); +ASSYM(TF_T, offsetof(struct trapframe, tf_t)); +ASSYM(TF_S, offsetof(struct trapframe, tf_s)); +ASSYM(TF_A, offsetof(struct trapframe, tf_a)); +ASSYM(TF_SEPC, offsetof(struct trapframe, tf_sepc)); +ASSYM(TF_SBADADDR, offsetof(struct trapframe, tf_sbadaddr)); +ASSYM(TF_SCAUSE, offsetof(struct trapframe, tf_scause)); +ASSYM(TF_SSTATUS, offsetof(struct trapframe, tf_sstatus)); Property changes on: user/ngie/make_check/sys/riscv/riscv/genassym.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/identcpu.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/identcpu.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/identcpu.c (revision 295078) @@ -0,0 +1,149 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include + +#include +#include +#include + +char machine[] = "riscv"; + +SYSCTL_STRING(_hw, HW_MACHINE, machine, CTLFLAG_RD, machine, 0, + "Machine class"); + +struct cpu_desc { + u_int cpu_impl; + u_int cpu_part_num; + const char *cpu_impl_name; + const char *cpu_part_name; +}; + +struct cpu_desc cpu_desc[MAXCPU]; + +struct cpu_parts { + u_int part_id; + const char *part_name; +}; +#define CPU_PART_NONE { -1, "Unknown Processor" } + +struct cpu_implementers { + u_int impl_id; + const char *impl_name; + /* + * Part number is implementation defined + * so each vendor will have its own set of values and names. + */ + const struct cpu_parts *cpu_parts; +}; +#define CPU_IMPLEMENTER_NONE { 0, "Unknown Implementer", cpu_parts_none } + +/* + * Per-implementer table of (PartNum, CPU Name) pairs. + */ +/* UC Berkeley */ +static const struct cpu_parts cpu_parts_ucb[] = { + { CPU_PART_RV32I, "RV32I" }, + { CPU_PART_RV32E, "RV32E" }, + { CPU_PART_RV64I, "RV64I" }, + { CPU_PART_RV128I, "RV128I" }, + CPU_PART_NONE, +}; + +/* Unknown */ +static const struct cpu_parts cpu_parts_none[] = { + CPU_PART_NONE, +}; + +/* + * Implementers table. + */ +const struct cpu_implementers cpu_implementers[] = { + { CPU_IMPL_UCB_ROCKET, "UC Berkeley Rocket", cpu_parts_ucb }, + CPU_IMPLEMENTER_NONE, +}; + +void +identify_cpu(void) +{ + const struct cpu_parts *cpu_partsp; + uint32_t part_id; + uint32_t impl_id; + uint64_t mimpid; + uint64_t mcpuid; + u_int cpu; + size_t i; + + cpu_partsp = NULL; + + mimpid = machine_command(ECALL_MIMPID_GET, 0); + mcpuid = machine_command(ECALL_MCPUID_GET, 0); + + /* SMPTODO: use mhartid ? */ + cpu = PCPU_GET(cpuid); + + impl_id = CPU_IMPL(mimpid); + for (i = 0; i < nitems(cpu_implementers); i++) { + if (impl_id == cpu_implementers[i].impl_id || + cpu_implementers[i].impl_id == 0) { + cpu_desc[cpu].cpu_impl = impl_id; + cpu_desc[cpu].cpu_impl_name = cpu_implementers[i].impl_name; + cpu_partsp = cpu_implementers[i].cpu_parts; + break; + } + } + + part_id = CPU_PART(mcpuid); + for (i = 0; &cpu_partsp[i] != NULL; i++) { + if (part_id == cpu_partsp[i].part_id || + cpu_partsp[i].part_id == -1) { + cpu_desc[cpu].cpu_part_num = part_id; + cpu_desc[cpu].cpu_part_name = cpu_partsp[i].part_name; + break; + } + } + + /* Print details for boot CPU or if we want verbose output */ + if (cpu == 0 || bootverbose) { + printf("CPU(%d): %s %s\n", cpu, + cpu_desc[cpu].cpu_impl_name, + cpu_desc[cpu].cpu_part_name); + } +} Property changes on: user/ngie/make_check/sys/riscv/riscv/identcpu.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/in_cksum.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/in_cksum.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/in_cksum.c (revision 295078) @@ -0,0 +1,241 @@ +/* $NetBSD: in_cksum.c,v 1.7 1997/09/02 13:18:15 thorpej Exp $ */ + +/*- + * Copyright (c) 1988, 1992, 1993 + * The Regents of the University of California. All rights reserved. + * Copyright (c) 1996 + * Matt Thomas + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by the University of + * California, Berkeley and its contributors. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * @(#)in_cksum.c 8.1 (Berkeley) 6/10/93 + */ + +#include /* RCS ID & Copyright macro defns */ +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +/* + * Checksum routine for Internet Protocol family headers + * (Portable Alpha version). + * + * This routine is very heavily used in the network + * code and should be modified for each CPU to be as fast as possible. + */ + +#define ADDCARRY(x) (x > 65535 ? x -= 65535 : x) +#define REDUCE32 \ + { \ + q_util.q = sum; \ + sum = q_util.s[0] + q_util.s[1] + q_util.s[2] + q_util.s[3]; \ + } +#define REDUCE16 \ + { \ + q_util.q = sum; \ + l_util.l = q_util.s[0] + q_util.s[1] + q_util.s[2] + q_util.s[3]; \ + sum = l_util.s[0] + l_util.s[1]; \ + ADDCARRY(sum); \ + } + +static const u_int32_t in_masks[] = { + /*0 bytes*/ /*1 byte*/ /*2 bytes*/ /*3 bytes*/ + 0x00000000, 0x000000FF, 0x0000FFFF, 0x00FFFFFF, /* offset 0 */ + 0x00000000, 0x0000FF00, 0x00FFFF00, 0xFFFFFF00, /* offset 1 */ + 0x00000000, 0x00FF0000, 0xFFFF0000, 0xFFFF0000, /* offset 2 */ + 0x00000000, 0xFF000000, 0xFF000000, 0xFF000000, /* offset 3 */ +}; + +union l_util { + u_int16_t s[2]; + u_int32_t l; +}; +union q_util { + u_int16_t s[4]; + u_int32_t l[2]; + u_int64_t q; +}; + +static u_int64_t +in_cksumdata(const void *buf, int len) +{ + const u_int32_t *lw = (const u_int32_t *) buf; + u_int64_t sum = 0; + u_int64_t prefilled; + int offset; + union q_util q_util; + + if ((3 & (long) lw) == 0 && len == 20) { + sum = (u_int64_t) lw[0] + lw[1] + lw[2] + lw[3] + lw[4]; + REDUCE32; + return sum; + } + + if ((offset = 3 & (long) lw) != 0) { + const u_int32_t *masks = in_masks + (offset << 2); + lw = (u_int32_t *) (((long) lw) - offset); + sum = *lw++ & masks[len >= 3 ? 3 : len]; + len -= 4 - offset; + if (len <= 0) { + REDUCE32; + return sum; + } + } +#if 0 + /* + * Force to cache line boundary. + */ + offset = 32 - (0x1f & (long) lw); + if (offset < 32 && len > offset) { + len -= offset; + if (4 & offset) { + sum += (u_int64_t) lw[0]; + lw += 1; + } + if (8 & offset) { + sum += (u_int64_t) lw[0] + lw[1]; + lw += 2; + } + if (16 & offset) { + sum += (u_int64_t) lw[0] + lw[1] + lw[2] + lw[3]; + lw += 4; + } + } +#endif + /* + * access prefilling to start load of next cache line. + * then add current cache line + * save result of prefilling for loop iteration. + */ + prefilled = lw[0]; + while ((len -= 32) >= 4) { + u_int64_t prefilling = lw[8]; + sum += prefilled + lw[1] + lw[2] + lw[3] + + lw[4] + lw[5] + lw[6] + lw[7]; + lw += 8; + prefilled = prefilling; + } + if (len >= 0) { + sum += prefilled + lw[1] + lw[2] + lw[3] + + lw[4] + lw[5] + lw[6] + lw[7]; + lw += 8; + } else { + len += 32; + } + while ((len -= 16) >= 0) { + sum += (u_int64_t) lw[0] + lw[1] + lw[2] + lw[3]; + lw += 4; + } + len += 16; + while ((len -= 4) >= 0) { + sum += (u_int64_t) *lw++; + } + len += 4; + if (len > 0) + sum += (u_int64_t) (in_masks[len] & *lw); + REDUCE32; + return sum; +} + +u_short +in_addword(u_short a, u_short b) +{ + u_int64_t sum = a + b; + + ADDCARRY(sum); + return (sum); +} + +u_short +in_pseudo(u_int32_t a, u_int32_t b, u_int32_t c) +{ + u_int64_t sum; + union q_util q_util; + union l_util l_util; + + sum = (u_int64_t) a + b + c; + REDUCE16; + return (sum); +} + +u_short +in_cksum_skip(struct mbuf *m, int len, int skip) +{ + u_int64_t sum = 0; + int mlen = 0; + int clen = 0; + caddr_t addr; + union q_util q_util; + union l_util l_util; + + len -= skip; + for (; skip && m; m = m->m_next) { + if (m->m_len > skip) { + mlen = m->m_len - skip; + addr = mtod(m, caddr_t) + skip; + goto skip_start; + } else { + skip -= m->m_len; + } + } + + for (; m && len; m = m->m_next) { + if (m->m_len == 0) + continue; + mlen = m->m_len; + addr = mtod(m, caddr_t); +skip_start: + if (len < mlen) + mlen = len; + if ((clen ^ (long) addr) & 1) + sum += in_cksumdata(addr, mlen) << 8; + else + sum += in_cksumdata(addr, mlen); + + clen += mlen; + len -= mlen; + } + REDUCE16; + return (~sum & 0xffff); +} + +u_int in_cksum_hdr(const struct ip *ip) +{ + u_int64_t sum = in_cksumdata(ip, sizeof(struct ip)); + union q_util q_util; + union l_util l_util; + REDUCE16; + return (~sum & 0xffff); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/in_cksum.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/intr_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/intr_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/intr_machdep.c (revision 295078) @@ -0,0 +1,223 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +enum { + IRQ_SOFTWARE, + IRQ_TIMER, + IRQ_HTIF, + NIRQS +}; + +u_long intrcnt[NIRQS]; +size_t sintrcnt = sizeof(intrcnt); + +char intrnames[NIRQS * (MAXCOMLEN + 1) * 2]; +size_t sintrnames = sizeof(intrnames); + +static struct intr_event *intr_events[NIRQS]; +static riscv_intrcnt_t riscv_intr_counters[NIRQS]; + +static int intrcnt_index; + +riscv_intrcnt_t +riscv_intrcnt_create(const char* name) +{ + riscv_intrcnt_t counter; + + counter = &intrcnt[intrcnt_index++]; + riscv_intrcnt_setname(counter, name); + + return (counter); +} + +void +riscv_intrcnt_setname(riscv_intrcnt_t counter, const char *name) +{ + int i; + + i = (counter - intrcnt); + + KASSERT(counter != NULL, ("riscv_intrcnt_setname: NULL counter")); + + snprintf(intrnames + (MAXCOMLEN + 1) * i, + MAXCOMLEN + 1, "%-*s", MAXCOMLEN, name); +} + +static void +riscv_mask_irq(void *source) +{ + uintptr_t irq; + + irq = (uintptr_t)source; + + switch (irq) { + case IRQ_TIMER: + csr_clear(sie, SIE_STIE); + break; + case IRQ_SOFTWARE: + csr_clear(sie, SIE_SSIE); + break; + default: + panic("Unknown irq %d\n", irq); + } +} + +static void +riscv_unmask_irq(void *source) +{ + uintptr_t irq; + + irq = (uintptr_t)source; + + switch (irq) { + case IRQ_TIMER: + csr_set(sie, SIE_STIE); + break; + case IRQ_SOFTWARE: + csr_set(sie, SIE_SSIE); + break; + default: + panic("Unknown irq %d\n", irq); + } +} + +void +riscv_init_interrupts(void) +{ + char name[MAXCOMLEN + 1]; + int i; + + for (i = 0; i < NIRQS; i++) { + snprintf(name, MAXCOMLEN + 1, "int%d:", i); + riscv_intr_counters[i] = riscv_intrcnt_create(name); + } +} + +int +riscv_setup_intr(const char *name, driver_filter_t *filt, + void (*handler)(void*), void *arg, int irq, int flags, void **cookiep) +{ + struct intr_event *event; + int error; + + if (irq < 0 || irq >= NIRQS) + panic("%s: unknown intr %d", __func__, irq); + + event = intr_events[irq]; + if (event == NULL) { + error = intr_event_create(&event, (void *)(uintptr_t)irq, 0, + irq, riscv_mask_irq, riscv_unmask_irq, + NULL, NULL, "int%d", irq); + if (error) + return (error); + intr_events[irq] = event; + riscv_unmask_irq((void*)(uintptr_t)irq); + } + + intr_event_add_handler(event, name, filt, handler, arg, + intr_priority(flags), flags, cookiep); + + riscv_intrcnt_setname(riscv_intr_counters[irq], + event->ie_fullname); + + return (0); +} + +int +riscv_teardown_intr(void *ih) +{ + + /* TODO */ + + return (0); +} + +int +riscv_config_intr(u_int irq, enum intr_trigger trig, enum intr_polarity pol) +{ + + /* There is no configuration for interrupts */ + + return (0); +} + +void +riscv_cpu_intr(struct trapframe *frame) +{ + struct intr_event *event; + int active_irq; + + critical_enter(); + + KASSERT(frame->tf_scause & EXCP_INTR, + ("riscv_cpu_intr: wrong frame passed")); + + active_irq = (frame->tf_scause & EXCP_MASK); + + switch (active_irq) { + case IRQ_SOFTWARE: + case IRQ_TIMER: + event = intr_events[active_irq]; + /* Update counters */ + atomic_add_long(riscv_intr_counters[active_irq], 1); + PCPU_INC(cnt.v_intr); + break; + case IRQ_HTIF: + /* HTIF interrupts are only handled in machine mode */ + panic("%s: HTIF interrupt", __func__); + break; + default: + event = NULL; + } + + if (!event || TAILQ_EMPTY(&event->ie_handlers) || + (intr_event_handle(event, frame) != 0)) + printf("stray interrupt %d\n", active_irq); + + critical_exit(); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/intr_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/locore.S =================================================================== --- user/ngie/make_check/sys/riscv/riscv/locore.S (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/locore.S (revision 295078) @@ -0,0 +1,274 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#include "assym.s" + +#include +#include +#include +#include +#include +#include + +#define HTIF_RING_SIZE (64) +#define HTIF_RING_LAST (24 * (HTIF_RING_SIZE - 1)) + + .globl kernbase + .set kernbase, KERNBASE + + /* Trap entries */ + .text + +mentry: + /* User mode entry point (mtvec + 0x000) */ + .align 6 + j user_trap + + /* Supervisor mode entry point (mtvec + 0x040) */ + .align 6 + j supervisor_trap + + /* Hypervisor mode entry point (mtvec + 0x080) */ + .align 6 + j bad_trap + + /* Machine mode entry point (mtvec + 0x0C0) */ + .align 6 + j bad_trap + + /* Reset vector */ + .text + .align 8 + .globl _start +_start: + li s11, KERNBASE + + /* Build ring */ + la t0, htif_ring + li t1, 0 + sd t1, 0(t0) /* zero data */ + sd t1, 8(t0) /* zero used */ + mv t2, t0 + mv t3, t0 + li t5, HTIF_RING_LAST + li t6, 0 + add t4, t0, t5 +1: + addi t3, t3, 24 /* pointer to next */ + beq t3, t4, 2f /* finish */ + sd t3, 16(t2) /* store pointer */ + addi t2, t2, 24 /* next entry */ + addi t6, t6, 1 /* counter */ + j 1b +2: + sd t0, 16(t3) /* last -> first */ + la t1, htif_ring_cursor + sd t0, 0(t1) + la t1, htif_ring_last + sd t0, 0(t1) + /* finish building ring */ + + la t0, hardstack_end + sub t0, t0, s11 + csrw mscratch, t0 + + la t0, mentry + csrw mtvec, t0 + + li t0, 0 + csrw sscratch, t0 + + li s10, PAGE_SIZE + li s9, (PAGE_SIZE * KSTACK_PAGES) + + /* Page tables */ + + /* Level 0 */ + la s1, pagetable_l0 + la s2, pagetable_l1 /* Link to next level PN */ + srli s2, s2, PAGE_SHIFT + + li t4, (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)) + slli t5, s2, PTE_PPN0_S /* (s2 << PTE_PPN0_S) */ + or t6, t4, t5 + + /* Store single level0 PTE entry to position */ + li a5, 0x1ff + li a6, PTE_SIZE + mulw a5, a5, a6 + add t0, s1, a5 + sd t6, 0(t0) + + /* Level 1 */ + la s1, pagetable_l1 + la s2, pagetable_l2 /* Link to next level PN */ + srli s2, s2, PAGE_SHIFT + + li a5, KERNBASE + srli a5, a5, 0x1e /* >> 30 */ + andi a5, a5, 0x1ff /* & 0x1ff */ + li t4, (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)) + slli t5, s2, PTE_PPN0_S /* (s2 << PTE_PPN0_S) */ + or t6, t4, t5 + + /* Store single level1 PTE entry to position */ + li a6, PTE_SIZE + mulw a5, a5, a6 + add t0, s1, a5 + sd t6, (t0) + + /* Level 2 superpages (512 x 2MiB) */ + la s1, pagetable_l2 + li t3, 512 /* Build 512 entries */ + li t4, 0 /* Counter */ + li t5, 0 +2: + li t0, (PTE_VALID | (PTE_TYPE_SRWX << PTE_TYPE_S)) + slli t2, t4, PTE_PPN1_S /* << PTE_PPN1_S */ + or t5, t0, t2 + sd t5, (s1) /* Store PTE entry to position */ + addi s1, s1, PTE_SIZE + + addi t4, t4, 1 + bltu t4, t3, 2b + + /* Set page tables base register */ + la s1, pagetable_l0 + csrw sptbr, s1 + + /* Page tables END */ + + /* Enter supervisor mode */ + li s0, ((MSTATUS_VM_SV48 << MSTATUS_VM_SHIFT) | \ + (MSTATUS_PRV_M << MSTATUS_PRV_SHIFT) | \ + (MSTATUS_PRV_S << MSTATUS_PRV1_SHIFT) | \ + (MSTATUS_PRV_U << MSTATUS_PRV2_SHIFT)); + csrw mstatus, s0 + + /* Exit from machine mode */ + la t0, .Lmmu_on + add t0, t0, s11 + csrw mepc, t0 + eret + +.Lmmu_on: + /* Initialize stack pointer */ + la s3, initstack_end + mv sp, s3 + addi sp, sp, -PCB_SIZE + + /* Clear BSS */ + la a0, _C_LABEL(__bss_start) + la s1, _C_LABEL(_end) +1: + sd zero, 0(a0) + addi a0, a0, 8 + bltu a0, s1, 1b + + /* Fill riscv_bootparams */ + addi sp, sp, -16 + la t0, pagetable_l1 + sd t0, 0(sp) /* kern_l1pt */ + la t0, initstack_end + sd t0, 8(sp) /* kern_stack */ + + mv a0, sp + call _C_LABEL(initriscv) /* Off we go */ + call _C_LABEL(mi_startup) + + .align 4 +initstack: + .space (PAGE_SIZE * KSTACK_PAGES) +initstack_end: +hardstack: + .space (PAGE_SIZE) +hardstack_end: + + .globl htif_ring +htif_ring: + .space (24 * 1024) + + .globl htif_ring_cursor +htif_ring_cursor: + .space (8) + + .globl htif_ring_last +htif_ring_last: + .space (8) + + .globl console_intr +console_intr: + .space (8) + +ENTRY(sigcode) + mv a0, sp + addi a0, a0, SF_UC + +1: + li t0, SYS_sigreturn + ecall + + /* sigreturn failed, exit */ + li t0, SYS_exit + ecall + + j 1b +END(sigcode) + /* This may be copied to the stack, keep it 16-byte aligned */ + .align 3 +esigcode: + + .data + .align 3 + .global szsigcode +szsigcode: + .quad esigcode - sigcode + + .align 12 + .globl pagetable_l0 +pagetable_l0: + .space PAGE_SIZE +pagetable_l1: + .space PAGE_SIZE +pagetable_l2: + .space PAGE_SIZE +pagetable_end: + + .globl init_pt_va +init_pt_va: + .quad pagetable_l2 /* XXX: Keep page tables VA */ + +#include "exception.S" Property changes on: user/ngie/make_check/sys/riscv/riscv/locore.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/machdep.c (revision 295078) @@ -0,0 +1,795 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include "opt_platform.h" + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#ifdef VFP +#include +#endif + +#ifdef FDT +#include +#include +#endif + +struct pcpu __pcpu[MAXCPU]; + +static struct trapframe proc0_tf; + +vm_paddr_t phys_avail[PHYS_AVAIL_SIZE + 2]; +vm_paddr_t dump_avail[PHYS_AVAIL_SIZE + 2]; + +int early_boot = 1; +int cold = 1; +long realmem = 0; +long Maxmem = 0; + +#define PHYSMAP_SIZE (2 * (VM_PHYSSEG_MAX - 1)) +vm_paddr_t physmap[PHYSMAP_SIZE]; +u_int physmap_idx; + +struct kva_md_info kmi; + +int64_t dcache_line_size; /* The minimum D cache line size */ +int64_t icache_line_size; /* The minimum I cache line size */ +int64_t idcache_line_size; /* The minimum cache line size */ + +extern int *end; +extern int *initstack_end; + +struct pcpu *pcpup; + +uintptr_t mcall_trap(uintptr_t mcause, uintptr_t* regs); + +uintptr_t +mcall_trap(uintptr_t mcause, uintptr_t* regs) +{ + + return (0); +} + +static void +cpu_startup(void *dummy) +{ + + identify_cpu(); + + vm_ksubmap_init(&kmi); + bufinit(); + vm_pager_bufferinit(); +} + +SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); + +int +cpu_idle_wakeup(int cpu) +{ + + return (0); +} + +void +bzero(void *buf, size_t len) +{ + uint8_t *p; + + p = buf; + while(len-- > 0) + *p++ = 0; +} + +int +fill_regs(struct thread *td, struct reg *regs) +{ + struct trapframe *frame; + + frame = td->td_frame; + regs->sepc = frame->tf_sepc; + regs->sstatus = frame->tf_sstatus; + regs->ra = frame->tf_ra; + regs->sp = frame->tf_sp; + regs->gp = frame->tf_gp; + regs->tp = frame->tf_tp; + + memcpy(regs->t, frame->tf_t, sizeof(regs->t)); + memcpy(regs->s, frame->tf_s, sizeof(regs->s)); + memcpy(regs->a, frame->tf_a, sizeof(regs->a)); + + return (0); +} + +int +set_regs(struct thread *td, struct reg *regs) +{ + struct trapframe *frame; + + frame = td->td_frame; + frame->tf_sepc = regs->sepc; + frame->tf_sstatus = regs->sstatus; + frame->tf_ra = regs->ra; + frame->tf_sp = regs->sp; + frame->tf_gp = regs->gp; + frame->tf_tp = regs->tp; + + memcpy(frame->tf_t, regs->t, sizeof(frame->tf_t)); + memcpy(frame->tf_s, regs->s, sizeof(frame->tf_s)); + memcpy(frame->tf_a, regs->a, sizeof(frame->tf_a)); + + return (0); +} + +int +fill_fpregs(struct thread *td, struct fpreg *regs) +{ + + /* TODO */ + bzero(regs, sizeof(*regs)); + return (0); +} + +int +set_fpregs(struct thread *td, struct fpreg *regs) +{ + + /* TODO */ + return (0); +} + +int +fill_dbregs(struct thread *td, struct dbreg *regs) +{ + + panic("fill_dbregs"); +} + +int +set_dbregs(struct thread *td, struct dbreg *regs) +{ + + panic("set_dbregs"); +} + +int +ptrace_set_pc(struct thread *td, u_long addr) +{ + + panic("ptrace_set_pc"); + return (0); +} + +int +ptrace_single_step(struct thread *td) +{ + + /* TODO; */ + return (0); +} + +int +ptrace_clear_single_step(struct thread *td) +{ + + /* TODO; */ + return (0); +} + +void +exec_setregs(struct thread *td, struct image_params *imgp, u_long stack) +{ + struct trapframe *tf = td->td_frame; + + memset(tf, 0, sizeof(struct trapframe)); + + /* + * We need to set a0 for init as it doesn't call + * cpu_set_syscall_retval to copy the value. We also + * need to set td_retval for the cases where we do. + */ + tf->tf_a[0] = td->td_retval[0] = stack; + tf->tf_sp = STACKALIGN(stack); + tf->tf_ra = imgp->entry_addr; + tf->tf_sepc = imgp->entry_addr; +} + +/* Sanity check these are the same size, they will be memcpy'd to and fro */ +CTASSERT(sizeof(((struct trapframe *)0)->tf_a) == + sizeof((struct gpregs *)0)->gp_a); +CTASSERT(sizeof(((struct trapframe *)0)->tf_s) == + sizeof((struct gpregs *)0)->gp_s); +CTASSERT(sizeof(((struct trapframe *)0)->tf_t) == + sizeof((struct gpregs *)0)->gp_t); +CTASSERT(sizeof(((struct trapframe *)0)->tf_a) == + sizeof((struct reg *)0)->a); +CTASSERT(sizeof(((struct trapframe *)0)->tf_s) == + sizeof((struct reg *)0)->s); +CTASSERT(sizeof(((struct trapframe *)0)->tf_t) == + sizeof((struct reg *)0)->t); + +int +get_mcontext(struct thread *td, mcontext_t *mcp, int clear_ret) +{ + struct trapframe *tf = td->td_frame; + + memcpy(mcp->mc_gpregs.gp_t, tf->tf_t, sizeof(mcp->mc_gpregs.gp_t)); + memcpy(mcp->mc_gpregs.gp_s, tf->tf_s, sizeof(mcp->mc_gpregs.gp_s)); + memcpy(mcp->mc_gpregs.gp_a, tf->tf_a, sizeof(mcp->mc_gpregs.gp_a)); + + if (clear_ret & GET_MC_CLEAR_RET) { + mcp->mc_gpregs.gp_a[0] = 0; + mcp->mc_gpregs.gp_t[0] = 0; /* clear syscall error */ + } + + mcp->mc_gpregs.gp_ra = tf->tf_ra; + mcp->mc_gpregs.gp_sp = tf->tf_sp; + mcp->mc_gpregs.gp_gp = tf->tf_gp; + mcp->mc_gpregs.gp_tp = tf->tf_tp; + mcp->mc_gpregs.gp_sepc = tf->tf_sepc; + mcp->mc_gpregs.gp_sstatus = tf->tf_sstatus; + + return (0); +} + +int +set_mcontext(struct thread *td, mcontext_t *mcp) +{ + struct trapframe *tf; + + tf = td->td_frame; + + memcpy(tf->tf_t, mcp->mc_gpregs.gp_t, sizeof(tf->tf_t)); + memcpy(tf->tf_s, mcp->mc_gpregs.gp_s, sizeof(tf->tf_s)); + memcpy(tf->tf_a, mcp->mc_gpregs.gp_a, sizeof(tf->tf_a)); + + tf->tf_ra = mcp->mc_gpregs.gp_ra; + tf->tf_sp = mcp->mc_gpregs.gp_sp; + tf->tf_gp = mcp->mc_gpregs.gp_gp; + tf->tf_tp = mcp->mc_gpregs.gp_tp; + tf->tf_sepc = mcp->mc_gpregs.gp_sepc; + tf->tf_sstatus = mcp->mc_gpregs.gp_sstatus; + + return (0); +} + +static void +get_fpcontext(struct thread *td, mcontext_t *mcp) +{ + /* TODO */ +} + +static void +set_fpcontext(struct thread *td, mcontext_t *mcp) +{ + /* TODO */ +} + +void +cpu_idle(int busy) +{ + + spinlock_enter(); + if (!busy) + cpu_idleclock(); + if (!sched_runnable()) + __asm __volatile( + "fence \n" + "wfi \n"); + if (!busy) + cpu_activeclock(); + spinlock_exit(); +} + +void +cpu_halt(void) +{ + + panic("cpu_halt"); +} + +/* + * Flush the D-cache for non-DMA I/O so that the I-cache can + * be made coherent later. + */ +void +cpu_flush_dcache(void *ptr, size_t len) +{ + + /* TBD */ +} + +/* Get current clock frequency for the given CPU ID. */ +int +cpu_est_clockrate(int cpu_id, uint64_t *rate) +{ + + panic("cpu_est_clockrate"); +} + +void +cpu_pcpu_init(struct pcpu *pcpu, int cpuid, size_t size) +{ + +} + +void +spinlock_enter(void) +{ + struct thread *td; + + td = curthread; + if (td->td_md.md_spinlock_count == 0) { + td->td_md.md_spinlock_count = 1; + td->td_md.md_saved_sstatus_ie = intr_disable(); + } else + td->td_md.md_spinlock_count++; + critical_enter(); +} + +void +spinlock_exit(void) +{ + struct thread *td; + register_t sstatus_ie; + + td = curthread; + critical_exit(); + sstatus_ie = td->td_md.md_saved_sstatus_ie; + td->td_md.md_spinlock_count--; + if (td->td_md.md_spinlock_count == 0) + intr_restore(sstatus_ie); +} + +#ifndef _SYS_SYSPROTO_H_ +struct sigreturn_args { + ucontext_t *ucp; +}; +#endif + +int +sys_sigreturn(struct thread *td, struct sigreturn_args *uap) +{ + uint64_t sstatus; + ucontext_t uc; + int error; + + if (uap == NULL) + return (EFAULT); + if (copyin(uap->sigcntxp, &uc, sizeof(uc))) + return (EFAULT); + + /* + * Make sure the processor mode has not been tampered with and + * interrupts have not been disabled. + */ + sstatus = uc.uc_mcontext.mc_gpregs.gp_sstatus; + if ((sstatus & SSTATUS_PS) != 0 || + (sstatus & SSTATUS_PIE) == 0) + return (EINVAL); + + error = set_mcontext(td, &uc.uc_mcontext); + if (error != 0) + return (error); + + set_fpcontext(td, &uc.uc_mcontext); + + /* Restore signal mask. */ + kern_sigprocmask(td, SIG_SETMASK, &uc.uc_sigmask, NULL, 0); + + return (EJUSTRETURN); +} + +/* + * Construct a PCB from a trapframe. This is called from kdb_trap() where + * we want to start a backtrace from the function that caused us to enter + * the debugger. We have the context in the trapframe, but base the trace + * on the PCB. The PCB doesn't have to be perfect, as long as it contains + * enough for a backtrace. + */ +void +makectx(struct trapframe *tf, struct pcb *pcb) +{ + + memcpy(pcb->pcb_t, tf->tf_t, sizeof(tf->tf_t)); + memcpy(pcb->pcb_s, tf->tf_s, sizeof(tf->tf_s)); + memcpy(pcb->pcb_a, tf->tf_a, sizeof(tf->tf_a)); + + pcb->pcb_ra = tf->tf_ra; + pcb->pcb_sp = tf->tf_sp; + pcb->pcb_gp = tf->tf_gp; + pcb->pcb_tp = tf->tf_tp; + pcb->pcb_sepc = tf->tf_sepc; +} + +void +sendsig(sig_t catcher, ksiginfo_t *ksi, sigset_t *mask) +{ + struct sigframe *fp, frame; + struct sysentvec *sysent; + struct trapframe *tf; + struct sigacts *psp; + struct thread *td; + struct proc *p; + int onstack; + int code; + int sig; + + td = curthread; + p = td->td_proc; + PROC_LOCK_ASSERT(p, MA_OWNED); + + sig = ksi->ksi_signo; + code = ksi->ksi_code; + psp = p->p_sigacts; + mtx_assert(&psp->ps_mtx, MA_OWNED); + + tf = td->td_frame; + onstack = sigonstack(tf->tf_sp); + + CTR4(KTR_SIG, "sendsig: td=%p (%s) catcher=%p sig=%d", td, p->p_comm, + catcher, sig); + + /* Allocate and validate space for the signal handler context. */ + if ((td->td_pflags & TDP_ALTSTACK) != 0 && !onstack && + SIGISMEMBER(psp->ps_sigonstack, sig)) { + fp = (struct sigframe *)((uintptr_t)td->td_sigstk.ss_sp + + td->td_sigstk.ss_size); + } else { + fp = (struct sigframe *)td->td_frame->tf_sp; + } + + /* Make room, keeping the stack aligned */ + fp--; + fp = (struct sigframe *)STACKALIGN(fp); + + /* Fill in the frame to copy out */ + get_mcontext(td, &frame.sf_uc.uc_mcontext, 0); + get_fpcontext(td, &frame.sf_uc.uc_mcontext); + frame.sf_si = ksi->ksi_info; + frame.sf_uc.uc_sigmask = *mask; + frame.sf_uc.uc_stack.ss_flags = (td->td_pflags & TDP_ALTSTACK) ? + ((onstack) ? SS_ONSTACK : 0) : SS_DISABLE; + frame.sf_uc.uc_stack = td->td_sigstk; + mtx_unlock(&psp->ps_mtx); + PROC_UNLOCK(td->td_proc); + + /* Copy the sigframe out to the user's stack. */ + if (copyout(&frame, fp, sizeof(*fp)) != 0) { + /* Process has trashed its stack. Kill it. */ + CTR2(KTR_SIG, "sendsig: sigexit td=%p fp=%p", td, fp); + PROC_LOCK(p); + sigexit(td, SIGILL); + } + + tf->tf_a[0] = sig; + tf->tf_a[1] = (register_t)&fp->sf_si; + tf->tf_a[2] = (register_t)&fp->sf_uc; + + tf->tf_sepc = (register_t)catcher; + tf->tf_sp = (register_t)fp; + + sysent = p->p_sysent; + if (sysent->sv_sigcode_base != 0) + tf->tf_ra = (register_t)sysent->sv_sigcode_base; + else + tf->tf_ra = (register_t)(sysent->sv_psstrings - + *(sysent->sv_szsigcode)); + + CTR3(KTR_SIG, "sendsig: return td=%p pc=%#x sp=%#x", td, tf->tf_elr, + tf->tf_sp); + + PROC_LOCK(p); + mtx_lock(&psp->ps_mtx); +} + +static void +init_proc0(vm_offset_t kstack) +{ + pcpup = &__pcpu[0]; + + proc_linkup0(&proc0, &thread0); + thread0.td_kstack = kstack; + thread0.td_pcb = (struct pcb *)(thread0.td_kstack) - 1; + thread0.td_frame = &proc0_tf; + pcpup->pc_curpcb = thread0.td_pcb; +} + +static int +add_physmap_entry(uint64_t base, uint64_t length, vm_paddr_t *physmap, + u_int *physmap_idxp) +{ + u_int i, insert_idx, _physmap_idx; + + _physmap_idx = *physmap_idxp; + + if (length == 0) + return (1); + + /* + * Find insertion point while checking for overlap. Start off by + * assuming the new entry will be added to the end. + */ + insert_idx = _physmap_idx; + for (i = 0; i <= _physmap_idx; i += 2) { + if (base < physmap[i + 1]) { + if (base + length <= physmap[i]) { + insert_idx = i; + break; + } + if (boothowto & RB_VERBOSE) + printf( + "Overlapping memory regions, ignoring second region\n"); + return (1); + } + } + + /* See if we can prepend to the next entry. */ + if (insert_idx <= _physmap_idx && + base + length == physmap[insert_idx]) { + physmap[insert_idx] = base; + return (1); + } + + /* See if we can append to the previous entry. */ + if (insert_idx > 0 && base == physmap[insert_idx - 1]) { + physmap[insert_idx - 1] += length; + return (1); + } + + _physmap_idx += 2; + *physmap_idxp = _physmap_idx; + if (_physmap_idx == PHYSMAP_SIZE) { + printf( + "Too many segments in the physical address map, giving up\n"); + return (0); + } + + /* + * Move the last 'N' entries down to make room for the new + * entry if needed. + */ + for (i = _physmap_idx; i > insert_idx; i -= 2) { + physmap[i] = physmap[i - 2]; + physmap[i + 1] = physmap[i - 1]; + } + + /* Insert the new entry. */ + physmap[insert_idx] = base; + physmap[insert_idx + 1] = base + length; + + printf("physmap[%d] = 0x%016lx\n", insert_idx, base); + printf("physmap[%d] = 0x%016lx\n", insert_idx + 1, base + length); + return (1); +} + +#ifdef FDT +static void +try_load_dtb(caddr_t kmdp) +{ + vm_offset_t dtbp; + + dtbp = (vm_offset_t)&fdt_static_dtb; + if (dtbp == (vm_offset_t)NULL) { + printf("ERROR loading DTB\n"); + return; + } + + if (OF_install(OFW_FDT, 0) == FALSE) + panic("Cannot install FDT"); + + if (OF_init((void *)dtbp) != 0) + panic("OF_init failed with the found device tree"); +} +#endif + +static void +cache_setup(void) +{ + + /* TODO */ +} + +/* + * Fake up a boot descriptor table. + * RISCVTODO: This needs to be done via loader (when it's available). + */ +vm_offset_t +fake_preload_metadata(struct riscv_bootparams *rvbp __unused) +{ +#ifdef DDB + vm_offset_t zstart = 0, zend = 0; +#endif + vm_offset_t lastaddr; + int i = 0; + static uint32_t fake_preload[35]; + + fake_preload[i++] = MODINFO_NAME; + fake_preload[i++] = strlen("kernel") + 1; + strcpy((char*)&fake_preload[i++], "kernel"); + i += 1; + fake_preload[i++] = MODINFO_TYPE; + fake_preload[i++] = strlen("elf64 kernel") + 1; + strcpy((char*)&fake_preload[i++], "elf64 kernel"); + i += 3; + fake_preload[i++] = MODINFO_ADDR; + fake_preload[i++] = sizeof(vm_offset_t); + fake_preload[i++] = (uint64_t)(KERNBASE + KERNENTRY); + i += 1; + fake_preload[i++] = MODINFO_SIZE; + fake_preload[i++] = sizeof(uint64_t); + printf("end is 0x%016lx\n", (uint64_t)&end); + fake_preload[i++] = (uint64_t)&end - (uint64_t)(KERNBASE + KERNENTRY); + i += 1; +#ifdef DDB +#if 0 + /* RISCVTODO */ + if (*(uint32_t *)KERNVIRTADDR == MAGIC_TRAMP_NUMBER) { + fake_preload[i++] = MODINFO_METADATA|MODINFOMD_SSYM; + fake_preload[i++] = sizeof(vm_offset_t); + fake_preload[i++] = *(uint32_t *)(KERNVIRTADDR + 4); + fake_preload[i++] = MODINFO_METADATA|MODINFOMD_ESYM; + fake_preload[i++] = sizeof(vm_offset_t); + fake_preload[i++] = *(uint32_t *)(KERNVIRTADDR + 8); + lastaddr = *(uint32_t *)(KERNVIRTADDR + 8); + zend = lastaddr; + zstart = *(uint32_t *)(KERNVIRTADDR + 4); + db_fetch_ksymtab(zstart, zend); + } else +#endif +#endif + lastaddr = (vm_offset_t)&end; + fake_preload[i++] = 0; + fake_preload[i] = 0; + preload_metadata = (void *)fake_preload; + + return (lastaddr); +} + +void +initriscv(struct riscv_bootparams *rvbp) +{ + vm_offset_t lastaddr; + vm_size_t kernlen; + caddr_t kmdp; + + /* Set the module data location */ + lastaddr = fake_preload_metadata(rvbp); + + /* Find the kernel address */ + kmdp = preload_search_by_type("elf kernel"); + if (kmdp == NULL) + kmdp = preload_search_by_type("elf64 kernel"); + + boothowto = 0; + + kern_envp = NULL; + +#ifdef FDT + try_load_dtb(kmdp); +#endif + + /* Load the physical memory ranges */ + physmap_idx = 0; + + /* + * RISCVTODO: figure out whether platform provides ranges, + * or grab from FDT. + */ + add_physmap_entry(0, 0x8000000, physmap, &physmap_idx); + + /* Set the pcpu data, this is needed by pmap_bootstrap */ + pcpup = &__pcpu[0]; + pcpu_init(pcpup, 0, sizeof(struct pcpu)); + + /* Set the pcpu pointer */ +#if 0 + /* SMP TODO: try re-use gp for pcpu pointer */ + __asm __volatile( + "mv gp, %0" :: "r"(pcpup)); +#endif + + PCPU_SET(curthread, &thread0); + + /* Do basic tuning, hz etc */ + init_param1(); + + cache_setup(); + + /* Bootstrap enough of pmap to enter the kernel proper */ + kernlen = (lastaddr - KERNBASE); + pmap_bootstrap(rvbp->kern_l1pt, KERNENTRY, kernlen); + + cninit(); + + init_proc0(rvbp->kern_stack); + + /* set page table base register for thread0 */ + thread0.td_pcb->pcb_l1addr = (rvbp->kern_l1pt - KERNBASE); + + msgbufinit(msgbufp, msgbufsize); + mutex_init(); + init_param2(physmem); + kdb_init(); + + riscv_init_interrupts(); + + early_boot = 0; +} Property changes on: user/ngie/make_check/sys/riscv/riscv/machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/mem.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/mem.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/mem.c (revision 295078) @@ -0,0 +1,124 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include + +struct mem_range_softc mem_range_softc; + +int +memrw(struct cdev *dev, struct uio *uio, int flags) +{ + ssize_t orig_resid; + vm_offset_t off, v; + struct iovec *iov; + struct vm_page m; + vm_page_t marr; + u_int cnt; + int error; + + error = 0; + orig_resid = uio->uio_resid; + while (uio->uio_resid > 0 && error == 0) { + iov = uio->uio_iov; + if (iov->iov_len == 0) { + uio->uio_iov++; + uio->uio_iovcnt--; + if (uio->uio_iovcnt < 0) + panic("memrw"); + continue; + } + + v = uio->uio_offset; + off = v & PAGE_MASK; + cnt = ulmin(iov->iov_len, PAGE_SIZE - (u_int)off); + if (cnt == 0) + continue; + + switch(dev2unit(dev)) { + case CDEV_MINOR_KMEM: + /* If the address is in the DMAP just copy it */ + if (VIRT_IN_DMAP(v)) { + error = uiomove((void *)v, cnt, uio); + break; + } + + if (!kernacc((void *)v, cnt, uio->uio_rw == UIO_READ ? + VM_PROT_READ : VM_PROT_WRITE)) { + error = EFAULT; + break; + } + + /* Get the physical address to read */ + v = pmap_extract(kernel_pmap, v); + if (v == 0) { + error = EFAULT; + break; + } + + /* FALLTHROUGH */ + case CDEV_MINOR_MEM: + /* If within the DMAP use this to copy from */ + if (PHYS_IN_DMAP(v)) { + v = PHYS_TO_DMAP(v); + error = uiomove((void *)v, cnt, uio); + break; + } + + /* Have uiomove_fromphys handle the data */ + m.phys_addr = trunc_page(v); + marr = &m; + uiomove_fromphys(&marr, off, cnt, uio); + break; + } + } + + /* + * Don't return error if any byte was written. Read and write + * can return error only if no i/o was performed. + */ + if (uio->uio_resid != orig_resid) + error = 0; + + return (error); +} + Property changes on: user/ngie/make_check/sys/riscv/riscv/mem.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/minidump_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/minidump_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/minidump_machdep.c (revision 295078) @@ -0,0 +1,59 @@ +/*- + * Copyright (c) 2006 Peter Wemm + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include "opt_watchdog.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +CTASSERT(sizeof(struct kerneldumpheader) == 512); +CTASSERT(sizeof(*vm_page_dump) == 8); + +int +minidumpsys(struct dumperinfo *di) +{ + + panic("minidumpsys"); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/minidump_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/nexus.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/nexus.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/nexus.c (revision 295078) @@ -0,0 +1,387 @@ +/*- + * Copyright 1998 Massachusetts Institute of Technology + * + * Permission to use, copy, modify, and distribute this software and + * its documentation for any purpose and without fee is hereby + * granted, provided that both the above copyright notice and this + * permission notice appear in all copies, that both the above + * copyright notice and this permission notice appear in all + * supporting documentation, and that the name of M.I.T. not be used + * in advertising or publicity pertaining to distribution of the + * software without specific, written prior permission. M.I.T. makes + * no representations about the suitability of this software for any + * purpose. It is provided "as is" without express or implied + * warranty. + * + * THIS SOFTWARE IS PROVIDED BY M.I.T. ``AS IS''. M.I.T. DISCLAIMS + * ALL EXPRESS OR IMPLIED WARRANTIES WITH REGARD TO THIS SOFTWARE, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT + * SHALL M.I.T. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +/* + * This code implements a `root nexus' for RISC-V Architecture + * machines. The function of the root nexus is to serve as an + * attachment point for both processors and buses, and to manage + * resources which are common to all of them. In particular, + * this code implements the core resource managers for interrupt + * requests, DMA requests (which rightfully should be a part of the + * ISA code but it's easier to do it here for now), I/O port addresses, + * and I/O memory address space. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include + +#include "opt_platform.h" + +#include +#include "ofw_bus_if.h" + +extern struct bus_space memmap_bus; + +static MALLOC_DEFINE(M_NEXUSDEV, "nexusdev", "Nexus device"); + +struct nexus_device { + struct resource_list nx_resources; +}; + +#define DEVTONX(dev) ((struct nexus_device *)device_get_ivars(dev)) + +static struct rman mem_rman; +static struct rman irq_rman; + +static device_probe_t nexus_fdt_probe; +static int nexus_attach(device_t); + +static int nexus_print_child(device_t, device_t); +static device_t nexus_add_child(device_t, u_int, const char *, int); +static struct resource *nexus_alloc_resource(device_t, device_t, int, int *, + u_long, u_long, u_long, u_int); +static int nexus_activate_resource(device_t, device_t, int, int, + struct resource *); +static int nexus_config_intr(device_t dev, int irq, enum intr_trigger trig, + enum intr_polarity pol); +static struct resource_list *nexus_get_reslist(device_t, device_t); +static int nexus_set_resource(device_t, device_t, int, int, u_long, u_long); +static int nexus_deactivate_resource(device_t, device_t, int, int, + struct resource *); + +static int nexus_setup_intr(device_t dev, device_t child, struct resource *res, + int flags, driver_filter_t *filt, driver_intr_t *intr, void *arg, void **cookiep); +static int nexus_teardown_intr(device_t, device_t, struct resource *, void *); + +static int nexus_ofw_map_intr(device_t dev, device_t child, phandle_t iparent, + int icells, pcell_t *intr); + +static device_method_t nexus_methods[] = { + /* Device interface */ + DEVMETHOD(device_probe, nexus_fdt_probe), + DEVMETHOD(device_attach, nexus_attach), + + /* OFW interface */ + DEVMETHOD(ofw_bus_map_intr, nexus_ofw_map_intr), + + /* Bus interface */ + DEVMETHOD(bus_print_child, nexus_print_child), + DEVMETHOD(bus_add_child, nexus_add_child), + DEVMETHOD(bus_alloc_resource, nexus_alloc_resource), + DEVMETHOD(bus_activate_resource, nexus_activate_resource), + DEVMETHOD(bus_config_intr, nexus_config_intr), + DEVMETHOD(bus_get_resource_list, nexus_get_reslist), + DEVMETHOD(bus_set_resource, nexus_set_resource), + DEVMETHOD(bus_deactivate_resource, nexus_deactivate_resource), + DEVMETHOD(bus_setup_intr, nexus_setup_intr), + DEVMETHOD(bus_teardown_intr, nexus_teardown_intr), + + { 0, 0 } +}; + +static driver_t nexus_fdt_driver = { + "nexus", + nexus_methods, + 1 /* no softc */ +}; + +static int +nexus_fdt_probe(device_t dev) +{ + + device_quiet(dev); + return (BUS_PROBE_DEFAULT); +} + +static int +nexus_attach(device_t dev) +{ + + mem_rman.rm_start = 0; + mem_rman.rm_end = ~0ul; + mem_rman.rm_type = RMAN_ARRAY; + mem_rman.rm_descr = "I/O memory addresses"; + if (rman_init(&mem_rman) || rman_manage_region(&mem_rman, 0, ~0)) + panic("nexus_attach mem_rman"); + irq_rman.rm_start = 0; + irq_rman.rm_end = ~0ul; + irq_rman.rm_type = RMAN_ARRAY; + irq_rman.rm_descr = "Interrupts"; + if (rman_init(&irq_rman) || rman_manage_region(&irq_rman, 0, ~0)) + panic("nexus_attach irq_rman"); + + nexus_add_child(dev, 10, "ofwbus", 0); + + bus_generic_probe(dev); + bus_generic_attach(dev); + + return (0); +} + +static int +nexus_print_child(device_t bus, device_t child) +{ + int retval = 0; + + retval += bus_print_child_header(bus, child); + retval += printf("\n"); + + return (retval); +} + +static device_t +nexus_add_child(device_t bus, u_int order, const char *name, int unit) +{ + device_t child; + struct nexus_device *ndev; + + ndev = malloc(sizeof(struct nexus_device), M_NEXUSDEV, M_NOWAIT|M_ZERO); + if (!ndev) + return (0); + resource_list_init(&ndev->nx_resources); + + child = device_add_child_ordered(bus, order, name, unit); + + /* should we free this in nexus_child_detached? */ + device_set_ivars(child, ndev); + + return (child); +} + + +/* + * Allocate a resource on behalf of child. NB: child is usually going to be a + * child of one of our descendants, not a direct child of nexus0. + * (Exceptions include footbridge.) + */ +static struct resource * +nexus_alloc_resource(device_t bus, device_t child, int type, int *rid, + u_long start, u_long end, u_long count, u_int flags) +{ + struct nexus_device *ndev = DEVTONX(child); + struct resource *rv; + struct resource_list_entry *rle; + struct rman *rm; + int needactivate = flags & RF_ACTIVE; + + /* + * If this is an allocation of the "default" range for a given + * RID, and we know what the resources for this device are + * (ie. they aren't maintained by a child bus), then work out + * the start/end values. + */ + if ((start == 0UL) && (end == ~0UL) && (count == 1)) { + if (device_get_parent(child) != bus || ndev == NULL) + return(NULL); + rle = resource_list_find(&ndev->nx_resources, type, *rid); + if (rle == NULL) + return(NULL); + start = rle->start; + end = rle->end; + count = rle->count; + } + + switch (type) { + case SYS_RES_IRQ: + rm = &irq_rman; + break; + + case SYS_RES_MEMORY: + case SYS_RES_IOPORT: + rm = &mem_rman; + break; + + default: + return (NULL); + } + + rv = rman_reserve_resource(rm, start, end, count, flags, child); + if (rv == 0) + return (NULL); + + rman_set_rid(rv, *rid); + rman_set_bushandle(rv, rman_get_start(rv)); + + if (needactivate) { + if (bus_activate_resource(child, type, *rid, rv)) { + rman_release_resource(rv); + return (NULL); + } + } + + return (rv); +} + +static int +nexus_config_intr(device_t dev, int irq, enum intr_trigger trig, + enum intr_polarity pol) +{ + + return (riscv_config_intr(irq, trig, pol)); +} + +static int +nexus_setup_intr(device_t dev, device_t child, struct resource *res, int flags, + driver_filter_t *filt, driver_intr_t *intr, void *arg, void **cookiep) +{ + int error; + + if ((rman_get_flags(res) & RF_SHAREABLE) == 0) + flags |= INTR_EXCL; + + /* We depend here on rman_activate_resource() being idempotent. */ + error = rman_activate_resource(res); + if (error) + return (error); + + error = riscv_setup_intr(device_get_nameunit(child), filt, intr, + arg, rman_get_start(res), flags, cookiep); + + return (error); +} + +static int +nexus_teardown_intr(device_t dev, device_t child, struct resource *r, void *ih) +{ + + return (riscv_teardown_intr(ih)); +} + +static int +nexus_activate_resource(device_t bus, device_t child, int type, int rid, + struct resource *r) +{ + int err; + bus_addr_t paddr; + bus_size_t psize; + bus_space_handle_t vaddr; + + if ((err = rman_activate_resource(r)) != 0) + return (err); + + /* + * If this is a memory resource, map it into the kernel. + */ + if (type == SYS_RES_MEMORY || type == SYS_RES_IOPORT) { + paddr = (bus_addr_t)rman_get_start(r); + psize = (bus_size_t)rman_get_size(r); + err = bus_space_map(&memmap_bus, paddr, psize, 0, &vaddr); + if (err != 0) { + rman_deactivate_resource(r); + return (err); + } + rman_set_bustag(r, &memmap_bus); + rman_set_virtual(r, (void *)vaddr); + rman_set_bushandle(r, vaddr); + } + return (0); +} + +static struct resource_list * +nexus_get_reslist(device_t dev, device_t child) +{ + struct nexus_device *ndev = DEVTONX(child); + + return (&ndev->nx_resources); +} + +static int +nexus_set_resource(device_t dev, device_t child, int type, int rid, + u_long start, u_long count) +{ + struct nexus_device *ndev = DEVTONX(child); + struct resource_list *rl = &ndev->nx_resources; + + /* XXX this should return a success/failure indicator */ + resource_list_add(rl, type, rid, start, start + count - 1, count); + + return(0); +} + + +static int +nexus_deactivate_resource(device_t bus, device_t child, int type, int rid, + struct resource *r) +{ + bus_size_t psize; + bus_space_handle_t vaddr; + + psize = (bus_size_t)rman_get_size(r); + vaddr = rman_get_bushandle(r); + + if (vaddr != 0) { + bus_space_unmap(&memmap_bus, vaddr, psize); + rman_set_virtual(r, NULL); + rman_set_bushandle(r, 0); + } + + return (rman_deactivate_resource(r)); +} + +static devclass_t nexus_fdt_devclass; + +EARLY_DRIVER_MODULE(nexus_fdt, root, nexus_fdt_driver, nexus_fdt_devclass, + 0, 0, BUS_PASS_BUS + BUS_PASS_ORDER_FIRST); + +static int +nexus_ofw_map_intr(device_t dev, device_t child, phandle_t iparent, int icells, + pcell_t *intr) +{ + int irq; + + if (icells == 3) { + irq = intr[1]; + if (intr[0] == 0) + irq += 32; /* SPI */ + else + irq += 16; /* PPI */ + } else + irq = intr[0]; + + return (irq); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/nexus.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/pmap.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/pmap.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/pmap.c (revision 295078) @@ -0,0 +1,3197 @@ +/*- + * Copyright (c) 1991 Regents of the University of California. + * All rights reserved. + * Copyright (c) 1994 John S. Dyson + * All rights reserved. + * Copyright (c) 1994 David Greenman + * All rights reserved. + * Copyright (c) 2003 Peter Wemm + * All rights reserved. + * Copyright (c) 2005-2010 Alan L. Cox + * All rights reserved. + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * the Systems Programming Group of the University of Utah Computer + * Science Department and William Jolitz of UUNET Technologies Inc. + * + * Portions of this software were developed by Andrew Turner under + * sponsorship from The FreeBSD Foundation. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by the University of + * California, Berkeley and its contributors. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * from: @(#)pmap.c 7.7 (Berkeley) 5/12/91 + */ +/*- + * Copyright (c) 2003 Networks Associates Technology, Inc. + * All rights reserved. + * + * This software was developed for the FreeBSD Project by Jake Burkholder, + * Safeport Network Services, and Network Associates Laboratories, the + * Security Research Division of Network Associates, Inc. under + * DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA + * CHATS research program. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +/* + * Manages physical address maps. + * + * Since the information managed by this module is + * also stored by the logical address mapping module, + * this module may throw away valid virtual-to-physical + * mappings at almost any time. However, invalidations + * of virtual-to-physical mappings must be done as + * requested. + * + * In order to cope with hardware architectures which + * make virtual-to-physical map invalidates expensive, + * this module may delay invalidate or reduced protection + * operations until such time as they are actually + * necessary. This module is given full information as + * to which processors are currently using which maps, + * and to when physical maps must be made correct. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#define NPDEPG (PAGE_SIZE/(sizeof (pd_entry_t))) +#define NUPDE (NPDEPG * NPDEPG) +#define NUSERPGTBLS (NUPDE + NPDEPG) + +#if !defined(DIAGNOSTIC) +#ifdef __GNUC_GNU_INLINE__ +#define PMAP_INLINE __attribute__((__gnu_inline__)) inline +#else +#define PMAP_INLINE extern inline +#endif +#else +#define PMAP_INLINE +#endif + +#ifdef PV_STATS +#define PV_STAT(x) do { x ; } while (0) +#else +#define PV_STAT(x) do { } while (0) +#endif + +#define pmap_l2_pindex(v) ((v) >> L2_SHIFT) + +#define NPV_LIST_LOCKS MAXCPU + +#define PHYS_TO_PV_LIST_LOCK(pa) \ + (&pv_list_locks[pa_index(pa) % NPV_LIST_LOCKS]) + +#define CHANGE_PV_LIST_LOCK_TO_PHYS(lockp, pa) do { \ + struct rwlock **_lockp = (lockp); \ + struct rwlock *_new_lock; \ + \ + _new_lock = PHYS_TO_PV_LIST_LOCK(pa); \ + if (_new_lock != *_lockp) { \ + if (*_lockp != NULL) \ + rw_wunlock(*_lockp); \ + *_lockp = _new_lock; \ + rw_wlock(*_lockp); \ + } \ +} while (0) + +#define CHANGE_PV_LIST_LOCK_TO_VM_PAGE(lockp, m) \ + CHANGE_PV_LIST_LOCK_TO_PHYS(lockp, VM_PAGE_TO_PHYS(m)) + +#define RELEASE_PV_LIST_LOCK(lockp) do { \ + struct rwlock **_lockp = (lockp); \ + \ + if (*_lockp != NULL) { \ + rw_wunlock(*_lockp); \ + *_lockp = NULL; \ + } \ +} while (0) + +#define VM_PAGE_TO_PV_LIST_LOCK(m) \ + PHYS_TO_PV_LIST_LOCK(VM_PAGE_TO_PHYS(m)) + +struct pmap kernel_pmap_store; + +vm_offset_t virtual_avail; /* VA of first avail page (after kernel bss) */ +vm_offset_t virtual_end; /* VA of last avail page (end of kernel AS) */ +vm_offset_t kernel_vm_end = 0; + +struct msgbuf *msgbufp = NULL; + +static struct rwlock_padalign pvh_global_lock; + +extern uint64_t pagetable_l0; + +/* + * Data for the pv entry allocation mechanism + */ +static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); +static struct mtx pv_chunks_mutex; +static struct rwlock pv_list_locks[NPV_LIST_LOCKS]; + +static void free_pv_chunk(struct pv_chunk *pc); +static void free_pv_entry(pmap_t pmap, pv_entry_t pv); +static pv_entry_t get_pv_entry(pmap_t pmap, struct rwlock **lockp); +static vm_page_t reclaim_pv_chunk(pmap_t locked_pmap, struct rwlock **lockp); +static void pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va); +static pv_entry_t pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, + vm_offset_t va); +static vm_page_t pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, + vm_page_t m, vm_prot_t prot, vm_page_t mpte, struct rwlock **lockp); +static int pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_t sva, + pd_entry_t ptepde, struct spglist *free, struct rwlock **lockp); +static boolean_t pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, + vm_page_t m, struct rwlock **lockp); + +static vm_page_t _pmap_alloc_l3(pmap_t pmap, vm_pindex_t ptepindex, + struct rwlock **lockp); + +static void _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, + struct spglist *free); +static int pmap_unuse_l3(pmap_t, vm_offset_t, pd_entry_t, struct spglist *); + +/* + * These load the old table data and store the new value. + * They need to be atomic as the System MMU may write to the table at + * the same time as the CPU. + */ +#define pmap_load_store(table, entry) atomic_swap_64(table, entry) +#define pmap_set(table, mask) atomic_set_64(table, mask) +#define pmap_load_clear(table) atomic_swap_64(table, 0) +#define pmap_load(table) (*table) + +/********************/ +/* Inline functions */ +/********************/ + +static __inline void +pagecopy(void *s, void *d) +{ + + memcpy(d, s, PAGE_SIZE); +} + +static __inline void +pagezero(void *p) +{ + + bzero(p, PAGE_SIZE); +} + +#define pmap_l1_index(va) (((va) >> L1_SHIFT) & Ln_ADDR_MASK) +#define pmap_l2_index(va) (((va) >> L2_SHIFT) & Ln_ADDR_MASK) +#define pmap_l3_index(va) (((va) >> L3_SHIFT) & Ln_ADDR_MASK) + +#define PTE_TO_PHYS(pte) ((pte >> PTE_PPN0_S) * PAGE_SIZE) + +static __inline pd_entry_t * +pmap_l1(pmap_t pmap, vm_offset_t va) +{ + + return (&pmap->pm_l1[pmap_l1_index(va)]); +} + +static __inline pd_entry_t * +pmap_l1_to_l2(pd_entry_t *l1, vm_offset_t va) +{ + vm_paddr_t phys; + pd_entry_t *l2; + + phys = PTE_TO_PHYS(pmap_load(l1)); + l2 = (pd_entry_t *)PHYS_TO_DMAP(phys); + + return (&l2[pmap_l2_index(va)]); +} + +static __inline pd_entry_t * +pmap_l2(pmap_t pmap, vm_offset_t va) +{ + pd_entry_t *l1; + + l1 = pmap_l1(pmap, va); + + if ((pmap_load(l1) & PTE_VALID) == 0) + return (NULL); + if ((pmap_load(l1) & PTE_TYPE_M) != (PTE_TYPE_PTR << PTE_TYPE_S)) + return (NULL); + + return (pmap_l1_to_l2(l1, va)); +} + +static __inline pt_entry_t * +pmap_l2_to_l3(pd_entry_t *l2, vm_offset_t va) +{ + vm_paddr_t phys; + pt_entry_t *l3; + + phys = PTE_TO_PHYS(pmap_load(l2)); + l3 = (pd_entry_t *)PHYS_TO_DMAP(phys); + + return (&l3[pmap_l3_index(va)]); +} + +static __inline pt_entry_t * +pmap_l3(pmap_t pmap, vm_offset_t va) +{ + pd_entry_t *l2; + + l2 = pmap_l2(pmap, va); + if (l2 == NULL) + return (NULL); + if ((pmap_load(l2) & PTE_VALID) == 0) + return (NULL); + if (l2 == NULL || (pmap_load(l2) & PTE_TYPE_M) != (PTE_TYPE_PTR << PTE_TYPE_S)) + return (NULL); + + return (pmap_l2_to_l3(l2, va)); +} + + +static __inline int +pmap_is_write(pt_entry_t entry) +{ + + if (entry & (1 << PTE_TYPE_S)) + return (1); + + return (0); +} + +static __inline int +pmap_is_current(pmap_t pmap) +{ + + return ((pmap == pmap_kernel()) || + (pmap == curthread->td_proc->p_vmspace->vm_map.pmap)); +} + +static __inline int +pmap_l3_valid(pt_entry_t l3) +{ + + return (l3 & PTE_VALID); +} + +static __inline int +pmap_l3_valid_cacheable(pt_entry_t l3) +{ + + /* TODO */ + + return (0); +} + +#define PTE_SYNC(pte) cpu_dcache_wb_range((vm_offset_t)pte, sizeof(*pte)) + +/* Checks if the page is dirty. */ +static inline int +pmap_page_dirty(pt_entry_t pte) +{ + + return (pte & PTE_DIRTY); +} + +static __inline void +pmap_resident_count_inc(pmap_t pmap, int count) +{ + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + pmap->pm_stats.resident_count += count; +} + +static __inline void +pmap_resident_count_dec(pmap_t pmap, int count) +{ + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + KASSERT(pmap->pm_stats.resident_count >= count, + ("pmap %p resident count underflow %ld %d", pmap, + pmap->pm_stats.resident_count, count)); + pmap->pm_stats.resident_count -= count; +} + +static pt_entry_t * +pmap_early_page_idx(vm_offset_t l1pt, vm_offset_t va, u_int *l1_slot, + u_int *l2_slot) +{ + pt_entry_t *l2; + pd_entry_t *l1; + + l1 = (pd_entry_t *)l1pt; + *l1_slot = (va >> L1_SHIFT) & Ln_ADDR_MASK; + + /* Check locore has used a table L1 map */ + KASSERT((l1[*l1_slot] & PTE_TYPE_M) == (PTE_TYPE_PTR << PTE_TYPE_S), + ("Invalid bootstrap L1 table")); + + /* Find the address of the L2 table */ + l2 = (pt_entry_t *)init_pt_va; + *l2_slot = pmap_l2_index(va); + + return (l2); +} + +static vm_paddr_t +pmap_early_vtophys(vm_offset_t l1pt, vm_offset_t va) +{ + u_int l1_slot, l2_slot; + pt_entry_t *l2; + u_int ret; + + l2 = pmap_early_page_idx(l1pt, va, &l1_slot, &l2_slot); + + /* L2 is superpages */ + ret = (l2[l2_slot] >> PTE_PPN1_S) << L2_SHIFT; + ret += (va & L2_OFFSET); + + return (ret); +} + +static void +pmap_bootstrap_dmap(vm_offset_t l2pt) +{ + vm_offset_t va; + vm_paddr_t pa; + pd_entry_t *l2; + u_int l2_slot; + pt_entry_t entry; + u_int pn; + + va = DMAP_MIN_ADDRESS; + l2 = (pd_entry_t *)l2pt; + l2_slot = pmap_l2_index(DMAP_MIN_ADDRESS); + + for (pa = 0; va < DMAP_MAX_ADDRESS; pa += L2_SIZE, va += L2_SIZE, l2_slot++) { + KASSERT(l2_slot < Ln_ENTRIES, ("Invalid L2 index")); + + /* superpages */ + pn = ((pa >> L2_SHIFT) & Ln_ADDR_MASK); + entry = (PTE_VALID | (PTE_TYPE_SRWX << PTE_TYPE_S)); + entry |= (pn << PTE_PPN1_S); + + pmap_load_store(&l2[l2_slot], entry); + } + + cpu_dcache_wb_range((vm_offset_t)l2, PAGE_SIZE); + cpu_tlb_flushID(); +} + +/* + * Bootstrap the system enough to run with virtual memory. + */ +void +pmap_bootstrap(vm_offset_t l1pt, vm_paddr_t kernstart, vm_size_t kernlen) +{ + u_int l1_slot, l2_slot, avail_slot, map_slot, used_map_slot; + uint64_t kern_delta; + pt_entry_t *l2; + vm_offset_t va, freemempos; + vm_offset_t dpcpu, msgbufpv; + vm_paddr_t pa, min_pa; + vm_offset_t l2pt; + int i; + + kern_delta = KERNBASE - kernstart; + physmem = 0; + + printf("pmap_bootstrap %lx %lx %lx\n", l1pt, kernstart, kernlen); + printf("%lx\n", l1pt); + printf("%lx\n", (KERNBASE >> L1_SHIFT) & Ln_ADDR_MASK); + + /* Set this early so we can use the pagetable walking functions */ + kernel_pmap_store.pm_l1 = (pd_entry_t *)l1pt; + PMAP_LOCK_INIT(kernel_pmap); + + /* + * Initialize the global pv list lock. + */ + rw_init(&pvh_global_lock, "pmap pv global"); + + /* Assume the address we were loaded to is a valid physical address */ + min_pa = KERNBASE - kern_delta; + + /* + * Find the minimum physical address. physmap is sorted, + * but may contain empty ranges. + */ + for (i = 0; i < (physmap_idx * 2); i += 2) { + if (physmap[i] == physmap[i + 1]) + continue; + if (physmap[i] <= min_pa) + min_pa = physmap[i]; + break; + } + + /* Create a direct map region early so we can use it for pa -> va */ + l2pt = (l1pt + PAGE_SIZE); + pmap_bootstrap_dmap(l2pt); + + va = KERNBASE; + pa = KERNBASE - kern_delta; + + /* + * Start to initialize phys_avail by copying from physmap + * up to the physical address KERNBASE points at. + */ + map_slot = avail_slot = 0; + for (; map_slot < (physmap_idx * 2); map_slot += 2) { + if (physmap[map_slot] == physmap[map_slot + 1]) + continue; + + phys_avail[avail_slot] = physmap[map_slot]; + phys_avail[avail_slot + 1] = physmap[map_slot + 1]; + physmem += (phys_avail[avail_slot + 1] - + phys_avail[avail_slot]) >> PAGE_SHIFT; + avail_slot += 2; + } + + /* Add the memory before the kernel */ + if (physmap[avail_slot] < pa) { + phys_avail[avail_slot] = physmap[map_slot]; + phys_avail[avail_slot + 1] = pa; + physmem += (phys_avail[avail_slot + 1] - + phys_avail[avail_slot]) >> PAGE_SHIFT; + avail_slot += 2; + } + used_map_slot = map_slot; + + /* + * Read the page table to find out what is already mapped. + * This assumes we have mapped a block of memory from KERNBASE + * using a single L1 entry. + */ + l2 = pmap_early_page_idx(l1pt, KERNBASE, &l1_slot, &l2_slot); + + /* Sanity check the index, KERNBASE should be the first VA */ + KASSERT(l2_slot == 0, ("The L2 index is non-zero")); + + /* Find how many pages we have mapped */ + for (; l2_slot < Ln_ENTRIES; l2_slot++) { + if ((l2[l2_slot] & PTE_VALID) == 0) + break; + + /* Check locore used L2 superpages */ + KASSERT((l2[l2_slot] & PTE_TYPE_M) != (PTE_TYPE_PTR << PTE_TYPE_S), + ("Invalid bootstrap L2 table")); + + va += L2_SIZE; + pa += L2_SIZE; + } + + va = roundup2(va, L2_SIZE); + + freemempos = KERNBASE + kernlen; + freemempos = roundup2(freemempos, PAGE_SIZE); + + cpu_tlb_flushID(); + +#define alloc_pages(var, np) \ + (var) = freemempos; \ + freemempos += (np * PAGE_SIZE); \ + memset((char *)(var), 0, ((np) * PAGE_SIZE)); + + /* Allocate dynamic per-cpu area. */ + alloc_pages(dpcpu, DPCPU_SIZE / PAGE_SIZE); + dpcpu_init((void *)dpcpu, 0); + + /* Allocate memory for the msgbuf, e.g. for /sbin/dmesg */ + alloc_pages(msgbufpv, round_page(msgbufsize) / PAGE_SIZE); + msgbufp = (void *)msgbufpv; + + virtual_avail = roundup2(freemempos, L2_SIZE); + virtual_end = VM_MAX_KERNEL_ADDRESS - L2_SIZE; + kernel_vm_end = virtual_avail; + + pa = pmap_early_vtophys(l1pt, freemempos); + + /* Finish initialising physmap */ + map_slot = used_map_slot; + for (; avail_slot < (PHYS_AVAIL_SIZE - 2) && + map_slot < (physmap_idx * 2); map_slot += 2) { + if (physmap[map_slot] == physmap[map_slot + 1]) + continue; + + /* Have we used the current range? */ + if (physmap[map_slot + 1] <= pa) + continue; + + /* Do we need to split the entry? */ + if (physmap[map_slot] < pa) { + phys_avail[avail_slot] = pa; + phys_avail[avail_slot + 1] = physmap[map_slot + 1]; + } else { + phys_avail[avail_slot] = physmap[map_slot]; + phys_avail[avail_slot + 1] = physmap[map_slot + 1]; + } + physmem += (phys_avail[avail_slot + 1] - + phys_avail[avail_slot]) >> PAGE_SHIFT; + + avail_slot += 2; + } + phys_avail[avail_slot] = 0; + phys_avail[avail_slot + 1] = 0; + + /* + * Maxmem isn't the "maximum memory", it's one larger than the + * highest page of the physical address space. It should be + * called something like "Maxphyspage". + */ + Maxmem = atop(phys_avail[avail_slot - 1]); + + cpu_tlb_flushID(); +} + +/* + * Initialize a vm_page's machine-dependent fields. + */ +void +pmap_page_init(vm_page_t m) +{ + + TAILQ_INIT(&m->md.pv_list); + m->md.pv_memattr = VM_MEMATTR_WRITE_BACK; +} + +/* + * Initialize the pmap module. + * Called by vm_init, to initialize any structures that the pmap + * system needs to map virtual memory. + */ +void +pmap_init(void) +{ + int i; + + /* + * Initialize the pv chunk list mutex. + */ + mtx_init(&pv_chunks_mutex, "pmap pv chunk list", NULL, MTX_DEF); + + /* + * Initialize the pool of pv list locks. + */ + for (i = 0; i < NPV_LIST_LOCKS; i++) + rw_init(&pv_list_locks[i], "pmap pv list"); +} + +/* + * Normal, non-SMP, invalidation functions. + * We inline these within pmap.c for speed. + */ +PMAP_INLINE void +pmap_invalidate_page(pmap_t pmap, vm_offset_t va) +{ + + /* TODO */ + + sched_pin(); + __asm __volatile("sfence.vm"); + sched_unpin(); +} + +PMAP_INLINE void +pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +{ + + /* TODO */ + + sched_pin(); + __asm __volatile("sfence.vm"); + sched_unpin(); +} + +PMAP_INLINE void +pmap_invalidate_all(pmap_t pmap) +{ + + /* TODO */ + + sched_pin(); + __asm __volatile("sfence.vm"); + sched_unpin(); +} + +/* + * Routine: pmap_extract + * Function: + * Extract the physical page address associated + * with the given map/virtual_address pair. + */ +vm_paddr_t +pmap_extract(pmap_t pmap, vm_offset_t va) +{ + pd_entry_t *l2p, l2; + pt_entry_t *l3p, l3; + vm_paddr_t pa; + + pa = 0; + PMAP_LOCK(pmap); + /* + * Start with the l2 tabel. We are unable to allocate + * pages in the l1 table. + */ + l2p = pmap_l2(pmap, va); + if (l2p != NULL) { + l2 = pmap_load(l2p); + if ((l2 & PTE_TYPE_M) == (PTE_TYPE_PTR << PTE_TYPE_S)) { + l3p = pmap_l2_to_l3(l2p, va); + if (l3p != NULL) { + l3 = pmap_load(l3p); + pa = PTE_TO_PHYS(l3); + pa |= (va & L3_OFFSET); + } + } else { + /* L2 is superpages */ + pa = (l2 >> PTE_PPN1_S) << L2_SHIFT; + pa |= (va & L2_OFFSET); + } + } + PMAP_UNLOCK(pmap); + return (pa); +} + +/* + * Routine: pmap_extract_and_hold + * Function: + * Atomically extract and hold the physical page + * with the given pmap and virtual address pair + * if that mapping permits the given protection. + */ +vm_page_t +pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) +{ + pt_entry_t *l3p, l3; + vm_paddr_t phys; + vm_paddr_t pa; + vm_page_t m; + + pa = 0; + m = NULL; + PMAP_LOCK(pmap); +retry: + l3p = pmap_l3(pmap, va); + if (l3p != NULL && (l3 = pmap_load(l3p)) != 0) { + if ((pmap_is_write(l3)) || ((prot & VM_PROT_WRITE) == 0)) { + phys = PTE_TO_PHYS(l3); + if (vm_page_pa_tryrelock(pmap, phys, &pa)) + goto retry; + m = PHYS_TO_VM_PAGE(phys); + vm_page_hold(m); + } + } + PA_UNLOCK_COND(pa); + PMAP_UNLOCK(pmap); + return (m); +} + +vm_paddr_t +pmap_kextract(vm_offset_t va) +{ + pd_entry_t *l2; + pt_entry_t *l3; + vm_paddr_t pa; + + if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) { + pa = DMAP_TO_PHYS(va); + } else { + l2 = pmap_l2(kernel_pmap, va); + if (l2 == NULL) + panic("pmap_kextract: No l2"); + if ((pmap_load(l2) & PTE_TYPE_M) != (PTE_TYPE_PTR << PTE_TYPE_S)) { + /* superpages */ + pa = (pmap_load(l2) >> PTE_PPN1_S) << L2_SHIFT; + pa |= (va & L2_OFFSET); + return (pa); + } + + l3 = pmap_l2_to_l3(l2, va); + if (l3 == NULL) + panic("pmap_kextract: No l3..."); + pa = PTE_TO_PHYS(pmap_load(l3)); + pa |= (va & PAGE_MASK); + } + return (pa); +} + +/*************************************************** + * Low level mapping routines..... + ***************************************************/ + +void +pmap_kenter_device(vm_offset_t sva, vm_size_t size, vm_paddr_t pa) +{ + pt_entry_t *l3; + vm_offset_t va; + + panic("%s: implement me\n", __func__); + + KASSERT((pa & L3_OFFSET) == 0, + ("pmap_kenter_device: Invalid physical address")); + KASSERT((sva & L3_OFFSET) == 0, + ("pmap_kenter_device: Invalid virtual address")); + KASSERT((size & PAGE_MASK) == 0, + ("pmap_kenter_device: Mapping is not page-sized")); + + va = sva; + while (size != 0) { + l3 = pmap_l3(kernel_pmap, va); + KASSERT(l3 != NULL, ("Invalid page table, va: 0x%lx", va)); + panic("%s: unimplemented", __func__); +#if 0 /* implement me */ + pmap_load_store(l3, (pa & ~L3_OFFSET) | ATTR_DEFAULT | + ATTR_IDX(DEVICE_MEMORY) | L3_PAGE); +#endif + PTE_SYNC(l3); + + va += PAGE_SIZE; + pa += PAGE_SIZE; + size -= PAGE_SIZE; + } + pmap_invalidate_range(kernel_pmap, sva, va); +} + +/* + * Remove a page from the kernel pagetables. + * Note: not SMP coherent. + */ +PMAP_INLINE void +pmap_kremove(vm_offset_t va) +{ + pt_entry_t *l3; + + l3 = pmap_l3(kernel_pmap, va); + KASSERT(l3 != NULL, ("pmap_kremove: Invalid address")); + + if (pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(va, L3_SIZE); + pmap_load_clear(l3); + PTE_SYNC(l3); + pmap_invalidate_page(kernel_pmap, va); +} + +void +pmap_kremove_device(vm_offset_t sva, vm_size_t size) +{ + pt_entry_t *l3; + vm_offset_t va; + + KASSERT((sva & L3_OFFSET) == 0, + ("pmap_kremove_device: Invalid virtual address")); + KASSERT((size & PAGE_MASK) == 0, + ("pmap_kremove_device: Mapping is not page-sized")); + + va = sva; + while (size != 0) { + l3 = pmap_l3(kernel_pmap, va); + KASSERT(l3 != NULL, ("Invalid page table, va: 0x%lx", va)); + pmap_load_clear(l3); + PTE_SYNC(l3); + + va += PAGE_SIZE; + size -= PAGE_SIZE; + } + pmap_invalidate_range(kernel_pmap, sva, va); +} + +/* + * Used to map a range of physical addresses into kernel + * virtual address space. + * + * The value passed in '*virt' is a suggested virtual address for + * the mapping. Architectures which can support a direct-mapped + * physical to virtual region can return the appropriate address + * within that region, leaving '*virt' unchanged. Other + * architectures should map the pages starting at '*virt' and + * update '*virt' with the first usable address after the mapped + * region. + */ +vm_offset_t +pmap_map(vm_offset_t *virt, vm_paddr_t start, vm_paddr_t end, int prot) +{ + + return PHYS_TO_DMAP(start); +} + + +/* + * Add a list of wired pages to the kva + * this routine is only used for temporary + * kernel mappings that do not need to have + * page modification or references recorded. + * Note that old mappings are simply written + * over. The page *must* be wired. + * Note: SMP coherent. Uses a ranged shootdown IPI. + */ +void +pmap_qenter(vm_offset_t sva, vm_page_t *ma, int count) +{ + pt_entry_t *l3, pa; + vm_offset_t va; + vm_page_t m; + pt_entry_t entry; + u_int pn; + int i; + + va = sva; + for (i = 0; i < count; i++) { + m = ma[i]; + pa = VM_PAGE_TO_PHYS(m); + pn = (pa / PAGE_SIZE); + l3 = pmap_l3(kernel_pmap, va); + + entry = (PTE_VALID | (PTE_TYPE_SRWX << PTE_TYPE_S)); + entry |= (pn << PTE_PPN0_S); + pmap_load_store(l3, entry); + + PTE_SYNC(l3); + va += L3_SIZE; + } + pmap_invalidate_range(kernel_pmap, sva, va); +} + +/* + * This routine tears out page mappings from the + * kernel -- it is meant only for temporary mappings. + * Note: SMP coherent. Uses a ranged shootdown IPI. + */ +void +pmap_qremove(vm_offset_t sva, int count) +{ + pt_entry_t *l3; + vm_offset_t va; + + KASSERT(sva >= VM_MIN_KERNEL_ADDRESS, ("usermode va %lx", sva)); + + va = sva; + while (count-- > 0) { + l3 = pmap_l3(kernel_pmap, va); + KASSERT(l3 != NULL, ("pmap_kremove: Invalid address")); + + if (pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(va, L3_SIZE); + pmap_load_clear(l3); + PTE_SYNC(l3); + + va += PAGE_SIZE; + } + pmap_invalidate_range(kernel_pmap, sva, va); +} + +/*************************************************** + * Page table page management routines..... + ***************************************************/ +static __inline void +pmap_free_zero_pages(struct spglist *free) +{ + vm_page_t m; + + while ((m = SLIST_FIRST(free)) != NULL) { + SLIST_REMOVE_HEAD(free, plinks.s.ss); + /* Preserve the page's PG_ZERO setting. */ + vm_page_free_toq(m); + } +} + +/* + * Schedule the specified unused page table page to be freed. Specifically, + * add the page to the specified list of pages that will be released to the + * physical memory manager after the TLB has been updated. + */ +static __inline void +pmap_add_delayed_free_list(vm_page_t m, struct spglist *free, + boolean_t set_PG_ZERO) +{ + + if (set_PG_ZERO) + m->flags |= PG_ZERO; + else + m->flags &= ~PG_ZERO; + SLIST_INSERT_HEAD(free, m, plinks.s.ss); +} + +/* + * Decrements a page table page's wire count, which is used to record the + * number of valid page table entries within the page. If the wire count + * drops to zero, then the page table page is unmapped. Returns TRUE if the + * page table page was unmapped and FALSE otherwise. + */ +static inline boolean_t +pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free) +{ + + --m->wire_count; + if (m->wire_count == 0) { + _pmap_unwire_l3(pmap, va, m, free); + return (TRUE); + } else { + return (FALSE); + } +} + +static void +_pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free) +{ + vm_paddr_t phys; + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + /* + * unmap the page table page + */ + if (m->pindex >= NUPDE) { + /* PD page */ + pd_entry_t *l1; + l1 = pmap_l1(pmap, va); + pmap_load_clear(l1); + PTE_SYNC(l1); + } else { + /* PTE page */ + pd_entry_t *l2; + l2 = pmap_l2(pmap, va); + pmap_load_clear(l2); + PTE_SYNC(l2); + } + pmap_resident_count_dec(pmap, 1); + if (m->pindex < NUPDE) { + pd_entry_t *l1; + /* We just released a PT, unhold the matching PD */ + vm_page_t pdpg; + + l1 = pmap_l1(pmap, va); + phys = PTE_TO_PHYS(pmap_load(l1)); + pdpg = PHYS_TO_VM_PAGE(phys); + pmap_unwire_l3(pmap, va, pdpg, free); + } + pmap_invalidate_page(pmap, va); + + /* + * This is a release store so that the ordinary store unmapping + * the page table page is globally performed before TLB shoot- + * down is begun. + */ + atomic_subtract_rel_int(&vm_cnt.v_wire_count, 1); + + /* + * Put page on a list so that it is released after + * *ALL* TLB shootdown is done + */ + pmap_add_delayed_free_list(m, free, TRUE); +} + +/* + * After removing an l3 entry, this routine is used to + * conditionally free the page, and manage the hold/wire counts. + */ +static int +pmap_unuse_l3(pmap_t pmap, vm_offset_t va, pd_entry_t ptepde, + struct spglist *free) +{ + vm_paddr_t phys; + vm_page_t mpte; + + if (va >= VM_MAXUSER_ADDRESS) + return (0); + KASSERT(ptepde != 0, ("pmap_unuse_pt: ptepde != 0")); + + phys = PTE_TO_PHYS(ptepde); + + mpte = PHYS_TO_VM_PAGE(phys); + return (pmap_unwire_l3(pmap, va, mpte, free)); +} + +void +pmap_pinit0(pmap_t pmap) +{ + + PMAP_LOCK_INIT(pmap); + bzero(&pmap->pm_stats, sizeof(pmap->pm_stats)); + pmap->pm_l1 = kernel_pmap->pm_l1; +} + +int +pmap_pinit(pmap_t pmap) +{ + vm_paddr_t l1phys; + vm_page_t l1pt; + + /* + * allocate the l1 page + */ + while ((l1pt = vm_page_alloc(NULL, 0xdeadbeef, VM_ALLOC_NORMAL | + VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) + VM_WAIT; + + l1phys = VM_PAGE_TO_PHYS(l1pt); + pmap->pm_l1 = (pd_entry_t *)PHYS_TO_DMAP(l1phys); + + if ((l1pt->flags & PG_ZERO) == 0) + pagezero(pmap->pm_l1); + + bzero(&pmap->pm_stats, sizeof(pmap->pm_stats)); + + return (1); +} + +/* + * This routine is called if the desired page table page does not exist. + * + * If page table page allocation fails, this routine may sleep before + * returning NULL. It sleeps only if a lock pointer was given. + * + * Note: If a page allocation fails at page table level two or three, + * one or two pages may be held during the wait, only to be released + * afterwards. This conservative approach is easily argued to avoid + * race conditions. + */ +static vm_page_t +_pmap_alloc_l3(pmap_t pmap, vm_pindex_t ptepindex, struct rwlock **lockp) +{ + vm_page_t m, /*pdppg, */pdpg; + pt_entry_t entry; + vm_paddr_t phys; + int pn; + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + + /* + * Allocate a page table page. + */ + if ((m = vm_page_alloc(NULL, ptepindex, VM_ALLOC_NOOBJ | + VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) { + if (lockp != NULL) { + RELEASE_PV_LIST_LOCK(lockp); + PMAP_UNLOCK(pmap); + rw_runlock(&pvh_global_lock); + VM_WAIT; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + } + + /* + * Indicate the need to retry. While waiting, the page table + * page may have been allocated. + */ + return (NULL); + } + + if ((m->flags & PG_ZERO) == 0) + pmap_zero_page(m); + + /* + * Map the pagetable page into the process address space, if + * it isn't already there. + */ + + if (ptepindex >= NUPDE) { + pd_entry_t *l1; + vm_pindex_t l1index; + + l1index = ptepindex - NUPDE; + l1 = &pmap->pm_l1[l1index]; + + pn = (VM_PAGE_TO_PHYS(m) / PAGE_SIZE); + entry = (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)); + entry |= (pn << PTE_PPN0_S); + pmap_load_store(l1, entry); + + PTE_SYNC(l1); + + } else { + vm_pindex_t l1index; + pd_entry_t *l1, *l2; + + l1index = ptepindex >> (L1_SHIFT - L2_SHIFT); + l1 = &pmap->pm_l1[l1index]; + if (pmap_load(l1) == 0) { + /* recurse for allocating page dir */ + if (_pmap_alloc_l3(pmap, NUPDE + l1index, + lockp) == NULL) { + --m->wire_count; + atomic_subtract_int(&vm_cnt.v_wire_count, 1); + vm_page_free_zero(m); + return (NULL); + } + } else { + phys = PTE_TO_PHYS(pmap_load(l1)); + pdpg = PHYS_TO_VM_PAGE(phys); + pdpg->wire_count++; + } + + phys = PTE_TO_PHYS(pmap_load(l1)); + l2 = (pd_entry_t *)PHYS_TO_DMAP(phys); + l2 = &l2[ptepindex & Ln_ADDR_MASK]; + + pn = (VM_PAGE_TO_PHYS(m) / PAGE_SIZE); + entry = (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)); + entry |= (pn << PTE_PPN0_S); + pmap_load_store(l2, entry); + + PTE_SYNC(l2); + } + + pmap_resident_count_inc(pmap, 1); + + return (m); +} + +static vm_page_t +pmap_alloc_l3(pmap_t pmap, vm_offset_t va, struct rwlock **lockp) +{ + vm_pindex_t ptepindex; + pd_entry_t *l2; + vm_paddr_t phys; + vm_page_t m; + + /* + * Calculate pagetable page index + */ + ptepindex = pmap_l2_pindex(va); +retry: + /* + * Get the page directory entry + */ + l2 = pmap_l2(pmap, va); + + /* + * If the page table page is mapped, we just increment the + * hold count, and activate it. + */ + if (l2 != NULL && pmap_load(l2) != 0) { + phys = PTE_TO_PHYS(pmap_load(l2)); + m = PHYS_TO_VM_PAGE(phys); + m->wire_count++; + } else { + /* + * Here if the pte page isn't mapped, or if it has been + * deallocated. + */ + m = _pmap_alloc_l3(pmap, ptepindex, lockp); + if (m == NULL && lockp != NULL) + goto retry; + } + return (m); +} + + +/*************************************************** + * Pmap allocation/deallocation routines. + ***************************************************/ + +/* + * Release any resources held by the given physical map. + * Called when a pmap initialized by pmap_pinit is being released. + * Should only be called if the map contains no valid mappings. + */ +void +pmap_release(pmap_t pmap) +{ + vm_page_t m; + + KASSERT(pmap->pm_stats.resident_count == 0, + ("pmap_release: pmap resident count %ld != 0", + pmap->pm_stats.resident_count)); + + m = PHYS_TO_VM_PAGE(DMAP_TO_PHYS((vm_offset_t)pmap->pm_l1)); + m->wire_count--; + atomic_subtract_int(&vm_cnt.v_wire_count, 1); + vm_page_free_zero(m); +} + +#if 0 +static int +kvm_size(SYSCTL_HANDLER_ARGS) +{ + unsigned long ksize = VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS; + + return sysctl_handle_long(oidp, &ksize, 0, req); +} +SYSCTL_PROC(_vm, OID_AUTO, kvm_size, CTLTYPE_LONG|CTLFLAG_RD, + 0, 0, kvm_size, "LU", "Size of KVM"); + +static int +kvm_free(SYSCTL_HANDLER_ARGS) +{ + unsigned long kfree = VM_MAX_KERNEL_ADDRESS - kernel_vm_end; + + return sysctl_handle_long(oidp, &kfree, 0, req); +} +SYSCTL_PROC(_vm, OID_AUTO, kvm_free, CTLTYPE_LONG|CTLFLAG_RD, + 0, 0, kvm_free, "LU", "Amount of KVM free"); +#endif /* 0 */ + +/* + * grow the number of kernel page table entries, if needed + */ +void +pmap_growkernel(vm_offset_t addr) +{ + vm_paddr_t paddr; + vm_page_t nkpg; + pd_entry_t *l1, *l2; + pt_entry_t entry; + int pn; + + mtx_assert(&kernel_map->system_mtx, MA_OWNED); + + addr = roundup2(addr, L2_SIZE); + if (addr - 1 >= kernel_map->max_offset) + addr = kernel_map->max_offset; + while (kernel_vm_end < addr) { + l1 = pmap_l1(kernel_pmap, kernel_vm_end); + if (pmap_load(l1) == 0) { + /* We need a new PDP entry */ + nkpg = vm_page_alloc(NULL, kernel_vm_end >> L1_SHIFT, + VM_ALLOC_INTERRUPT | VM_ALLOC_NOOBJ | + VM_ALLOC_WIRED | VM_ALLOC_ZERO); + if (nkpg == NULL) + panic("pmap_growkernel: no memory to grow kernel"); + if ((nkpg->flags & PG_ZERO) == 0) + pmap_zero_page(nkpg); + paddr = VM_PAGE_TO_PHYS(nkpg); + + panic("%s: implement grow l1\n", __func__); +#if 0 + pmap_load_store(l1, paddr | L1_TABLE); +#endif + PTE_SYNC(l1); + continue; /* try again */ + } + l2 = pmap_l1_to_l2(l1, kernel_vm_end); + if ((pmap_load(l2) & PTE_REF) != 0) { + kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET; + if (kernel_vm_end - 1 >= kernel_map->max_offset) { + kernel_vm_end = kernel_map->max_offset; + break; + } + continue; + } + + nkpg = vm_page_alloc(NULL, kernel_vm_end >> L2_SHIFT, + VM_ALLOC_INTERRUPT | VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | + VM_ALLOC_ZERO); + if (nkpg == NULL) + panic("pmap_growkernel: no memory to grow kernel"); + if ((nkpg->flags & PG_ZERO) == 0) + pmap_zero_page(nkpg); + paddr = VM_PAGE_TO_PHYS(nkpg); + + pn = (paddr / PAGE_SIZE); + entry = (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)); + entry |= (pn << PTE_PPN0_S); + pmap_load_store(l2, entry); + + PTE_SYNC(l2); + pmap_invalidate_page(kernel_pmap, kernel_vm_end); + + kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET; + if (kernel_vm_end - 1 >= kernel_map->max_offset) { + kernel_vm_end = kernel_map->max_offset; + break; + } + } +} + + +/*************************************************** + * page management routines. + ***************************************************/ + +CTASSERT(sizeof(struct pv_chunk) == PAGE_SIZE); +CTASSERT(_NPCM == 3); +CTASSERT(_NPCPV == 168); + +static __inline struct pv_chunk * +pv_to_chunk(pv_entry_t pv) +{ + + return ((struct pv_chunk *)((uintptr_t)pv & ~(uintptr_t)PAGE_MASK)); +} + +#define PV_PMAP(pv) (pv_to_chunk(pv)->pc_pmap) + +#define PC_FREE0 0xfffffffffffffffful +#define PC_FREE1 0xfffffffffffffffful +#define PC_FREE2 0x000000fffffffffful + +static const uint64_t pc_freemask[_NPCM] = { PC_FREE0, PC_FREE1, PC_FREE2 }; + +#if 0 +#ifdef PV_STATS +static int pc_chunk_count, pc_chunk_allocs, pc_chunk_frees, pc_chunk_tryfail; + +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_count, CTLFLAG_RD, &pc_chunk_count, 0, + "Current number of pv entry chunks"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_allocs, CTLFLAG_RD, &pc_chunk_allocs, 0, + "Current number of pv entry chunks allocated"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_frees, CTLFLAG_RD, &pc_chunk_frees, 0, + "Current number of pv entry chunks frees"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_tryfail, CTLFLAG_RD, &pc_chunk_tryfail, 0, + "Number of times tried to get a chunk page but failed."); + +static long pv_entry_frees, pv_entry_allocs, pv_entry_count; +static int pv_entry_spare; + +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_frees, CTLFLAG_RD, &pv_entry_frees, 0, + "Current number of pv entry frees"); +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_allocs, CTLFLAG_RD, &pv_entry_allocs, 0, + "Current number of pv entry allocs"); +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_count, CTLFLAG_RD, &pv_entry_count, 0, + "Current number of pv entries"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_spare, CTLFLAG_RD, &pv_entry_spare, 0, + "Current number of spare pv entries"); +#endif +#endif /* 0 */ + +/* + * We are in a serious low memory condition. Resort to + * drastic measures to free some pages so we can allocate + * another pv entry chunk. + * + * Returns NULL if PV entries were reclaimed from the specified pmap. + * + * We do not, however, unmap 2mpages because subsequent accesses will + * allocate per-page pv entries until repromotion occurs, thereby + * exacerbating the shortage of free pv entries. + */ +static vm_page_t +reclaim_pv_chunk(pmap_t locked_pmap, struct rwlock **lockp) +{ + + panic("RISCVTODO: reclaim_pv_chunk"); +} + +/* + * free the pv_entry back to the free list + */ +static void +free_pv_entry(pmap_t pmap, pv_entry_t pv) +{ + struct pv_chunk *pc; + int idx, field, bit; + + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + PV_STAT(atomic_add_long(&pv_entry_frees, 1)); + PV_STAT(atomic_add_int(&pv_entry_spare, 1)); + PV_STAT(atomic_subtract_long(&pv_entry_count, 1)); + pc = pv_to_chunk(pv); + idx = pv - &pc->pc_pventry[0]; + field = idx / 64; + bit = idx % 64; + pc->pc_map[field] |= 1ul << bit; + if (pc->pc_map[0] != PC_FREE0 || pc->pc_map[1] != PC_FREE1 || + pc->pc_map[2] != PC_FREE2) { + /* 98% of the time, pc is already at the head of the list. */ + if (__predict_false(pc != TAILQ_FIRST(&pmap->pm_pvchunk))) { + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); + } + return; + } + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + free_pv_chunk(pc); +} + +static void +free_pv_chunk(struct pv_chunk *pc) +{ + vm_page_t m; + + mtx_lock(&pv_chunks_mutex); + TAILQ_REMOVE(&pv_chunks, pc, pc_lru); + mtx_unlock(&pv_chunks_mutex); + PV_STAT(atomic_subtract_int(&pv_entry_spare, _NPCPV)); + PV_STAT(atomic_subtract_int(&pc_chunk_count, 1)); + PV_STAT(atomic_add_int(&pc_chunk_frees, 1)); + /* entire chunk is free, return it */ + m = PHYS_TO_VM_PAGE(DMAP_TO_PHYS((vm_offset_t)pc)); +#if 0 /* TODO: For minidump */ + dump_drop_page(m->phys_addr); +#endif + vm_page_unwire(m, PQ_INACTIVE); + vm_page_free(m); +} + +/* + * Returns a new PV entry, allocating a new PV chunk from the system when + * needed. If this PV chunk allocation fails and a PV list lock pointer was + * given, a PV chunk is reclaimed from an arbitrary pmap. Otherwise, NULL is + * returned. + * + * The given PV list lock may be released. + */ +static pv_entry_t +get_pv_entry(pmap_t pmap, struct rwlock **lockp) +{ + int bit, field; + pv_entry_t pv; + struct pv_chunk *pc; + vm_page_t m; + + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + PV_STAT(atomic_add_long(&pv_entry_allocs, 1)); +retry: + pc = TAILQ_FIRST(&pmap->pm_pvchunk); + if (pc != NULL) { + for (field = 0; field < _NPCM; field++) { + if (pc->pc_map[field]) { + bit = ffsl(pc->pc_map[field]) - 1; + break; + } + } + if (field < _NPCM) { + pv = &pc->pc_pventry[field * 64 + bit]; + pc->pc_map[field] &= ~(1ul << bit); + /* If this was the last item, move it to tail */ + if (pc->pc_map[0] == 0 && pc->pc_map[1] == 0 && + pc->pc_map[2] == 0) { + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + TAILQ_INSERT_TAIL(&pmap->pm_pvchunk, pc, + pc_list); + } + PV_STAT(atomic_add_long(&pv_entry_count, 1)); + PV_STAT(atomic_subtract_int(&pv_entry_spare, 1)); + return (pv); + } + } + /* No free items, allocate another chunk */ + m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ | + VM_ALLOC_WIRED); + if (m == NULL) { + if (lockp == NULL) { + PV_STAT(pc_chunk_tryfail++); + return (NULL); + } + m = reclaim_pv_chunk(pmap, lockp); + if (m == NULL) + goto retry; + } + PV_STAT(atomic_add_int(&pc_chunk_count, 1)); + PV_STAT(atomic_add_int(&pc_chunk_allocs, 1)); +#if 0 /* TODO: This is for minidump */ + dump_add_page(m->phys_addr); +#endif + pc = (void *)PHYS_TO_DMAP(m->phys_addr); + pc->pc_pmap = pmap; + pc->pc_map[0] = PC_FREE0 & ~1ul; /* preallocated bit 0 */ + pc->pc_map[1] = PC_FREE1; + pc->pc_map[2] = PC_FREE2; + mtx_lock(&pv_chunks_mutex); + TAILQ_INSERT_TAIL(&pv_chunks, pc, pc_lru); + mtx_unlock(&pv_chunks_mutex); + pv = &pc->pc_pventry[0]; + TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); + PV_STAT(atomic_add_long(&pv_entry_count, 1)); + PV_STAT(atomic_add_int(&pv_entry_spare, _NPCPV - 1)); + return (pv); +} + +/* + * First find and then remove the pv entry for the specified pmap and virtual + * address from the specified pv list. Returns the pv entry if found and NULL + * otherwise. This operation can be performed on pv lists for either 4KB or + * 2MB page mappings. + */ +static __inline pv_entry_t +pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +{ + pv_entry_t pv; + + rw_assert(&pvh_global_lock, RA_LOCKED); + TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { + if (pmap == PV_PMAP(pv) && va == pv->pv_va) { + TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); + pvh->pv_gen++; + break; + } + } + return (pv); +} + +/* + * First find and then destroy the pv entry for the specified pmap and virtual + * address. This operation can be performed on pv lists for either 4KB or 2MB + * page mappings. + */ +static void +pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +{ + pv_entry_t pv; + + pv = pmap_pvh_remove(pvh, pmap, va); + + KASSERT(pv != NULL, ("pmap_pvh_free: pv not found")); + free_pv_entry(pmap, pv); +} + +/* + * Conditionally create the PV entry for a 4KB page mapping if the required + * memory can be allocated without resorting to reclamation. + */ +static boolean_t +pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, vm_page_t m, + struct rwlock **lockp) +{ + pv_entry_t pv; + + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + /* Pass NULL instead of the lock pointer to disable reclamation. */ + if ((pv = get_pv_entry(pmap, NULL)) != NULL) { + pv->pv_va = va; + CHANGE_PV_LIST_LOCK_TO_VM_PAGE(lockp, m); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + return (TRUE); + } else + return (FALSE); +} + +/* + * pmap_remove_l3: do the things to unmap a page in a process + */ +static int +pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_t va, + pd_entry_t l2e, struct spglist *free, struct rwlock **lockp) +{ + pt_entry_t old_l3; + vm_paddr_t phys; + vm_page_t m; + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + if (pmap_is_current(pmap) && pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(va, L3_SIZE); + old_l3 = pmap_load_clear(l3); + PTE_SYNC(l3); + pmap_invalidate_page(pmap, va); + if (old_l3 & PTE_SW_WIRED) + pmap->pm_stats.wired_count -= 1; + pmap_resident_count_dec(pmap, 1); + if (old_l3 & PTE_SW_MANAGED) { + phys = PTE_TO_PHYS(old_l3); + m = PHYS_TO_VM_PAGE(phys); + if (pmap_page_dirty(old_l3)) + vm_page_dirty(m); + if (old_l3 & PTE_REF) + vm_page_aflag_set(m, PGA_REFERENCED); + CHANGE_PV_LIST_LOCK_TO_VM_PAGE(lockp, m); + pmap_pvh_free(&m->md, pmap, va); + } + + return (pmap_unuse_l3(pmap, va, l2e, free)); +} + +/* + * Remove the given range of addresses from the specified map. + * + * It is assumed that the start and end are properly + * rounded to the page size. + */ +void +pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +{ + struct rwlock *lock; + vm_offset_t va, va_next; + pd_entry_t *l1, *l2; + pt_entry_t l3_pte, *l3; + struct spglist free; + int anyvalid; + + /* + * Perform an unsynchronized read. This is, however, safe. + */ + if (pmap->pm_stats.resident_count == 0) + return; + + anyvalid = 0; + SLIST_INIT(&free); + + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + + lock = NULL; + for (; sva < eva; sva = va_next) { + if (pmap->pm_stats.resident_count == 0) + break; + + l1 = pmap_l1(pmap, sva); + if (pmap_load(l1) == 0) { + va_next = (sva + L1_SIZE) & ~L1_OFFSET; + if (va_next < sva) + va_next = eva; + continue; + } + + /* + * Calculate index for next page table. + */ + va_next = (sva + L2_SIZE) & ~L2_OFFSET; + if (va_next < sva) + va_next = eva; + + l2 = pmap_l1_to_l2(l1, sva); + if (l2 == NULL) + continue; + + l3_pte = pmap_load(l2); + + /* + * Weed out invalid mappings. + */ + if (l3_pte == 0) + continue; + if ((pmap_load(l2) & PTE_TYPE_M) != (PTE_TYPE_PTR << PTE_TYPE_S)) + continue; + + /* + * Limit our scan to either the end of the va represented + * by the current page table page, or to the end of the + * range being removed. + */ + if (va_next > eva) + va_next = eva; + + va = va_next; + for (l3 = pmap_l2_to_l3(l2, sva); sva != va_next; l3++, + sva += L3_SIZE) { + if (l3 == NULL) + panic("l3 == NULL"); + if (pmap_load(l3) == 0) { + if (va != va_next) { + pmap_invalidate_range(pmap, va, sva); + va = va_next; + } + continue; + } + if (va == va_next) + va = sva; + if (pmap_remove_l3(pmap, l3, sva, l3_pte, &free, + &lock)) { + sva += L3_SIZE; + break; + } + } + if (va != va_next) + pmap_invalidate_range(pmap, va, sva); + } + if (lock != NULL) + rw_wunlock(lock); + if (anyvalid) + pmap_invalidate_all(pmap); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + pmap_free_zero_pages(&free); +} + +/* + * Routine: pmap_remove_all + * Function: + * Removes this physical page from + * all physical maps in which it resides. + * Reflects back modify bits to the pager. + * + * Notes: + * Original versions of this routine were very + * inefficient because they iteratively called + * pmap_remove (slow...) + */ + +void +pmap_remove_all(vm_page_t m) +{ + pv_entry_t pv; + pmap_t pmap; + pt_entry_t *l3, tl3; + pd_entry_t *l2, tl2; + struct spglist free; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_remove_all: page %p is not managed", m)); + SLIST_INIT(&free); + rw_wlock(&pvh_global_lock); + while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pmap_resident_count_dec(pmap, 1); + l2 = pmap_l2(pmap, pv->pv_va); + KASSERT(l2 != NULL, ("pmap_remove_all: no l2 table found")); + tl2 = pmap_load(l2); + + KASSERT((tl2 & PTE_TYPE_M) == (PTE_TYPE_PTR << PTE_TYPE_S), + ("pmap_remove_all: found a table when expecting " + "a block in %p's pv list", m)); + + l3 = pmap_l2_to_l3(l2, pv->pv_va); + if (pmap_is_current(pmap) && + pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(pv->pv_va, L3_SIZE); + tl3 = pmap_load_clear(l3); + PTE_SYNC(l3); + pmap_invalidate_page(pmap, pv->pv_va); + if (tl3 & PTE_SW_WIRED) + pmap->pm_stats.wired_count--; + if ((tl3 & PTE_REF) != 0) + vm_page_aflag_set(m, PGA_REFERENCED); + + /* + * Update the vm_page_t clean and reference bits. + */ + if (pmap_page_dirty(tl3)) + vm_page_dirty(m); + pmap_unuse_l3(pmap, pv->pv_va, pmap_load(l2), &free); + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + free_pv_entry(pmap, pv); + PMAP_UNLOCK(pmap); + } + vm_page_aflag_clear(m, PGA_WRITEABLE); + rw_wunlock(&pvh_global_lock); + pmap_free_zero_pages(&free); +} + +/* + * Set the physical protection on the + * specified range of this map as requested. + */ +void +pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) +{ + vm_offset_t va, va_next; + pd_entry_t *l1, *l2; + pt_entry_t *l3p, l3; + pt_entry_t entry; + + if ((prot & VM_PROT_READ) == VM_PROT_NONE) { + pmap_remove(pmap, sva, eva); + return; + } + + if ((prot & VM_PROT_WRITE) == VM_PROT_WRITE) + return; + + PMAP_LOCK(pmap); + for (; sva < eva; sva = va_next) { + + l1 = pmap_l1(pmap, sva); + if (pmap_load(l1) == 0) { + va_next = (sva + L1_SIZE) & ~L1_OFFSET; + if (va_next < sva) + va_next = eva; + continue; + } + + va_next = (sva + L2_SIZE) & ~L2_OFFSET; + if (va_next < sva) + va_next = eva; + + l2 = pmap_l1_to_l2(l1, sva); + if (l2 == NULL) + continue; + if ((pmap_load(l2) & PTE_TYPE_M) != (PTE_TYPE_PTR << PTE_TYPE_S)) + continue; + + if (va_next > eva) + va_next = eva; + + va = va_next; + for (l3p = pmap_l2_to_l3(l2, sva); sva != va_next; l3p++, + sva += L3_SIZE) { + l3 = pmap_load(l3p); + if (pmap_l3_valid(l3)) { + entry = pmap_load(l3p); + entry &= ~(1 << PTE_TYPE_S); + pmap_load_store(l3p, entry); + PTE_SYNC(l3p); + /* XXX: Use pmap_invalidate_range */ + pmap_invalidate_page(pmap, va); + } + } + } + PMAP_UNLOCK(pmap); + + /* TODO: Only invalidate entries we are touching */ + pmap_invalidate_all(pmap); +} + +/* + * Insert the given physical page (p) at + * the specified virtual address (v) in the + * target physical map with the protection requested. + * + * If specified, the page will be wired down, meaning + * that the related pte can not be reclaimed. + * + * NB: This is the only routine which MAY NOT lazy-evaluate + * or lose information. That is, this routine must actually + * insert this page into the given map NOW. + */ +int +pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, + u_int flags, int8_t psind __unused) +{ + struct rwlock *lock; + pd_entry_t *l1, *l2; + pt_entry_t new_l3, orig_l3; + pt_entry_t *l3; + pv_entry_t pv; + vm_paddr_t opa, pa, l2_pa, l3_pa; + vm_page_t mpte, om, l2_m, l3_m; + boolean_t nosleep; + pt_entry_t entry; + int l2_pn; + int l3_pn; + int pn; + + va = trunc_page(va); + if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m)) + VM_OBJECT_ASSERT_LOCKED(m->object); + pa = VM_PAGE_TO_PHYS(m); + pn = (pa / PAGE_SIZE); + + new_l3 = PTE_VALID; + + if ((prot & VM_PROT_WRITE) == 0) { /* Read-only */ + if ((va >> 63) == 0) /* USER */ + new_l3 |= (PTE_TYPE_SURX << PTE_TYPE_S); + else /* KERNEL */ + new_l3 |= (PTE_TYPE_SRX << PTE_TYPE_S); + } else { + if ((va >> 63) == 0) /* USER */ + new_l3 |= (PTE_TYPE_SURWX << PTE_TYPE_S); + else /* KERNEL */ + new_l3 |= (PTE_TYPE_SRWX << PTE_TYPE_S); + } + + new_l3 |= (pn << PTE_PPN0_S); + if ((flags & PMAP_ENTER_WIRED) != 0) + new_l3 |= PTE_SW_WIRED; + + CTR2(KTR_PMAP, "pmap_enter: %.16lx -> %.16lx", va, pa); + + mpte = NULL; + + lock = NULL; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + + if (va < VM_MAXUSER_ADDRESS) { + nosleep = (flags & PMAP_ENTER_NOSLEEP) != 0; + mpte = pmap_alloc_l3(pmap, va, nosleep ? NULL : &lock); + if (mpte == NULL && nosleep) { + CTR0(KTR_PMAP, "pmap_enter: mpte == NULL"); + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + return (KERN_RESOURCE_SHORTAGE); + } + l3 = pmap_l3(pmap, va); + } else { + l3 = pmap_l3(pmap, va); + /* TODO: This is not optimal, but should mostly work */ + if (l3 == NULL) { + l2 = pmap_l2(pmap, va); + if (l2 == NULL) { + l2_m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | + VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | + VM_ALLOC_ZERO); + if (l2_m == NULL) + panic("pmap_enter: l2 pte_m == NULL"); + if ((l2_m->flags & PG_ZERO) == 0) + pmap_zero_page(l2_m); + + l2_pa = VM_PAGE_TO_PHYS(l2_m); + l2_pn = (l2_pa / PAGE_SIZE); + + l1 = pmap_l1(pmap, va); + entry = (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)); + entry |= (l2_pn << PTE_PPN0_S); + pmap_load_store(l1, entry); + PTE_SYNC(l1); + + l2 = pmap_l1_to_l2(l1, va); + } + + KASSERT(l2 != NULL, + ("No l2 table after allocating one")); + + l3_m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | + VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO); + if (l3_m == NULL) + panic("pmap_enter: l3 pte_m == NULL"); + if ((l3_m->flags & PG_ZERO) == 0) + pmap_zero_page(l3_m); + + l3_pa = VM_PAGE_TO_PHYS(l3_m); + l3_pn = (l3_pa / PAGE_SIZE); + entry = (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)); + entry |= (l3_pn << PTE_PPN0_S); + pmap_load_store(l2, entry); + PTE_SYNC(l2); + l3 = pmap_l2_to_l3(l2, va); + } + pmap_invalidate_page(pmap, va); + } + + om = NULL; + orig_l3 = pmap_load(l3); + opa = PTE_TO_PHYS(orig_l3); + + /* + * Is the specified virtual address already mapped? + */ + if (pmap_l3_valid(orig_l3)) { + /* + * Wiring change, just update stats. We don't worry about + * wiring PT pages as they remain resident as long as there + * are valid mappings in them. Hence, if a user page is wired, + * the PT page will be also. + */ + if ((flags & PMAP_ENTER_WIRED) != 0 && + (orig_l3 & PTE_SW_WIRED) == 0) + pmap->pm_stats.wired_count++; + else if ((flags & PMAP_ENTER_WIRED) == 0 && + (orig_l3 & PTE_SW_WIRED) != 0) + pmap->pm_stats.wired_count--; + + /* + * Remove the extra PT page reference. + */ + if (mpte != NULL) { + mpte->wire_count--; + KASSERT(mpte->wire_count > 0, + ("pmap_enter: missing reference to page table page," + " va: 0x%lx", va)); + } + + /* + * Has the physical page changed? + */ + if (opa == pa) { + /* + * No, might be a protection or wiring change. + */ + if ((orig_l3 & PTE_SW_MANAGED) != 0) { + new_l3 |= PTE_SW_MANAGED; + if (pmap_is_write(new_l3)) + vm_page_aflag_set(m, PGA_WRITEABLE); + } + goto validate; + } + + /* Flush the cache, there might be uncommitted data in it */ + if (pmap_is_current(pmap) && pmap_l3_valid_cacheable(orig_l3)) + cpu_dcache_wb_range(va, L3_SIZE); + } else { + /* + * Increment the counters. + */ + if ((new_l3 & PTE_SW_WIRED) != 0) + pmap->pm_stats.wired_count++; + pmap_resident_count_inc(pmap, 1); + } + /* + * Enter on the PV list if part of our managed memory. + */ + if ((m->oflags & VPO_UNMANAGED) == 0) { + new_l3 |= PTE_SW_MANAGED; + pv = get_pv_entry(pmap, &lock); + pv->pv_va = va; + CHANGE_PV_LIST_LOCK_TO_PHYS(&lock, pa); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + if (pmap_is_write(new_l3)) + vm_page_aflag_set(m, PGA_WRITEABLE); + } + + /* + * Update the L3 entry. + */ + if (orig_l3 != 0) { +validate: + orig_l3 = pmap_load_store(l3, new_l3); + PTE_SYNC(l3); + opa = PTE_TO_PHYS(orig_l3); + + if (opa != pa) { + if ((orig_l3 & PTE_SW_MANAGED) != 0) { + om = PHYS_TO_VM_PAGE(opa); + if (pmap_page_dirty(orig_l3)) + vm_page_dirty(om); + if ((orig_l3 & PTE_REF) != 0) + vm_page_aflag_set(om, PGA_REFERENCED); + CHANGE_PV_LIST_LOCK_TO_PHYS(&lock, opa); + pmap_pvh_free(&om->md, pmap, va); + } + } else if (pmap_page_dirty(orig_l3)) { + if ((orig_l3 & PTE_SW_MANAGED) != 0) + vm_page_dirty(m); + } + } else { + pmap_load_store(l3, new_l3); + PTE_SYNC(l3); + } + pmap_invalidate_page(pmap, va); + if ((pmap != pmap_kernel()) && (pmap == &curproc->p_vmspace->vm_pmap)) + cpu_icache_sync_range(va, PAGE_SIZE); + + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + return (KERN_SUCCESS); +} + +/* + * Maps a sequence of resident pages belonging to the same object. + * The sequence begins with the given page m_start. This page is + * mapped at the given virtual address start. Each subsequent page is + * mapped at a virtual address that is offset from start by the same + * amount as the page is offset from m_start within the object. The + * last page in the sequence is the page with the largest offset from + * m_start that can be mapped at a virtual address less than the given + * virtual address end. Not every virtual page between start and end + * is mapped; only those for which a resident page exists with the + * corresponding offset from m_start are mapped. + */ +void +pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, + vm_page_t m_start, vm_prot_t prot) +{ + struct rwlock *lock; + vm_offset_t va; + vm_page_t m, mpte; + vm_pindex_t diff, psize; + + VM_OBJECT_ASSERT_LOCKED(m_start->object); + + psize = atop(end - start); + mpte = NULL; + m = m_start; + lock = NULL; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { + va = start + ptoa(diff); + mpte = pmap_enter_quick_locked(pmap, va, m, prot, mpte, &lock); + m = TAILQ_NEXT(m, listq); + } + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); +} + +/* + * this code makes some *MAJOR* assumptions: + * 1. Current pmap & pmap exists. + * 2. Not wired. + * 3. Read access. + * 4. No page table pages. + * but is *MUCH* faster than pmap_enter... + */ + +void +pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) +{ + struct rwlock *lock; + + lock = NULL; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + (void)pmap_enter_quick_locked(pmap, va, m, prot, NULL, &lock); + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); +} + +static vm_page_t +pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, + vm_prot_t prot, vm_page_t mpte, struct rwlock **lockp) +{ + struct spglist free; + vm_paddr_t phys; + pd_entry_t *l2; + pt_entry_t *l3; + vm_paddr_t pa; + pt_entry_t entry; + int pn; + + KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva || + (m->oflags & VPO_UNMANAGED) != 0, + ("pmap_enter_quick_locked: managed mapping within the clean submap")); + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + + CTR2(KTR_PMAP, "pmap_enter_quick_locked: %p %lx", pmap, va); + /* + * In the case that a page table page is not + * resident, we are creating it here. + */ + if (va < VM_MAXUSER_ADDRESS) { + vm_pindex_t l2pindex; + + /* + * Calculate pagetable page index + */ + l2pindex = pmap_l2_pindex(va); + if (mpte && (mpte->pindex == l2pindex)) { + mpte->wire_count++; + } else { + /* + * Get the l2 entry + */ + l2 = pmap_l2(pmap, va); + + /* + * If the page table page is mapped, we just increment + * the hold count, and activate it. Otherwise, we + * attempt to allocate a page table page. If this + * attempt fails, we don't retry. Instead, we give up. + */ + if (l2 != NULL && pmap_load(l2) != 0) { + phys = PTE_TO_PHYS(pmap_load(l2)); + mpte = PHYS_TO_VM_PAGE(phys); + mpte->wire_count++; + } else { + /* + * Pass NULL instead of the PV list lock + * pointer, because we don't intend to sleep. + */ + mpte = _pmap_alloc_l3(pmap, l2pindex, NULL); + if (mpte == NULL) + return (mpte); + } + } + l3 = (pt_entry_t *)PHYS_TO_DMAP(VM_PAGE_TO_PHYS(mpte)); + l3 = &l3[pmap_l3_index(va)]; + } else { + mpte = NULL; + l3 = pmap_l3(kernel_pmap, va); + } + if (l3 == NULL) + panic("pmap_enter_quick_locked: No l3"); + if (pmap_load(l3) != 0) { + if (mpte != NULL) { + mpte->wire_count--; + mpte = NULL; + } + return (mpte); + } + + /* + * Enter on the PV list if part of our managed memory. + */ + if ((m->oflags & VPO_UNMANAGED) == 0 && + !pmap_try_insert_pv_entry(pmap, va, m, lockp)) { + if (mpte != NULL) { + SLIST_INIT(&free); + if (pmap_unwire_l3(pmap, va, mpte, &free)) { + pmap_invalidate_page(pmap, va); + pmap_free_zero_pages(&free); + } + mpte = NULL; + } + return (mpte); + } + + /* + * Increment counters + */ + pmap_resident_count_inc(pmap, 1); + + pa = VM_PAGE_TO_PHYS(m); + pn = (pa / PAGE_SIZE); + + /* RISCVTODO: check permissions */ + entry = (PTE_VALID | (PTE_TYPE_SRWX << PTE_TYPE_S)); + entry |= (pn << PTE_PPN0_S); + + /* + * Now validate mapping with RO protection + */ + if ((m->oflags & VPO_UNMANAGED) == 0) + entry |= PTE_SW_MANAGED; + pmap_load_store(l3, entry); + + PTE_SYNC(l3); + pmap_invalidate_page(pmap, va); + return (mpte); +} + +/* + * This code maps large physical mmap regions into the + * processor address space. Note that some shortcuts + * are taken, but the code works. + */ +void +pmap_object_init_pt(pmap_t pmap, vm_offset_t addr, vm_object_t object, + vm_pindex_t pindex, vm_size_t size) +{ + + VM_OBJECT_ASSERT_WLOCKED(object); + KASSERT(object->type == OBJT_DEVICE || object->type == OBJT_SG, + ("pmap_object_init_pt: non-device object")); +} + +/* + * Clear the wired attribute from the mappings for the specified range of + * addresses in the given pmap. Every valid mapping within that range + * must have the wired attribute set. In contrast, invalid mappings + * cannot have the wired attribute set, so they are ignored. + * + * The wired attribute of the page table entry is not a hardware feature, + * so there is no need to invalidate any TLB entries. + */ +void +pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +{ + vm_offset_t va_next; + pd_entry_t *l1, *l2; + pt_entry_t *l3; + boolean_t pv_lists_locked; + + pv_lists_locked = FALSE; + PMAP_LOCK(pmap); + for (; sva < eva; sva = va_next) { + l1 = pmap_l1(pmap, sva); + if (pmap_load(l1) == 0) { + va_next = (sva + L1_SIZE) & ~L1_OFFSET; + if (va_next < sva) + va_next = eva; + continue; + } + + va_next = (sva + L2_SIZE) & ~L2_OFFSET; + if (va_next < sva) + va_next = eva; + + l2 = pmap_l1_to_l2(l1, sva); + if (pmap_load(l2) == 0) + continue; + + if (va_next > eva) + va_next = eva; + for (l3 = pmap_l2_to_l3(l2, sva); sva != va_next; l3++, + sva += L3_SIZE) { + if (pmap_load(l3) == 0) + continue; + if ((pmap_load(l3) & PTE_SW_WIRED) == 0) + panic("pmap_unwire: l3 %#jx is missing " + "PTE_SW_WIRED", (uintmax_t)*l3); + + /* + * PG_W must be cleared atomically. Although the pmap + * lock synchronizes access to PG_W, another processor + * could be setting PG_M and/or PG_A concurrently. + */ + atomic_clear_long(l3, PTE_SW_WIRED); + pmap->pm_stats.wired_count--; + } + } + if (pv_lists_locked) + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); +} + +/* + * Copy the range specified by src_addr/len + * from the source map to the range dst_addr/len + * in the destination map. + * + * This routine is only advisory and need not do anything. + */ + +void +pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t len, + vm_offset_t src_addr) +{ + +} + +/* + * pmap_zero_page zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. + */ +void +pmap_zero_page(vm_page_t m) +{ + vm_offset_t va = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m)); + + pagezero((void *)va); +} + +/* + * pmap_zero_page_area zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. + * + * off and size may not cover an area beyond a single hardware page. + */ +void +pmap_zero_page_area(vm_page_t m, int off, int size) +{ + vm_offset_t va = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m)); + + if (off == 0 && size == PAGE_SIZE) + pagezero((void *)va); + else + bzero((char *)va + off, size); +} + +/* + * pmap_zero_page_idle zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. This + * is intended to be called from the vm_pagezero process only and + * outside of Giant. + */ +void +pmap_zero_page_idle(vm_page_t m) +{ + vm_offset_t va = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m)); + + pagezero((void *)va); +} + +/* + * pmap_copy_page copies the specified (machine independent) + * page by mapping the page into virtual memory and using + * bcopy to copy the page, one machine dependent page at a + * time. + */ +void +pmap_copy_page(vm_page_t msrc, vm_page_t mdst) +{ + vm_offset_t src = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(msrc)); + vm_offset_t dst = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(mdst)); + + pagecopy((void *)src, (void *)dst); +} + +int unmapped_buf_allowed = 1; + +void +pmap_copy_pages(vm_page_t ma[], vm_offset_t a_offset, vm_page_t mb[], + vm_offset_t b_offset, int xfersize) +{ + void *a_cp, *b_cp; + vm_page_t m_a, m_b; + vm_paddr_t p_a, p_b; + vm_offset_t a_pg_offset, b_pg_offset; + int cnt; + + while (xfersize > 0) { + a_pg_offset = a_offset & PAGE_MASK; + m_a = ma[a_offset >> PAGE_SHIFT]; + p_a = m_a->phys_addr; + b_pg_offset = b_offset & PAGE_MASK; + m_b = mb[b_offset >> PAGE_SHIFT]; + p_b = m_b->phys_addr; + cnt = min(xfersize, PAGE_SIZE - a_pg_offset); + cnt = min(cnt, PAGE_SIZE - b_pg_offset); + if (__predict_false(!PHYS_IN_DMAP(p_a))) { + panic("!DMAP a %lx", p_a); + } else { + a_cp = (char *)PHYS_TO_DMAP(p_a) + a_pg_offset; + } + if (__predict_false(!PHYS_IN_DMAP(p_b))) { + panic("!DMAP b %lx", p_b); + } else { + b_cp = (char *)PHYS_TO_DMAP(p_b) + b_pg_offset; + } + bcopy(a_cp, b_cp, cnt); + a_offset += cnt; + b_offset += cnt; + xfersize -= cnt; + } +} + +vm_offset_t +pmap_quick_enter_page(vm_page_t m) +{ + + return (PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m))); +} + +void +pmap_quick_remove_page(vm_offset_t addr) +{ +} + +/* + * Returns true if the pmap's pv is one of the first + * 16 pvs linked to from this page. This count may + * be changed upwards or downwards in the future; it + * is only necessary that true be returned for a small + * subset of pmaps for proper page aging. + */ +boolean_t +pmap_page_exists_quick(pmap_t pmap, vm_page_t m) +{ + struct rwlock *lock; + pv_entry_t pv; + int loops = 0; + boolean_t rv; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_page_exists_quick: page %p is not managed", m)); + rv = FALSE; + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); + rw_rlock(lock); + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + if (PV_PMAP(pv) == pmap) { + rv = TRUE; + break; + } + loops++; + if (loops >= 16) + break; + } + rw_runlock(lock); + rw_runlock(&pvh_global_lock); + return (rv); +} + +/* + * pmap_page_wired_mappings: + * + * Return the number of managed mappings to the given physical page + * that are wired. + */ +int +pmap_page_wired_mappings(vm_page_t m) +{ + struct rwlock *lock; + pmap_t pmap; + pt_entry_t *l3; + pv_entry_t pv; + int count, md_gen; + + if ((m->oflags & VPO_UNMANAGED) != 0) + return (0); + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); + rw_rlock(lock); +restart: + count = 0; + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_runlock(lock); + PMAP_LOCK(pmap); + rw_rlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + goto restart; + } + } + l3 = pmap_l3(pmap, pv->pv_va); + if (l3 != NULL && (pmap_load(l3) & PTE_SW_WIRED) != 0) + count++; + PMAP_UNLOCK(pmap); + } + rw_runlock(lock); + rw_runlock(&pvh_global_lock); + return (count); +} + +/* + * Destroy all managed, non-wired mappings in the given user-space + * pmap. This pmap cannot be active on any processor besides the + * caller. + * + * This function cannot be applied to the kernel pmap. Moreover, it + * is not intended for general use. It is only to be used during + * process termination. Consequently, it can be implemented in ways + * that make it faster than pmap_remove(). First, it can more quickly + * destroy mappings by iterating over the pmap's collection of PV + * entries, rather than searching the page table. Second, it doesn't + * have to test and clear the page table entries atomically, because + * no processor is currently accessing the user address space. In + * particular, a page table entry's dirty bit won't change state once + * this function starts. + */ +void +pmap_remove_pages(pmap_t pmap) +{ + pd_entry_t ptepde, *l2; + pt_entry_t *l3, tl3; + struct spglist free; + vm_page_t m; + pv_entry_t pv; + struct pv_chunk *pc, *npc; + struct rwlock *lock; + int64_t bit; + uint64_t inuse, bitmask; + int allfree, field, freed, idx; + vm_paddr_t pa; + + lock = NULL; + + SLIST_INIT(&free); + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { + allfree = 1; + freed = 0; + for (field = 0; field < _NPCM; field++) { + inuse = ~pc->pc_map[field] & pc_freemask[field]; + while (inuse != 0) { + bit = ffsl(inuse) - 1; + bitmask = 1UL << bit; + idx = field * 64 + bit; + pv = &pc->pc_pventry[idx]; + inuse &= ~bitmask; + + l2 = pmap_l2(pmap, pv->pv_va); + ptepde = pmap_load(l2); + l3 = pmap_l2_to_l3(l2, pv->pv_va); + tl3 = pmap_load(l3); + +/* + * We cannot remove wired pages from a process' mapping at this time + */ + if (tl3 & PTE_SW_WIRED) { + allfree = 0; + continue; + } + + pa = PTE_TO_PHYS(tl3); + m = PHYS_TO_VM_PAGE(pa); + KASSERT(m->phys_addr == pa, + ("vm_page_t %p phys_addr mismatch %016jx %016jx", + m, (uintmax_t)m->phys_addr, + (uintmax_t)tl3)); + + KASSERT((m->flags & PG_FICTITIOUS) != 0 || + m < &vm_page_array[vm_page_array_size], + ("pmap_remove_pages: bad l3 %#jx", + (uintmax_t)tl3)); + + if (pmap_is_current(pmap) && + pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(pv->pv_va, L3_SIZE); + pmap_load_clear(l3); + PTE_SYNC(l3); + pmap_invalidate_page(pmap, pv->pv_va); + + /* + * Update the vm_page_t clean/reference bits. + */ + if (pmap_page_dirty(tl3)) + vm_page_dirty(m); + + CHANGE_PV_LIST_LOCK_TO_VM_PAGE(&lock, m); + + /* Mark free */ + pc->pc_map[field] |= bitmask; + + pmap_resident_count_dec(pmap, 1); + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + + pmap_unuse_l3(pmap, pv->pv_va, ptepde, &free); + freed++; + } + } + PV_STAT(atomic_add_long(&pv_entry_frees, freed)); + PV_STAT(atomic_add_int(&pv_entry_spare, freed)); + PV_STAT(atomic_subtract_long(&pv_entry_count, freed)); + if (allfree) { + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + free_pv_chunk(pc); + } + } + pmap_invalidate_all(pmap); + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + pmap_free_zero_pages(&free); +} + +/* + * This is used to check if a page has been accessed or modified. As we + * don't have a bit to see if it has been modified we have to assume it + * has been if the page is read/write. + */ +static boolean_t +pmap_page_test_mappings(vm_page_t m, boolean_t accessed, boolean_t modified) +{ + struct rwlock *lock; + pv_entry_t pv; + pt_entry_t *l3, mask, value; + pmap_t pmap; + int md_gen; + boolean_t rv; + + rv = FALSE; + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); + rw_rlock(lock); +restart: + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_runlock(lock); + PMAP_LOCK(pmap); + rw_rlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + goto restart; + } + } + l3 = pmap_l3(pmap, pv->pv_va); + mask = 0; + value = 0; + if (modified) { + mask |= PTE_DIRTY; + value |= PTE_DIRTY; + } + if (accessed) { + mask |= PTE_REF; + value |= PTE_REF; + } + +#if 0 + if (modified) { + mask |= ATTR_AP_RW_BIT; + value |= ATTR_AP(ATTR_AP_RW); + } + if (accessed) { + mask |= ATTR_AF | ATTR_DESCR_MASK; + value |= ATTR_AF | L3_PAGE; + } +#endif + + rv = (pmap_load(l3) & mask) == value; + PMAP_UNLOCK(pmap); + if (rv) + goto out; + } +out: + rw_runlock(lock); + rw_runlock(&pvh_global_lock); + return (rv); +} + +/* + * pmap_is_modified: + * + * Return whether or not the specified physical page was modified + * in any physical maps. + */ +boolean_t +pmap_is_modified(vm_page_t m) +{ + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_is_modified: page %p is not managed", m)); + + /* + * If the page is not exclusive busied, then PGA_WRITEABLE cannot be + * concurrently set while the object is locked. Thus, if PGA_WRITEABLE + * is clear, no PTEs can have PG_M set. + */ + VM_OBJECT_ASSERT_WLOCKED(m->object); + if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) + return (FALSE); + return (pmap_page_test_mappings(m, FALSE, TRUE)); +} + +/* + * pmap_is_prefaultable: + * + * Return whether or not the specified virtual address is eligible + * for prefault. + */ +boolean_t +pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr) +{ + pt_entry_t *l3; + boolean_t rv; + + rv = FALSE; + PMAP_LOCK(pmap); + l3 = pmap_l3(pmap, addr); + if (l3 != NULL && pmap_load(l3) != 0) { + rv = TRUE; + } + PMAP_UNLOCK(pmap); + return (rv); +} + +/* + * pmap_is_referenced: + * + * Return whether or not the specified physical page was referenced + * in any physical maps. + */ +boolean_t +pmap_is_referenced(vm_page_t m) +{ + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_is_referenced: page %p is not managed", m)); + return (pmap_page_test_mappings(m, TRUE, FALSE)); +} + +/* + * Clear the write and modified bits in each of the given page's mappings. + */ +void +pmap_remove_write(vm_page_t m) +{ + pmap_t pmap; + struct rwlock *lock; + pv_entry_t pv; + pt_entry_t *l3, oldl3; + pt_entry_t newl3; + int md_gen; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_remove_write: page %p is not managed", m)); + + /* + * If the page is not exclusive busied, then PGA_WRITEABLE cannot be + * set by another thread while the object is locked. Thus, + * if PGA_WRITEABLE is clear, no page table entries need updating. + */ + VM_OBJECT_ASSERT_WLOCKED(m->object); + if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) + return; + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); +retry_pv_loop: + rw_wlock(lock); + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_wunlock(lock); + PMAP_LOCK(pmap); + rw_wlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + rw_wunlock(lock); + goto retry_pv_loop; + } + } + l3 = pmap_l3(pmap, pv->pv_va); +retry: + oldl3 = pmap_load(l3); + + if (pmap_is_write(oldl3)) { + newl3 = oldl3 & ~(1 << PTE_TYPE_S); + if (!atomic_cmpset_long(l3, oldl3, newl3)) + goto retry; + /* TODO: use pmap_page_dirty(oldl3) ? */ + if ((oldl3 & PTE_REF) != 0) + vm_page_dirty(m); + pmap_invalidate_page(pmap, pv->pv_va); + } + PMAP_UNLOCK(pmap); + } + rw_wunlock(lock); + vm_page_aflag_clear(m, PGA_WRITEABLE); + rw_runlock(&pvh_global_lock); +} + +static __inline boolean_t +safe_to_clear_referenced(pmap_t pmap, pt_entry_t pte) +{ + + return (FALSE); +} + +#define PMAP_TS_REFERENCED_MAX 5 + +/* + * pmap_ts_referenced: + * + * Return a count of reference bits for a page, clearing those bits. + * It is not necessary for every reference bit to be cleared, but it + * is necessary that 0 only be returned when there are truly no + * reference bits set. + * + * XXX: The exact number of bits to check and clear is a matter that + * should be tested and standardized at some point in the future for + * optimal aging of shared pages. + */ +int +pmap_ts_referenced(vm_page_t m) +{ + pv_entry_t pv, pvf; + pmap_t pmap; + struct rwlock *lock; + pd_entry_t *l2; + pt_entry_t *l3; + vm_paddr_t pa; + int cleared, md_gen, not_cleared; + struct spglist free; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_ts_referenced: page %p is not managed", m)); + SLIST_INIT(&free); + cleared = 0; + pa = VM_PAGE_TO_PHYS(m); + lock = PHYS_TO_PV_LIST_LOCK(pa); + rw_rlock(&pvh_global_lock); + rw_wlock(lock); +retry: + not_cleared = 0; + if ((pvf = TAILQ_FIRST(&m->md.pv_list)) == NULL) + goto out; + pv = pvf; + do { + if (pvf == NULL) + pvf = pv; + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_wunlock(lock); + PMAP_LOCK(pmap); + rw_wlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + goto retry; + } + } + l2 = pmap_l2(pmap, pv->pv_va); + + KASSERT((pmap_load(l2) & PTE_TYPE_M) == (PTE_TYPE_PTR << PTE_TYPE_S), + ("pmap_ts_referenced: found an invalid l2 table")); + + l3 = pmap_l2_to_l3(l2, pv->pv_va); + if ((pmap_load(l3) & PTE_REF) != 0) { + if (safe_to_clear_referenced(pmap, pmap_load(l3))) { + /* + * TODO: We don't handle the access flag + * at all. We need to be able to set it in + * the exception handler. + */ + panic("RISCVTODO: safe_to_clear_referenced\n"); + } else if ((pmap_load(l3) & PTE_SW_WIRED) == 0) { + /* + * Wired pages cannot be paged out so + * doing accessed bit emulation for + * them is wasted effort. We do the + * hard work for unwired pages only. + */ + pmap_remove_l3(pmap, l3, pv->pv_va, + pmap_load(l2), &free, &lock); + pmap_invalidate_page(pmap, pv->pv_va); + cleared++; + if (pvf == pv) + pvf = NULL; + pv = NULL; + KASSERT(lock == VM_PAGE_TO_PV_LIST_LOCK(m), + ("inconsistent pv lock %p %p for page %p", + lock, VM_PAGE_TO_PV_LIST_LOCK(m), m)); + } else + not_cleared++; + } + PMAP_UNLOCK(pmap); + /* Rotate the PV list if it has more than one entry. */ + if (pv != NULL && TAILQ_NEXT(pv, pv_next) != NULL) { + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + } + } while ((pv = TAILQ_FIRST(&m->md.pv_list)) != pvf && cleared + + not_cleared < PMAP_TS_REFERENCED_MAX); +out: + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + pmap_free_zero_pages(&free); + return (cleared + not_cleared); +} + +/* + * Apply the given advice to the specified range of addresses within the + * given pmap. Depending on the advice, clear the referenced and/or + * modified flags in each mapping and set the mapped page's dirty field. + */ +void +pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice) +{ +} + +/* + * Clear the modify bits on the specified physical page. + */ +void +pmap_clear_modify(vm_page_t m) +{ + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_clear_modify: page %p is not managed", m)); + VM_OBJECT_ASSERT_WLOCKED(m->object); + KASSERT(!vm_page_xbusied(m), + ("pmap_clear_modify: page %p is exclusive busied", m)); + + /* + * If the page is not PGA_WRITEABLE, then no PTEs can have PG_M set. + * If the object containing the page is locked and the page is not + * exclusive busied, then PGA_WRITEABLE cannot be concurrently set. + */ + if ((m->aflags & PGA_WRITEABLE) == 0) + return; + + /* RISCVTODO: We lack support for tracking if a page is modified */ +} + +void * +pmap_mapbios(vm_paddr_t pa, vm_size_t size) +{ + + return ((void *)PHYS_TO_DMAP(pa)); +} + +void +pmap_unmapbios(vm_paddr_t pa, vm_size_t size) +{ +} + +/* + * Sets the memory attribute for the specified page. + */ +void +pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma) +{ + + m->md.pv_memattr = ma; + + /* + * RISCVTODO: Implement the below (from the amd64 pmap) + * If "m" is a normal page, update its direct mapping. This update + * can be relied upon to perform any cache operations that are + * required for data coherence. + */ + if ((m->flags & PG_FICTITIOUS) == 0 && + PHYS_IN_DMAP(VM_PAGE_TO_PHYS(m))) + panic("RISCVTODO: pmap_page_set_memattr"); +} + +/* + * perform the pmap work for mincore + */ +int +pmap_mincore(pmap_t pmap, vm_offset_t addr, vm_paddr_t *locked_pa) +{ + + panic("RISCVTODO: pmap_mincore"); +} + +void +pmap_activate(struct thread *td) +{ + uint64_t entry; + uint64_t pn; + pmap_t pmap; + + critical_enter(); + pmap = vmspace_pmap(td->td_proc->p_vmspace); + td->td_pcb->pcb_l1addr = vtophys(pmap->pm_l1); + + pn = (td->td_pcb->pcb_l1addr / PAGE_SIZE); + entry = (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)); + entry |= (pn << PTE_PPN0_S); + pmap_load_store(&pagetable_l0, entry); + + pmap_invalidate_all(pmap); + critical_exit(); +} + +void +pmap_sync_icache(pmap_t pm, vm_offset_t va, vm_size_t sz) +{ + + panic("RISCVTODO: pmap_sync_icache"); +} + +/* + * Increase the starting virtual address of the given mapping if a + * different alignment might result in more superpage mappings. + */ +void +pmap_align_superpage(vm_object_t object, vm_ooffset_t offset, + vm_offset_t *addr, vm_size_t size) +{ +} + +/** + * Get the kernel virtual address of a set of physical pages. If there are + * physical addresses not covered by the DMAP perform a transient mapping + * that will be removed when calling pmap_unmap_io_transient. + * + * \param page The pages the caller wishes to obtain the virtual + * address on the kernel memory map. + * \param vaddr On return contains the kernel virtual memory address + * of the pages passed in the page parameter. + * \param count Number of pages passed in. + * \param can_fault TRUE if the thread using the mapped pages can take + * page faults, FALSE otherwise. + * + * \returns TRUE if the caller must call pmap_unmap_io_transient when + * finished or FALSE otherwise. + * + */ +boolean_t +pmap_map_io_transient(vm_page_t page[], vm_offset_t vaddr[], int count, + boolean_t can_fault) +{ + vm_paddr_t paddr; + boolean_t needs_mapping; + int error, i; + + /* + * Allocate any KVA space that we need, this is done in a separate + * loop to prevent calling vmem_alloc while pinned. + */ + needs_mapping = FALSE; + for (i = 0; i < count; i++) { + paddr = VM_PAGE_TO_PHYS(page[i]); + if (__predict_false(paddr >= DMAP_MAX_PHYSADDR)) { + error = vmem_alloc(kernel_arena, PAGE_SIZE, + M_BESTFIT | M_WAITOK, &vaddr[i]); + KASSERT(error == 0, ("vmem_alloc failed: %d", error)); + needs_mapping = TRUE; + } else { + vaddr[i] = PHYS_TO_DMAP(paddr); + } + } + + /* Exit early if everything is covered by the DMAP */ + if (!needs_mapping) + return (FALSE); + + if (!can_fault) + sched_pin(); + for (i = 0; i < count; i++) { + paddr = VM_PAGE_TO_PHYS(page[i]); + if (paddr >= DMAP_MAX_PHYSADDR) { + panic( + "pmap_map_io_transient: TODO: Map out of DMAP data"); + } + } + + return (needs_mapping); +} + +void +pmap_unmap_io_transient(vm_page_t page[], vm_offset_t vaddr[], int count, + boolean_t can_fault) +{ + vm_paddr_t paddr; + int i; + + if (!can_fault) + sched_unpin(); + for (i = 0; i < count; i++) { + paddr = VM_PAGE_TO_PHYS(page[i]); + if (paddr >= DMAP_MAX_PHYSADDR) { + panic("RISCVTODO: pmap_unmap_io_transient: Unmap data"); + } + } +} Property changes on: user/ngie/make_check/sys/riscv/riscv/pmap.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/support.S =================================================================== --- user/ngie/make_check/sys/riscv/riscv/support.S (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/support.S (revision 295078) @@ -0,0 +1,295 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include + +#include "assym.s" + +/* + * One of the fu* or su* functions failed, return -1. + */ +ENTRY(fsu_fault) + SET_FAULT_HANDLER(x0, a1) /* Reset the handler function */ +fsu_fault_nopcb: + li a0, -1 + ret +END(fsu_fault) + +/* + * int casueword32(volatile uint32_t *, uint32_t, uint32_t *, uint32_t) + */ +ENTRY(casueword32) + li a4, (VM_MAXUSER_ADDRESS-3) + bgt a0, a4, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a4) /* And set it */ +1: lr.w a4, 0(a0) /* Load-exclusive the data */ + bne a4, a1, 2f /* If not equal then exit */ + sc.w a5, a3, 0(a0) /* Store the new data */ + bnez a5, 1b /* Retry on failure */ +2: SET_FAULT_HANDLER(x0, a5) /* Reset the fault handler */ + sw a4, 0(a2) /* Store the read data */ + li a0, 0 /* Success */ + ret /* Return */ +END(casueword32) + +/* + * int casueword(volatile u_long *, u_long, u_long *, u_long) + */ +ENTRY(casueword) + li a4, (VM_MAXUSER_ADDRESS-7) + bgt a0, a4, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a4) /* And set it */ +1: lr.d a4, 0(a0) /* Load-exclusive the data */ + bne a4, a1, 2f /* If not equal then exit */ + sc.d a5, a3, 0(a0) /* Store the new data */ + bnez a5, 1b /* Retry on failure */ +2: SET_FAULT_HANDLER(x0, a5) /* Reset the fault handler */ + sd a4, 0(a2) /* Store the read data */ + li a0, 0 /* Success */ + ret /* Return */ +END(casueword) + +/* + * int fubyte(volatile const void *) + */ +ENTRY(fubyte) + li a1, VM_MAXUSER_ADDRESS + bgt a0, a1, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a1) /* And set it */ + lb a0, 0(a0) /* Try loading the data */ + SET_FAULT_HANDLER(x0, a1) /* Reset the fault handler */ + ret /* Return */ +END(fubyte) + +/* + * int fuword(volatile const void *) + */ +ENTRY(fuword16) + li a1, (VM_MAXUSER_ADDRESS-1) + bgt a0, a1, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a1) /* And set it */ + lh a0, 0(a0) /* Try loading the data */ + SET_FAULT_HANDLER(x0, a1) /* Reset the fault handler */ + ret /* Return */ +END(fuword16) + +/* + * int32_t fueword32(volatile const void *, int32_t *) + */ +ENTRY(fueword32) + li a2, (VM_MAXUSER_ADDRESS-3) + bgt a0, a2, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a2) /* And set it */ + lw a0, 0(a0) /* Try loading the data */ + SET_FAULT_HANDLER(x0, a2) /* Reset the fault handler */ + sw a0, 0(a1) /* Save the data in kernel space */ + li a0, 0 /* Success */ + ret /* Return */ +END(fueword32) + +/* + * long fueword(volatile const void *, int64_t *) + * int64_t fueword64(volatile const void *, int64_t *) + */ +ENTRY(fueword) +EENTRY(fueword64) + li a2, (VM_MAXUSER_ADDRESS-7) + bgt a0, a2, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a2) /* And set it */ + ld a0, 0(a0) /* Try loading the data */ + SET_FAULT_HANDLER(x0, a2) /* Reset the fault handler */ + sd a0, 0(a1) /* Save the data in kernel space */ + li a0, 0 /* Success */ + ret /* Return */ +EEND(fueword64) +END(fueword) + +/* + * int subyte(volatile void *, int) + */ +ENTRY(subyte) + li a2, VM_MAXUSER_ADDRESS + bgt a0, a2, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a2) /* And set it */ + sb a1, 0(a0) /* Try storing the data */ + SET_FAULT_HANDLER(x0, a2) /* Reset the fault handler */ + li a0, 0 /* Success */ + ret /* Return */ +END(subyte) + +/* + * int suword16(volatile void *, int) + */ +ENTRY(suword16) + li a2, (VM_MAXUSER_ADDRESS-1) + bgt a0, a2, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a2) /* And set it */ + sh a1, 0(a0) /* Try storing the data */ + SET_FAULT_HANDLER(x0, a2) /* Reset the fault handler */ + li a0, 0 /* Success */ + ret /* Return */ +END(suword16) + +/* + * int suword32(volatile void *, int) + */ +ENTRY(suword32) + li a2, (VM_MAXUSER_ADDRESS-3) + bgt a0, a2, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a2) /* And set it */ + sw a1, 0(a0) /* Try storing the data */ + SET_FAULT_HANDLER(x0, a2) /* Reset the fault handler */ + li a0, 0 /* Success */ + ret /* Return */ +END(suword32) + +/* + * int suword(volatile void *, long) + */ +ENTRY(suword) +EENTRY(suword64) + li a2, (VM_MAXUSER_ADDRESS-7) + bgt a0, a2, fsu_fault_nopcb + la a6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a2) /* And set it */ + sd a1, 0(a0) /* Try storing the data */ + SET_FAULT_HANDLER(x0, a2) /* Reset the fault handler */ + li a0, 0 /* Success */ + ret /* Return */ +EEND(suword64) +END(suword) + +/* + * fuswintr and suswintr are just like fusword and susword except that if + * the page is not in memory or would cause a trap, then we return an error. + * The important thing is to prevent sleep() and switch(). + */ + +/* + * Special handler so the trap code knows not to sleep. + */ +ENTRY(fsu_intr_fault) + SET_FAULT_HANDLER(x0, a1) /* Reset the handler function */ + li a0, -1 + ret +END(fsu_fault) + +/* + * int fuswintr(void *) + */ +ENTRY(fuswintr) + li a1, (VM_MAXUSER_ADDRESS-3) + bgt a0, a1, fsu_fault_nopcb + la a6, fsu_intr_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a1) /* And set it */ + lw a0, 0(a0) /* Try loading the data */ + SET_FAULT_HANDLER(x0, x1) /* Reset the fault handler */ + ret /* Return */ +END(fuswintr) + +/* + * int suswintr(void *base, int word) + */ +ENTRY(suswintr) + li a2, (VM_MAXUSER_ADDRESS-3) + bgt a0, a2, fsu_fault_nopcb + la a6, fsu_intr_fault /* Load the fault handler */ + SET_FAULT_HANDLER(a6, a2) /* And set it */ + sw a1, 0(a0) /* Try storing the data */ + SET_FAULT_HANDLER(x0, a2) /* Reset the fault handler */ + li a0, 0 /* Success */ + ret /* Return */ +END(suswintr) + +ENTRY(setjmp) + /* Store the stack pointer */ + sd sp, 0(a0) + addi a0, a0, 8 + + /* Store the general purpose registers and ra */ + sd s0, (0 * 8)(a0) + sd s1, (1 * 8)(a0) + sd s2, (2 * 8)(a0) + sd s3, (3 * 8)(a0) + sd s4, (4 * 8)(a0) + sd s5, (5 * 8)(a0) + sd s6, (6 * 8)(a0) + sd s7, (7 * 8)(a0) + sd s8, (8 * 8)(a0) + sd s9, (9 * 8)(a0) + sd s10, (10 * 8)(a0) + sd s11, (11 * 8)(a0) + sd ra, (12 * 8)(a0) + + /* Return value */ + li a0, 0 + ret +END(setjmp) + +ENTRY(longjmp) + /* Restore the stack pointer */ + ld sp, 0(a0) + addi a0, a0, 8 + + /* Restore the general purpose registers and ra */ + ld s0, (0 * 8)(a0) + ld s1, (1 * 8)(a0) + ld s2, (2 * 8)(a0) + ld s3, (3 * 8)(a0) + ld s4, (4 * 8)(a0) + ld s5, (5 * 8)(a0) + ld s6, (6 * 8)(a0) + ld s7, (7 * 8)(a0) + ld s8, (8 * 8)(a0) + ld s9, (9 * 8)(a0) + ld s10, (10 * 8)(a0) + ld s11, (11 * 8)(a0) + ld ra, (12 * 8)(a0) + + /* Load the return value */ + mv a0, a1 + ret +END(longjmp) Property changes on: user/ngie/make_check/sys/riscv/riscv/support.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/swtch.S =================================================================== --- user/ngie/make_check/sys/riscv/riscv/swtch.S (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/swtch.S (revision 295078) @@ -0,0 +1,272 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include "assym.s" +#include "opt_sched.h" + +#include +#include +#include +#include + +__FBSDID("$FreeBSD$"); + +/* + * void cpu_throw(struct thread *old, struct thread *new) + */ +ENTRY(cpu_throw) + /* Load pcpu */ + la x14, pcpup + ld x14, 0(x14) + /* Store the new curthread */ + sd a1, PC_CURTHREAD(x14) + /* And the new pcb */ + ld x13, TD_PCB(a1) + sd x13, PC_CURPCB(x14) + + sfence.vm + + /* Switch to the new pmap */ + la t0, pagetable_l0 + ld t1, PCB_L1ADDR(x13) /* Link to next level PN */ + srli t1, t1, PAGE_SHIFT /* PN no */ + li t2, (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)) + slli t3, t1, PTE_PPN0_S /* (t1 << PTE_PPN0_S) */ + or t4, t2, t3 + /* Store single level0 PTE entry to position */ + sd t4, 0(t0) + + /* TODO: Invalidate the TLB */ + + sfence.vm + + /* Load registers */ + ld ra, (PCB_RA)(x13) + ld sp, (PCB_SP)(x13) + ld gp, (PCB_GP)(x13) + ld tp, (PCB_TP)(x13) + + /* s[0-11] */ + ld s0, (PCB_S + 0 * 8)(x13) + ld s1, (PCB_S + 1 * 8)(x13) + ld s2, (PCB_S + 2 * 8)(x13) + ld s3, (PCB_S + 3 * 8)(x13) + ld s4, (PCB_S + 4 * 8)(x13) + ld s5, (PCB_S + 5 * 8)(x13) + ld s6, (PCB_S + 6 * 8)(x13) + ld s7, (PCB_S + 7 * 8)(x13) + ld s8, (PCB_S + 8 * 8)(x13) + ld s9, (PCB_S + 9 * 8)(x13) + ld s10, (PCB_S + 10 * 8)(x13) + ld s11, (PCB_S + 11 * 8)(x13) + ret + +.Lcpu_throw_panic_str: + .asciz "cpu_throw: %p\0" +END(cpu_throw) + +/* + * void cpu_switch(struct thread *old, struct thread *new, struct mtx *mtx) + * + * a0 = old + * a1 = new + * a2 = mtx + * x3 to x7, x16 and x17 are caller saved + */ +ENTRY(cpu_switch) + /* Load pcpu */ + la x14, pcpup + ld x14, 0(x14) + /* Store the new curthread */ + sd a1, PC_CURTHREAD(x14) + /* And the new pcb */ + ld x13, TD_PCB(a1) + sd x13, PC_CURPCB(x14) + + /* Save the old context. */ + ld x13, TD_PCB(a0) + + /* Store the callee-saved registers */ + sd ra, (PCB_RA)(x13) + sd sp, (PCB_SP)(x13) + sd gp, (PCB_GP)(x13) + sd tp, (PCB_TP)(x13) + + /* We use these in fork_trampoline */ + sd t0, (PCB_T + 0 * 8)(x13) + sd t1, (PCB_T + 1 * 8)(x13) + + /* s[0-11] */ + sd s0, (PCB_S + 0 * 8)(x13) + sd s1, (PCB_S + 1 * 8)(x13) + sd s2, (PCB_S + 2 * 8)(x13) + sd s3, (PCB_S + 3 * 8)(x13) + sd s4, (PCB_S + 4 * 8)(x13) + sd s5, (PCB_S + 5 * 8)(x13) + sd s6, (PCB_S + 6 * 8)(x13) + sd s7, (PCB_S + 7 * 8)(x13) + sd s8, (PCB_S + 8 * 8)(x13) + sd s9, (PCB_S + 9 * 8)(x13) + sd s10, (PCB_S + 10 * 8)(x13) + sd s11, (PCB_S + 11 * 8)(x13) + + /* + * Restore the saved context. + */ + ld x13, TD_PCB(a1) + + /* + * TODO: We may need to flush the cache here if switching + * to a user process. + */ + + sfence.vm + + /* Switch to the new pmap */ + la t0, pagetable_l0 + ld t1, PCB_L1ADDR(x13) /* Link to next level PN */ + srli t1, t1, PAGE_SHIFT /* PN no */ + li t2, (PTE_VALID | (PTE_TYPE_PTR << PTE_TYPE_S)) + slli t3, t1, PTE_PPN0_S /* (t1 << PTE_PPN0_S) */ + or t4, t2, t3 + /* Store single level0 PTE entry to position */ + sd t4, 0(t0) + + /* TODO: Invalidate the TLB */ + + sfence.vm + + /* Release the old thread */ + sd a2, TD_LOCK(a0) +#if defined(SCHED_ULE) && defined(SMP) + /* TODO */ +#endif + + /* Restore the registers */ + ld ra, (PCB_RA)(x13) + ld sp, (PCB_SP)(x13) + ld gp, (PCB_GP)(x13) + ld tp, (PCB_TP)(x13) + + /* We use these in fork_trampoline */ + ld t0, (PCB_T + 0 * 8)(x13) + ld t1, (PCB_T + 1 * 8)(x13) + + /* s[0-11] */ + ld s0, (PCB_S + 0 * 8)(x13) + ld s1, (PCB_S + 1 * 8)(x13) + ld s2, (PCB_S + 2 * 8)(x13) + ld s3, (PCB_S + 3 * 8)(x13) + ld s4, (PCB_S + 4 * 8)(x13) + ld s5, (PCB_S + 5 * 8)(x13) + ld s6, (PCB_S + 6 * 8)(x13) + ld s7, (PCB_S + 7 * 8)(x13) + ld s8, (PCB_S + 8 * 8)(x13) + ld s9, (PCB_S + 9 * 8)(x13) + ld s10, (PCB_S + 10 * 8)(x13) + ld s11, (PCB_S + 11 * 8)(x13) + ret +.Lcpu_switch_panic_str: + .asciz "cpu_switch: %p\0" +END(cpu_switch) + +/* + * fork_exit(void (*callout)(void *, struct trapframe *), void *arg, + * struct trapframe *frame) + */ + +ENTRY(fork_trampoline) + mv a0, x5 + mv a1, x6 + mv a2, sp + call _C_LABEL(fork_exit) + + /* Restore sstatus */ + ld t0, (TF_SSTATUS)(sp) + /* Ensure interrupts disabled */ + li t1, ~SSTATUS_IE + and t0, t0, t1 + csrw sstatus, t0 + + /* Restore exception program counter */ + ld t0, (TF_SEPC)(sp) + csrw sepc, t0 + + /* Restore the registers */ + ld t0, (TF_T + 0 * 8)(sp) + ld t1, (TF_T + 1 * 8)(sp) + ld t2, (TF_T + 2 * 8)(sp) + ld t3, (TF_T + 3 * 8)(sp) + ld t4, (TF_T + 4 * 8)(sp) + ld t5, (TF_T + 5 * 8)(sp) + ld t6, (TF_T + 6 * 8)(sp) + + ld s0, (TF_S + 0 * 8)(sp) + ld s1, (TF_S + 1 * 8)(sp) + ld s2, (TF_S + 2 * 8)(sp) + ld s3, (TF_S + 3 * 8)(sp) + ld s4, (TF_S + 4 * 8)(sp) + ld s5, (TF_S + 5 * 8)(sp) + ld s6, (TF_S + 6 * 8)(sp) + ld s7, (TF_S + 7 * 8)(sp) + ld s8, (TF_S + 8 * 8)(sp) + ld s9, (TF_S + 9 * 8)(sp) + ld s10, (TF_S + 10 * 8)(sp) + ld s11, (TF_S + 11 * 8)(sp) + + ld a0, (TF_A + 0 * 8)(sp) + ld a1, (TF_A + 1 * 8)(sp) + ld a2, (TF_A + 2 * 8)(sp) + ld a3, (TF_A + 3 * 8)(sp) + ld a4, (TF_A + 4 * 8)(sp) + ld a5, (TF_A + 5 * 8)(sp) + ld a6, (TF_A + 6 * 8)(sp) + ld a7, (TF_A + 7 * 8)(sp) + + /* Save kernel stack so we can use it doing a user trap */ + csrw sscratch, sp + + /* Load user ra and sp */ + ld ra, (TF_RA)(sp) + ld sp, (TF_SP)(sp) + + eret +END(fork_trampoline) + +ENTRY(savectx) + la a0, .Lsavectx_panic_str + call panic +.Lsavectx_panic_str: + .asciz "savectx_panic: %p\0" +END(savectx) Property changes on: user/ngie/make_check/sys/riscv/riscv/swtch.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/sys_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/sys_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/sys_machdep.c (revision 295078) @@ -0,0 +1,49 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include + +#include + +int +sysarch(struct thread *td, struct sysarch_args *uap) +{ + + return (ENOTSUP); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/sys_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/timer.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/timer.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/timer.c (revision 295078) @@ -0,0 +1,298 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +/* + * RISC-V Timer + */ + +#include "opt_platform.h" + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#define DEFAULT_FREQ 1000000 + +struct riscv_tmr_softc { + struct resource *res[1]; + void *ihl[1]; + uint32_t clkfreq; + struct eventtimer et; +}; + +static struct riscv_tmr_softc *riscv_tmr_sc = NULL; + +static struct resource_spec timer_spec[] = { + { SYS_RES_IRQ, 0, RF_ACTIVE }, + { -1, 0 } +}; + +static timecounter_get_t riscv_tmr_get_timecount; + +static struct timecounter riscv_tmr_timecount = { + .tc_name = "RISC-V Timecounter", + .tc_get_timecount = riscv_tmr_get_timecount, + .tc_poll_pps = NULL, + .tc_counter_mask = ~0u, + .tc_frequency = 0, + .tc_quality = 1000, +}; + +static long +get_counts(void) +{ + + return (csr_read(stime)); +} + +static unsigned +riscv_tmr_get_timecount(struct timecounter *tc) +{ + + return (get_counts()); +} + +static int +riscv_tmr_start(struct eventtimer *et, sbintime_t first, sbintime_t period) +{ + struct riscv_tmr_softc *sc; + int counts; + + sc = (struct riscv_tmr_softc *)et->et_priv; + + if (first != 0) { + counts = ((uint32_t)et->et_frequency * first) >> 32; + machine_command(ECALL_MTIMECMP, counts); + return (0); + } + + return (EINVAL); + +} + +static int +riscv_tmr_stop(struct eventtimer *et) +{ + struct riscv_tmr_softc *sc; + + sc = (struct riscv_tmr_softc *)et->et_priv; + + /* TODO */ + + return (0); +} + +static int +riscv_tmr_intr(void *arg) +{ + struct riscv_tmr_softc *sc; + + sc = (struct riscv_tmr_softc *)arg; + + /* + * Clear interrupt pending bit. + * Note sip register is unimplemented in Spike simulator, + * so use machine command to clear in mip. + */ + machine_command(ECALL_CLEAR_PENDING, 0); + + if (sc->et.et_active) + sc->et.et_event_cb(&sc->et, sc->et.et_arg); + + return (FILTER_HANDLED); +} + +static int +riscv_tmr_fdt_probe(device_t dev) +{ + + if (!ofw_bus_status_okay(dev)) + return (ENXIO); + + if (ofw_bus_is_compatible(dev, "riscv,timer")) { + device_set_desc(dev, "RISC-V Timer"); + return (BUS_PROBE_DEFAULT); + } + + return (ENXIO); +} + +static int +riscv_tmr_attach(device_t dev) +{ + struct riscv_tmr_softc *sc; + phandle_t node; + pcell_t clock; + int error; + + sc = device_get_softc(dev); + if (riscv_tmr_sc) + return (ENXIO); + + /* Get the base clock frequency */ + node = ofw_bus_get_node(dev); + if (node > 0) { + error = OF_getprop(node, "clock-frequency", &clock, + sizeof(clock)); + if (error > 0) { + sc->clkfreq = fdt32_to_cpu(clock); + } + } + + if (sc->clkfreq == 0) + sc->clkfreq = DEFAULT_FREQ; + + if (sc->clkfreq == 0) { + device_printf(dev, "No clock frequency specified\n"); + return (ENXIO); + } + + if (bus_alloc_resources(dev, timer_spec, sc->res)) { + device_printf(dev, "could not allocate resources\n"); + return (ENXIO); + } + + riscv_tmr_sc = sc; + + /* Setup IRQs handler */ + error = bus_setup_intr(dev, sc->res[0], INTR_TYPE_CLK, + riscv_tmr_intr, NULL, sc, &sc->ihl[0]); + if (error) { + device_printf(dev, "Unable to alloc int resource.\n"); + return (ENXIO); + } + + riscv_tmr_timecount.tc_frequency = sc->clkfreq; + tc_init(&riscv_tmr_timecount); + + sc->et.et_name = "RISC-V Eventtimer"; + sc->et.et_flags = ET_FLAGS_ONESHOT | ET_FLAGS_PERCPU; + sc->et.et_quality = 1000; + + sc->et.et_frequency = sc->clkfreq; + sc->et.et_min_period = (0x00000002LLU << 32) / sc->et.et_frequency; + sc->et.et_max_period = (0xfffffffeLLU << 32) / sc->et.et_frequency; + sc->et.et_start = riscv_tmr_start; + sc->et.et_stop = riscv_tmr_stop; + sc->et.et_priv = sc; + et_register(&sc->et); + + return (0); +} + +static device_method_t riscv_tmr_fdt_methods[] = { + DEVMETHOD(device_probe, riscv_tmr_fdt_probe), + DEVMETHOD(device_attach, riscv_tmr_attach), + { 0, 0 } +}; + +static driver_t riscv_tmr_fdt_driver = { + "timer", + riscv_tmr_fdt_methods, + sizeof(struct riscv_tmr_softc), +}; + +static devclass_t riscv_tmr_fdt_devclass; + +EARLY_DRIVER_MODULE(timer, simplebus, riscv_tmr_fdt_driver, riscv_tmr_fdt_devclass, + 0, 0, BUS_PASS_TIMER + BUS_PASS_ORDER_MIDDLE); +EARLY_DRIVER_MODULE(timer, ofwbus, riscv_tmr_fdt_driver, riscv_tmr_fdt_devclass, + 0, 0, BUS_PASS_TIMER + BUS_PASS_ORDER_MIDDLE); + +void +DELAY(int usec) +{ + int32_t counts, counts_per_usec; + uint32_t first, last; + + /* + * Check the timers are setup, if not just + * use a for loop for the meantime + */ + if (riscv_tmr_sc == NULL) { + for (; usec > 0; usec--) + for (counts = 200; counts > 0; counts--) + /* + * Prevent the compiler from optimizing + * out the loop + */ + cpufunc_nullop(); + return; + } + + /* Get the number of times to count */ + counts_per_usec = ((riscv_tmr_timecount.tc_frequency / 1000000) + 1); + + /* + * Clamp the timeout at a maximum value (about 32 seconds with + * a 66MHz clock). *Nobody* should be delay()ing for anywhere + * near that length of time and if they are, they should be hung + * out to dry. + */ + if (usec >= (0x80000000U / counts_per_usec)) + counts = (0x80000000U / counts_per_usec) - 1; + else + counts = usec * counts_per_usec; + + first = get_counts(); + + while (counts > 0) { + last = get_counts(); + counts -= (int32_t)(last - first); + first = last; + } +} Property changes on: user/ngie/make_check/sys/riscv/riscv/timer.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/trap.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/trap.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/trap.c (revision 295078) @@ -0,0 +1,311 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include + +extern register_t fsu_intr_fault; + +/* Called from exception.S */ +void do_trap_supervisor(struct trapframe *); +void do_trap_user(struct trapframe *); + +static __inline void +call_trapsignal(struct thread *td, int sig, int code, void *addr) +{ + ksiginfo_t ksi; + + ksiginfo_init_trap(&ksi); + ksi.ksi_signo = sig; + ksi.ksi_code = code; + ksi.ksi_addr = addr; + trapsignal(td, &ksi); +} + +int +cpu_fetch_syscall_args(struct thread *td, struct syscall_args *sa) +{ + struct proc *p; + register_t *ap; + int nap; + + nap = 8; + p = td->td_proc; + ap = &td->td_frame->tf_a[0]; + + sa->code = td->td_frame->tf_t[0]; + + if (sa->code == SYS_syscall || sa->code == SYS___syscall) { + sa->code = *ap++; + nap--; + } + + if (p->p_sysent->sv_mask) + sa->code &= p->p_sysent->sv_mask; + if (sa->code >= p->p_sysent->sv_size) + sa->callp = &p->p_sysent->sv_table[0]; + else + sa->callp = &p->p_sysent->sv_table[sa->code]; + + sa->narg = sa->callp->sy_narg; + memcpy(sa->args, ap, nap * sizeof(register_t)); + if (sa->narg > nap) + panic("TODO: Could we have more then 8 args?"); + + td->td_retval[0] = 0; + td->td_retval[1] = 0; + + return (0); +} + +#include "../../kern/subr_syscall.c" + +static void +dump_regs(struct trapframe *frame) +{ + int n; + int i; + + n = (sizeof(frame->tf_t) / sizeof(frame->tf_t[0])); + for (i = 0; i < n; i++) + printf("t[%d] == 0x%016lx\n", i, frame->tf_t[i]); + + n = (sizeof(frame->tf_s) / sizeof(frame->tf_s[0])); + for (i = 0; i < n; i++) + printf("s[%d] == 0x%016lx\n", i, frame->tf_s[i]); + + n = (sizeof(frame->tf_a) / sizeof(frame->tf_a[0])); + for (i = 0; i < n; i++) + printf("a[%d] == 0x%016lx\n", i, frame->tf_a[i]); + + printf("sepc == 0x%016lx\n", frame->tf_sepc); + printf("sstatus == 0x%016lx\n", frame->tf_sstatus); +} + +static void +svc_handler(struct trapframe *frame) +{ + struct syscall_args sa; + struct thread *td; + int error; + + td = curthread; + td->td_frame = frame; + + error = syscallenter(td, &sa); + syscallret(td, error, &sa); +} + +static void +data_abort(struct trapframe *frame, int lower) +{ + struct vm_map *map; + uint64_t sbadaddr; + struct thread *td; + struct pcb *pcb; + vm_prot_t ftype; + vm_offset_t va; + struct proc *p; + int ucode; + int error; + int sig; + + td = curthread; + pcb = td->td_pcb; + + /* + * Special case for fuswintr and suswintr. These can't sleep so + * handle them early on in the trap handler. + */ + if (__predict_false(pcb->pcb_onfault == (vm_offset_t)&fsu_intr_fault)) { + frame->tf_sepc = pcb->pcb_onfault; + return; + } + + sbadaddr = frame->tf_sbadaddr; + + p = td->td_proc; + + if (lower) + map = &td->td_proc->p_vmspace->vm_map; + else { + /* The top bit tells us which range to use */ + if ((sbadaddr >> 63) == 1) + map = kernel_map; + else + map = &td->td_proc->p_vmspace->vm_map; + } + + va = trunc_page(sbadaddr); + + if (frame->tf_scause == EXCP_STORE_ACCESS_FAULT) { + ftype = (VM_PROT_READ | VM_PROT_WRITE); + } else { + ftype = (VM_PROT_READ); + } + + if (map != kernel_map) { + /* + * Keep swapout from messing with us during this + * critical time. + */ + PROC_LOCK(p); + ++p->p_lock; + PROC_UNLOCK(p); + + /* Fault in the user page: */ + error = vm_fault(map, va, ftype, VM_FAULT_NORMAL); + + PROC_LOCK(p); + --p->p_lock; + PROC_UNLOCK(p); + } else { + /* + * Don't have to worry about process locking or stacks in the + * kernel. + */ + error = vm_fault(map, va, ftype, VM_FAULT_NORMAL); + } + + if (error != KERN_SUCCESS) { + if (lower) { + sig = SIGSEGV; + if (error == KERN_PROTECTION_FAILURE) + ucode = SEGV_ACCERR; + else + ucode = SEGV_MAPERR; + call_trapsignal(td, sig, ucode, (void *)sbadaddr); + } else { + if (td->td_intr_nesting_level == 0 && + pcb->pcb_onfault != 0) { + frame->tf_a[0] = error; + frame->tf_sepc = pcb->pcb_onfault; + return; + } + dump_regs(frame); + panic("vm_fault failed: %lx, va 0x%016lx", + frame->tf_sepc, sbadaddr); + } + } + + if (lower) + userret(td, frame); +} + +void +do_trap_supervisor(struct trapframe *frame) +{ + uint64_t exception; + + exception = (frame->tf_scause & EXCP_MASK); + if (frame->tf_scause & EXCP_INTR) { + /* Interrupt */ + riscv_cpu_intr(frame); + return; + } + + CTR3(KTR_TRAP, "do_trap_supervisor: curthread: %p, sepc: %lx, frame: %p", + curthread, frame->tf_sepc, frame); + + switch(exception) { + case EXCP_LOAD_ACCESS_FAULT: + case EXCP_STORE_ACCESS_FAULT: + case EXCP_INSTR_ACCESS_FAULT: + data_abort(frame, 0); + break; + default: + dump_regs(frame); + panic("Unknown kernel exception %x badaddr %lx\n", + exception, frame->tf_sbadaddr); + } +} + +void +do_trap_user(struct trapframe *frame) +{ + uint64_t exception; + + exception = (frame->tf_scause & EXCP_MASK); + if (frame->tf_scause & EXCP_INTR) { + /* Interrupt */ + riscv_cpu_intr(frame); + return; + } + + CTR3(KTR_TRAP, "do_trap_user: curthread: %p, sepc: %lx, frame: %p", + curthread, frame->tf_sepc, frame); + + switch(exception) { + case EXCP_LOAD_ACCESS_FAULT: + case EXCP_STORE_ACCESS_FAULT: + case EXCP_INSTR_ACCESS_FAULT: + data_abort(frame, 1); + break; + case EXCP_UMODE_ENV_CALL: + frame->tf_sepc += 4; /* Next instruction */ + svc_handler(frame); + break; + default: + dump_regs(frame); + panic("Unknown userland exception %x badaddr %lx\n", + exception, frame->tf_sbadaddr); + } +} Property changes on: user/ngie/make_check/sys/riscv/riscv/trap.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/uio_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/uio_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/uio_machdep.c (revision 295078) @@ -0,0 +1,134 @@ +/*- + * Copyright (c) 2004 Alan L. Cox + * Copyright (c) 1982, 1986, 1991, 1993 + * The Regents of the University of California. All rights reserved. + * (c) UNIX System Laboratories, Inc. + * All or some portions of this file are derived from material licensed + * to the University of California by American Telephone and Telegraph + * Co. or Unix System Laboratories, Inc. and are reproduced herein with + * the permission of UNIX System Laboratories, Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * @(#)kern_subr.c 8.3 (Berkeley) 1/21/94 + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include + +/* + * Implement uiomove(9) from physical memory using the direct map to + * avoid the creation and destruction of ephemeral mappings. + */ +int +uiomove_fromphys(vm_page_t ma[], vm_offset_t offset, int n, struct uio *uio) +{ + struct thread *td = curthread; + struct iovec *iov; + void *cp; + vm_offset_t page_offset, vaddr; + size_t cnt; + int error = 0; + int save = 0; + boolean_t mapped; + + KASSERT(uio->uio_rw == UIO_READ || uio->uio_rw == UIO_WRITE, + ("uiomove_fromphys: mode")); + KASSERT(uio->uio_segflg != UIO_USERSPACE || uio->uio_td == curthread, + ("uiomove_fromphys proc")); + save = td->td_pflags & TDP_DEADLKTREAT; + td->td_pflags |= TDP_DEADLKTREAT; + mapped = FALSE; + while (n > 0 && uio->uio_resid) { + iov = uio->uio_iov; + cnt = iov->iov_len; + if (cnt == 0) { + uio->uio_iov++; + uio->uio_iovcnt--; + continue; + } + if (cnt > n) + cnt = n; + page_offset = offset & PAGE_MASK; + cnt = min(cnt, PAGE_SIZE - page_offset); + if (uio->uio_segflg != UIO_NOCOPY) { + mapped = pmap_map_io_transient( + &ma[offset >> PAGE_SHIFT], &vaddr, 1, TRUE); + cp = (char *)vaddr + page_offset; + } + switch (uio->uio_segflg) { + case UIO_USERSPACE: + maybe_yield(); + if (uio->uio_rw == UIO_READ) + error = copyout(cp, iov->iov_base, cnt); + else + error = copyin(iov->iov_base, cp, cnt); + if (error) + goto out; + break; + case UIO_SYSSPACE: + if (uio->uio_rw == UIO_READ) + bcopy(cp, iov->iov_base, cnt); + else + bcopy(iov->iov_base, cp, cnt); + break; + case UIO_NOCOPY: + break; + } + if (__predict_false(mapped)) { + pmap_unmap_io_transient(&ma[offset >> PAGE_SHIFT], + &vaddr, 1, TRUE); + mapped = FALSE; + } + iov->iov_base = (char *)iov->iov_base + cnt; + iov->iov_len -= cnt; + uio->uio_resid -= cnt; + uio->uio_offset += cnt; + offset += cnt; + n -= cnt; + } +out: + if (__predict_false(mapped)) { + panic("TODO 3"); + pmap_unmap_io_transient(&ma[offset >> PAGE_SHIFT], &vaddr, 1, + TRUE); + } + if (save == 0) + td->td_pflags &= ~TDP_DEADLKTREAT; + return (error); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/uio_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/uma_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/uma_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/uma_machdep.c (revision 295078) @@ -0,0 +1,55 @@ +/*- + * Copyright (c) 2003 Alan L. Cox + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +void * +uma_small_alloc(uma_zone_t zone, vm_size_t bytes, u_int8_t *flags, int wait) +{ + + panic("uma_small_alloc"); +} + +void +uma_small_free(void *mem, vm_size_t size, u_int8_t flags) +{ + + panic("uma_small_free"); +} Property changes on: user/ngie/make_check/sys/riscv/riscv/uma_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/riscv/riscv/vm_machdep.c =================================================================== --- user/ngie/make_check/sys/riscv/riscv/vm_machdep.c (nonexistent) +++ user/ngie/make_check/sys/riscv/riscv/vm_machdep.c (revision 295078) @@ -0,0 +1,259 @@ +/*- + * Copyright (c) 2015 Ruslan Bukin + * All rights reserved. + * + * Portions of this software were developed by SRI International and the + * University of Cambridge Computer Laboratory under DARPA/AFRL contract + * FA8750-10-C-0237 ("CTSRD"), as part of the DARPA CRASH research programme. + * + * Portions of this software were developed by the University of Cambridge + * Computer Laboratory as part of the CTSRD Project, with support from the + * UK Higher Education Innovation Fund (HEIF). + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +/* + * Finish a fork operation, with process p2 nearly set up. + * Copy and update the pcb, set up the stack so that the child + * ready to run and return to user mode. + */ +void +cpu_fork(struct thread *td1, struct proc *p2, struct thread *td2, int flags) +{ + struct pcb *pcb2; + struct trapframe *tf; + uint64_t val; + + if ((flags & RFPROC) == 0) + return; + + if (td1 == curthread) { + __asm __volatile("mv %0, tp" : "=&r"(val)); + td1->td_pcb->pcb_tp = val; + } + + pcb2 = (struct pcb *)(td2->td_kstack + + td2->td_kstack_pages * PAGE_SIZE) - 1; + + td2->td_pcb = pcb2; + bcopy(td1->td_pcb, pcb2, sizeof(*pcb2)); + + td2->td_pcb->pcb_l1addr = + vtophys(vmspace_pmap(td2->td_proc->p_vmspace)->pm_l1); + + tf = (struct trapframe *)STACKALIGN((struct trapframe *)pcb2 - 1); + bcopy(td1->td_frame, tf, sizeof(*tf)); + + /* Clear syscall error flag */ + tf->tf_t[0] = 0; + + /* Arguments for child */ + tf->tf_a[0] = 0; + tf->tf_a[1] = 0; + tf->tf_sstatus = SSTATUS_PIE; + + td2->td_frame = tf; + + /* Set the return value registers for fork() */ + td2->td_pcb->pcb_t[0] = (uintptr_t)fork_return; + td2->td_pcb->pcb_t[1] = (uintptr_t)td2; + td2->td_pcb->pcb_ra = (uintptr_t)fork_trampoline; + td2->td_pcb->pcb_sp = (uintptr_t)td2->td_frame; + + /* Setup to release spin count in fork_exit(). */ + td2->td_md.md_spinlock_count = 1; + td2->td_md.md_saved_sstatus_ie = 1; +} + +void +cpu_reset(void) +{ + + printf("cpu_reset"); + while(1) + __asm volatile("wfi" ::: "memory"); +} + +void +cpu_thread_swapin(struct thread *td) +{ +} + +void +cpu_thread_swapout(struct thread *td) +{ +} + +void +cpu_set_syscall_retval(struct thread *td, int error) +{ + struct trapframe *frame; + + frame = td->td_frame; + + switch (error) { + case 0: + frame->tf_a[0] = td->td_retval[0]; + frame->tf_a[1] = td->td_retval[1]; + frame->tf_t[0] = 0; /* syscall succeeded */ + break; + case ERESTART: + frame->tf_sepc -= 4; /* prev instruction */ + break; + case EJUSTRETURN: + break; + default: + frame->tf_a[0] = error; + frame->tf_t[0] = 1; /* syscall error */ + break; + } +} + +/* + * Initialize machine state (pcb and trap frame) for a new thread about to + * upcall. Put enough state in the new thread's PCB to get it to go back + * userret(), where we can intercept it again to set the return (upcall) + * Address and stack, along with those from upcals that are from other sources + * such as those generated in thread_userret() itself. + */ +void +cpu_set_upcall(struct thread *td, struct thread *td0) +{ + + bcopy(td0->td_frame, td->td_frame, sizeof(struct trapframe)); + bcopy(td0->td_pcb, td->td_pcb, sizeof(struct pcb)); + + td->td_pcb->pcb_t[0] = (uintptr_t)fork_return; + td->td_pcb->pcb_t[1] = (uintptr_t)td; + td->td_pcb->pcb_ra = (uintptr_t)fork_trampoline; + td->td_pcb->pcb_sp = (uintptr_t)td->td_frame; + + /* Setup to release spin count in fork_exit(). */ + td->td_md.md_spinlock_count = 1; + td->td_md.md_saved_sstatus_ie = 1; +} + +/* + * Set that machine state for performing an upcall that has to + * be done in thread_userret() so that those upcalls generated + * in thread_userret() itself can be done as well. + */ +void +cpu_set_upcall_kse(struct thread *td, void (*entry)(void *), void *arg, + stack_t *stack) +{ + struct trapframe *tf = td->td_frame; + + tf->tf_sp = STACKALIGN((uintptr_t)stack->ss_sp + stack->ss_size); + tf->tf_sepc = (register_t)entry; + tf->tf_a[0] = (register_t)arg; +} + +int +cpu_set_user_tls(struct thread *td, void *tls_base) +{ + struct pcb *pcb; + + if ((uintptr_t)tls_base >= VM_MAXUSER_ADDRESS) + return (EINVAL); + + pcb = td->td_pcb; + pcb->pcb_tp = (register_t)tls_base; + + return (0); +} + +void +cpu_thread_exit(struct thread *td) +{ +} + +void +cpu_thread_alloc(struct thread *td) +{ + + td->td_pcb = (struct pcb *)(td->td_kstack + + td->td_kstack_pages * PAGE_SIZE) - 1; + td->td_frame = (struct trapframe *)STACKALIGN( + td->td_pcb - 1); +} + +void +cpu_thread_free(struct thread *td) +{ +} + +void +cpu_thread_clean(struct thread *td) +{ +} + +/* + * Intercept the return address from a freshly forked process that has NOT + * been scheduled yet. + * + * This is needed to make kernel threads stay in kernel mode. + */ +void +cpu_set_fork_handler(struct thread *td, void (*func)(void *), void *arg) +{ + + td->td_pcb->pcb_t[0] = (uintptr_t)func; + td->td_pcb->pcb_t[1] = (uintptr_t)arg; + td->td_pcb->pcb_ra = (uintptr_t)fork_trampoline; + td->td_pcb->pcb_sp = (uintptr_t)td->td_frame; +} + +void +cpu_exit(struct thread *td) +{ +} + +void +swi_vm(void *v) +{ + + /* Nothing to do here - busdma bounce buffers are not implemented. */ +} Property changes on: user/ngie/make_check/sys/riscv/riscv/vm_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/sys/sys/cdefs.h =================================================================== --- user/ngie/make_check/sys/sys/cdefs.h (revision 295077) +++ user/ngie/make_check/sys/sys/cdefs.h (revision 295078) @@ -1,858 +1,858 @@ /*- * Copyright (c) 1991, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * Berkeley Software Design, Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)cdefs.h 8.8 (Berkeley) 1/9/95 * $FreeBSD$ */ #ifndef _SYS_CDEFS_H_ #define _SYS_CDEFS_H_ /* * Testing against Clang-specific extensions. */ #ifndef __has_attribute #define __has_attribute(x) 0 #endif #ifndef __has_extension #define __has_extension __has_feature #endif #ifndef __has_feature #define __has_feature(x) 0 #endif #ifndef __has_include #define __has_include(x) 0 #endif #ifndef __has_builtin #define __has_builtin(x) 0 #endif #if defined(__cplusplus) #define __BEGIN_DECLS extern "C" { #define __END_DECLS } #else #define __BEGIN_DECLS #define __END_DECLS #endif /* * This code has been put in place to help reduce the addition of * compiler specific defines in FreeBSD code. It helps to aid in * having a compiler-agnostic source tree. */ #if defined(__GNUC__) || defined(__INTEL_COMPILER) #if __GNUC__ >= 3 || defined(__INTEL_COMPILER) #define __GNUCLIKE_ASM 3 #define __GNUCLIKE_MATH_BUILTIN_CONSTANTS #else #define __GNUCLIKE_ASM 2 #endif #define __GNUCLIKE___TYPEOF 1 #define __GNUCLIKE___OFFSETOF 1 #define __GNUCLIKE___SECTION 1 #ifndef __INTEL_COMPILER #define __GNUCLIKE_CTOR_SECTION_HANDLING 1 #endif #define __GNUCLIKE_BUILTIN_CONSTANT_P 1 #if defined(__INTEL_COMPILER) && defined(__cplusplus) && \ __INTEL_COMPILER < 800 #undef __GNUCLIKE_BUILTIN_CONSTANT_P #endif #if (__GNUC_MINOR__ > 95 || __GNUC__ >= 3) #define __GNUCLIKE_BUILTIN_VARARGS 1 #define __GNUCLIKE_BUILTIN_STDARG 1 #define __GNUCLIKE_BUILTIN_VAALIST 1 #endif #if defined(__GNUC__) #define __GNUC_VA_LIST_COMPATIBILITY 1 #endif /* * Compiler memory barriers, specific to gcc and clang. */ #if defined(__GNUC__) #define __compiler_membar() __asm __volatile(" " : : : "memory") #endif #ifndef __INTEL_COMPILER #define __GNUCLIKE_BUILTIN_NEXT_ARG 1 #define __GNUCLIKE_MATH_BUILTIN_RELOPS #endif #define __GNUCLIKE_BUILTIN_MEMCPY 1 /* XXX: if __GNUC__ >= 2: not tested everywhere originally, where replaced */ #define __CC_SUPPORTS_INLINE 1 #define __CC_SUPPORTS___INLINE 1 #define __CC_SUPPORTS___INLINE__ 1 #define __CC_SUPPORTS___FUNC__ 1 #define __CC_SUPPORTS_WARNING 1 #define __CC_SUPPORTS_VARADIC_XXX 1 /* see varargs.h */ #define __CC_SUPPORTS_DYNAMIC_ARRAY_INIT 1 #endif /* __GNUC__ || __INTEL_COMPILER */ /* * Macro to test if we're using a specific version of gcc or later. */ #if defined(__GNUC__) && !defined(__INTEL_COMPILER) #define __GNUC_PREREQ__(ma, mi) \ (__GNUC__ > (ma) || __GNUC__ == (ma) && __GNUC_MINOR__ >= (mi)) #else #define __GNUC_PREREQ__(ma, mi) 0 #endif /* * The __CONCAT macro is used to concatenate parts of symbol names, e.g. * with "#define OLD(foo) __CONCAT(old,foo)", OLD(foo) produces oldfoo. * The __CONCAT macro is a bit tricky to use if it must work in non-ANSI * mode -- there must be no spaces between its arguments, and for nested * __CONCAT's, all the __CONCAT's must be at the left. __CONCAT can also * concatenate double-quoted strings produced by the __STRING macro, but * this only works with ANSI C. * * __XSTRING is like __STRING, but it expands any macros in its argument * first. It is only available with ANSI C. */ #if defined(__STDC__) || defined(__cplusplus) #define __P(protos) protos /* full-blown ANSI C */ #define __CONCAT1(x,y) x ## y #define __CONCAT(x,y) __CONCAT1(x,y) #define __STRING(x) #x /* stringify without expanding x */ #define __XSTRING(x) __STRING(x) /* expand x, then stringify */ #define __const const /* define reserved names to standard */ #define __signed signed #define __volatile volatile #if defined(__cplusplus) #define __inline inline /* convert to C++ keyword */ #else #if !(defined(__CC_SUPPORTS___INLINE)) #define __inline /* delete GCC keyword */ #endif /* ! __CC_SUPPORTS___INLINE */ #endif /* !__cplusplus */ #else /* !(__STDC__ || __cplusplus) */ #define __P(protos) () /* traditional C preprocessor */ #define __CONCAT(x,y) x/**/y #define __STRING(x) "x" #if !defined(__CC_SUPPORTS___INLINE) #define __const /* delete pseudo-ANSI C keywords */ #define __inline #define __signed #define __volatile /* * In non-ANSI C environments, new programs will want ANSI-only C keywords * deleted from the program and old programs will want them left alone. * When using a compiler other than gcc, programs using the ANSI C keywords * const, inline etc. as normal identifiers should define -DNO_ANSI_KEYWORDS. * When using "gcc -traditional", we assume that this is the intent; if * __GNUC__ is defined but __STDC__ is not, we leave the new keywords alone. */ #ifndef NO_ANSI_KEYWORDS #define const /* delete ANSI C keywords */ #define inline #define signed #define volatile #endif /* !NO_ANSI_KEYWORDS */ #endif /* !__CC_SUPPORTS___INLINE */ #endif /* !(__STDC__ || __cplusplus) */ /* * Compiler-dependent macros to help declare dead (non-returning) and * pure (no side effects) functions, and unused variables. They are * null except for versions of gcc that are known to support the features * properly (old versions of gcc-2 supported the dead and pure features * in a different (wrong) way). If we do not provide an implementation * for a given compiler, let the compile fail if it is told to use * a feature that we cannot live without. */ #ifdef lint #define __dead2 #define __pure2 #define __unused #define __packed #define __aligned(x) #define __alloc_align(x) #define __alloc_size(x) #define __section(x) #define __weak_symbol #else #define __weak_symbol __attribute__((__weak__)) #if !__GNUC_PREREQ__(2, 5) && !defined(__INTEL_COMPILER) #define __dead2 #define __pure2 #define __unused #endif #if __GNUC__ == 2 && __GNUC_MINOR__ >= 5 && __GNUC_MINOR__ < 7 && !defined(__INTEL_COMPILER) #define __dead2 __attribute__((__noreturn__)) #define __pure2 __attribute__((__const__)) #define __unused /* XXX Find out what to do for __packed, __aligned and __section */ #endif #if __GNUC_PREREQ__(2, 7) || defined(__INTEL_COMPILER) #define __dead2 __attribute__((__noreturn__)) #define __pure2 __attribute__((__const__)) #define __unused __attribute__((__unused__)) #define __used __attribute__((__used__)) #define __packed __attribute__((__packed__)) #define __aligned(x) __attribute__((__aligned__(x))) #define __section(x) __attribute__((__section__(x))) #endif #if __GNUC_PREREQ__(4, 3) || __has_attribute(__alloc_size__) #define __alloc_size(x) __attribute__((__alloc_size__(x))) #else #define __alloc_size(x) #endif #if __GNUC_PREREQ__(4, 9) || __has_attribute(__alloc_align__) #define __alloc_align(x) __attribute__((__alloc_align__(x))) #else #define __alloc_align(x) #endif #endif /* lint */ #if !__GNUC_PREREQ__(2, 95) #define __alignof(x) __offsetof(struct { char __a; x __b; }, __b) #endif /* * Keywords added in C11. */ #if !defined(__STDC_VERSION__) || __STDC_VERSION__ < 201112L || defined(lint) #if !__has_extension(c_alignas) #if (defined(__cplusplus) && __cplusplus >= 201103L) || \ __has_extension(cxx_alignas) #define _Alignas(x) alignas(x) #else /* XXX: Only emulates _Alignas(constant-expression); not _Alignas(type-name). */ #define _Alignas(x) __aligned(x) #endif #endif #if defined(__cplusplus) && __cplusplus >= 201103L #define _Alignof(x) alignof(x) #else #define _Alignof(x) __alignof(x) #endif #if !__has_extension(c_atomic) && !__has_extension(cxx_atomic) /* * No native support for _Atomic(). Place object in structure to prevent * most forms of direct non-atomic access. */ #define _Atomic(T) struct { T volatile __val; } #endif #if defined(__cplusplus) && __cplusplus >= 201103L #define _Noreturn [[noreturn]] #else #define _Noreturn __dead2 #endif #if !__has_extension(c_static_assert) #if (defined(__cplusplus) && __cplusplus >= 201103L) || \ __has_extension(cxx_static_assert) #define _Static_assert(x, y) static_assert(x, y) #elif __GNUC_PREREQ__(4,6) /* Nothing, gcc 4.6 and higher has _Static_assert built-in */ #elif defined(__COUNTER__) #define _Static_assert(x, y) __Static_assert(x, __COUNTER__) #define __Static_assert(x, y) ___Static_assert(x, y) #define ___Static_assert(x, y) typedef char __assert_ ## y[(x) ? 1 : -1] \ __unused #else #define _Static_assert(x, y) struct __hack #endif #endif #if !__has_extension(c_thread_local) /* * XXX: Some compilers (Clang 3.3, GCC 4.7) falsely announce C++11 mode * without actually supporting the thread_local keyword. Don't check for * the presence of C++11 when defining _Thread_local. */ #if /* (defined(__cplusplus) && __cplusplus >= 201103L) || */ \ __has_extension(cxx_thread_local) #define _Thread_local thread_local #else #define _Thread_local __thread #endif #endif #endif /* __STDC_VERSION__ || __STDC_VERSION__ < 201112L */ /* * Emulation of C11 _Generic(). Unlike the previously defined C11 * keywords, it is not possible to implement this using exactly the same * syntax. Therefore implement something similar under the name * __generic(). Unlike _Generic(), this macro can only distinguish * between a single type, so it requires nested invocations to * distinguish multiple cases. */ #if (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L) || \ __has_extension(c_generic_selections) #define __generic(expr, t, yes, no) \ _Generic(expr, t: yes, default: no) #elif __GNUC_PREREQ__(3, 1) && !defined(__cplusplus) #define __generic(expr, t, yes, no) \ __builtin_choose_expr( \ __builtin_types_compatible_p(__typeof(expr), t), yes, no) #endif #if __GNUC_PREREQ__(2, 96) #define __malloc_like __attribute__((__malloc__)) #define __pure __attribute__((__pure__)) #else #define __malloc_like #define __pure #endif #if __GNUC_PREREQ__(3, 1) || (defined(__INTEL_COMPILER) && __INTEL_COMPILER >= 800) #define __always_inline __attribute__((__always_inline__)) #else #define __always_inline #endif #if __GNUC_PREREQ__(3, 1) #define __noinline __attribute__ ((__noinline__)) #else #define __noinline #endif #if __GNUC_PREREQ__(3, 3) #define __nonnull(x) __attribute__((__nonnull__(x))) #define __nonnull_all __attribute__((__nonnull__)) #else #define __nonnull(x) #define __nonnull_all #endif #if __GNUC_PREREQ__(3, 4) #define __fastcall __attribute__((__fastcall__)) #define __result_use_check __attribute__((__warn_unused_result__)) #else #define __fastcall #define __result_use_check #endif #if __GNUC_PREREQ__(4, 1) #define __returns_twice __attribute__((__returns_twice__)) #else #define __returns_twice #endif #if __GNUC_PREREQ__(4, 6) || __has_builtin(__builtin_unreachable) #define __unreachable() __builtin_unreachable() #else #define __unreachable() ((void)0) #endif /* XXX: should use `#if __STDC_VERSION__ < 199901'. */ #if !__GNUC_PREREQ__(2, 7) && !defined(__INTEL_COMPILER) #define __func__ NULL #endif #if (defined(__INTEL_COMPILER) || (defined(__GNUC__) && __GNUC__ >= 2)) && !defined(__STRICT_ANSI__) || __STDC_VERSION__ >= 199901 #define __LONG_LONG_SUPPORTED #endif /* C++11 exposes a load of C99 stuff */ #if defined(__cplusplus) && __cplusplus >= 201103L #define __LONG_LONG_SUPPORTED #ifndef __STDC_LIMIT_MACROS #define __STDC_LIMIT_MACROS #endif #ifndef __STDC_CONSTANT_MACROS #define __STDC_CONSTANT_MACROS #endif #endif /* * GCC 2.95 provides `__restrict' as an extension to C90 to support the * C99-specific `restrict' type qualifier. We happen to use `__restrict' as * a way to define the `restrict' type qualifier without disturbing older * software that is unaware of C99 keywords. */ #if !(__GNUC__ == 2 && __GNUC_MINOR__ == 95) #if !defined(__STDC_VERSION__) || __STDC_VERSION__ < 199901 || defined(lint) #define __restrict #else #define __restrict restrict #endif #endif /* * GNU C version 2.96 adds explicit branch prediction so that * the CPU back-end can hint the processor and also so that * code blocks can be reordered such that the predicted path * sees a more linear flow, thus improving cache behavior, etc. * * The following two macros provide us with a way to utilize this * compiler feature. Use __predict_true() if you expect the expression * to evaluate to true, and __predict_false() if you expect the * expression to evaluate to false. * * A few notes about usage: * * * Generally, __predict_false() error condition checks (unless * you have some _strong_ reason to do otherwise, in which case * document it), and/or __predict_true() `no-error' condition * checks, assuming you want to optimize for the no-error case. * * * Other than that, if you don't know the likelihood of a test * succeeding from empirical or other `hard' evidence, don't * make predictions. * * * These are meant to be used in places that are run `a lot'. * It is wasteful to make predictions in code that is run * seldomly (e.g. at subsystem initialization time) as the * basic block reordering that this affects can often generate * larger code. */ #if __GNUC_PREREQ__(2, 96) #define __predict_true(exp) __builtin_expect((exp), 1) #define __predict_false(exp) __builtin_expect((exp), 0) #else #define __predict_true(exp) (exp) #define __predict_false(exp) (exp) #endif #if __GNUC_PREREQ__(4, 0) #define __null_sentinel __attribute__((__sentinel__)) #define __exported __attribute__((__visibility__("default"))) #define __hidden __attribute__((__visibility__("hidden"))) #else #define __null_sentinel #define __exported #define __hidden #endif /* * We define this here since , , and * require it. */ #if __GNUC_PREREQ__(4, 1) #define __offsetof(type, field) __builtin_offsetof(type, field) #else #ifndef __cplusplus #define __offsetof(type, field) \ ((__size_t)(__uintptr_t)((const volatile void *)&((type *)0)->field)) #else #define __offsetof(type, field) \ (__offsetof__ (reinterpret_cast <__size_t> \ (&reinterpret_cast \ (static_cast (0)->field)))) #endif #endif #define __rangeof(type, start, end) \ (__offsetof(type, end) - __offsetof(type, start)) /* * Given the pointer x to the member m of the struct s, return * a pointer to the containing structure. When using GCC, we first * assign pointer x to a local variable, to check that its type is * compatible with member m. */ #if __GNUC_PREREQ__(3, 1) #define __containerof(x, s, m) ({ \ const volatile __typeof(((s *)0)->m) *__x = (x); \ __DEQUALIFY(s *, (const volatile char *)__x - __offsetof(s, m));\ }) #else #define __containerof(x, s, m) \ __DEQUALIFY(s *, (const volatile char *)(x) - __offsetof(s, m)) #endif /* * Compiler-dependent macros to declare that functions take printf-like * or scanf-like arguments. They are null except for versions of gcc * that are known to support the features properly (old versions of gcc-2 * didn't permit keeping the keywords out of the application namespace). */ #if !__GNUC_PREREQ__(2, 7) && !defined(__INTEL_COMPILER) #define __printflike(fmtarg, firstvararg) #define __scanflike(fmtarg, firstvararg) #define __format_arg(fmtarg) #define __strfmonlike(fmtarg, firstvararg) #define __strftimelike(fmtarg, firstvararg) #else #define __printflike(fmtarg, firstvararg) \ __attribute__((__format__ (__printf__, fmtarg, firstvararg))) #define __scanflike(fmtarg, firstvararg) \ __attribute__((__format__ (__scanf__, fmtarg, firstvararg))) #define __format_arg(fmtarg) __attribute__((__format_arg__ (fmtarg))) #define __strfmonlike(fmtarg, firstvararg) \ __attribute__((__format__ (__strfmon__, fmtarg, firstvararg))) #define __strftimelike(fmtarg, firstvararg) \ __attribute__((__format__ (__strftime__, fmtarg, firstvararg))) #endif /* * FORTIFY_SOURCE, and perhaps other compiler-specific features, require * the use of non-standard inlining. In general we should try to avoid * using these but GCC-compatible compilers tend to support the extensions * well enough to use them in limited cases. */ #if defined(__GNUC_GNU_INLINE__) || defined(__GNUC_STDC_INLINE__) #if __GNUC_PREREQ__(4, 3) || __has_attribute(__artificial__) #define __gnu_inline __attribute__((__gnu_inline__, __artificial__)) #else #define __gnu_inline __attribute__((__gnu_inline__)) #endif /* artificial */ #else #define __gnu_inline #endif /* Compiler-dependent macros that rely on FreeBSD-specific extensions. */ #if defined(__FreeBSD_cc_version) && __FreeBSD_cc_version >= 300001 && \ defined(__GNUC__) && !defined(__INTEL_COMPILER) #define __printf0like(fmtarg, firstvararg) \ __attribute__((__format__ (__printf0__, fmtarg, firstvararg))) #else #define __printf0like(fmtarg, firstvararg) #endif #if defined(__GNUC__) || defined(__INTEL_COMPILER) #ifndef __INTEL_COMPILER #define __strong_reference(sym,aliassym) \ extern __typeof (sym) aliassym __attribute__ ((__alias__ (#sym))) #endif #ifdef __STDC__ #define __weak_reference(sym,alias) \ __asm__(".weak " #alias); \ __asm__(".equ " #alias ", " #sym) #define __warn_references(sym,msg) \ __asm__(".section .gnu.warning." #sym); \ __asm__(".asciz \"" msg "\""); \ __asm__(".previous") #define __sym_compat(sym,impl,verid) \ __asm__(".symver " #impl ", " #sym "@" #verid) #define __sym_default(sym,impl,verid) \ __asm__(".symver " #impl ", " #sym "@@" #verid) #else #define __weak_reference(sym,alias) \ __asm__(".weak alias"); \ __asm__(".equ alias, sym") #define __warn_references(sym,msg) \ __asm__(".section .gnu.warning.sym"); \ __asm__(".asciz \"msg\""); \ __asm__(".previous") #define __sym_compat(sym,impl,verid) \ __asm__(".symver impl, sym@verid") #define __sym_default(impl,sym,verid) \ __asm__(".symver impl, sym@@verid") #endif /* __STDC__ */ #endif /* __GNUC__ || __INTEL_COMPILER */ #define __GLOBL1(sym) __asm__(".globl " #sym) #define __GLOBL(sym) __GLOBL1(sym) #if defined(__GNUC__) || defined(__INTEL_COMPILER) #define __IDSTRING(name,string) __asm__(".ident\t\"" string "\"") #else /* * The following definition might not work well if used in header files, * but it should be better than nothing. If you want a "do nothing" * version, then it should generate some harmless declaration, such as: * #define __IDSTRING(name,string) struct __hack */ #define __IDSTRING(name,string) static const char name[] __unused = string #endif /* * Embed the rcs id of a source file in the resulting library. Note that in * more recent ELF binutils, we use .ident allowing the ID to be stripped. * Usage: * __FBSDID("$FreeBSD$"); */ #ifndef __FBSDID #if !defined(lint) && !defined(STRIP_FBSDID) #define __FBSDID(s) __IDSTRING(__CONCAT(__rcsid_,__LINE__),s) #else #define __FBSDID(s) struct __hack #endif #endif #ifndef __RCSID #ifndef NO__RCSID #define __RCSID(s) __IDSTRING(__CONCAT(__rcsid_,__LINE__),s) #else #define __RCSID(s) struct __hack #endif #endif #ifndef __RCSID_SOURCE #ifndef NO__RCSID_SOURCE #define __RCSID_SOURCE(s) __IDSTRING(__CONCAT(__rcsid_source_,__LINE__),s) #else #define __RCSID_SOURCE(s) struct __hack #endif #endif #ifndef __SCCSID #ifndef NO__SCCSID #define __SCCSID(s) __IDSTRING(__CONCAT(__sccsid_,__LINE__),s) #else #define __SCCSID(s) struct __hack #endif #endif #ifndef __COPYRIGHT #ifndef NO__COPYRIGHT #define __COPYRIGHT(s) __IDSTRING(__CONCAT(__copyright_,__LINE__),s) #else #define __COPYRIGHT(s) struct __hack #endif #endif #ifndef __DECONST #define __DECONST(type, var) ((type)(__uintptr_t)(const void *)(var)) #endif #ifndef __DEVOLATILE #define __DEVOLATILE(type, var) ((type)(__uintptr_t)(volatile void *)(var)) #endif #ifndef __DEQUALIFY #define __DEQUALIFY(type, var) ((type)(__uintptr_t)(const volatile void *)(var)) #endif /*- * The following definitions are an extension of the behavior originally * implemented in , but with a different level of granularity. * POSIX.1 requires that the macros we test be defined before any standard * header file is included. * * Here's a quick run-down of the versions: * defined(_POSIX_SOURCE) 1003.1-1988 * _POSIX_C_SOURCE == 1 1003.1-1990 * _POSIX_C_SOURCE == 2 1003.2-1992 C Language Binding Option * _POSIX_C_SOURCE == 199309 1003.1b-1993 * _POSIX_C_SOURCE == 199506 1003.1c-1995, 1003.1i-1995, * and the omnibus ISO/IEC 9945-1: 1996 * _POSIX_C_SOURCE == 200112 1003.1-2001 * _POSIX_C_SOURCE == 200809 1003.1-2008 * * In addition, the X/Open Portability Guide, which is now the Single UNIX * Specification, defines a feature-test macro which indicates the version of * that specification, and which subsumes _POSIX_C_SOURCE. * * Our macros begin with two underscores to avoid namespace screwage. */ /* Deal with IEEE Std. 1003.1-1990, in which _POSIX_C_SOURCE == 1. */ #if defined(_POSIX_C_SOURCE) && _POSIX_C_SOURCE == 1 #undef _POSIX_C_SOURCE /* Probably illegal, but beyond caring now. */ #define _POSIX_C_SOURCE 199009 #endif /* Deal with IEEE Std. 1003.2-1992, in which _POSIX_C_SOURCE == 2. */ #if defined(_POSIX_C_SOURCE) && _POSIX_C_SOURCE == 2 #undef _POSIX_C_SOURCE #define _POSIX_C_SOURCE 199209 #endif /* Deal with various X/Open Portability Guides and Single UNIX Spec. */ #ifdef _XOPEN_SOURCE #if _XOPEN_SOURCE - 0 >= 700 #define __XSI_VISIBLE 700 #undef _POSIX_C_SOURCE #define _POSIX_C_SOURCE 200809 #elif _XOPEN_SOURCE - 0 >= 600 #define __XSI_VISIBLE 600 #undef _POSIX_C_SOURCE #define _POSIX_C_SOURCE 200112 #elif _XOPEN_SOURCE - 0 >= 500 #define __XSI_VISIBLE 500 #undef _POSIX_C_SOURCE #define _POSIX_C_SOURCE 199506 #endif #endif /* * Deal with all versions of POSIX. The ordering relative to the tests above is * important. */ #if defined(_POSIX_SOURCE) && !defined(_POSIX_C_SOURCE) #define _POSIX_C_SOURCE 198808 #endif #ifdef _POSIX_C_SOURCE #if _POSIX_C_SOURCE >= 200809 #define __POSIX_VISIBLE 200809 #define __ISO_C_VISIBLE 1999 #elif _POSIX_C_SOURCE >= 200112 #define __POSIX_VISIBLE 200112 #define __ISO_C_VISIBLE 1999 #elif _POSIX_C_SOURCE >= 199506 #define __POSIX_VISIBLE 199506 #define __ISO_C_VISIBLE 1990 #elif _POSIX_C_SOURCE >= 199309 #define __POSIX_VISIBLE 199309 #define __ISO_C_VISIBLE 1990 #elif _POSIX_C_SOURCE >= 199209 #define __POSIX_VISIBLE 199209 #define __ISO_C_VISIBLE 1990 #elif _POSIX_C_SOURCE >= 199009 #define __POSIX_VISIBLE 199009 #define __ISO_C_VISIBLE 1990 #else #define __POSIX_VISIBLE 198808 #define __ISO_C_VISIBLE 0 #endif /* _POSIX_C_SOURCE */ #else /*- * Deal with _ANSI_SOURCE: * If it is defined, and no other compilation environment is explicitly * requested, then define our internal feature-test macros to zero. This * makes no difference to the preprocessor (undefined symbols in preprocessing * expressions are defined to have value zero), but makes it more convenient for * a test program to print out the values. * * If a program mistakenly defines _ANSI_SOURCE and some other macro such as * _POSIX_C_SOURCE, we will assume that it wants the broader compilation * environment (and in fact we will never get here). */ #if defined(_ANSI_SOURCE) /* Hide almost everything. */ #define __POSIX_VISIBLE 0 #define __XSI_VISIBLE 0 #define __BSD_VISIBLE 0 #define __ISO_C_VISIBLE 1990 #elif defined(_C99_SOURCE) /* Localism to specify strict C99 env. */ #define __POSIX_VISIBLE 0 #define __XSI_VISIBLE 0 #define __BSD_VISIBLE 0 #define __ISO_C_VISIBLE 1999 #elif defined(_C11_SOURCE) /* Localism to specify strict C11 env. */ #define __POSIX_VISIBLE 0 #define __XSI_VISIBLE 0 #define __BSD_VISIBLE 0 #define __ISO_C_VISIBLE 2011 #else /* Default environment: show everything. */ #define __POSIX_VISIBLE 200809 #define __XSI_VISIBLE 700 #define __BSD_VISIBLE 1 #define __ISO_C_VISIBLE 2011 #endif #endif -#if defined(__mips) || defined(__powerpc64__) +#if defined(__mips) || defined(__powerpc64__) || defined(__riscv__) #define __NO_TLS 1 #endif /* * Type Safety Checking * * Clang provides additional attributes to enable checking type safety * properties that cannot be enforced by the C type system. */ #if __has_attribute(__argument_with_type_tag__) && \ __has_attribute(__type_tag_for_datatype__) && !defined(lint) #define __arg_type_tag(arg_kind, arg_idx, type_tag_idx) \ __attribute__((__argument_with_type_tag__(arg_kind, arg_idx, type_tag_idx))) #define __datatype_type_tag(kind, type) \ __attribute__((__type_tag_for_datatype__(kind, type))) #else #define __arg_type_tag(arg_kind, arg_idx, type_tag_idx) #define __datatype_type_tag(kind, type) #endif /* * Lock annotations. * * Clang provides support for doing basic thread-safety tests at * compile-time, by marking which locks will/should be held when * entering/leaving a functions. * * Furthermore, it is also possible to annotate variables and structure * members to enforce that they are only accessed when certain locks are * held. */ #if __has_extension(c_thread_safety_attributes) #define __lock_annotate(x) __attribute__((x)) #else #define __lock_annotate(x) #endif /* Structure implements a lock. */ #define __lockable __lock_annotate(lockable) /* Function acquires an exclusive or shared lock. */ #define __locks_exclusive(...) \ __lock_annotate(exclusive_lock_function(__VA_ARGS__)) #define __locks_shared(...) \ __lock_annotate(shared_lock_function(__VA_ARGS__)) /* Function attempts to acquire an exclusive or shared lock. */ #define __trylocks_exclusive(...) \ __lock_annotate(exclusive_trylock_function(__VA_ARGS__)) #define __trylocks_shared(...) \ __lock_annotate(shared_trylock_function(__VA_ARGS__)) /* Function releases a lock. */ #define __unlocks(...) __lock_annotate(unlock_function(__VA_ARGS__)) /* Function asserts that an exclusive or shared lock is held. */ #define __asserts_exclusive(...) \ __lock_annotate(assert_exclusive_lock(__VA_ARGS__)) #define __asserts_shared(...) \ __lock_annotate(assert_shared_lock(__VA_ARGS__)) /* Function requires that an exclusive or shared lock is or is not held. */ #define __requires_exclusive(...) \ __lock_annotate(exclusive_locks_required(__VA_ARGS__)) #define __requires_shared(...) \ __lock_annotate(shared_locks_required(__VA_ARGS__)) #define __requires_unlocked(...) \ __lock_annotate(locks_excluded(__VA_ARGS__)) /* Function should not be analyzed. */ #define __no_lock_analysis __lock_annotate(no_thread_safety_analysis) /* Guard variables and structure members by lock. */ #define __guarded_by(x) __lock_annotate(guarded_by(x)) #define __pt_guarded_by(x) __lock_annotate(pt_guarded_by(x)) #endif /* !_SYS_CDEFS_H_ */ Index: user/ngie/make_check/sys/sys/kerneldump.h =================================================================== --- user/ngie/make_check/sys/sys/kerneldump.h (revision 295077) +++ user/ngie/make_check/sys/sys/kerneldump.h (revision 295078) @@ -1,130 +1,131 @@ /*- * Copyright (c) 2002 Poul-Henning Kamp * Copyright (c) 2002 Networks Associates Technology, Inc. * All rights reserved. * * This software was developed for the FreeBSD Project by Poul-Henning Kamp * and NAI Labs, the Security Research Division of Network Associates, Inc. * under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the * DARPA CHATS research program. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The names of the authors may not be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _SYS_KERNELDUMP_H #define _SYS_KERNELDUMP_H #include #if BYTE_ORDER == LITTLE_ENDIAN #define dtoh32(x) __bswap32(x) #define dtoh64(x) __bswap64(x) #define htod32(x) __bswap32(x) #define htod64(x) __bswap64(x) #elif BYTE_ORDER == BIG_ENDIAN #define dtoh32(x) (x) #define dtoh64(x) (x) #define htod32(x) (x) #define htod64(x) (x) #endif /* * All uintX_t fields are in dump byte order, which is the same as * network byte order. Use the macros defined above to read or * write the fields. */ struct kerneldumpheader { char magic[20]; #define KERNELDUMPMAGIC "FreeBSD Kernel Dump" #define TEXTDUMPMAGIC "FreeBSD Text Dump" #define KERNELDUMPMAGIC_CLEARED "Cleared Kernel Dump" char architecture[12]; uint32_t version; #define KERNELDUMPVERSION 1 uint32_t architectureversion; #define KERNELDUMP_AARCH64_VERSION 1 #define KERNELDUMP_AMD64_VERSION 2 #define KERNELDUMP_ARM_VERSION 1 #define KERNELDUMP_I386_VERSION 2 #define KERNELDUMP_MIPS_VERSION 1 #define KERNELDUMP_POWERPC_VERSION 1 +#define KERNELDUMP_RISCV_VERSION 1 #define KERNELDUMP_SPARC64_VERSION 1 #define KERNELDUMP_TEXT_VERSION 1 uint64_t dumplength; /* excl headers */ uint64_t dumptime; uint32_t blocksize; char hostname[64]; char versionstring[192]; char panicstring[192]; uint32_t parity; }; /* * Parity calculation is endian insensitive. */ static __inline u_int32_t kerneldump_parity(struct kerneldumpheader *kdhp) { uint32_t *up, parity; u_int i; up = (uint32_t *)kdhp; parity = 0; for (i = 0; i < sizeof *kdhp; i += sizeof *up) parity ^= *up++; return (parity); } #ifdef _KERNEL struct dump_pa { vm_paddr_t pa_start; vm_paddr_t pa_size; }; void mkdumpheader(struct kerneldumpheader *kdh, char *magic, uint32_t archver, uint64_t dumplen, uint32_t blksz); int dumpsys_generic(struct dumperinfo *); void dumpsys_map_chunk(vm_paddr_t, size_t, void **); typedef int dumpsys_callback_t(struct dump_pa *, int, void *); int dumpsys_foreach_chunk(dumpsys_callback_t, void *); int dumpsys_cb_dumpdata(struct dump_pa *, int, void *); int dumpsys_buf_write(struct dumperinfo *, char *, size_t); int dumpsys_buf_flush(struct dumperinfo *); void dumpsys_gen_pa_init(void); struct dump_pa *dumpsys_gen_pa_next(struct dump_pa *); void dumpsys_gen_wbinv_all(void); void dumpsys_gen_unmap_chunk(vm_paddr_t, size_t, void *); int dumpsys_gen_write_aux_headers(struct dumperinfo *); extern int do_minidump; #endif #endif /* _SYS_KERNELDUMP_H */ Index: user/ngie/make_check/sys/sys/socket.h =================================================================== --- user/ngie/make_check/sys/sys/socket.h (revision 295077) +++ user/ngie/make_check/sys/sys/socket.h (revision 295078) @@ -1,675 +1,690 @@ /*- * Copyright (c) 1982, 1985, 1986, 1988, 1993, 1994 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * @(#)socket.h 8.4 (Berkeley) 2/21/94 * $FreeBSD$ */ #ifndef _SYS_SOCKET_H_ #define _SYS_SOCKET_H_ #include #include #include #include /* * Definitions related to sockets: types, address families, options. */ /* * Data types. */ #if __BSD_VISIBLE #ifndef _GID_T_DECLARED typedef __gid_t gid_t; #define _GID_T_DECLARED #endif #ifndef _OFF_T_DECLARED typedef __off_t off_t; #define _OFF_T_DECLARED #endif #ifndef _PID_T_DECLARED typedef __pid_t pid_t; #define _PID_T_DECLARED #endif #endif #ifndef _SA_FAMILY_T_DECLARED typedef __sa_family_t sa_family_t; #define _SA_FAMILY_T_DECLARED #endif #ifndef _SOCKLEN_T_DECLARED typedef __socklen_t socklen_t; #define _SOCKLEN_T_DECLARED #endif #ifndef _SSIZE_T_DECLARED typedef __ssize_t ssize_t; #define _SSIZE_T_DECLARED #endif #if __BSD_VISIBLE #ifndef _UID_T_DECLARED typedef __uid_t uid_t; #define _UID_T_DECLARED #endif #endif #ifndef _UINT32_T_DECLARED typedef __uint32_t uint32_t; #define _UINT32_T_DECLARED #endif #ifndef _UINTPTR_T_DECLARED typedef __uintptr_t uintptr_t; #define _UINTPTR_T_DECLARED #endif /* * Types */ #define SOCK_STREAM 1 /* stream socket */ #define SOCK_DGRAM 2 /* datagram socket */ #define SOCK_RAW 3 /* raw-protocol interface */ #if __BSD_VISIBLE #define SOCK_RDM 4 /* reliably-delivered message */ #endif #define SOCK_SEQPACKET 5 /* sequenced packet stream */ #if __BSD_VISIBLE /* * Creation flags, OR'ed into socket() and socketpair() type argument. */ #define SOCK_CLOEXEC 0x10000000 #define SOCK_NONBLOCK 0x20000000 #endif /* * Option flags per-socket. */ #define SO_DEBUG 0x0001 /* turn on debugging info recording */ #define SO_ACCEPTCONN 0x0002 /* socket has had listen() */ #define SO_REUSEADDR 0x0004 /* allow local address reuse */ #define SO_KEEPALIVE 0x0008 /* keep connections alive */ #define SO_DONTROUTE 0x0010 /* just use interface addresses */ #define SO_BROADCAST 0x0020 /* permit sending of broadcast msgs */ #if __BSD_VISIBLE #define SO_USELOOPBACK 0x0040 /* bypass hardware when possible */ #endif #define SO_LINGER 0x0080 /* linger on close if data present */ #define SO_OOBINLINE 0x0100 /* leave received OOB data in line */ #if __BSD_VISIBLE #define SO_REUSEPORT 0x0200 /* allow local address & port reuse */ #define SO_TIMESTAMP 0x0400 /* timestamp received dgram traffic */ #define SO_NOSIGPIPE 0x0800 /* no SIGPIPE from EPIPE */ #define SO_ACCEPTFILTER 0x1000 /* there is an accept filter */ #define SO_BINTIME 0x2000 /* timestamp received dgram traffic */ #endif #define SO_NO_OFFLOAD 0x4000 /* socket cannot be offloaded */ #define SO_NO_DDP 0x8000 /* disable direct data placement */ /* * Additional options, not kept in so_options. */ #define SO_SNDBUF 0x1001 /* send buffer size */ #define SO_RCVBUF 0x1002 /* receive buffer size */ #define SO_SNDLOWAT 0x1003 /* send low-water mark */ #define SO_RCVLOWAT 0x1004 /* receive low-water mark */ #define SO_SNDTIMEO 0x1005 /* send timeout */ #define SO_RCVTIMEO 0x1006 /* receive timeout */ #define SO_ERROR 0x1007 /* get error status and clear */ #define SO_TYPE 0x1008 /* get socket type */ #if __BSD_VISIBLE #define SO_LABEL 0x1009 /* socket's MAC label */ #define SO_PEERLABEL 0x1010 /* socket's peer's MAC label */ #define SO_LISTENQLIMIT 0x1011 /* socket's backlog limit */ #define SO_LISTENQLEN 0x1012 /* socket's complete queue length */ #define SO_LISTENINCQLEN 0x1013 /* socket's incomplete queue length */ #define SO_SETFIB 0x1014 /* use this FIB to route */ #define SO_USER_COOKIE 0x1015 /* user cookie (dummynet etc.) */ #define SO_PROTOCOL 0x1016 /* get socket protocol (Linux name) */ #define SO_PROTOTYPE SO_PROTOCOL /* alias for SO_PROTOCOL (SunOS name) */ #endif /* * Space reserved for new socket options added by third-party vendors. * This range applies to all socket option levels. New socket options * in FreeBSD should always use an option value less than SO_VENDOR. */ #if __BSD_VISIBLE #define SO_VENDOR 0x80000000 #endif /* * Structure used for manipulating linger option. */ struct linger { int l_onoff; /* option on/off */ int l_linger; /* linger time */ }; #if __BSD_VISIBLE struct accept_filter_arg { char af_name[16]; char af_arg[256-16]; }; #endif /* * Level number for (get/set)sockopt() to apply to socket itself. */ #define SOL_SOCKET 0xffff /* options for socket level */ /* * Address families. */ #define AF_UNSPEC 0 /* unspecified */ #if __BSD_VISIBLE #define AF_LOCAL AF_UNIX /* local to host (pipes, portals) */ #endif #define AF_UNIX 1 /* standardized name for AF_LOCAL */ #define AF_INET 2 /* internetwork: UDP, TCP, etc. */ #if __BSD_VISIBLE #define AF_IMPLINK 3 /* arpanet imp addresses */ #define AF_PUP 4 /* pup protocols: e.g. BSP */ #define AF_CHAOS 5 /* mit CHAOS protocols */ #define AF_NETBIOS 6 /* SMB protocols */ #define AF_ISO 7 /* ISO protocols */ #define AF_OSI AF_ISO #define AF_ECMA 8 /* European computer manufacturers */ #define AF_DATAKIT 9 /* datakit protocols */ #define AF_CCITT 10 /* CCITT protocols, X.25 etc */ #define AF_SNA 11 /* IBM SNA */ #define AF_DECnet 12 /* DECnet */ #define AF_DLI 13 /* DEC Direct data link interface */ #define AF_LAT 14 /* LAT */ #define AF_HYLINK 15 /* NSC Hyperchannel */ #define AF_APPLETALK 16 /* Apple Talk */ #define AF_ROUTE 17 /* Internal Routing Protocol */ #define AF_LINK 18 /* Link layer interface */ #define pseudo_AF_XTP 19 /* eXpress Transfer Protocol (no AF) */ #define AF_COIP 20 /* connection-oriented IP, aka ST II */ #define AF_CNT 21 /* Computer Network Technology */ #define pseudo_AF_RTIP 22 /* Help Identify RTIP packets */ #define AF_IPX 23 /* Novell Internet Protocol */ #define AF_SIP 24 /* Simple Internet Protocol */ #define pseudo_AF_PIP 25 /* Help Identify PIP packets */ #define AF_ISDN 26 /* Integrated Services Digital Network*/ #define AF_E164 AF_ISDN /* CCITT E.164 recommendation */ #define pseudo_AF_KEY 27 /* Internal key-management function */ #endif #define AF_INET6 28 /* IPv6 */ #if __BSD_VISIBLE #define AF_NATM 29 /* native ATM access */ #define AF_ATM 30 /* ATM */ #define pseudo_AF_HDRCMPLT 31 /* Used by BPF to not rewrite headers * in interface output routine */ #define AF_NETGRAPH 32 /* Netgraph sockets */ #define AF_SLOW 33 /* 802.3ad slow protocol */ #define AF_SCLUSTER 34 /* Sitara cluster protocol */ #define AF_ARP 35 #define AF_BLUETOOTH 36 /* Bluetooth sockets */ #define AF_IEEE80211 37 /* IEEE 802.11 protocol */ #define AF_INET_SDP 40 /* OFED Socket Direct Protocol ipv4 */ #define AF_INET6_SDP 42 /* OFED Socket Direct Protocol ipv6 */ #define AF_MAX 42 /* * When allocating a new AF_ constant, please only allocate * even numbered constants for FreeBSD until 134 as odd numbered AF_ * constants 39-133 are now reserved for vendors. */ #define AF_VENDOR00 39 #define AF_VENDOR01 41 #define AF_VENDOR02 43 #define AF_VENDOR03 45 #define AF_VENDOR04 47 #define AF_VENDOR05 49 #define AF_VENDOR06 51 #define AF_VENDOR07 53 #define AF_VENDOR08 55 #define AF_VENDOR09 57 #define AF_VENDOR10 59 #define AF_VENDOR11 61 #define AF_VENDOR12 63 #define AF_VENDOR13 65 #define AF_VENDOR14 67 #define AF_VENDOR15 69 #define AF_VENDOR16 71 #define AF_VENDOR17 73 #define AF_VENDOR18 75 #define AF_VENDOR19 77 #define AF_VENDOR20 79 #define AF_VENDOR21 81 #define AF_VENDOR22 83 #define AF_VENDOR23 85 #define AF_VENDOR24 87 #define AF_VENDOR25 89 #define AF_VENDOR26 91 #define AF_VENDOR27 93 #define AF_VENDOR28 95 #define AF_VENDOR29 97 #define AF_VENDOR30 99 #define AF_VENDOR31 101 #define AF_VENDOR32 103 #define AF_VENDOR33 105 #define AF_VENDOR34 107 #define AF_VENDOR35 109 #define AF_VENDOR36 111 #define AF_VENDOR37 113 #define AF_VENDOR38 115 #define AF_VENDOR39 117 #define AF_VENDOR40 119 #define AF_VENDOR41 121 #define AF_VENDOR42 123 #define AF_VENDOR43 125 #define AF_VENDOR44 127 #define AF_VENDOR45 129 #define AF_VENDOR46 131 #define AF_VENDOR47 133 #endif /* * Structure used by kernel to store most * addresses. */ struct sockaddr { unsigned char sa_len; /* total length */ sa_family_t sa_family; /* address family */ char sa_data[14]; /* actually longer; address value */ }; #if __BSD_VISIBLE #define SOCK_MAXADDRLEN 255 /* longest possible addresses */ /* * Structure used by kernel to pass protocol * information in raw sockets. */ struct sockproto { unsigned short sp_family; /* address family */ unsigned short sp_protocol; /* protocol */ }; #endif #include #if __BSD_VISIBLE /* * Protocol families, same as address families for now. */ #define PF_UNSPEC AF_UNSPEC #define PF_LOCAL AF_LOCAL #define PF_UNIX PF_LOCAL /* backward compatibility */ #define PF_INET AF_INET #define PF_IMPLINK AF_IMPLINK #define PF_PUP AF_PUP #define PF_CHAOS AF_CHAOS #define PF_NETBIOS AF_NETBIOS #define PF_ISO AF_ISO #define PF_OSI AF_ISO #define PF_ECMA AF_ECMA #define PF_DATAKIT AF_DATAKIT #define PF_CCITT AF_CCITT #define PF_SNA AF_SNA #define PF_DECnet AF_DECnet #define PF_DLI AF_DLI #define PF_LAT AF_LAT #define PF_HYLINK AF_HYLINK #define PF_APPLETALK AF_APPLETALK #define PF_ROUTE AF_ROUTE #define PF_LINK AF_LINK #define PF_XTP pseudo_AF_XTP /* really just proto family, no AF */ #define PF_COIP AF_COIP #define PF_CNT AF_CNT #define PF_SIP AF_SIP #define PF_IPX AF_IPX #define PF_RTIP pseudo_AF_RTIP /* same format as AF_INET */ #define PF_PIP pseudo_AF_PIP #define PF_ISDN AF_ISDN #define PF_KEY pseudo_AF_KEY #define PF_INET6 AF_INET6 #define PF_NATM AF_NATM #define PF_ATM AF_ATM #define PF_NETGRAPH AF_NETGRAPH #define PF_SLOW AF_SLOW #define PF_SCLUSTER AF_SCLUSTER #define PF_ARP AF_ARP #define PF_BLUETOOTH AF_BLUETOOTH #define PF_IEEE80211 AF_IEEE80211 #define PF_INET_SDP AF_INET_SDP #define PF_INET6_SDP AF_INET6_SDP #define PF_MAX AF_MAX /* * Definitions for network related sysctl, CTL_NET. * * Second level is protocol family. * Third level is protocol number. * * Further levels are defined by the individual families. */ /* * PF_ROUTE - Routing table * * Three additional levels are defined: * Fourth: address family, 0 is wildcard * Fifth: type of info, defined below * Sixth: flag(s) to mask with for NET_RT_FLAGS */ #define NET_RT_DUMP 1 /* dump; may limit to a.f. */ #define NET_RT_FLAGS 2 /* by flags, e.g. RESOLVING */ #define NET_RT_IFLIST 3 /* survey interface list */ #define NET_RT_IFMALIST 4 /* return multicast address list */ #define NET_RT_IFLISTL 5 /* Survey interface list, using 'l'en * versions of msghdr structs. */ #endif /* __BSD_VISIBLE */ /* * Maximum queue length specifiable by listen. */ #define SOMAXCONN 128 /* * Message header for recvmsg and sendmsg calls. * Used value-result for recvmsg, value only for sendmsg. */ struct msghdr { void *msg_name; /* optional address */ socklen_t msg_namelen; /* size of address */ struct iovec *msg_iov; /* scatter/gather array */ int msg_iovlen; /* # elements in msg_iov */ void *msg_control; /* ancillary data, see below */ socklen_t msg_controllen; /* ancillary data buffer len */ int msg_flags; /* flags on received message */ }; #define MSG_OOB 0x1 /* process out-of-band data */ #define MSG_PEEK 0x2 /* peek at incoming message */ #define MSG_DONTROUTE 0x4 /* send without using routing tables */ #define MSG_EOR 0x8 /* data completes record */ #define MSG_TRUNC 0x10 /* data discarded before delivery */ #define MSG_CTRUNC 0x20 /* control data lost before delivery */ #define MSG_WAITALL 0x40 /* wait for full request or error */ #if __POSIX_VISIBLE >= 200809 #define MSG_NOSIGNAL 0x20000 /* do not generate SIGPIPE on EOF */ #endif #if __BSD_VISIBLE #define MSG_DONTWAIT 0x80 /* this message should be nonblocking */ #define MSG_EOF 0x100 /* data completes connection */ #define MSG_NOTIFICATION 0x2000 /* SCTP notification */ #define MSG_NBIO 0x4000 /* FIONBIO mode, used by fifofs */ #define MSG_COMPAT 0x8000 /* used in sendit() */ #define MSG_CMSG_CLOEXEC 0x40000 /* make received fds close-on-exec */ +#define MSG_WAITFORONE 0x80000 /* for recvmmsg() */ #endif #ifdef _KERNEL #define MSG_SOCALLBCK 0x10000 /* for use by socket callbacks - soreceive (TCP) */ #endif /* * Header for ancillary data objects in msg_control buffer. * Used for additional information with/about a datagram * not expressible by flags. The format is a sequence * of message elements headed by cmsghdr structures. */ struct cmsghdr { socklen_t cmsg_len; /* data byte count, including hdr */ int cmsg_level; /* originating protocol */ int cmsg_type; /* protocol-specific type */ /* followed by u_char cmsg_data[]; */ }; #if __BSD_VISIBLE /* * While we may have more groups than this, the cmsgcred struct must * be able to fit in an mbuf and we have historically supported a * maximum of 16 groups. */ #define CMGROUP_MAX 16 /* * Credentials structure, used to verify the identity of a peer * process that has sent us a message. This is allocated by the * peer process but filled in by the kernel. This prevents the * peer from lying about its identity. (Note that cmcred_groups[0] * is the effective GID.) */ struct cmsgcred { pid_t cmcred_pid; /* PID of sending process */ uid_t cmcred_uid; /* real UID of sending process */ uid_t cmcred_euid; /* effective UID of sending process */ gid_t cmcred_gid; /* real GID of sending process */ short cmcred_ngroups; /* number or groups */ gid_t cmcred_groups[CMGROUP_MAX]; /* groups */ }; /* * Socket credentials. */ struct sockcred { uid_t sc_uid; /* real user id */ uid_t sc_euid; /* effective user id */ gid_t sc_gid; /* real group id */ gid_t sc_egid; /* effective group id */ int sc_ngroups; /* number of supplemental groups */ gid_t sc_groups[1]; /* variable length */ }; /* * Compute size of a sockcred structure with groups. */ #define SOCKCREDSIZE(ngrps) \ (sizeof(struct sockcred) + (sizeof(gid_t) * ((ngrps) - 1))) #endif /* __BSD_VISIBLE */ /* given pointer to struct cmsghdr, return pointer to data */ #define CMSG_DATA(cmsg) ((unsigned char *)(cmsg) + \ _ALIGN(sizeof(struct cmsghdr))) /* given pointer to struct cmsghdr, return pointer to next cmsghdr */ #define CMSG_NXTHDR(mhdr, cmsg) \ ((char *)(cmsg) == NULL ? CMSG_FIRSTHDR(mhdr) : \ ((char *)(cmsg) + _ALIGN(((struct cmsghdr *)(cmsg))->cmsg_len) + \ _ALIGN(sizeof(struct cmsghdr)) > \ (char *)(mhdr)->msg_control + (mhdr)->msg_controllen) ? \ (struct cmsghdr *)0 : \ (struct cmsghdr *)(void *)((char *)(cmsg) + \ _ALIGN(((struct cmsghdr *)(cmsg))->cmsg_len))) /* * RFC 2292 requires to check msg_controllen, in case that the kernel returns * an empty list for some reasons. */ #define CMSG_FIRSTHDR(mhdr) \ ((mhdr)->msg_controllen >= sizeof(struct cmsghdr) ? \ (struct cmsghdr *)(mhdr)->msg_control : \ (struct cmsghdr *)NULL) #if __BSD_VISIBLE /* RFC 2292 additions */ #define CMSG_SPACE(l) (_ALIGN(sizeof(struct cmsghdr)) + _ALIGN(l)) #define CMSG_LEN(l) (_ALIGN(sizeof(struct cmsghdr)) + (l)) #endif #ifdef _KERNEL #define CMSG_ALIGN(n) _ALIGN(n) #endif /* "Socket"-level control message types: */ #define SCM_RIGHTS 0x01 /* access rights (array of int) */ #if __BSD_VISIBLE #define SCM_TIMESTAMP 0x02 /* timestamp (struct timeval) */ #define SCM_CREDS 0x03 /* process creds (struct cmsgcred) */ #define SCM_BINTIME 0x04 /* timestamp (struct bintime) */ #endif #if __BSD_VISIBLE /* * 4.3 compat sockaddr, move to compat file later */ struct osockaddr { unsigned short sa_family; /* address family */ char sa_data[14]; /* up to 14 bytes of direct address */ }; /* * 4.3-compat message header (move to compat file later). */ struct omsghdr { char *msg_name; /* optional address */ int msg_namelen; /* size of address */ struct iovec *msg_iov; /* scatter/gather array */ int msg_iovlen; /* # elements in msg_iov */ char *msg_accrights; /* access rights sent/received */ int msg_accrightslen; }; #endif /* * howto arguments for shutdown(2), specified by Posix.1g. */ #define SHUT_RD 0 /* shut down the reading side */ #define SHUT_WR 1 /* shut down the writing side */ #define SHUT_RDWR 2 /* shut down both sides */ #if __BSD_VISIBLE /* for SCTP */ /* we cheat and use the SHUT_XX defines for these */ #define PRU_FLUSH_RD SHUT_RD #define PRU_FLUSH_WR SHUT_WR #define PRU_FLUSH_RDWR SHUT_RDWR #endif #if __BSD_VISIBLE /* * sendfile(2) header/trailer struct */ struct sf_hdtr { struct iovec *headers; /* pointer to an array of header struct iovec's */ int hdr_cnt; /* number of header iovec's */ struct iovec *trailers; /* pointer to an array of trailer struct iovec's */ int trl_cnt; /* number of trailer iovec's */ }; /* * Sendfile-specific flag(s) */ #define SF_NODISKIO 0x00000001 #define SF_MNOWAIT 0x00000002 /* obsolete */ #define SF_SYNC 0x00000004 #define SF_NOCACHE 0x00000010 #define SF_FLAGS(rh, flags) (((rh) << 16) | (flags)) #ifdef _KERNEL #define SFK_COMPAT 0x00000001 #define SF_READAHEAD(flags) ((flags) >> 16) #endif /* _KERNEL */ + +/* + * Sendmmsg/recvmmsg specific structure(s) + */ +struct mmsghdr { + struct msghdr msg_hdr; /* message header */ + ssize_t msg_len; /* message length */ +}; #endif /* __BSD_VISIBLE */ #ifndef _KERNEL #include __BEGIN_DECLS int accept(int, struct sockaddr * __restrict, socklen_t * __restrict); int bind(int, const struct sockaddr *, socklen_t); int connect(int, const struct sockaddr *, socklen_t); #if __BSD_VISIBLE int accept4(int, struct sockaddr * __restrict, socklen_t * __restrict, int); int bindat(int, int, const struct sockaddr *, socklen_t); int connectat(int, int, const struct sockaddr *, socklen_t); #endif int getpeername(int, struct sockaddr * __restrict, socklen_t * __restrict); int getsockname(int, struct sockaddr * __restrict, socklen_t * __restrict); int getsockopt(int, int, int, void * __restrict, socklen_t * __restrict); int listen(int, int); ssize_t recv(int, void *, size_t, int); ssize_t recvfrom(int, void *, size_t, int, struct sockaddr * __restrict, socklen_t * __restrict); ssize_t recvmsg(int, struct msghdr *, int); +#if __BSD_VISIBLE +struct timespec; +ssize_t recvmmsg(int, struct mmsghdr * __restrict, size_t, int, + const struct timespec * __restrict); +#endif ssize_t send(int, const void *, size_t, int); ssize_t sendto(int, const void *, size_t, int, const struct sockaddr *, socklen_t); ssize_t sendmsg(int, const struct msghdr *, int); #if __BSD_VISIBLE int sendfile(int, int, off_t, size_t, struct sf_hdtr *, off_t *, int); +ssize_t sendmmsg(int, struct mmsghdr * __restrict, size_t, int); int setfib(int); #endif int setsockopt(int, int, int, const void *, socklen_t); int shutdown(int, int); int sockatmark(int); int socket(int, int, int); int socketpair(int, int, int, int *); __END_DECLS #endif /* !_KERNEL */ #ifdef _KERNEL struct socket; struct tcpcb *so_sototcpcb(struct socket *so); struct inpcb *so_sotoinpcb(struct socket *so); struct sockbuf *so_sockbuf_snd(struct socket *); struct sockbuf *so_sockbuf_rcv(struct socket *); int so_state_get(const struct socket *); void so_state_set(struct socket *, int); int so_options_get(const struct socket *); void so_options_set(struct socket *, int); int so_error_get(const struct socket *); void so_error_set(struct socket *, int); int so_linger_get(const struct socket *); void so_linger_set(struct socket *, int); struct protosw *so_protosw_get(const struct socket *); void so_protosw_set(struct socket *, struct protosw *); void so_sorwakeup_locked(struct socket *so); void so_sowwakeup_locked(struct socket *so); void so_sorwakeup(struct socket *so); void so_sowwakeup(struct socket *so); void so_lock(struct socket *so); void so_unlock(struct socket *so); void so_listeners_apply_all(struct socket *so, void (*func)(struct socket *, void *), void *arg); #endif #endif /* !_SYS_SOCKET_H_ */ Index: user/ngie/make_check/sys =================================================================== --- user/ngie/make_check/sys (revision 295077) +++ user/ngie/make_check/sys (revision 295078) Property changes on: user/ngie/make_check/sys ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys:r295034-295077 Index: user/ngie/make_check/tools/tools/nanobsd/defaults.sh =================================================================== --- user/ngie/make_check/tools/tools/nanobsd/defaults.sh (revision 295077) +++ user/ngie/make_check/tools/tools/nanobsd/defaults.sh (revision 295078) @@ -1,1073 +1,1075 @@ #!/bin/sh # # Copyright (c) 2005 Poul-Henning Kamp. # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # $FreeBSD$ # set -e ####################################################################### # # Setup default values for all controlling variables. # These values can be overridden from the config file(s) # ####################################################################### # Name of this NanoBSD build. (Used to construct workdir names) NANO_NAME=full # Source tree directory NANO_SRC=/usr/src # Where nanobsd additional files live under the source tree NANO_TOOLS=tools/tools/nanobsd # Where cust_pkgng() finds packages to install NANO_PACKAGE_DIR=${NANO_SRC}/${NANO_TOOLS}/Pkg NANO_PACKAGE_LIST="*" # where package metadata gets placed NANO_PKG_META_BASE=/var/db # Object tree directory # default is subdir of /usr/obj #NANO_OBJ="" # The directory to put the final images # default is ${NANO_OBJ} #NANO_DISKIMGDIR="" # Make & parallel Make NANO_MAKE="make" NANO_PMAKE="make -j 3" # The default name for any image we create. NANO_IMGNAME="_.disk.full" # Options to put in make.conf during buildworld only CONF_BUILD=' ' # Options to put in make.conf during installworld only CONF_INSTALL=' ' # Options to put in make.conf during both build- & installworld. CONF_WORLD=' ' # Kernel config file to use NANO_KERNEL=GENERIC # Kernel modules to install. If empty, no modules are installed. # Use "default" to install all built modules. NANO_MODULES= # Customize commands. NANO_CUSTOMIZE="" # Late customize commands. NANO_LATE_CUSTOMIZE="" # Newfs paramters to use NANO_NEWFS="-b 4096 -f 512 -i 8192 -U" # The drive name of the media at runtime NANO_DRIVE=ada0 # Target media size in 512 bytes sectors NANO_MEDIASIZE=2000000 # Number of code images on media (1 or 2) NANO_IMAGES=2 # 0 -> Leave second image all zeroes so it compresses better. # 1 -> Initialize second image with a copy of the first NANO_INIT_IMG2=1 # Size of code file system in 512 bytes sectors # If zero, size will be as large as possible. NANO_CODESIZE=0 # Size of configuration file system in 512 bytes sectors # Cannot be zero. NANO_CONFSIZE=2048 # Size of data file system in 512 bytes sectors # If zero: no partition configured. # If negative: max size possible NANO_DATASIZE=0 # Size of the /etc ramdisk in 512 bytes sectors NANO_RAM_ETCSIZE=10240 # Size of the /tmp+/var ramdisk in 512 bytes sectors NANO_RAM_TMPVARSIZE=10240 # Media geometry, only relevant if bios doesn't understand LBA. NANO_SECTS=63 NANO_HEADS=16 # boot0 flags/options and configuration NANO_BOOT0CFG="-o packet -s 1 -m 3" NANO_BOOTLOADER="boot/boot0sio" # boot2 flags/options # default force serial console NANO_BOOT2CFG="-h -S115200" # Backing type of md(4) device # Can be "file" or "swap" NANO_MD_BACKING="file" # for swap type md(4) backing, write out the mbr only NANO_IMAGE_MBRONLY=true # Progress Print level PPLEVEL=3 # Set NANO_LABEL to non-blank to form the basis for using /dev/ufs/label # in preference to /dev/${NANO_DRIVE} # Root partition will be ${NANO_LABEL}s{1,2} # /cfg partition will be ${NANO_LABEL}s3 # /data partition will be ${NANO_LABEL}s4 NANO_LABEL="" NANO_SLICE_ROOT=s1 NANO_SLICE_ALTROOT=s2 NANO_SLICE_CFG=s3 NANO_SLICE_DATA=s4 +NANO_ROOT=s1a +NANO_ALTROOT=s2a # Default ownwership for nopriv build NANO_DEF_UNAME=root NANO_DEF_GNAME=wheel ####################################################################### # Architecture to build. Corresponds to TARGET_ARCH in a buildworld. # Unfortunately, there's no way to set TARGET at this time, and it # conflates the two, so architectures where TARGET != TARGET_ARCH and # TARGET can't be guessed from TARGET_ARCH do not work. This defaults # to the arch of the current machine. NANO_ARCH=`uname -p` # CPUTYPE defaults to "" which is the default when CPUTYPE isn't # defined. NANO_CPUTYPE="" # Directory to populate /cfg from NANO_CFGDIR="" # Directory to populate /data from NANO_DATADIR="" # We don't need SRCCONF or SRC_ENV_CONF. NanoBSD puts everything we # need for the build in files included with __MAKE_CONF. Override in your # config file if you really must. We set them unconditionally here, though # in case they are stray in the build environment SRCCONF=/dev/null SRC_ENV_CONF=/dev/null ####################################################################### # # The functions which do the real work. # Can be overridden from the config file(s) # ####################################################################### # Export values into the shell. Must use { } instead of ( ) like # other functions to avoid a subshell. # We set __MAKE_CONF as a global since it is easier to get quoting # right for paths with spaces in them. make_export ( ) { # Similar to export_var, except puts the data out to stdout var=$1 eval val=\$$var echo "Setting variable: $var=\"$val\"" export $1 } nano_make_build_env ( ) { __MAKE_CONF="${NANO_MAKE_CONF_BUILD}" make_export __MAKE_CONF } nano_make_install_env ( ) { __MAKE_CONF="${NANO_MAKE_CONF_INSTALL}" make_export __MAKE_CONF } # Extra environment variables for kernel builds nano_make_kernel_env ( ) { if [ -f ${NANO_KERNEL} ] ; then KERNCONFDIR="$(realpath $(dirname ${NANO_KERNEL}))" KERNCONF="$(basename ${NANO_KERNEL})" make_export KERNCONFDIR make_export KERNCONF else export KERNCONF="${NANO_KERNEL}" make_export KERNCONF fi } nano_global_make_env ( ) ( # global settings for the make.conf file, if set [ -z "${NANO_ARCH}" ] || echo TARGET_ARCH="${NANO_ARCH}" [ -z "${NANO_CPUTYPE}" ] || echo TARGET_CPUTYPE="${NANO_CPUTYPE}" ) # rm doesn't know -x prior to FreeBSD 10, so cope with a variety of build # hosts for now. This will go away when support in the base goes away. rm ( ) { echo "NANO RM $*" case $(uname -r) in 7*|8*|9*) command rm $* ;; *) command rm -x $* ;; esac } # # Create empty files in the target tree, and record the fact. All paths # are relative to NANO_WORLDDIR. # tgt_touch ( ) ( cd "${NANO_WORLDDIR}" for i; do touch $i echo "./${i} type=file" >> ${NANO_METALOG} done ) # # Convert a directory into a symlink. Takes two arguments, the # current directory and what it should become a symlink to. The # directory is removed and a symlink is created. If we're doing # a nopriv build, then append this fact to the metalog # tgt_dir2symlink () ( dir=$1 symlink=$2 cd "${NANO_WORLDDIR}" rm -rf "$dir" ln -s "$symlink" "$dir" if [ -n "$NANO_METALOG" ]; then echo "./${dir} type=link mode=0777 link=${symlink}" >> ${NANO_METALOG} fi ) # run in the world chroot, errors fatal CR ( ) { chroot "${NANO_WORLDDIR}" /bin/sh -exc "$*" } # run in the world chroot, errors not fatal CR0 ( ) { chroot "${NANO_WORLDDIR}" /bin/sh -c "$*" || true } nano_cleanup ( ) ( [ $? -eq 0 ] || echo "Error encountered. Check for errors in last log file." 1>&2 exit $? ) clean_build ( ) ( pprint 2 "Clean and create object directory (${MAKEOBJDIRPREFIX})" if ! rm -rf ${MAKEOBJDIRPREFIX}/ > /dev/null 2>&1 ; then chflags -R noschg ${MAKEOBJDIRPREFIX}/ rm -r ${MAKEOBJDIRPREFIX}/ fi ) make_conf_build ( ) ( pprint 2 "Construct build make.conf ($NANO_MAKE_CONF_BUILD)" mkdir -p ${MAKEOBJDIRPREFIX} printenv > ${MAKEOBJDIRPREFIX}/_.env # Make sure we get all the global settings that NanoBSD wants # in addition to the user's global settings ( nano_global_make_env echo "${CONF_WORLD}" echo "${CONF_BUILD}" ) > ${NANO_MAKE_CONF_BUILD} ) build_world ( ) ( pprint 2 "run buildworld" pprint 3 "log: ${MAKEOBJDIRPREFIX}/_.bw" ( nano_make_build_env set -o xtrace cd "${NANO_SRC}" ${NANO_PMAKE} buildworld ) > ${MAKEOBJDIRPREFIX}/_.bw 2>&1 ) build_kernel ( ) ( local extra pprint 2 "build kernel ($NANO_KERNEL)" pprint 3 "log: ${MAKEOBJDIRPREFIX}/_.bk" ( nano_make_build_env nano_make_kernel_env # Note: We intentionally build all modules, not only the ones in # NANO_MODULES so the built world can be reused by multiple images. # Although MODULES_OVERRIDE can be defined in the kenrel config # file to override this behavior. Just set NANO_MODULES=default. set -o xtrace cd "${NANO_SRC}" ${NANO_PMAKE} buildkernel ) > ${MAKEOBJDIRPREFIX}/_.bk 2>&1 ) clean_world ( ) ( if [ "${NANO_OBJ}" != "${MAKEOBJDIRPREFIX}" ]; then pprint 2 "Clean and create object directory (${NANO_OBJ})" if ! rm -rf ${NANO_OBJ}/ > /dev/null 2>&1 ; then chflags -R noschg ${NANO_OBJ} rm -r ${NANO_OBJ}/ fi mkdir -p "${NANO_OBJ}" "${NANO_WORLDDIR}" printenv > ${NANO_OBJ}/_.env else pprint 2 "Clean and create world directory (${NANO_WORLDDIR})" if ! rm -rf "${NANO_WORLDDIR}/" > /dev/null 2>&1 ; then chflags -R noschg "${NANO_WORLDDIR}" rm -rf "${NANO_WORLDDIR}/" fi mkdir -p "${NANO_WORLDDIR}" fi ) make_conf_install ( ) ( pprint 2 "Construct install make.conf ($NANO_MAKE_CONF_INSTALL)" # Make sure we get all the global settings that NanoBSD wants # in addition to the user's global settings ( nano_global_make_env echo "${CONF_WORLD}" echo "${CONF_INSTALL}" if [ -n "${NANO_NOPRIV_BUILD}" ]; then echo NO_ROOT=t echo METALOG=${NANO_METALOG} fi ) > ${NANO_MAKE_CONF_INSTALL} ) install_world ( ) ( pprint 2 "installworld" pprint 3 "log: ${NANO_OBJ}/_.iw" ( nano_make_install_env set -o xtrace cd "${NANO_SRC}" ${NANO_MAKE} installworld DESTDIR="${NANO_WORLDDIR}" chflags -R noschg "${NANO_WORLDDIR}" ) > ${NANO_OBJ}/_.iw 2>&1 ) install_etc ( ) ( pprint 2 "install /etc" pprint 3 "log: ${NANO_OBJ}/_.etc" ( nano_make_install_env set -o xtrace cd "${NANO_SRC}" ${NANO_MAKE} distribution DESTDIR="${NANO_WORLDDIR}" # make.conf doesn't get created by default, but some ports need it # so they can spam it. cp /dev/null "${NANO_WORLDDIR}"/etc/make.conf ) > ${NANO_OBJ}/_.etc 2>&1 ) install_kernel ( ) ( local extra pprint 2 "install kernel ($NANO_KERNEL)" pprint 3 "log: ${NANO_OBJ}/_.ik" ( nano_make_install_env nano_make_kernel_env if [ "${NANO_MODULES}" != "default" ]; then MODULES_OVERRIDE="${NANO_MODULES}" make_export MODULES_OVERRIDE fi set -o xtrace cd "${NANO_SRC}" ${NANO_MAKE} installkernel DESTDIR="${NANO_WORLDDIR}" ) > ${NANO_OBJ}/_.ik 2>&1 ) native_xtools ( ) ( print 2 "Installing the optimized native build tools for cross env" pprint 3 "log: ${NANO_OBJ}/_.native_xtools" ( nano_make_install_env set -o xtrace cd "${NANO_SRC}" ${NANO_MAKE} native-xtools DESTDIR="${NANO_WORLDDIR}" ) > ${NANO_OBJ}/_.native_xtools 2>&1 ) # # Run the requested set of customization scripts, run after we've # done an installworld, installed the etc files, installed the kernel # and tweaked them in the standard way. # run_customize ( ) ( pprint 2 "run customize scripts" for c in $NANO_CUSTOMIZE do pprint 2 "customize \"$c\"" pprint 3 "log: ${NANO_OBJ}/_.cust.$c" pprint 4 "`type $c`" ( set -x ; $c ) > ${NANO_OBJ}/_.cust.$c 2>&1 done ) # # Run any last-minute customization commands after we've had a chance to # setup nanobsd, prune empty dirs from /usr, etc # run_late_customize ( ) ( pprint 2 "run late customize scripts" for c in $NANO_LATE_CUSTOMIZE do pprint 2 "late customize \"$c\"" pprint 3 "log: ${NANO_OBJ}/_.late_cust.$c" pprint 4 "`type $c`" ( set -x ; $c ) > ${NANO_OBJ}/_.late_cust.$c 2>&1 done ) # # Hook called after we run all the late customize commands, but # before we invoke the disk imager. The nopriv build uses it to # read in the meta log, apply the changes other parts of nanobsd # have been recording their actions. It's not anticipated that # a user's cfg file would override this. # fixup_before_diskimage ( ) ( # Run the deduplication script that takes the matalog journal and # combines multiple entries for the same file (see source for # details). We take the extra step of removing the size keywords. This # script, and many of the user scripts, copies, appeneds and otherwise # modifies files in the build, changing their sizes. These actions are # impossible to trap, so go ahead remove the size= keyword. For this # narrow use, it doesn't buy us any protection and just gets in the way. # The dedup tool's output must be sorted due to limitations in awk. if [ -n "${NANO_METALOG}" ]; then pprint 2 "Fixing metalog" cp ${NANO_METALOG} ${NANO_METALOG}.pre echo "/set uname=${NANO_DEF_UNAME} gname=${NANO_DEF_GNAME}" > ${NANO_METALOG} cat ${NANO_METALOG}.pre | ${NANO_TOOLS}/mtree-dedup.awk | \ sed -e 's/ size=[0-9][0-9]*//' | sort >> ${NANO_METALOG} fi ) setup_nanobsd ( ) ( pprint 2 "configure nanobsd setup" pprint 3 "log: ${NANO_OBJ}/_.dl" ( cd "${NANO_WORLDDIR}" # Move /usr/local/etc to /etc/local so that the /cfg stuff # can stomp on it. Otherwise packages like ipsec-tools which # have hardcoded paths under ${prefix}/etc are not tweakable. if [ -d usr/local/etc ] ; then ( mkdir -p etc/local cd usr/local/etc find . -print | cpio -dumpl ../../../etc/local cd .. rm -rf etc ln -s ../../etc/local etc ) fi for d in var etc do # link /$d under /conf # we use hard links so we have them both places. # the files in /$d will be hidden by the mount. mkdir -p conf/base/$d conf/default/$d find $d -print | cpio -dumpl conf/base/ done echo "$NANO_RAM_ETCSIZE" > conf/base/etc/md_size echo "$NANO_RAM_TMPVARSIZE" > conf/base/var/md_size # pick up config files from the special partition echo "mount -o ro /dev/${NANO_DRIVE}${NANO_SLICE_CFG}" > conf/default/etc/remount # Put /tmp on the /var ramdisk (could be symlink already) tgt_dir2symlink tmp var/tmp ) > ${NANO_OBJ}/_.dl 2>&1 ) setup_nanobsd_etc ( ) ( pprint 2 "configure nanobsd /etc" ( cd "${NANO_WORLDDIR}" # create diskless marker file touch etc/diskless [ -n "${NANO_NOPRIV_BUILD}" ] && chmod 666 etc/defaults/rc.conf # Make root filesystem R/O by default echo "root_rw_mount=NO" >> etc/defaults/rc.conf # Disable entropy file, since / is read-only /var/db/entropy should be enough? echo "entropy_file=NO" >> etc/defaults/rc.conf [ -n "${NANO_NOPRIV_BUILD}" ] && chmod 444 etc/defaults/rc.conf # save config file for scripts echo "NANO_DRIVE=${NANO_DRIVE}" > etc/nanobsd.conf - echo "/dev/${NANO_DRIVE}${NANO_SLICE_ROOT}a / ufs ro 1 1" > etc/fstab + echo "/dev/${NANO_DRIVE}${NANO_ROOT} / ufs ro 1 1" > etc/fstab echo "/dev/${NANO_DRIVE}${NANO_SLICE_CFG} /cfg ufs rw,noauto 2 2" >> etc/fstab mkdir -p cfg ) ) prune_usr ( ) ( # Remove all empty directories in /usr find "${NANO_WORLDDIR}"/usr -type d -depth -print | while read d do rmdir $d > /dev/null 2>&1 || true done ) newfs_part ( ) ( local dev mnt lbl dev=$1 mnt=$2 lbl=$3 echo newfs ${NANO_NEWFS} ${NANO_LABEL:+-L${NANO_LABEL}${lbl}} ${dev} newfs ${NANO_NEWFS} ${NANO_LABEL:+-L${NANO_LABEL}${lbl}} ${dev} mount -o async ${dev} ${mnt} ) # Convenient spot to work around any umount issues that your build environment # hits by overriding this method. nano_umount ( ) ( umount ${1} ) populate_slice ( ) ( local dev dir mnt lbl dev=$1 dir=$2 mnt=$3 lbl=$4 echo "Creating ${dev} (mounting on ${mnt})" newfs_part ${dev} ${mnt} ${lbl} if [ -n "${dir}" -a -d "${dir}" ]; then echo "Populating ${lbl} from ${dir}" cd "${dir}" find . -print | grep -Ev '/(CVS|\.svn|\.hg|\.git)' | cpio -dumpv ${mnt} fi df -i ${mnt} nano_umount ${mnt} ) populate_cfg_slice ( ) ( populate_slice "$1" "$2" "$3" "$4" ) populate_data_slice ( ) ( populate_slice "$1" "$2" "$3" "$4" ) create_diskimage ( ) ( pprint 2 "build diskimage" pprint 3 "log: ${NANO_OBJ}/_.di" ( echo $NANO_MEDIASIZE $NANO_IMAGES \ $NANO_SECTS $NANO_HEADS \ $NANO_CODESIZE $NANO_CONFSIZE $NANO_DATASIZE | awk ' { printf "# %s\n", $0 # size of cylinder in sectors cs = $3 * $4 # number of full cylinders on media cyl = int ($1 / cs) # output fdisk geometry spec, truncate cyls to 1023 if (cyl <= 1023) print "g c" cyl " h" $4 " s" $3 else print "g c" 1023 " h" $4 " s" $3 if ($7 > 0) { # size of data partition in full cylinders dsl = int (($7 + cs - 1) / cs) } else { dsl = 0; } # size of config partition in full cylinders csl = int (($6 + cs - 1) / cs) if ($5 == 0) { # size of image partition(s) in full cylinders isl = int ((cyl - dsl - csl) / $2) } else { isl = int (($5 + cs - 1) / cs) } # First image partition start at second track print "p 1 165 " $3, isl * cs - $3 c = isl * cs; # Second image partition (if any) also starts offset one # track to keep them identical. if ($2 > 1) { print "p 2 165 " $3 + c, isl * cs - $3 c += isl * cs; } # Config partition starts at cylinder boundary. print "p 3 165 " c, csl * cs c += csl * cs # Data partition (if any) starts at cylinder boundary. if ($7 > 0) { print "p 4 165 " c, dsl * cs } else if ($7 < 0 && $1 > c) { print "p 4 165 " c, $1 - c } else if ($1 < c) { print "Disk space overcommitted by", \ c - $1, "sectors" > "/dev/stderr" exit 2 } # Force slice 1 to be marked active. This is necessary # for booting the image from a USB device to work. print "a 1" } ' > ${NANO_OBJ}/_.fdisk IMG=${NANO_DISKIMGDIR}/${NANO_IMGNAME} MNT=${NANO_OBJ}/_.mnt mkdir -p ${MNT} if [ "${NANO_MD_BACKING}" = "swap" ] ; then MD=`mdconfig -a -t swap -s ${NANO_MEDIASIZE} -x ${NANO_SECTS} \ -y ${NANO_HEADS}` else echo "Creating md backing file..." rm -f ${IMG} dd if=/dev/zero of=${IMG} seek=${NANO_MEDIASIZE} count=0 MD=`mdconfig -a -t vnode -f ${IMG} -x ${NANO_SECTS} \ -y ${NANO_HEADS}` fi trap "echo 'Running exit trap code' ; df -i ${MNT} ; nano_umount ${MNT} || true ; mdconfig -d -u $MD" 1 2 15 EXIT fdisk -i -f ${NANO_OBJ}/_.fdisk ${MD} fdisk ${MD} # XXX: params # XXX: pick up cached boot* files, they may not be in image anymore. if [ -f ${NANO_WORLDDIR}/${NANO_BOOTLOADER} ]; then boot0cfg -B -b ${NANO_WORLDDIR}/${NANO_BOOTLOADER} ${NANO_BOOT0CFG} ${MD} fi if [ -f ${NANO_WORLDDIR}/boot/boot ]; then bsdlabel -w -B -b ${NANO_WORLDDIR}/boot/boot ${MD}${NANO_SLICE_ROOT} else bsdlabel -w ${MD}${NANO_SLICE_ROOT} fi bsdlabel ${MD}${NANO_SLICE_ROOT} # Create first image - populate_slice /dev/${MD}${NANO_SLICE_ROOT}a ${NANO_WORLDDIR} ${MNT} "${NANO_SLICE_ROOT}a" - mount /dev/${MD}${NANO_SLICE_ROOT}a ${MNT} + populate_slice /dev/${MD}${NANO_ROOT} ${NANO_WORLDDIR} ${MNT} "${NANO_ROOT}" + mount /dev/${MD}${NANO_ROOT} ${MNT} echo "Generating mtree..." ( cd "${MNT}" && mtree -c ) > ${NANO_OBJ}/_.mtree ( cd "${MNT}" && du -k ) > ${NANO_OBJ}/_.du nano_umount "${MNT}" if [ $NANO_IMAGES -gt 1 -a $NANO_INIT_IMG2 -gt 0 ] ; then # Duplicate to second image (if present) echo "Duplicating to second image..." dd conv=sparse if=/dev/${MD}${NANO_SLICE_ROOT} of=/dev/${MD}${NANO_SLICE_ALTROOT} bs=64k - mount /dev/${MD}${NANO_SLICE_ALTROOT}a ${MNT} + mount /dev/${MD}${NANO_ALTROOT} ${MNT} for f in ${MNT}/etc/fstab ${MNT}/conf/base/etc/fstab do sed -i "" "s=${NANO_DRIVE}${NANO_SLICE_ROOT}=${NANO_DRIVE}${NANO_SLICE_ALTROOT}=g" $f done nano_umount ${MNT} # Override the label from the first partition so we # don't confuse glabel with duplicates. if [ -n "${NANO_LABEL}" ]; then - tunefs -L ${NANO_LABEL}"${NANO_SLICE_ALTROOT}a" /dev/${MD}${NANO_SLICE_ALTROOT}a + tunefs -L ${NANO_LABEL}"${NANO_ALTROOT}" /dev/${MD}${NANO_ALTROOT} fi fi # Create Config slice populate_cfg_slice /dev/${MD}${NANO_SLICE_CFG} "${NANO_CFGDIR}" ${MNT} "${NANO_SLICE_CFG}" # Create Data slice, if any. if [ -n "$NANO_SLICE_DATA" -a "$NANO_SLICE_CFG" = "$NANO_SLICE_DATA" -a \ "$NANO_DATASIZE" -ne 0 ]; then pprint 2 "NANO_SLICE_DATA is the same as NANO_SLICE_CFG, fix." exit 2 fi if [ $NANO_DATASIZE -ne 0 -a -n "$NANO_SLICE_DATA" ] ; then populate_data_slice /dev/${MD}${NANO_SLICE_DATA} "${NANO_DATADIR}" ${MNT} "${NANO_SLICE_DATA}" fi if [ "${NANO_MD_BACKING}" = "swap" ] ; then if [ ${NANO_IMAGE_MBRONLY} ]; then echo "Writing out _.disk.mbr..." dd if=/dev/${MD} of=${NANO_DISKIMGDIR}/_.disk.mbr bs=512 count=1 else echo "Writing out ${NANO_IMGNAME}..." dd if=/dev/${MD} of=${IMG} bs=64k fi echo "Writing out ${NANO_IMGNAME}..." dd conv=sparse if=/dev/${MD} of=${IMG} bs=64k fi if ${do_copyout_partition} ; then echo "Writing out _.disk.image..." dd conv=sparse if=/dev/${MD}${NANO_SLICE_ROOT} of=${NANO_DISKIMGDIR}/_.disk.image bs=64k fi mdconfig -d -u $MD trap - 1 2 15 trap nano_cleanup EXIT ) > ${NANO_OBJ}/_.di 2>&1 ) last_orders ( ) ( # Redefine this function with any last orders you may have # after the build completed, for instance to copy the finished # image to a more convenient place: # cp ${NANO_DISKIMGDIR}/_.disk.image /home/ftp/pub/nanobsd.disk true ) ####################################################################### # # Optional convenience functions. # ####################################################################### ####################################################################### # Common Flash device geometries # FlashDevice ( ) { if [ -d ${NANO_TOOLS} ] ; then . ${NANO_TOOLS}/FlashDevice.sub else . ${NANO_SRC}/${NANO_TOOLS}/FlashDevice.sub fi sub_FlashDevice $1 $2 } ####################################################################### # USB device geometries # # Usage: # UsbDevice Generic 1000 # a generic flash key sold as having 1GB # # This function will set NANO_MEDIASIZE, NANO_HEADS and NANO_SECTS for you. # # Note that the capacity of a flash key is usually advertised in MB or # GB, *not* MiB/GiB. As such, the precise number of cylinders available # for C/H/S geometry may vary depending on the actual flash geometry. # # The following generic device layouts are understood: # generic An alias for generic-hdd. # generic-hdd 255H 63S/T xxxxC with no MBR restrictions. # generic-fdd 64H 32S/T xxxxC with no MBR restrictions. # # The generic-hdd device is preferred for flash devices larger than 1GB. # UsbDevice ( ) { a1=`echo $1 | tr '[:upper:]' '[:lower:]'` case $a1 in generic-fdd) NANO_HEADS=64 NANO_SECTS=32 NANO_MEDIASIZE=$(( $2 * 1000 * 1000 / 512 )) ;; generic|generic-hdd) NANO_HEADS=255 NANO_SECTS=63 NANO_MEDIASIZE=$(( $2 * 1000 * 1000 / 512 )) ;; *) echo "Unknown USB flash device" exit 2 ;; esac } ####################################################################### # Setup serial console cust_comconsole ( ) ( # Enable getty on console sed -i "" -e /tty[du]0/s/off/on/ ${NANO_WORLDDIR}/etc/ttys # Disable getty on syscons devices sed -i "" -e '/^ttyv[0-8]/s/ on/ off/' ${NANO_WORLDDIR}/etc/ttys # Tell loader to use serial console early. echo "${NANO_BOOT2CFG}" > ${NANO_WORLDDIR}/boot.config ) ####################################################################### # Allow root login via ssh cust_allow_ssh_root ( ) ( sed -i "" -e '/PermitRootLogin/s/.*/PermitRootLogin yes/' \ ${NANO_WORLDDIR}/etc/ssh/sshd_config ) ####################################################################### # Install the stuff under ./Files cust_install_files ( ) ( cd "${NANO_TOOLS}/Files" find . -print | grep -Ev '/(CVS|\.svn|\.hg|\.git)' | cpio -Ldumpv ${NANO_WORLDDIR} ) ####################################################################### # Install packages from ${NANO_PACKAGE_DIR} cust_pkgng ( ) ( # If the package directory doesn't exist, we're done. if [ ! -d ${NANO_PACKAGE_DIR} ]; then echo "DONE 0 packages" return 0 fi # Find a pkg-* package for x in `find -s ${NANO_PACKAGE_DIR} -iname 'pkg-*'`; do _NANO_PKG_PACKAGE=`basename "$x"` done if [ -z "${_NANO_PKG_PACKAGE}" -o ! -f "${NANO_PACKAGE_DIR}/${_NANO_PKG_PACKAGE}" ]; then echo "FAILED: need a pkg/ package for bootstrapping" exit 2 fi # Copy packages into chroot mkdir -p ${NANO_WORLDDIR}/Pkg ( cd "${NANO_PACKAGE_DIR}" find "${NANO_PACKAGE_LIST}" -print | cpio -Ldumpv ${NANO_WORLDDIR}/Pkg ) #Bootstrap pkg CR env ASSUME_ALWAYS_YES=YES SIGNATURE_TYPE=none /usr/sbin/pkg add /Pkg/${_NANO_PKG_PACKAGE} CR pkg -N >/dev/null 2>&1 if [ "$?" -ne "0" ]; then echo "FAILED: pkg bootstrapping faied" exit 2 fi rm -f ${NANO_WORLDDIR}/Pkg/pkg-* # Count & report how many we have to install todo=`ls ${NANO_WORLDDIR}/Pkg | /usr/bin/wc -l` todo=$(expr $todo + 1) # add one for pkg since it is installed already echo "=== TODO: $todo" ls ${NANO_WORLDDIR}/Pkg echo "===" while true do # Record how many we have now have=$(CR env ASSUME_ALWAYS_YES=YES /usr/sbin/pkg info | /usr/bin/wc -l) # Attempt to install more packages CR0 'ls 'Pkg/*txz' | xargs env ASSUME_ALWAYS_YES=YES /usr/sbin/pkg add' # See what that got us now=$(CR env ASSUME_ALWAYS_YES=YES /usr/sbin/pkg info | /usr/bin/wc -l) echo "=== NOW $now" CR env ASSUME_ALWAYS_YES=YES /usr/sbin/pkg info echo "===" if [ $now -eq $todo ] ; then echo "DONE $now packages" break elif [ $now -eq $have ] ; then echo "FAILED: Nothing happened on this pass" exit 2 fi done rm -rf ${NANO_WORLDDIR}/Pkg ) ####################################################################### # Convenience function: # Register all args as customize function. customize_cmd ( ) { NANO_CUSTOMIZE="$NANO_CUSTOMIZE $*" } ####################################################################### # Convenience function: # Register all args as late customize function to run just before # image creation. late_customize_cmd ( ) { NANO_LATE_CUSTOMIZE="$NANO_LATE_CUSTOMIZE $*" } ####################################################################### # # All set up to go... # ####################################################################### # Progress Print # Print $2 at level $1. pprint ( ) ( if [ "$1" -le $PPLEVEL ]; then runtime=$(( `date +%s` - $NANO_STARTTIME )) printf "%s %.${1}s %s\n" "`date -u -r $runtime +%H:%M:%S`" "#####" "$2" 1>&3 fi ) usage ( ) { ( echo "Usage: $0 [-bfiKknqvw] [-c config_file]" echo " -b suppress builds (both kernel and world)" echo " -c specify config file" echo " -f suppress code slice extraction" echo " -i suppress disk image build" echo " -K suppress installkernel" echo " -k suppress buildkernel" echo " -n add -DNO_CLEAN to buildworld, buildkernel, etc" echo " -q make output more quiet" echo " -v make output more verbose" echo " -w suppress buildworld" ) 1>&2 exit 2 } ####################################################################### # Setup and Export Internal variables # export_var ( ) { # Don't wawnt a subshell var=$1 # Lookup value of the variable. eval val=\$$var pprint 3 "Setting variable: $var=\"$val\"" export $1 } # Call this function to set defaults _after_ parsing options. # dont want a subshell otherwise variable setting is thrown away. set_defaults_and_export ( ) { : ${NANO_OBJ:=/usr/obj/nanobsd.${NANO_NAME}} : ${MAKEOBJDIRPREFIX:=${NANO_OBJ}} : ${NANO_DISKIMGDIR:=${NANO_OBJ}} NANO_WORLDDIR=${NANO_OBJ}/_.w NANO_MAKE_CONF_BUILD=${MAKEOBJDIRPREFIX}/make.conf.build NANO_MAKE_CONF_INSTALL=${NANO_OBJ}/make.conf.install # Override user's NANO_DRIVE if they specified a NANO_LABEL [ -n "${NANO_LABEL}" ] && NANO_DRIVE="ufs/${NANO_LABEL}" || true # Set a default NANO_TOOLS to NANO_SRC/NANO_TOOLS if it exists. [ ! -d "${NANO_TOOLS}" ] && [ -d "${NANO_SRC}/${NANO_TOOLS}" ] && \ NANO_TOOLS="${NANO_SRC}/${NANO_TOOLS}" || true [ -n "${NANO_NOPRIV_BUILD}" ] && [ -z "${NANO_METALOG}" ] && \ NANO_METALOG=${NANO_OBJ}/_.metalog || true NANO_STARTTIME=`date +%s` pprint 3 "Exporting NanoBSD variables" export_var MAKEOBJDIRPREFIX export_var NANO_ARCH export_var NANO_CODESIZE export_var NANO_CONFSIZE export_var NANO_CUSTOMIZE export_var NANO_DATASIZE export_var NANO_DRIVE export_var NANO_HEADS export_var NANO_IMAGES export_var NANO_IMGNAME export_var NANO_MAKE export_var NANO_MAKE_CONF_BUILD export_var NANO_MAKE_CONF_INSTALL export_var NANO_MEDIASIZE export_var NANO_NAME export_var NANO_NEWFS export_var NANO_OBJ export_var NANO_PMAKE export_var NANO_SECTS export_var NANO_SRC export_var NANO_TOOLS export_var NANO_WORLDDIR export_var NANO_BOOT0CFG export_var NANO_BOOTLOADER export_var NANO_LABEL export_var NANO_MODULES export_var NANO_NOPRIV_BUILD export_var NANO_METALOG export_var SRCCONF export_var SRC_ENV_CONF } Index: user/ngie/make_check/tools/tools/nanobsd/embedded/README =================================================================== --- user/ngie/make_check/tools/tools/nanobsd/embedded/README (revision 295077) +++ user/ngie/make_check/tools/tools/nanobsd/embedded/README (revision 295078) @@ -1,51 +1,49 @@ $FreeBSD$ Example for creating many different builds (including different arch) from a common set of files, as well as building natively using qemu user space emulation. This creates a simple appliance that uses dnsmasq to serve DNS and DHCPd. This is a work in progress. Generally, to build this you should cd tools/tools/nanobsd/embedded sudo sh ../nanobsd.sh -c foo.cfg -but do be careful if things are interrupted. There may still be -bugs lurking that cause your entire FreeBSD tree to disappear. Some features: Image size is minimal, we grow the last partition on first boot to fill the media. Images are both as easy as possible to construct, as well as easy as possible to expand. Config Short description beaglebone.cfg Create a bootable beaglebone image qemu-amd64.cfg Create a bootable amd64 image for qemu (W) qemu-i386.cfg Create a bootable i386 image for qemu (W) qemu-mips.cfg Create a bootable mips malta board image for qemu qemu-mips64.cfg Create a bootable mips malta board (64-bit mode) image for qemu qemu-powerpc.cfg Create a bootable 32-bit powerpc image for qemu qemu-powerpc64.cfg Create a bootable 64-bit IBM-flavor image for qemu qemu-sparc64.cfg Create a bootable sparc64 image for qemu rpi.cfg Create a bootable image for Raspberry Pi B rpi2.cfg Create a bootable image for Raspberry Pi2 sam9260ek.cfg Create a bootable image for an Atmel SAM9260-EK evaluation board (still needs a kenrel loaded into dataflash or NAND, so experimental). sam9g20ek.cfg Create a bootable image for an Atmel SAM9G20-EK evaluation board (still needs a kenrel loaded into dataflash or NAND, so experimental). Also works on many after-market boards that are somewhat compatible with the refernce board. QEMU command lines for serial console access i386: qemu-system-i386 -m 512 -hda _.disk.image.qemu-i386.qcow2 -nographic amd64: qemu-system-amd64 -m 512 -hda _.disk.image.qemu-amd64.qcow2 -nographic Index: user/ngie/make_check/tools/tools/nanobsd/embedded/common =================================================================== --- user/ngie/make_check/tools/tools/nanobsd/embedded/common (revision 295077) +++ user/ngie/make_check/tools/tools/nanobsd/embedded/common (revision 295078) @@ -1,674 +1,723 @@ # $FreeBSD$ #- # Copyright (c) 2015 Warner Losh. All Rights Reserved. # Copyright (c) 2010-2011 iXsystems, Inc., All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL iXsystems, Inc OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # # This file is heavily derived from both Sam Leffler's Avilia config, # as well as the BSDRP project's config file. Neither of these have # an explicit copyright/license statement, but are implicitly BSDL. This # example has been taken from the FreeNAS project (an early version) and # simplified to meet the needs of the example. # # NB: You want the other file as the command line arg :) # Missing in base: # o mkimg setting active partition # o mkimg starting at arbitrary offset (needed for A10, et al) # o mtools still needed becuase we have no makefs -t msdos # o nanobsd doesn't record changes to WORLDTEMP in customization # scripts yet, so we have kludge to pick up all files # o easy way for pkg to grab files from other repos and put that # data on the image # o Need to have some way to create / resize the s4 slice to do ping # pong bouncing between s3 and s4 for an arbitrary image. we can resize # one slice, not two # o hints in the uboot ports for how to create the image # # Missing here # o documentation for how to run the qemu images # o msdos mtools fallback # o special boot for !x86 !arm platforms # o qemu image for arm # o qemu image for aarch64 # o qemu image for armv6/armv7 # o easy support for different image / vm formats # o need to promote much of this to nanobsd.sh and friends # o uses old kludge to build image w/o ownership being right # for many files. Should move to mtree-dedup.awk. # o support for EFI images # o support for GPT partitioning # # Long Term # o common tooling for creating images for odd-ball platforms # o support for boot loaders other than uboot in the image # or via special insturctions # o No pony support. Sadly, you cannot have a pony. # if [ -z $NANO_NAME ]; then echo "NANO_NAME not defined. Use foo.cfg instead." fi NANO_SLICE_FAT_SIZE=32m NANO_SLICE_CFG_SIZE=32m +NANO_BOOT2CFG="-D -h -S115200 comconsole_port=0x3e8" + NANO_RAM_ETCSIZE=8192 NANO_RAM_TMPVARSIZE=8192 NANO_IMAGES=2 NANO_PMAKE="make -j $(sysctl -n hw.ncpu)" NANO_CFG_BASE=$(pwd) NANO_CFG_BASE=$(realpath ${NANO_CFG_BASE}/..) NANO_SRC=$(realpath ${NANO_CFG_BASE}/../../..) #### XXX share obj NANO_OBJ=${NANO_SRC}/../$NANO_NAME/obj # Where cust_pkg() finds packages to install #XXX: Is this the right place? #NANO_PORTS=$(realpath ${NANO_SRC}/../ports) NANO_PORTS=/usr/ports NANO_PACKAGE_DIR=${NANO_SRC}/${NANO_TOOLS}/Pkg NANO_INIT_IMG2=0 NANO_NOPRIV_BUILD=t unset MAKEOBJDIRPREFIX # this to go into nanobsd.sh NANO_PORTS=${NANO_PORTS:-/usr/ports} mkdir -p ${NANO_OBJ} NANO_OBJ=$(realpath ${NANO_OBJ}) NANO_FAT_DIR=${NANO_OBJ}/_.fat customize_cmd cust_allow_ssh_root add_etc_make_conf ( ) ( touch ${NANO_WORLDDIR}/etc/make.conf ) customize_cmd add_etc_make_conf cust_install_machine_files ( ) ( echo "cd ${NANO_CFG_BASE}/Files" cd ${NANO_CFG_BASE}/Files find . -print | grep -Ev '/(CVS|\.svn|\.hg|\.git)' | cpio -dumpv ${NANO_WORLDDIR} ) customize_cmd cust_install_files customize_cmd cust_install_machine_files # NB: leave c++ enabled so devd can be built CONF_BUILD=" WITHOUT_ACPI=true WITHOUT_ATM=true WITHOUT_AUDIT=true WITHOUT_BIND_DNSSEC=true WITHOUT_BIND_ETC=true WITHOUT_BIND_LIBS_LWRES=true WITHOUT_BLUETOOTH=true WITHOUT_CALENDAR=true WITHOUT_CVS=true WITHOUT_DICT=true WITHOUT_EXAMPLES=true WITHOUT_FORTRAN=true WITHOUT_GAMES=true WITHOUT_GCOV=true WITHOUT_GPIB=true WITHOUT_HTML=true WITHOUT_I4B=true WITHOUT_IPFILTER=true WITHOUT_IPX=true WITHOUT_LIBKSE=true WITHOUT_LOCALES=true WITHOUT_LPR=true WITHOUT_MAN=true WITHOUT_NETCAT=true WITHOUT_NIS=true WITHOUT_NLS=true WITHOUT_NS_CACHING=true WITHOUT_OBJC=true WITHOUT_PROFILE=true WITHOUT_RCMDS=true WITHOUT_SENDMAIL=true WITHOUT_SHAREDOCS=true WITHOUT_SYSCONS=true WITHOUT_LIB32=true WITHOUT_TESTS=true " CONF_INSTALL="$CONF_BUILD INSTALL_NODEBUG=t NOPORTDOCS=t NO_INSTALL_MANPAGES=t " # The following would help... # WITHOUT_TOOLCHAIN=true can't build ports # WITHOUT_INSTALLLIB=true libgcc.a # # from the build # WITHOUT_INFO=true makeinfo # WITHOUT_RCS=true PKG_ONLY_MAKE_CONF=" WITHOUT_TOOLCHAIN=true WITHOUT_INSTALLLIB=true WITHOUT_INFO=true WITHOUT_RCS=true " NANO_PACKAGE_ONLY=1 # install a package from a pre-built binary do_add_pkg ( ) ( # Need to create ${NANO_OBJ}/ports in this add_pkg_${port} function set -x mkdir -p ${NANO_OBJ}/ports/distfiles mkdir -p ${NANO_OBJ}/ports/packages mkdir -p ${NANO_WORLDDIR}/usr/ports/packages mkdir -p ${NANO_WORLDDIR}/usr/ports/distfiles mount -t nullfs -o noatime ${NANO_OBJ}/ports/packages \ ${NANO_WORLDDIR}/usr/ports/packages mount -t nullfs -o noatime ${NANO_OBJ}/ports/distfiles \ ${NANO_WORLDDIR}/usr/ports/distfiles CR env ASSUME_ALWAYS_YES=YES SIGNATURE_TYPE=none /usr/sbin/pkg add /usr/ports/packages/All/$1.txz umount ${NANO_WORLDDIR}/usr/ports/distfiles umount ${NANO_WORLDDIR}/usr/ports/packages rmdir ${NANO_WORLDDIR}/usr/ports/packages rmdir ${NANO_WORLDDIR}/usr/ports/distfiles rmdir ${NANO_WORLDDIR}/usr/ports set +x ) # Build a port (with the side effect of creating a package) do_add_port ( ) ( local port_path port_path=$1 shift set -x # Need to create ${NANO_OBJ}/ports in this add_port_${port} function mkdir -p ${NANO_OBJ}/ports/distfiles mkdir -p ${NANO_OBJ}/ports/packages mkdir -p ${NANO_PORTS}/packages mkdir -p ${NANO_PORTS}/distfiles mkdir -p ${NANO_WORLDDIR}/usr/src mkdir -p ${NANO_WORLDDIR}/usr/ports mount -t nullfs -o noatime ${NANO_SRC} ${NANO_WORLDDIR}/usr/src mount -t nullfs -o noatime ${NANO_PORTS} ${NANO_WORLDDIR}/usr/ports mount -t nullfs -o noatime ${NANO_OBJ}/ports/packages \ ${NANO_WORLDDIR}/usr/ports/packages mount -t nullfs -o noatime ${NANO_OBJ}/ports/distfiles \ ${NANO_WORLDDIR}/usr/ports/distfiles mkdir -p ${NANO_WORLDDIR}/dev mount -t devfs devfs ${NANO_WORLDDIR}/dev mkdir -p ${NANO_WORLDDIR}/usr/workdir cp /etc/resolv.conf ${NANO_WORLDDIR}/etc/resolv.conf # OK, a little inefficient, but likely not enough to worry about. CR ldconfig /lib /usr/lib /usr/local/lib CR ldconfig -R CR ldconfig -r # Improvement: Don't know why package-recursive don't works here CR "env UNAME_p=${NANO_ARCH} TARGET=${NANO_ARCH} \ TARGET_ARCH=${NANO_ARCH} PORTSDIR=${NANO_PORTS} make \ __MAKE_CONF=${NANO_MAKE_CONF_BUILD} \ WRKDIRPREFIX=/usr/workdir -C /usr/ports/$port_path \ package-recursive BATCH=yes $* clean FORCE_PKG_REGISTER=t" rm ${NANO_WORLDDIR}/etc/resolv.conf rm -rf ${NANO_WORLDDIR}/usr/obj rm -rf ${NANO_WORLDDIR}/usr/workdir umount ${NANO_WORLDDIR}/dev umount ${NANO_WORLDDIR}/usr/ports/packages umount ${NANO_WORLDDIR}/usr/ports/distfiles umount ${NANO_WORLDDIR}/usr/ports umount ${NANO_WORLDDIR}/usr/src set +x ) # Need to check if this function works with cross-compiling architecture!!!! # Recursive complex fonction: Generate one function for each ports # writes shell functions called later, so don't do in subshell. add_port ( ) { local port_path=$1 local port=`echo $1 | sed -e 's/\//_/'` shift # Check if package allready exist # Need to: # 1. check ARCH of this package! # 2. Add a trap cd ${NANO_PORTS}/${port_path} PKG_NAME=`env PORTSDIR=${NANO_PORTS} make __MAKE_CONF=${NANO_MAKE_CONF_BUILD} package-name` if [ -f ${NANO_OBJ}/ports/packages/All/${PKG_NAME}.txz ]; then # Pkg file found: Generate add_pkg_NAME function eval " add_pkg_${port} ( ) { do_add_pkg ${PKG_NAME} } customize_cmd add_pkg_${port} " else # No pkg file: Generate add_port_NAME function eval " add_port_${port} ( ) { do_add_port ${port_path} $* } customize_cmd add_port_${port} " NANO_PACKAGE_ONLY=0 fi } # Creates images for all the formats that use MBR create_diskimage_mbr ( ) ( pprint 2 "build diskimage ${NANO_NAME}" pprint 3 "log: ${NANO_OBJ}/_.di" ( local extra i sz fmt fmtarg bootmbr bootbsd skiparg set -o xtrace [ -z ${NANO_DISKIMAGE_FORMAT} ] || fmtarg="-f ${NANO_DISKIMAGE_FORMAT}" [ -z ${NANO_DISKIMAGE_FORMAT} ] || fmt=".${NANO_DISKIMAGE_FORMAT}" bootmbr=${NANO_BOOT_MBR:+-b ${NANO_BOOT_MBR}} bootbsd=${NANO_BOOT_BSD:+-b ${NANO_BOOT_BSD}} skiparg=${NANO_MBR_FIRST_SKIP:+-S ${NANO_MBR_FIRST_SKIP}} - for i in s1 s2 s3 s4 empty; do + for i in s1 s2 s3 s4 p1 p2 p3 p4 p5 empty; do rm -fr ${NANO_OBJ}/_.${i}* done # Populate the FAT partition, if needed if [ -n "${NANO_SLICE_FAT}" ]; then echo Creating MSDOS partition for kernel newfs_msdos -C ${NANO_SLICE_FAT_SIZE} -F 16 -L ${NANO_NAME} \ ${NANO_OBJ}/_.${NANO_SLICE_FAT} if [ -d ${NANO_FAT_DIR} ]; then # Need to copy files from ${NANO_FATDIR} with mtools, or use # makefs -t msdos once that's supported mcopy -i ${NANO_OBJ}/_.${NANO_SLICE_FAT} ${NANO_FAT_DIR}/* :: fi fi # Populate the Powerpc boot image, if needed if [ "${NANO_LAYOUT}" = powerpc64-ibm ]; then dd if=${NANO_WORLDDIR}/boot/boot1.elf of=${NANO_OBJ}/_.s1 bs=800k count=1 conv=sync fi # Populate the / partition, and place it into a slice with a # bsd label [ -z ${NANO_NOPRIV_BUILD} ] || extra="-F ${NANO_METALOG}" sz=${NANO_SLICE_ROOT_SIZE:+-s ${NANO_SLICE_ROOT_SIZE}} - eval "${NANO_MAKEFS_UFS}" ${extra} $sz "${NANO_OBJ}/_.${NANO_SLICE_ROOT}a" \ + eval "${NANO_MAKEFS_UFS}" ${extra} $sz "${NANO_OBJ}/_.${NANO_SLICE_ROOT}" \ "${NANO_WORLDDIR}" - mkimg -s bsd ${bootbsd} -p freebsd-ufs:=${NANO_OBJ}/_.${NANO_SLICE_ROOT}a \ - -o ${NANO_OBJ}/_.${NANO_SLICE_ROOT} +# mkimg -s bsd ${bootbsd} -p freebsd-ufs:=${NANO_OBJ}/_.${NANO_SLICE_ROOT} \ +# -o ${NANO_OBJ}/_.${NANO_SLICE_ROOT} # Populate the /cfg partition, empty if none given if [ -z "${NANO_CFGDIR}" ]; then echo "Faking cfg dir, it's empty" NANO_CFGDIR=${NANO_OBJ}/_.empty mkdir -p ${NANO_CFGDIR} fi # XXX -F cfg-mtree eval "${NANO_MAKEFS_UFS}" -s ${NANO_SLICE_CFG_SIZE} \ "${NANO_OBJ}/_.${NANO_SLICE_CFG}" "${NANO_CFGDIR}" # data slice not supported since we need the part for FAT for # booting # Now shuffle all the slices together into the proper layout if [ -n "$NANO_SLICE_FAT" ]; then eval $NANO_SLICE_FAT=fat16b fi - eval $NANO_SLICE_CFG=freebsd - eval $NANO_SLICE_ROOT=freebsd + case ${NANO_SLICE_CFG} in + s*) + echo slice + eval $NANO_SLICE_CFG=freebsd + eval $NANO_SLICE_ROOT=freebsd + ;; + p*) + echo part + eval $NANO_SLICE_CFG=freebsd-ufs + eval $NANO_SLICE_ROOT=freebsd-ufs + ;; + esac # below depends on https://reviews.freebsd.org/D4403 not yet in the tree # but there's problems: it marks all partitions as active, so you have to # boot off parittion 3 or 2 by hand if you're playing around with this WIP case ${NANO_LAYOUT} in std-embedded) mkimg -a 3 ${skiparg} ${fmtarg} ${bootmbr} -s mbr -p ${s1}:=${NANO_OBJ}/_.s1 \ -p ${s2}:=${NANO_OBJ}/_.s2 \ -p ${s3}:=${NANO_OBJ}/_.s3 \ -o ${NANO_OBJ}/_.disk.image.${NANO_NAME}${fmt} ;; std-x86) # s1 is cfg, s2 is /, not sure how to make that # boot (marked as active) with mkimg yet mkimg -a 2 ${fmtarg} ${bootmbr} -s mbr -p ${s1}:=${NANO_OBJ}/_.s1 \ -p ${s2}:=${NANO_OBJ}/_.s2 \ -o ${NANO_OBJ}/_.disk.image.${NANO_NAME}${fmt} ;; + std-uefi) + # s1 is boot, s2 is cfg, s3 is /, not sure how to make that + # boot (marked as active) with mkimg yet + mkimg -a 2 ${fmtarg} ${bootmbr} -s mbr \ + -p efi:=${NANO_WORLDDIR}/boot/boot1.efifat \ + -p ${s2}:=${NANO_OBJ}/_.s2 \ + -p ${s3}:=${NANO_OBJ}/_.s3 \ + -o ${NANO_OBJ}/_.disk.image.${NANO_NAME}${fmt} + ;; + std-uefi-bios) + # p1 is boot for uefi, p2 is boot for gpt, p3 is cfg, p4 is / + # and p5 is alt-root (after resize) + mkimg -a 2 ${fmtarg} ${bootmbr} -s gpt \ + -p efi:=${NANO_WORLDDIR}/boot/boot1.efifat \ + -p freebsd-boot:=${NAANO_WORLDDIR}/boot/gptboot \ + -p ${p3}:=${NANO_OBJ}/_.p3 \ + -p ${p4}:=${NANO_OBJ}/_.p4 \ + -o ${NANO_OBJ}/_.disk.image.${NANO_NAME}${fmt} + ;; powerpc64-ibm) # A lie to make the boot loader work, it boots the first BSD partition # it finds, regardless of the active flag. s2=fat16b # boot image is on a special partition, ala std-embedded, but that # partition isn't FAT with special files, but a copy of the boot # loader itself. mkimg -a 1 ${fmtarg} -s mbr -p prepboot:=${NANO_OBJ}/_.s1 \ -p ${s2}:=${NANO_OBJ}/_.s2 \ -p ${s3}:=${NANO_OBJ}/_.s3a \ -o ${NANO_OBJ}/_.disk.image.${NANO_NAME}${fmt} ;; esac ) > ${NANO_OBJ}/_.di 2>&1 ) die( ) { echo "$*" exit 1 } # Automatically include the packaging port here so it is always first so it # builds the port and adds the package so we can add other packages. #XXX Doesn't work for cross build, so punting until I can integreate qemu-static #XXX or poudere, both of which require priv'd execution. Or qemu-system, which #XXX is super, super slow. #add_port ports-mgmt/pkg #add_port security/sudo #add_port ftp/curl rp=$(realpath ${NANO_OBJ}/) __a=`mount | grep ${rp} | awk '{print length($3), $3;}' | sort -rn | awk '{$1=""; print;}'` if [ -n "$__a" ]; then echo "unmounting $__a" umount $__a fi if [ "$DEBUG" = 1 ]; then DEBUG_BUILD=" DEBUG_FLAGS= -g " else DEBUG_INSTALL=" INSTALL_NODEBUG= t " fi CONF_INSTALL="$CONF_BUILD ${DEBUG_BUILD} " CONF_INSTALL="$CONF_INSTALL ${DEBUG_INSTALL} " if [ "${NANO_PACKAGE_ONLY}" -eq 1 ]; then CONF_INSTALL="${CONF_INSTALL} ${PKG_ONLY_MAKE_CONF} " echo "Automatically building a thin image with packages" else echo "Automatically building a * * F A T * * image so we can build ports" fi VARS="MASTER_SITE_BACKUP MASTER_SITE_OVERRIDE PACKAGEROOT PACKAGESITE" for var in $VARS; do val=$(eval echo "\$$var") if [ -n "$val" ]; then CONF_INSTALL="${CONF_INSTALL} $var=$val" fi done typical_embedded ( ) ( # Need to create rc.conf before we copy over /etc to /conf/base/etc # so now's a good time. local rc=${NANO_WORLDDIR}/etc/rc.conf echo "hostname=nanobsd-${NANO_NAME}" > $rc echo "growfs_enable=YES" >> $rc echo "growfs_type=nanobsd-pingpong" >> $rc echo "ntpdate_enable=YES" >> $rc echo "ifconfig_DEFAULT=DHCP" >> $rc echo "ntpdate_hosts='0.freebsd.pool.ntp.org 1.freebsd.pool.ntp.org'" >> $rc # Make sure that firstboot scripts run so growfs works. # Note: still some issues remvoing this XXX touch ${NANO_WORLDDIR}/firstboot ) customize_cmd typical_embedded fix_pkg ( ) ( chdir ${NANO_WORLDDIR} mkdir -p pkg mkdir -p pkg/db mkdir -p pkg/cache mkdir -p pkg/tmp # Needed for pkg bootstrap mkdir -p usr/local/etc # Will get moved to local/etc ( echo 'PKG_DBDIR = "/pkg/db"' echo 'PKG_CACHEDIR = "/pkg/cache"' echo 'DEFAULT_ALWAYS_YES = "yes"' echo 'ASSUME_ALWAYS_YES = "yes"' ) >> usr/local/etc/pkg.conf [ -z ${NANO_NOPRIV_BUILD} ] || ( echo "./pkg type=dir uname=root gname=wheel mode=0755" echo "./pkg/cache type=dir uname=root gname=wheel mode=0755" echo "./pkg/db type=dir uname=root gname=wheel mode=0755" echo "./pkg/tmp type=dir uname=root gname=wheel mode=0755" ) >> ${NANO_METALOG} ) customize_cmd fix_pkg save_build ( ) ( VERSION_FILE=${NANO_WORLDDIR}/etc/version if [ "${SVNREVISION}" = "${REVISION}" ]; then echo "${NANO_NAME}" > "${VERSION_FILE}" else echo "${NANO_NAME} (${SVNREVISION})" > "${VERSION_FILE}" fi ) customize_cmd save_build shrink_md_fbsize ( ) ( # We have a lot of little files on our memory disks. Let's decrease # the block and frag size to fit more little files on them (this # halves our space requirement by ~50% on /etc and /var on 8.x -- # and gives us more back on 9.x as the default block and frag size # are 4 times larger). sed -i '' -e 's,-S -i 4096,-S -i 4096 -b 4096 -f 512,' \ ${NANO_WORLDDIR}/etc/rc.initdiskless ) customize_cmd shrink_md_fbsize customize_cmd cust_comconsole dos_boot_part ( ) ( local d=/usr/local/share/u-boot/${NANO_BOOT_PKG} local f=${NANO_FAT_DIR} # For now, just copy all the files. However, for iMX6 and Allwinner, # we'll need to put a special boot block at a fixed location # on the disk as well. rm -rf $f mkdir $f chdir $f cp ${d}/* . # Also copy ubldr. u-boot will load it and it will load the kernel # from the ufs partition cp ${NANO_WORLDDIR}/boot/ubldr . cp ${NANO_WORLDDIR}/boot/ubldr.bin . # We have to touch the saveenv file touch uEnv.txt # Now we need to copy over dtb files from the build. cp ${NANO_WORLDDIR}/boot/dtb/*.dtb . ) if [ -n "$NANO_BOOT_PKG" ]; then if [ ! -d ${d} ]; then echo ${NANO_BOOT_PKG} not installed. Sadly, it must be. exit 1 fi if [ ! -x /usr/local/bin/mcopy ]; then echo mtools not installed. Sadly, we gotta have it. exit 1 fi customize_cmd dos_boot_part fi product_custom ( ) ( # not quie ready to tweak these in nopriv build if [ -z ${NANO_NOPRIV_BUILD} ]; then # Last second tweaks -- generally not needed chown -R root:wheel ${NANO_WORLDDIR}/root chmod 0755 ${NANO_WORLDDIR}/root/* chmod 0755 ${NANO_WORLDDIR}/* chown -R root:wheel ${NANO_WORLDDIR}/etc chown -R root:wheel ${NANO_WORLDDIR}/boot chown root:wheel ${NANO_WORLDDIR}/ chown root:wheel ${NANO_WORLDDIR}/usr fi ) late_customize_cmd product_custom # # Convenience routines to bring up many settings for standard environments # # # For each architecture, we have a standard set of settings to the extent # it makes sense. For architectures that don't have a standard environment # cfg files need to either define a lot of settings, provide their own disk # imaging, or set which of the variants we support. No subshells, since we # set global variables # std_aarch64 ( ) { } std_amd64 ( ) { std_i386 } std_arm ( ) { } std_armeb ( ) { NANO_ENDIAN=big } std_armv6 ( ) { } std_i386 ( ) { # Default values, if not overridden in .cfg file : ${NANO_KERNEL:=GENERIC} : ${NANO_DRIVE:=ada0} : ${NANO_LAYOUT:=std-x86} : ${NANO_BOOT_MBR:=${NANO_WORLDDIR}/boot/boot0sio} : ${NANO_BOOT_BSD:=${NANO_WORLDDIR}/boot/boot} } std_mips ( ) { NANO_ENDIAN=big } std_mipsel ( ) { } std_mips64 ( ) { NANO_ENDIAN=big } std_mips64el ( ) { } std_powerpc ( ) { NANO_ENDIAN=big } std_powerpc64 ( ) { NANO_LAYOUT=powerpc64-ibm NANO_ENDIAN=big } std_sparc64 ( ) { NANO_ENDIAN=big } # # QEMU settings for the standard environments # qemu_env ( ) { NANO_DISKIMAGE_FORMAT=qcow2 } # other standard environments will go here eval std_${NANO_ARCH} # Slice 1: FAT for ROM loading bootstrap # Slice 2: Config partition # Slice 3: FreeBSD partition (sized exactly) # Slice 4: FreeBSD partition (empty) # on first boot, we resize slice 3 & 4 to be 1/2 of what's # left over on the SD card after slice 1 and 2 are taken # off the top. We also resize the 'a' partion on first boot # to the size of the partition for the ping/pong upgrade. # This feature needs support in the rc.d bootup script. : ${NANO_ENDIAN:=little} # make -V something to figure it out? : ${NANO_LAYOUT:=std-embedded} : ${NANO_MAKEFS_UFS:=makefs -t ffs -B ${NANO_ENDIAN}} : ${NANO_DISK_SCHEME:=mbr} # No others really supported ATM case ${NANO_LAYOUT} in std-embedded) NANO_SLICE_FAT=s1 NANO_SLICE_CFG=s2 NANO_SLICE_ROOT=s3 NANO_SLICE_ALTROOT=s4 ;; std-x86) NANO_SLICE_CFG=s1 NANO_SLICE_ROOT=s2 NANO_SLICE_ALTROOT=s3 ;; powerpc64-ibm) NANO_SLICE_PPCBOOT=s1 NANO_SLICE_CFG=s2 NANO_SLICE_ROOT=s3 NANO_SLICE_ALTROOT=s4 ;; powerpc64-apple) echo Not yet exit 1 ;; +std-uefi) + NANO_SLICE_UEFI=s1 + NANO_SLICE_CFG=s2 + NANO_SLICE_ROOT=s3 + NANO_SLICE_ALTROOT=s4 + ;; +std-uefi-bios) + NANO_SLICE_UEFI=p1 + NANO_SLICE_BOOT=p2 + NANO_SLICE_CFG=p3 + NANO_SLICE_ROOT=p4 + NANO_SLICE_ALTROOT=p5 + ;; *) echo Unknown Layout ${NANO_LAYOUT} exit 1 ;; esac + +# For this config, no BSD labels so NANO_ROOT and NANO_ALTROOT need to be +# adjusted +NANO_ROOT=${NANO_SLICE_ROOT} +NANO_ALTROOT=${NANO_SLICE_ALTROOT} NANO_SLICE_DATA= # Not included # Each major disk scheme has its own routine. Generally # this is for mbr, gpt, etc. These are generally are widely # shared, but some specialized formats won't be shared. create_diskimage ( ) ( eval create_diskimage_${NANO_DISK_SCHEME} ) Index: user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi-bios.cfg =================================================================== --- user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi-bios.cfg (nonexistent) +++ user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi-bios.cfg (revision 295078) @@ -0,0 +1,43 @@ +# $FreeBSD$ + +#- +# Copyright (c) 2015 Warner Losh. All Rights Reserved. +# Copyright (c) 2010-2011 iXsystems, Inc., All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL iXsystems, Inc. OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# + +NANO_ARCH=amd64 +NANO_NAME=qemu-amd64-uefi-bios +NANO_LAYOUT=std-uefi-bios + +. common # Pull in common definitions + +qemu_env + +# +# Run with +# qemu-system-x86_64 -serial stdio -bios OVMF.fd \ +# -hda _.disk.image.qemu-amd64-uefi.qcow2 +# OVMF.fd is from +# http://sourceforge.net/projects/edk2/files/OVMF/OVMF-X64-r15214.zip +# Property changes on: user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi-bios.cfg ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi.cfg =================================================================== --- user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi.cfg (nonexistent) +++ user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi.cfg (revision 295078) @@ -0,0 +1,43 @@ +# $FreeBSD$ + +#- +# Copyright (c) 2015 Warner Losh. All Rights Reserved. +# Copyright (c) 2010-2011 iXsystems, Inc., All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL iXsystems, Inc. OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# + +NANO_ARCH=amd64 +NANO_NAME=qemu-amd64-uefi +NANO_LAYOUT=std-uefi + +. common # Pull in common definitions + +qemu_env + +# +# Run with +# qemu-system-x86_64 -serial stdio -bios OVMF.fd \ +# -hda _.disk.image.qemu-amd64-uefi.qcow2 +# OVMF.fd is from +# http://sourceforge.net/projects/edk2/files/OVMF/OVMF-X64-r15214.zip +# Property changes on: user/ngie/make_check/tools/tools/nanobsd/embedded/qemu-amd64-uefi.cfg ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/make_check/usr.bin/kdump/kdump.c =================================================================== --- user/ngie/make_check/usr.bin/kdump/kdump.c (revision 295077) +++ user/ngie/make_check/usr.bin/kdump/kdump.c (revision 295078) @@ -1,1899 +1,1899 @@ /*- * Copyright (c) 1988, 1993 * The Regents of the University of California. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #ifndef lint static const char copyright[] = "@(#) Copyright (c) 1988, 1993\n\ The Regents of the University of California. All rights reserved.\n"; #endif /* not lint */ #ifndef lint #if 0 static char sccsid[] = "@(#)kdump.c 8.1 (Berkeley) 6/6/93"; #endif #endif /* not lint */ #include __FBSDID("$FreeBSD$"); #define _KERNEL extern int errno; #include #undef _KERNEL #include #include #include #define _KERNEL #include #undef _KERNEL #include #include #include #include #include #include #include #include #include #include #ifdef HAVE_LIBCAPSICUM #include #endif #include #include #include #include #include #include #ifdef HAVE_LIBCAPSICUM #include #include #include #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include "ktrace.h" #include "kdump_subr.h" u_int abidump(struct ktr_header *); int fetchprocinfo(struct ktr_header *, u_int *); int fread_tail(void *, int, int); void dumpheader(struct ktr_header *); void ktrsyscall(struct ktr_syscall *, u_int); void ktrsysret(struct ktr_sysret *, u_int); void ktrnamei(char *, int); void hexdump(char *, int, int); void visdump(char *, int, int); void ktrgenio(struct ktr_genio *, int); void ktrpsig(struct ktr_psig *); void ktrcsw(struct ktr_csw *); void ktrcsw_old(struct ktr_csw_old *); void ktruser_malloc(void *); void ktruser_rtld(int, void *); void ktruser(int, void *); void ktrcaprights(cap_rights_t *); void ktrsockaddr(struct sockaddr *); void ktrstat(struct stat *); void ktrstruct(char *, size_t); void ktrcapfail(struct ktr_cap_fail *); void ktrfault(struct ktr_fault *); void ktrfaultend(struct ktr_faultend *); void limitfd(int fd); void usage(void); #define TIMESTAMP_NONE 0x0 #define TIMESTAMP_ABSOLUTE 0x1 #define TIMESTAMP_ELAPSED 0x2 #define TIMESTAMP_RELATIVE 0x4 extern const char *signames[]; static int timestamp, decimal, fancy = 1, suppressdata, tail, threads, maxdata, resolv = 0, abiflag = 0, syscallno = 0; static const char *tracefile = DEF_TRACEFILE; static struct ktr_header ktr_header; #define TIME_FORMAT "%b %e %T %Y" #define eqs(s1, s2) (strcmp((s1), (s2)) == 0) #define print_number(i,n,c) do { \ if (decimal) \ printf("%c%jd", c, (intmax_t)*i); \ else \ printf("%c%#jx", c, (uintmax_t)(u_register_t)*i); \ i++; \ n--; \ c = ','; \ } while (0) #if defined(__amd64__) || defined(__i386__) void linux_ktrsysret(struct ktr_sysret *, u_int); /* * from linux.h * Linux syscalls return negative errno's, we do positive and map them */ static int bsd_to_linux_errno[ELAST + 1] = { -0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, -35, -12, -13, -14, -15, -16, -17, -18, -19, -20, -21, -22, -23, -24, -25, -26, -27, -28, -29, -30, -31, -32, -33, -34, -11,-115,-114, -88, -89, -90, -91, -92, -93, -94, -95, -96, -97, -98, -99, -100,-101,-102,-103,-104,-105,-106,-107,-108,-109, -110,-111, -40, -36,-112,-113, -39, -11, -87,-122, -116, -66, -6, -6, -6, -6, -6, -37, -38, -9, -6, -6, -43, -42, -75,-125, -84, -95, -16, -74, -72, -67, -71 }; #endif struct proc_info { TAILQ_ENTRY(proc_info) info; u_int sv_flags; pid_t pid; }; static TAILQ_HEAD(trace_procs, proc_info) trace_procs; #ifdef HAVE_LIBCAPSICUM static cap_channel_t *cappwd, *capgrp; #endif static void strerror_init(void) { /* * Cache NLS data before entering capability mode. * XXXPJD: There should be strerror_init() and strsignal_init() in libc. */ (void)catopen("libc", NL_CAT_LOCALE); } static void localtime_init(void) { time_t ltime; /* * Allow localtime(3) to cache /etc/localtime content before entering * capability mode. * XXXPJD: There should be localtime_init() in libc. */ (void)time(<ime); (void)localtime(<ime); } #ifdef HAVE_LIBCAPSICUM static int cappwdgrp_setup(cap_channel_t **cappwdp, cap_channel_t **capgrpp) { cap_channel_t *capcas, *cappwdloc, *capgrploc; const char *cmds[1], *fields[1]; capcas = cap_init(); if (capcas == NULL) { warn("unable to contact casperd"); return (-1); } cappwdloc = cap_service_open(capcas, "system.pwd"); capgrploc = cap_service_open(capcas, "system.grp"); /* Casper capability no longer needed. */ cap_close(capcas); if (cappwdloc == NULL || capgrploc == NULL) { if (cappwdloc == NULL) warn("unable to open system.pwd service"); if (capgrploc == NULL) warn("unable to open system.grp service"); exit(1); } /* Limit system.pwd to only getpwuid() function and pw_name field. */ cmds[0] = "getpwuid"; if (cap_pwd_limit_cmds(cappwdloc, cmds, 1) < 0) err(1, "unable to limit system.pwd service"); fields[0] = "pw_name"; if (cap_pwd_limit_fields(cappwdloc, fields, 1) < 0) err(1, "unable to limit system.pwd service"); /* Limit system.grp to only getgrgid() function and gr_name field. */ cmds[0] = "getgrgid"; if (cap_grp_limit_cmds(capgrploc, cmds, 1) < 0) err(1, "unable to limit system.grp service"); fields[0] = "gr_name"; if (cap_grp_limit_fields(capgrploc, fields, 1) < 0) err(1, "unable to limit system.grp service"); *cappwdp = cappwdloc; *capgrpp = capgrploc; return (0); } #endif /* HAVE_LIBCAPSICUM */ int main(int argc, char *argv[]) { int ch, ktrlen, size; void *m; int trpoints = ALL_POINTS; int drop_logged; pid_t pid = 0; u_int sv_flags; setlocale(LC_CTYPE, ""); timestamp = TIMESTAMP_NONE; while ((ch = getopt(argc,argv,"f:dElm:np:AHRrSsTt:")) != -1) switch (ch) { case 'A': abiflag = 1; break; case 'f': tracefile = optarg; break; case 'd': decimal = 1; break; case 'l': tail = 1; break; case 'm': maxdata = atoi(optarg); break; case 'n': fancy = 0; break; case 'p': pid = atoi(optarg); break; case 'r': resolv = 1; break; case 'S': syscallno = 1; break; case 's': suppressdata = 1; break; case 'E': timestamp |= TIMESTAMP_ELAPSED; break; case 'H': threads = 1; break; case 'R': timestamp |= TIMESTAMP_RELATIVE; break; case 'T': timestamp |= TIMESTAMP_ABSOLUTE; break; case 't': trpoints = getpoints(optarg); if (trpoints < 0) errx(1, "unknown trace point in %s", optarg); break; default: usage(); } if (argc > optind) usage(); m = malloc(size = 1025); if (m == NULL) errx(1, "%s", strerror(ENOMEM)); if (!freopen(tracefile, "r", stdin)) err(1, "%s", tracefile); strerror_init(); localtime_init(); #ifdef HAVE_LIBCAPSICUM if (resolv != 0) { if (cappwdgrp_setup(&cappwd, &capgrp) < 0) { cappwd = NULL; capgrp = NULL; } } if (resolv == 0 || (cappwd != NULL && capgrp != NULL)) { if (cap_enter() < 0 && errno != ENOSYS) err(1, "unable to enter capability mode"); } #else if (resolv == 0) { if (cap_enter() < 0 && errno != ENOSYS) err(1, "unable to enter capability mode"); } #endif limitfd(STDIN_FILENO); limitfd(STDOUT_FILENO); limitfd(STDERR_FILENO); TAILQ_INIT(&trace_procs); drop_logged = 0; while (fread_tail(&ktr_header, sizeof(struct ktr_header), 1)) { if (ktr_header.ktr_type & KTR_DROP) { ktr_header.ktr_type &= ~KTR_DROP; if (!drop_logged && threads) { printf( "%6jd %6jd %-8.*s Events dropped.\n", (intmax_t)ktr_header.ktr_pid, ktr_header.ktr_tid > 0 ? (intmax_t)ktr_header.ktr_tid : 0, MAXCOMLEN, ktr_header.ktr_comm); drop_logged = 1; } else if (!drop_logged) { printf("%6jd %-8.*s Events dropped.\n", (intmax_t)ktr_header.ktr_pid, MAXCOMLEN, ktr_header.ktr_comm); drop_logged = 1; } } if (trpoints & (1< size) { m = realloc(m, ktrlen+1); if (m == NULL) errx(1, "%s", strerror(ENOMEM)); size = ktrlen; } if (ktrlen && fread_tail(m, ktrlen, 1) == 0) errx(1, "data too short"); if (fetchprocinfo(&ktr_header, (u_int *)m) != 0) continue; sv_flags = abidump(&ktr_header); if (pid && ktr_header.ktr_pid != pid && ktr_header.ktr_tid != pid) continue; if ((trpoints & (1<ktr_type) { case KTR_PROCCTOR: TAILQ_FOREACH(pi, &trace_procs, info) { if (pi->pid == kth->ktr_pid) { TAILQ_REMOVE(&trace_procs, pi, info); break; } } pi = malloc(sizeof(struct proc_info)); if (pi == NULL) errx(1, "%s", strerror(ENOMEM)); pi->sv_flags = *flags; pi->pid = kth->ktr_pid; TAILQ_INSERT_TAIL(&trace_procs, pi, info); return (1); case KTR_PROCDTOR: TAILQ_FOREACH(pi, &trace_procs, info) { if (pi->pid == kth->ktr_pid) { TAILQ_REMOVE(&trace_procs, pi, info); free(pi); break; } } return (1); } return (0); } u_int abidump(struct ktr_header *kth) { struct proc_info *pi; const char *abi; const char *arch; u_int flags = 0; TAILQ_FOREACH(pi, &trace_procs, info) { if (pi->pid == kth->ktr_pid) { flags = pi->sv_flags; break; } } if (abiflag == 0) return (flags); switch (flags & SV_ABI_MASK) { case SV_ABI_LINUX: abi = "L"; break; case SV_ABI_FREEBSD: abi = "F"; break; default: abi = "U"; break; } if (flags != 0) { if (flags & SV_LP64) arch = "64"; else arch = "32"; } else arch = "00"; printf("%s%s ", abi, arch); return (flags); } void dumpheader(struct ktr_header *kth) { static char unknown[64]; static struct timeval prevtime, prevtime_e, temp; const char *type; const char *sign; switch (kth->ktr_type) { case KTR_SYSCALL: type = "CALL"; break; case KTR_SYSRET: type = "RET "; break; case KTR_NAMEI: type = "NAMI"; break; case KTR_GENIO: type = "GIO "; break; case KTR_PSIG: type = "PSIG"; break; case KTR_CSW: type = "CSW "; break; case KTR_USER: type = "USER"; break; case KTR_STRUCT: type = "STRU"; break; case KTR_SYSCTL: type = "SCTL"; break; case KTR_PROCCTOR: /* FALLTHROUGH */ case KTR_PROCDTOR: return; case KTR_CAPFAIL: type = "CAP "; break; case KTR_FAULT: type = "PFLT"; break; case KTR_FAULTEND: type = "PRET"; break; default: sprintf(unknown, "UNKNOWN(%d)", kth->ktr_type); type = unknown; } /* * The ktr_tid field was previously the ktr_buffer field, which held * the kernel pointer value for the buffer associated with data * following the record header. It now holds a threadid, but only * for trace files after the change. Older trace files still contain * kernel pointers. Detect this and suppress the results by printing * negative tid's as 0. */ if (threads) printf("%6jd %6jd %-8.*s ", (intmax_t)kth->ktr_pid, kth->ktr_tid > 0 ? (intmax_t)kth->ktr_tid : 0, MAXCOMLEN, kth->ktr_comm); else printf("%6jd %-8.*s ", (intmax_t)kth->ktr_pid, MAXCOMLEN, kth->ktr_comm); if (timestamp) { if (timestamp & TIMESTAMP_ABSOLUTE) { printf("%jd.%06ld ", (intmax_t)kth->ktr_time.tv_sec, kth->ktr_time.tv_usec); } if (timestamp & TIMESTAMP_ELAPSED) { if (prevtime_e.tv_sec == 0) prevtime_e = kth->ktr_time; timevalsub(&kth->ktr_time, &prevtime_e); printf("%jd.%06ld ", (intmax_t)kth->ktr_time.tv_sec, kth->ktr_time.tv_usec); timevaladd(&kth->ktr_time, &prevtime_e); } if (timestamp & TIMESTAMP_RELATIVE) { if (prevtime.tv_sec == 0) prevtime = kth->ktr_time; temp = kth->ktr_time; timevalsub(&kth->ktr_time, &prevtime); if ((intmax_t)kth->ktr_time.tv_sec < 0) { kth->ktr_time = prevtime; prevtime = temp; timevalsub(&kth->ktr_time, &prevtime); sign = "-"; } else { prevtime = temp; sign = ""; } printf("%s%jd.%06ld ", sign, (intmax_t)kth->ktr_time.tv_sec, kth->ktr_time.tv_usec); } } printf("%s ", type); } #include static void ioctlname(unsigned long val) { const char *str; str = sysdecode_ioctlname(val); if (str != NULL) printf("%s", str); else if (decimal) printf("%lu", val); else printf("%#lx", val); } static enum sysdecode_abi syscallabi(u_int sv_flags) { if (sv_flags == 0) - return (FREEBSD); + return (SYSDECODE_ABI_FREEBSD); switch (sv_flags & SV_ABI_MASK) { case SV_ABI_FREEBSD: - return (FREEBSD); + return (SYSDECODE_ABI_FREEBSD); #if defined(__amd64__) || defined(__i386__) case SV_ABI_LINUX: #ifdef __amd64__ if (sv_flags & SV_ILP32) - return (LINUX32); + return (SYSDECODE_ABI_LINUX32); #endif - return (LINUX); + return (SYSDECODE_ABI_LINUX); #endif default: - return (UNKNOWN_ABI); + return (SYSDECODE_ABI_UNKNOWN); } } static void syscallname(u_int code, u_int sv_flags) { const char *name; name = sysdecode_syscallname(syscallabi(sv_flags), code); if (name == NULL) printf("[%d]", code); else { printf("%s", name); if (syscallno) printf("[%d]", code); } } void ktrsyscall(struct ktr_syscall *ktr, u_int sv_flags) { int narg = ktr->ktr_narg; register_t *ip; intmax_t arg; syscallname(ktr->ktr_code, sv_flags); ip = &ktr->ktr_args[0]; if (narg) { char c = '('; if (fancy && (sv_flags == 0 || (sv_flags & SV_ABI_MASK) == SV_ABI_FREEBSD)) { switch (ktr->ktr_code) { case SYS_bindat: case SYS_connectat: case SYS_faccessat: case SYS_fchmodat: case SYS_fchownat: case SYS_fstatat: case SYS_futimesat: case SYS_linkat: case SYS_mkdirat: case SYS_mkfifoat: case SYS_mknodat: case SYS_openat: case SYS_readlinkat: case SYS_renameat: case SYS_unlinkat: case SYS_utimensat: putchar('('); atfdname(*ip, decimal); c = ','; ip++; narg--; break; } switch (ktr->ktr_code) { case SYS_ioctl: { print_number(ip, narg, c); putchar(c); ioctlname(*ip); c = ','; ip++; narg--; break; } case SYS_ptrace: putchar('('); ptraceopname(*ip); c = ','; ip++; narg--; break; case SYS_access: case SYS_eaccess: case SYS_faccessat: print_number(ip, narg, c); putchar(','); accessmodename(*ip); ip++; narg--; break; case SYS_open: case SYS_openat: print_number(ip, narg, c); putchar(','); flagsandmodename(ip[0], ip[1], decimal); ip += 2; narg -= 2; break; case SYS_wait4: print_number(ip, narg, c); print_number(ip, narg, c); /* * A flags value of zero is valid for * wait4() but not for wait6(), so * handle zero special here. */ if (*ip == 0) { print_number(ip, narg, c); } else { putchar(','); wait6optname(*ip); ip++; narg--; } break; case SYS_wait6: putchar('('); idtypename(*ip, decimal); c = ','; ip++; narg--; print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); wait6optname(*ip); ip++; narg--; break; case SYS_chmod: case SYS_fchmod: case SYS_lchmod: print_number(ip, narg, c); putchar(','); modename(*ip); ip++; narg--; break; case SYS_mknod: case SYS_mknodat: print_number(ip, narg, c); putchar(','); modename(*ip); ip++; narg--; break; case SYS_getfsstat: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); getfsstatflagsname(*ip); ip++; narg--; break; case SYS_mount: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); mountflagsname(*ip); ip++; narg--; break; case SYS_unmount: print_number(ip, narg, c); putchar(','); mountflagsname(*ip); ip++; narg--; break; case SYS_recvmsg: case SYS_sendmsg: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); sendrecvflagsname(*ip); ip++; narg--; break; case SYS_recvfrom: case SYS_sendto: print_number(ip, narg, c); print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); sendrecvflagsname(*ip); ip++; narg--; break; case SYS_chflags: case SYS_fchflags: case SYS_lchflags: print_number(ip, narg, c); putchar(','); modename(*ip); ip++; narg--; break; case SYS_kill: print_number(ip, narg, c); putchar(','); signame(*ip); ip++; narg--; break; case SYS_reboot: putchar('('); rebootoptname(*ip); ip++; narg--; break; case SYS_umask: putchar('('); modename(*ip); ip++; narg--; break; case SYS_msync: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); msyncflagsname(*ip); ip++; narg--; break; #ifdef SYS_freebsd6_mmap case SYS_freebsd6_mmap: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); mmapprotname(*ip); putchar(','); ip++; narg--; mmapflagsname(*ip); ip++; narg--; break; #endif case SYS_mmap: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); mmapprotname(*ip); putchar(','); ip++; narg--; mmapflagsname(*ip); ip++; narg--; break; case SYS_mprotect: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); mmapprotname(*ip); ip++; narg--; break; case SYS_madvise: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); madvisebehavname(*ip); ip++; narg--; break; case SYS_setpriority: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); prioname(*ip); ip++; narg--; break; case SYS_fcntl: print_number(ip, narg, c); putchar(','); fcntlcmdname(ip[0], ip[1], decimal); ip += 2; narg -= 2; break; case SYS_socket: { int sockdomain; putchar('('); sockdomain = *ip; sockdomainname(sockdomain); ip++; narg--; putchar(','); socktypenamewithflags(*ip); ip++; narg--; if (sockdomain == PF_INET || sockdomain == PF_INET6) { putchar(','); sockipprotoname(*ip); ip++; narg--; } c = ','; break; } case SYS_setsockopt: case SYS_getsockopt: print_number(ip, narg, c); putchar(','); sockoptlevelname(*ip, decimal); if (*ip == SOL_SOCKET) { ip++; narg--; putchar(','); sockoptname(*ip); } ip++; narg--; break; #ifdef SYS_freebsd6_lseek case SYS_freebsd6_lseek: print_number(ip, narg, c); /* Hidden 'pad' argument, not in lseek(2) */ print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); whencename(*ip); ip++; narg--; break; #endif case SYS_lseek: print_number(ip, narg, c); /* Hidden 'pad' argument, not in lseek(2) */ print_number(ip, narg, c); putchar(','); whencename(*ip); ip++; narg--; break; case SYS_flock: print_number(ip, narg, c); putchar(','); flockname(*ip); ip++; narg--; break; case SYS_mkfifo: case SYS_mkfifoat: case SYS_mkdir: case SYS_mkdirat: print_number(ip, narg, c); putchar(','); modename(*ip); ip++; narg--; break; case SYS_shutdown: print_number(ip, narg, c); putchar(','); shutdownhowname(*ip); ip++; narg--; break; case SYS_socketpair: putchar('('); sockdomainname(*ip); ip++; narg--; putchar(','); socktypenamewithflags(*ip); ip++; narg--; c = ','; break; case SYS_getrlimit: case SYS_setrlimit: putchar('('); rlimitname(*ip); ip++; narg--; c = ','; break; case SYS_quotactl: print_number(ip, narg, c); putchar(','); quotactlname(*ip); ip++; narg--; c = ','; break; case SYS_nfssvc: putchar('('); nfssvcname(*ip); ip++; narg--; c = ','; break; case SYS_rtprio: putchar('('); rtprioname(*ip); ip++; narg--; c = ','; break; case SYS___semctl: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); semctlname(*ip); ip++; narg--; break; case SYS_semget: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); semgetname(*ip); ip++; narg--; break; case SYS_msgctl: print_number(ip, narg, c); putchar(','); shmctlname(*ip); ip++; narg--; break; case SYS_shmat: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); shmatname(*ip); ip++; narg--; break; case SYS_shmctl: print_number(ip, narg, c); putchar(','); shmctlname(*ip); ip++; narg--; break; case SYS_shm_open: print_number(ip, narg, c); putchar(','); flagsname(ip[0]); printf(",0%o", (unsigned int)ip[1]); ip += 3; narg -= 3; break; case SYS_minherit: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); minheritname(*ip); ip++; narg--; break; case SYS_rfork: putchar('('); rforkname(*ip); ip++; narg--; c = ','; break; case SYS_lio_listio: putchar('('); lio_listioname(*ip); ip++; narg--; c = ','; break; case SYS_mlockall: putchar('('); mlockallname(*ip); ip++; narg--; break; case SYS_sched_setscheduler: print_number(ip, narg, c); putchar(','); schedpolicyname(*ip); ip++; narg--; break; case SYS_sched_get_priority_max: case SYS_sched_get_priority_min: putchar('('); schedpolicyname(*ip); ip++; narg--; break; case SYS_sendfile: print_number(ip, narg, c); print_number(ip, narg, c); print_number(ip, narg, c); print_number(ip, narg, c); print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); sendfileflagsname(*(int *)ip); ip++; narg--; break; case SYS_kldsym: print_number(ip, narg, c); putchar(','); kldsymcmdname(*ip); ip++; narg--; break; case SYS_sigprocmask: putchar('('); sigprocmaskhowname(*ip); ip++; narg--; c = ','; break; case SYS___acl_get_file: case SYS___acl_set_file: case SYS___acl_get_fd: case SYS___acl_set_fd: case SYS___acl_delete_file: case SYS___acl_delete_fd: case SYS___acl_aclcheck_file: case SYS___acl_aclcheck_fd: case SYS___acl_get_link: case SYS___acl_set_link: case SYS___acl_delete_link: case SYS___acl_aclcheck_link: print_number(ip, narg, c); putchar(','); acltypename(*ip); ip++; narg--; break; case SYS_sigaction: putchar('('); signame(*ip); ip++; narg--; c = ','; break; case SYS_extattrctl: print_number(ip, narg, c); putchar(','); extattrctlname(*ip); ip++; narg--; break; case SYS_nmount: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); mountflagsname(*ip); ip++; narg--; break; case SYS_thr_create: print_number(ip, narg, c); print_number(ip, narg, c); putchar(','); thrcreateflagsname(*ip); ip++; narg--; break; case SYS_thr_kill: print_number(ip, narg, c); putchar(','); signame(*ip); ip++; narg--; break; case SYS_kldunloadf: print_number(ip, narg, c); putchar(','); kldunloadfflagsname(*ip); ip++; narg--; break; case SYS_linkat: case SYS_renameat: case SYS_symlinkat: print_number(ip, narg, c); putchar(','); atfdname(*ip, decimal); ip++; narg--; break; case SYS_cap_fcntls_limit: print_number(ip, narg, c); putchar(','); arg = *ip; ip++; narg--; capfcntlname(arg); break; case SYS_posix_fadvise: print_number(ip, narg, c); print_number(ip, narg, c); print_number(ip, narg, c); (void)putchar(','); fadvisebehavname((int)*ip); ip++; narg--; break; case SYS_procctl: putchar('('); idtypename(*ip, decimal); c = ','; ip++; narg--; print_number(ip, narg, c); putchar(','); procctlcmdname(*ip); ip++; narg--; break; case SYS__umtx_op: print_number(ip, narg, c); putchar(','); umtxopname(*ip); switch (*ip) { case UMTX_OP_CV_WAIT: ip++; narg--; putchar(','); umtxcvwaitflags(*ip); break; case UMTX_OP_RW_RDLOCK: ip++; narg--; putchar(','); umtxrwlockflags(*ip); break; } ip++; narg--; } } while (narg > 0) { print_number(ip, narg, c); } putchar(')'); } putchar('\n'); } void ktrsysret(struct ktr_sysret *ktr, u_int sv_flags) { register_t ret = ktr->ktr_retval; int error = ktr->ktr_error; syscallname(ktr->ktr_code, sv_flags); printf(" "); if (error == 0) { if (fancy) { printf("%ld", (long)ret); if (ret < 0 || ret > 9) printf("/%#lx", (unsigned long)ret); } else { if (decimal) printf("%ld", (long)ret); else printf("%#lx", (unsigned long)ret); } } else if (error == ERESTART) printf("RESTART"); else if (error == EJUSTRETURN) printf("JUSTRETURN"); else { printf("-1 errno %d", ktr->ktr_error); if (fancy) printf(" %s", strerror(ktr->ktr_error)); } putchar('\n'); } void ktrnamei(char *cp, int len) { printf("\"%.*s\"\n", len, cp); } void hexdump(char *p, int len, int screenwidth) { int n, i; int width; width = 0; do { width += 2; i = 13; /* base offset */ i += (width / 2) + 1; /* spaces every second byte */ i += (width * 2); /* width of bytes */ i += 3; /* " |" */ i += width; /* each byte */ i += 1; /* "|" */ } while (i < screenwidth); width -= 2; for (n = 0; n < len; n += width) { for (i = n; i < n + width; i++) { if ((i % width) == 0) { /* beginning of line */ printf(" 0x%04x", i); } if ((i % 2) == 0) { printf(" "); } if (i < len) printf("%02x", p[i] & 0xff); else printf(" "); } printf(" |"); for (i = n; i < n + width; i++) { if (i >= len) break; if (p[i] >= ' ' && p[i] <= '~') printf("%c", p[i]); else printf("."); } printf("|\n"); } if ((i % width) != 0) printf("\n"); } void visdump(char *dp, int datalen, int screenwidth) { int col = 0; char *cp; int width; char visbuf[5]; printf(" \""); col = 8; for (;datalen > 0; datalen--, dp++) { vis(visbuf, *dp, VIS_CSTYLE, *(dp+1)); cp = visbuf; /* * Keep track of printables and * space chars (like fold(1)). */ if (col == 0) { putchar('\t'); col = 8; } switch(*cp) { case '\n': col = 0; putchar('\n'); continue; case '\t': width = 8 - (col&07); break; default: width = strlen(cp); } if (col + width > (screenwidth-2)) { printf("\\\n\t"); col = 8; } col += width; do { putchar(*cp++); } while (*cp); } if (col == 0) printf(" "); printf("\"\n"); } void ktrgenio(struct ktr_genio *ktr, int len) { int datalen = len - sizeof (struct ktr_genio); char *dp = (char *)ktr + sizeof (struct ktr_genio); static int screenwidth = 0; int i, binary; printf("fd %d %s %d byte%s\n", ktr->ktr_fd, ktr->ktr_rw == UIO_READ ? "read" : "wrote", datalen, datalen == 1 ? "" : "s"); if (suppressdata) return; if (screenwidth == 0) { struct winsize ws; if (fancy && ioctl(fileno(stderr), TIOCGWINSZ, &ws) != -1 && ws.ws_col > 8) screenwidth = ws.ws_col; else screenwidth = 80; } if (maxdata && datalen > maxdata) datalen = maxdata; for (i = 0, binary = 0; i < datalen && binary == 0; i++) { if (dp[i] >= 32 && dp[i] < 127) continue; if (dp[i] == 10 || dp[i] == 13 || dp[i] == 0 || dp[i] == 9) continue; binary = 1; } if (binary) hexdump(dp, datalen, screenwidth); else visdump(dp, datalen, screenwidth); } const char *signames[] = { "NULL", "HUP", "INT", "QUIT", "ILL", "TRAP", "IOT", /* 1 - 6 */ "EMT", "FPE", "KILL", "BUS", "SEGV", "SYS", /* 7 - 12 */ "PIPE", "ALRM", "TERM", "URG", "STOP", "TSTP", /* 13 - 18 */ "CONT", "CHLD", "TTIN", "TTOU", "IO", "XCPU", /* 19 - 24 */ "XFSZ", "VTALRM", "PROF", "WINCH", "29", "USR1", /* 25 - 30 */ "USR2", NULL, /* 31 - 32 */ }; void ktrpsig(struct ktr_psig *psig) { if (psig->signo > 0 && psig->signo < NSIG) printf("SIG%s ", signames[psig->signo]); else printf("SIG %d ", psig->signo); if (psig->action == SIG_DFL) { printf("SIG_DFL code="); sigcodename(psig->signo, psig->code); putchar('\n'); } else { printf("caught handler=0x%lx mask=0x%x code=", (u_long)psig->action, psig->mask.__bits[0]); sigcodename(psig->signo, psig->code); putchar('\n'); } } void ktrcsw_old(struct ktr_csw_old *cs) { printf("%s %s\n", cs->out ? "stop" : "resume", cs->user ? "user" : "kernel"); } void ktrcsw(struct ktr_csw *cs) { printf("%s %s \"%s\"\n", cs->out ? "stop" : "resume", cs->user ? "user" : "kernel", cs->wmesg); } void ktruser(int len, void *p) { unsigned char *cp; if (sysdecode_utrace(stdout, p, len)) { printf("\n"); return; } printf("%d ", len); cp = p; while (len--) if (decimal) printf(" %d", *cp++); else printf(" %02x", *cp++); printf("\n"); } void ktrcaprights(cap_rights_t *rightsp) { printf("cap_rights_t "); capname(rightsp); printf("\n"); } void ktrsockaddr(struct sockaddr *sa) { /* TODO: Support additional address families #include struct sockaddr_natm *natm; #include struct sockaddr_nb *nb; */ char addr[64]; /* * note: ktrstruct() has already verified that sa points to a * buffer at least sizeof(struct sockaddr) bytes long and exactly * sa->sa_len bytes long. */ printf("struct sockaddr { "); sockfamilyname(sa->sa_family); printf(", "); #define check_sockaddr_len(n) \ if (sa_##n.s##n##_len < sizeof(struct sockaddr_##n)) { \ printf("invalid"); \ break; \ } switch(sa->sa_family) { case AF_INET: { struct sockaddr_in sa_in; memset(&sa_in, 0, sizeof(sa_in)); memcpy(&sa_in, sa, sa->sa_len); check_sockaddr_len(in); inet_ntop(AF_INET, &sa_in.sin_addr, addr, sizeof addr); printf("%s:%u", addr, ntohs(sa_in.sin_port)); break; } case AF_INET6: { struct sockaddr_in6 sa_in6; memset(&sa_in6, 0, sizeof(sa_in6)); memcpy(&sa_in6, sa, sa->sa_len); check_sockaddr_len(in6); getnameinfo((struct sockaddr *)&sa_in6, sizeof(sa_in6), addr, sizeof(addr), NULL, 0, NI_NUMERICHOST); printf("[%s]:%u", addr, htons(sa_in6.sin6_port)); break; } case AF_UNIX: { struct sockaddr_un sa_un; memset(&sa_un, 0, sizeof(sa_un)); memcpy(&sa_un, sa, sa->sa_len); printf("%.*s", (int)sizeof(sa_un.sun_path), sa_un.sun_path); break; } default: printf("unknown address family"); } printf(" }\n"); } void ktrstat(struct stat *statp) { char mode[12], timestr[PATH_MAX + 4]; struct passwd *pwd; struct group *grp; struct tm *tm; /* * note: ktrstruct() has already verified that statp points to a * buffer exactly sizeof(struct stat) bytes long. */ printf("struct stat {"); printf("dev=%ju, ino=%ju, ", (uintmax_t)statp->st_dev, (uintmax_t)statp->st_ino); if (resolv == 0) printf("mode=0%jo, ", (uintmax_t)statp->st_mode); else { strmode(statp->st_mode, mode); printf("mode=%s, ", mode); } printf("nlink=%ju, ", (uintmax_t)statp->st_nlink); if (resolv == 0) { pwd = NULL; } else { #ifdef HAVE_LIBCAPSICUM if (cappwd != NULL) pwd = cap_getpwuid(cappwd, statp->st_uid); else #endif pwd = getpwuid(statp->st_uid); } if (pwd == NULL) printf("uid=%ju, ", (uintmax_t)statp->st_uid); else printf("uid=\"%s\", ", pwd->pw_name); if (resolv == 0) { grp = NULL; } else { #ifdef HAVE_LIBCAPSICUM if (capgrp != NULL) grp = cap_getgrgid(capgrp, statp->st_gid); else #endif grp = getgrgid(statp->st_gid); } if (grp == NULL) printf("gid=%ju, ", (uintmax_t)statp->st_gid); else printf("gid=\"%s\", ", grp->gr_name); printf("rdev=%ju, ", (uintmax_t)statp->st_rdev); printf("atime="); if (resolv == 0) printf("%jd", (intmax_t)statp->st_atim.tv_sec); else { tm = localtime(&statp->st_atim.tv_sec); strftime(timestr, sizeof(timestr), TIME_FORMAT, tm); printf("\"%s\"", timestr); } if (statp->st_atim.tv_nsec != 0) printf(".%09ld, ", statp->st_atim.tv_nsec); else printf(", "); printf("stime="); if (resolv == 0) printf("%jd", (intmax_t)statp->st_mtim.tv_sec); else { tm = localtime(&statp->st_mtim.tv_sec); strftime(timestr, sizeof(timestr), TIME_FORMAT, tm); printf("\"%s\"", timestr); } if (statp->st_mtim.tv_nsec != 0) printf(".%09ld, ", statp->st_mtim.tv_nsec); else printf(", "); printf("ctime="); if (resolv == 0) printf("%jd", (intmax_t)statp->st_ctim.tv_sec); else { tm = localtime(&statp->st_ctim.tv_sec); strftime(timestr, sizeof(timestr), TIME_FORMAT, tm); printf("\"%s\"", timestr); } if (statp->st_ctim.tv_nsec != 0) printf(".%09ld, ", statp->st_ctim.tv_nsec); else printf(", "); printf("birthtime="); if (resolv == 0) printf("%jd", (intmax_t)statp->st_birthtim.tv_sec); else { tm = localtime(&statp->st_birthtim.tv_sec); strftime(timestr, sizeof(timestr), TIME_FORMAT, tm); printf("\"%s\"", timestr); } if (statp->st_birthtim.tv_nsec != 0) printf(".%09ld, ", statp->st_birthtim.tv_nsec); else printf(", "); printf("size=%jd, blksize=%ju, blocks=%jd, flags=0x%x", (uintmax_t)statp->st_size, (uintmax_t)statp->st_blksize, (intmax_t)statp->st_blocks, statp->st_flags); printf(" }\n"); } void ktrstruct(char *buf, size_t buflen) { char *name, *data; size_t namelen, datalen; int i; cap_rights_t rights; struct stat sb; struct sockaddr_storage ss; for (name = buf, namelen = 0; namelen < buflen && name[namelen] != '\0'; ++namelen) /* nothing */; if (namelen == buflen) goto invalid; if (name[namelen] != '\0') goto invalid; data = buf + namelen + 1; datalen = buflen - namelen - 1; if (datalen == 0) goto invalid; /* sanity check */ for (i = 0; i < (int)namelen; ++i) if (!isalpha(name[i])) goto invalid; if (strcmp(name, "caprights") == 0) { if (datalen != sizeof(cap_rights_t)) goto invalid; memcpy(&rights, data, datalen); ktrcaprights(&rights); } else if (strcmp(name, "stat") == 0) { if (datalen != sizeof(struct stat)) goto invalid; memcpy(&sb, data, datalen); ktrstat(&sb); } else if (strcmp(name, "sockaddr") == 0) { if (datalen > sizeof(ss)) goto invalid; memcpy(&ss, data, datalen); if (datalen != ss.ss_len) goto invalid; ktrsockaddr((struct sockaddr *)&ss); } else { printf("unknown structure\n"); } return; invalid: printf("invalid record\n"); } void ktrcapfail(struct ktr_cap_fail *ktr) { switch (ktr->cap_type) { case CAPFAIL_NOTCAPABLE: /* operation on fd with insufficient capabilities */ printf("operation requires "); capname(&ktr->cap_needed); printf(", descriptor holds "); capname(&ktr->cap_held); break; case CAPFAIL_INCREASE: /* requested more capabilities than fd already has */ printf("attempt to increase capabilities from "); capname(&ktr->cap_held); printf(" to "); capname(&ktr->cap_needed); break; case CAPFAIL_SYSCALL: /* called restricted syscall */ printf("disallowed system call"); break; case CAPFAIL_LOOKUP: /* used ".." in strict-relative mode */ printf("restricted VFS lookup"); break; default: printf("unknown capability failure: "); capname(&ktr->cap_needed); printf(" "); capname(&ktr->cap_held); break; } printf("\n"); } void ktrfault(struct ktr_fault *ktr) { printf("0x%jx ", (uintmax_t)ktr->vaddr); vmprotname(ktr->type); printf("\n"); } void ktrfaultend(struct ktr_faultend *ktr) { vmresultname(ktr->result); printf("\n"); } #if defined(__amd64__) || defined(__i386__) void linux_ktrsysret(struct ktr_sysret *ktr, u_int sv_flags) { register_t ret = ktr->ktr_retval; int error = ktr->ktr_error; syscallname(ktr->ktr_code, sv_flags); printf(" "); if (error == 0) { if (fancy) { printf("%ld", (long)ret); if (ret < 0 || ret > 9) printf("/%#lx", (unsigned long)ret); } else { if (decimal) printf("%ld", (long)ret); else printf("%#lx", (unsigned long)ret); } } else if (error == ERESTART) printf("RESTART"); else if (error == EJUSTRETURN) printf("JUSTRETURN"); else { if (ktr->ktr_error <= ELAST + 1) error = abs(bsd_to_linux_errno[ktr->ktr_error]); else error = 999; printf("-1 errno %d", error); if (fancy) printf(" %s", strerror(ktr->ktr_error)); } putchar('\n'); } #endif void usage(void) { fprintf(stderr, "usage: kdump [-dEnlHRrSsTA] [-f trfile] " "[-m maxdata] [-p pid] [-t trstr]\n"); exit(1); } Index: user/ngie/make_check/usr.bin/truss/aarch64-cloudabi64.c =================================================================== --- user/ngie/make_check/usr.bin/truss/aarch64-cloudabi64.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/aarch64-cloudabi64.c (revision 295078) @@ -1,89 +1,89 @@ /*- * Copyright (c) 2015 Nuxi, https://nuxi.nl/ * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include "cloudabi.h" #include "truss.h" static int aarch64_cloudabi64_fetch_args(struct trussinfo *trussinfo, unsigned int narg) { struct current_syscall *cs; struct reg regs; lwpid_t tid; unsigned int i; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) == -1) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } cs = &trussinfo->curthread->cs; for (i = 0; i < narg && i < 8; i++) cs->args[i] = regs.x[i]; return (0); } static int aarch64_cloudabi64_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) == -1) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.x[0]; retval[1] = regs.x[1]; *errorp = (regs.spsr & PSR_C) != 0; if (*errorp) retval[0] = cloudabi_convert_errno(retval[0]); return (0); } static struct procabi aarch64_cloudabi64 = { "CloudABI ELF64", - CLOUDABI64, + SYSDECODE_ABI_CLOUDABI64, aarch64_cloudabi64_fetch_args, aarch64_cloudabi64_fetch_retval }; PROCABI(aarch64_cloudabi64); Index: user/ngie/make_check/usr.bin/truss/aarch64-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/aarch64-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/aarch64-freebsd.c (revision 295078) @@ -1,107 +1,107 @@ /* * Copyright (c) 2015 The FreeBSD Foundation * * Portions of this software were developed by Konstantin Belousov * under sponsorship from the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/arm64-specific system call handling. */ #include #include #include #include #include #include #include #include "truss.h" static int aarch64_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg, syscall_num; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ syscall_num = regs.x[8]; if (syscall_num == SYS_syscall || syscall_num == SYS___syscall) { reg = 1; syscall_num = regs.x[0]; } else { reg = 0; } for (i = 0; i < narg && reg < 8; i++, reg++) cs->args[i] = regs.x[reg]; return (0); } static int aarch64_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.x[0]; retval[1] = regs.x[1]; *errorp = !!(regs.spsr & PSR_C); return (0); } static struct procabi aarch64_freebsd = { "FreeBSD ELF64", - FREEBSD, + SYSDECODE_ABI_FREEBSD, aarch64_fetch_args, aarch64_fetch_retval }; PROCABI(aarch64_freebsd); Index: user/ngie/make_check/usr.bin/truss/amd64-cloudabi64.c =================================================================== --- user/ngie/make_check/usr.bin/truss/amd64-cloudabi64.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/amd64-cloudabi64.c (revision 295078) @@ -1,98 +1,98 @@ /*- * Copyright (c) 2015 Nuxi, https://nuxi.nl/ * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include "cloudabi.h" #include "truss.h" static int amd64_cloudabi64_fetch_args(struct trussinfo *trussinfo, unsigned int narg) { struct current_syscall *cs; struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) == -1) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } cs = &trussinfo->curthread->cs; if (narg >= 1) cs->args[0] = regs.r_rdi; if (narg >= 2) cs->args[1] = regs.r_rsi; if (narg >= 3) cs->args[2] = regs.r_rdx; if (narg >= 4) cs->args[3] = regs.r_rcx; if (narg >= 5) cs->args[4] = regs.r_r8; if (narg >= 6) cs->args[5] = regs.r_r9; return (0); } static int amd64_cloudabi64_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) == -1) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.r_rax; retval[1] = regs.r_rdx; *errorp = (regs.r_rflags & PSL_C) != 0; if (*errorp) retval[0] = cloudabi_convert_errno(retval[0]); return (0); } static struct procabi amd64_cloudabi64 = { "CloudABI ELF64", - CLOUDABI64, + SYSDECODE_ABI_CLOUDABI64, amd64_cloudabi64_fetch_args, amd64_cloudabi64_fetch_retval }; PROCABI(amd64_cloudabi64); Index: user/ngie/make_check/usr.bin/truss/amd64-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/amd64-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/amd64-freebsd.c (revision 295078) @@ -1,129 +1,129 @@ /* * Copyright 1997 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/amd64-specific system call handling. */ #include #include #include #include #include #include #include "truss.h" static int amd64_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over %rax if it contains * either of these values. */ reg = 0; switch (regs.r_rax) { case SYS_syscall: case SYS___syscall: reg++; break; } for (i = 0; i < narg && reg < 6; i++, reg++) { switch (reg) { case 0: cs->args[i] = regs.r_rdi; break; case 1: cs->args[i] = regs.r_rsi; break; case 2: cs->args[i] = regs.r_rdx; break; case 3: cs->args[i] = regs.r_rcx; break; case 4: cs->args[i] = regs.r_r8; break; case 5: cs->args[i] = regs.r_r9; break; } } if (narg > i) { iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)(regs.r_rsp + sizeof(register_t)); iorequest.piod_addr = &cs->args[i]; iorequest.piod_len = (narg - i) * sizeof(register_t); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); } return (0); } static int amd64_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.r_rax; retval[1] = regs.r_rdx; *errorp = !!(regs.r_rflags & PSL_C); return (0); } static struct procabi amd64_freebsd = { "FreeBSD ELF64", - FREEBSD, + SYSDECODE_ABI_FREEBSD, amd64_fetch_args, amd64_fetch_retval }; PROCABI(amd64_freebsd); Index: user/ngie/make_check/usr.bin/truss/amd64-freebsd32.c =================================================================== --- user/ngie/make_check/usr.bin/truss/amd64-freebsd32.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/amd64-freebsd32.c (revision 295078) @@ -1,134 +1,134 @@ /* * Copyright 1997 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/amd64-freebsd32-specific system call handling. */ #include #include #include #include #include #include #include #include "truss.h" static int amd64_freebsd32_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; unsigned int args32[narg]; unsigned long parm_offset; lwpid_t tid; u_int i; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } parm_offset = regs.r_rsp + sizeof(int); /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ switch (regs.r_rax) { case SYS_syscall: parm_offset += sizeof(int); break; case SYS___syscall: parm_offset += sizeof(quad_t); break; } iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)parm_offset; iorequest.piod_addr = args32; iorequest.piod_len = sizeof(args32); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) { return (-1); } for (i = 0; i < narg; i++) cs->args[i] = args32[i]; return (0); } static int amd64_freebsd32_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.r_rax & 0xffffffff; retval[1] = regs.r_rdx & 0xffffffff; *errorp = !!(regs.r_rflags & PSL_C); return (0); } static struct procabi amd64_freebsd32 = { "FreeBSD ELF32", - FREEBSD32, + SYSDECODE_ABI_FREEBSD32, amd64_freebsd32_fetch_args, amd64_freebsd32_fetch_retval }; PROCABI(amd64_freebsd32); static struct procabi amd64_freebsd32_aout = { "FreeBSD a.out", - FREEBSD32, + SYSDECODE_ABI_FREEBSD32, amd64_freebsd32_fetch_args, amd64_freebsd32_fetch_retval }; PROCABI(amd64_freebsd32_aout); Index: user/ngie/make_check/usr.bin/truss/amd64-linux32.c =================================================================== --- user/ngie/make_check/usr.bin/truss/amd64-linux32.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/amd64-linux32.c (revision 295078) @@ -1,139 +1,139 @@ /* * Copyright 1997 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* Linux/i386-specific system call handling. */ #include #include #include #include #include #include "truss.h" static int amd64_linux32_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct reg regs; struct current_syscall *cs; lwpid_t tid; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * Linux passes syscall arguments in registers, not * on the stack. Fortunately, we've got access to the * register set. Note that we don't bother checking the * number of arguments. And what does linux do for syscalls * that have more than five arguments? */ switch (narg) { default: cs->args[5] = regs.r_rbp; /* Unconfirmed */ case 5: cs->args[4] = regs.r_rdi; case 4: cs->args[3] = regs.r_rsi; case 3: cs->args[2] = regs.r_rdx; case 2: cs->args[1] = regs.r_rcx; case 1: cs->args[0] = regs.r_rbx; } return (0); } /* * Linux syscalls return negative errno's, we do positive and map them */ static const int bsd_to_linux_errno[] = { -0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, -35, -12, -13, -14, -15, -16, -17, -18, -19, -20, -21, -22, -23, -24, -25, -26, -27, -28, -29, -30, -31, -32, -33, -34, -11,-115,-114, -88, -89, -90, -91, -92, -93, -94, -95, -96, -97, -98, -99, -100,-101,-102,-103,-104,-105,-106,-107,-108,-109, -110,-111, -40, -36,-112,-113, -39, -11, -87,-122, -116, -66, -6, -6, -6, -6, -6, -37, -38, -9, -6, }; static int amd64_linux32_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; size_t i; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.r_rax & 0xffffffff; retval[1] = regs.r_rdx & 0xffffffff; *errorp = !!(regs.r_rflags & PSL_C); if (*errorp) { for (i = 0; i < nitems(bsd_to_linux_errno); i++) { if (retval[0] == bsd_to_linux_errno[i]) { retval[0] = i; return (0); } } /* XXX: How to handle unknown errors? */ } return (0); } static struct procabi amd64_linux32 = { "Linux ELF32", - LINUX32, + SYSDECODE_ABI_LINUX32, amd64_linux32_fetch_args, amd64_linux32_fetch_retval }; PROCABI(amd64_linux32); Index: user/ngie/make_check/usr.bin/truss/arm-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/arm-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/arm-freebsd.c (revision 295078) @@ -1,136 +1,136 @@ /* * Copyright 1997 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/arm-specific system call handling. */ #include #include #include #include #include #include #include #include "truss.h" static int arm_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg, syscall_num; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ #ifdef __ARM_EABI__ syscall_num = regs.r[7]; #else if ((syscall_num = ptrace(PT_READ_I, tid, (caddr_t)(regs.r[_REG_PC] - INSN_SIZE), 0)) == -1) { fprintf(trussinfo->outfile, "-- CANNOT READ PC --\n"); return (-1); } syscall_num = syscall_num & 0x000fffff; #endif reg = 0; switch (syscall_num) { case SYS_syscall: reg = 1; break; case SYS___syscall: reg = 2; break; } for (i = 0; i < narg && reg < 4; i++, reg++) cs->args[i] = regs.r[reg]; if (narg > i) { iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)(regs.r_sp + 4 * sizeof(uint32_t)); iorequest.piod_addr = &cs->args[i]; iorequest.piod_len = (narg - i) * sizeof(cs->args[0]); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); } return (0); } static int arm_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* XXX: Does not have the __ARMEB__ handling for __syscall(). */ retval[0] = regs.r[0]; retval[1] = regs.r[1]; *errorp = !!(regs.r_cpsr & PSR_C); return (0); } static struct procabi arm_freebsd = { "FreeBSD ELF32", - FREEBSD, + SYSDECODE_ABI_FREEBSD, arm_fetch_args, arm_fetch_retval }; PROCABI(arm_freebsd); Index: user/ngie/make_check/usr.bin/truss/i386-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/i386-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/i386-freebsd.c (revision 295078) @@ -1,128 +1,128 @@ /* * Copyright 1997 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/i386-specific system call handling. */ #include #include #include #include #include #include #include "truss.h" static int i386_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; unsigned int parm_offset; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } parm_offset = regs.r_esp + sizeof(int); /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ switch (regs.r_eax) { case SYS_syscall: parm_offset += sizeof(int); break; case SYS___syscall: parm_offset += sizeof(quad_t); break; } iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)parm_offset; iorequest.piod_addr = cs->args; iorequest.piod_len = narg * sizeof(unsigned long); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); return (0); } static int i386_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.r_eax; retval[1] = regs.r_edx; *errorp = !!(regs.r_eflags & PSL_C); return (0); } static struct procabi i386_freebsd = { "FreeBSD ELF32", - FREEBSD, + SYSDECODE_ABI_FREEBSD, i386_fetch_args, i386_fetch_retval }; PROCABI(i386_freebsd); static struct procabi i386_freebsd_aout = { "FreeBSD a.out", - FREEBSD, + SYSDECODE_ABI_FREEBSD, i386_fetch_args, i386_fetch_retval }; PROCABI(i386_freebsd_aout); Index: user/ngie/make_check/usr.bin/truss/i386-linux.c =================================================================== --- user/ngie/make_check/usr.bin/truss/i386-linux.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/i386-linux.c (revision 295078) @@ -1,138 +1,138 @@ /* * Copyright 1997 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* Linux/i386-specific system call handling. */ #include #include #include #include #include #include "truss.h" static int i386_linux_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct reg regs; struct current_syscall *cs; lwpid_t tid; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * Linux passes syscall arguments in registers, not * on the stack. Fortunately, we've got access to the * register set. Note that we don't bother checking the * number of arguments. And what does linux do for syscalls * that have more than five arguments? */ switch (narg) { default: cs->args[5] = regs.r_ebp; /* Unconfirmed */ case 5: cs->args[4] = regs.r_edi; case 4: cs->args[3] = regs.r_esi; case 3: cs->args[2] = regs.r_edx; case 2: cs->args[1] = regs.r_ecx; case 1: cs->args[0] = regs.r_ebx; } return (0); } /* * Linux syscalls return negative errno's, we do positive and map them */ static const int bsd_to_linux_errno[] = { -0, -1, -2, -3, -4, -5, -6, -7, -8, -9, -10, -35, -12, -13, -14, -15, -16, -17, -18, -19, -20, -21, -22, -23, -24, -25, -26, -27, -28, -29, -30, -31, -32, -33, -34, -11,-115,-114, -88, -89, -90, -91, -92, -93, -94, -95, -96, -97, -98, -99, -100,-101,-102,-103,-104,-105,-106,-107,-108,-109, -110,-111, -40, -36,-112,-113, -39, -11, -87,-122, -116, -66, -6, -6, -6, -6, -6, -37, -38, -9, -6, }; static int i386_linux_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; size_t i; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.r_eax; retval[1] = regs.r_edx; *errorp = !!(regs.r_eflags & PSL_C); if (*errorp) { for (i = 0; i < nitems(bsd_to_linux_errno); i++) { if (retval[0] == bsd_to_linux_errno[i]) { retval[0] = i; return (0); } } /* XXX: How to handle unknown errors? */ } return (0); } static struct procabi i386_linux = { "Linux ELF32", - LINUX, + SYSDECODE_ABI_LINUX, i386_linux_fetch_args, i386_linux_fetch_retval }; PROCABI(i386_linux); Index: user/ngie/make_check/usr.bin/truss/mips-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/mips-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/mips-freebsd.c (revision 295078) @@ -1,139 +1,139 @@ /* * Copyright 1998 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/mips-specific system call handling. */ #include #include #include #include #include #include #include "truss.h" static int mips_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ reg = A0; switch (regs.r_regs[V0]) { case SYS_syscall: reg = A1; break; case SYS___syscall: #if defined(__mips_n32) || defined(__mips_n64) reg = A1; #else reg = A2; #endif break; } #if defined(__mips_n32) || defined(__mips_n64) #define MAXREG A7 #else #define MAXREG A3 #endif for (i = 0; i < narg && reg <= MAXREG; i++, reg++) cs->args[i] = regs.r_regs[reg]; if (narg > i) { iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)((uintptr_t)regs.r_regs[SP] + 4 * sizeof(cs->args[0])); iorequest.piod_addr = &cs->args[i]; iorequest.piod_len = (narg - i) * sizeof(cs->args[0]); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); } return (0); } static int mips_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* XXX: Does not have special handling for __syscall(). */ retval[0] = regs.r_regs[V0]; retval[1] = regs.r_regs[V1]; *errorp = !!regs.r_regs[A3]; return (0); } static struct procabi mips_freebsd = { #ifdef __mips_n64 "FreeBSD ELF64", #else "FreeBSD ELF32", #endif - FREEBSD, + SYSDECODE_ABI_FREEBSD, mips_fetch_args, mips_fetch_retval }; PROCABI(mips_freebsd); Index: user/ngie/make_check/usr.bin/truss/powerpc-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/powerpc-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/powerpc-freebsd.c (revision 295078) @@ -1,120 +1,120 @@ /* * Copyright 2006 Peter Grehan * Copyright 2005 Orlando Bassotto * Copyright 1998 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/powerpc-specific system call handling. */ #include #include #include #include #include #include #include "truss.h" static int powerpc_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ reg = 0; switch (regs.fixreg[0]) { case SYS_syscall: reg += 1; break; case SYS___syscall: reg += 2; break; } for (i = 0; i < narg && reg < NARGREG; i++, reg++) { cs->args[i] = regs.fixreg[FIRSTARG + reg]; } if (narg > i) { iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)(regs.fixreg[1] + 8); iorequest.piod_addr = &cs->args[i]; iorequest.piod_len = (narg - i) * sizeof(cs->args[0]); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); } return (0); } static int powerpc_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* XXX: Does not have fixup for __syscall(). */ retval[0] = regs.fixreg[3]; retval[1] = regs.fixreg[4]; *errorp = !!(regs.cr & 0x10000000); return (0); } static struct procabi powerpc_freebsd = { "FreeBSD ELF32", - FREEBSD, + SYSDECODE_ABI_FREEBSD, powerpc_fetch_args, powerpc_fetch_retval }; PROCABI(powerpc_freebsd); Index: user/ngie/make_check/usr.bin/truss/powerpc64-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/powerpc64-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/powerpc64-freebsd.c (revision 295078) @@ -1,116 +1,116 @@ /* * Copyright 2006 Peter Grehan * Copyright 2005 Orlando Bassotto * Copyright 1998 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/powerpc64-specific system call handling. */ #include #include #include #include #include #include #include "truss.h" static int powerpc64_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ reg = 0; switch (regs.fixreg[0]) { case SYS_syscall: case SYS___syscall: reg += 1; break; } for (i = 0; i < narg && reg < NARGREG; i++, reg++) cs->args[i] = regs.fixreg[FIRSTARG + reg]; if (narg > i) { iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)(regs.fixreg[1] + 48); iorequest.piod_addr = &cs->args[i]; iorequest.piod_len = (narg - i) * sizeof(cs->args[0]); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); } return (0); } static int powerpc64_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.fixreg[3]; retval[1] = regs.fixreg[4]; *errorp = !!(regs.cr & 0x10000000); return (0); } static struct procabi powerpc64_freebsd = { "FreeBSD ELF64", - FREEBSD, + SYSDECODE_ABI_FREEBSD, powerpc64_fetch_args, powerpc64_fetch_retval }; PROCABI(powerpc64_freebsd); Index: user/ngie/make_check/usr.bin/truss/powerpc64-freebsd32.c =================================================================== --- user/ngie/make_check/usr.bin/truss/powerpc64-freebsd32.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/powerpc64-freebsd32.c (revision 295078) @@ -1,125 +1,125 @@ /* * Copyright 2006 Peter Grehan * Copyright 2005 Orlando Bassotto * Copyright 1998 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/powerpc64-freebsd32-specific system call handling. */ #include #include #include #include #include #include #include "truss.h" static int powerpc64_freebsd32_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ reg = 0; switch (regs.fixreg[0]) { case SYS_syscall: reg += 1; break; case SYS___syscall: reg += 2; break; } for (i = 0; i < narg && reg < NARGREG; i++, reg++) { cs->args[i] = regs.fixreg[FIRSTARG + reg] & 0xffffffff; } if (narg > i) { uint32_t args32[narg - i]; u_int j; iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)(regs.fixreg[1] + 8); iorequest.piod_addr = args32; iorequest.piod_len = sizeof(args32); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); for (j = 0; j < narg - i; j++) cs->args[i + j] = args32[j]; } return (0); } static int powerpc64_freebsd32_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* XXX: Does not have fixup for __syscall(). */ retval[0] = regs.fixreg[3] & 0xffffffff; retval[1] = regs.fixreg[4] & 0xffffffff; *errorp = !!(regs.cr & 0x10000000); return (0); } static struct procabi powerpc64_freebsd32 = { "FreeBSD ELF32", - FREEBSD32, + SYSDECODE_ABI_FREEBSD32, powerpc64_freebsd32_fetch_args, powerpc64_freebsd32_fetch_retval }; PROCABI(powerpc64_freebsd32); Index: user/ngie/make_check/usr.bin/truss/sparc64-freebsd.c =================================================================== --- user/ngie/make_check/usr.bin/truss/sparc64-freebsd.c (revision 295077) +++ user/ngie/make_check/usr.bin/truss/sparc64-freebsd.c (revision 295078) @@ -1,123 +1,123 @@ /* * Copyright 1998 Sean Eric Fagan * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by Sean Eric Fagan * 4. Neither the name of the author may be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* FreeBSD/sparc64-specific system call handling. */ #include #include #include #include #include #include #include #include #include "truss.h" static int sparc64_fetch_args(struct trussinfo *trussinfo, u_int narg) { struct ptrace_io_desc iorequest; struct reg regs; struct current_syscall *cs; lwpid_t tid; u_int i, reg; tid = trussinfo->curthread->tid; cs = &trussinfo->curthread->cs; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } /* * FreeBSD has two special kinds of system call redirections -- * SYS_syscall, and SYS___syscall. The former is the old syscall() * routine, basically; the latter is for quad-aligned arguments. * * The system call argument count and code from ptrace() already * account for these, but we need to skip over the first argument. */ reg = 0; switch (regs.r_global[1]) { case SYS_syscall: case SYS___syscall: reg = 1; break; } for (i = 0; i < narg && reg < 6; i++, reg++) cs->args[i] = regs.r_out[reg]; if (narg > i) { iorequest.piod_op = PIOD_READ_D; iorequest.piod_offs = (void *)(regs.r_out[6] + SPOFF + offsetof(struct frame, fr_pad[6])); iorequest.piod_addr = &cs->args[i]; iorequest.piod_len = (narg - i) * sizeof(cs->args[0]); ptrace(PT_IO, tid, (caddr_t)&iorequest, 0); if (iorequest.piod_len == 0) return (-1); } return (0); } static int sparc64_fetch_retval(struct trussinfo *trussinfo, long *retval, int *errorp) { struct reg regs; lwpid_t tid; tid = trussinfo->curthread->tid; if (ptrace(PT_GETREGS, tid, (caddr_t)®s, 0) < 0) { fprintf(trussinfo->outfile, "-- CANNOT READ REGISTERS --\n"); return (-1); } retval[0] = regs.r_out[0]; retval[1] = regs.r_out[1]; *errorp = !!(regs.r_tstate & TSTATE_XCC_C); return (0); } static struct procabi sparc64_freebsd = { "FreeBSD ELF64", - FREEBSD, + SYSDECODE_ABI_FREEBSD, sparc64_fetch_args, sparc64_fetch_retval }; PROCABI(sparc64_freebsd); Index: user/ngie/make_check/usr.sbin/bsdinstall/scripts/zfsboot =================================================================== --- user/ngie/make_check/usr.sbin/bsdinstall/scripts/zfsboot (revision 295077) +++ user/ngie/make_check/usr.sbin/bsdinstall/scripts/zfsboot (revision 295078) @@ -1,1656 +1,1635 @@ #!/bin/sh #- # Copyright (c) 2013-2015 Allan Jude # Copyright (c) 2013-2015 Devin Teske # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. # # $FreeBSD$ # ############################################################ INCLUDES BSDCFG_SHARE="/usr/share/bsdconfig" . $BSDCFG_SHARE/common.subr || exit 1 f_dprintf "%s: loading includes..." "$0" f_include $BSDCFG_SHARE/device.subr f_include $BSDCFG_SHARE/dialog.subr f_include $BSDCFG_SHARE/password/password.subr f_include $BSDCFG_SHARE/variable.subr ############################################################ CONFIGURATION # # Default name of the boot-pool # : ${ZFSBOOT_POOL_NAME:=zroot} # # Default options to use when creating zroot pool # : ${ZFSBOOT_POOL_CREATE_OPTIONS:=-O compress=lz4 -O atime=off} # # Default name for the boot environment parent dataset # : ${ZFSBOOT_BEROOT_NAME:=ROOT} # # Default name for the primany boot environment # : ${ZFSBOOT_BOOTFS_NAME:=default} # # Default Virtual Device (vdev) type to create # : ${ZFSBOOT_VDEV_TYPE:=stripe} # # Should we use sysctl(8) vfs.zfs.min_auto_ashift=12 to force 4K sectors? # : ${ZFSBOOT_FORCE_4K_SECTORS:=1} # # Should we use geli(8) to encrypt the drives? # NB: Automatically enables ZFSBOOT_BOOT_POOL # : ${ZFSBOOT_GELI_ENCRYPTION=} # # Default path to the geli(8) keyfile used in drive encryption # : ${ZFSBOOT_GELI_KEY_FILE:=/boot/encryption.key} # # Create a separate boot pool? # NB: Automatically set when using geli(8) or MBR # : ${ZFSBOOT_BOOT_POOL=} # # Options to use when creating separate boot pool (if any) # : ${ZFSBOOT_BOOT_POOL_CREATE_OPTIONS:=} # # Default name for boot pool when enabled (e.g., geli(8) or MBR) # : ${ZFSBOOT_BOOT_POOL_NAME:=bootpool} # # Default size for boot pool when enabled (e.g., geli(8) or MBR) # : ${ZFSBOOT_BOOT_POOL_SIZE:=2g} # # Default disks to use (always empty unless being scripted) # : ${ZFSBOOT_DISKS:=} # # Default partitioning scheme to use on disks # -: ${ZFSBOOT_PARTITION_SCHEME:=GPT} +: ${ZFSBOOT_PARTITION_SCHEME:=} # +# Default partitioning scheme to use on disks +# +: ${ZFSBOOT_BOOT_TYPE:=} + +# # How much swap to put on each block device in the boot zpool # NOTE: Value passed to gpart(8); which supports SI unit suffixes. # : ${ZFSBOOT_SWAP_SIZE:=2g} # # Should we use geli(8) to encrypt the swap? # : ${ZFSBOOT_SWAP_ENCRYPTION=} # # Should we use gmirror(8) to mirror the swap? # : ${ZFSBOOT_SWAP_MIRROR=} # # Default ZFS datasets for root zpool # # NOTE: Requires /tmp, /var/tmp, /$ZFSBOOT_BOOTFS_NAME/$ZFSBOOT_BOOTFS_NAME # NOTE: Anything after pound/hash character [#] is ignored as a comment. # f_isset ZFSBOOT_DATASETS || ZFSBOOT_DATASETS=" # DATASET OPTIONS (comma or space separated; or both) # Boot Environment [BE] root and default boot dataset /$ZFSBOOT_BEROOT_NAME mountpoint=none /$ZFSBOOT_BEROOT_NAME/$ZFSBOOT_BOOTFS_NAME mountpoint=/ # Compress /tmp, allow exec but not setuid /tmp mountpoint=/tmp,exec=on,setuid=off # Don't mount /usr so that 'base' files go to the BEROOT /usr mountpoint=/usr,canmount=off # Home directories separated so they are common to all BEs /usr/home # NB: /home is a symlink to /usr/home # Ports tree /usr/ports setuid=off # Source tree (compressed) /usr/src # Create /var and friends /var mountpoint=/var,canmount=off /var/audit exec=off,setuid=off /var/crash exec=off,setuid=off /var/log exec=off,setuid=off /var/mail atime=on /var/tmp setuid=off " # END-QUOTE # # If interactive and the user has not explicitly chosen a vdev type or disks, # make the user confirm scripted/default choices when proceeding to install. # : ${ZFSBOOT_CONFIRM_LAYOUT:=1} ############################################################ GLOBALS # # Format of a line in printf(1) syntax to add to fstab(5) # FSTAB_FMT="%s\t\t%s\t%s\t%s\t\t%s\t%s\n" # # Command strings for various tasks # CHMOD_MODE='chmod %s "%s"' DD_WITH_OPTIONS='dd if="%s" of="%s" %s' ECHO_APPEND='echo "%s" >> "%s"' GELI_ATTACH='geli attach -j - -k "%s" "%s"' GELI_DETACH_F='geli detach -f "%s"' GELI_PASSWORD_INIT='geli init -b -B "%s" -e %s -J - -K "%s" -l 256 -s 4096 "%s"' GPART_ADD_ALIGN='gpart add %s -t %s "%s"' GPART_ADD_ALIGN_INDEX='gpart add %s -i %s -t %s "%s"' GPART_ADD_ALIGN_INDEX_WITH_SIZE='gpart add %s -i %s -t %s -s %s "%s"' GPART_ADD_ALIGN_LABEL='gpart add %s -l %s -t %s "%s"' GPART_ADD_ALIGN_LABEL_WITH_SIZE='gpart add %s -l %s -t %s -s %s "%s"' GPART_BOOTCODE='gpart bootcode -b "%s" "%s"' GPART_BOOTCODE_PART='gpart bootcode -b "%s" -p "%s" -i %s "%s"' +GPART_BOOTCODE_PARTONLY='gpart bootcode -p "%s" -i %s "%s"' GPART_CREATE='gpart create -s %s "%s"' GPART_DESTROY_F='gpart destroy -F "%s"' GPART_SET_ACTIVE='gpart set -a active -i %s "%s"' GPART_SET_LENOVOFIX='gpart set -a lenovofix "%s"' GPART_SET_PMBR_ACTIVE='gpart set -a active "%s"' GRAID_DELETE='graid delete "%s"' LN_SF='ln -sf "%s" "%s"' MKDIR_P='mkdir -p "%s"' MOUNT_TYPE='mount -t %s "%s" "%s"' PRINTF_CONF="printf '%s=\"%%s\"\\\n' %s >> \"%s\"" PRINTF_FSTAB='printf "$FSTAB_FMT" "%s" "%s" "%s" "%s" "%s" "%s" >> "%s"' SHELL_TRUNCATE=':> "%s"' SWAP_GMIRROR_LABEL='gmirror label swap %s' SYSCTL_ZFS_MIN_ASHIFT_12='sysctl vfs.zfs.min_auto_ashift=12' UMOUNT='umount "%s"' ZFS_CREATE_WITH_OPTIONS='zfs create %s "%s"' ZFS_SET='zfs set "%s" "%s"' ZFS_UNMOUNT='zfs unmount "%s"' ZPOOL_CREATE_WITH_OPTIONS='zpool create %s "%s" %s %s' ZPOOL_DESTROY='zpool destroy "%s"' ZPOOL_EXPORT='zpool export "%s"' ZPOOL_IMPORT_WITH_OPTIONS='zpool import %s "%s"' ZPOOL_LABELCLEAR_F='zpool labelclear -f "%s"' ZPOOL_SET='zpool set %s "%s"' # # Strings that should be moved to an i18n file and loaded with f_include_lang() # hline_alnum_arrows_punc_tab_enter="Use alnum, arrows, punctuation, TAB or ENTER" hline_arrows_space_tab_enter="Use arrows, SPACE, TAB or ENTER" hline_arrows_tab_enter="Press arrows, TAB or ENTER" msg_an_unknown_error_occurred="An unknown error occurred" msg_back="Back" msg_cancel="Cancel" msg_change_selection="Change Selection" msg_configure_options="Configure Options:" msg_detailed_disk_info="gpart(8) show %s:\n%s\n\ncamcontrol(8) inquiry %s:\n%s\n\n\ncamcontrol(8) identify %s:\n%s\n" msg_disk_info="Disk Info" msg_disk_info_help="Get detailed information on disk device(s)" msg_disk_singular="disk" msg_disk_plural="disks" msg_encrypt_disks="Encrypt Disks?" msg_encrypt_disks_help="Use geli(8) to encrypt all data partitions" msg_error="Error" msg_force_4k_sectors="Force 4K Sectors?" msg_force_4k_sectors_help="Align partitions to 4K sector boundries and set vfs.zfs.min_auto_ashift=12" msg_freebsd_installer="FreeBSD Installer" msg_geli_password="Enter a strong passphrase, used to protect your encryption keys. You will be required to enter this passphrase each time the system is booted" msg_geli_setup="Initializing encryption on selected disks,\n this will take several seconds per disk" msg_install="Install" msg_install_desc="Proceed with Installation" msg_install_help="Create ZFS boot pool with displayed options" msg_invalid_boot_pool_size="Invalid boot pool size \`%s'" msg_invalid_disk_argument="Invalid disk argument \`%s'" msg_invalid_index_argument="Invalid index argument \`%s'" msg_invalid_swap_size="Invalid swap size \`%s'" msg_invalid_virtual_device_type="Invalid Virtual Device type \`%s'" msg_last_chance_are_you_sure="Last Chance! Are you sure you want to destroy\nthe current contents of the following disks:\n\n %s" msg_last_chance_are_you_sure_color='\\ZrLast Chance!\\ZR Are you \\Z1sure\\Zn you want to \\Zr\\Z1destroy\\Zn\nthe current contents of the following disks:\n\n %s' msg_mirror_desc="Mirror - n-Way Mirroring" msg_mirror_help="[2+ Disks] Mirroring provides the best performance, but the least storage" msg_missing_disk_arguments="missing disk arguments" msg_missing_one_or_more_scripted_disks="Missing one or more scripted disks!" msg_no="NO" msg_no_disks_present_to_configure="No disk(s) present to configure" msg_no_disks_selected="No disks selected." msg_not_enough_disks_selected="Not enough disks selected. (%u < %u minimum)" msg_null_disk_argument="NULL disk argument" msg_null_index_argument="NULL index argument" msg_null_poolname="NULL poolname" msg_ok="OK" msg_partition_scheme="Partition Scheme" msg_partition_scheme_help="Select partitioning scheme. GPT is recommended." msg_please_enter_a_name_for_your_zpool="Please enter a name for your zpool:" msg_please_enter_amount_of_swap_space="Please enter amount of swap space (SI-Unit suffixes\nrecommended; e.g., \`2g' for 2 Gigabytes):" msg_please_select_one_or_more_disks="Please select one or more disks to create a zpool:" msg_pool_name="Pool Name" msg_pool_name_cannot_be_empty="Pool name cannot be empty." msg_pool_name_help="Customize the name of the zpool to be created (Required)" msg_pool_type_disks="Pool Type/Disks:" msg_pool_type_disks_help="Choose type of ZFS Virtual Device and disks to use (Required)" msg_processing_selection="Processing selection..." msg_raidz1_desc="RAID-Z1 - Single Redundant RAID" msg_raidz1_help="[3+ Disks] Withstand failure of 1 disk. Recommended for: 3, 5 or 9 disks" msg_raidz2_desc="RAID-Z2 - Double Redundant RAID" msg_raidz2_help="[4+ Disks] Withstand failure of 2 disks. Recommended for: 4, 6 or 10 disks" msg_raidz3_desc="RAID-Z3 - Triple Redundant RAID" msg_raidz3_help="[5+ Disks] Withstand failure of 3 disks. Recommended for: 5, 7 or 11 disks" msg_rescan_devices="Rescan Devices" msg_rescan_devices_help="Scan for device changes" msg_select="Select" msg_select_a_disk_device="Select a disk device" msg_select_virtual_device_type="Select Virtual Device type:" msg_stripe_desc="Stripe - No Redundancy" msg_stripe_help="[1+ Disks] Striping provides maximum storage but no redundancy" msg_swap_encrypt="Encrypt Swap?" msg_swap_encrypt_help="Encrypt swap partitions with temporary keys, discarded on reboot" msg_swap_invalid="The selected swap size (%s) is invalid. Enter a number optionally followed by units. Example: 2G" msg_swap_mirror="Mirror Swap?" msg_swap_mirror_help="Mirror swap partitions for redundancy, breaks crash dumps" msg_swap_size="Swap Size" msg_swap_size_help="Customize how much swap space is allocated to each selected disk" msg_swap_toosmall="The selected swap size (%s) is to small. Please enter a value greater than 100MB or enter 0 for no swap" msg_these_disks_are_too_small="These disks are too small given the amount of requested\nswap (%s) and/or geli(8) (%s) partitions, which would\ntake 50%% or more of each of the following selected disk\ndevices (not recommended):\n\n %s\n\nRecommend changing partition size(s) and/or selecting a\ndifferent set of devices." -msg_uefi_not_supported="The FreeBSD UEFI loader does not currently support booting root-on-ZFS. Your system will need to boot in legacy (CSM) mode.\nDo you want to continue?" msg_unable_to_get_disk_capacity="Unable to get disk capacity of \`%s'" msg_unsupported_partition_scheme="%s is an unsupported partition scheme" msg_user_cancelled="User Cancelled." msg_yes="YES" msg_zfs_configuration="ZFS Configuration" ############################################################ FUNCTIONS # dialog_menu_main # # Display the dialog(1)-based application main menu. # dialog_menu_main() { local title="$DIALOG_TITLE" local btitle="$DIALOG_BACKTITLE" local prompt="$msg_configure_options" local force4k="$msg_no" local usegeli="$msg_no" local swapgeli="$msg_no" local swapmirror="$msg_no" [ "$ZFSBOOT_FORCE_4K_SECTORS" ] && force4k="$msg_yes" [ "$ZFSBOOT_GELI_ENCRYPTION" ] && usegeli="$msg_yes" [ "$ZFSBOOT_SWAP_ENCRYPTION" ] && swapgeli="$msg_yes" [ "$ZFSBOOT_SWAP_MIRROR" ] && swapmirror="$msg_yes" local disks n disks_grammar f_count n $ZFSBOOT_DISKS { [ $n -eq 1 ] && disks_grammar=$msg_disk_singular; } || disks_grammar=$msg_disk_plural # grammar local menu_list=" '>>> $msg_install' '$msg_install_desc' '$msg_install_help' 'T $msg_pool_type_disks' '$ZFSBOOT_VDEV_TYPE: $n $disks_grammar' '$msg_pool_type_disks_help' '- $msg_rescan_devices' '*' '$msg_rescan_devices_help' '- $msg_disk_info' '*' '$msg_disk_info_help' 'N $msg_pool_name' '$ZFSBOOT_POOL_NAME' '$msg_pool_name_help' '4 $msg_force_4k_sectors' '$force4k' '$msg_force_4k_sectors_help' 'E $msg_encrypt_disks' '$usegeli' '$msg_encrypt_disks_help' 'P $msg_partition_scheme' - '$ZFSBOOT_PARTITION_SCHEME' + '$ZFSBOOT_PARTITION_SCHEME ($ZFSBOOT_BOOT_TYPE)' '$msg_partition_scheme_help' 'S $msg_swap_size' '$ZFSBOOT_SWAP_SIZE' '$msg_swap_size_help' 'M $msg_swap_mirror' '$swapmirror' '$msg_swap_mirror_help' 'W $msg_swap_encrypt' '$swapgeli' '$msg_swap_encrypt_help' " # END-QUOTE local defaultitem= # Calculated below local hline="$hline_alnum_arrows_punc_tab_enter" local height width rows eval f_dialog_menu_with_help_size height width rows \ \"\$title\" \"\$btitle\" \"\$prompt\" \"\$hline\" $menu_list # Obtain default-item from previously stored selection f_dialog_default_fetch defaultitem local menu_choice menu_choice=$( eval $DIALOG \ --title \"\$title\" \ --backtitle \"\$btitle\" \ --hline \"\$hline\" \ --item-help \ --ok-label \"\$msg_select\" \ --cancel-label \"\$msg_cancel\" \ --default-item \"\$defaultitem\" \ --menu \"\$prompt\" \ $height $width $rows \ $menu_list \ 2>&1 >&$DIALOG_TERMINAL_PASSTHRU_FD ) local retval=$? f_dialog_data_sanitize menu_choice f_dialog_menutag_store "$menu_choice" # Only update default-item on success [ $retval -eq $DIALOG_OK ] && f_dialog_default_store "$menu_choice" return $retval } # dialog_last_chance $disks ... # # Display a list of the disks that the user is about to destroy. The default # action is to return error status unless the user explicitly (non-default) # selects "Yes" from the noyes dialog. # dialog_last_chance() { local title="$DIALOG_TITLE" local btitle="$DIALOG_BACKTITLE" local prompt # Calculated below local hline="$hline_arrows_tab_enter" local height=8 width=50 prefix=" " local plen=${#prefix} list= line= local max_width=$(( $width - 3 - $plen )) local yes no defaultno extra_args format if [ "$USE_XDIALOG" ]; then yes=ok no=cancel defaultno=default-no extra_args="--wrap --left" format="$msg_last_chance_are_you_sure" else yes=yes no=no defaultno=defaultno extra_args="--colors --cr-wrap" format="$msg_last_chance_are_you_sure_color" fi local disk line_width for disk in $*; do if [ "$line" ]; then line_width=${#line} else line_width=$plen fi line_width=$(( $line_width + 1 + ${#disk} )) # Add newline before disk if it would exceed max_width if [ $line_width -gt $max_width ]; then list="$list$line\n" line="$prefix" height=$(( $height + 1 )) fi # Add the disk to the list line="$line $disk" done # Append the left-overs if [ "${line#$prefix}" ]; then list="$list$line" height=$(( $height + 1 )) fi # Add height for Xdialog(1) [ "$USE_XDIALOG" ] && height=$(( $height + $height / 5 + 3 )) prompt=$( printf "$format" "$list" ) f_dprintf "%s: Last Chance!" "$0" $DIALOG \ --title "$title" \ --backtitle "$btitle" \ --hline "$hline" \ --$defaultno \ --$yes-label "$msg_yes" \ --$no-label "$msg_no" \ $extra_args \ --yesno "$prompt" $height $width } # dialog_menu_layout # # Configure Virtual Device type and disks to use for the ZFS boot pool. User # must select enough disks to satisfy the chosen vdev type. # dialog_menu_layout() { local funcname=dialog_menu_layout local title="$DIALOG_TITLE" local btitle="$DIALOG_BACKTITLE" local vdev_prompt="$msg_select_virtual_device_type" local disk_prompt="$msg_please_select_one_or_more_disks" local vdev_menu_list=" 'stripe' '$msg_stripe_desc' '$msg_stripe_help' 'mirror' '$msg_mirror_desc' '$msg_mirror_help' 'raidz1' '$msg_raidz1_desc' '$msg_raidz1_help' 'raidz2' '$msg_raidz2_desc' '$msg_raidz2_help' 'raidz3' '$msg_raidz3_desc' '$msg_raidz3_help' " # END-QUOTE local disk_check_list= # Calculated below local vdev_hline="$hline_arrows_tab_enter" local disk_hline="$hline_arrows_space_tab_enter" # Warn the user if vdev type is not valid case "$ZFSBOOT_VDEV_TYPE" in stripe|mirror|raidz1|raidz2|raidz3) : known good ;; *) f_dprintf "%s: Invalid virtual device type \`%s'" \ $funcname "$ZFSBOOT_VDEV_TYPE" f_show_err "$msg_invalid_virtual_device_type" \ "$ZFSBOOT_VDEV_TYPE" f_interactive || return $FAILURE esac # Calculate size of vdev menu once only local vheight vwidth vrows eval f_dialog_menu_with_help_size vheight vwidth vrows \ \"\$title\" \"\$btitle\" \"\$vdev_prompt\" \"\$vdev_hline\" \ $vdev_menu_list # Get a list of probed disk devices local disks= debug= f_device_find "" $DEVICE_TYPE_DISK disks # Prune out mounted md(4) devices that may be part of the boot process local disk name new_list= for disk in $disks; do debug= $disk get name name case "$name" in md[0-9]*) f_mounted -b "/dev/$name" && continue ;; esac new_list="$new_list $disk" done disks="${new_list# }" # Debugging if [ "$debug" ]; then local disk_names= for disk in $disks; do debug= $disk get name name disk_names="$disk_names $name" done f_dprintf "$funcname: disks=[%s]" "${disk_names# }" fi if [ ! "$disks" ]; then f_dprintf "No disk(s) present to configure" f_show_err "$msg_no_disks_present_to_configure" return $FAILURE fi # Lets sort the disks array to be more user friendly f_device_sort_by name disks disks # # Operate in a loop so we can (if interactive) repeat if not enough # disks are selected to satisfy the chosen vdev type or user wants to # back-up to the previous menu. # local vardisk ndisks onoff selections vdev_choice breakout device local valid_disks all_valid want_disks desc height width rows while :; do # # Confirm the vdev type that was selected # if f_interactive && [ "$ZFSBOOT_CONFIRM_LAYOUT" ]; then vdev_choice=$( eval $DIALOG \ --title \"\$title\" \ --backtitle \"\$btitle\" \ --hline \"\$vdev_hline\" \ --ok-label \"\$msg_ok\" \ --cancel-label \"\$msg_cancel\" \ --item-help \ --default-item \"\$ZFSBOOT_VDEV_TYPE\" \ --menu \"\$vdev_prompt\" \ $vheight $vwidth $vrows \ $vdev_menu_list \ 2>&1 >&$DIALOG_TERMINAL_PASSTHRU_FD ) || return $? # Exit if user pressed ESC or chose Cancel/No f_dialog_data_sanitize vdev_choice ZFSBOOT_VDEV_TYPE="$vdev_choice" f_dprintf "$funcname: ZFSBOOT_VDEV_TYPE=[%s]" \ "$ZFSBOOT_VDEV_TYPE" fi # Determine the number of disks needed for this vdev type want_disks=0 case "$ZFSBOOT_VDEV_TYPE" in stripe) want_disks=1 ;; mirror) want_disks=2 ;; raidz1) want_disks=3 ;; raidz2) want_disks=4 ;; raidz3) want_disks=5 ;; esac # # Warn the user if any scripted disks are invalid # valid_disks= all_valid=${ZFSBOOT_DISKS:+1} # optimism for disk in $ZFSBOOT_DISKS; do if debug= f_device_find -1 \ $disk $DEVICE_TYPE_DISK device then valid_disks="$valid_disks $disk" continue fi f_dprintf "$funcname: \`%s' is not a real disk" "$disk" all_valid= done if [ ! "$all_valid" ]; then if [ "$ZFSBOOT_DISKS" ]; then f_show_err \ "$msg_missing_one_or_more_scripted_disks" else f_dprintf "No disks selected." f_interactive || f_show_err "$msg_no_disks_selected" fi f_interactive || return $FAILURE fi ZFSBOOT_DISKS="${valid_disks# }" # # Short-circuit if we're running non-interactively # if ! f_interactive || [ ! "$ZFSBOOT_CONFIRM_LAYOUT" ]; then f_count ndisks $ZFSBOOT_DISKS [ $ndisks -ge $want_disks ] && break # to success # Not enough disks selected f_dprintf "$funcname: %s: %s (%u < %u minimum)" \ "$ZFSBOOT_VDEV_TYPE" \ "Not enough disks selected." \ $ndisks $want_disks f_interactive || return $FAILURE msg_yes="$msg_change_selection" msg_no="$msg_cancel" \ f_yesno "%s: $msg_not_enough_disks_selected" \ "$ZFSBOOT_VDEV_TYPE" $ndisks $want_disks || return $FAILURE fi # # Confirm the disks that were selected # Loop until the user cancels or selects enough disks # breakout= while :; do # Loop over list of available disks, resetting state for disk in $disks; do f_isset _${disk}_status && _${disk}_status= done # Loop over list of selected disks and create temporary # locals to map statuses onto up-to-date list of disks for disk in $ZFSBOOT_DISKS; do debug= f_device_find -1 \ $disk $DEVICE_TYPE_DISK disk f_isset _${disk}_status || local _${disk}_status _${disk}_status=on done # Create the checklist menu of discovered disk devices disk_check_list= for disk in $disks; do desc= $disk get name name $disk get desc desc f_shell_escape "$desc" desc f_getvar _${disk}_status:-off onoff disk_check_list="$disk_check_list $name '$desc' $onoff" done eval f_dialog_checklist_size height width rows \ \"\$title\" \"\$btitle\" \"\$prompt\" \ \"\$hline\" $disk_check_list selections=$( eval $DIALOG \ --title \"\$DIALOG_TITLE\" \ --backtitle \"\$DIALOG_BACKTITLE\" \ --separate-output \ --hline \"\$hline\" \ --ok-label \"\$msg_ok\" \ --cancel-label \"\$msg_back\" \ --checklist \"\$prompt\" \ $height $width $rows \ $disk_check_list \ 2>&1 >&$DIALOG_TERMINAL_PASSTHRU_FD ) || break # Loop if user pressed ESC or chose Cancel/No f_dialog_data_sanitize selections ZFSBOOT_DISKS="$selections" f_dprintf "$funcname: ZFSBOOT_DISKS=[%s]" \ "$ZFSBOOT_DISKS" f_count ndisks $ZFSBOOT_DISKS [ $ndisks -ge $want_disks ] && breakout=break && break # Not enough disks selected f_dprintf "$funcname: %s: %s (%u < %u minimum)" \ "$ZFSBOOT_VDEV_TYPE" \ "Not enough disks selected." \ $ndisks $want_disks msg_yes="$msg_change_selection" msg_no="$msg_cancel" \ f_yesno "%s: $msg_not_enough_disks_selected" \ "$ZFSBOOT_VDEV_TYPE" $ndisks $want_disks || break done [ "$breakout" = "break" ] && break [ "$ZFSBOOT_CONFIRM_LAYOUT" ] || return $FAILURE done return $DIALOG_OK } -# dialog_uefi_prompt -# -# Confirm that the user wants to continue with the installation on a BIOS -# system when they have booted with UEFI -# -dialog_uefi_prompt() -{ - local title="$DIALOG_TITLE" - local btitle="$DIALOG_BACKTITLE" - local prompt # Calculated below - local hline="$hline_arrows_tab_enter" - - local height=8 width=50 prefix=" " - local plen=${#prefix} list= line= - local max_width=$(( $width - 3 - $plen )) - - local yes no defaultno extra_args format - if [ "$USE_XDIALOG" ]; then - yes=ok no=cancel defaultno=default-no - extra_args="--wrap --left" - format="$msg_uefi_not_supported" - else - yes=yes no=no defaultno=defaultno - extra_args="--cr-wrap" - format="$msg_uefi_not_supported" - fi - - # Add height for Xdialog(1) - [ "$USE_XDIALOG" ] && height=$(( $height + $height / 5 + 3 )) - - prompt=$( printf "$format" ) - f_dprintf "%s: UEFI prompt" "$0" - $DIALOG \ - --title "$title" \ - --backtitle "$btitle" \ - --hline "$hline" \ - --$yes-label "$msg_yes" \ - --$no-label "$msg_no" \ - $extra_args \ - --yesno "$prompt" $height $width -} - # zfs_create_diskpart $disk $index # # For each block device to be used in the zpool, rather than just create the # zpool with the raw block devices (e.g., da0, da1, etc.) we create partitions # so we can have some real swap. This also provides wiggle room incase your # replacement drivers do not have the exact same sector counts. # # NOTE: $swapsize and $bootsize should be defined by the calling function. # NOTE: Sets $bootpart and $targetpart for the calling function. # zfs_create_diskpart() { local funcname=zfs_create_diskpart local disk="$1" index="$2" # Check arguments if [ ! "$disk" ]; then f_dprintf "$funcname: NULL disk argument" msg_error="$msg_error: $funcname" \ f_show_err "$msg_null_disk_argument" return $FAILURE fi if [ "${disk#*[$IFS]}" != "$disk" ]; then f_dprintf "$funcname: Invalid disk argument \`%s'" "$disk" msg_error="$msg_error: $funcname" \ f_show_err "$msg_invalid_disk_argument" "$disk" return $FAILURE fi if [ ! "$index" ]; then f_dprintf "$funcname: NULL index argument" msg_error="$msg_error: $funcname" \ f_show_err "$msg_null_index_argument" return $FAILURE fi if ! f_isinteger "$index"; then f_dprintf "$funcname: Invalid index argument \`%s'" "$index" msg_error="$msg_error: $funcname" \ f_show_err "$msg_invalid_index_argument" "$index" return $FAILURE fi f_dprintf "$funcname: disk=[%s] index=[%s]" "$disk" "$index" # Check for unknown partition scheme before proceeding further case "$ZFSBOOT_PARTITION_SCHEME" in ""|MBR|GPT*) : known good ;; *) f_dprintf "$funcname: %s is an unsupported partition scheme" \ "$ZFSBOOT_PARTITION_SCHEME" msg_error="$msg_error: $funcname" f_show_err \ "$msg_unsupported_partition_scheme" \ "$ZFSBOOT_PARTITION_SCHEME" return $FAILURE esac # # Destroy whatever partition layout is currently on disk. # NOTE: `-F' required to destroy if partitions still exist. # NOTE: Failure is ok here, blank disk will have nothing to destroy. # f_dprintf "$funcname: Destroying all data/layouts on \`%s'..." "$disk" f_eval_catch -d $funcname gpart "$GPART_DESTROY_F" $disk f_eval_catch -d $funcname graid "$GRAID_DELETE" $disk f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" /dev/$disk # Make doubly-sure backup GPT is destroyed f_eval_catch -d $funcname gpart "$GPART_CREATE" gpt $disk f_eval_catch -d $funcname gpart "$GPART_DESTROY_F" $disk # # Enable boot pool if encryption is desired # [ "$ZFSBOOT_GELI_ENCRYPTION" ] && ZFSBOOT_BOOT_POOL=1 # # Lay down the desired type of partition scheme # local setsize mbrindex align_small align_big # # If user has requested 4 K alignment, add these params to the # gpart add calls. With GPT, we align large partitions to 1 M for # improved performance on SSDs. MBR does not always play well with gaps # between partitions, so all alignment is only 4k for that case. # With MBR, we align the BSD partition that contains the MBR, otherwise # the system fails to boot. # if [ "$ZFSBOOT_FORCE_4K_SECTORS" ]; then align_small="-a 4k" align_big="-a 1m" sysctl kern.geom.part.mbr.enforce_chs=0 fi case "$ZFSBOOT_PARTITION_SCHEME" in ""|GPT*) f_dprintf "$funcname: Creating GPT layout..." # # 1. Create GPT layout using labels # f_eval_catch $funcname gpart "$GPART_CREATE" gpt $disk || return $FAILURE # # Apply workarounds if requested by the user # if [ "$ZFSBOOT_PARTITION_SCHEME" = "GPT + Lenovo Fix" ]; then f_eval_catch $funcname gpart "$GPART_SET_LENOVOFIX" \ $disk || return $FAILURE elif [ "$ZFSBOOT_PARTITION_SCHEME" = "GPT + Active" ]; then f_eval_catch $funcname gpart "$GPART_SET_PMBR_ACTIVE" \ $disk || return $FAILURE fi # - # 2. Add small freebsd-boot partition labeled `boot#' + # 2. Add small freebsd-boot or efi partition # - f_eval_catch $funcname gpart "$GPART_ADD_ALIGN_LABEL_WITH_SIZE" \ - "$align_small" gptboot$index freebsd-boot 512k $disk || - return $FAILURE - f_eval_catch $funcname gpart "$GPART_BOOTCODE_PART" \ - /boot/pmbr /boot/gptzfsboot 1 $disk || - return $FAILURE + if [ "$ZFSBOOT_BOOT_TYPE" = "UEFI" ]; then + f_eval_catch $funcname gpart \ + "$GPART_ADD_ALIGN_LABEL_WITH_SIZE" \ + "$align_small" efiboot$index efi 800k $disk || + return $FAILURE + f_eval_catch $funcname gpart "$GPART_BOOTCODE_PARTONLY" \ + /boot/boot1.efifat 1 $disk || + return $FAILURE + else + f_eval_catch $funcname gpart \ + "$GPART_ADD_ALIGN_LABEL_WITH_SIZE" \ + "$align_small" gptboot$index freebsd-boot \ + 512k $disk || return $FAILURE + f_eval_catch $funcname gpart "$GPART_BOOTCODE_PART" \ + /boot/pmbr /boot/gptzfsboot 1 $disk || + return $FAILURE + fi # NB: zpool will use the `zfs#' GPT labels bootpart=p2 swappart=p2 targetpart=p2 [ ${swapsize:-0} -gt 0 ] && targetpart=p3 # # Prepare boot pool if enabled (e.g., for geli(8)) # if [ "$ZFSBOOT_BOOT_POOL" ]; then bootpart=p2 swappart=p3 targetpart=p3 [ ${swapsize:-0} -gt 0 ] && targetpart=p4 f_eval_catch $funcname gpart \ "$GPART_ADD_ALIGN_LABEL_WITH_SIZE" \ "$align_big" boot$index freebsd-zfs \ ${bootsize}b $disk || return $FAILURE # Pedantically nuke any old labels f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" \ /dev/$disk$bootpart if [ "$ZFSBOOT_GELI_ENCRYPTION" ]; then # Pedantically detach targetpart for later f_eval_catch -d $funcname geli \ "$GELI_DETACH_F" \ /dev/$disk$targetpart fi fi # # 3. Add freebsd-swap partition labeled `swap#' # if [ ${swapsize:-0} -gt 0 ]; then f_eval_catch $funcname gpart \ "$GPART_ADD_ALIGN_LABEL_WITH_SIZE" \ "$align_big" swap$index freebsd-swap \ ${swapsize}b $disk || return $FAILURE # Pedantically nuke any old labels on the swap f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" \ /dev/$disk$swappart fi # # 4. Add freebsd-zfs partition labeled `zfs#' for zroot # f_eval_catch $funcname gpart "$GPART_ADD_ALIGN_LABEL" \ "$align_big" zfs$index freebsd-zfs $disk || return $FAILURE f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" \ /dev/$disk$targetpart ;; MBR) f_dprintf "$funcname: Creating MBR layout..." # # 1. Create MBR layout (no labels) # f_eval_catch $funcname gpart "$GPART_CREATE" mbr $disk || return $FAILURE f_eval_catch $funcname gpart "$GPART_BOOTCODE" /boot/mbr \ $disk || return $FAILURE # # 2. Add freebsd slice with all available space # f_eval_catch $funcname gpart "$GPART_ADD_ALIGN" "$align_small" \ freebsd $disk || return $FAILURE f_eval_catch $funcname gpart "$GPART_SET_ACTIVE" 1 $disk || return $FAILURE # Pedantically nuke any old labels f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" \ /dev/${disk}s1 # Pedantically nuke any old scheme f_eval_catch -d $funcname gpart "$GPART_DESTROY_F" ${disk}s1 # # 3. Write BSD scheme to the freebsd slice # f_eval_catch $funcname gpart "$GPART_CREATE" BSD ${disk}s1 || return $FAILURE # NB: zpool will use s1a (no labels) bootpart=s1a swappart=s1b targetpart=s1d mbrindex=4 # # Always prepare a boot pool on MBR # Do not align this partition, there must not be a gap # ZFSBOOT_BOOT_POOL=1 f_eval_catch $funcname gpart \ "$GPART_ADD_ALIGN_INDEX_WITH_SIZE" \ "" 1 freebsd-zfs ${bootsize}b ${disk}s1 || return $FAILURE # Pedantically nuke any old labels f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" \ /dev/$disk$bootpart if [ "$ZFSBOOT_GELI_ENCRYPTION" ]; then # Pedantically detach targetpart for later f_eval_catch -d $funcname geli \ "$GELI_DETACH_F" \ /dev/$disk$targetpart fi # # 4. Add freebsd-swap partition # if [ ${swapsize:-0} -gt 0 ]; then f_eval_catch $funcname gpart \ "$GPART_ADD_ALIGN_INDEX_WITH_SIZE" \ "$align_small" 2 freebsd-swap ${swapsize}b ${disk}s1 || return $FAILURE # Pedantically nuke any old labels on the swap f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" \ /dev/${disk}s1b fi # # 5. Add freebsd-zfs partition for zroot # f_eval_catch $funcname gpart "$GPART_ADD_ALIGN_INDEX" \ "$align_small" $mbrindex freebsd-zfs ${disk}s1 || return $FAILURE f_eval_catch -d $funcname zpool "$ZPOOL_LABELCLEAR_F" \ /dev/$disk$targetpart # Pedantic f_eval_catch $funcname dd "$DD_WITH_OPTIONS" \ /boot/zfsboot /dev/${disk}s1 count=1 || return $FAILURE ;; esac # $ZFSBOOT_PARTITION_SCHEME # Update fstab(5) local swapsize f_expand_number "$ZFSBOOT_SWAP_SIZE" swapsize if [ "$isswapmirror" ]; then # This is not the first disk in the mirror, do nothing elif [ "$ZFSBOOT_SWAP_ENCRYPTION" -a "$ZFSBOOT_SWAP_MIRROR" ]; then f_eval_catch $funcname printf "$PRINTF_FSTAB" \ /dev/mirror/swap.eli none swap sw 0 0 \ $BSDINSTALL_TMPETC/fstab || return $FAILURE isswapmirror=1 elif [ "$ZFSBOOT_SWAP_MIRROR" ]; then f_eval_catch $funcname printf "$PRINTF_FSTAB" \ /dev/mirror/swap none swap sw 0 0 \ $BSDINSTALL_TMPETC/fstab || return $FAILURE isswapmirror=1 elif [ "$ZFSBOOT_SWAP_ENCRYPTION" ]; then f_eval_catch $funcname printf "$PRINTF_FSTAB" \ /dev/$disk${swappart}.eli none swap sw 0 0 \ $BSDINSTALL_TMPETC/fstab || return $FAILURE elif [ ${swapsize:-0} -eq 0 ]; then # If swap is 0 sized, don't add it to fstab else f_eval_catch $funcname printf "$PRINTF_FSTAB" \ /dev/$disk$swappart none swap sw 0 0 \ $BSDINSTALL_TMPETC/fstab || return $FAILURE fi return $SUCCESS } # zfs_create_boot $poolname $vdev_type $disks ... # # Creates boot pool and dataset layout. Returns error if something goes wrong. # Errors are printed to stderr for collection and display. # zfs_create_boot() { local funcname=zfs_create_boot local zroot_name="$1" local zroot_vdevtype="$2" local zroot_vdevs= # Calculated below local swap_devs= # Calculated below local boot_vdevs= # Used for geli(8) and/or MBR layouts shift 2 # poolname vdev_type local disks="$*" disk local isswapmirror local bootpart targetpart swappart # Set by zfs_create_diskpart() below local create_options # # Pedantic checks; should never be seen # if [ ! "$zroot_name" ]; then f_dprintf "$funcname: NULL poolname" msg_error="$msg_error: $funcname" \ f_show_err "$msg_null_poolname" return $FAILURE fi if [ $# -lt 1 ]; then f_dprintf "$funcname: missing disk arguments" msg_error="$msg_error: $funcname" \ f_show_err "$msg_missing_disk_arguments" return $FAILURE fi f_dprintf "$funcname: poolname=[%s] vdev_type=[%s]" \ "$zroot_name" "$zroot_vdevtype" # # Initialize fstab(5) # f_dprintf "$funcname: Initializing temporary fstab(5) file..." f_eval_catch $funcname sh "$SHELL_TRUNCATE" $BSDINSTALL_TMPETC/fstab || return $FAILURE f_eval_catch $funcname printf "$PRINTF_FSTAB" \ "# Device" Mountpoint FStype Options Dump "Pass#" \ $BSDINSTALL_TMPETC/fstab || return $FAILURE # # Expand SI units in desired sizes # f_dprintf "$funcname: Expanding supplied size values..." local swapsize bootsize if ! f_expand_number "$ZFSBOOT_SWAP_SIZE" swapsize; then f_dprintf "$funcname: Invalid swap size \`%s'" \ "$ZFSBOOT_SWAP_SIZE" f_show_err "$msg_invalid_swap_size" "$ZFSBOOT_SWAP_SIZE" return $FAILURE fi if ! f_expand_number "$ZFSBOOT_BOOT_POOL_SIZE" bootsize; then f_dprintf "$funcname: Invalid boot pool size \`%s'" \ "$ZFSBOOT_BOOT_POOL_SIZE" f_show_err "$msg_invalid_boot_pool_size" \ "$ZFSBOOT_BOOT_POOL_SIZE" return $FAILURE fi f_dprintf "$funcname: ZFSBOOT_SWAP_SIZE=[%s] swapsize=[%s]" \ "$ZFSBOOT_SWAP_SIZE" "$swapsize" f_dprintf "$funcname: ZFSBOOT_BOOT_POOL_SIZE=[%s] bootsize=[%s]" \ "$ZFSBOOT_BOOT_POOL_SIZE" "$bootsize" # # Destroy the pool in-case this is our second time 'round (case of # failure and installer presented ``Retry'' option to come back). # # NB: If we don't destroy the pool, later gpart(8) destroy commands # that try to clear existing partitions (see zfs_create_diskpart()) # will fail with a `Device Busy' error, leading to `GEOM exists'. # f_eval_catch -d $funcname zpool "$ZPOOL_DESTROY" "$zroot_name" # # Prepare the disks and build pool device list(s) # f_dprintf "$funcname: Preparing disk partitions for ZFS pool..." # Force 4K sectors using vfs.zfs.min_auto_ashift=12 if [ "$ZFSBOOT_FORCE_4K_SECTORS" ]; then f_dprintf "$funcname: With 4K sectors..." f_eval_catch $funcname sysctl "$SYSCTL_ZFS_MIN_ASHIFT_12" \ || return $FAILURE fi local n=0 for disk in $disks; do zfs_create_diskpart $disk $n || return $FAILURE # Now $bootpart, $targetpart, and $swappart are set (suffix # for $disk) if [ "$ZFSBOOT_BOOT_POOL" ]; then boot_vdevs="$boot_vdevs $disk$bootpart" fi zroot_vdevs="$zroot_vdevs $disk$targetpart" if [ "$ZFSBOOT_GELI_ENCRYPTION" ]; then zroot_vdevs="$zroot_vdevs.eli" fi n=$(( $n + 1 )) done # disks # # If we need/want a boot pool, create it # if [ "$ZFSBOOT_BOOT_POOL" ]; then local bootpool_vdevtype= # Calculated below local bootpool_options= # Calculated below local bootpool_name="$ZFSBOOT_BOOT_POOL_NAME" local bootpool="$BSDINSTALL_CHROOT/$bootpool_name" local zroot_key="${ZFSBOOT_GELI_KEY_FILE#/}" f_dprintf "$funcname: Setting up boot pool..." [ "$ZFSBOOT_GELI_ENCRYPTION" ] && f_dprintf "$funcname: For encrypted root disk..." # Create parent directory for boot pool f_eval_catch -d $funcname umount "$UMOUNT" /mnt f_eval_catch $funcname mount "$MOUNT_TYPE" tmpfs none \ $BSDINSTALL_CHROOT || return $FAILURE # Create mirror across the boot partition on all disks local nvdevs f_count nvdevs $boot_vdevs [ $nvdevs -gt 1 ] && bootpool_vdevtype=mirror create_options="$ZFSBOOT_BOOT_POOL_CREATE_OPTIONS" bootpool_options="-o altroot=$BSDINSTALL_CHROOT" bootpool_options="$bootpool_options $create_options" bootpool_options="$bootpool_options -m \"/$bootpool_name\" -f" f_eval_catch $funcname zpool "$ZPOOL_CREATE_WITH_OPTIONS" \ "$bootpool_options" "$bootpool_name" \ "$bootpool_vdevtype" "$boot_vdevs" || return $FAILURE f_eval_catch $funcname mkdir "$MKDIR_P" "$bootpool/boot" || return $FAILURE if [ "$ZFSBOOT_GELI_ENCRYPTION" ]; then # Generate an encryption key using random(4) f_eval_catch $funcname dd "$DD_WITH_OPTIONS" \ /dev/random "$bootpool/$zroot_key" \ "bs=4096 count=1" || return $FAILURE f_eval_catch $funcname chmod "$CHMOD_MODE" \ go-wrx "$bootpool/$zroot_key" || return $FAILURE else # Clean up f_eval_catch $funcname zfs "$ZFS_UNMOUNT" \ "$bootpool_name" || return $FAILURE f_eval_catch -d $funcname umount "$UMOUNT" /mnt # tmpfs fi fi # # Create the geli(8) GEOMS # if [ "$ZFSBOOT_GELI_ENCRYPTION" ]; then # Prompt user for password (twice) if ! msg_enter_new_password="$msg_geli_password" \ f_dialog_input_password then f_dprintf "$funcname: User cancelled" f_show_err "$msg_user_cancelled" return $FAILURE fi # Initialize geli(8) on each of the target partitions for disk in $disks; do f_dialog_info "$msg_geli_setup" \ 2>&1 >&$DIALOG_TERMINAL_PASSTHRU_FD if ! echo "$pw_password" | f_eval_catch \ $funcname geli "$GELI_PASSWORD_INIT" \ "$bootpool/boot/$disk$targetpart.eli" \ AES-XTS "$bootpool/$zroot_key" \ $disk$targetpart then f_interactive || f_die unset pw_password # Sensitive info return $FAILURE fi if ! echo "$pw_password" | f_eval_catch \ $funcname geli "$GELI_ATTACH" \ "$bootpool/$zroot_key" $disk$targetpart then f_interactive || f_die unset pw_password # Sensitive info return $FAILURE fi done unset pw_password # Sensitive info # Clean up f_eval_catch $funcname zfs "$ZFS_UNMOUNT" "$bootpool_name" || return $FAILURE f_eval_catch -d $funcname umount "$UMOUNT" /mnt # tmpfs fi # # Create the gmirror(8) GEOMS for swap # if [ "$ZFSBOOT_SWAP_MIRROR" ]; then for disk in $disks; do swap_devs="$swap_devs $disk$swappart" done f_eval_catch $funcname gmirror "$SWAP_GMIRROR_LABEL" \ "$swap_devs" || return $FAILURE fi # # Create the ZFS root pool with desired type and disk devices # f_dprintf "$funcname: Creating root pool..." create_options="$ZFSBOOT_POOL_CREATE_OPTIONS" f_eval_catch $funcname zpool "$ZPOOL_CREATE_WITH_OPTIONS" \ "-o altroot=$BSDINSTALL_CHROOT $create_options -m none -f" \ "$zroot_name" "$zroot_vdevtype" "$zroot_vdevs" || return $FAILURE # # Create ZFS dataset layout within the new root pool # f_dprintf "$funcname: Creating ZFS datasets..." echo "$ZFSBOOT_DATASETS" | while read dataset options; do # Skip blank lines and comments case "$dataset" in "#"*|"") continue; esac # Remove potential inline comments in options options="${options%%#*}" # Replace tabs with spaces f_replaceall "$options" " " " " options # Reduce contiguous runs of space to one single space oldoptions= while [ "$oldoptions" != "$options" ]; do oldoptions="$options" f_replaceall "$options" " " " " options done # Replace both commas and spaces with ` -o ' f_replaceall "$options" "[ ,]" " -o " options # Create the dataset with desired options f_eval_catch $funcname zfs "$ZFS_CREATE_WITH_OPTIONS" \ "${options:+-o $options}" "$zroot_name$dataset" || return $FAILURE done # # Set a mountpoint for the root of the pool so newly created datasets # have a mountpoint to inherit # f_dprintf "$funcname: Setting mountpoint for root of the pool..." f_eval_catch $funcname zfs "$ZFS_SET" \ "mountpoint=/$zroot_name" "$zroot_name" || return $FAILURE # Touch up permissions on the tmp directories f_dprintf "$funcname: Modifying directory permissions..." local dir for dir in /tmp /var/tmp; do f_eval_catch $funcname chmod "$CHMOD_MODE" 1777 \ $BSDINSTALL_CHROOT$dir || return $FAILURE done # Create symlink(s) if [ "$ZFSBOOT_BOOT_POOL" ]; then f_dprintf "$funcname: Creating /boot symlink for boot pool..." f_eval_catch $funcname ln "$LN_SF" "$bootpool_name/boot" \ $BSDINSTALL_CHROOT/boot || return $FAILURE fi # Set bootfs property local zroot_bootfs="$ZFSBOOT_BEROOT_NAME/$ZFSBOOT_BOOTFS_NAME" f_dprintf "$funcname: Setting bootfs property..." f_eval_catch $funcname zpool "$ZPOOL_SET" \ "bootfs=\"$zroot_name/$zroot_bootfs\"" "$zroot_name" || return $FAILURE # Export the pool(s) f_dprintf "$funcname: Temporarily exporting ZFS pool(s)..." f_eval_catch $funcname zpool "$ZPOOL_EXPORT" "$zroot_name" || return $FAILURE if [ "$ZFSBOOT_BOOT_POOL" ]; then f_eval_catch $funcname zpool "$ZPOOL_EXPORT" \ "$bootpool_name" || return $FAILURE fi # MBR boot loader touch-up if [ "$ZFSBOOT_PARTITION_SCHEME" = "MBR" ]; then f_dprintf "$funcname: Updating MBR boot loader on disks..." # Stick the ZFS boot loader in the "convienient hole" after # the ZFS internal metadata for disk in $disks; do f_eval_catch $funcname dd "$DD_WITH_OPTIONS" \ /boot/zfsboot /dev/$disk$bootpart \ "skip=1 seek=1024" || return $FAILURE done fi # Re-import the ZFS pool(s) f_dprintf "$funcname: Re-importing ZFS pool(s)..." f_eval_catch $funcname zpool "$ZPOOL_IMPORT_WITH_OPTIONS" \ "-o altroot=\"$BSDINSTALL_CHROOT\"" "$zroot_name" || return $FAILURE if [ "$ZFSBOOT_BOOT_POOL" ]; then f_eval_catch $funcname zpool "$ZPOOL_IMPORT_WITH_OPTIONS" \ "-o altroot=\"$BSDINSTALL_CHROOT\"" \ "$bootpool_name" || return $FAILURE fi # While this is apparently not needed, it seems to help MBR f_dprintf "$funcname: Configuring zpool.cache for zroot..." f_eval_catch $funcname mkdir "$MKDIR_P" $BSDINSTALL_CHROOT/boot/zfs || return $FAILURE f_eval_catch $funcname zpool "$ZPOOL_SET" \ "cachefile=\"$BSDINSTALL_CHROOT/boot/zfs/zpool.cache\"" \ "$zroot_name" || return $FAILURE # Last, but not least... required lines for rc.conf(5)/loader.conf(5) # NOTE: We later concatenate these into their destination f_dprintf "%s: Configuring rc.conf(5)/loader.conf(5) additions..." \ "$funcname" f_eval_catch $funcname echo "$ECHO_APPEND" 'zfs_enable=\"YES\"' \ $BSDINSTALL_TMPETC/rc.conf.zfs || return $FAILURE f_eval_catch $funcname echo "$ECHO_APPEND" \ 'kern.geom.label.disk_ident.enable=\"0\"' \ $BSDINSTALL_TMPBOOT/loader.conf.zfs || return $FAILURE f_eval_catch $funcname echo "$ECHO_APPEND" \ 'kern.geom.label.gptid.enable=\"0\"' \ $BSDINSTALL_TMPBOOT/loader.conf.zfs || return $FAILURE if [ "$ZFSBOOT_SWAP_MIRROR" ]; then f_eval_catch $funcname echo "$ECHO_APPEND" \ 'geom_mirror_load=\"YES\"' \ $BSDINSTALL_TMPBOOT/loader.conf.gmirror || return $FAILURE fi # We're all done unless we should go on for boot pool [ "$ZFSBOOT_BOOT_POOL" ] || return $SUCCESS # Set cachefile for boot pool so it auto-imports at system start f_dprintf "$funcname: Configuring zpool.cache for boot pool..." f_eval_catch $funcname zpool "$ZPOOL_SET" \ "cachefile=\"$BSDINSTALL_CHROOT/boot/zfs/zpool.cache\"" \ "$bootpool_name" || return $FAILURE # Some additional geli(8) requirements for loader.conf(5) for option in \ 'zpool_cache_load=\"YES\"' \ 'zpool_cache_type=\"/boot/zfs/zpool.cache\"' \ 'zpool_cache_name=\"/boot/zfs/zpool.cache\"' \ ; do f_eval_catch $funcname echo "$ECHO_APPEND" "$option" \ $BSDINSTALL_TMPBOOT/loader.conf.zfs || return $FAILURE done f_eval_catch $funcname printf "$PRINTF_CONF" vfs.root.mountfrom \ "\"zfs:$zroot_name/$zroot_bootfs\"" \ $BSDINSTALL_TMPBOOT/loader.conf.root || return $FAILURE # We're all done unless we should go on to do encryption [ "$ZFSBOOT_GELI_ENCRYPTION" ] || return $SUCCESS # # Configure geli(8)-based encryption # f_dprintf "$funcname: Configuring disk encryption..." f_eval_catch $funcname echo "$ECHO_APPEND" 'aesni_load=\"YES\"' \ $BSDINSTALL_TMPBOOT/loader.conf.aesni || return $FAILURE f_eval_catch $funcname echo "$ECHO_APPEND" 'geom_eli_load=\"YES\"' \ $BSDINSTALL_TMPBOOT/loader.conf.geli || return $FAILURE f_eval_catch $funcname echo "$ECHO_APPEND" \ 'geom_eli_passphrase_prompt=\"YES\"' \ $BSDINSTALL_TMPBOOT/loader.conf.geli || return $FAILURE for disk in $disks; do f_eval_catch $funcname printf "$PRINTF_CONF" \ geli_%s_keyfile0_load "$disk$targetpart YES" \ $BSDINSTALL_TMPBOOT/loader.conf.$disk$targetpart || return $FAILURE f_eval_catch $funcname printf "$PRINTF_CONF" \ geli_%s_keyfile0_type \ "$disk$targetpart $disk$targetpart:geli_keyfile0" \ $BSDINSTALL_TMPBOOT/loader.conf.$disk$targetpart || return $FAILURE f_eval_catch $funcname printf "$PRINTF_CONF" \ geli_%s_keyfile0_name \ "$disk$targetpart \"$ZFSBOOT_GELI_KEY_FILE\"" \ $BSDINSTALL_TMPBOOT/loader.conf.$disk$targetpart || return $FAILURE done return $SUCCESS } # dialog_menu_diskinfo # # Prompt the user to select a disk and then provide detailed info on it. # dialog_menu_diskinfo() { local device disk # # Break from loop when user cancels disk selection # while :; do device=$( msg_cancel="$msg_back" f_device_menu \ "$DIALOG_TITLE" "$msg_select_a_disk_device" "" \ $DEVICE_TYPE_DISK 2>&1 ) || break $device get name disk # Show gpart(8) `show' and camcontrol(8) `inquiry' data f_show_msg "$msg_detailed_disk_info" \ "$disk" "$( gpart show $disk 2> /dev/null )" \ "$disk" "$( camcontrol inquiry $disk 2> /dev/null )" \ "$disk" "$( camcontrol identify $disk 2> /dev/null )" done return $SUCCESS } ############################################################ MAIN # # Initialize # f_dialog_title "$msg_zfs_configuration" f_dialog_backtitle "$msg_freebsd_installer" # User may have specifically requested ZFS-related operations be interactive ! f_interactive && f_zfsinteractive && unset $VAR_NONINTERACTIVE # # Debugging # f_dprintf "BSDINSTALL_CHROOT=[%s]" "$BSDINSTALL_CHROOT" f_dprintf "BSDINSTALL_TMPETC=[%s]" "$BSDINSTALL_TMPETC" f_dprintf "FSTAB_FMT=[%s]" "$FSTAB_FMT" # -# If the system was booted with UEFI, warn the user that FreeBSD can't do -# ZFS with UEFI yet +# If the system was booted with UEFI, set the default boot type to UEFI # -if f_interactive; then - bootmethod=$( sysctl -n machdep.bootmethod ) - f_dprintf "machdep.bootmethod=[%s]" "$bootmethod" - if [ "$bootmethod" != "BIOS" ]; then - dialog_uefi_prompt - retval=$? - f_dprintf "uefi_prompt=[%s]" "$retval" - [ $retval -eq $DIALOG_OK ] || f_die - fi +bootmethod=$( sysctl -n machdep.bootmethod ) +f_dprintf "machdep.bootmethod=[%s]" "$bootmethod" +if [ "$bootmethod" = "UEFI" ]; then + : ${ZFSBOOT_BOOT_TYPE:=UEFI} + : ${ZFSBOOT_PARTITION_SCHEME:=GPT} +else + : ${ZFSBOOT_BOOT_TYPE:=BIOS} + : ${ZFSBOOT_PARTITION_SCHEME:=GPT} fi # # Loop over the main menu until we've accomplished what we came here to do # while :; do if ! f_interactive; then retval=$DIALOG_OK mtag=">>> $msg_install" else dialog_menu_main retval=$? f_dialog_menutag_fetch mtag fi f_dprintf "retval=%u mtag=[%s]" $retval "$mtag" [ $retval -eq $DIALOG_OK ] || f_die case "$mtag" in ">>> $msg_install") # # First, validate the user's selections # # Make sure they gave us a name for the pool if [ ! "$ZFSBOOT_POOL_NAME" ]; then f_dprintf "Pool name cannot be empty." f_show_err "$msg_pool_name_cannot_be_empty" continue fi # Validate vdev type against number of disks selected/scripted # (also validates that ZFSBOOT_DISKS are real [probed] disks) # NB: dialog_menu_layout supports running non-interactively dialog_menu_layout || continue # Make sure each disk will be at least 50% ZFS if f_expand_number "$ZFSBOOT_SWAP_SIZE" swapsize && f_expand_number "$ZFSBOOT_BOOT_POOL_SIZE" bootsize then minsize=$swapsize teeny_disks= [ "$ZFSBOOT_BOOT_POOL" ] && minsize=$(( $minsize + $bootsize )) for disk in $ZFSBOOT_DISKS; do debug= f_device_find -1 \ $disk $DEVICE_TYPE_DISK device $device get capacity disksize || continue [ ${disksize:-0} -ge 0 ] || disksize=0 disksize=$(( $disksize - $minsize )) [ $disksize -lt $minsize ] && teeny_disks="$teeny_disks $disk" done if [ "$teeny_disks" ]; then f_dprintf "swapsize=[%s] bootsize[%s] %s" \ "$ZFSBOOT_SWAP_SIZE" \ "$ZFSBOOT_BOOT_POOL_SIZE" \ "minsize=[$minsize]" f_dprintf "These disks are too small: %s" \ "$teeny_disks" f_show_err "$msg_these_disks_are_too_small" \ "$ZFSBOOT_SWAP_SIZE" \ "$ZFSBOOT_BOOT_POOL_SIZE" \ "$teeny_disks" continue fi fi # # Last Chance! # if f_interactive; then dialog_last_chance $ZFSBOOT_DISKS || continue fi # # Let's do this # vdev_type="$ZFSBOOT_VDEV_TYPE" # Blank the vdev type for the default layout [ "$vdev_type" = "stripe" ] && vdev_type= zfs_create_boot "$ZFSBOOT_POOL_NAME" \ "$vdev_type" $ZFSBOOT_DISKS || continue break # to success ;; ?" $msg_pool_type_disks") ZFSBOOT_CONFIRM_LAYOUT=1 dialog_menu_layout # User has poked settings, disable later confirmation ZFSBOOT_CONFIRM_LAYOUT= ;; "- $msg_rescan_devices") f_device_rescan ;; "- $msg_disk_info") dialog_menu_diskinfo ;; ?" $msg_pool_name") # Prompt the user to input/change the name for the new pool f_dialog_input input \ "$msg_please_enter_a_name_for_your_zpool" \ "$ZFSBOOT_POOL_NAME" && ZFSBOOT_POOL_NAME="$input" ;; ?" $msg_force_4k_sectors") # Toggle the variable referenced both by the menu and later if [ "$ZFSBOOT_FORCE_4K_SECTORS" ]; then ZFSBOOT_FORCE_4K_SECTORS= else ZFSBOOT_FORCE_4K_SECTORS=1 fi ;; ?" $msg_encrypt_disks") # Toggle the variable referenced both by the menu and later if [ "$ZFSBOOT_GELI_ENCRYPTION" ]; then ZFSBOOT_GELI_ENCRYPTION= else ZFSBOOT_FORCE_4K_SECTORS=1 ZFSBOOT_GELI_ENCRYPTION=1 fi ;; ?" $msg_partition_scheme") - # Toggle between GPT and MBR - if [ "$ZFSBOOT_PARTITION_SCHEME" = GPT ]; then - ZFSBOOT_PARTITION_SCHEME=MBR - elif [ "$ZFSBOOT_PARTITION_SCHEME" = MBR ]; then + # Toggle between GPT (BIOS), GPT (UEFI) and MBR + if [ "$ZFSBOOT_PARTITION_SCHEME" = "GPT" -a "$ZFSBOOT_BOOT_TYPE" = "BIOS" ]; then + ZFSBOOT_PARTITION_SCHEME="GPT" + ZFSBOOT_BOOT_TYPE="UEFI" + elif [ "$ZFSBOOT_PARTITION_SCHEME" = "GPT" ]; then + ZFSBOOT_PARTITION_SCHEME="MBR" + ZFSBOOT_BOOT_TYPE="BIOS" + elif [ "$ZFSBOOT_PARTITION_SCHEME" = "MBR" ]; then ZFSBOOT_PARTITION_SCHEME="GPT + Active" + ZFSBOOT_BOOT_TYPE="BIOS" elif [ "$ZFSBOOT_PARTITION_SCHEME" = "GPT + Active" ]; then ZFSBOOT_PARTITION_SCHEME="GPT + Lenovo Fix" + ZFSBOOT_BOOT_TYPE="BIOS" else - ZFSBOOT_PARTITION_SCHEME=GPT + ZFSBOOT_PARTITION_SCHEME="GPT" + ZFSBOOT_BOOT_TYPE="BIOS" fi ;; ?" $msg_swap_size") # Prompt the user to input/change the swap size for each disk while :; do f_dialog_input input \ "$msg_please_enter_amount_of_swap_space" \ "$ZFSBOOT_SWAP_SIZE" && ZFSBOOT_SWAP_SIZE="${input:-0}" if f_expand_number "$ZFSBOOT_SWAP_SIZE" swapsize then if [ $swapsize -ne 0 -a $swapsize -lt 104857600 ]; then f_show_err "$msg_swap_toosmall" \ "$ZFSBOOT_SWAP_SIZE" continue; else break; fi else f_show_err "$msg_swap_invalid" \ "$ZFSBOOT_SWAP_SIZE" continue; fi done ;; ?" $msg_swap_mirror") # Toggle the variable referenced both by the menu and later if [ "$ZFSBOOT_SWAP_MIRROR" ]; then ZFSBOOT_SWAP_MIRROR= else ZFSBOOT_SWAP_MIRROR=1 fi ;; ?" $msg_swap_encrypt") # Toggle the variable referenced both by the menu and later if [ "$ZFSBOOT_SWAP_ENCRYPTION" ]; then ZFSBOOT_SWAP_ENCRYPTION= else ZFSBOOT_SWAP_ENCRYPTION=1 fi ;; esac done exit $SUCCESS ################################################################################ # END ################################################################################ Index: user/ngie/make_check =================================================================== --- user/ngie/make_check (revision 295077) +++ user/ngie/make_check (revision 295078) Property changes on: user/ngie/make_check ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head:r295034-295077