Index: user/ngie/more-tests/README =================================================================== --- user/ngie/more-tests/README (revision 281503) +++ user/ngie/more-tests/README (revision 281504) @@ -1,90 +1,91 @@ This is the top level of the FreeBSD source directory. This file was last revised on: $FreeBSD$ For copyright information, please see the file COPYRIGHT in this directory (additional copyright information also exists for some sources in this tree - please see the specific source directories for more information). The Makefile in this directory supports a number of targets for building components (or all) of the FreeBSD source tree, the most commonly used one being ``world'', which rebuilds and installs everything in the FreeBSD system from the source tree except the kernel, the kernel-modules and the contents of /etc. The ``world'' target should only be used in cases where the source tree has not changed from the currently running version. See: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html for more information, including setting make(1) variables. The ``buildkernel'' and ``installkernel'' targets build and install the kernel and the modules (see below). Please see the top of the Makefile in this directory for more information on the standard build targets and compile-time flags. Building a kernel is a somewhat more involved process, documentation for which can be found at: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig.html And in the config(8) man page. Note: If you want to build and install the kernel with the ``buildkernel'' and ``installkernel'' targets, you might need to build world before. More information is available in the handbook. The sample kernel configuration files reside in the sys//conf sub-directory (assuming that you've installed the kernel sources), the file named GENERIC being the one used to build your initial installation kernel. The file NOTES contains entries and documentation for all possible -devices, not just those commonly used. It is the successor of the ancient -LINT file, but in contrast to LINT, it is not buildable as a kernel but a -pure reference and documentation file. +devices, not just those commonly used. Source Roadmap: --------------- bin System/user commands. cddl Various commands and libraries under the Common Development and Distribution License. contrib Packages contributed by 3rd parties. crypto Cryptography stuff (see crypto/README). etc Template files for /etc. games Amusements. gnu Various commands and libraries under the GNU Public License. Please see gnu/COPYING* for more information. include System include files. kerberos5 Kerberos5 (Heimdal) package. lib System libraries. libexec System daemons. release Release building Makefile & associated tools. rescue Build system for statically linked /rescue utilities. sbin System commands. secure Cryptographic libraries and commands. share Shared resources. sys Kernel sources. + +tests Regression tests which can be run by Kyua. See tests/README + for additional information. tools Utilities for regression testing and miscellaneous tasks. usr.bin User commands. usr.sbin System administration commands. For information on synchronizing your source tree with one or more of the FreeBSD Project's development branches, please see: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/synching.html Index: user/ngie/more-tests/share/mk/bsd.progs.mk =================================================================== --- user/ngie/more-tests/share/mk/bsd.progs.mk (revision 281503) +++ user/ngie/more-tests/share/mk/bsd.progs.mk (revision 281504) @@ -1,129 +1,129 @@ # $FreeBSD$ # $Id: progs.mk,v 1.11 2012/11/06 17:18:54 sjg Exp $ # # @(#) Copyright (c) 2006, Simon J. Gerraty # # This file is provided in the hope that it will # be of use. There is absolutely NO WARRANTY. # Permission to copy, redistribute or otherwise # use this file is hereby granted provided that # the above copyright notice and this notice are # left intact. # # Please send copies of changes and bug-fixes to: # sjg@crufty.net # .MAIN: all .if defined(PROGS) || defined(PROGS_CXX) # we really only use PROGS below... PROGS += ${PROGS_CXX} # In meta mode, we can capture dependenices for _one_ of the progs. # if makefile doesn't nominate one, we use the first. .if defined(.PARSEDIR) .ifndef UPDATE_DEPENDFILE_PROG UPDATE_DEPENDFILE_PROG = ${PROGS:[1]} .export UPDATE_DEPENDFILE_PROG .endif .else UPDATE_DEPENDFILE_PROG?= no .endif .ifndef PROG # They may have asked us to build just one .for t in ${PROGS} .if make($t) PROG ?= $t .endif .endfor .endif .if defined(PROG) # just one of many PROG_OVERRIDE_VARS += BINDIR DPSRCS MAN SRCS PROG_VARS += CFLAGS CPPFLAGS CXXFLAGS DPADD DPLIBS LDADD LIBADD LDFLAGS ${PROG_OVERRIDE_VARS} .for v in ${PROG_VARS:O:u} .if empty(${PROG_OVERRIDE_VARS:M$v}) .if defined(${v}.${PROG}) $v += ${${v}.${PROG}} .elif defined(${v}_${PROG}) $v += ${${v}_${PROG}} .endif .else $v ?= .endif .endfor # for meta mode, there can be only one! .if ${PROG} == ${UPDATE_DEPENDFILE_PROG} UPDATE_DEPENDFILE ?= yes .endif UPDATE_DEPENDFILE ?= NO # ensure that we don't clobber each other's dependencies DEPENDFILE?= .depend.${PROG} # prog.mk will do the rest .else all: ${FILES} ${PROGS} ${SCRIPTS} # We cannot capture dependencies for meta mode here UPDATE_DEPENDFILE = NO # nor can we safely run in parallel. .NOTPARALLEL: .endif .endif # The non-recursive call to bsd.progs.mk will handle FILES; NUL out # FILESGROUPS so recursive calls don't duplicate the work .ifdef _RECURSING_PROGS FILESGROUPS= .endif # handle being called [bsd.]progs.mk .include .ifndef _RECURSING_PROGS # tell progs.mk we might want to install things PROGS_TARGETS+= checkdpadd clean cleandepend cleandir cleanobj depend install .for p in ${PROGS} .if defined(PROGS_CXX) && !empty(PROGS_CXX:M$p) # bsd.prog.mk may need to know this x.$p= PROG_CXX=$p .endif $p ${p}_p: .PHONY .MAKE (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} _RECURSING_PROGS= \ - SUBDIR= SCRIPTS= PROG=$p \ + SUBDIR= PROG=$p \ DEPENDFILE=.depend.$p .MAKE.DEPENDFILE=.depend.$p \ ${x.$p}) .for t in ${PROGS_TARGETS:O:u} $p.$t: .PHONY .MAKE (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} _RECURSING_PROGS= \ - SUBDIR= SCRIPTS= PROG=$p \ + SUBDIR= PROG=$p \ DEPENDFILE=.depend.$p .MAKE.DEPENDFILE=.depend.$p \ ${x.$p} ${@:E}) .endfor .endfor .if !empty(PROGS) .for t in ${PROGS_TARGETS:O:u} $t: ${PROGS:%=%.$t} .endfor .endif .if empty(PROGS) && !empty(SCRIPTS) .for t in ${PROGS_TARGETS:O:u} scripts.$t: .PHONY .MAKE (cd ${.CURDIR} && ${MAKE} -f ${MAKEFILE} SUBDIR= _RECURSING_PROGS= \ $t) $t: scripts.$t .endfor .endif .endif Index: user/ngie/more-tests/share =================================================================== --- user/ngie/more-tests/share (revision 281503) +++ user/ngie/more-tests/share (revision 281504) Property changes on: user/ngie/more-tests/share ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/share:r281414-281503 Index: user/ngie/more-tests/sys/arm/arm/devmap.c =================================================================== --- user/ngie/more-tests/sys/arm/arm/devmap.c (revision 281503) +++ user/ngie/more-tests/sys/arm/arm/devmap.c (revision 281504) @@ -1,298 +1,327 @@ /*- * Copyright (c) 2013 Ian Lepore * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* * Routines for mapping device memory. + * + * This is used on both arm and arm64. */ #include "opt_ddb.h" #include #include #include #include #include #include #include +#include static const struct arm_devmap_entry *devmap_table; static boolean_t devmap_bootstrap_done = false; +#if defined(__aarch64__) +#define MAX_VADDR VM_MAX_KERNEL_ADDRESS +#define PTE_DEVICE VM_MEMATTR_DEVICE +#elif defined(__arm__) +#define MAX_VADDR ARM_VECTORS_HIGH +#endif + /* * The allocated-kva (akva) devmap table and metadata. Platforms can call * arm_devmap_add_entry() to add static device mappings to this table using * automatically allocated virtual addresses carved out of the top of kva space. * Allocation begins immediately below the ARM_VECTORS_HIGH address. */ #define AKVA_DEVMAP_MAX_ENTRIES 32 static struct arm_devmap_entry akva_devmap_entries[AKVA_DEVMAP_MAX_ENTRIES]; static u_int akva_devmap_idx; -static vm_offset_t akva_devmap_vaddr = ARM_VECTORS_HIGH; +static vm_offset_t akva_devmap_vaddr = MAX_VADDR; +#ifdef __aarch64__ +extern int early_boot; +#endif + /* * Print the contents of the static mapping table using the provided printf-like * output function (which will be either printf or db_printf). */ static void devmap_dump_table(int (*prfunc)(const char *, ...)) { const struct arm_devmap_entry *pd; if (devmap_table == NULL || devmap_table[0].pd_size == 0) { prfunc("No static device mappings.\n"); return; } prfunc("Static device mappings:\n"); for (pd = devmap_table; pd->pd_size != 0; ++pd) { prfunc(" 0x%08x - 0x%08x mapped at VA 0x%08x\n", pd->pd_pa, pd->pd_pa + pd->pd_size - 1, pd->pd_va); } } /* * Print the contents of the static mapping table. Used for bootverbose. */ void arm_devmap_print_table() { devmap_dump_table(printf); } /* * Return the "last" kva address used by the registered devmap table. It's * actually the lowest address used by the static mappings, i.e., the address of * the first unusable byte of KVA. */ vm_offset_t arm_devmap_lastaddr() { const struct arm_devmap_entry *pd; vm_offset_t lowaddr; if (akva_devmap_idx > 0) return (akva_devmap_vaddr); - lowaddr = ARM_VECTORS_HIGH; + lowaddr = MAX_VADDR; for (pd = devmap_table; pd != NULL && pd->pd_size != 0; ++pd) { if (lowaddr > pd->pd_va) lowaddr = pd->pd_va; } return (lowaddr); } /* * Add an entry to the internal "akva" static devmap table using the given * physical address and size and a virtual address allocated from the top of * kva. This automatically registers the akva table on the first call, so all a * platform has to do is call this routine to install as many mappings as it * needs and when initarm() calls arm_devmap_bootstrap() it will pick up all the * entries in the akva table automatically. */ void arm_devmap_add_entry(vm_paddr_t pa, vm_size_t sz) { struct arm_devmap_entry *m; if (devmap_bootstrap_done) panic("arm_devmap_add_entry() after arm_devmap_bootstrap()"); if (akva_devmap_idx == (AKVA_DEVMAP_MAX_ENTRIES - 1)) panic("AKVA_DEVMAP_MAX_ENTRIES is too small"); if (akva_devmap_idx == 0) arm_devmap_register_table(akva_devmap_entries); /* * Allocate virtual address space from the top of kva downwards. If the * range being mapped is aligned and sized to 1MB boundaries then also * align the virtual address to the next-lower 1MB boundary so that we * end up with a nice efficient section mapping. */ +#ifdef __arm__ if ((pa & 0x000fffff) == 0 && (sz & 0x000fffff) == 0) { akva_devmap_vaddr = trunc_1mpage(akva_devmap_vaddr - sz); - } else { + } else +#endif + { akva_devmap_vaddr = trunc_page(akva_devmap_vaddr - sz); } m = &akva_devmap_entries[akva_devmap_idx++]; m->pd_va = akva_devmap_vaddr; m->pd_pa = pa; m->pd_size = sz; m->pd_prot = VM_PROT_READ | VM_PROT_WRITE; m->pd_cache = PTE_DEVICE; } /* * Register the given table as the one to use in arm_devmap_bootstrap(). */ void arm_devmap_register_table(const struct arm_devmap_entry *table) { devmap_table = table; } /* * Map all of the static regions in the devmap table, and remember the devmap * table so the mapdev, ptov, and vtop functions can do lookups later. * * If a non-NULL table pointer is given it is used unconditionally, otherwise * the previously-registered table is used. This smooths transition from legacy * code that fills in a local table then calls this function passing that table, * and newer code that uses arm_devmap_register_table() in platform-specific * code, then lets the common initarm() call this function with a NULL pointer. */ void arm_devmap_bootstrap(vm_offset_t l1pt, const struct arm_devmap_entry *table) { const struct arm_devmap_entry *pd; devmap_bootstrap_done = true; /* * If given a table pointer, use it. Otherwise, if a table was * previously registered, use it. Otherwise, no work to do. */ if (table != NULL) devmap_table = table; else if (devmap_table == NULL) return; for (pd = devmap_table; pd->pd_size != 0; ++pd) { +#if defined(__arm__) pmap_map_chunk(l1pt, pd->pd_va, pd->pd_pa, pd->pd_size, pd->pd_prot,pd->pd_cache); +#elif defined(__aarch64__) + pmap_kenter_device(pd->pd_va, pd->pd_size, pd->pd_pa); +#endif } } /* * Look up the given physical address in the static mapping data and return the * corresponding virtual address, or NULL if not found. */ void * arm_devmap_ptov(vm_paddr_t pa, vm_size_t size) { const struct arm_devmap_entry *pd; if (devmap_table == NULL) return (NULL); for (pd = devmap_table; pd->pd_size != 0; ++pd) { if (pa >= pd->pd_pa && pa + size <= pd->pd_pa + pd->pd_size) return ((void *)(pd->pd_va + (pa - pd->pd_pa))); } return (NULL); } /* * Look up the given virtual address in the static mapping data and return the * corresponding physical address, or DEVMAP_PADDR_NOTFOUND if not found. */ vm_paddr_t arm_devmap_vtop(void * vpva, vm_size_t size) { const struct arm_devmap_entry *pd; vm_offset_t va; if (devmap_table == NULL) return (DEVMAP_PADDR_NOTFOUND); va = (vm_offset_t)vpva; for (pd = devmap_table; pd->pd_size != 0; ++pd) { if (va >= pd->pd_va && va + size <= pd->pd_va + pd->pd_size) return ((vm_paddr_t)(pd->pd_pa + (va - pd->pd_va))); } return (DEVMAP_PADDR_NOTFOUND); } /* * Map a set of physical memory pages into the kernel virtual address space. * Return a pointer to where it is mapped. * * This uses a pre-established static mapping if one exists for the requested * range, otherwise it allocates kva space and maps the physical pages into it. * * This routine is intended to be used for mapping device memory, NOT real * memory; the mapping type is inherently PTE_DEVICE in pmap_kenter_device(). */ void * pmap_mapdev(vm_offset_t pa, vm_size_t size) { vm_offset_t va, offset; void * rva; /* First look in the static mapping table. */ if ((rva = arm_devmap_ptov(pa, size)) != NULL) return (rva); - + offset = pa & PAGE_MASK; pa = trunc_page(pa); size = round_page(size + offset); - - va = kva_alloc(size); + +#ifdef __aarch64__ + if (early_boot) { + akva_devmap_vaddr = trunc_page(akva_devmap_vaddr - size); + va = akva_devmap_vaddr; + KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE, + ("Too many early devmap mappings")); + } else +#endif + va = kva_alloc(size); if (!va) panic("pmap_mapdev: Couldn't alloc kernel virtual memory"); pmap_kenter_device(va, size, pa); - + return ((void *)(va + offset)); } /* * Unmap device memory and free the kva space. */ void pmap_unmapdev(vm_offset_t va, vm_size_t size) { vm_offset_t offset; /* Nothing to do if we find the mapping in the static table. */ if (arm_devmap_vtop((void*)va, size) != DEVMAP_PADDR_NOTFOUND) return; offset = va & PAGE_MASK; va = trunc_page(va); size = round_page(size + offset); pmap_kremove_device(va, size); kva_free(va, size); } #ifdef DDB #include DB_SHOW_COMMAND(devmap, db_show_devmap) { devmap_dump_table(db_printf); } #endif /* DDB */ Index: user/ngie/more-tests/sys/arm64/arm64/autoconf.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/autoconf.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/autoconf.c (revision 281504) @@ -0,0 +1,90 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +/* + * Setup the system to run on the current machine. + * + * Configure() is called at boot time and initializes the vba + * device tables and the memory controller monitoring. Available + * devices are determined (from possibilities mentioned in ioconf.c), + * and the drivers are initialized. + */ + +#include +#include +#include +#include +#include + +#include + +static void configure_first(void *); +static void configure(void *); +static void configure_final(void *); + +SYSINIT(configure1, SI_SUB_CONFIGURE, SI_ORDER_FIRST, configure_first, NULL); +/* SI_ORDER_SECOND is hookable */ +SYSINIT(configure2, SI_SUB_CONFIGURE, SI_ORDER_THIRD, configure, NULL); +/* SI_ORDER_MIDDLE is hookable */ +SYSINIT(configure3, SI_SUB_CONFIGURE, SI_ORDER_ANY, configure_final, NULL); + +/* + * Determine i/o configuration for a machine. + */ +static void +configure_first(void *dummy) +{ + + /* nexus0 is the top of the device tree */ + device_add_child(root_bus, "nexus", 0); +} + +static void +configure(void *dummy) +{ + + /* initialize new bus architecture */ + root_bus_configure(); +} + +static void +configure_final(void *dummy) +{ + arm_enable_intr(); + cninit_finish(); + + if (bootverbose) + printf("Device configuration finished.\n"); + + cold = 0; +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/autoconf.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/bcopy.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/bcopy.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/bcopy.c (revision 281504) @@ -0,0 +1,139 @@ +/*- + * Copyright (c) 1990 The Regents of the University of California. + * All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * Chris Torek. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * From: sys/powerpc/powerpc/bcopy.c + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +/* + * sizeof(word) MUST BE A POWER OF TWO + * SO THAT wmask BELOW IS ALL ONES + */ +typedef long word; /* "word" used for optimal copy speed */ + +#define wsize sizeof(word) +#define wmask (wsize - 1) + +/* + * Copy a block of memory, handling overlap. + * This is the routine that actually implements + * (the portable versions of) bcopy, memcpy, and memmove. + */ +void * +memcpy(void *dst0, const void *src0, size_t length) +{ + char *dst; + const char *src; + size_t t; + + dst = dst0; + src = src0; + + if (length == 0 || dst == src) { /* nothing to do */ + goto done; + } + + /* + * Macros: loop-t-times; and loop-t-times, t>0 + */ +#define TLOOP(s) if (t) TLOOP1(s) +#define TLOOP1(s) do { s; } while (--t) + + if ((unsigned long)dst < (unsigned long)src) { + /* + * Copy forward. + */ + t = (size_t)src; /* only need low bits */ + + if ((t | (uintptr_t)dst) & wmask) { + /* + * Try to align operands. This cannot be done + * unless the low bits match. + */ + if ((t ^ (uintptr_t)dst) & wmask || length < wsize) { + t = length; + } else { + t = wsize - (t & wmask); + } + + length -= t; + TLOOP1(*dst++ = *src++); + } + /* + * Copy whole words, then mop up any trailing bytes. + */ + t = length / wsize; + TLOOP(*(word *)dst = *(const word *)src; src += wsize; + dst += wsize); + t = length & wmask; + TLOOP(*dst++ = *src++); + } else { + /* + * Copy backwards. Otherwise essentially the same. + * Alignment works as before, except that it takes + * (t&wmask) bytes to align, not wsize-(t&wmask). + */ + src += length; + dst += length; + t = (uintptr_t)src; + + if ((t | (uintptr_t)dst) & wmask) { + if ((t ^ (uintptr_t)dst) & wmask || length <= wsize) { + t = length; + } else { + t &= wmask; + } + + length -= t; + TLOOP1(*--dst = *--src); + } + t = length / wsize; + TLOOP(src -= wsize; dst -= wsize; + *(word *)dst = *(const word *)src); + t = length & wmask; + TLOOP(*--dst = *--src); + } +done: + return (dst0); +} + +void +bcopy(const void *src0, void *dst0, size_t length) +{ + + memcpy(dst0, src0, length); +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/bcopy.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/bus_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/bus_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/bus_machdep.c (revision 281504) @@ -0,0 +1,204 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include "opt_platform.h" + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +#include + +uint8_t generic_bs_r_1(void *, bus_space_handle_t, bus_size_t); +uint16_t generic_bs_r_2(void *, bus_space_handle_t, bus_size_t); +uint32_t generic_bs_r_4(void *, bus_space_handle_t, bus_size_t); +uint64_t generic_bs_r_8(void *, bus_space_handle_t, bus_size_t); + +void generic_bs_rm_1(void *, bus_space_handle_t, bus_size_t, uint8_t *, + bus_size_t); +void generic_bs_rm_2(void *, bus_space_handle_t, bus_size_t, uint16_t *, + bus_size_t); +void generic_bs_rm_4(void *, bus_space_handle_t, bus_size_t, uint32_t *, + bus_size_t); +void generic_bs_rm_8(void *, bus_space_handle_t, bus_size_t, uint64_t *, + bus_size_t); + +void generic_bs_w_1(void *, bus_space_handle_t, bus_size_t, uint8_t); +void generic_bs_w_2(void *, bus_space_handle_t, bus_size_t, uint16_t); +void generic_bs_w_4(void *, bus_space_handle_t, bus_size_t, uint32_t); +void generic_bs_w_8(void *, bus_space_handle_t, bus_size_t, uint64_t); + +void generic_bs_wm_1(void *, bus_space_handle_t, bus_size_t, const uint8_t *, + bus_size_t); +void generic_bs_wm_2(void *, bus_space_handle_t, bus_size_t, const uint16_t *, + bus_size_t); +void generic_bs_wm_4(void *, bus_space_handle_t, bus_size_t, const uint32_t *, + bus_size_t); +void generic_bs_wm_8(void *, bus_space_handle_t, bus_size_t, const uint64_t *, + bus_size_t); + +static int +generic_bs_map(void *t, bus_addr_t bpa, bus_size_t size, int flags, + bus_space_handle_t *bshp) +{ + void *va; + + va = pmap_mapdev(bpa, size); + if (va == NULL) + return (ENOMEM); + *bshp = (bus_space_handle_t)va; + return (0); +} + +static void +generic_bs_unmap(void *t, bus_space_handle_t bsh, bus_size_t size) +{ + + pmap_unmapdev(bsh, size); +} + +static void +generic_bs_barrier(void *t, bus_space_handle_t bsh, bus_size_t offset, + bus_size_t size, int flags) +{ +} + +static int +generic_bs_subregion(void *t, bus_space_handle_t bsh, bus_size_t offset, + bus_size_t size, bus_space_handle_t *nbshp) +{ + + *nbshp = bsh + offset; + return (0); +} + +struct bus_space memmap_bus = { + /* cookie */ + .bs_cookie = NULL, + + /* mapping/unmapping */ + .bs_map = generic_bs_map, + .bs_unmap = generic_bs_unmap, + .bs_subregion = generic_bs_subregion, + + /* allocation/deallocation */ + .bs_alloc = NULL, + .bs_free = NULL, + + /* barrier */ + .bs_barrier = generic_bs_barrier, + + /* read single */ + .bs_r_1 = generic_bs_r_1, + .bs_r_2 = generic_bs_r_2, + .bs_r_4 = generic_bs_r_4, + .bs_r_8 = generic_bs_r_8, + + /* read multiple */ + .bs_rm_1 = generic_bs_rm_1, + .bs_rm_2 = generic_bs_rm_2, + .bs_rm_4 = generic_bs_rm_4, + .bs_rm_8 = generic_bs_rm_8, + + /* write single */ + .bs_w_1 = generic_bs_w_1, + .bs_w_2 = generic_bs_w_2, + .bs_w_4 = generic_bs_w_4, + .bs_w_8 = generic_bs_w_8, + + /* write multiple */ + .bs_wm_1 = generic_bs_wm_1, + .bs_wm_2 = generic_bs_wm_2, + .bs_wm_4 = generic_bs_wm_4, + .bs_wm_8 = generic_bs_wm_8, + + /* write region */ + .bs_wr_1 = NULL, + .bs_wr_2 = NULL, + .bs_wr_4 = NULL, + .bs_wr_8 = NULL, + + /* set multiple */ + .bs_sm_1 = NULL, + .bs_sm_2 = NULL, + .bs_sm_4 = NULL, + .bs_sm_8 = NULL, + + /* set region */ + .bs_sr_1 = NULL, + .bs_sr_2 = NULL, + .bs_sr_4 = NULL, + .bs_sr_8 = NULL, + + /* copy */ + .bs_c_1 = NULL, + .bs_c_2 = NULL, + .bs_c_4 = NULL, + .bs_c_8 = NULL, + + /* read single stream */ + .bs_r_1_s = NULL, + .bs_r_2_s = NULL, + .bs_r_4_s = NULL, + .bs_r_8_s = NULL, + + /* read multiple stream */ + .bs_rm_1_s = NULL, + .bs_rm_2_s = NULL, + .bs_rm_4_s = NULL, + .bs_rm_8_s = NULL, + + /* read region stream */ + .bs_rr_1_s = NULL, + .bs_rr_2_s = NULL, + .bs_rr_4_s = NULL, + .bs_rr_8_s = NULL, + + /* write single stream */ + .bs_w_1_s = NULL, + .bs_w_2_s = NULL, + .bs_w_4_s = NULL, + .bs_w_8_s = NULL, + + /* write multiple stream */ + .bs_wm_1_s = NULL, + .bs_wm_2_s = NULL, + .bs_wm_4_s = NULL, + .bs_wm_8_s = NULL, + + /* write region stream */ + .bs_wr_1_s = NULL, + .bs_wr_2_s = NULL, + .bs_wr_4_s = NULL, + .bs_wr_8_s = NULL, +}; + +#ifdef FDT +bus_space_tag_t fdtbus_bs_tag = &memmap_bus; +#endif Property changes on: user/ngie/more-tests/sys/arm64/arm64/bus_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/bus_space_asm.S =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/bus_space_asm.S (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/bus_space_asm.S (revision 281504) @@ -0,0 +1,235 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include + +__FBSDID("$FreeBSD$"); + +ENTRY(generic_bs_r_1) + ldrb w0, [x1, x2] + ret +END(generic_bs_r_1) + +ENTRY(generic_bs_r_2) + ldrh w0, [x1, x2] + ret +END(generic_bs_r_2) + +ENTRY(generic_bs_r_4) + ldr w0, [x1, x2] + ret +END(generic_bs_r_4) + +ENTRY(generic_bs_r_8) + ldr x0, [x1, x2] + ret +END(generic_bs_r_8) + +ENTRY(generic_bs_rm_1) + /* If there is anything to read. */ + cbz x4, 2f + + /* Calculate the device address. */ + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Read the data. */ +1: ldrb w1, [x0] + strb w1, [x3], #2 + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_rm_1) + +ENTRY(generic_bs_rm_2) + /* If there is anything to read. */ + cbz x4, 2f + + /* Calculate the device address. */ + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Read the data. */ +1: ldrh w1, [x0] + strh w1, [x3], #2 + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_rm_2) + +ENTRY(generic_bs_rm_4) + /* If there is anything to read. */ + cbz x4, 2f + + /* Calculate the device address. */ + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Read the data. */ +1: ldr w1, [x0] + str w1, [x3], #2 + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_rm_4) + +ENTRY(generic_bs_rm_8) + /* If there is anything to read. */ + cbz x4, 2f + + /* Calculate the device address. */ + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Read the data. */ +1: ldr x1, [x0] + str x1, [x3], #2 + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_rm_8) + + +ENTRY(generic_bs_w_1) + strb w3, [x1, x2] + ret +END(generic_bs_w_1) + +ENTRY(generic_bs_w_2) + strh w3, [x1, x2] + ret +END(generic_bs_w_2) + +ENTRY(generic_bs_w_4) + str w3, [x1, x2] + ret +END(generic_bs_w_4) + +ENTRY(generic_bs_w_8) + str x3, [x1, x2] + ret +END(generic_bs_w_8) + +ENTRY(generic_bs_wm_1) + /* If there is anything to write. */ + cbz x4, 2f + + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Write the data */ +1: ldrb w1, [x3], #1 + strb w1, [x0] + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_wm_1) + +ENTRY(generic_bs_wm_2) + /* If there is anything to write. */ + cbz x4, 2f + + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Write the data */ +1: ldrh w1, [x3], #2 + strh w1, [x0] + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_wm_2) + +ENTRY(generic_bs_wm_4) + /* If there is anything to write. */ + cbz x4, 2f + + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Write the data */ +1: ldr w1, [x3], #4 + str w1, [x0] + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_wm_4) + +ENTRY(generic_bs_wm_8) + /* If there is anything to write. */ + cbz x4, 2f + + add x0, x1, x2 + /* + * x0 = The device address. + * x3 = The kernel address. + * x4 = Count + */ + + /* Write the data */ +1: ldr x1, [x3], #8 + str x1, [x0] + subs x4, x4, #1 + b.ne 1b + +2: ret +END(generic_bs_wm_8) Property changes on: user/ngie/more-tests/sys/arm64/arm64/bus_space_asm.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/busdma_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/busdma_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/busdma_machdep.c (revision 281504) @@ -0,0 +1,68 @@ +/* $FreeBSD$ */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +#include +#include + +#include + +int +_bus_dmamap_load_phys(bus_dma_tag_t dmat, bus_dmamap_t map, vm_paddr_t buf, + bus_size_t buflen, int flags, bus_dma_segment_t *segs, int *segp) +{ + + panic("_bus_dmamap_load_phys"); +} + +int +_bus_dmamap_load_ma(bus_dma_tag_t dmat, bus_dmamap_t map, struct vm_page **ma, + bus_size_t tlen, int ma_offs, int flags, bus_dma_segment_t *segs, + int *segp) +{ + + panic("_bus_dmamap_load_ma"); +} + +int +_bus_dmamap_load_buffer(bus_dma_tag_t dmat, bus_dmamap_t map, void *buf, + bus_size_t buflen, pmap_t pmap, int flags, bus_dma_segment_t *segs, + int *segp) +{ + + panic("_bus_dmamap_load_buffer"); +} + +void +__bus_dmamap_waitok(bus_dma_tag_t dmat, bus_dmamap_t map, + struct memdesc *mem, bus_dmamap_callback_t *callback, void *callback_arg) +{ + + panic("__bus_dmamap_waitok"); +} + +bus_dma_segment_t * +_bus_dmamap_complete(bus_dma_tag_t dmat, bus_dmamap_t map, + bus_dma_segment_t *segs, int nsegs, int error) +{ + + panic("_bus_dmamap_complete"); +} + +void +_bus_dmamap_unload(bus_dma_tag_t dmat, bus_dmamap_t map) +{ + + panic("_bus_dmamap_unload"); +} + +void +_bus_dmamap_sync(bus_dma_tag_t dmat, bus_dmamap_t map, bus_dmasync_op_t op) +{ + + panic("_bus_dmamap_sync"); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/busdma_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/clock.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/clock.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/clock.c (revision 281504) @@ -0,0 +1,39 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +void +cpu_initclocks(void) +{ + + cpu_initclocks_bsp(); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/clock.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/copyinout.S =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/copyinout.S (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/copyinout.S (revision 281504) @@ -0,0 +1,118 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include + +#include "assym.s" + +/* + * Fault handler for the copy{in,out} functions below. + */ +ENTRY(copyio_fault) + SET_FAULT_HANDLER(xzr, x1) /* Clear the handler */ + mov x0, #EFAULT + ret +END(copyio_fault) + +/* + * Copies from a kernel to user address + * + * int copyout(const void *kaddr, void *udaddr, size_t len) + */ +ENTRY(copyout) + cbz x2, 2f /* If len == 0 then skip loop */ + + adr x6, copyio_fault /* Get the handler address */ + SET_FAULT_HANDLER(x6, x7) /* Set the handler */ + +1: ldrb w4, [x0], #1 /* Load from kaddr */ + strb w4, [x1], #1 /* Store in uaddr */ + sub x2, x2, #1 /* len-- */ + cbnz x2, 1b + + SET_FAULT_HANDLER(xzr, x7) /* Clear the handler */ + +2: mov x0, xzr /* return 0 */ + ret +END(copyout) + +/* + * Copies from a user to kernel address + * + * int copyin(const void *uaddr, void *kdaddr, size_t len) + */ +ENTRY(copyin) + cbz x2, 2f /* If len == 0 then skip loop */ + + adr x6, copyio_fault /* Get the handler address */ + SET_FAULT_HANDLER(x6, x7) /* Set the handler */ + +1: ldrb w4, [x0], #1 /* Load from uaddr */ + strb w4, [x1], #1 /* Store in kaddr */ + sub x2, x2, #1 /* len-- */ + cbnz x2, 1b + + SET_FAULT_HANDLER(xzr, x7) /* Clear the handler */ + +2: mov x0, xzr /* return 0 */ + ret +END(copyin) + +/* + * Copies a string from a user to kernel address + * + * int copyinstr(const void *udaddr, void *kaddr, size_t len, size_t *done) + */ +ENTRY(copyinstr) + mov x5, xzr /* count = 0 */ + cbz x2, 3f /* If len == 0 then skip loop */ + + adr x6, copyio_fault /* Get the handler address */ + SET_FAULT_HANDLER(x6, x7) /* Set the handler */ + +1: ldrb w4, [x0], #1 /* Load from uaddr */ + strb w4, [x1], #1 /* Store in kaddr */ + cbz w4, 2f /* If == 0 then break */ + sub x2, x2, #1 /* len-- */ + add x5, x5, #1 /* count++ */ + cbnz x2, 1b + +2: SET_FAULT_HANDLER(xzr, x7) /* Clear the handler */ + +3: cbz x3, 4f /* Check if done != NULL */ + add x5, x5, #1 /* count++ */ + str x5, [x3] /* done = count */ + +4: mov x0, xzr /* return 0 */ + ret +END(copyinstr) Property changes on: user/ngie/more-tests/sys/arm64/arm64/copyinout.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/copystr.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/copystr.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/copystr.c (revision 281504) @@ -0,0 +1,61 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include + +int +copystr(const void * __restrict kfaddr, void * __restrict kdaddr, size_t len, + size_t * __restrict lencopied) +{ + const char *src; + size_t pos; + char *dst; + int error; + + error = ENAMETOOLONG; + src = kfaddr; + dst = kdaddr; + for (pos = 0; pos < len; pos++) { + dst[pos] = src[pos]; + if (src[pos] == '\0') { + /* Increment pos to hold the number of bytes copied */ + pos++; + error = 0; + break; + } + } + + if (lencopied != NULL) + *lencopied = pos; + + return (0); +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/copystr.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/cpufunc_asm.S =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/cpufunc_asm.S (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/cpufunc_asm.S (revision 281504) @@ -0,0 +1,154 @@ +/*- + * Copyright (c) 2014 Robin Randhawa + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * Portions of this software were developed by Andrew Turner + * under sponsorship from the FreeBSD Foundation + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +#include +__FBSDID("$FreeBSD$"); + +/* + * FIXME: + * Need big.LITTLE awareness at some point. + * Using arm64_p[id]cache_line_size may not be the best option. + * Need better SMP awareness. + */ + .text + .align 2 + +.Lpage_mask: + .word PAGE_MASK + +/* + * Macro to handle the cache. This takes the start address in x0, length + * in x1. It will corrupt x0, x1, x2, and x3. + */ +.macro cache_handle_range dcop = 0, ic = 0, icop = 0 +.if \ic == 0 + ldr x3, =dcache_line_size /* Load the D cache line size */ +.else + ldr x3, =idcache_line_size /* Load the I & D cache line size */ +.endif + ldr x3, [x3] + sub x4, x3, #1 /* Get the address mask */ + and x2, x0, x4 /* Get the low bits of the address */ + add x1, x1, x2 /* Add these to the size */ + bic x0, x0, x4 /* Clear the low bit of the address */ +1: +.if \ic != 0 + ic \icop, x0 +.endif + dc \dcop, x0 + add x0, x0, x3 /* Move to the next line */ + subs x1, x1, x3 /* Reduce the size */ + b.hi 1b /* Check if we are done */ +.if \ic != 0 + isb +.endif + dsb ish + ret +.endm + +ENTRY(arm64_nullop) + ret +END(arm64_nullop) + +/* + * Generic functions to read/modify/write the internal coprocessor registers + */ + +ENTRY(arm64_setttb) + dsb ish + msr ttbr0_el1, x0 + dsb ish + isb + ret +END(arm64_setttb) + +ENTRY(arm64_tlb_flushID) +#ifdef SMP + tlbi vmalle1is +#else + tlbi vmalle1 +#endif + dsb ish + isb + ret +END(arm64_tlb_flushID) + +ENTRY(arm64_tlb_flushID_SE) + ldr x1, .Lpage_mask + bic x0, x0, x1 +#ifdef SMP + tlbi vae1is, x0 +#else + tlbi vae1, x0 +#endif + dsb ish + isb + ret +END(arm64_tlb_flushID_SE) + +/* + * void arm64_dcache_wb_range(vm_offset_t, vm_size_t) + */ +ENTRY(arm64_dcache_wb_range) + cache_handle_range dcop = cvac +END(arm64_dcache_wb_range) + +/* + * void arm64_dcache_wbinv_range(vm_offset_t, vm_size_t) + */ +ENTRY(arm64_dcache_wbinv_range) + cache_handle_range dcop = civac +END(arm64_dcache_wbinv_range) + +/* + * void arm64_dcache_inv_range(vm_offset_t, vm_size_t) + * + * Note, we must not invalidate everything. If the range is too big we + * must use wb-inv of the entire cache. + */ +ENTRY(arm64_dcache_inv_range) + cache_handle_range dcop = ivac +END(arm64_dcache_inv_range) + +/* + * void arm64_idcache_wbinv_range(vm_offset_t, vm_size_t) + */ +ENTRY(arm64_idcache_wbinv_range) + cache_handle_range dcop = civac, ic = 1, icop = ivau +END(arm64_idcache_wbinv_range) + +/* + * void arm64_icache_sync_range(vm_offset_t, vm_size_t) + */ +ENTRY(arm64_icache_sync_range) + cache_handle_range dcop = cvac, ic = 1, icop = ivau +END(arm64_icache_sync_range) Property changes on: user/ngie/more-tests/sys/arm64/arm64/cpufunc_asm.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/dump_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/dump_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/dump_machdep.c (revision 281504) @@ -0,0 +1,73 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include + +int do_minidump = 1; +TUNABLE_INT("debug.minidump", &do_minidump); +SYSCTL_INT(_debug, OID_AUTO, minidump, CTLFLAG_RW, &do_minidump, 0, + "Enable mini crash dumps"); + +void +dumpsys_wbinv_all(void) +{ + + printf("dumpsys_wbinv_all\n"); +} + +void +dumpsys_map_chunk(vm_paddr_t pa, size_t chunk __unused, void **va) +{ + + printf("dumpsys_map_chunk\n"); + while(1); +} + +/* + * Add a header to be used by libkvm to get the va to pa delta + */ +int +dumpsys_write_aux_headers(struct dumperinfo *di) +{ + + printf("dumpsys_map_chunk\n"); + return (0); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/dump_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/elf_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/elf_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/elf_machdep.c (revision 281504) @@ -0,0 +1,164 @@ +/*- + * Copyright (c) 2014 The FreeBSD Foundation. + * Copyright (c) 2014 Andrew Turner. + * Copyright (c) 2001 Jake Burkholder. + * Copyright (c) 2000 Eduardo Horvath. + * Copyright (c) 1999 The NetBSD Foundation, Inc. + * All rights reserved. + * + * This code is derived from software contributed to The NetBSD Foundation + * by Paul Kranenburg. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + * + * from: NetBSD: mdreloc.c,v 1.42 2008/04/28 20:23:04 martin Exp + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include + +#include "linker_if.h" + +static struct sysentvec elf64_freebsd_sysvec = { + .sv_size = SYS_MAXSYSCALL, + .sv_table = sysent, + .sv_mask = 0, + .sv_sigsize = 0, + .sv_sigtbl = NULL, + .sv_errsize = 0, + .sv_errtbl = NULL, + .sv_transtrap = NULL, + .sv_fixup = __elfN(freebsd_fixup), + .sv_sendsig = sendsig, + .sv_sigcode = sigcode, + .sv_szsigcode = &szsigcode, + .sv_prepsyscall = NULL, + .sv_name = "FreeBSD ELF64", + .sv_coredump = __elfN(coredump), + .sv_imgact_try = NULL, + .sv_minsigstksz = MINSIGSTKSZ, + .sv_pagesize = PAGE_SIZE, + .sv_minuser = VM_MIN_ADDRESS, + .sv_maxuser = VM_MAXUSER_ADDRESS, + .sv_usrstack = USRSTACK, + .sv_psstrings = PS_STRINGS, + .sv_stackprot = VM_PROT_READ | VM_PROT_WRITE, + .sv_copyout_strings = exec_copyout_strings, + .sv_setregs = exec_setregs, + .sv_fixlimit = NULL, + .sv_maxssiz = NULL, + .sv_flags = SV_ABI_FREEBSD | SV_LP64, + .sv_set_syscall_retval = cpu_set_syscall_retval, + .sv_fetch_syscall_args = cpu_fetch_syscall_args, + .sv_syscallnames = syscallnames, + .sv_schedtail = NULL, +}; + +static Elf64_Brandinfo freebsd_brand_info = { + .brand = ELFOSABI_FREEBSD, + .machine = EM_AARCH64, + .compat_3_brand = "FreeBSD", + .emul_path = NULL, + .interp_path = "/libexec/ld-elf.so.1", + .sysvec = &elf64_freebsd_sysvec, + .interp_newpath = NULL, + .brand_note = &elf64_freebsd_brandnote, + .flags = BI_CAN_EXEC_DYN | BI_BRAND_NOTE +}; + +SYSINIT(elf64, SI_SUB_EXEC, SI_ORDER_FIRST, + (sysinit_cfunc_t)elf64_insert_brand_entry, &freebsd_brand_info); + +static Elf64_Brandinfo freebsd_brand_oinfo = { + .brand = ELFOSABI_FREEBSD, + .machine = EM_AARCH64, + .compat_3_brand = "FreeBSD", + .emul_path = NULL, + .interp_path = "/usr/libexec/ld-elf.so.1", + .sysvec = &elf64_freebsd_sysvec, + .interp_newpath = NULL, + .brand_note = &elf64_freebsd_brandnote, + .flags = BI_CAN_EXEC_DYN | BI_BRAND_NOTE +}; + +SYSINIT(oelf64, SI_SUB_EXEC, SI_ORDER_ANY, + (sysinit_cfunc_t)elf64_insert_brand_entry, &freebsd_brand_oinfo); + +void +elf64_dump_thread(struct thread *td __unused, void *dst __unused, + size_t *off __unused) +{ + +} + +int +elf_reloc_local(linker_file_t lf, Elf_Addr relocbase, const void *data, + int type, elf_lookup_fn lookup __unused) +{ + + panic("elf_reloc_local"); +} + +/* Process one elf relocation with addend. */ +int +elf_reloc(linker_file_t lf, Elf_Addr relocbase, const void *data, int type, + elf_lookup_fn lookup) +{ + + panic("elf_reloc"); +} + +int +elf_cpu_load_file(linker_file_t lf __unused) +{ + + return (0); +} + +int +elf_cpu_unload_file(linker_file_t lf __unused) +{ + + return (0); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/elf_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/exception.S =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/exception.S (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/exception.S (revision 281504) @@ -0,0 +1,198 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include "assym.s" + + .text + +.macro save_registers el + stp x28, x29, [sp, #-16]! + stp x26, x27, [sp, #-16]! + stp x24, x25, [sp, #-16]! + stp x22, x23, [sp, #-16]! + stp x20, x21, [sp, #-16]! + stp x18, x19, [sp, #-16]! + stp x16, x17, [sp, #-16]! + stp x14, x15, [sp, #-16]! + stp x12, x13, [sp, #-16]! + stp x10, x11, [sp, #-16]! + stp x8, x9, [sp, #-16]! + stp x6, x7, [sp, #-16]! + stp x4, x5, [sp, #-16]! + stp x2, x3, [sp, #-16]! + stp x0, x1, [sp, #-16]! + mrs x10, elr_el1 + mrs x11, spsr_el1 +.if \el == 0 + mrs x12, sp_el0 +.else + mov x12, sp +.endif + stp x10, x11, [sp, #-16]! + stp x12, lr, [sp, #-16]! + mrs x18, tpidr_el1 +.endm + +.macro restore_registers el + ldp x12, lr, [sp], #16 + ldp x10, x11, [sp], #16 +.if \el == 0 + msr sp_el0, x12 +.else + mov sp, x12 +.endif + msr spsr_el1, x11 + msr elr_el1, x10 + ldp x0, x1, [sp], #16 + ldp x2, x3, [sp], #16 + ldp x4, x5, [sp], #16 + ldp x6, x7, [sp], #16 + ldp x8, x9, [sp], #16 + ldp x10, x11, [sp], #16 + ldp x12, x13, [sp], #16 + ldp x14, x15, [sp], #16 + ldp x16, x17, [sp], #16 +.if \el == 0 + ldp x18, x19, [sp], #16 +.else + ldp xzr, x19, [sp], #16 +.endif + ldp x20, x21, [sp], #16 + ldp x22, x23, [sp], #16 + ldp x24, x25, [sp], #16 + ldp x26, x27, [sp], #16 + ldp x28, x29, [sp], #16 +.endm + +.macro do_ast + /* Disable interrupts */ + mrs x19, daif + msr daifset, #2 + + /* Read the current thread flags */ +1: ldr x1, [x18, #PC_CURTHREAD] /* Load curthread */ + ldr x2, [x1, #TD_FLAGS]! /* TODO: No need for the ! but clang fails without it */ + + /* Check if we have either bits set */ + mov x3, #((TDF_ASTPENDING|TDF_NEEDRESCHED) >> 8) + lsl x3, x3, #8 + and x2, x2, x3 + cmp x2, #0 + b.eq 2f + + /* Restore interrupts */ + msr daif, x19 + + /* handle the ast */ + mov x0, sp + bl _C_LABEL(ast) + + /* Disable interrupts */ + mrs x19, daif + msr daifset, #2 + +2: + /* Restore interrupts */ + msr daif, x19 +.endm + +handle_el1h_sync: + save_registers 1 + mov x0, sp + bl do_el1h_sync + restore_registers 1 + eret + +handle_el1h_irq: + save_registers 1 + mov x0, sp + bl arm_cpu_intr + restore_registers 1 + eret + +handle_el1h_error: + brk 0xf13 + +handle_el0_sync: + save_registers 0 + mov x0, sp + bl do_el0_sync + do_ast + restore_registers 0 + eret + +handle_el0_irq: + save_registers 0 + mov x0, sp + bl arm_cpu_intr + restore_registers 0 + eret + +handle_el0_error: + save_registers 0 + mov x0, sp + bl do_el0_error + brk 0xf23 + 1: b 1b + +.macro vempty + .align 7 + brk 0xfff + 1: b 1b +.endm + +.macro vector name + .align 7 + b handle_\name +.endm + + .align 11 + .globl exception_vectors +exception_vectors: + vempty /* Synchronous EL1t */ + vempty /* IRQ EL1t */ + vempty /* FIQ EL1t */ + vempty /* Error EL1t */ + + vector el1h_sync /* Synchronous EL1h */ + vector el1h_irq /* IRQ EL1h */ + vempty /* FIQ EL1h */ + vector el1h_error /* Error EL1h */ + + vector el0_sync /* Synchronous 64-bit EL0 */ + vector el0_irq /* IRQ 64-bit EL0 */ + vempty /* FIQ 64-bit EL0 */ + vector el0_error /* Error 64-bit EL0 */ + + vempty /* Synchronous 32-bit EL0 */ + vempty /* IRQ 32-bit EL0 */ + vempty /* FIQ 32-bit EL0 */ + vempty /* Error 32-bit EL0 */ + Property changes on: user/ngie/more-tests/sys/arm64/arm64/exception.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/genassym.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/genassym.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/genassym.c (revision 281504) @@ -0,0 +1,61 @@ +/*- + * Copyright (c) 2004 Olivier Houchard + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); +#include +#include +#include +#include + +#include +#include +#include + +ASSYM(KERNBASE, KERNBASE); +ASSYM(TDF_ASTPENDING, TDF_ASTPENDING); +ASSYM(TDF_NEEDRESCHED, TDF_NEEDRESCHED); + +ASSYM(PCPU_SIZE, sizeof(struct pcpu)); +ASSYM(PC_CURPCB, offsetof(struct pcpu, pc_curpcb)); +ASSYM(PC_CURTHREAD, offsetof(struct pcpu, pc_curthread)); + +/* Size of pcb, rounded to keep stack alignment */ +ASSYM(PCB_SIZE, roundup2(sizeof(struct pcb), STACKALIGNBYTES + 1)); +ASSYM(PCB_REGS, offsetof(struct pcb, pcb_x)); +ASSYM(PCB_SP, offsetof(struct pcb, pcb_sp)); +ASSYM(PCB_L1ADDR, offsetof(struct pcb, pcb_l1addr)); +ASSYM(PCB_ONFAULT, offsetof(struct pcb, pcb_onfault)); + +ASSYM(SF_UC, offsetof(struct sigframe, sf_uc)); + +ASSYM(TD_PCB, offsetof(struct thread, td_pcb)); +ASSYM(TD_FLAGS, offsetof(struct thread, td_flags)); +ASSYM(TD_LOCK, offsetof(struct thread, td_lock)); + +ASSYM(TF_X, offsetof(struct trapframe, tf_x)); Property changes on: user/ngie/more-tests/sys/arm64/arm64/genassym.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/gic.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/gic.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/gic.c (revision 281504) @@ -0,0 +1,398 @@ +/*- + * Copyright (c) 2011 The FreeBSD Foundation + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Developed by Damjan Marion + * + * Based on OMAP4 GIC code by Ben Gray + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. The name of the company nor the name of the author may be used to + * endorse or promote products derived from this software without specific + * prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + + +#include "pic_if.h" + +/* We are using GICv2 register naming */ + +/* Distributor Registers */ +#define GICD_CTLR 0x000 /* v1 ICDDCR */ +#define GICD_TYPER 0x004 /* v1 ICDICTR */ +#define GICD_IIDR 0x008 /* v1 ICDIIDR */ +#define GICD_IGROUPR(n) (0x0080 + ((n) * 4)) /* v1 ICDISER */ +#define GICD_ISENABLER(n) (0x0100 + ((n) * 4)) /* v1 ICDISER */ +#define GICD_ICENABLER(n) (0x0180 + ((n) * 4)) /* v1 ICDICER */ +#define GICD_ISPENDR(n) (0x0200 + ((n) * 4)) /* v1 ICDISPR */ +#define GICD_ICPENDR(n) (0x0280 + ((n) * 4)) /* v1 ICDICPR */ +#define GICD_ICACTIVER(n) (0x0380 + ((n) * 4)) /* v1 ICDABR */ +#define GICD_IPRIORITYR(n) (0x0400 + ((n) * 4)) /* v1 ICDIPR */ +#define GICD_ITARGETSR(n) (0x0800 + ((n) * 4)) /* v1 ICDIPTR */ +#define GICD_ICFGR(n) (0x0C00 + ((n) * 4)) /* v1 ICDICFR */ +#define GICD_SGIR(n) (0x0F00 + ((n) * 4)) /* v1 ICDSGIR */ + +/* CPU Registers */ +#define GICC_CTLR 0x0000 /* v1 ICCICR */ +#define GICC_PMR 0x0004 /* v1 ICCPMR */ +#define GICC_BPR 0x0008 /* v1 ICCBPR */ +#define GICC_IAR 0x000C /* v1 ICCIAR */ +#define GICC_EOIR 0x0010 /* v1 ICCEOIR */ +#define GICC_RPR 0x0014 /* v1 ICCRPR */ +#define GICC_HPPIR 0x0018 /* v1 ICCHPIR */ +#define GICC_ABPR 0x001C /* v1 ICCABPR */ +#define GICC_IIDR 0x00FC /* v1 ICCIIDR*/ + +#define GIC_FIRST_IPI 0 /* Irqs 0-15 are SGIs/IPIs. */ +#define GIC_LAST_IPI 15 +#define GIC_FIRST_PPI 16 /* Irqs 16-31 are private (per */ +#define GIC_LAST_PPI 31 /* core) peripheral interrupts. */ +#define GIC_FIRST_SPI 32 /* Irqs 32+ are shared peripherals. */ + +/* First bit is a polarity bit (0 - low, 1 - high) */ +#define GICD_ICFGR_POL_LOW (0 << 0) +#define GICD_ICFGR_POL_HIGH (1 << 0) +#define GICD_ICFGR_POL_MASK 0x1 +/* Second bit is a trigger bit (0 - level, 1 - edge) */ +#define GICD_ICFGR_TRIG_LVL (0 << 1) +#define GICD_ICFGR_TRIG_EDGE (1 << 1) +#define GICD_ICFGR_TRIG_MASK 0x2 + +struct arm_gic_softc { + device_t gic_dev; + struct resource * gic_res[3]; + bus_space_tag_t gic_c_bst; + bus_space_tag_t gic_d_bst; + bus_space_handle_t gic_c_bsh; + bus_space_handle_t gic_d_bsh; + uint8_t ver; + struct mtx mutex; + uint32_t nirqs; +}; + +static struct resource_spec arm_gic_spec[] = { + { SYS_RES_MEMORY, 0, RF_ACTIVE }, /* Distributor registers */ + { SYS_RES_MEMORY, 1, RF_ACTIVE }, /* CPU Interrupt Intf. registers */ + { -1, 0 } +}; + +static struct arm_gic_softc *arm_gic_sc = NULL; + +#define gic_c_read_4(_sc, _reg) \ + bus_space_read_4((_sc)->gic_c_bst, (_sc)->gic_c_bsh, (_reg)) +#define gic_c_write_4(_sc, _reg, _val) \ + bus_space_write_4((_sc)->gic_c_bst, (_sc)->gic_c_bsh, (_reg), (_val)) +#define gic_d_read_4(_sc, _reg) \ + bus_space_read_4((_sc)->gic_d_bst, (_sc)->gic_d_bsh, (_reg)) +#define gic_d_write_4(_sc, _reg, _val) \ + bus_space_write_4((_sc)->gic_d_bst, (_sc)->gic_d_bsh, (_reg), (_val)) + +static pic_dispatch_t gic_dispatch; +static pic_eoi_t gic_eoi; +static pic_mask_t gic_mask_irq; +static pic_unmask_t gic_unmask_irq; + +static struct ofw_compat_data compat_data[] = { + {"arm,gic", true}, /* Non-standard, used in FreeBSD dts. */ + {"arm,gic-400", true}, + {"arm,cortex-a15-gic", true}, + {"arm,cortex-a9-gic", true}, + {"arm,cortex-a7-gic", true}, + {"arm,arm11mp-gic", true}, + {"brcm,brahma-b15-gic", true}, + {NULL, false} +}; + +static int +arm_gic_probe(device_t dev) +{ + + if (!ofw_bus_status_okay(dev)) + return (ENXIO); + + if (!ofw_bus_search_compatible(dev, compat_data)->ocd_data) + return (ENXIO); + + device_set_desc(dev, "ARM Generic Interrupt Controller"); + return (BUS_PROBE_DEFAULT); +} + +#ifdef SMP +static void +gic_init_secondary(device_t dev) +{ + struct arm_gic_softc *sc = device_get_softc(dev); + int i; + + for (i = 0; i < sc->nirqs; i += 4) + gic_d_write_4(sc, GICD_IPRIORITYR(i >> 2), 0); + + /* Set all the interrupts to be in Group 0 (secure) */ + for (i = 0; i < sc->nirqs; i += 32) { + gic_d_write_4(sc, GICD_IGROUPR(i >> 5), 0); + } + + /* Enable CPU interface */ + gic_c_write_4(sc, GICC_CTLR, 1); + + /* Set priority mask register. */ + gic_c_write_4(sc, GICC_PMR, 0xff); + + /* Enable interrupt distribution */ + gic_d_write_4(sc, GICD_CTLR, 0x01); + + /* + * Activate the timer interrupts: virtual, secure, and non-secure. + */ + gic_d_write_4(sc, GICD_ISENABLER(27 >> 5), (1UL << (27 & 0x1F))); + gic_d_write_4(sc, GICD_ISENABLER(29 >> 5), (1UL << (29 & 0x1F))); + gic_d_write_4(sc, GICD_ISENABLER(30 >> 5), (1UL << (30 & 0x1F))); +} +#endif + +static int +arm_gic_attach(device_t dev) +{ + struct arm_gic_softc *sc; + int i; + uint32_t icciidr; + + if (arm_gic_sc) + return (ENXIO); + + sc = device_get_softc(dev); + + if (bus_alloc_resources(dev, arm_gic_spec, sc->gic_res)) { + device_printf(dev, "could not allocate resources\n"); + return (ENXIO); + } + + sc->gic_dev = dev; + arm_gic_sc = sc; + + /* Initialize mutex */ + mtx_init(&sc->mutex, "GIC lock", "", MTX_SPIN); + + /* Distributor Interface */ + sc->gic_d_bst = rman_get_bustag(sc->gic_res[0]); + sc->gic_d_bsh = rman_get_bushandle(sc->gic_res[0]); + + /* CPU Interface */ + sc->gic_c_bst = rman_get_bustag(sc->gic_res[1]); + sc->gic_c_bsh = rman_get_bushandle(sc->gic_res[1]); + + /* Disable interrupt forwarding to the CPU interface */ + gic_d_write_4(sc, GICD_CTLR, 0x00); + + /* Get the number of interrupts */ + sc->nirqs = gic_d_read_4(sc, GICD_TYPER); + sc->nirqs = 32 * ((sc->nirqs & 0x1f) + 1); + + arm_register_root_pic(dev, sc->nirqs); + + icciidr = gic_c_read_4(sc, GICC_IIDR); + device_printf(dev,"pn 0x%x, arch 0x%x, rev 0x%x, implementer 0x%x irqs %u\n", + icciidr>>20, (icciidr>>16) & 0xF, (icciidr>>12) & 0xf, + (icciidr & 0xfff), sc->nirqs); + + /* Set all global interrupts to be level triggered, active low. */ + for (i = 32; i < sc->nirqs; i += 16) { + gic_d_write_4(sc, GICD_ICFGR(i >> 4), 0x00000000); + } + + /* Disable all interrupts. */ + for (i = 32; i < sc->nirqs; i += 32) { + gic_d_write_4(sc, GICD_ICENABLER(i >> 5), 0xFFFFFFFF); + } + + for (i = 0; i < sc->nirqs; i += 4) { + gic_d_write_4(sc, GICD_IPRIORITYR(i >> 2), 0); + gic_d_write_4(sc, GICD_ITARGETSR(i >> 2), + 1 << 0 | 1 << 8 | 1 << 16 | 1 << 24); + } + + /* Set all the interrupts to be in Group 0 (secure) */ + for (i = 0; i < sc->nirqs; i += 32) { + gic_d_write_4(sc, GICD_IGROUPR(i >> 5), 0); + } + + /* Enable CPU interface */ + gic_c_write_4(sc, GICC_CTLR, 1); + + /* Set priority mask register. */ + gic_c_write_4(sc, GICC_PMR, 0xff); + + /* Enable interrupt distribution */ + gic_d_write_4(sc, GICD_CTLR, 0x01); + + return (0); +} + +static void gic_dispatch(device_t dev, struct trapframe *frame) +{ + struct arm_gic_softc *sc = device_get_softc(dev); + uint32_t active_irq; + int first = 1; + + while (1) { + active_irq = gic_c_read_4(sc, GICC_IAR); + + /* + * Immediatly EOIR the SGIs, because doing so requires the other + * bits (ie CPU number), not just the IRQ number, and we do not + * have this information later. + */ + + if ((active_irq & 0x3ff) <= GIC_LAST_IPI) + gic_c_write_4(sc, GICC_EOIR, active_irq); + active_irq &= 0x3FF; + + if (active_irq == 0x3FF) { + if (first) + printf("Spurious interrupt detected\n"); + return; + } + + arm_dispatch_intr(active_irq, frame); + first = 0; + } +} + +static void +gic_eoi(device_t dev, u_int irq) +{ + struct arm_gic_softc *sc = device_get_softc(dev); + + gic_c_write_4(sc, GICC_EOIR, irq); +} + +void +gic_mask_irq(device_t dev, u_int irq) +{ + struct arm_gic_softc *sc = device_get_softc(dev); + + gic_d_write_4(sc, GICD_ICENABLER(irq >> 5), (1UL << (irq & 0x1F))); + gic_c_write_4(sc, GICC_EOIR, irq); +} + +void +gic_unmask_irq(device_t dev, u_int irq) +{ + struct arm_gic_softc *sc = device_get_softc(dev); + + gic_d_write_4(sc, GICD_ISENABLER(irq >> 5), (1UL << (irq & 0x1F))); +} + +#ifdef SMP +static void +gic_ipi_send(device_t dev, cpuset_t cpus, u_int ipi) +{ + struct arm_gic_softc *sc = device_get_softc(dev); + uint32_t val = 0, i; + + for (i = 0; i < MAXCPU; i++) + if (CPU_ISSET(i, &cpus)) + val |= 1 << (16 + i); + + gic_d_write_4(sc, GICD_SGIR(0), val | ipi); +} + +static int +arm_gic_ipi_read(device_t dev, int i) +{ + + if (i != -1) { + /* + * The intr code will automagically give the frame pointer + * if the interrupt argument is 0. + */ + if ((unsigned int)i > 16) + return (0); + return (i); + } + + return (0x3ff); +} + +static void +arm_gic_ipi_clear(device_t dev, int ipi) +{ + /* no-op */ +} +#endif + +static device_method_t arm_gic_methods[] = { + /* Device interface */ + DEVMETHOD(device_probe, arm_gic_probe), + DEVMETHOD(device_attach, arm_gic_attach), + + /* pic_if */ + DEVMETHOD(pic_dispatch, gic_dispatch), + DEVMETHOD(pic_eoi, gic_eoi), + DEVMETHOD(pic_mask, gic_mask_irq), + DEVMETHOD(pic_unmask, gic_unmask_irq), + +#ifdef SMP + DEVMETHOD(pic_init_secondary, gic_init_secondary), + DEVMETHOD(pic_ipi_send, gic_ipi_send), +#endif + + { 0, 0 } +}; + +static driver_t arm_gic_driver = { + "gic", + arm_gic_methods, + sizeof(struct arm_gic_softc), +}; + +static devclass_t arm_gic_devclass; + +EARLY_DRIVER_MODULE(gic, simplebus, arm_gic_driver, arm_gic_devclass, 0, 0, + BUS_PASS_INTERRUPT + BUS_PASS_ORDER_MIDDLE); +EARLY_DRIVER_MODULE(gic, ofwbus, arm_gic_driver, arm_gic_devclass, 0, 0, + BUS_PASS_INTERRUPT + BUS_PASS_ORDER_MIDDLE); Property changes on: user/ngie/more-tests/sys/arm64/arm64/gic.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/identcpu.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/identcpu.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/identcpu.c (revision 281504) @@ -0,0 +1,199 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * Portions of this software were developed by Semihalf + * under sponsorship of the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include + +#include +#include + +char machine[] = "arm64"; + +SYSCTL_STRING(_hw, HW_MACHINE, machine, CTLFLAG_RD, machine, 0, + "Machine class"); + +/* + * Per-CPU affinity as provided in MPIDR_EL1 + * Indexed by CPU number in logical order selected by the system. + * Relevant fields can be extracetd using CPU_AFFn macros, + * Aff3.Aff2.Aff1.Aff0 construct a unique CPU address in the system. + * + * Fields used by us: + * Aff1 - Cluster number + * Aff0 - CPU number in Aff1 cluster + */ +uint64_t __cpu_affinity[MAXCPU]; + +#define CPU_IMPL_ARM 0x41 +#define CPU_IMPL_BROADCOM 0x42 +#define CPU_IMPL_CAVIUM 0x43 +#define CPU_IMPL_DEC 0x44 +#define CPU_IMPL_INFINEON 0x49 +#define CPU_IMPL_FREESCALE 0x4D +#define CPU_IMPL_NVIDIA 0x4E +#define CPU_IMPL_APM 0x50 +#define CPU_IMPL_QUALCOMM 0x51 +#define CPU_IMPL_MARVELL 0x56 +#define CPU_IMPL_INTEL 0x69 + +#define CPU_PART_THUNDER 0x0A1 +#define CPU_PART_FOUNDATION 0xD00 +#define CPU_PART_CORTEX_A53 0xD03 +#define CPU_PART_CORTEX_A57 0xD07 + +#define CPU_IMPL(midr) (((midr) >> 24) & 0xff) +#define CPU_PART(midr) (((midr) >> 4) & 0xfff) +#define CPU_VAR(midr) (((midr) >> 20) & 0xf) +#define CPU_REV(midr) (((midr) >> 0) & 0xf) + +struct cpu_desc { + u_int cpu_impl; + u_int cpu_part_num; + u_int cpu_variant; + u_int cpu_revision; + const char *cpu_impl_name; + const char *cpu_part_name; +}; + +struct cpu_desc cpu_desc[MAXCPU]; + +struct cpu_parts { + u_int part_id; + const char *part_name; +}; +#define CPU_PART_NONE { 0, "Unknown Processor" } + +struct cpu_implementers { + u_int impl_id; + const char *impl_name; + /* + * Part number is implementation defined + * so each vendor will have its own set of values and names. + */ + const struct cpu_parts *cpu_parts; +}; +#define CPU_IMPLEMENTER_NONE { 0, "Unknown Implementer", cpu_parts_none } + +/* + * Per-implementer table of (PartNum, CPU Name) pairs. + */ +/* ARM Ltd. */ +static const struct cpu_parts cpu_parts_arm[] = { + { 0xD00, "Foundation-Model" }, + { 0xD03, "Cortex-A53" }, + { 0xD07, "Cortex-A57" }, + CPU_PART_NONE, +}; +/* Cavium */ +static const struct cpu_parts cpu_parts_cavium[] = { + { 0x0A1, "Thunder" }, + CPU_PART_NONE, +}; + +/* Unknown */ +static const struct cpu_parts cpu_parts_none[] = { + CPU_PART_NONE, +}; + +/* + * Implementers table. + */ +const struct cpu_implementers cpu_implementers[] = { + { CPU_IMPL_ARM, "ARM", cpu_parts_arm }, + { CPU_IMPL_BROADCOM, "Broadcom", cpu_parts_none }, + { CPU_IMPL_CAVIUM, "Cavium", cpu_parts_cavium }, + { CPU_IMPL_DEC, "DEC", cpu_parts_none }, + { CPU_IMPL_INFINEON, "IFX", cpu_parts_none }, + { CPU_IMPL_FREESCALE, "Freescale", cpu_parts_none }, + { CPU_IMPL_NVIDIA, "NVIDIA", cpu_parts_none }, + { CPU_IMPL_APM, "APM", cpu_parts_none }, + { CPU_IMPL_QUALCOMM, "Qualcomm", cpu_parts_none }, + { CPU_IMPL_MARVELL, "Marvell", cpu_parts_none }, + { CPU_IMPL_INTEL, "Intel", cpu_parts_none }, + CPU_IMPLEMENTER_NONE, +}; + +void identify_cpu(void); + +void +identify_cpu(void) +{ + u_int midr; + u_int impl_id; + u_int part_id; + u_int cpu; + uint64_t mpidr; + size_t i; + const struct cpu_parts *cpu_partsp = NULL; + + cpu = PCPU_GET(cpuid); + midr = get_midr(); + + impl_id = CPU_IMPL(midr); + for (i = 0; i < nitems(cpu_implementers); i++) { + if (impl_id == cpu_implementers[i].impl_id || + cpu_implementers[i].impl_id == 0) { + cpu_desc[cpu].cpu_impl = impl_id; + cpu_desc[cpu].cpu_impl_name = cpu_implementers[i].impl_name; + cpu_partsp = cpu_implementers[i].cpu_parts; + break; + } + } + + part_id = CPU_PART(midr); + for (i = 0; &cpu_partsp[i] != NULL; i++) { + if (part_id == cpu_partsp[i].part_id || + cpu_partsp[i].part_id == 0) { + cpu_desc[cpu].cpu_part_num = part_id; + cpu_desc[cpu].cpu_part_name = cpu_partsp[i].part_name; + break; + } + } + + printf("CPU: %s %s r%dp%d\n", cpu_desc[cpu].cpu_impl_name, + cpu_desc[cpu].cpu_part_name, CPU_VAR(midr), CPU_REV(midr)); + + /* + * Save affinity for the boot CPU. + * (CPU0 in the internal system enumeration. + */ + mpidr = get_mpidr(); + CPU_AFFINITY(0) = mpidr & CPU_AFF_MASK; + + if (bootverbose) + printf("CPU%u affinity: %u.%u.%u.%u\n", 0, CPU_AFF0(mpidr), + CPU_AFF1(mpidr), CPU_AFF2(mpidr), CPU_AFF3(mpidr)); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/identcpu.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/in_cksum.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/in_cksum.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/in_cksum.c (revision 281504) @@ -0,0 +1,241 @@ +/* $NetBSD: in_cksum.c,v 1.7 1997/09/02 13:18:15 thorpej Exp $ */ + +/*- + * Copyright (c) 1988, 1992, 1993 + * The Regents of the University of California. All rights reserved. + * Copyright (c) 1996 + * Matt Thomas + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by the University of + * California, Berkeley and its contributors. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * @(#)in_cksum.c 8.1 (Berkeley) 6/10/93 + */ + +#include /* RCS ID & Copyright macro defns */ +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +/* + * Checksum routine for Internet Protocol family headers + * (Portable Alpha version). + * + * This routine is very heavily used in the network + * code and should be modified for each CPU to be as fast as possible. + */ + +#define ADDCARRY(x) (x > 65535 ? x -= 65535 : x) +#define REDUCE32 \ + { \ + q_util.q = sum; \ + sum = q_util.s[0] + q_util.s[1] + q_util.s[2] + q_util.s[3]; \ + } +#define REDUCE16 \ + { \ + q_util.q = sum; \ + l_util.l = q_util.s[0] + q_util.s[1] + q_util.s[2] + q_util.s[3]; \ + sum = l_util.s[0] + l_util.s[1]; \ + ADDCARRY(sum); \ + } + +static const u_int32_t in_masks[] = { + /*0 bytes*/ /*1 byte*/ /*2 bytes*/ /*3 bytes*/ + 0x00000000, 0x000000FF, 0x0000FFFF, 0x00FFFFFF, /* offset 0 */ + 0x00000000, 0x0000FF00, 0x00FFFF00, 0xFFFFFF00, /* offset 1 */ + 0x00000000, 0x00FF0000, 0xFFFF0000, 0xFFFF0000, /* offset 2 */ + 0x00000000, 0xFF000000, 0xFF000000, 0xFF000000, /* offset 3 */ +}; + +union l_util { + u_int16_t s[2]; + u_int32_t l; +}; +union q_util { + u_int16_t s[4]; + u_int32_t l[2]; + u_int64_t q; +}; + +static u_int64_t +in_cksumdata(const void *buf, int len) +{ + const u_int32_t *lw = (const u_int32_t *) buf; + u_int64_t sum = 0; + u_int64_t prefilled; + int offset; + union q_util q_util; + + if ((3 & (long) lw) == 0 && len == 20) { + sum = (u_int64_t) lw[0] + lw[1] + lw[2] + lw[3] + lw[4]; + REDUCE32; + return sum; + } + + if ((offset = 3 & (long) lw) != 0) { + const u_int32_t *masks = in_masks + (offset << 2); + lw = (u_int32_t *) (((long) lw) - offset); + sum = *lw++ & masks[len >= 3 ? 3 : len]; + len -= 4 - offset; + if (len <= 0) { + REDUCE32; + return sum; + } + } +#if 0 + /* + * Force to cache line boundary. + */ + offset = 32 - (0x1f & (long) lw); + if (offset < 32 && len > offset) { + len -= offset; + if (4 & offset) { + sum += (u_int64_t) lw[0]; + lw += 1; + } + if (8 & offset) { + sum += (u_int64_t) lw[0] + lw[1]; + lw += 2; + } + if (16 & offset) { + sum += (u_int64_t) lw[0] + lw[1] + lw[2] + lw[3]; + lw += 4; + } + } +#endif + /* + * access prefilling to start load of next cache line. + * then add current cache line + * save result of prefilling for loop iteration. + */ + prefilled = lw[0]; + while ((len -= 32) >= 4) { + u_int64_t prefilling = lw[8]; + sum += prefilled + lw[1] + lw[2] + lw[3] + + lw[4] + lw[5] + lw[6] + lw[7]; + lw += 8; + prefilled = prefilling; + } + if (len >= 0) { + sum += prefilled + lw[1] + lw[2] + lw[3] + + lw[4] + lw[5] + lw[6] + lw[7]; + lw += 8; + } else { + len += 32; + } + while ((len -= 16) >= 0) { + sum += (u_int64_t) lw[0] + lw[1] + lw[2] + lw[3]; + lw += 4; + } + len += 16; + while ((len -= 4) >= 0) { + sum += (u_int64_t) *lw++; + } + len += 4; + if (len > 0) + sum += (u_int64_t) (in_masks[len] & *lw); + REDUCE32; + return sum; +} + +u_short +in_addword(u_short a, u_short b) +{ + u_int64_t sum = a + b; + + ADDCARRY(sum); + return (sum); +} + +u_short +in_pseudo(u_int32_t a, u_int32_t b, u_int32_t c) +{ + u_int64_t sum; + union q_util q_util; + union l_util l_util; + + sum = (u_int64_t) a + b + c; + REDUCE16; + return (sum); +} + +u_short +in_cksum_skip(struct mbuf *m, int len, int skip) +{ + u_int64_t sum = 0; + int mlen = 0; + int clen = 0; + caddr_t addr; + union q_util q_util; + union l_util l_util; + + len -= skip; + for (; skip && m; m = m->m_next) { + if (m->m_len > skip) { + mlen = m->m_len - skip; + addr = mtod(m, caddr_t) + skip; + goto skip_start; + } else { + skip -= m->m_len; + } + } + + for (; m && len; m = m->m_next) { + if (m->m_len == 0) + continue; + mlen = m->m_len; + addr = mtod(m, caddr_t); +skip_start: + if (len < mlen) + mlen = len; + if ((clen ^ (long) addr) & 1) + sum += in_cksumdata(addr, mlen) << 8; + else + sum += in_cksumdata(addr, mlen); + + clen += mlen; + len -= mlen; + } + REDUCE16; + return (~sum & 0xffff); +} + +u_int in_cksum_hdr(const struct ip *ip) +{ + u_int64_t sum = in_cksumdata(ip, sizeof(struct ip)); + union q_util q_util; + union l_util l_util; + REDUCE16; + return (~sum & 0xffff); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/in_cksum.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/intr_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/intr_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/intr_machdep.c (revision 281504) @@ -0,0 +1,503 @@ +/*- + * Copyright (c) 1991 The Regents of the University of California. + * Copyright (c) 2002 Benno Rice. + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Semihalf under + * the sponsorship of the FreeBSD Foundation. + * + * This code is derived from software contributed by + * William Jolitz (Berkeley) and Benno Rice. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * form: src/sys/powerpc/powerpc/intr_machdep.c, r271712 2014/09/17 + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#ifdef SMP +#include +#endif + +#include "pic_if.h" + +#define MAX_STRAY_LOG 5 +#define INTRNAME_LEN (MAXCOMLEN + 1) + +#define NIRQS 1024 /* Maximum number of interrupts in the system */ + +static MALLOC_DEFINE(M_INTR, "intr", "Interrupt Services"); + +/* + * Linked list of interrupts that have been set-up. + * Each element holds the interrupt description + * and has to be allocated and freed dynamically. + */ +static SLIST_HEAD(, arm64_intr_entry) irq_slist_head = + SLIST_HEAD_INITIALIZER(irq_slist_head); + +struct arm64_intr_entry { + SLIST_ENTRY(arm64_intr_entry) entries; + struct intr_event *i_event; + + enum intr_trigger i_trig; + enum intr_polarity i_pol; + + u_int i_hw_irq; /* Physical interrupt number */ + u_int i_cntidx; /* Index in intrcnt table */ + u_int i_handlers; /* Allocated handlers */ + u_long *i_cntp; /* Interrupt hit counter */ +}; + +/* Counts and names for statistics - see sys/sys/interrupt.h */ +/* Tables are indexed by i_cntidx */ +u_long intrcnt[NIRQS]; +char intrnames[NIRQS * INTRNAME_LEN]; +size_t sintrcnt = sizeof(intrcnt); +size_t sintrnames = sizeof(intrnames); + +static u_int intrcntidx; /* Current index into intrcnt table */ +static u_int arm64_nintrs; /* Max interrupts number of the root PIC */ +static u_int arm64_nstray; /* Number of received stray interrupts */ +static device_t root_pic; /* PIC device for all incoming interrupts */ +static device_t msi_pic; /* Device which handles MSI/MSI-X interrupts */ +static struct mtx intr_list_lock; + +static void +intr_init(void *dummy __unused) +{ + + mtx_init(&intr_list_lock, "intr sources lock", NULL, MTX_DEF); +} +SYSINIT(intr_init, SI_SUB_INTR, SI_ORDER_FIRST, intr_init, NULL); + +/* + * Helper routines. + */ + +/* Set interrupt name for statistics */ +static void +intrcnt_setname(const char *name, u_int idx) +{ + + snprintf(&intrnames[idx * INTRNAME_LEN], INTRNAME_LEN, "%-*s", + INTRNAME_LEN - 1, name); +} + +/* + * Get intr structure for the given interrupt number. + * Allocate one if this is the first time. + * (Similar to ppc's intr_lookup() but without actual + * lookup since irq number is an index in arm64_intrs[]). + */ +static struct arm64_intr_entry * +intr_acquire(u_int hw_irq) +{ + struct arm64_intr_entry *intr; + + mtx_lock(&intr_list_lock); + + SLIST_FOREACH(intr, &irq_slist_head, entries) { + if (intr->i_hw_irq == hw_irq) { + break; + } + } + if (intr != NULL) + goto out; + + /* Do not alloc another intr when max number of IRQs has been reached */ + if (intrcntidx >= NIRQS) + goto out; + + intr = malloc(sizeof(*intr), M_INTR, M_NOWAIT); + if (intr == NULL) + goto out; + + intr->i_event = NULL; + intr->i_handlers = 0; + intr->i_trig = INTR_TRIGGER_CONFORM; + intr->i_pol = INTR_POLARITY_CONFORM; + intr->i_cntidx = atomic_fetchadd_int(&intrcntidx, 1); + intr->i_cntp = &intrcnt[intr->i_cntidx]; + intr->i_hw_irq = hw_irq; + SLIST_INSERT_HEAD(&irq_slist_head, intr, entries); +out: + mtx_unlock(&intr_list_lock); + return intr; +} + +static void +intr_pre_ithread(void *arg) +{ + struct arm64_intr_entry *intr = arg; + + PIC_PRE_ITHREAD(root_pic, intr->i_hw_irq); +} + +static void +intr_post_ithread(void *arg) +{ + struct arm64_intr_entry *intr = arg; + + PIC_POST_ITHREAD(root_pic, intr->i_hw_irq); +} + +static void +intr_post_filter(void *arg) +{ + struct arm64_intr_entry *intr = arg; + + PIC_POST_FILTER(root_pic, intr->i_hw_irq); +} + +/* + * Register PIC driver. + * This is intended to be called by the very first PIC driver + * at the end of the successful attach. + * Note that during boot this can be called after first references + * to bus_setup_intr() so it is required to not use root_pic if it + * is not 100% safe. + */ +void +arm_register_root_pic(device_t dev, u_int nirq) +{ + + KASSERT(root_pic == NULL, ("Unable to set the pic twice")); + KASSERT(nirq <= NIRQS, ("PIC is trying to handle too many IRQs")); + + arm64_nintrs = NIRQS; /* Number of IRQs limited only by array size */ + root_pic = dev; +} + +/* Register device which allocates MSI interrupts */ +void +arm_register_msi_pic(device_t dev) +{ + + KASSERT(msi_pic == NULL, ("Unable to set msi_pic twice")); + msi_pic = dev; +} + +int +arm_alloc_msi(device_t pci_dev, int count, int *irqs) +{ + + return PIC_ALLOC_MSI(msi_pic, pci_dev, count, irqs); +} + +int +arm_release_msi(device_t pci_dev, int count, int *irqs) +{ + + return PIC_RELEASE_MSI(msi_pic, pci_dev, count, irqs); +} + +int +arm_map_msi(device_t pci_dev, int irq, uint64_t *addr, uint32_t *data) +{ + + return PIC_MAP_MSI(msi_pic, pci_dev, irq, addr, data); +} + +int +arm_alloc_msix(device_t pci_dev, int *irq) +{ + + return PIC_ALLOC_MSIX(msi_pic, pci_dev, irq); +} + +int +arm_release_msix(device_t pci_dev, int irq) +{ + + return PIC_RELEASE_MSIX(msi_pic, pci_dev, irq); +} + + +int +arm_map_msix(device_t pci_dev, int irq, uint64_t *addr, uint32_t *data) +{ + + return PIC_MAP_MSIX(msi_pic, pci_dev, irq, addr, data); +} + +/* + * Finalize interrupts bring-up (should be called from configure_final()). + * Enables all interrupts registered by bus_setup_intr() during boot + * as well as unlocks interrups reception on primary CPU. + */ +int +arm_enable_intr(void) +{ + struct arm64_intr_entry *intr; + + if (root_pic == NULL) + panic("Cannot enable interrupts. No PIC configured"); + + /* + * Iterate through all possible interrupts and perform + * configuration if the interrupt is registered. + */ + SLIST_FOREACH(intr, &irq_slist_head, entries) { + /* + * XXX: In case we allowed to set up interrupt whose number + * exceeds maximum number of interrupts for the root PIC + * disable it and print proper error message. + * + * This can happen only when calling bus_setup_intr() + * before the interrupt controller is attached. + */ + if (intr->i_cntidx >= arm64_nintrs) { + /* Better fail when IVARIANTS enabled */ + KASSERT(0, ("%s: Interrupt %u cannot be handled by the " + "registered PIC. Max interrupt number: %u", __func__, + intr->i_cntidx, arm64_nintrs - 1)); + /* Print message and disable otherwise */ + printf("ERROR: Cannot enable irq %u. Disabling.\n", + intr->i_cntidx); + PIC_MASK(root_pic, intr->i_hw_irq); + } + + if (intr->i_trig != INTR_TRIGGER_CONFORM || + intr->i_pol != INTR_POLARITY_CONFORM) { + PIC_CONFIG(root_pic, intr->i_hw_irq, + intr->i_trig, intr->i_pol); + } + + if (intr->i_handlers > 0) + PIC_UNMASK(root_pic, intr->i_hw_irq); + + } + /* Enable interrupt reception on this CPU */ + intr_enable(); + + return (0); +} + +int +arm_setup_intr(const char *name, driver_filter_t *filt, driver_intr_t handler, + void *arg, u_int hw_irq, enum intr_type flags, void **cookiep) +{ + struct arm64_intr_entry *intr; + int error; + + intr = intr_acquire(hw_irq); + if (intr == NULL) + return (ENOMEM); + + /* + * Watch out for interrupts' numbers. + * If this is a system boot then don't allow to overfill interrupts + * table (the interrupts will be deconfigured in arm_enable_intr()). + */ + if (intr->i_cntidx >= NIRQS) + return (EINVAL); + + if (intr->i_event == NULL) { + error = intr_event_create(&intr->i_event, (void *)intr, 0, + hw_irq, intr_pre_ithread, intr_post_ithread, + intr_post_filter, NULL, "irq%u", hw_irq); + if (error) + return (error); + } + + error = intr_event_add_handler(intr->i_event, name, filt, handler, arg, + intr_priority(flags), flags, cookiep); + + if (!error) { + mtx_lock(&intr_list_lock); + intrcnt_setname(intr->i_event->ie_fullname, intr->i_cntidx); + intr->i_handlers++; + + if (!cold && intr->i_handlers == 1) { + if (intr->i_trig != INTR_TRIGGER_CONFORM || + intr->i_pol != INTR_POLARITY_CONFORM) { + PIC_CONFIG(root_pic, intr->i_hw_irq, intr->i_trig, + intr->i_pol); + } + + PIC_UNMASK(root_pic, intr->i_hw_irq); + } + mtx_unlock(&intr_list_lock); + } + + return (error); +} + +int +arm_teardown_intr(void *cookie) +{ + struct arm64_intr_entry *intr; + int error; + + intr = intr_handler_source(cookie); + error = intr_event_remove_handler(cookie); + if (!error) { + mtx_lock(&intr_list_lock); + intr->i_handlers--; + if (intr->i_handlers == 0) + PIC_MASK(root_pic, intr->i_hw_irq); + intrcnt_setname(intr->i_event->ie_fullname, intr->i_cntidx); + mtx_unlock(&intr_list_lock); + } + + return (error); +} + +int +arm_config_intr(u_int hw_irq, enum intr_trigger trig, enum intr_polarity pol) +{ + struct arm64_intr_entry *intr; + + intr = intr_acquire(hw_irq); + if (intr == NULL) + return (ENOMEM); + + intr->i_trig = trig; + intr->i_pol = pol; + + if (!cold && root_pic != NULL) + PIC_CONFIG(root_pic, intr->i_hw_irq, trig, pol); + + return (0); +} + +void +arm_dispatch_intr(u_int hw_irq, struct trapframe *tf) +{ + struct arm64_intr_entry *intr; + + SLIST_FOREACH(intr, &irq_slist_head, entries) { + if (intr->i_hw_irq == hw_irq) { + break; + } + } + + if (intr == NULL) + goto stray; + + (*intr->i_cntp)++; + + if (!intr_event_handle(intr->i_event, tf)) + return; + +stray: + if (arm64_nstray < MAX_STRAY_LOG) { + arm64_nstray++; + printf("Stray IRQ %u\n", hw_irq); + if (arm64_nstray >= MAX_STRAY_LOG) { + printf("Got %d stray IRQs. Not logging anymore.\n", + MAX_STRAY_LOG); + } + } + + if (intr != NULL) + PIC_MASK(root_pic, intr->i_hw_irq); +} + +void +arm_cpu_intr(struct trapframe *tf) +{ + + critical_enter(); + PIC_DISPATCH(root_pic, tf); + critical_exit(); +} + +#ifdef SMP +void +arm_setup_ipihandler(driver_filter_t *filt, u_int ipi) +{ + + /* ARM64TODO: The hard coded 16 will be fixed with am_intrng */ + arm_setup_intr("ipi", filt, NULL, (void *)((uintptr_t)ipi | 1<<16), ipi + 16, + INTR_TYPE_MISC | INTR_EXCL, NULL); + arm_unmask_ipi(ipi); +} + +void +arm_unmask_ipi(u_int ipi) +{ + PIC_UNMASK(root_pic, ipi + 16); +} + +void +arm_init_secondary(void) +{ + + PIC_INIT_SECONDARY(root_pic); +} + +/* Sending IPI */ +void +ipi_all_but_self(u_int ipi) +{ + + /* ARM64TODO: We should support this */ + panic("ipi_all_but_self"); +} + +void +ipi_cpu(int cpu, u_int ipi) +{ + cpuset_t cpus; + + CPU_ZERO(&cpus); + CPU_SET(cpu, &cpus); + + /* ARM64TODO: This will be fixed with arm_intrng */ + ipi += 16; + + CTR2(KTR_SMP, "ipi_cpu: cpu: %d, ipi: %x", cpu, ipi); + PIC_IPI_SEND(root_pic, cpus, ipi); +} + +void +ipi_selected(cpuset_t cpus, u_int ipi) +{ + + /* ARM64TODO: This will be fixed with arm_intrng */ + ipi += 16; + + CTR1(KTR_SMP, "ipi_selected: ipi: %x", ipi); + PIC_IPI_SEND(root_pic, cpus, ipi); +} + +#endif Property changes on: user/ngie/more-tests/sys/arm64/arm64/intr_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/locore.S =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/locore.S (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/locore.S (revision 281504) @@ -0,0 +1,544 @@ +/*- + * Copyright (c) 2012-2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#include "assym.s" +#include +#include +#include +#include +#include +#include + +#define VIRT_BITS 39 + + .globl kernbase + .set kernbase, KERNBASE + +#define DEVICE_MEM 0 +#define NORMAL_UNCACHED 1 +#define NORMAL_MEM 2 + +/* + * We assume: + * MMU on with an identity map, or off + * D-Cache: off + * I-Cache: on or off + * We are loaded at a 2MiB aligned address + */ + +#define INIT_STACK_SIZE (PAGE_SIZE * 4) + + .text + .globl _start +_start: + /* Drop to EL1 */ + bl drop_to_el1 + + /* + * Disable the MMU. We may have entered the kernel with it on and + * will need to update the tables later. If this has been set up + * with anything other than a VA == PA map then this will fail, + * but in this case the code to find where we are running from + * would have also failed. + */ + dsb sy + mrs x2, sctlr_el1 + bic x2, x2, SCTLR_M + msr sctlr_el1, x2 + isb + + + /* Get the virt -> phys offset */ + bl get_virt_delta + + /* + * At this point: + * x29 = PA - VA + * x28 = Our physical load address + */ + + /* Create the page tables */ + bl create_pagetables + + /* + * At this point: + * x27 = TTBR0 table + * x26 = TTBR1 table + */ + + /* Enable the mmu */ + bl start_mmu + + /* Jump to the virtual address space */ + ldr x15, .Lvirtdone + br x15 + +virtdone: + /* Set up the stack */ + adr x25, initstack_end + mov sp, x25 + sub sp, sp, #PCB_SIZE + + /* Zero the BSS */ + ldr x15, .Lbss + ldr x14, .Lend +1: + str xzr, [x15], #8 + cmp x15, x14 + b.lo 1b + + /* Backup the module pointer */ + mov x1, x0 + + /* Make the page table base a virtual address */ + sub x26, x26, x29 + + sub sp, sp, #(64 * 4) + mov x0, sp + + /* Degate the delda so it is VA -> PA */ + neg x29, x29 + + str x1, [x0] /* modulep */ + str x26, [x0, 8] /* kern_l1pt */ + str x29, [x0, 16] /* kern_delta */ + str x25, [x0, 24] /* kern_stack */ + + /* trace back starts here */ + mov fp, #0 + /* Branch to C code */ + bl initarm + bl mi_startup + + /* We should not get here */ + brk 0 + + .align 3 +.Lvirtdone: + .quad virtdone +.Lbss: + .quad __bss_start +.Lend: + .quad _end + +/* + * If we are started in EL2, configure the required hypervisor + * registers and drop to EL1. + */ +drop_to_el1: + mrs x1, CurrentEL + lsr x1, x1, #2 + cmp x1, #0x2 + b.eq 1f + ret +1: + /* Configure the Hypervisor */ + mov x2, #(HCR_RW) + msr hcr_el2, x2 + + /* Load the Virtualization Process ID Register */ + mrs x2, midr_el1 + msr vpidr_el2, x2 + + /* Load the Virtualization Multiprocess ID Register */ + mrs x2, mpidr_el1 + msr vmpidr_el2, x2 + + /* Set the bits that need to be 1 in sctlr_el1 */ + ldr x2, .Lsctlr_res1 + msr sctlr_el1, x2 + + /* Don't trap to EL2 for exceptions */ + mov x2, #CPTR_RES1 + msr cptr_el2, x2 + + /* Don't trap to EL2 for CP15 traps */ + msr hstr_el2, xzr + + /* Hypervisor trap functions */ + adr x2, hyp_vectors + msr vbar_el2, x2 + + mov x2, #(PSR_F | PSR_I | PSR_A | PSR_D | PSR_M_EL1h) + msr spsr_el2, x2 + + /* Set the address to return to our return address */ + msr elr_el2, x30 + + eret + + .align 3 +.Lsctlr_res1: + .quad SCTLR_RES1 + +#define VECT_EMPTY \ + .align 7; \ + 1: b 1b + + .align 11 +hyp_vectors: + VECT_EMPTY /* Synchronous EL2t */ + VECT_EMPTY /* IRQ EL2t */ + VECT_EMPTY /* FIQ EL2t */ + VECT_EMPTY /* Error EL2t */ + + VECT_EMPTY /* Synchronous EL2h */ + VECT_EMPTY /* IRQ EL2h */ + VECT_EMPTY /* FIQ EL2h */ + VECT_EMPTY /* Error EL2h */ + + VECT_EMPTY /* Synchronous 64-bit EL1 */ + VECT_EMPTY /* IRQ 64-bit EL1 */ + VECT_EMPTY /* FIQ 64-bit EL1 */ + VECT_EMPTY /* Error 64-bit EL1 */ + + VECT_EMPTY /* Synchronous 32-bit EL1 */ + VECT_EMPTY /* IRQ 32-bit EL1 */ + VECT_EMPTY /* FIQ 32-bit EL1 */ + VECT_EMPTY /* Error 32-bit EL1 */ + +/* + * Get the delta between the physical address we were loaded to and the + * virtual address we expect to run from. This is used when building the + * initial page table. + */ +get_virt_delta: + /* Load the physical address of virt_map */ + adr x29, virt_map + /* Load the virtual address of virt_map stored in virt_map */ + ldr x28, [x29] + /* Find PA - VA as PA' = VA' - VA + PA = VA' + (PA - VA) = VA' + x29 */ + sub x29, x29, x28 + /* Find the load address for the kernel */ + mov x28, #(KERNBASE) + add x28, x28, x29 + ret + + .align 3 +virt_map: + .quad virt_map + +/* + * This builds the page tables containing the identity map, and the kernel + * virtual map. + * + * It relys on: + * We were loaded to an address that is on a 2MiB boundary + * All the memory must not cross a 1GiB boundaty + * x28 contains the physical address we were loaded from + * + * TODO: This is out of date. + * There are at least 5 pages before that address for the page tables + * The pages used are: + * - The identity (PA = VA) table (TTBR0) + * - The Kernel L1 table (TTBR1)(not yet) + * - The PA != VA L2 table to jump into (not yet) + * - The FDT L2 table (not yet) + */ +create_pagetables: + /* Save the Link register */ + mov x5, x30 + + /* Clean the page table */ + adr x6, pagetable + mov x26, x6 + adr x27, pagetable_end +1: + stp xzr, xzr, [x6], #16 + stp xzr, xzr, [x6], #16 + stp xzr, xzr, [x6], #16 + stp xzr, xzr, [x6], #16 + cmp x6, x27 + b.lo 1b + + /* + * Build the TTBR1 maps. + */ + + /* Find the size of the kernel */ + mov x6, #(KERNBASE) + ldr x7, .Lend + /* Find the end - begin */ + sub x8, x7, x6 + /* Get the number of l2 pages to allocate, rounded down */ + lsr x10, x8, #(L2_SHIFT) + /* Add 4 MiB for any rounding above and the module data */ + add x10, x10, #2 + + /* Create the kernel space L2 table */ + mov x6, x26 + mov x7, #NORMAL_MEM + mov x8, #(KERNBASE & L2_BLOCK_MASK) + mov x9, x28 + bl build_block_pagetable + + /* Move to the l1 table */ + add x26, x26, #PAGE_SIZE + + /* Link the l1 -> l2 table */ + mov x9, x6 + mov x6, x26 + bl link_l1_pagetable + + + /* + * Build the TTBR0 maps. + */ + add x27, x26, #PAGE_SIZE + +#if defined(SOCDEV_PA) && defined(SOCDEV_VA) + /* Create a table for the UART */ + mov x6, x27 /* The initial page table */ + mov x7, #DEVICE_MEM + mov x8, #(SOCDEV_VA) /* VA start */ + mov x9, #(SOCDEV_PA) /* PA start */ + bl build_section_pagetable +#endif + + /* Create the VA = PA map */ + mov x6, x27 /* The initial page table */ + mov x7, #NORMAL_UNCACHED /* Uncached as it's only needed early on */ + mov x9, x27 + mov x8, x9 /* VA start (== PA start) */ + bl build_section_pagetable + + /* Restore the Link register */ + mov x30, x5 + ret + +/* + * Builds a 1 GiB page table entry + * x6 = L1 table + * x7 = Type (0 = Device, 1 = Normal) + * x8 = VA start + * x9 = PA start (trashed) + * x11, x12 and x13 are trashed + */ +build_section_pagetable: + /* + * Build the L1 table entry. + */ + /* Find the table index */ + lsr x11, x8, #L1_SHIFT + and x11, x11, #Ln_ADDR_MASK + + /* Build the L1 block entry */ + lsl x12, x7, #2 + orr x12, x12, #L1_BLOCK + orr x12, x12, #(ATTR_AF) + + /* Only use the output address bits */ + lsr x9, x9, #L1_SHIFT + orr x12, x12, x9, lsl #L1_SHIFT + + /* Store the entry */ + str x12, [x6, x11, lsl #3] + + ret + +/* + * Builds an L1 -> L2 table descriptor + * + * This is a link for a 1GiB block of memory with up to 2MiB regions mapped + * within it by build_block_pagetable. + * + * x6 = L1 table + * x8 = Virtual Address + * x9 = L2 PA (trashed) + * x11, x12 and x13 are trashed + */ +link_l1_pagetable: + /* + * Link an L1 -> L2 table entry. + */ + /* Find the table index */ + lsr x11, x8, #L1_SHIFT + and x11, x11, #Ln_ADDR_MASK + + /* Build the L1 block entry */ + mov x12, #L1_TABLE + + /* Only use the output address bits */ + lsr x9, x9, #12 + orr x12, x12, x9, lsl #12 + + /* Store the entry */ + str x12, [x6, x11, lsl #3] + + ret + +/* + * Builds count 2 MiB page table entry + * x6 = L2 table + * x7 = Type (0 = Device, 1 = Normal) + * x8 = VA start + * x9 = PA start (trashed) + * x10 = Entry count (TODO) + * x11, x12 and x13 are trashed + */ +build_block_pagetable: + /* + * Build the L2 table entry. + */ + /* Find the table index */ + lsr x11, x8, #L2_SHIFT + and x11, x11, #Ln_ADDR_MASK + + /* Build the L2 block entry */ + lsl x12, x7, #2 + orr x12, x12, #L2_BLOCK + orr x12, x12, #(ATTR_AF) + + /* Only use the output address bits */ + lsr x9, x9, #L2_SHIFT + + /* Set the physical address for this virtual address */ +1: orr x12, x12, x9, lsl #L2_SHIFT + + /* Store the entry */ + str x12, [x6, x11, lsl #3] + + /* Clear the address bits */ + and x12, x12, #ATTR_MASK_L + + sub x10, x10, #1 + add x11, x11, #1 + add x9, x9, #1 + cbnz x10, 1b + +2: ret + +start_mmu: + dsb sy + + /* Load the exception vectors */ + ldr x2, =exception_vectors + msr vbar_el1, x2 + + /* Load ttbr0 and ttbr1 */ + msr ttbr0_el1, x27 + msr ttbr1_el1, x26 + isb + + /* Clear the Monitor Debug System control register */ + msr mdscr_el1, xzr + + /* Invalidate the TLB */ + tlbi vmalle1is + + ldr x2, mair + msr mair_el1, x2 + + /* Setup TCR according to PARange bits from ID_AA64MMFR0_EL1 */ + ldr x2, tcr + mrs x3, id_aa64mmfr0_el1 + bfi x2, x3, #32, #3 + msr tcr_el1, x2 + + /* Setup SCTLR */ + ldr x2, sctlr_set + ldr x3, sctlr_clear + mrs x1, sctlr_el1 + bic x1, x1, x3 /* Clear the required bits */ + orr x1, x1, x2 /* Set the required bits */ + msr sctlr_el1, x1 + isb + + ret + + .align 3 +mair: + /* Device Normal, no cache Normal, write-back */ + .quad MAIR_ATTR(0x00, 0) | MAIR_ATTR(0x44, 1) | MAIR_ATTR(0xff, 2) +tcr: + .quad (TCR_TxSZ(64 - VIRT_BITS) | TCR_ASID_16 | TCR_TG1_4K) +sctlr_set: + /* Bits to set */ + .quad (SCTLR_UCI | SCTLR_nTWE | SCTLR_nTWI | SCTLR_UCT | SCTLR_DZE | \ + SCTLR_I | SCTLR_SED | SCTLR_C | SCTLR_M) +sctlr_clear: + /* Bits to clear */ + .quad (SCTLR_EE | SCTLR_EOE | SCTLR_WXN | SCTLR_UMA | SCTLR_ITD | \ + SCTLR_THEE | SCTLR_CP15BEN | SCTLR_SA0 | SCTLR_SA | SCTLR_A) + + .globl abort +abort: + b abort + + //.section .init_pagetable + .align 12 /* 4KiB aligned */ + /* + * 3 initial tables (in the following order): + * L2 for kernel (High addresses) + * L1 for kernel + * L1 for user (Low addresses) + */ +pagetable: + .space PAGE_SIZE +pagetable_l1_ttbr1: + .space PAGE_SIZE +pagetable_l1_ttbr0: + .space PAGE_SIZE +pagetable_end: + +el2_pagetable: + .space PAGE_SIZE + + .globl init_pt_va +init_pt_va: + .quad pagetable /* XXX: Keep page tables VA */ + + .align 4 +initstack: + .space (PAGE_SIZE * KSTACK_PAGES) +initstack_end: + + +ENTRY(sigcode) + mov x0, sp + add x0, x0, #SF_UC + +1: + mov x8, #SYS_sigreturn + svc 0 + + /* sigreturn failed, exit */ + mov x8, #SYS_exit + svc 0 + + b 1b +END(sigcode) + /* This may be copied to the stack, keep it 16-byte aligned */ + .align 3 +esigcode: + + .data + .align 3 + .global szsigcode +szsigcode: + .quad esigcode - sigcode Property changes on: user/ngie/more-tests/sys/arm64/arm64/locore.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/machdep.c (revision 281504) @@ -0,0 +1,854 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include "opt_platform.h" + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef VFP +#include +#endif + +#ifdef FDT +#include +#include +#endif + +struct pcpu __pcpu[MAXCPU]; + +static struct trapframe proc0_tf; + +vm_paddr_t phys_avail[PHYS_AVAIL_SIZE + 2]; +vm_paddr_t dump_avail[PHYS_AVAIL_SIZE + 2]; + +int early_boot = 1; +int cold = 1; +long realmem = 0; +long Maxmem = 0; + +#define PHYSMAP_SIZE (2 * (VM_PHYSSEG_MAX - 1)) +vm_paddr_t physmap[PHYSMAP_SIZE]; +u_int physmap_idx; + +struct kva_md_info kmi; + +int64_t dcache_line_size; /* The minimum D cache line size */ +int64_t icache_line_size; /* The minimum I cache line size */ +int64_t idcache_line_size; /* The minimum cache line size */ + +static void +cpu_startup(void *dummy) +{ + + identify_cpu(); + + vm_ksubmap_init(&kmi); + bufinit(); + vm_pager_bufferinit(); +} + +SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); + +void +bzero(void *buf, size_t len) +{ + uint8_t *p; + + p = buf; + while(len-- > 0) + *p++ = 0; +} + +int +fill_regs(struct thread *td, struct reg *regs) +{ + struct trapframe *frame; + + frame = td->td_frame; + regs->sp = frame->tf_sp; + regs->lr = frame->tf_lr; + regs->elr = frame->tf_elr; + regs->spsr = frame->tf_spsr; + + memcpy(regs->x, frame->tf_x, sizeof(regs->x)); + + return (0); +} + +int +set_regs(struct thread *td, struct reg *regs) +{ + struct trapframe *frame; + + frame = td->td_frame; + frame->tf_sp = regs->sp; + frame->tf_lr = regs->lr; + frame->tf_elr = regs->elr; + frame->tf_spsr = regs->spsr; + + memcpy(frame->tf_x, regs->x, sizeof(frame->tf_x)); + + return (0); +} + +int +fill_fpregs(struct thread *td, struct fpreg *regs) +{ +#ifdef VFP + struct pcb *pcb; + + pcb = td->td_pcb; + if ((pcb->pcb_fpflags & PCB_FP_STARTED) != 0) { + /* + * If we have just been running VFP instructions we will + * need to save the state to memcpy it below. + */ + vfp_save_state(td); + + memcpy(regs->fp_q, pcb->pcb_vfp, sizeof(regs->fp_q)); + regs->fp_cr = pcb->pcb_fpcr; + regs->fp_sr = pcb->pcb_fpsr; + } else +#endif + memset(regs->fp_q, 0, sizeof(regs->fp_q)); + return (0); +} + +int +set_fpregs(struct thread *td, struct fpreg *regs) +{ +#ifdef VFP + struct pcb *pcb; + + pcb = td->td_pcb; + memcpy(pcb->pcb_vfp, regs->fp_q, sizeof(regs->fp_q)); + pcb->pcb_fpcr = regs->fp_cr; + pcb->pcb_fpsr = regs->fp_sr; +#endif + return (0); +} + +int +fill_dbregs(struct thread *td, struct dbreg *regs) +{ + + panic("fill_dbregs"); +} + +int +set_dbregs(struct thread *td, struct dbreg *regs) +{ + + panic("set_dbregs"); +} + +int +ptrace_set_pc(struct thread *td, u_long addr) +{ + + panic("ptrace_set_pc"); + return (0); +} + +int +ptrace_single_step(struct thread *td) +{ + + /* TODO; */ + return (0); +} + +int +ptrace_clear_single_step(struct thread *td) +{ + + /* TODO; */ + return (0); +} + +void +exec_setregs(struct thread *td, struct image_params *imgp, u_long stack) +{ + struct trapframe *tf = td->td_frame; + + memset(tf, 0, sizeof(struct trapframe)); + + tf->tf_sp = stack; + tf->tf_lr = imgp->entry_addr; + tf->tf_elr = imgp->entry_addr; +} + +/* Sanity check these are the same size, they will be memcpy'd to and fro */ +CTASSERT(sizeof(((struct trapframe *)0)->tf_x) == + sizeof((struct gpregs *)0)->gp_x); +CTASSERT(sizeof(((struct trapframe *)0)->tf_x) == + sizeof((struct reg *)0)->x); + +int +get_mcontext(struct thread *td, mcontext_t *mcp, int clear_ret) +{ + struct trapframe *tf = td->td_frame; + + if (clear_ret & GET_MC_CLEAR_RET) + mcp->mc_gpregs.gp_x[0] = 0; + else + mcp->mc_gpregs.gp_x[0] = tf->tf_x[0]; + + memcpy(&mcp->mc_gpregs.gp_x[1], &tf->tf_x[1], + sizeof(mcp->mc_gpregs.gp_x[1]) * (nitems(mcp->mc_gpregs.gp_x) - 1)); + + mcp->mc_gpregs.gp_sp = tf->tf_sp; + mcp->mc_gpregs.gp_lr = tf->tf_lr; + mcp->mc_gpregs.gp_elr = tf->tf_elr; + mcp->mc_gpregs.gp_spsr = tf->tf_spsr; + + return (0); +} + +int +set_mcontext(struct thread *td, mcontext_t *mcp) +{ + struct trapframe *tf = td->td_frame; + + memcpy(tf->tf_x, mcp->mc_gpregs.gp_x, sizeof(tf->tf_x)); + + tf->tf_sp = mcp->mc_gpregs.gp_sp; + tf->tf_lr = mcp->mc_gpregs.gp_lr; + tf->tf_elr = mcp->mc_gpregs.gp_elr; + tf->tf_spsr = mcp->mc_gpregs.gp_spsr; + + return (0); +} + +static void +get_fpcontext(struct thread *td, mcontext_t *mcp) +{ +#ifdef VFP + struct pcb *curpcb; + + critical_enter(); + + curpcb = curthread->td_pcb; + + if ((curpcb->pcb_fpflags & PCB_FP_STARTED) != 0) { + /* + * If we have just been running VFP instructions we will + * need to save the state to memcpy it below. + */ + vfp_save_state(td); + + memcpy(mcp->mc_fpregs.fp_q, curpcb->pcb_vfp, + sizeof(mcp->mc_fpregs)); + mcp->mc_fpregs.fp_cr = curpcb->pcb_fpcr; + mcp->mc_fpregs.fp_sr = curpcb->pcb_fpsr; + mcp->mc_fpregs.fp_flags = curpcb->pcb_fpflags; + mcp->mc_flags |= _MC_FP_VALID; + } + + critical_exit(); +#endif +} + +static void +set_fpcontext(struct thread *td, mcontext_t *mcp) +{ +#ifdef VFP + struct pcb *curpcb; + + critical_enter(); + + if ((mcp->mc_flags & _MC_FP_VALID) != 0) { + curpcb = curthread->td_pcb; + + /* + * Discard any vfp state for the current thread, we + * are about to override it. + */ + vfp_discard(td); + + memcpy(curpcb->pcb_vfp, mcp->mc_fpregs.fp_q, + sizeof(mcp->mc_fpregs)); + curpcb->pcb_fpcr = mcp->mc_fpregs.fp_cr; + curpcb->pcb_fpsr = mcp->mc_fpregs.fp_sr; + curpcb->pcb_fpflags = mcp->mc_fpregs.fp_flags; + } + + critical_exit(); +#endif +} + +void +cpu_idle(int busy) +{ + + spinlock_enter(); + if (!busy) + cpu_idleclock(); + if (!sched_runnable()) + __asm __volatile( + "dsb sy \n" + "wfi \n"); + if (!busy) + cpu_activeclock(); + spinlock_exit(); +} + +void +cpu_halt(void) +{ + + panic("cpu_halt"); +} + +/* + * Flush the D-cache for non-DMA I/O so that the I-cache can + * be made coherent later. + */ +void +cpu_flush_dcache(void *ptr, size_t len) +{ + + /* TBD */ +} + +/* Get current clock frequency for the given CPU ID. */ +int +cpu_est_clockrate(int cpu_id, uint64_t *rate) +{ + + panic("cpu_est_clockrate"); +} + +void +cpu_pcpu_init(struct pcpu *pcpu, int cpuid, size_t size) +{ +} + +void +spinlock_enter(void) +{ + struct thread *td; + register_t daif; + + td = curthread; + if (td->td_md.md_spinlock_count == 0) { + daif = intr_disable(); + td->td_md.md_spinlock_count = 1; + td->td_md.md_saved_daif = daif; + } else + td->td_md.md_spinlock_count++; + critical_enter(); +} + +void +spinlock_exit(void) +{ + struct thread *td; + register_t daif; + + td = curthread; + critical_exit(); + daif = td->td_md.md_saved_daif; + td->td_md.md_spinlock_count--; + if (td->td_md.md_spinlock_count == 0) + intr_restore(daif); +} + +#ifndef _SYS_SYSPROTO_H_ +struct sigreturn_args { + ucontext_t *ucp; +}; +#endif + +int +sys_sigreturn(struct thread *td, struct sigreturn_args *uap) +{ + ucontext_t uc; + uint32_t spsr; + + if (uap == NULL) + return (EFAULT); + if (copyin(uap->sigcntxp, &uc, sizeof(uc))) + return (EFAULT); + + spsr = uc.uc_mcontext.mc_gpregs.gp_spsr; + if ((spsr & PSR_M_MASK) != PSR_M_EL0t || + (spsr & (PSR_F | PSR_I | PSR_A | PSR_D)) != 0) + return (EINVAL); + + set_mcontext(td, &uc.uc_mcontext); + set_fpcontext(td, &uc.uc_mcontext); + + /* Restore signal mask. */ + kern_sigprocmask(td, SIG_SETMASK, &uc.uc_sigmask, NULL, 0); + + return (EJUSTRETURN); +} + +/* + * Construct a PCB from a trapframe. This is called from kdb_trap() where + * we want to start a backtrace from the function that caused us to enter + * the debugger. We have the context in the trapframe, but base the trace + * on the PCB. The PCB doesn't have to be perfect, as long as it contains + * enough for a backtrace. + */ +void +makectx(struct trapframe *tf, struct pcb *pcb) +{ + int i; + + for (i = 0; i < PCB_LR; i++) + pcb->pcb_x[i] = tf->tf_x[i]; + + pcb->pcb_x[PCB_LR] = tf->tf_lr; + pcb->pcb_pc = tf->tf_elr; + pcb->pcb_sp = tf->tf_sp; +} + +void +sendsig(sig_t catcher, ksiginfo_t *ksi, sigset_t *mask) +{ + struct thread *td; + struct proc *p; + struct trapframe *tf; + struct sigframe *fp, frame; + struct sigacts *psp; + int code, onstack, sig; + + td = curthread; + p = td->td_proc; + PROC_LOCK_ASSERT(p, MA_OWNED); + + sig = ksi->ksi_signo; + code = ksi->ksi_code; + psp = p->p_sigacts; + mtx_assert(&psp->ps_mtx, MA_OWNED); + + tf = td->td_frame; + onstack = sigonstack(tf->tf_sp); + + CTR4(KTR_SIG, "sendsig: td=%p (%s) catcher=%p sig=%d", td, p->p_comm, + catcher, sig); + + /* Allocate and validate space for the signal handler context. */ + if ((td->td_pflags & TDP_ALTSTACK) != 0 && !onstack && + SIGISMEMBER(psp->ps_sigonstack, sig)) { + fp = (struct sigframe *)(td->td_sigstk.ss_sp + + td->td_sigstk.ss_size); +#if defined(COMPAT_43) + td->td_sigstk.ss_flags |= SS_ONSTACK; +#endif + } else { + fp = (struct sigframe *)td->td_frame->tf_sp; + } + + /* Make room, keeping the stack aligned */ + fp--; + fp = (struct sigframe *)STACKALIGN(fp); + + /* Fill in the frame to copy out */ + get_mcontext(td, &frame.sf_uc.uc_mcontext, 0); + get_fpcontext(td, &frame.sf_uc.uc_mcontext); + frame.sf_si = ksi->ksi_info; + frame.sf_uc.uc_sigmask = *mask; + frame.sf_uc.uc_stack.ss_flags = (td->td_pflags & TDP_ALTSTACK) ? + ((onstack) ? SS_ONSTACK : 0) : SS_DISABLE; + frame.sf_uc.uc_stack = td->td_sigstk; + mtx_unlock(&psp->ps_mtx); + PROC_UNLOCK(td->td_proc); + + /* Copy the sigframe out to the user's stack. */ + if (copyout(&frame, fp, sizeof(*fp)) != 0) { + /* Process has trashed its stack. Kill it. */ + CTR2(KTR_SIG, "sendsig: sigexit td=%p fp=%p", td, fp); + PROC_LOCK(p); + sigexit(td, SIGILL); + } + + /* Translate the signal if appropriate. */ + if (p->p_sysent->sv_sigtbl && sig <= p->p_sysent->sv_sigsize) + sig = p->p_sysent->sv_sigtbl[_SIG_IDX(sig)]; + + tf->tf_x[0]= sig; + tf->tf_x[1] = (register_t)&fp->sf_si; + tf->tf_x[2] = (register_t)&fp->sf_uc; + + tf->tf_elr = (register_t)catcher; + tf->tf_sp = (register_t)fp; + tf->tf_lr = (register_t)(PS_STRINGS - *(p->p_sysent->sv_szsigcode)); + + CTR3(KTR_SIG, "sendsig: return td=%p pc=%#x sp=%#x", td, tf->tf_elr, + tf->tf_sp); + + PROC_LOCK(p); + mtx_lock(&psp->ps_mtx); +} + +static void +init_proc0(vm_offset_t kstack) +{ + struct pcpu *pcpup = &__pcpu[0]; + + proc_linkup0(&proc0, &thread0); + thread0.td_kstack = kstack; + thread0.td_pcb = (struct pcb *)(thread0.td_kstack) - 1; + thread0.td_pcb->pcb_fpflags = 0; + thread0.td_pcb->pcb_vfpcpu = UINT_MAX; + thread0.td_frame = &proc0_tf; + pcpup->pc_curpcb = thread0.td_pcb; +} + +typedef struct { + uint32_t type; + uint64_t phys_start; + uint64_t virt_start; + uint64_t num_pages; + uint64_t attr; +} EFI_MEMORY_DESCRIPTOR; + +static int +add_physmap_entry(uint64_t base, uint64_t length, vm_paddr_t *physmap, + u_int *physmap_idxp) +{ + u_int i, insert_idx, _physmap_idx; + + _physmap_idx = *physmap_idxp; + + if (length == 0) + return (1); + + /* + * Find insertion point while checking for overlap. Start off by + * assuming the new entry will be added to the end. + */ + insert_idx = _physmap_idx; + for (i = 0; i <= _physmap_idx; i += 2) { + if (base < physmap[i + 1]) { + if (base + length <= physmap[i]) { + insert_idx = i; + break; + } + if (boothowto & RB_VERBOSE) + printf( + "Overlapping memory regions, ignoring second region\n"); + return (1); + } + } + + /* See if we can prepend to the next entry. */ + if (insert_idx <= _physmap_idx && + base + length == physmap[insert_idx]) { + physmap[insert_idx] = base; + return (1); + } + + /* See if we can append to the previous entry. */ + if (insert_idx > 0 && base == physmap[insert_idx - 1]) { + physmap[insert_idx - 1] += length; + return (1); + } + + _physmap_idx += 2; + *physmap_idxp = _physmap_idx; + if (_physmap_idx == PHYSMAP_SIZE) { + printf( + "Too many segments in the physical address map, giving up\n"); + return (0); + } + + /* + * Move the last 'N' entries down to make room for the new + * entry if needed. + */ + for (i = _physmap_idx; i > insert_idx; i -= 2) { + physmap[i] = physmap[i - 2]; + physmap[i + 1] = physmap[i - 1]; + } + + /* Insert the new entry. */ + physmap[insert_idx] = base; + physmap[insert_idx + 1] = base + length; + return (1); +} + +#define efi_next_descriptor(ptr, size) \ + ((struct efi_md *)(((uint8_t *) ptr) + size)) + +static void +add_efi_map_entries(struct efi_map_header *efihdr, vm_paddr_t *physmap, + u_int *physmap_idxp) +{ + struct efi_md *map, *p; + const char *type; + size_t efisz; + int ndesc, i; + + static const char *types[] = { + "Reserved", + "LoaderCode", + "LoaderData", + "BootServicesCode", + "BootServicesData", + "RuntimeServicesCode", + "RuntimeServicesData", + "ConventionalMemory", + "UnusableMemory", + "ACPIReclaimMemory", + "ACPIMemoryNVS", + "MemoryMappedIO", + "MemoryMappedIOPortSpace", + "PalCode" + }; + + /* + * Memory map data provided by UEFI via the GetMemoryMap + * Boot Services API. + */ + efisz = (sizeof(struct efi_map_header) + 0xf) & ~0xf; + map = (struct efi_md *)((uint8_t *)efihdr + efisz); + + if (efihdr->descriptor_size == 0) + return; + ndesc = efihdr->memory_size / efihdr->descriptor_size; + + if (boothowto & RB_VERBOSE) + printf("%23s %12s %12s %8s %4s\n", + "Type", "Physical", "Virtual", "#Pages", "Attr"); + + for (i = 0, p = map; i < ndesc; i++, + p = efi_next_descriptor(p, efihdr->descriptor_size)) { + if (boothowto & RB_VERBOSE) { + if (p->md_type <= EFI_MD_TYPE_PALCODE) + type = types[p->md_type]; + else + type = ""; + printf("%23s %012lx %12p %08lx ", type, p->md_phys, + p->md_virt, p->md_pages); + if (p->md_attr & EFI_MD_ATTR_UC) + printf("UC "); + if (p->md_attr & EFI_MD_ATTR_WC) + printf("WC "); + if (p->md_attr & EFI_MD_ATTR_WT) + printf("WT "); + if (p->md_attr & EFI_MD_ATTR_WB) + printf("WB "); + if (p->md_attr & EFI_MD_ATTR_UCE) + printf("UCE "); + if (p->md_attr & EFI_MD_ATTR_WP) + printf("WP "); + if (p->md_attr & EFI_MD_ATTR_RP) + printf("RP "); + if (p->md_attr & EFI_MD_ATTR_XP) + printf("XP "); + if (p->md_attr & EFI_MD_ATTR_RT) + printf("RUNTIME"); + printf("\n"); + } + + switch (p->md_type) { + case EFI_MD_TYPE_CODE: + case EFI_MD_TYPE_DATA: + case EFI_MD_TYPE_BS_CODE: + case EFI_MD_TYPE_BS_DATA: + case EFI_MD_TYPE_FREE: + /* + * We're allowed to use any entry with these types. + */ + break; + default: + continue; + } + + if (!add_physmap_entry(p->md_phys, (p->md_pages * PAGE_SIZE), + physmap, physmap_idxp)) + break; + } +} + +#ifdef FDT +static void +try_load_dtb(caddr_t kmdp) +{ + vm_offset_t dtbp; + + dtbp = MD_FETCH(kmdp, MODINFOMD_DTBP, vm_offset_t); + if (dtbp == (vm_offset_t)NULL) { + printf("ERROR loading DTB\n"); + return; + } + + if (OF_install(OFW_FDT, 0) == FALSE) + panic("Cannot install FDT"); + + if (OF_init((void *)dtbp) != 0) + panic("OF_init failed with the found device tree"); +} +#endif + +static void +cache_setup(void) +{ + int dcache_line_shift, icache_line_shift; + uint32_t ctr_el0; + + ctr_el0 = READ_SPECIALREG(ctr_el0); + + /* Read the log2 words in each D cache line */ + dcache_line_shift = CTR_DLINE_SIZE(ctr_el0); + /* Get the D cache line size */ + dcache_line_size = sizeof(int) << dcache_line_shift; + + /* And the same for the I cache */ + icache_line_shift = CTR_ILINE_SIZE(ctr_el0); + icache_line_size = sizeof(int) << icache_line_shift; + + idcache_line_size = MIN(dcache_line_size, icache_line_size); +} + +void +initarm(struct arm64_bootparams *abp) +{ + struct efi_map_header *efihdr; + struct pcpu *pcpup; + vm_offset_t lastaddr; + caddr_t kmdp; + vm_paddr_t mem_len; + int i; + + /* Set the module data location */ + preload_metadata = (caddr_t)(uintptr_t)(abp->modulep); + + /* Find the kernel address */ + kmdp = preload_search_by_type("elf kernel"); + if (kmdp == NULL) + kmdp = preload_search_by_type("elf64 kernel"); + + boothowto = MD_FETCH(kmdp, MODINFOMD_HOWTO, int); + kern_envp = MD_FETCH(kmdp, MODINFOMD_ENVP, char *); + +#ifdef FDT + try_load_dtb(kmdp); +#endif + + /* Find the address to start allocating from */ + lastaddr = MD_FETCH(kmdp, MODINFOMD_KERNEND, vm_offset_t); + + /* Load the physical memory ranges */ + physmap_idx = 0; + efihdr = (struct efi_map_header *)preload_search_info(kmdp, + MODINFO_METADATA | MODINFOMD_EFI_MAP); + add_efi_map_entries(efihdr, physmap, &physmap_idx); + + /* Print the memory map */ + mem_len = 0; + for (i = 0; i < physmap_idx; i += 2) + mem_len += physmap[i + 1] - physmap[i]; + + /* Set the pcpu data, this is needed by pmap_bootstrap */ + pcpup = &__pcpu[0]; + pcpu_init(pcpup, 0, sizeof(struct pcpu)); + + /* + * Set the pcpu pointer with a backup in tpidr_el1 to be + * loaded when entering the kernel from userland. + */ + __asm __volatile( + "mov x18, %0 \n" + "msr tpidr_el1, %0" :: "r"(pcpup)); + + PCPU_SET(curthread, &thread0); + + /* Do basic tuning, hz etc */ + init_param1(); + + cache_setup(); + + /* Bootstrap enough of pmap to enter the kernel proper */ + pmap_bootstrap(abp->kern_l1pt, KERNBASE - abp->kern_delta, + lastaddr - KERNBASE); + + arm_devmap_bootstrap(0, NULL); + + cninit(); + + init_proc0(abp->kern_stack); + msgbufinit(msgbufp, msgbufsize); + mutex_init(); + init_param2(physmem); + + dbg_monitor_init(); + kdb_init(); + + early_boot = 0; +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/mem.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/mem.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/mem.c (revision 281504) @@ -0,0 +1,47 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include + +#include + +struct mem_range_softc mem_range_softc; + +int +memrw(struct cdev *dev, struct uio *uio, int flags) +{ + + panic("memrw"); +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/mem.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/minidump_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/minidump_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/minidump_machdep.c (revision 281504) @@ -0,0 +1,50 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include "opt_watchdog.h" + +#include +#include +#include +#include +#include + +#include + +int +minidumpsys(struct dumperinfo *di) +{ + + printf("minidumpsys\n"); + while (1); +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/minidump_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/nexus.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/nexus.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/nexus.c (revision 281504) @@ -0,0 +1,334 @@ +/*- + * Copyright 1998 Massachusetts Institute of Technology + * + * Permission to use, copy, modify, and distribute this software and + * its documentation for any purpose and without fee is hereby + * granted, provided that both the above copyright notice and this + * permission notice appear in all copies, that both the above + * copyright notice and this permission notice appear in all + * supporting documentation, and that the name of M.I.T. not be used + * in advertising or publicity pertaining to distribution of the + * software without specific, written prior permission. M.I.T. makes + * no representations about the suitability of this software for any + * purpose. It is provided "as is" without express or implied + * warranty. + * + * THIS SOFTWARE IS PROVIDED BY M.I.T. ``AS IS''. M.I.T. DISCLAIMS + * ALL EXPRESS OR IMPLIED WARRANTIES WITH REGARD TO THIS SOFTWARE, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT + * SHALL M.I.T. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +/* + * This code implements a `root nexus' for Arm Architecture + * machines. The function of the root nexus is to serve as an + * attachment point for both processors and buses, and to manage + * resources which are common to all of them. In particular, + * this code implements the core resource managers for interrupt + * requests, DMA requests (which rightfully should be a part of the + * ISA code but it's easier to do it here for now), I/O port addresses, + * and I/O memory address space. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#include +#include + +#include "opt_platform.h" + +#ifdef FDT +#include +#include "ofw_bus_if.h" +#endif + +extern struct bus_space memmap_bus; + +static MALLOC_DEFINE(M_NEXUSDEV, "nexusdev", "Nexus device"); + +struct nexus_device { + struct resource_list nx_resources; +}; + +#define DEVTONX(dev) ((struct nexus_device *)device_get_ivars(dev)) + +static struct rman mem_rman; + +static int nexus_probe(device_t); +static int nexus_attach(device_t); +static int nexus_print_child(device_t, device_t); +static device_t nexus_add_child(device_t, u_int, const char *, int); +static struct resource *nexus_alloc_resource(device_t, device_t, int, int *, + u_long, u_long, u_long, u_int); +static int nexus_activate_resource(device_t, device_t, int, int, + struct resource *); +static int nexus_config_intr(device_t dev, int irq, enum intr_trigger trig, + enum intr_polarity pol); +static int nexus_deactivate_resource(device_t, device_t, int, int, + struct resource *); + +static int nexus_setup_intr(device_t dev, device_t child, struct resource *res, + int flags, driver_filter_t *filt, driver_intr_t *intr, void *arg, void **cookiep); +static int nexus_teardown_intr(device_t, device_t, struct resource *, void *); + +#ifdef FDT +static int nexus_ofw_map_intr(device_t dev, device_t child, phandle_t iparent, + int icells, pcell_t *intr); +#endif + +static device_method_t nexus_methods[] = { + /* Device interface */ + DEVMETHOD(device_probe, nexus_probe), + DEVMETHOD(device_attach, nexus_attach), + /* Bus interface */ + DEVMETHOD(bus_print_child, nexus_print_child), + DEVMETHOD(bus_add_child, nexus_add_child), + DEVMETHOD(bus_alloc_resource, nexus_alloc_resource), + DEVMETHOD(bus_activate_resource, nexus_activate_resource), + DEVMETHOD(bus_config_intr, nexus_config_intr), + DEVMETHOD(bus_deactivate_resource, nexus_deactivate_resource), + DEVMETHOD(bus_setup_intr, nexus_setup_intr), + DEVMETHOD(bus_teardown_intr, nexus_teardown_intr), +#ifdef FDT + DEVMETHOD(ofw_bus_map_intr, nexus_ofw_map_intr), +#endif + { 0, 0 } +}; + +static devclass_t nexus_devclass; +static driver_t nexus_driver = { + "nexus", + nexus_methods, + 1 /* no softc */ +}; +DRIVER_MODULE(nexus, root, nexus_driver, nexus_devclass, 0, 0); + +static int +nexus_probe(device_t dev) +{ + + device_quiet(dev); /* suppress attach message for neatness */ + + return (BUS_PROBE_DEFAULT); +} + +static int +nexus_attach(device_t dev) +{ + + mem_rman.rm_start = 0; + mem_rman.rm_end = ~0ul; + mem_rman.rm_type = RMAN_ARRAY; + mem_rman.rm_descr = "I/O memory addresses"; + if (rman_init(&mem_rman) || rman_manage_region(&mem_rman, 0, ~0)) + panic("nexus_probe mem_rman"); + + /* + * First, deal with the children we know about already + */ + bus_generic_probe(dev); + bus_generic_attach(dev); + + return (0); +} + +static int +nexus_print_child(device_t bus, device_t child) +{ + int retval = 0; + + retval += bus_print_child_header(bus, child); + retval += printf("\n"); + + return (retval); +} + +static device_t +nexus_add_child(device_t bus, u_int order, const char *name, int unit) +{ + device_t child; + struct nexus_device *ndev; + + ndev = malloc(sizeof(struct nexus_device), M_NEXUSDEV, M_NOWAIT|M_ZERO); + if (!ndev) + return (0); + resource_list_init(&ndev->nx_resources); + + child = device_add_child_ordered(bus, order, name, unit); + + /* should we free this in nexus_child_detached? */ + device_set_ivars(child, ndev); + + return (child); +} + + +/* + * Allocate a resource on behalf of child. NB: child is usually going to be a + * child of one of our descendants, not a direct child of nexus0. + * (Exceptions include footbridge.) + */ +static struct resource * +nexus_alloc_resource(device_t bus, device_t child, int type, int *rid, + u_long start, u_long end, u_long count, u_int flags) +{ + struct resource *rv; + struct rman *rm; + int needactivate = flags & RF_ACTIVE; + + switch (type) { + case SYS_RES_MEMORY: + case SYS_RES_IOPORT: + rm = &mem_rman; + break; + + default: + return (0); + } + + rv = rman_reserve_resource(rm, start, end, count, flags, child); + if (rv == 0) + return (0); + + rman_set_rid(rv, *rid); + rman_set_bushandle(rv, rman_get_start(rv)); + + if (needactivate) { + if (bus_activate_resource(child, type, *rid, rv)) { + rman_release_resource(rv); + return (0); + } + } + + return (rv); +} + +static int +nexus_config_intr(device_t dev, int irq, enum intr_trigger trig, + enum intr_polarity pol) +{ + + return (arm_config_intr(irq, trig, pol)); +} + +static int +nexus_setup_intr(device_t dev, device_t child, struct resource *res, int flags, + driver_filter_t *filt, driver_intr_t *intr, void *arg, void **cookiep) +{ + int error; + + if ((rman_get_flags(res) & RF_SHAREABLE) == 0) + flags |= INTR_EXCL; + + /* We depend here on rman_activate_resource() being idempotent. */ + error = rman_activate_resource(res); + if (error) + return (error); + + error = arm_setup_intr(device_get_nameunit(child), filt, intr, + arg, rman_get_start(res), flags, cookiep); + + return (error); +} + +static int +nexus_teardown_intr(device_t dev, device_t child, struct resource *r, void *ih) +{ + + return (arm_teardown_intr(ih)); +} + +static int +nexus_activate_resource(device_t bus, device_t child, int type, int rid, + struct resource *r) +{ + int err; + bus_addr_t paddr; + bus_size_t psize; + bus_space_handle_t vaddr; + + if ((err = rman_activate_resource(r)) != 0) + return (err); + + /* + * If this is a memory resource, map it into the kernel. + */ + if (type == SYS_RES_MEMORY || type == SYS_RES_IOPORT) { + paddr = (bus_addr_t)rman_get_start(r); + psize = (bus_size_t)rman_get_size(r); + err = bus_space_map(&memmap_bus, paddr, psize, 0, &vaddr); + if (err != 0) { + rman_deactivate_resource(r); + return (err); + } + rman_set_bustag(r, &memmap_bus); + rman_set_virtual(r, (void *)vaddr); + rman_set_bushandle(r, vaddr); + } + return (0); +} + +static int +nexus_deactivate_resource(device_t bus, device_t child, int type, int rid, + struct resource *r) +{ + bus_size_t psize; + bus_space_handle_t vaddr; + + psize = (bus_size_t)rman_get_size(r); + vaddr = rman_get_bushandle(r); + + if (vaddr != 0) { + bus_space_unmap(&memmap_bus, vaddr, psize); + rman_set_virtual(r, NULL); + rman_set_bushandle(r, 0); + } + + return (rman_deactivate_resource(r)); +} + +#ifdef FDT +static int +nexus_ofw_map_intr(device_t dev, device_t child, phandle_t iparent, int icells, + pcell_t *intr) +{ + int irq; + + if (icells == 3) { + irq = intr[1]; + if (intr[0] == 0) + irq += 32; /* SPI */ + else + irq += 16; /* PPI */ + } else + irq = intr[0]; + + return (irq); +} +#endif + Property changes on: user/ngie/more-tests/sys/arm64/arm64/nexus.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/pic_if.m =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/pic_if.m (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/pic_if.m (revision 281504) @@ -0,0 +1,180 @@ +#- +# Copyright (c) 1998 Doug Rabson +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# 1. Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +# SUCH DAMAGE. +# +# from: src/sys/kern/bus_if.m,v 1.21 2002/04/21 11:16:10 markm Exp +# $FreeBSD$ +# + +#include +#include +#include + +INTERFACE pic; + +CODE { + static pic_translate_code_t pic_translate_code_default; + + static void pic_translate_code_default(device_t dev, u_int irq, + int code, enum intr_trigger *trig, enum intr_polarity *pol) + { + *trig = INTR_TRIGGER_CONFORM; + *pol = INTR_POLARITY_CONFORM; + } + + static void pic_pre_ithread(device_t dev, u_int irq) + { + PIC_MASK(dev, irq); + PIC_EOI(dev, irq); + } + + static void pic_post_ithread(device_t dev, u_int irq) + { + PIC_UNMASK(dev, irq); + } + + static void pic_post_filter(device_t dev, u_int irq) + { + PIC_EOI(dev, irq); + } +}; + +METHOD void bind { + device_t dev; + u_int irq; + cpuset_t cpumask; +}; + +METHOD void translate_code { + device_t dev; + u_int irq; + int code; + enum intr_trigger *trig; + enum intr_polarity *pol; +} DEFAULT pic_translate_code_default; + +METHOD void config { + device_t dev; + u_int irq; + enum intr_trigger trig; + enum intr_polarity pol; +}; + +METHOD void dispatch { + device_t dev; + struct trapframe *tf; +}; + +METHOD void enable { + device_t dev; + u_int irq; + u_int vector; +}; + +METHOD void pre_ithread { + device_t dev; + u_int irq; +} DEFAULT pic_pre_ithread; + +METHOD void post_ithread { + device_t dev; + u_int irq; +} DEFAULT pic_post_ithread; + +METHOD void post_filter { + device_t dev; + u_int irq; +} DEFAULT pic_post_filter; + +METHOD void eoi { + device_t dev; + u_int irq; +}; + +METHOD void ipi { + device_t dev; + u_int cpu; +}; + +METHOD void mask { + device_t dev; + u_int irq; +}; + +METHOD void unmask { + device_t dev; + u_int irq; +}; + +METHOD void init_secondary { + device_t dev; +}; + +METHOD void ipi_send { + device_t dev; + cpuset_t cpus; + u_int ipi; +}; + +METHOD int alloc_msi { + device_t dev; + device_t pci_dev; + int count; + int *irqs; +}; + +METHOD int alloc_msix { + device_t dev; + device_t pci_dev; + int *irq; +}; + +METHOD int map_msi { + device_t dev; + device_t pci_dev; + int irq; + uint64_t *addr; + uint32_t *data; +}; + +METHOD int map_msix { + device_t dev; + device_t pci_dev; + int irq; + uint64_t *addr; + uint32_t *data; +}; + +METHOD int release_msi { + device_t dev; + device_t pci_dev; + int count; + int *irqs; +}; + +METHOD int release_msix { + device_t dev; + device_t pci_dev; + int irq; +}; Property changes on: user/ngie/more-tests/sys/arm64/arm64/pic_if.m ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/pmap.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/pmap.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/pmap.c (revision 281504) @@ -0,0 +1,3067 @@ +/*- + * Copyright (c) 1991 Regents of the University of California. + * All rights reserved. + * Copyright (c) 1994 John S. Dyson + * All rights reserved. + * Copyright (c) 1994 David Greenman + * All rights reserved. + * Copyright (c) 2003 Peter Wemm + * All rights reserved. + * Copyright (c) 2005-2010 Alan L. Cox + * All rights reserved. + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This code is derived from software contributed to Berkeley by + * the Systems Programming Group of the University of Utah Computer + * Science Department and William Jolitz of UUNET Technologies Inc. + * + * This software was developed by Andrew Turner under sponsorship from + * the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by the University of + * California, Berkeley and its contributors. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * from: @(#)pmap.c 7.7 (Berkeley) 5/12/91 + */ +/*- + * Copyright (c) 2003 Networks Associates Technology, Inc. + * All rights reserved. + * + * This software was developed for the FreeBSD Project by Jake Burkholder, + * Safeport Network Services, and Network Associates Laboratories, the + * Security Research Division of Network Associates, Inc. under + * DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA + * CHATS research program. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#define AMD64_NPT_AWARE + +#include +__FBSDID("$FreeBSD$"); + +/* + * Manages physical address maps. + * + * Since the information managed by this module is + * also stored by the logical address mapping module, + * this module may throw away valid virtual-to-physical + * mappings at almost any time. However, invalidations + * of virtual-to-physical mappings must be done as + * requested. + * + * In order to cope with hardware architectures which + * make virtual-to-physical map invalidates expensive, + * this module may delay invalidate or reduced protection + * operations until such time as they are actually + * necessary. This module is given full information as + * to which processors are currently using which maps, + * and to when physical maps must be made correct. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +#define NPDEPG (PAGE_SIZE/(sizeof (pd_entry_t))) +#define NUPDE (NPDEPG * NPDEPG) +#define NUSERPGTBLS (NUPDE + NPDEPG) + +#if !defined(DIAGNOSTIC) +#ifdef __GNUC_GNU_INLINE__ +#define PMAP_INLINE __attribute__((__gnu_inline__)) inline +#else +#define PMAP_INLINE extern inline +#endif +#else +#define PMAP_INLINE +#endif + +/* + * These are configured by the mair_el1 register. This is set up in locore.S + */ +#define DEVICE_MEMORY 0 +#define UNCACHED_MEMORY 1 +#define CACHED_MEMORY 2 + + +#ifdef PV_STATS +#define PV_STAT(x) do { x ; } while (0) +#else +#define PV_STAT(x) do { } while (0) +#endif + +#define pmap_l2_pindex(v) ((v) >> L2_SHIFT) + +#define NPV_LIST_LOCKS MAXCPU + +#define PHYS_TO_PV_LIST_LOCK(pa) \ + (&pv_list_locks[pa_index(pa) % NPV_LIST_LOCKS]) + +#define CHANGE_PV_LIST_LOCK_TO_PHYS(lockp, pa) do { \ + struct rwlock **_lockp = (lockp); \ + struct rwlock *_new_lock; \ + \ + _new_lock = PHYS_TO_PV_LIST_LOCK(pa); \ + if (_new_lock != *_lockp) { \ + if (*_lockp != NULL) \ + rw_wunlock(*_lockp); \ + *_lockp = _new_lock; \ + rw_wlock(*_lockp); \ + } \ +} while (0) + +#define CHANGE_PV_LIST_LOCK_TO_VM_PAGE(lockp, m) \ + CHANGE_PV_LIST_LOCK_TO_PHYS(lockp, VM_PAGE_TO_PHYS(m)) + +#define RELEASE_PV_LIST_LOCK(lockp) do { \ + struct rwlock **_lockp = (lockp); \ + \ + if (*_lockp != NULL) { \ + rw_wunlock(*_lockp); \ + *_lockp = NULL; \ + } \ +} while (0) + +#define VM_PAGE_TO_PV_LIST_LOCK(m) \ + PHYS_TO_PV_LIST_LOCK(VM_PAGE_TO_PHYS(m)) + +struct pmap kernel_pmap_store; + +vm_offset_t virtual_avail; /* VA of first avail page (after kernel bss) */ +vm_offset_t virtual_end; /* VA of last avail page (end of kernel AS) */ +vm_offset_t kernel_vm_end = 0; + +struct msgbuf *msgbufp = NULL; + +static struct rwlock_padalign pvh_global_lock; + +/* + * Data for the pv entry allocation mechanism + */ +static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); +static struct mtx pv_chunks_mutex; +static struct rwlock pv_list_locks[NPV_LIST_LOCKS]; + +static void free_pv_chunk(struct pv_chunk *pc); +static void free_pv_entry(pmap_t pmap, pv_entry_t pv); +static pv_entry_t get_pv_entry(pmap_t pmap, struct rwlock **lockp); +static vm_page_t reclaim_pv_chunk(pmap_t locked_pmap, struct rwlock **lockp); +static void pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va); +static pv_entry_t pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, + vm_offset_t va); +static vm_page_t pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, + vm_page_t m, vm_prot_t prot, vm_page_t mpte, struct rwlock **lockp); +static int pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_t sva, + pd_entry_t ptepde, struct spglist *free, struct rwlock **lockp); +static boolean_t pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, + vm_page_t m, struct rwlock **lockp); + +static vm_page_t _pmap_alloc_l3(pmap_t pmap, vm_pindex_t ptepindex, + struct rwlock **lockp); + +static void _pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, + struct spglist *free); +static int pmap_unuse_l3(pmap_t, vm_offset_t, pd_entry_t, struct spglist *); + +/********************/ +/* Inline functions */ +/********************/ + +static __inline void +pagecopy(void *s, void *d) +{ + + memcpy(d, s, PAGE_SIZE); +} + +static __inline void +pagezero(void *p) +{ + + bzero(p, PAGE_SIZE); +} + +#define pmap_l1_index(va) (((va) >> L1_SHIFT) & Ln_ADDR_MASK) +#define pmap_l2_index(va) (((va) >> L2_SHIFT) & Ln_ADDR_MASK) +#define pmap_l3_index(va) (((va) >> L3_SHIFT) & Ln_ADDR_MASK) + +static __inline pd_entry_t * +pmap_l1(pmap_t pmap, vm_offset_t va) +{ + + return (&pmap->pm_l1[pmap_l1_index(va)]); +} + +static __inline pd_entry_t * +pmap_l1_to_l2(pd_entry_t *l1, vm_offset_t va) +{ + pd_entry_t *l2; + + l2 = (pd_entry_t *)PHYS_TO_DMAP(*l1 & ~ATTR_MASK); + return (&l2[pmap_l2_index(va)]); +} + +static __inline pd_entry_t * +pmap_l2(pmap_t pmap, vm_offset_t va) +{ + pd_entry_t *l1; + + l1 = pmap_l1(pmap, va); + if ((*l1 & ATTR_DESCR_MASK) != L1_TABLE) + return (NULL); + + return (pmap_l1_to_l2(l1, va)); +} + +static __inline pt_entry_t * +pmap_l2_to_l3(pd_entry_t *l2, vm_offset_t va) +{ + pt_entry_t *l3; + + l3 = (pd_entry_t *)PHYS_TO_DMAP(*l2 & ~ATTR_MASK); + return (&l3[pmap_l3_index(va)]); +} + +static __inline pt_entry_t * +pmap_l3(pmap_t pmap, vm_offset_t va) +{ + pd_entry_t *l2; + + l2 = pmap_l2(pmap, va); + if (l2 == NULL || (*l2 & ATTR_DESCR_MASK) != L2_TABLE) + return (NULL); + + return (pmap_l2_to_l3(l2, va)); +} + +/* + * These load the old table data and store the new value. + * They need to be atomic as the System MMU may write to the table at + * the same time as the CPU. + */ +#define pmap_load_store(table, entry) atomic_swap_64(table, entry) +#define pmap_set(table, mask) atomic_set_64(table, mask) +#define pmap_load_clear(table) atomic_swap_64(table, 0) +#define pmap_load(table) (*table) + +static __inline int +pmap_is_current(pmap_t pmap) +{ + + return ((pmap == pmap_kernel()) || + (pmap == curthread->td_proc->p_vmspace->vm_map.pmap)); +} + +static __inline int +pmap_l3_valid(pt_entry_t l3) +{ + + return ((l3 & ATTR_DESCR_MASK) == L3_PAGE); +} + +static __inline int +pmap_l3_valid_cacheable(pt_entry_t l3) +{ + + return (((l3 & ATTR_DESCR_MASK) == L3_PAGE) && + ((l3 & ATTR_IDX_MASK) == ATTR_IDX(CACHED_MEMORY))); +} + +#define PTE_SYNC(pte) cpu_dcache_wb_range((vm_offset_t)pte, sizeof(*pte)) + +/* + * Checks if the page is dirty. We currently lack proper tracking of this on + * arm64 so for now assume is a page mapped as rw was accessed it is. + */ +static inline int +pmap_page_dirty(pt_entry_t pte) +{ + + return ((pte & (ATTR_AF | ATTR_AP_RW_BIT)) == + (ATTR_AF | ATTR_AP(ATTR_AP_RW))); +} + +static __inline void +pmap_resident_count_inc(pmap_t pmap, int count) +{ + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + pmap->pm_stats.resident_count += count; +} + +static __inline void +pmap_resident_count_dec(pmap_t pmap, int count) +{ + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + KASSERT(pmap->pm_stats.resident_count >= count, + ("pmap %p resident count underflow %ld %d", pmap, + pmap->pm_stats.resident_count, count)); + pmap->pm_stats.resident_count -= count; +} + +static pt_entry_t * +pmap_early_page_idx(vm_offset_t l1pt, vm_offset_t va, u_int *l1_slot, + u_int *l2_slot) +{ + pt_entry_t *l2; + pd_entry_t *l1; + + l1 = (pd_entry_t *)l1pt; + *l1_slot = (va >> L1_SHIFT) & Ln_ADDR_MASK; + + /* Check locore has used a table L1 map */ + KASSERT((l1[*l1_slot] & ATTR_DESCR_MASK) == L1_TABLE, + ("Invalid bootstrap L1 table")); + /* Find the address of the L2 table */ + l2 = (pt_entry_t *)init_pt_va; + *l2_slot = pmap_l2_index(va); + + return (l2); +} + +static vm_paddr_t +pmap_early_vtophys(vm_offset_t l1pt, vm_offset_t va) +{ + u_int l1_slot, l2_slot; + pt_entry_t *l2; + + l2 = pmap_early_page_idx(l1pt, va, &l1_slot, &l2_slot); + + return ((l2[l2_slot] & ~ATTR_MASK) + (va & L2_OFFSET)); +} + +static void +pmap_bootstrap_dmap(vm_offset_t l1pt) +{ + vm_offset_t va; + vm_paddr_t pa; + pd_entry_t *l1; + u_int l1_slot; + + va = DMAP_MIN_ADDRESS; + l1 = (pd_entry_t *)l1pt; + l1_slot = pmap_l1_index(DMAP_MIN_ADDRESS); + + for (pa = 0; va < DMAP_MAX_ADDRESS; + pa += L1_SIZE, va += L1_SIZE, l1_slot++) { + KASSERT(l1_slot < Ln_ENTRIES, ("Invalid L1 index")); + + /* + * TODO: Turn the cache on here when we have cache + * flushing code. + */ + pmap_load_store(&l1[l1_slot], + (pa & ~L1_OFFSET) | ATTR_AF | L1_BLOCK | + ATTR_IDX(CACHED_MEMORY)); + } + + cpu_dcache_wb_range((vm_offset_t)l1, PAGE_SIZE); + cpu_tlb_flushID(); +} + +static vm_offset_t +pmap_bootstrap_l2(vm_offset_t l1pt, vm_offset_t va, vm_offset_t l2_start) +{ + vm_offset_t l2pt; + vm_paddr_t pa; + pd_entry_t *l1; + u_int l1_slot; + + KASSERT((va & L1_OFFSET) == 0, ("Invalid virtual address")); + + l1 = (pd_entry_t *)l1pt; + l1_slot = pmap_l1_index(va); + l2pt = l2_start; + + for (; va < VM_MAX_KERNEL_ADDRESS; l1_slot++, va += L1_SIZE) { + KASSERT(l1_slot < Ln_ENTRIES, ("Invalid L1 index")); + + pa = pmap_early_vtophys(l1pt, l2pt); + pmap_load_store(&l1[l1_slot], + (pa & ~Ln_TABLE_MASK) | L1_TABLE); + l2pt += PAGE_SIZE; + } + + /* Clean the L2 page table */ + memset((void *)l2_start, 0, l2pt - l2_start); + cpu_dcache_wb_range(l2_start, l2pt - l2_start); + + /* Flush the l1 table to ram */ + cpu_dcache_wb_range((vm_offset_t)l1, PAGE_SIZE); + + return l2pt; +} + +static vm_offset_t +pmap_bootstrap_l3(vm_offset_t l1pt, vm_offset_t va, vm_offset_t l3_start) +{ + vm_offset_t l2pt, l3pt; + vm_paddr_t pa; + pd_entry_t *l2; + u_int l2_slot; + + KASSERT((va & L2_OFFSET) == 0, ("Invalid virtual address")); + + l2 = pmap_l2(kernel_pmap, va); + l2 = (pd_entry_t *)((uintptr_t)l2 & ~(PAGE_SIZE - 1)); + l2pt = (vm_offset_t)l2; + l2_slot = pmap_l2_index(va); + l3pt = l3_start; + + for (; va < VM_MAX_KERNEL_ADDRESS; l2_slot++, va += L2_SIZE) { + KASSERT(l2_slot < Ln_ENTRIES, ("Invalid L2 index")); + + pa = pmap_early_vtophys(l1pt, l3pt); + pmap_load_store(&l2[l2_slot], + (pa & ~Ln_TABLE_MASK) | L2_TABLE); + l3pt += PAGE_SIZE; + } + + /* Clean the L2 page table */ + memset((void *)l3_start, 0, l3pt - l3_start); + cpu_dcache_wb_range(l3_start, l3pt - l3_start); + + cpu_dcache_wb_range((vm_offset_t)l2, PAGE_SIZE); + + return l3pt; +} + +/* + * Bootstrap the system enough to run with virtual memory. + */ +void +pmap_bootstrap(vm_offset_t l1pt, vm_paddr_t kernstart, vm_size_t kernlen) +{ + u_int l1_slot, l2_slot, avail_slot, map_slot, used_map_slot; + uint64_t kern_delta; + pt_entry_t *l2; + vm_offset_t va, freemempos; + vm_offset_t dpcpu, msgbufpv; + vm_paddr_t pa; + + kern_delta = KERNBASE - kernstart; + physmem = 0; + + printf("pmap_bootstrap %lx %lx %lx\n", l1pt, kernstart, kernlen); + printf("%lx\n", l1pt); + printf("%lx\n", (KERNBASE >> L1_SHIFT) & Ln_ADDR_MASK); + + /* Set this early so we can use the pagetable walking functions */ + kernel_pmap_store.pm_l1 = (pd_entry_t *)l1pt; + PMAP_LOCK_INIT(kernel_pmap); + + /* + * Initialize the global pv list lock. + */ + rw_init(&pvh_global_lock, "pmap pv global"); + + /* Create a direct map region early so we can use it for pa -> va */ + pmap_bootstrap_dmap(l1pt); + + va = KERNBASE; + pa = KERNBASE - kern_delta; + + /* + * Start to initialise phys_avail by copying from physmap + * up to the physical address KERNBASE points at. + */ + map_slot = avail_slot = 0; + for (; map_slot < (physmap_idx * 2); map_slot += 2) { + if (physmap[map_slot] == physmap[map_slot + 1]) + continue; + + if (physmap[map_slot] <= pa && + physmap[map_slot + 1] > pa) + break; + + phys_avail[avail_slot] = physmap[map_slot]; + phys_avail[avail_slot + 1] = physmap[map_slot + 1]; + physmem += (phys_avail[avail_slot + 1] - + phys_avail[avail_slot]) >> PAGE_SHIFT; + avail_slot += 2; + } + + /* Add the memory before the kernel */ + if (physmap[avail_slot] < pa) { + phys_avail[avail_slot] = physmap[map_slot]; + phys_avail[avail_slot + 1] = pa; + physmem += (phys_avail[avail_slot + 1] - + phys_avail[avail_slot]) >> PAGE_SHIFT; + avail_slot += 2; + } + used_map_slot = map_slot; + + /* + * Read the page table to find out what is already mapped. + * This assumes we have mapped a block of memory from KERNBASE + * using a single L1 entry. + */ + l2 = pmap_early_page_idx(l1pt, KERNBASE, &l1_slot, &l2_slot); + + /* Sanity check the index, KERNBASE should be the first VA */ + KASSERT(l2_slot == 0, ("The L2 index is non-zero")); + + /* Find how many pages we have mapped */ + for (; l2_slot < Ln_ENTRIES; l2_slot++) { + if ((l2[l2_slot] & ATTR_DESCR_MASK) == 0) + break; + + /* Check locore used L2 blocks */ + KASSERT((l2[l2_slot] & ATTR_DESCR_MASK) == L2_BLOCK, + ("Invalid bootstrap L2 table")); + KASSERT((l2[l2_slot] & ~ATTR_MASK) == pa, + ("Incorrect PA in L2 table")); + + va += L2_SIZE; + pa += L2_SIZE; + } + + va = roundup2(va, L1_SIZE); + + freemempos = KERNBASE + kernlen; + freemempos = roundup2(freemempos, PAGE_SIZE); + /* Create the l2 tables up to VM_MAX_KERNEL_ADDRESS */ + freemempos = pmap_bootstrap_l2(l1pt, va, freemempos); + /* And the l3 tables for the early devmap */ + freemempos = pmap_bootstrap_l3(l1pt, + VM_MAX_KERNEL_ADDRESS - L2_SIZE, freemempos); + + cpu_tlb_flushID(); + +#define alloc_pages(var, np) \ + (var) = freemempos; \ + freemempos += (np * PAGE_SIZE); \ + memset((char *)(var), 0, ((np) * PAGE_SIZE)); + + /* Allocate dynamic per-cpu area. */ + alloc_pages(dpcpu, DPCPU_SIZE / PAGE_SIZE); + dpcpu_init((void *)dpcpu, 0); + + /* Allocate memory for the msgbuf, e.g. for /sbin/dmesg */ + alloc_pages(msgbufpv, round_page(msgbufsize) / PAGE_SIZE); + msgbufp = (void *)msgbufpv; + + virtual_avail = roundup2(freemempos, L1_SIZE); + virtual_end = VM_MAX_KERNEL_ADDRESS - L2_SIZE; + kernel_vm_end = virtual_avail; + + pa = pmap_early_vtophys(l1pt, freemempos); + + /* Finish initialising physmap */ + map_slot = used_map_slot; + for (; avail_slot < (PHYS_AVAIL_SIZE - 2) && + map_slot < (physmap_idx * 2); map_slot += 2) { + if (physmap[map_slot] == physmap[map_slot + 1]) + continue; + + /* Have we used the current range? */ + if (physmap[map_slot + 1] <= pa) + continue; + + /* Do we need to split the entry? */ + if (physmap[map_slot] < pa) { + phys_avail[avail_slot] = pa; + phys_avail[avail_slot + 1] = physmap[map_slot + 1]; + } else { + phys_avail[avail_slot] = physmap[map_slot]; + phys_avail[avail_slot + 1] = physmap[map_slot + 1]; + } + physmem += (phys_avail[avail_slot + 1] - + phys_avail[avail_slot]) >> PAGE_SHIFT; + + avail_slot += 2; + } + phys_avail[avail_slot] = 0; + phys_avail[avail_slot + 1] = 0; + + /* + * Maxmem isn't the "maximum memory", it's one larger than the + * highest page of the physical address space. It should be + * called something like "Maxphyspage". + */ + Maxmem = atop(phys_avail[avail_slot - 1]); + + cpu_tlb_flushID(); +} + +/* + * Initialize a vm_page's machine-dependent fields. + */ +void +pmap_page_init(vm_page_t m) +{ + + TAILQ_INIT(&m->md.pv_list); + m->md.pv_memattr = VM_MEMATTR_WRITE_BACK; +} + +/* + * Initialize the pmap module. + * Called by vm_init, to initialize any structures that the pmap + * system needs to map virtual memory. + */ +void +pmap_init(void) +{ + int i; + + /* + * Initialize the pv chunk list mutex. + */ + mtx_init(&pv_chunks_mutex, "pmap pv chunk list", NULL, MTX_DEF); + + /* + * Initialize the pool of pv list locks. + */ + for (i = 0; i < NPV_LIST_LOCKS; i++) + rw_init(&pv_list_locks[i], "pmap pv list"); +} + +/* + * Normal, non-SMP, invalidation functions. + * We inline these within pmap.c for speed. + */ +PMAP_INLINE void +pmap_invalidate_page(pmap_t pmap, vm_offset_t va) +{ + + sched_pin(); + __asm __volatile( + "dsb sy \n" + "tlbi vaae1is, %0 \n" + "dsb sy \n" + "isb \n" + : : "r"(va >> PAGE_SHIFT)); + sched_unpin(); +} + +PMAP_INLINE void +pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +{ + vm_offset_t addr; + + sched_pin(); + sva >>= PAGE_SHIFT; + eva >>= PAGE_SHIFT; + __asm __volatile("dsb sy"); + for (addr = sva; addr < eva; addr++) { + __asm __volatile( + "tlbi vaae1is, %0" : : "r"(addr)); + } + __asm __volatile( + "dsb sy \n" + "isb \n"); + sched_unpin(); +} + +PMAP_INLINE void +pmap_invalidate_all(pmap_t pmap) +{ + + sched_pin(); + __asm __volatile( + "dsb sy \n" + "tlbi vmalle1is \n" + "dsb sy \n" + "isb \n"); + sched_unpin(); +} + +/* + * Routine: pmap_extract + * Function: + * Extract the physical page address associated + * with the given map/virtual_address pair. + */ +vm_paddr_t +pmap_extract(pmap_t pmap, vm_offset_t va) +{ + pd_entry_t *l2p, l2; + pt_entry_t *l3p, l3; + vm_paddr_t pa; + + pa = 0; + PMAP_LOCK(pmap); + /* + * Start with the l2 tabel. We are unable to allocate + * pages in the l1 table. + */ + l2p = pmap_l2(pmap, va); + if (l2p != NULL) { + l2 = *l2p; + if ((l2 & ATTR_DESCR_MASK) == L2_TABLE) { + l3p = pmap_l2_to_l3(l2p, va); + if (l3p != NULL) { + l3 = *l3p; + + if ((l3 & ATTR_DESCR_MASK) == L3_PAGE) + pa = (l3 & ~ATTR_MASK) | + (va & L3_OFFSET); + } + } else if ((l2 & ATTR_DESCR_MASK) == L2_BLOCK) + pa = (l2 & ~ATTR_MASK) | (va & L2_OFFSET); + } + PMAP_UNLOCK(pmap); + return (pa); +} + +/* + * Routine: pmap_extract_and_hold + * Function: + * Atomically extract and hold the physical page + * with the given pmap and virtual address pair + * if that mapping permits the given protection. + */ +vm_page_t +pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) +{ + pt_entry_t *l3p, l3; + vm_paddr_t pa; + vm_page_t m; + + pa = 0; + m = NULL; + PMAP_LOCK(pmap); +retry: + l3p = pmap_l3(pmap, va); + if (l3p != NULL && (l3 = *l3p) != 0) { + if (((l3 & ATTR_AP_RW_BIT) == ATTR_AP(ATTR_AP_RW)) || + ((prot & VM_PROT_WRITE) == 0)) { + if (vm_page_pa_tryrelock(pmap, l3 & ~ATTR_MASK, &pa)) + goto retry; + m = PHYS_TO_VM_PAGE(l3 & ~ATTR_MASK); + vm_page_hold(m); + } + } + PA_UNLOCK_COND(pa); + PMAP_UNLOCK(pmap); + return (m); +} + +vm_paddr_t +pmap_kextract(vm_offset_t va) +{ + pd_entry_t *l2; + pt_entry_t *l3; + vm_paddr_t pa; + + if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) { + pa = DMAP_TO_PHYS(va); + } else { + l2 = pmap_l2(kernel_pmap, va); + if (l2 == NULL) + panic("pmap_kextract: No l2"); + if ((*l2 & ATTR_DESCR_MASK) == L2_BLOCK) + return ((*l2 & ~ATTR_MASK) | (va & L2_OFFSET)); + + l3 = pmap_l2_to_l3(l2, va); + if (l3 == NULL) + panic("pmap_kextract: No l3..."); + pa = (*l3 & ~ATTR_MASK) | (va & PAGE_MASK); + } + return (pa); +} + +/*************************************************** + * Low level mapping routines..... + ***************************************************/ + +void +pmap_kenter_device(vm_offset_t va, vm_size_t size, vm_paddr_t pa) +{ + pt_entry_t *l3; + + KASSERT((pa & L3_OFFSET) == 0, + ("pmap_kenter_device: Invalid physical address")); + KASSERT((va & L3_OFFSET) == 0, + ("pmap_kenter_device: Invalid virtual address")); + KASSERT((size & PAGE_MASK) == 0, + ("pmap_kenter_device: Mapping is not page-sized")); + + while (size != 0) { + l3 = pmap_l3(kernel_pmap, va); + KASSERT(l3 != NULL, ("Invalid page table, va: 0x%lx", va)); + pmap_load_store(l3, (pa & ~L3_OFFSET) | ATTR_AF | L3_PAGE | + ATTR_IDX(DEVICE_MEMORY)); + PTE_SYNC(l3); + + va += PAGE_SIZE; + pa += PAGE_SIZE; + size -= PAGE_SIZE; + } +} + +/* + * Remove a page from the kernel pagetables. + * Note: not SMP coherent. + */ +PMAP_INLINE void +pmap_kremove(vm_offset_t va) +{ + pt_entry_t *l3; + + l3 = pmap_l3(kernel_pmap, va); + KASSERT(l3 != NULL, ("pmap_kremove: Invalid address")); + + if (pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(va, L3_SIZE); + pmap_load_clear(l3); + PTE_SYNC(l3); +} + +void +pmap_kremove_device(vm_offset_t va, vm_size_t size) +{ + pt_entry_t *l3; + + KASSERT((va & L3_OFFSET) == 0, + ("pmap_kremove_device: Invalid virtual address")); + KASSERT((size & PAGE_MASK) == 0, + ("pmap_kremove_device: Mapping is not page-sized")); + + while (size != 0) { + l3 = pmap_l3(kernel_pmap, va); + KASSERT(l3 != NULL, ("Invalid page table, va: 0x%lx", va)); + pmap_load_clear(l3); + PTE_SYNC(l3); + + va += PAGE_SIZE; + size -= PAGE_SIZE; + } +} + +/* + * Used to map a range of physical addresses into kernel + * virtual address space. + * + * The value passed in '*virt' is a suggested virtual address for + * the mapping. Architectures which can support a direct-mapped + * physical to virtual region can return the appropriate address + * within that region, leaving '*virt' unchanged. Other + * architectures should map the pages starting at '*virt' and + * update '*virt' with the first usable address after the mapped + * region. + */ +vm_offset_t +pmap_map(vm_offset_t *virt, vm_paddr_t start, vm_paddr_t end, int prot) +{ + return PHYS_TO_DMAP(start); +} + + +/* + * Add a list of wired pages to the kva + * this routine is only used for temporary + * kernel mappings that do not need to have + * page modification or references recorded. + * Note that old mappings are simply written + * over. The page *must* be wired. + * Note: SMP coherent. Uses a ranged shootdown IPI. + */ +void +pmap_qenter(vm_offset_t sva, vm_page_t *ma, int count) +{ + pt_entry_t *l3, pa; + vm_offset_t va; + vm_page_t m; + int i; + + va = sva; + for (i = 0; i < count; i++) { + m = ma[i]; + pa = VM_PAGE_TO_PHYS(m) | ATTR_AF | + ATTR_IDX(m->md.pv_memattr) | ATTR_AP(ATTR_AP_RW) | L3_PAGE; + l3 = pmap_l3(kernel_pmap, va); + pmap_load_store(l3, pa); + PTE_SYNC(l3); + + va += L3_SIZE; + } +} + +/* + * This routine tears out page mappings from the + * kernel -- it is meant only for temporary mappings. + * Note: SMP coherent. Uses a ranged shootdown IPI. + */ +void +pmap_qremove(vm_offset_t sva, int count) +{ + vm_offset_t va; + + va = sva; + while (count-- > 0) { + KASSERT(va >= VM_MIN_KERNEL_ADDRESS, ("usermode va %lx", va)); + pmap_kremove(va); + va += PAGE_SIZE; + } + pmap_invalidate_range(kernel_pmap, sva, va); +} + +/*************************************************** + * Page table page management routines..... + ***************************************************/ +static __inline void +pmap_free_zero_pages(struct spglist *free) +{ + vm_page_t m; + + while ((m = SLIST_FIRST(free)) != NULL) { + SLIST_REMOVE_HEAD(free, plinks.s.ss); + /* Preserve the page's PG_ZERO setting. */ + vm_page_free_toq(m); + } +} + +/* + * Schedule the specified unused page table page to be freed. Specifically, + * add the page to the specified list of pages that will be released to the + * physical memory manager after the TLB has been updated. + */ +static __inline void +pmap_add_delayed_free_list(vm_page_t m, struct spglist *free, + boolean_t set_PG_ZERO) +{ + + if (set_PG_ZERO) + m->flags |= PG_ZERO; + else + m->flags &= ~PG_ZERO; + SLIST_INSERT_HEAD(free, m, plinks.s.ss); +} + +/* + * Decrements a page table page's wire count, which is used to record the + * number of valid page table entries within the page. If the wire count + * drops to zero, then the page table page is unmapped. Returns TRUE if the + * page table page was unmapped and FALSE otherwise. + */ +static inline boolean_t +pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free) +{ + + --m->wire_count; + if (m->wire_count == 0) { + _pmap_unwire_l3(pmap, va, m, free); + return (TRUE); + } else + return (FALSE); +} + +static void +_pmap_unwire_l3(pmap_t pmap, vm_offset_t va, vm_page_t m, struct spglist *free) +{ + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + /* + * unmap the page table page + */ + if (m->pindex >= NUPDE) { + /* PD page */ + pd_entry_t *l1; + l1 = pmap_l1(pmap, va); + pmap_load_clear(l1); + PTE_SYNC(l1); + } else { + /* PTE page */ + pd_entry_t *l2; + l2 = pmap_l2(pmap, va); + pmap_load_clear(l2); + PTE_SYNC(l2); + } + pmap_resident_count_dec(pmap, 1); + if (m->pindex < NUPDE) { + /* We just released a PT, unhold the matching PD */ + vm_page_t pdpg; + + pdpg = PHYS_TO_VM_PAGE(*pmap_l1(pmap, va) & ~ATTR_MASK); + pmap_unwire_l3(pmap, va, pdpg, free); + } + + /* + * This is a release store so that the ordinary store unmapping + * the page table page is globally performed before TLB shoot- + * down is begun. + */ + atomic_subtract_rel_int(&vm_cnt.v_wire_count, 1); + + /* + * Put page on a list so that it is released after + * *ALL* TLB shootdown is done + */ + pmap_add_delayed_free_list(m, free, TRUE); +} + +/* + * After removing an l3 entry, this routine is used to + * conditionally free the page, and manage the hold/wire counts. + */ +static int +pmap_unuse_l3(pmap_t pmap, vm_offset_t va, pd_entry_t ptepde, + struct spglist *free) +{ + vm_page_t mpte; + + if (va >= VM_MAXUSER_ADDRESS) + return (0); + KASSERT(ptepde != 0, ("pmap_unuse_pt: ptepde != 0")); + mpte = PHYS_TO_VM_PAGE(ptepde & ~ATTR_MASK); + return (pmap_unwire_l3(pmap, va, mpte, free)); +} + +void +pmap_pinit0(pmap_t pmap) +{ + + PMAP_LOCK_INIT(pmap); + bzero(&pmap->pm_stats, sizeof(pmap->pm_stats)); + pmap->pm_l1 = kernel_pmap->pm_l1; +} + +int +pmap_pinit(pmap_t pmap) +{ + vm_paddr_t l1phys; + vm_page_t l1pt; + + /* + * allocate the l1 page + */ + while ((l1pt = vm_page_alloc(NULL, 0xdeadbeef, VM_ALLOC_NORMAL | + VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) + VM_WAIT; + + l1phys = VM_PAGE_TO_PHYS(l1pt); + pmap->pm_l1 = (pd_entry_t *)PHYS_TO_DMAP(l1phys); + + if ((l1pt->flags & PG_ZERO) == 0) + pagezero(pmap->pm_l1); + + bzero(&pmap->pm_stats, sizeof(pmap->pm_stats)); + + return (1); +} + +/* + * This routine is called if the desired page table page does not exist. + * + * If page table page allocation fails, this routine may sleep before + * returning NULL. It sleeps only if a lock pointer was given. + * + * Note: If a page allocation fails at page table level two or three, + * one or two pages may be held during the wait, only to be released + * afterwards. This conservative approach is easily argued to avoid + * race conditions. + */ +static vm_page_t +_pmap_alloc_l3(pmap_t pmap, vm_pindex_t ptepindex, struct rwlock **lockp) +{ + vm_page_t m, /*pdppg, */pdpg; + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + + /* + * Allocate a page table page. + */ + if ((m = vm_page_alloc(NULL, ptepindex, VM_ALLOC_NOOBJ | + VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) { + if (lockp != NULL) { + RELEASE_PV_LIST_LOCK(lockp); + PMAP_UNLOCK(pmap); + rw_runlock(&pvh_global_lock); + VM_WAIT; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + } + + /* + * Indicate the need to retry. While waiting, the page table + * page may have been allocated. + */ + return (NULL); + } + if ((m->flags & PG_ZERO) == 0) + pmap_zero_page(m); + + /* + * Map the pagetable page into the process address space, if + * it isn't already there. + */ + + if (ptepindex >= NUPDE) { + pd_entry_t *l1; + vm_pindex_t l1index; + + l1index = ptepindex - NUPDE; + l1 = &pmap->pm_l1[l1index]; + pmap_load_store(l1, VM_PAGE_TO_PHYS(m) | L1_TABLE); + PTE_SYNC(l1); + + } else { + vm_pindex_t l1index; + pd_entry_t *l1, *l2; + + l1index = ptepindex >> (L1_SHIFT - L2_SHIFT); + l1 = &pmap->pm_l1[l1index]; + if (*l1 == 0) { + /* recurse for allocating page dir */ + if (_pmap_alloc_l3(pmap, NUPDE + l1index, + lockp) == NULL) { + --m->wire_count; + atomic_subtract_int(&vm_cnt.v_wire_count, 1); + vm_page_free_zero(m); + return (NULL); + } + } else { + pdpg = PHYS_TO_VM_PAGE(*l1 & ~ATTR_MASK); + pdpg->wire_count++; + } + + l2 = (pd_entry_t *)PHYS_TO_DMAP(*l1 & ~ATTR_MASK); + l2 = &l2[ptepindex & Ln_ADDR_MASK]; + pmap_load_store(l2, VM_PAGE_TO_PHYS(m) | ATTR_AF | + ATTR_IDX(CACHED_MEMORY) | L2_TABLE); + PTE_SYNC(l2); + } + + pmap_resident_count_inc(pmap, 1); + + return (m); +} + +static vm_page_t +pmap_alloc_l3(pmap_t pmap, vm_offset_t va, struct rwlock **lockp) +{ + vm_pindex_t ptepindex; + pd_entry_t *l2; + vm_page_t m; + + /* + * Calculate pagetable page index + */ + ptepindex = pmap_l2_pindex(va); +retry: + /* + * Get the page directory entry + */ + l2 = pmap_l2(pmap, va); + + /* + * If the page table page is mapped, we just increment the + * hold count, and activate it. + */ + if (l2 != NULL && *l2 != 0) { + m = PHYS_TO_VM_PAGE(*l2 & ~ATTR_MASK); + m->wire_count++; + } else { + /* + * Here if the pte page isn't mapped, or if it has been + * deallocated. + */ + m = _pmap_alloc_l3(pmap, ptepindex, lockp); + if (m == NULL && lockp != NULL) + goto retry; + } + /* + * XXXARM64: I'm not sure why we need this but it fixes a crash + * when running things from a shell script. + */ + pmap_invalidate_all(pmap); + return (m); +} + + +/*************************************************** + * Pmap allocation/deallocation routines. + ***************************************************/ + +/* + * Release any resources held by the given physical map. + * Called when a pmap initialized by pmap_pinit is being released. + * Should only be called if the map contains no valid mappings. + */ +void +pmap_release(pmap_t pmap) +{ + vm_page_t m; + + KASSERT(pmap->pm_stats.resident_count == 0, + ("pmap_release: pmap resident count %ld != 0", + pmap->pm_stats.resident_count)); + + m = PHYS_TO_VM_PAGE(DMAP_TO_PHYS((vm_offset_t)pmap->pm_l1)); + + m->wire_count--; + atomic_subtract_int(&vm_cnt.v_wire_count, 1); + vm_page_free_zero(m); +} + +#if 0 +static int +kvm_size(SYSCTL_HANDLER_ARGS) +{ + unsigned long ksize = VM_MAX_KERNEL_ADDRESS - VM_MIN_KERNEL_ADDRESS; + + return sysctl_handle_long(oidp, &ksize, 0, req); +} +SYSCTL_PROC(_vm, OID_AUTO, kvm_size, CTLTYPE_LONG|CTLFLAG_RD, + 0, 0, kvm_size, "LU", "Size of KVM"); + +static int +kvm_free(SYSCTL_HANDLER_ARGS) +{ + unsigned long kfree = VM_MAX_KERNEL_ADDRESS - kernel_vm_end; + + return sysctl_handle_long(oidp, &kfree, 0, req); +} +SYSCTL_PROC(_vm, OID_AUTO, kvm_free, CTLTYPE_LONG|CTLFLAG_RD, + 0, 0, kvm_free, "LU", "Amount of KVM free"); +#endif /* 0 */ + +/* + * grow the number of kernel page table entries, if needed + */ +void +pmap_growkernel(vm_offset_t addr) +{ + vm_paddr_t paddr; + vm_page_t nkpg; + pd_entry_t *l1, *l2; + + mtx_assert(&kernel_map->system_mtx, MA_OWNED); + + addr = roundup2(addr, L2_SIZE); + if (addr - 1 >= kernel_map->max_offset) + addr = kernel_map->max_offset; + while (kernel_vm_end < addr) { + l1 = pmap_l1(kernel_pmap, kernel_vm_end); + if (*l1 == 0) { + /* We need a new PDP entry */ + nkpg = vm_page_alloc(NULL, kernel_vm_end >> L1_SHIFT, + VM_ALLOC_INTERRUPT | VM_ALLOC_NOOBJ | + VM_ALLOC_WIRED | VM_ALLOC_ZERO); + if (nkpg == NULL) + panic("pmap_growkernel: no memory to grow kernel"); + if ((nkpg->flags & PG_ZERO) == 0) + pmap_zero_page(nkpg); + paddr = VM_PAGE_TO_PHYS(nkpg); + pmap_load_store(l1, paddr | L1_TABLE); + PTE_SYNC(l1); + continue; /* try again */ + } + l2 = pmap_l1_to_l2(l1, kernel_vm_end); + if ((*l2 & ATTR_AF) != 0) { + kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET; + if (kernel_vm_end - 1 >= kernel_map->max_offset) { + kernel_vm_end = kernel_map->max_offset; + break; + } + continue; + } + + nkpg = vm_page_alloc(NULL, kernel_vm_end >> L2_SHIFT, + VM_ALLOC_INTERRUPT | VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | + VM_ALLOC_ZERO); + if (nkpg == NULL) + panic("pmap_growkernel: no memory to grow kernel"); + if ((nkpg->flags & PG_ZERO) == 0) + pmap_zero_page(nkpg); + paddr = VM_PAGE_TO_PHYS(nkpg); + pmap_load_store(l2, paddr | L2_TABLE); + PTE_SYNC(l2); + + kernel_vm_end = (kernel_vm_end + L2_SIZE) & ~L2_OFFSET; + if (kernel_vm_end - 1 >= kernel_map->max_offset) { + kernel_vm_end = kernel_map->max_offset; + break; + } + } +} + + +/*************************************************** + * page management routines. + ***************************************************/ + +CTASSERT(sizeof(struct pv_chunk) == PAGE_SIZE); +CTASSERT(_NPCM == 3); +CTASSERT(_NPCPV == 168); + +static __inline struct pv_chunk * +pv_to_chunk(pv_entry_t pv) +{ + + return ((struct pv_chunk *)((uintptr_t)pv & ~(uintptr_t)PAGE_MASK)); +} + +#define PV_PMAP(pv) (pv_to_chunk(pv)->pc_pmap) + +#define PC_FREE0 0xfffffffffffffffful +#define PC_FREE1 0xfffffffffffffffful +#define PC_FREE2 0x000000fffffffffful + +static const uint64_t pc_freemask[_NPCM] = { PC_FREE0, PC_FREE1, PC_FREE2 }; + +#if 0 +#ifdef PV_STATS +static int pc_chunk_count, pc_chunk_allocs, pc_chunk_frees, pc_chunk_tryfail; + +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_count, CTLFLAG_RD, &pc_chunk_count, 0, + "Current number of pv entry chunks"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_allocs, CTLFLAG_RD, &pc_chunk_allocs, 0, + "Current number of pv entry chunks allocated"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_frees, CTLFLAG_RD, &pc_chunk_frees, 0, + "Current number of pv entry chunks frees"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_tryfail, CTLFLAG_RD, &pc_chunk_tryfail, 0, + "Number of times tried to get a chunk page but failed."); + +static long pv_entry_frees, pv_entry_allocs, pv_entry_count; +static int pv_entry_spare; + +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_frees, CTLFLAG_RD, &pv_entry_frees, 0, + "Current number of pv entry frees"); +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_allocs, CTLFLAG_RD, &pv_entry_allocs, 0, + "Current number of pv entry allocs"); +SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_count, CTLFLAG_RD, &pv_entry_count, 0, + "Current number of pv entries"); +SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_spare, CTLFLAG_RD, &pv_entry_spare, 0, + "Current number of spare pv entries"); +#endif +#endif /* 0 */ + +/* + * We are in a serious low memory condition. Resort to + * drastic measures to free some pages so we can allocate + * another pv entry chunk. + * + * Returns NULL if PV entries were reclaimed from the specified pmap. + * + * We do not, however, unmap 2mpages because subsequent accesses will + * allocate per-page pv entries until repromotion occurs, thereby + * exacerbating the shortage of free pv entries. + */ +static vm_page_t +reclaim_pv_chunk(pmap_t locked_pmap, struct rwlock **lockp) +{ + + panic("reclaim_pv_chunk"); +} + +/* + * free the pv_entry back to the free list + */ +static void +free_pv_entry(pmap_t pmap, pv_entry_t pv) +{ + struct pv_chunk *pc; + int idx, field, bit; + + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + PV_STAT(atomic_add_long(&pv_entry_frees, 1)); + PV_STAT(atomic_add_int(&pv_entry_spare, 1)); + PV_STAT(atomic_subtract_long(&pv_entry_count, 1)); + pc = pv_to_chunk(pv); + idx = pv - &pc->pc_pventry[0]; + field = idx / 64; + bit = idx % 64; + pc->pc_map[field] |= 1ul << bit; + if (pc->pc_map[0] != PC_FREE0 || pc->pc_map[1] != PC_FREE1 || + pc->pc_map[2] != PC_FREE2) { + /* 98% of the time, pc is already at the head of the list. */ + if (__predict_false(pc != TAILQ_FIRST(&pmap->pm_pvchunk))) { + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); + } + return; + } + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + free_pv_chunk(pc); +} + +static void +free_pv_chunk(struct pv_chunk *pc) +{ + vm_page_t m; + + mtx_lock(&pv_chunks_mutex); + TAILQ_REMOVE(&pv_chunks, pc, pc_lru); + mtx_unlock(&pv_chunks_mutex); + PV_STAT(atomic_subtract_int(&pv_entry_spare, _NPCPV)); + PV_STAT(atomic_subtract_int(&pc_chunk_count, 1)); + PV_STAT(atomic_add_int(&pc_chunk_frees, 1)); + /* entire chunk is free, return it */ + m = PHYS_TO_VM_PAGE(DMAP_TO_PHYS((vm_offset_t)pc)); +#if 0 /* TODO: For minidump */ + dump_drop_page(m->phys_addr); +#endif + vm_page_unwire(m, PQ_INACTIVE); + vm_page_free(m); +} + +/* + * Returns a new PV entry, allocating a new PV chunk from the system when + * needed. If this PV chunk allocation fails and a PV list lock pointer was + * given, a PV chunk is reclaimed from an arbitrary pmap. Otherwise, NULL is + * returned. + * + * The given PV list lock may be released. + */ +static pv_entry_t +get_pv_entry(pmap_t pmap, struct rwlock **lockp) +{ + int bit, field; + pv_entry_t pv; + struct pv_chunk *pc; + vm_page_t m; + + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + PV_STAT(atomic_add_long(&pv_entry_allocs, 1)); +retry: + pc = TAILQ_FIRST(&pmap->pm_pvchunk); + if (pc != NULL) { + for (field = 0; field < _NPCM; field++) { + if (pc->pc_map[field]) { + bit = ffsl(pc->pc_map[field]) - 1; + break; + } + } + if (field < _NPCM) { + pv = &pc->pc_pventry[field * 64 + bit]; + pc->pc_map[field] &= ~(1ul << bit); + /* If this was the last item, move it to tail */ + if (pc->pc_map[0] == 0 && pc->pc_map[1] == 0 && + pc->pc_map[2] == 0) { + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + TAILQ_INSERT_TAIL(&pmap->pm_pvchunk, pc, + pc_list); + } + PV_STAT(atomic_add_long(&pv_entry_count, 1)); + PV_STAT(atomic_subtract_int(&pv_entry_spare, 1)); + return (pv); + } + } + /* No free items, allocate another chunk */ + m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ | + VM_ALLOC_WIRED); + if (m == NULL) { + if (lockp == NULL) { + PV_STAT(pc_chunk_tryfail++); + return (NULL); + } + m = reclaim_pv_chunk(pmap, lockp); + if (m == NULL) + goto retry; + } + PV_STAT(atomic_add_int(&pc_chunk_count, 1)); + PV_STAT(atomic_add_int(&pc_chunk_allocs, 1)); +#if 0 /* TODO: This is for minidump */ + dump_add_page(m->phys_addr); +#endif + pc = (void *)PHYS_TO_DMAP(m->phys_addr); + pc->pc_pmap = pmap; + pc->pc_map[0] = PC_FREE0 & ~1ul; /* preallocated bit 0 */ + pc->pc_map[1] = PC_FREE1; + pc->pc_map[2] = PC_FREE2; + mtx_lock(&pv_chunks_mutex); + TAILQ_INSERT_TAIL(&pv_chunks, pc, pc_lru); + mtx_unlock(&pv_chunks_mutex); + pv = &pc->pc_pventry[0]; + TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); + PV_STAT(atomic_add_long(&pv_entry_count, 1)); + PV_STAT(atomic_add_int(&pv_entry_spare, _NPCPV - 1)); + return (pv); +} + +/* + * First find and then remove the pv entry for the specified pmap and virtual + * address from the specified pv list. Returns the pv entry if found and NULL + * otherwise. This operation can be performed on pv lists for either 4KB or + * 2MB page mappings. + */ +static __inline pv_entry_t +pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +{ + pv_entry_t pv; + + rw_assert(&pvh_global_lock, RA_LOCKED); + TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { + if (pmap == PV_PMAP(pv) && va == pv->pv_va) { + TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); + pvh->pv_gen++; + break; + } + } + return (pv); +} + +/* + * First find and then destroy the pv entry for the specified pmap and virtual + * address. This operation can be performed on pv lists for either 4KB or 2MB + * page mappings. + */ +static void +pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va) +{ + pv_entry_t pv; + + pv = pmap_pvh_remove(pvh, pmap, va); + KASSERT(pv != NULL, ("pmap_pvh_free: pv not found")); + free_pv_entry(pmap, pv); +} + +/* + * Conditionally create the PV entry for a 4KB page mapping if the required + * memory can be allocated without resorting to reclamation. + */ +static boolean_t +pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, vm_page_t m, + struct rwlock **lockp) +{ + pv_entry_t pv; + + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + /* Pass NULL instead of the lock pointer to disable reclamation. */ + if ((pv = get_pv_entry(pmap, NULL)) != NULL) { + pv->pv_va = va; + CHANGE_PV_LIST_LOCK_TO_VM_PAGE(lockp, m); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + return (TRUE); + } else + return (FALSE); +} + +/* + * pmap_remove_l3: do the things to unmap a page in a process + */ +static int +pmap_remove_l3(pmap_t pmap, pt_entry_t *l3, vm_offset_t va, + pd_entry_t l2e, struct spglist *free, struct rwlock **lockp) +{ + pt_entry_t old_l3; + vm_page_t m; + + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + if (pmap_is_current(pmap) && pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(va, L3_SIZE); + old_l3 = pmap_load_clear(l3); + PTE_SYNC(l3); + if (old_l3 & ATTR_SW_WIRED) + pmap->pm_stats.wired_count -= 1; + pmap_resident_count_dec(pmap, 1); + if (old_l3 & ATTR_SW_MANAGED) { + m = PHYS_TO_VM_PAGE(old_l3 & ~ATTR_MASK); + if (pmap_page_dirty(old_l3)) + vm_page_dirty(m); + if (old_l3 & ATTR_AF) + vm_page_aflag_set(m, PGA_REFERENCED); + CHANGE_PV_LIST_LOCK_TO_VM_PAGE(lockp, m); + pmap_pvh_free(&m->md, pmap, va); + } + return (pmap_unuse_l3(pmap, va, l2e, free)); +} + +/* + * Remove the given range of addresses from the specified map. + * + * It is assumed that the start and end are properly + * rounded to the page size. + */ +void +pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +{ + struct rwlock *lock; + vm_offset_t va, va_next; + pd_entry_t *l1, *l2; + pt_entry_t l3_paddr, *l3; + struct spglist free; + int anyvalid; + + /* + * Perform an unsynchronized read. This is, however, safe. + */ + if (pmap->pm_stats.resident_count == 0) + return; + + anyvalid = 0; + SLIST_INIT(&free); + + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + + lock = NULL; + for (; sva < eva; sva = va_next) { + + if (pmap->pm_stats.resident_count == 0) + break; + + l1 = pmap_l1(pmap, sva); + if (*l1 == 0) { + va_next = (sva + L1_SIZE) & ~L1_OFFSET; + if (va_next < sva) + va_next = eva; + continue; + } + + /* + * Calculate index for next page table. + */ + va_next = (sva + L2_SIZE) & ~L2_OFFSET; + if (va_next < sva) + va_next = eva; + + l2 = pmap_l1_to_l2(l1, sva); + if (l2 == NULL) + continue; + + l3_paddr = *l2; + + /* + * Weed out invalid mappings. + */ + if ((l3_paddr & ATTR_DESCR_MASK) != L2_TABLE) + continue; + + /* + * Limit our scan to either the end of the va represented + * by the current page table page, or to the end of the + * range being removed. + */ + if (va_next > eva) + va_next = eva; + + va = va_next; + for (l3 = pmap_l2_to_l3(l2, sva); sva != va_next; l3++, + sva += L3_SIZE) { + if (l3 == NULL) + panic("l3 == NULL"); + if (*l3 == 0) { + if (va != va_next) { + pmap_invalidate_range(pmap, va, sva); + va = va_next; + } + continue; + } + if (va == va_next) + va = sva; + if (pmap_remove_l3(pmap, l3, sva, l3_paddr, &free, + &lock)) { + sva += L3_SIZE; + break; + } + } + if (va != va_next) + pmap_invalidate_range(pmap, va, sva); + } + if (lock != NULL) + rw_wunlock(lock); + if (anyvalid) + pmap_invalidate_all(pmap); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + pmap_free_zero_pages(&free); +} + +/* + * Routine: pmap_remove_all + * Function: + * Removes this physical page from + * all physical maps in which it resides. + * Reflects back modify bits to the pager. + * + * Notes: + * Original versions of this routine were very + * inefficient because they iteratively called + * pmap_remove (slow...) + */ + +void +pmap_remove_all(vm_page_t m) +{ + pv_entry_t pv; + pmap_t pmap; + pt_entry_t *l3, tl3; + pd_entry_t *l2; + struct spglist free; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_remove_all: page %p is not managed", m)); + SLIST_INIT(&free); + rw_wlock(&pvh_global_lock); + while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { + pmap = PV_PMAP(pv); + PMAP_LOCK(pmap); + pmap_resident_count_dec(pmap, 1); + l2 = pmap_l2(pmap, pv->pv_va); + KASSERT((*l2 & ATTR_DESCR_MASK) == L2_TABLE, + ("pmap_remove_all: found a table when expecting " + "a block in %p's pv list", m)); + l3 = pmap_l2_to_l3(l2, pv->pv_va); + if (pmap_is_current(pmap) && + pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(pv->pv_va, L3_SIZE); + tl3 = pmap_load_clear(l3); + PTE_SYNC(l3); + if (tl3 & ATTR_SW_WIRED) + pmap->pm_stats.wired_count--; + if ((tl3 & ATTR_AF) != 0) + vm_page_aflag_set(m, PGA_REFERENCED); + + /* + * Update the vm_page_t clean and reference bits. + */ + if (pmap_page_dirty(tl3)) + vm_page_dirty(m); + pmap_unuse_l3(pmap, pv->pv_va, *l2, &free); + pmap_invalidate_page(pmap, pv->pv_va); + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + free_pv_entry(pmap, pv); + PMAP_UNLOCK(pmap); + } + vm_page_aflag_clear(m, PGA_WRITEABLE); + rw_wunlock(&pvh_global_lock); + pmap_free_zero_pages(&free); +} + +/* + * Set the physical protection on the + * specified range of this map as requested. + */ +void +pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) +{ + vm_offset_t va, va_next; + pd_entry_t *l1, *l2; + pt_entry_t *l3p, l3; + + if ((prot & VM_PROT_READ) == VM_PROT_NONE) { + pmap_remove(pmap, sva, eva); + return; + } + + if ((prot & VM_PROT_WRITE) == VM_PROT_WRITE) + return; + + PMAP_LOCK(pmap); + for (; sva < eva; sva = va_next) { + + l1 = pmap_l1(pmap, sva); + if (*l1 == 0) { + va_next = (sva + L1_SIZE) & ~L1_OFFSET; + if (va_next < sva) + va_next = eva; + continue; + } + + va_next = (sva + L2_SIZE) & ~L2_OFFSET; + if (va_next < sva) + va_next = eva; + + l2 = pmap_l1_to_l2(l1, sva); + if (l2 == NULL || (*l2 & ATTR_DESCR_MASK) != L2_TABLE) + continue; + + if (va_next > eva) + va_next = eva; + + va = va_next; + for (l3p = pmap_l2_to_l3(l2, sva); sva != va_next; l3p++, + sva += L3_SIZE) { + l3 = pmap_load(l3p); + if (pmap_l3_valid(l3)) { + pmap_set(l3p, ATTR_AP(ATTR_AP_RO)); + PTE_SYNC(l3p); + } + } + } + PMAP_UNLOCK(pmap); + + /* TODO: Only invalidate entries we are touching */ + pmap_invalidate_all(pmap); +} + +/* + * Insert the given physical page (p) at + * the specified virtual address (v) in the + * target physical map with the protection requested. + * + * If specified, the page will be wired down, meaning + * that the related pte can not be reclaimed. + * + * NB: This is the only routine which MAY NOT lazy-evaluate + * or lose information. That is, this routine must actually + * insert this page into the given map NOW. + */ +int +pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, + u_int flags, int8_t psind __unused) +{ + struct rwlock *lock; + pd_entry_t *l1, *l2; + pt_entry_t new_l3, orig_l3; + pt_entry_t *l3; + pv_entry_t pv; + vm_paddr_t opa, pa, l2_pa, l3_pa; + vm_page_t mpte, om, l2_m, l3_m; + boolean_t nosleep; + + va = trunc_page(va); + if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m)) + VM_OBJECT_ASSERT_LOCKED(m->object); + pa = VM_PAGE_TO_PHYS(m); + new_l3 = (pt_entry_t)(pa | ATTR_AF | L3_PAGE); + if ((prot & VM_PROT_WRITE) == 0) + new_l3 |= ATTR_AP(ATTR_AP_RO); + if ((flags & PMAP_ENTER_WIRED) != 0) + new_l3 |= ATTR_SW_WIRED; + if ((va >> 63) == 0) + new_l3 |= ATTR_AP(ATTR_AP_USER); + new_l3 |= ATTR_IDX(m->md.pv_memattr); + + mpte = NULL; + + lock = NULL; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + + if (va < VM_MAXUSER_ADDRESS) { + nosleep = (flags & PMAP_ENTER_NOSLEEP) != 0; + mpte = pmap_alloc_l3(pmap, va, nosleep ? NULL : &lock); + if (mpte == NULL && nosleep) { + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + return (KERN_RESOURCE_SHORTAGE); + } + l3 = pmap_l3(pmap, va); + } else { + l3 = pmap_l3(pmap, va); + /* TODO: This is not optimal, but should mostly work */ + if (l3 == NULL) { + l2 = pmap_l2(pmap, va); + + if (l2 == NULL) { + l2_m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | + VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | + VM_ALLOC_ZERO); + if (l2_m == NULL) + panic("pmap_enter: l2 pte_m == NULL"); + if ((l2_m->flags & PG_ZERO) == 0) + pmap_zero_page(l2_m); + + l2_pa = VM_PAGE_TO_PHYS(l2_m); + l1 = pmap_l1(pmap, va); + pmap_load_store(l1, l2_pa | L1_TABLE); + PTE_SYNC(l1); + l2 = pmap_l1_to_l2(l1, va); + } + + KASSERT(l2 != NULL, + ("No l2 table after allocating one")); + + l3_m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | + VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO); + if (l3_m == NULL) + panic("pmap_enter: l3 pte_m == NULL"); + if ((l3_m->flags & PG_ZERO) == 0) + pmap_zero_page(l3_m); + + l3_pa = VM_PAGE_TO_PHYS(l3_m); + pmap_load_store(l2, l3_pa | L2_TABLE); + PTE_SYNC(l2); + l3 = pmap_l2_to_l3(l2, va); + } + } + + om = NULL; + orig_l3 = pmap_load(l3); + opa = orig_l3 & ~ATTR_MASK; + + /* + * Is the specified virtual address already mapped? + */ + if (pmap_l3_valid(orig_l3)) { + /* + * Wiring change, just update stats. We don't worry about + * wiring PT pages as they remain resident as long as there + * are valid mappings in them. Hence, if a user page is wired, + * the PT page will be also. + */ + if ((flags & PMAP_ENTER_WIRED) != 0 && + (orig_l3 & ATTR_SW_WIRED) == 0) + pmap->pm_stats.wired_count++; + else if ((flags & PMAP_ENTER_WIRED) == 0 && + (orig_l3 & ATTR_SW_WIRED) != 0) + pmap->pm_stats.wired_count--; + + /* + * Remove the extra PT page reference. + */ + if (mpte != NULL) { + mpte->wire_count--; + KASSERT(mpte->wire_count > 0, + ("pmap_enter: missing reference to page table page," + " va: 0x%lx", va)); + } + + /* + * Has the physical page changed? + */ + if (opa == pa) { + /* + * No, might be a protection or wiring change. + */ + if ((orig_l3 & ATTR_SW_MANAGED) != 0) { + new_l3 |= ATTR_SW_MANAGED; + if ((new_l3 & ATTR_AP(ATTR_AP_RW)) == + ATTR_AP(ATTR_AP_RW)) { + vm_page_aflag_set(m, PGA_WRITEABLE); + } + } + goto validate; + } + + /* Flush the cache, there might be uncommitted data in it */ + if (pmap_is_current(pmap) && pmap_l3_valid_cacheable(orig_l3)) + cpu_dcache_wb_range(va, L3_SIZE); + } else { + /* + * Increment the counters. + */ + if ((new_l3 & ATTR_SW_WIRED) != 0) + pmap->pm_stats.wired_count++; + pmap_resident_count_inc(pmap, 1); + } + /* + * Enter on the PV list if part of our managed memory. + */ + if ((m->oflags & VPO_UNMANAGED) == 0) { + new_l3 |= ATTR_SW_MANAGED; + pv = get_pv_entry(pmap, &lock); + pv->pv_va = va; + CHANGE_PV_LIST_LOCK_TO_PHYS(&lock, pa); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + if ((new_l3 & ATTR_AP_RW_BIT) == ATTR_AP(ATTR_AP_RW)) + vm_page_aflag_set(m, PGA_WRITEABLE); + } + + /* + * Update the L3 entry. + */ + if (orig_l3 != 0) { +validate: + orig_l3 = pmap_load_store(l3, new_l3); + PTE_SYNC(l3); + opa = orig_l3 & ~ATTR_MASK; + + if (opa != pa) { + if ((orig_l3 & ATTR_SW_MANAGED) != 0) { + om = PHYS_TO_VM_PAGE(opa); + if (pmap_page_dirty(orig_l3)) + vm_page_dirty(om); + if ((orig_l3 & ATTR_AF) != 0) + vm_page_aflag_set(om, PGA_REFERENCED); + CHANGE_PV_LIST_LOCK_TO_PHYS(&lock, opa); + pmap_pvh_free(&om->md, pmap, va); + } + } else if (pmap_page_dirty(orig_l3)) { + if ((orig_l3 & ATTR_SW_MANAGED) != 0) + vm_page_dirty(m); + } + if ((orig_l3 & ATTR_AF) != 0) + pmap_invalidate_page(pmap, va); + } else { + pmap_load_store(l3, new_l3); + PTE_SYNC(l3); + } + if ((pmap != pmap_kernel()) && (pmap == &curproc->p_vmspace->vm_pmap)) + cpu_icache_sync_range(va, PAGE_SIZE); + + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + return (KERN_SUCCESS); +} + +/* + * Maps a sequence of resident pages belonging to the same object. + * The sequence begins with the given page m_start. This page is + * mapped at the given virtual address start. Each subsequent page is + * mapped at a virtual address that is offset from start by the same + * amount as the page is offset from m_start within the object. The + * last page in the sequence is the page with the largest offset from + * m_start that can be mapped at a virtual address less than the given + * virtual address end. Not every virtual page between start and end + * is mapped; only those for which a resident page exists with the + * corresponding offset from m_start are mapped. + */ +void +pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, + vm_page_t m_start, vm_prot_t prot) +{ + struct rwlock *lock; + vm_offset_t va; + vm_page_t m, mpte; + vm_pindex_t diff, psize; + + VM_OBJECT_ASSERT_LOCKED(m_start->object); + + psize = atop(end - start); + mpte = NULL; + m = m_start; + lock = NULL; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { + va = start + ptoa(diff); + mpte = pmap_enter_quick_locked(pmap, va, m, prot, mpte, &lock); + m = TAILQ_NEXT(m, listq); + } + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); +} + +/* + * this code makes some *MAJOR* assumptions: + * 1. Current pmap & pmap exists. + * 2. Not wired. + * 3. Read access. + * 4. No page table pages. + * but is *MUCH* faster than pmap_enter... + */ + +void +pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) +{ + struct rwlock *lock; + + lock = NULL; + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + (void)pmap_enter_quick_locked(pmap, va, m, prot, NULL, &lock); + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); +} + +static vm_page_t +pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, + vm_prot_t prot, vm_page_t mpte, struct rwlock **lockp) +{ + struct spglist free; + pd_entry_t *l2; + pt_entry_t *l3; + vm_paddr_t pa; + + KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva || + (m->oflags & VPO_UNMANAGED) != 0, + ("pmap_enter_quick_locked: managed mapping within the clean submap")); + rw_assert(&pvh_global_lock, RA_LOCKED); + PMAP_LOCK_ASSERT(pmap, MA_OWNED); + + /* + * In the case that a page table page is not + * resident, we are creating it here. + */ + if (va < VM_MAXUSER_ADDRESS) { + vm_pindex_t l2pindex; + + /* + * Calculate pagetable page index + */ + l2pindex = pmap_l2_pindex(va); + if (mpte && (mpte->pindex == l2pindex)) { + mpte->wire_count++; + } else { + /* + * Get the l2 entry + */ + l2 = pmap_l2(pmap, va); + + /* + * If the page table page is mapped, we just increment + * the hold count, and activate it. Otherwise, we + * attempt to allocate a page table page. If this + * attempt fails, we don't retry. Instead, we give up. + */ + if (l2 != NULL && *l2 != 0) { + mpte = PHYS_TO_VM_PAGE(*l2 & ~ATTR_MASK); + mpte->wire_count++; + } else { + /* + * Pass NULL instead of the PV list lock + * pointer, because we don't intend to sleep. + */ + mpte = _pmap_alloc_l3(pmap, l2pindex, NULL); + if (mpte == NULL) + return (mpte); + } + } + l3 = (pt_entry_t *)PHYS_TO_DMAP(VM_PAGE_TO_PHYS(mpte)); + l3 = &l3[pmap_l3_index(va)]; + } else { + mpte = NULL; + l3 = pmap_l3(kernel_pmap, va); + } + if (l3 == NULL) + panic("pmap_enter_quick_locked: No l3"); + if (*l3) { + if (mpte != NULL) { + mpte->wire_count--; + mpte = NULL; + } + return (mpte); + } + + /* + * Enter on the PV list if part of our managed memory. + */ + if ((m->oflags & VPO_UNMANAGED) == 0 && + !pmap_try_insert_pv_entry(pmap, va, m, lockp)) { + if (mpte != NULL) { + SLIST_INIT(&free); + if (pmap_unwire_l3(pmap, va, mpte, &free)) { + pmap_invalidate_page(pmap, va); + pmap_free_zero_pages(&free); + } + mpte = NULL; + } + return (mpte); + } + + /* + * Increment counters + */ + pmap_resident_count_inc(pmap, 1); + + pa = VM_PAGE_TO_PHYS(m) | ATTR_AF | ATTR_IDX(m->md.pv_memattr) | + ATTR_AP(ATTR_AP_RW) | L3_PAGE; + + /* + * Now validate mapping with RO protection + */ + if ((m->oflags & VPO_UNMANAGED) == 0) + pa |= ATTR_SW_MANAGED; + pmap_load_store(l3, pa); + PTE_SYNC(l3); + pmap_invalidate_page(pmap, va); + return (mpte); +} + +/* + * This code maps large physical mmap regions into the + * processor address space. Note that some shortcuts + * are taken, but the code works. + */ +void +pmap_object_init_pt(pmap_t pmap, vm_offset_t addr, vm_object_t object, + vm_pindex_t pindex, vm_size_t size) +{ + + panic("pmap_object_init_pt"); +} + +/* + * Clear the wired attribute from the mappings for the specified range of + * addresses in the given pmap. Every valid mapping within that range + * must have the wired attribute set. In contrast, invalid mappings + * cannot have the wired attribute set, so they are ignored. + * + * The wired attribute of the page table entry is not a hardware feature, + * so there is no need to invalidate any TLB entries. + */ +void +pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) +{ + vm_offset_t va_next; + pd_entry_t *l1, *l2; + pt_entry_t *l3; + boolean_t pv_lists_locked; + + pv_lists_locked = FALSE; + PMAP_LOCK(pmap); + for (; sva < eva; sva = va_next) { + l1 = pmap_l1(pmap, sva); + if (*l1 == 0) { + va_next = (sva + L1_SIZE) & ~L1_OFFSET; + if (va_next < sva) + va_next = eva; + continue; + } + + va_next = (sva + L2_SIZE) & ~L2_OFFSET; + if (va_next < sva) + va_next = eva; + + l2 = pmap_l1_to_l2(l1, sva); + if (*l2 == 0) + continue; + + if (va_next > eva) + va_next = eva; + for (l3 = pmap_l2_to_l3(l2, sva); sva != va_next; l3++, + sva += L3_SIZE) { + if (*l3 == 0) + continue; + if ((*l3 & ATTR_SW_WIRED) == 0) + panic("pmap_unwire: l3 %#jx is missing " + "ATTR_SW_WIRED", (uintmax_t)*l3); + + /* + * PG_W must be cleared atomically. Although the pmap + * lock synchronizes access to PG_W, another processor + * could be setting PG_M and/or PG_A concurrently. + */ + atomic_clear_long(l3, ATTR_SW_WIRED); + pmap->pm_stats.wired_count--; + } + } + if (pv_lists_locked) + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); +} + +/* + * Copy the range specified by src_addr/len + * from the source map to the range dst_addr/len + * in the destination map. + * + * This routine is only advisory and need not do anything. + */ + +void +pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t len, + vm_offset_t src_addr) +{ +} + +/* + * pmap_zero_page zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. + */ +void +pmap_zero_page(vm_page_t m) +{ + vm_offset_t va = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m)); + + pagezero((void *)va); +} + +/* + * pmap_zero_page_area zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. + * + * off and size may not cover an area beyond a single hardware page. + */ +void +pmap_zero_page_area(vm_page_t m, int off, int size) +{ + vm_offset_t va = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m)); + + if (off == 0 && size == PAGE_SIZE) + pagezero((void *)va); + else + bzero((char *)va + off, size); +} + +/* + * pmap_zero_page_idle zeros the specified hardware page by mapping + * the page into KVM and using bzero to clear its contents. This + * is intended to be called from the vm_pagezero process only and + * outside of Giant. + */ +void +pmap_zero_page_idle(vm_page_t m) +{ + vm_offset_t va = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m)); + + pagezero((void *)va); +} + +/* + * pmap_copy_page copies the specified (machine independent) + * page by mapping the page into virtual memory and using + * bcopy to copy the page, one machine dependent page at a + * time. + */ +void +pmap_copy_page(vm_page_t msrc, vm_page_t mdst) +{ + vm_offset_t src = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(msrc)); + vm_offset_t dst = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(mdst)); + + pagecopy((void *)src, (void *)dst); +} + +int unmapped_buf_allowed = 1; + +void +pmap_copy_pages(vm_page_t ma[], vm_offset_t a_offset, vm_page_t mb[], + vm_offset_t b_offset, int xfersize) +{ + void *a_cp, *b_cp; + vm_page_t m_a, m_b; + vm_paddr_t p_a, p_b; + vm_offset_t a_pg_offset, b_pg_offset; + int cnt; + + while (xfersize > 0) { + a_pg_offset = a_offset & PAGE_MASK; + m_a = ma[a_offset >> PAGE_SHIFT]; + p_a = m_a->phys_addr; + b_pg_offset = b_offset & PAGE_MASK; + m_b = mb[b_offset >> PAGE_SHIFT]; + p_b = m_b->phys_addr; + cnt = min(xfersize, PAGE_SIZE - a_pg_offset); + cnt = min(cnt, PAGE_SIZE - b_pg_offset); + if (__predict_false(!PHYS_IN_DMAP(p_a))) { + panic("!DMAP a %lx", p_a); + } else { + a_cp = (char *)PHYS_TO_DMAP(p_a) + a_pg_offset; + } + if (__predict_false(!PHYS_IN_DMAP(p_b))) { + panic("!DMAP b %lx", p_b); + } else { + b_cp = (char *)PHYS_TO_DMAP(p_b) + b_pg_offset; + } + bcopy(a_cp, b_cp, cnt); + a_offset += cnt; + b_offset += cnt; + xfersize -= cnt; + } +} + +/* + * Returns true if the pmap's pv is one of the first + * 16 pvs linked to from this page. This count may + * be changed upwards or downwards in the future; it + * is only necessary that true be returned for a small + * subset of pmaps for proper page aging. + */ +boolean_t +pmap_page_exists_quick(pmap_t pmap, vm_page_t m) +{ + struct rwlock *lock; + pv_entry_t pv; + int loops = 0; + boolean_t rv; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_page_exists_quick: page %p is not managed", m)); + rv = FALSE; + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); + rw_rlock(lock); + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + if (PV_PMAP(pv) == pmap) { + rv = TRUE; + break; + } + loops++; + if (loops >= 16) + break; + } + rw_runlock(lock); + rw_runlock(&pvh_global_lock); + return (rv); +} + +/* + * pmap_page_wired_mappings: + * + * Return the number of managed mappings to the given physical page + * that are wired. + */ +int +pmap_page_wired_mappings(vm_page_t m) +{ + struct rwlock *lock; + pmap_t pmap; + pt_entry_t *l3; + pv_entry_t pv; + int count, md_gen; + + if ((m->oflags & VPO_UNMANAGED) != 0) + return (0); + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); + rw_rlock(lock); +restart: + count = 0; + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_runlock(lock); + PMAP_LOCK(pmap); + rw_rlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + goto restart; + } + } + l3 = pmap_l3(pmap, pv->pv_va); + if (l3 != NULL && (*l3 & ATTR_SW_WIRED) != 0) + count++; + PMAP_UNLOCK(pmap); + } + rw_runlock(lock); + rw_runlock(&pvh_global_lock); + return (count); +} + +/* + * Destroy all managed, non-wired mappings in the given user-space + * pmap. This pmap cannot be active on any processor besides the + * caller. + * + * This function cannot be applied to the kernel pmap. Moreover, it + * is not intended for general use. It is only to be used during + * process termination. Consequently, it can be implemented in ways + * that make it faster than pmap_remove(). First, it can more quickly + * destroy mappings by iterating over the pmap's collection of PV + * entries, rather than searching the page table. Second, it doesn't + * have to test and clear the page table entries atomically, because + * no processor is currently accessing the user address space. In + * particular, a page table entry's dirty bit won't change state once + * this function starts. + */ +void +pmap_remove_pages(pmap_t pmap) +{ + pd_entry_t ptepde, *l2; + pt_entry_t *l3, tl3; + struct spglist free; + vm_page_t m; + pv_entry_t pv; + struct pv_chunk *pc, *npc; + struct rwlock *lock; + int64_t bit; + uint64_t inuse, bitmask; + int allfree, field, freed, idx; + vm_paddr_t pa; + + lock = NULL; + + SLIST_INIT(&free); + rw_rlock(&pvh_global_lock); + PMAP_LOCK(pmap); + TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { + allfree = 1; + freed = 0; + for (field = 0; field < _NPCM; field++) { + inuse = ~pc->pc_map[field] & pc_freemask[field]; + while (inuse != 0) { + bit = ffsl(inuse) - 1; + bitmask = 1UL << bit; + idx = field * 64 + bit; + pv = &pc->pc_pventry[idx]; + inuse &= ~bitmask; + + l2 = pmap_l2(pmap, pv->pv_va); + ptepde = pmap_load(l2); + l3 = pmap_l2_to_l3(l2, pv->pv_va); + tl3 = pmap_load(l3); + +/* + * We cannot remove wired pages from a process' mapping at this time + */ + if (tl3 & ATTR_SW_WIRED) { + allfree = 0; + continue; + } + + pa = tl3 & ~ATTR_MASK; + + m = PHYS_TO_VM_PAGE(pa); + KASSERT(m->phys_addr == pa, + ("vm_page_t %p phys_addr mismatch %016jx %016jx", + m, (uintmax_t)m->phys_addr, + (uintmax_t)tl3)); + + KASSERT((m->flags & PG_FICTITIOUS) != 0 || + m < &vm_page_array[vm_page_array_size], + ("pmap_remove_pages: bad l3 %#jx", + (uintmax_t)tl3)); + + if (pmap_is_current(pmap) && + pmap_l3_valid_cacheable(pmap_load(l3))) + cpu_dcache_wb_range(pv->pv_va, L3_SIZE); + pmap_load_clear(l3); + PTE_SYNC(l3); + + /* + * Update the vm_page_t clean/reference bits. + */ + if ((tl3 & ATTR_AP_RW_BIT) == + ATTR_AP(ATTR_AP_RW)) + vm_page_dirty(m); + + CHANGE_PV_LIST_LOCK_TO_VM_PAGE(&lock, m); + + /* Mark free */ + pc->pc_map[field] |= bitmask; + + pmap_resident_count_dec(pmap, 1); + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + + pmap_unuse_l3(pmap, pv->pv_va, ptepde, &free); + freed++; + } + } + PV_STAT(atomic_add_long(&pv_entry_frees, freed)); + PV_STAT(atomic_add_int(&pv_entry_spare, freed)); + PV_STAT(atomic_subtract_long(&pv_entry_count, freed)); + if (allfree) { + TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); + free_pv_chunk(pc); + } + } + pmap_invalidate_all(pmap); + if (lock != NULL) + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + PMAP_UNLOCK(pmap); + pmap_free_zero_pages(&free); +} + +/* + * This is used to check if a page has been accessed or modified. As we + * don't have a bit to see if it has been modified we have to assume it + * has been if the page is read/write. + */ +static boolean_t +pmap_page_test_mappings(vm_page_t m, boolean_t accessed, boolean_t modified) +{ + struct rwlock *lock; + pv_entry_t pv; + pt_entry_t *l3, mask, value; + pmap_t pmap; + int md_gen; + boolean_t rv; + + rv = FALSE; + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); + rw_rlock(lock); +restart: + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_runlock(lock); + PMAP_LOCK(pmap); + rw_rlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + goto restart; + } + } + l3 = pmap_l3(pmap, pv->pv_va); + mask = 0; + value = 0; + if (modified) { + mask |= ATTR_AP_RW_BIT; + value |= ATTR_AP(ATTR_AP_RW); + } + if (accessed) { + mask |= ATTR_AF | ATTR_DESCR_MASK; + value |= ATTR_AF | L3_PAGE; + } + rv = (pmap_load(l3) & mask) == value; + PMAP_UNLOCK(pmap); + if (rv) + goto out; + } +out: + rw_runlock(lock); + rw_runlock(&pvh_global_lock); + return (rv); +} + +/* + * pmap_is_modified: + * + * Return whether or not the specified physical page was modified + * in any physical maps. + */ +boolean_t +pmap_is_modified(vm_page_t m) +{ + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_is_modified: page %p is not managed", m)); + + /* + * If the page is not exclusive busied, then PGA_WRITEABLE cannot be + * concurrently set while the object is locked. Thus, if PGA_WRITEABLE + * is clear, no PTEs can have PG_M set. + */ + VM_OBJECT_ASSERT_WLOCKED(m->object); + if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) + return (FALSE); + return (pmap_page_test_mappings(m, FALSE, TRUE)); +} + +/* + * pmap_is_prefaultable: + * + * Return whether or not the specified virtual address is eligible + * for prefault. + */ +boolean_t +pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr) +{ + pt_entry_t *l3; + boolean_t rv; + + rv = FALSE; + PMAP_LOCK(pmap); + l3 = pmap_l3(pmap, addr); + if (l3 != NULL && *l3 != 0) { + rv = TRUE; + } + PMAP_UNLOCK(pmap); + return (rv); +} + +/* + * pmap_is_referenced: + * + * Return whether or not the specified physical page was referenced + * in any physical maps. + */ +boolean_t +pmap_is_referenced(vm_page_t m) +{ + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_is_referenced: page %p is not managed", m)); + return (pmap_page_test_mappings(m, TRUE, FALSE)); +} + +/* + * Clear the write and modified bits in each of the given page's mappings. + */ +void +pmap_remove_write(vm_page_t m) +{ + pmap_t pmap; + struct rwlock *lock; + pv_entry_t pv; + pt_entry_t *l3, oldl3; + int md_gen; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_remove_write: page %p is not managed", m)); + + /* + * If the page is not exclusive busied, then PGA_WRITEABLE cannot be + * set by another thread while the object is locked. Thus, + * if PGA_WRITEABLE is clear, no page table entries need updating. + */ + VM_OBJECT_ASSERT_WLOCKED(m->object); + if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) + return; + rw_rlock(&pvh_global_lock); + lock = VM_PAGE_TO_PV_LIST_LOCK(m); +retry_pv_loop: + rw_wlock(lock); + TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_wunlock(lock); + PMAP_LOCK(pmap); + rw_wlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + rw_wunlock(lock); + goto retry_pv_loop; + } + } + l3 = pmap_l3(pmap, pv->pv_va); +retry: + oldl3 = *l3; + if ((oldl3 & ATTR_AP_RW_BIT) == ATTR_AP(ATTR_AP_RW)) { + if (!atomic_cmpset_long(l3, oldl3, + oldl3 | ATTR_AP(ATTR_AP_RO))) + goto retry; + if ((oldl3 & ATTR_AF) != 0) + vm_page_dirty(m); + pmap_invalidate_page(pmap, pv->pv_va); + } + PMAP_UNLOCK(pmap); + } + rw_wunlock(lock); + vm_page_aflag_clear(m, PGA_WRITEABLE); + rw_runlock(&pvh_global_lock); +} + +static __inline boolean_t +safe_to_clear_referenced(pmap_t pmap, pt_entry_t pte) +{ + + return (FALSE); +} + +#define PMAP_TS_REFERENCED_MAX 5 + +/* + * pmap_ts_referenced: + * + * Return a count of reference bits for a page, clearing those bits. + * It is not necessary for every reference bit to be cleared, but it + * is necessary that 0 only be returned when there are truly no + * reference bits set. + * + * XXX: The exact number of bits to check and clear is a matter that + * should be tested and standardized at some point in the future for + * optimal aging of shared pages. + */ +int +pmap_ts_referenced(vm_page_t m) +{ + pv_entry_t pv, pvf; + pmap_t pmap; + struct rwlock *lock; + pd_entry_t *l2; + pt_entry_t *l3; + vm_paddr_t pa; + int cleared, md_gen, not_cleared; + struct spglist free; + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_ts_referenced: page %p is not managed", m)); + SLIST_INIT(&free); + cleared = 0; + pa = VM_PAGE_TO_PHYS(m); + lock = PHYS_TO_PV_LIST_LOCK(pa); + rw_rlock(&pvh_global_lock); + rw_wlock(lock); +retry: + not_cleared = 0; + if ((pvf = TAILQ_FIRST(&m->md.pv_list)) == NULL) + goto out; + pv = pvf; + do { + if (pvf == NULL) + pvf = pv; + pmap = PV_PMAP(pv); + if (!PMAP_TRYLOCK(pmap)) { + md_gen = m->md.pv_gen; + rw_wunlock(lock); + PMAP_LOCK(pmap); + rw_wlock(lock); + if (md_gen != m->md.pv_gen) { + PMAP_UNLOCK(pmap); + goto retry; + } + } + l2 = pmap_l2(pmap, pv->pv_va); + KASSERT((*l2 & ATTR_DESCR_MASK) == L2_TABLE, + ("pmap_ts_referenced: found an invalid l2 table")); + l3 = pmap_l2_to_l3(l2, pv->pv_va); + if ((*l3 & ATTR_AF) != 0) { + if (safe_to_clear_referenced(pmap, *l3)) { + /* + * TODO: We don't handle the access flag + * at all. We need to be able to set it in + * the exception handler. + */ + panic("TODO: safe_to_clear_referenced\n"); + } else if ((*l3 & ATTR_SW_WIRED) == 0) { + /* + * Wired pages cannot be paged out so + * doing accessed bit emulation for + * them is wasted effort. We do the + * hard work for unwired pages only. + */ + pmap_remove_l3(pmap, l3, pv->pv_va, + *l2, &free, &lock); + pmap_invalidate_page(pmap, pv->pv_va); + cleared++; + if (pvf == pv) + pvf = NULL; + pv = NULL; + KASSERT(lock == VM_PAGE_TO_PV_LIST_LOCK(m), + ("inconsistent pv lock %p %p for page %p", + lock, VM_PAGE_TO_PV_LIST_LOCK(m), m)); + } else + not_cleared++; + } + PMAP_UNLOCK(pmap); + /* Rotate the PV list if it has more than one entry. */ + if (pv != NULL && TAILQ_NEXT(pv, pv_next) != NULL) { + TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); + TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); + m->md.pv_gen++; + } + } while ((pv = TAILQ_FIRST(&m->md.pv_list)) != pvf && cleared + + not_cleared < PMAP_TS_REFERENCED_MAX); +out: + rw_wunlock(lock); + rw_runlock(&pvh_global_lock); + pmap_free_zero_pages(&free); + return (cleared + not_cleared); +} + +/* + * Apply the given advice to the specified range of addresses within the + * given pmap. Depending on the advice, clear the referenced and/or + * modified flags in each mapping and set the mapped page's dirty field. + */ +void +pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice) +{ +} + +/* + * Clear the modify bits on the specified physical page. + */ +void +pmap_clear_modify(vm_page_t m) +{ + + KASSERT((m->oflags & VPO_UNMANAGED) == 0, + ("pmap_clear_modify: page %p is not managed", m)); + VM_OBJECT_ASSERT_WLOCKED(m->object); + KASSERT(!vm_page_xbusied(m), + ("pmap_clear_modify: page %p is exclusive busied", m)); + + /* + * If the page is not PGA_WRITEABLE, then no PTEs can have PG_M set. + * If the object containing the page is locked and the page is not + * exclusive busied, then PGA_WRITEABLE cannot be concurrently set. + */ + if ((m->aflags & PGA_WRITEABLE) == 0) + return; + panic("pmap_clear_modify"); +} + +/* + * Sets the memory attribute for the specified page. + */ +void +pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma) +{ + + panic("pmap_page_set_memattr"); +} + +/* + * perform the pmap work for mincore + */ +int +pmap_mincore(pmap_t pmap, vm_offset_t addr, vm_paddr_t *locked_pa) +{ + + panic("pmap_mincore"); +} + +void +pmap_activate(struct thread *td) +{ + pmap_t pmap; + + critical_enter(); + pmap = vmspace_pmap(td->td_proc->p_vmspace); + td->td_pcb->pcb_l1addr = vtophys(pmap->pm_l1); + __asm __volatile("msr ttbr0_el1, %0" : : "r"(td->td_pcb->pcb_l1addr)); + critical_exit(); +} + +void +pmap_sync_icache(pmap_t pm, vm_offset_t va, vm_size_t sz) +{ + + panic("pmap_sync_icache"); +} + +/* + * Increase the starting virtual address of the given mapping if a + * different alignment might result in more superpage mappings. + */ +void +pmap_align_superpage(vm_object_t object, vm_ooffset_t offset, + vm_offset_t *addr, vm_size_t size) +{ +} + +/** + * Get the kernel virtual address of a set of physical pages. If there are + * physical addresses not covered by the DMAP perform a transient mapping + * that will be removed when calling pmap_unmap_io_transient. + * + * \param page The pages the caller wishes to obtain the virtual + * address on the kernel memory map. + * \param vaddr On return contains the kernel virtual memory address + * of the pages passed in the page parameter. + * \param count Number of pages passed in. + * \param can_fault TRUE if the thread using the mapped pages can take + * page faults, FALSE otherwise. + * + * \returns TRUE if the caller must call pmap_unmap_io_transient when + * finished or FALSE otherwise. + * + */ +boolean_t +pmap_map_io_transient(vm_page_t page[], vm_offset_t vaddr[], int count, + boolean_t can_fault) +{ + vm_paddr_t paddr; + boolean_t needs_mapping; + int error, i; + + /* + * Allocate any KVA space that we need, this is done in a separate + * loop to prevent calling vmem_alloc while pinned. + */ + needs_mapping = FALSE; + for (i = 0; i < count; i++) { + paddr = VM_PAGE_TO_PHYS(page[i]); + if (__predict_false(paddr >= DMAP_MAX_PHYSADDR)) { + error = vmem_alloc(kernel_arena, PAGE_SIZE, + M_BESTFIT | M_WAITOK, &vaddr[i]); + KASSERT(error == 0, ("vmem_alloc failed: %d", error)); + needs_mapping = TRUE; + } else { + vaddr[i] = PHYS_TO_DMAP(paddr); + } + } + + /* Exit early if everything is covered by the DMAP */ + if (!needs_mapping) + return (FALSE); + + /* + * NB: The sequence of updating a page table followed by accesses + * to the corresponding pages used in the !DMAP case is subject to + * the situation described in the "AMD64 Architecture Programmer's + * Manual Volume 2: System Programming" rev. 3.23, "7.3.1 Special + * Coherency Considerations". Therefore, issuing the INVLPG right + * after modifying the PTE bits is crucial. + */ + if (!can_fault) + sched_pin(); + for (i = 0; i < count; i++) { + paddr = VM_PAGE_TO_PHYS(page[i]); + if (paddr >= DMAP_MAX_PHYSADDR) { + panic( + "pmap_map_io_transient: TODO: Map out of DMAP data"); + } + } + + return (needs_mapping); +} + +void +pmap_unmap_io_transient(vm_page_t page[], vm_offset_t vaddr[], int count, + boolean_t can_fault) +{ + vm_paddr_t paddr; + int i; + + if (!can_fault) + sched_unpin(); + for (i = 0; i < count; i++) { + paddr = VM_PAGE_TO_PHYS(page[i]); + if (paddr >= DMAP_MAX_PHYSADDR) { + panic("pmap_unmap_io_transient: TODO: Unmap data"); + } + } +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/pmap.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/stack_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/stack_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/stack_machdep.c (revision 281504) @@ -0,0 +1,53 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include + +#include +#include +#include + +void +stack_save_td(struct stack *st, struct thread *td) +{ + panic("stack_save_td"); +} + +void +stack_save(struct stack *st) +{ + panic("stack_save"); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/stack_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/support.S =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/support.S (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/support.S (revision 281504) @@ -0,0 +1,255 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * Copyright (c) 2014-2015 The FreeBSD Foundation + * All rights reserved. + * + * Portions of this software were developed by Andrew Turner + * under sponsorship from the FreeBSD Foundation + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include + +#include "assym.s" + +/* + * One of the fu* or su* functions failed, return -1. + */ +ENTRY(fsu_fault) + SET_FAULT_HANDLER(xzr, x1) /* Reset the handler function */ + mov x0, #-1 + ret +END(fsu_fault) + +/* + * int casueword32(volatile uint32_t *, uint32_t, uint32_t *, uint32_t) + */ +ENTRY(casueword32) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x4) /* And set it */ +1: ldxr w4, [x0] /* Load-exclusive the data */ + cmp w4, w1 /* Compare */ + b.ne 2f /* Not equal, exit */ + stxr w5, w3, [x0] /* Store the new data */ + cbnz w5, 1b /* Retry on failure */ + ldrb w0, [x0] /* Try loading the data */ +2: SET_FAULT_HANDLER(xzr, x5) /* Reset the fault handler */ + str w4, [x2] /* Store the read data */ + ret /* Return */ +END(casueword32) + +/* + * int casueword(volatile u_long *, u_long, u_long *, u_long) + */ +ENTRY(casueword) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x4) /* And set it */ +1: ldxr x4, [x0] /* Load-exclusive the data */ + cmp x4, x1 /* Compare */ + b.ne 2f /* Not equal, exit */ + stxr w5, x3, [x0] /* Store the new data */ + cbnz w5, 1b /* Retry on failure */ + ldrb w0, [x0] /* Try loading the data */ +2: SET_FAULT_HANDLER(xzr, x5) /* Reset the fault handler */ + str x4, [x2] /* Store the read data */ + ret /* Return */ +END(casueword) + +/* + * int fubyte(volatile const void *) + */ +ENTRY(fubyte) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x1) /* And set it */ + ldrb w0, [x0] /* Try loading the data */ + SET_FAULT_HANDLER(xzr, x1) /* Reset the fault handler */ + ret /* Return */ +END(fubyte) + +/* + * int fuword(volatile const void *) + */ +ENTRY(fuword16) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x1) /* And set it */ + ldrh w0, [x0] /* Try loading the data */ + SET_FAULT_HANDLER(xzr, x1) /* Reset the fault handler */ + ret /* Return */ +END(fuword16) + +/* + * int32_t fueword32(volatile const void *, int32_t *) + */ +ENTRY(fueword32) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x2) /* And set it */ + ldr w0, [x0] /* Try loading the data */ + SET_FAULT_HANDLER(xzr, x2) /* Reset the fault handler */ + str w0, [x1] /* Save the data in kernel space */ + mov w0, #0 /* Success */ + ret /* Return */ +END(fueword32) + +/* + * long fueword(volatile const void *, int64_t *) + * int64_t fueword64(volatile const void *, int64_t *) + */ +ENTRY(fueword) +EENTRY(fueword64) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x2) /* And set it */ + ldr x0, [x0] /* Try loading the data */ + SET_FAULT_HANDLER(xzr, x2) /* Reset the fault handler */ + str x0, [x1] /* Save the data in kernel space */ + mov x0, #0 /* Success */ + ret /* Return */ +EEND(fueword64) +END(fueword) + +/* + * int subyte(volatile void *, int) + */ +ENTRY(subyte) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x2) /* And set it */ + strb w1, [x0] /* Try storing the data */ + SET_FAULT_HANDLER(xzr, x2) /* Reset the fault handler */ + mov x0, #0 /* Success */ + ret /* Return */ +END(subyte) + +/* + * int suword16(volatile void *, int) + */ +ENTRY(suword16) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x2) /* And set it */ + strh w1, [x0] /* Try storing the data */ + SET_FAULT_HANDLER(xzr, x2) /* Reset the fault handler */ + mov x0, #0 /* Success */ + ret /* Return */ +END(suword16) + +/* + * int suword32(volatile void *, int) + */ +ENTRY(suword32) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x2) /* And set it */ + str w1, [x0] /* Try storing the data */ + SET_FAULT_HANDLER(xzr, x2) /* Reset the fault handler */ + mov x0, #0 /* Success */ + ret /* Return */ +END(suword32) + +/* + * int suword(volatile void *, long) + */ +ENTRY(suword) +EENTRY(suword64) + adr x6, fsu_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x2) /* And set it */ + str x1, [x0] /* Try storing the data */ + SET_FAULT_HANDLER(xzr, x2) /* Reset the fault handler */ + mov x0, #0 /* Success */ + ret /* Return */ +EEND(suword64) +END(suword) + +/* + * fuswintr and suswintr are just like fusword and susword except that if + * the page is not in memory or would cause a trap, then we return an error. + * The important thing is to prevent sleep() and switch(). + */ + +/* + * Special handler so the trap code knows not to sleep. + */ +ENTRY(fsu_intr_fault) + SET_FAULT_HANDLER(xzr, x1) /* Reset the handler function */ + mov x0, #-1 + ret +END(fsu_fault) + +/* + * int fuswintr(void *) + */ +ENTRY(fuswintr) + adr x6, fsu_intr_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x1) /* And set it */ + ldr w0, [x0] /* Try loading the data */ + SET_FAULT_HANDLER(xzr, x1) /* Reset the fault handler */ + ret /* Return */ +END(fuswintr) + +/* + * int suswintr(void *base, int word) + */ +ENTRY(suswintr) + adr x6, fsu_intr_fault /* Load the fault handler */ + SET_FAULT_HANDLER(x6, x2) /* And set it */ + str w1, [x0] /* Try storing the data */ + SET_FAULT_HANDLER(xzr, x2) /* Reset the fault handler */ + mov x0, #0 /* Success */ + ret /* Return */ +END(suswintr) + +ENTRY(setjmp) + /* Store the stack pointer */ + mov x8, sp + str x8, [x0] + + /* Store the general purpose registers and lr */ + stp x19, x20, [x0], #16 + stp x21, x22, [x0], #16 + stp x23, x24, [x0], #16 + stp x25, x26, [x0], #16 + stp x27, x28, [x0], #16 + stp x29, lr, [x0], #16 + + /* Return value */ + mov x0, #0 + ret +END(setjmp) + +ENTRY(longjmp) + /* Restore the stack pointer */ + ldr x8, [x0], #8 + mov sp, x8 + + /* Restore the general purpose registers and lr */ + ldp x19, x20, [x0], #16 + ldp x21, x22, [x0], #16 + ldp x23, x24, [x0], #16 + ldp x25, x26, [x0], #16 + ldp x27, x28, [x0], #16 + ldp x29, lr, [x0], #16 + + /* Load the return value */ + mov x0, x1 + ret +END(longjmp) Property changes on: user/ngie/more-tests/sys/arm64/arm64/support.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/swtch.S =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/swtch.S (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/swtch.S (revision 281504) @@ -0,0 +1,255 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under sponsorship from + * the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include "assym.s" + +#include + +__FBSDID("$FreeBSD$"); + +/* + * void cpu_throw(struct thread *old, struct thread *new) + */ +ENTRY(cpu_throw) +#ifdef SMP +#error cpu_throw needs to be ported to support SMP +#endif + +#ifdef VFP + /* Backup the new thread pointer around a call to C code */ + mov x19, x1 + bl vfp_discard + mov x1, x19 +#endif + + /* Store the new curthread */ + str x1, [x18, #PC_CURTHREAD] + /* And the new pcb */ + ldr x4, [x1, #TD_PCB] + str x4, [x18, #PC_CURPCB] + + /* + * TODO: We may need to flush the cache here. + */ + + /* Switch to the new pmap */ + ldr x5, [x4, #PCB_L1ADDR] + msr ttbr0_el1, x5 + isb + + /* Invalidate the TLB */ + dsb sy + tlbi vmalle1is + dsb sy + isb + + /* Restore the registers */ + ldp x5, x6, [x4, #PCB_SP] + mov sp, x5 + msr tpidr_el0, x6 + ldp x8, x9, [x4, #PCB_REGS + 8 * 8] + ldp x10, x11, [x4, #PCB_REGS + 10 * 8] + ldp x12, x13, [x4, #PCB_REGS + 12 * 8] + ldp x14, x15, [x4, #PCB_REGS + 14 * 8] + ldp x16, x17, [x4, #PCB_REGS + 16 * 8] + ldr x19, [x4, #PCB_REGS + 19 * 8] + ldp x20, x21, [x4, #PCB_REGS + 20 * 8] + ldp x22, x23, [x4, #PCB_REGS + 22 * 8] + ldp x24, x25, [x4, #PCB_REGS + 24 * 8] + ldp x26, x27, [x4, #PCB_REGS + 26 * 8] + ldp x28, x29, [x4, #PCB_REGS + 28 * 8] + ldr x30, [x4, #PCB_REGS + 30 * 8] + + ret +END(cpu_throw) + +/* + * void cpu_switch(struct thread *old, struct thread *new, struct mtx *mtx) + * + * x0 = old + * x1 = new + * x2 = mtx + * x3 to x7, x16 and x17 are caller saved + */ +ENTRY(cpu_switch) +#ifdef SMP +#error cpu_switch needs to be ported to support SMP +#endif + + /* Store the new curthread */ + str x1, [x18, #PC_CURTHREAD] + /* And the new pcb */ + ldr x4, [x1, #TD_PCB] + str x4, [x18, #PC_CURPCB] + + /* + * Save the old context. + */ + ldr x4, [x0, #TD_PCB] + + /* Store the callee-saved registers */ + stp x8, x9, [x4, #PCB_REGS + 8 * 8] + stp x10, x11, [x4, #PCB_REGS + 10 * 8] + stp x12, x13, [x4, #PCB_REGS + 12 * 8] + stp x14, x15, [x4, #PCB_REGS + 14 * 8] + stp x16, x17, [x4, #PCB_REGS + 16 * 8] + stp x18, x19, [x4, #PCB_REGS + 18 * 8] + stp x20, x21, [x4, #PCB_REGS + 20 * 8] + stp x22, x23, [x4, #PCB_REGS + 22 * 8] + stp x24, x25, [x4, #PCB_REGS + 24 * 8] + stp x26, x27, [x4, #PCB_REGS + 26 * 8] + stp x28, x29, [x4, #PCB_REGS + 28 * 8] + str x30, [x4, #PCB_REGS + 30 * 8] + /* And the old stack pointer */ + mov x5, sp + mrs x6, tpidr_el0 + stp x5, x6, [x4, #PCB_SP] + +#ifdef VFP + mov x19, x0 + mov x20, x1 + mov x21, x2 + bl vfp_save_state + mov x2, x21 + mov x1, x20 + mov x0, x19 +#endif + + /* + * Restore the saved context. + */ + ldr x4, [x1, #TD_PCB] + + /* + * TODO: We may need to flush the cache here if switching + * to a user process. + */ + + /* Switch to the new pmap */ + ldr x5, [x4, #PCB_L1ADDR] + msr ttbr0_el1, x5 + isb + + /* Invalidate the TLB */ + dsb sy + tlbi vmalle1is + dsb sy + isb + + /* Release the old thread */ + str x2, [x0, #TD_LOCK] +#if defined(SCHED_ULE) && defined(SMP) +#error We may need to wait for the lock here +#endif + + /* Restore the registers */ + ldp x5, x6, [x4, #PCB_SP] + mov sp, x5 + msr tpidr_el0, x6 + ldp x8, x9, [x4, #PCB_REGS + 8 * 8] + ldp x10, x11, [x4, #PCB_REGS + 10 * 8] + ldp x12, x13, [x4, #PCB_REGS + 12 * 8] + ldp x14, x15, [x4, #PCB_REGS + 14 * 8] + ldp x16, x17, [x4, #PCB_REGS + 16 * 8] + ldr x19, [x4, #PCB_REGS + 19 * 8] + ldp x20, x21, [x4, #PCB_REGS + 20 * 8] + ldp x22, x23, [x4, #PCB_REGS + 22 * 8] + ldp x24, x25, [x4, #PCB_REGS + 24 * 8] + ldp x26, x27, [x4, #PCB_REGS + 26 * 8] + ldp x28, x29, [x4, #PCB_REGS + 28 * 8] + ldr x30, [x4, #PCB_REGS + 30 * 8] + + str xzr, [x4, #PCB_REGS + 18 * 8] + ret +.Lcpu_switch_panic_str: + .asciz "cpu_switch: %p\0" +END(cpu_switch) + +ENTRY(fork_trampoline) + mov x0, x8 + mov x1, x9 + mov x2, sp + mov fp, #0 /* Stack traceback stops here. */ + bl _C_LABEL(fork_exit) + + /* Restore sp and lr */ + ldp x0, x1, [sp] + msr sp_el0, x0 + mov lr, x1 + + /* Restore the registers other than x0 and x1 */ + ldp x2, x3, [sp, #TF_X + 2 * 8] + ldp x4, x5, [sp, #TF_X + 4 * 8] + ldp x6, x7, [sp, #TF_X + 6 * 8] + ldp x8, x9, [sp, #TF_X + 8 * 8] + ldp x10, x11, [sp, #TF_X + 10 * 8] + ldp x12, x13, [sp, #TF_X + 12 * 8] + ldp x14, x15, [sp, #TF_X + 14 * 8] + ldp x16, x17, [sp, #TF_X + 16 * 8] + ldr x19, [sp, #TF_X + 19 * 8] + ldp x20, x21, [sp, #TF_X + 20 * 8] + ldp x22, x23, [sp, #TF_X + 22 * 8] + ldp x24, x25, [sp, #TF_X + 24 * 8] + ldp x26, x27, [sp, #TF_X + 26 * 8] + ldp x28, x29, [sp, #TF_X + 28 * 8] + /* Skip x30 as it was restored above as lr */ + + /* + * Disable interrupts to avoid + * overwriting spsr_el1 by an IRQ exception. + */ + msr daifset, #2 + + /* Restore elr and spsr */ + ldp x0, x1, [sp, #16] + msr elr_el1, x0 + msr spsr_el1, x1 + + /* Finally x0 and x1 */ + ldp x0, x1, [sp, #TF_X + 0 * 8] + ldr x18, [sp, #TF_X + 18 * 8] + + /* + * No need for interrupts reenabling since PSR + * will be set to the desired value anyway. + */ + eret + +END(fork_trampoline) + +ENTRY(savectx) + adr x0, .Lsavectx_panic_str + bl panic + ret +.Lsavectx_panic_str: + .asciz "savectx" +END(savectx) + Property changes on: user/ngie/more-tests/sys/arm64/arm64/swtch.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/sys_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/sys_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/sys_machdep.c (revision 281504) @@ -0,0 +1,47 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include + +#include + + +int +sysarch(struct thread *td, struct sysarch_args *uap) +{ + + return (ENOTSUP); +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/sys_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/trap.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/trap.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/trap.c (revision 281504) @@ -0,0 +1,311 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#ifdef KDB +#include +#endif + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +#ifdef VFP +#include +#endif + +#ifdef KDB +#include +#endif + +#ifdef DDB +#include +#endif + +extern uintptr_t fsu_intr_fault; + +/* Called from exception.S */ +void do_el1h_sync(struct trapframe *); +void do_el0_sync(struct trapframe *); +void do_el0_error(struct trapframe *); + +static __inline void +call_trapsignal(struct thread *td, int sig, u_long code) +{ + ksiginfo_t ksi; + + ksiginfo_init_trap(&ksi); + ksi.ksi_signo = sig; + ksi.ksi_code = (int)code; + trapsignal(td, &ksi); +} + +int +cpu_fetch_syscall_args(struct thread *td, struct syscall_args *sa) +{ + struct proc *p; + register_t *ap; + int nap; + + nap = 8; + p = td->td_proc; + ap = td->td_frame->tf_x; + + sa->code = td->td_frame->tf_x[8]; + + if (sa->code == SYS_syscall || sa->code == SYS___syscall) { + sa->code = *ap++; + nap--; + } + + if (p->p_sysent->sv_mask) + sa->code &= p->p_sysent->sv_mask; + if (sa->code >= p->p_sysent->sv_size) + sa->callp = &p->p_sysent->sv_table[0]; + else + sa->callp = &p->p_sysent->sv_table[sa->code]; + + sa->narg = sa->callp->sy_narg; + memcpy(sa->args, ap, nap * sizeof(register_t)); + if (sa->narg > nap) + panic("TODO: Could we have more then 8 args?"); + + td->td_retval[0] = 0; + td->td_retval[1] = 0; + + return (0); +} + +#include "../../kern/subr_syscall.c" + +static void +svc_handler(struct trapframe *frame) +{ + struct syscall_args sa; + struct thread *td; + int error; + + td = curthread; + td->td_frame = frame; + + error = syscallenter(td, &sa); + syscallret(td, error, &sa); +} + +static void +data_abort(struct trapframe *frame, uint64_t esr, int lower) +{ + struct vm_map *map; + struct thread *td; + struct proc *p; + struct pcb *pcb; + vm_prot_t ftype; + vm_offset_t va; + uint64_t far; + int error, sig; + + td = curthread; + pcb = td->td_pcb; + + /* + * Special case for fuswintr and suswintr. These can't sleep so + * handle them early on in the trap handler. + */ + if (__predict_false(pcb->pcb_onfault == fsu_intr_fault)) { + frame->tf_elr = pcb->pcb_onfault; + return; + } + + far = READ_SPECIALREG(far_el1); + p = td->td_proc; + + if (lower) + map = &td->td_proc->p_vmspace->vm_map; + else { + /* The top bit tells us which range to use */ + if ((far >> 63) == 1) + map = kernel_map; + else + map = &td->td_proc->p_vmspace->vm_map; + } + + va = trunc_page(far); + ftype = ((esr >> 6) & 1) ? VM_PROT_READ | VM_PROT_WRITE : VM_PROT_READ; + + if (map != kernel_map) { + /* + * Keep swapout from messing with us during this + * critical time. + */ + PROC_LOCK(p); + ++p->p_lock; + PROC_UNLOCK(p); + + /* Fault in the user page: */ + error = vm_fault(map, va, ftype, VM_FAULT_NORMAL); + + PROC_LOCK(p); + --p->p_lock; + PROC_UNLOCK(p); + } else { + /* + * Don't have to worry about process locking or stacks in the + * kernel. + */ + error = vm_fault(map, va, ftype, VM_FAULT_NORMAL); + } + + if (error != 0) { + if (lower) { + if (error == ENOMEM) + sig = SIGKILL; + else + sig = SIGSEGV; + call_trapsignal(td, sig, 0); + } else { + if (td->td_intr_nesting_level == 0 && + pcb->pcb_onfault != 0) { + frame->tf_x[0] = error; + frame->tf_elr = pcb->pcb_onfault; + return; + } + panic("vm_fault failed: %lx", frame->tf_elr); + } + } + + if (lower) + userret(td, frame); +} + +void +do_el1h_sync(struct trapframe *frame) +{ + uint32_t exception; + uint64_t esr; + + /* Read the esr register to get the exception details */ + esr = READ_SPECIALREG(esr_el1); + exception = ESR_ELx_EXCEPTION(esr); + + /* + * Sanity check we are in an exception er can handle. The IL bit + * is used to indicate the instruction length, except in a few + * exceptions described in the ARMv8 ARM. + * + * It is unclear in some cases if the bit is implementation defined. + * The Foundation Model and QEMU disagree on if the IL bit should + * be set when we are in a data fault from the same EL and the ISV + * bit (bit 24) is also set. + */ + KASSERT((esr & ESR_ELx_IL) == ESR_ELx_IL || + (exception == EXCP_DATA_ABORT && ((esr & ISS_DATA_ISV) == 0)), + ("Invalid instruction length in exception")); + + switch(exception) { + case EXCP_FP_SIMD: + case EXCP_TRAP_FP: + panic("VFP exception in the kernel"); + case EXCP_DATA_ABORT: + data_abort(frame, esr, 0); + break; + case EXCP_BRK: + case EXCP_WATCHPT_EL1: + case EXCP_SOFTSTP_EL1: +#ifdef KDB + kdb_trap(exception, 0, frame); +#else + panic("No debugger in kernel.\n"); +#endif + break; + default: + panic("Unknown kernel exception %x esr_el1 %lx\n", exception, + esr); + } +} + +void +do_el0_sync(struct trapframe *frame) +{ + uint32_t exception; + uint64_t esr; + + /* Check we have a sane environment when entering from userland */ + KASSERT((uintptr_t)get_pcpu() >= VM_MIN_KERNEL_ADDRESS, + ("Invalid pcpu address from userland: %p (tpidr %lx)", + get_pcpu(), READ_SPECIALREG(tpidr_el1))); + + esr = READ_SPECIALREG(esr_el1); + exception = ESR_ELx_EXCEPTION(esr); + + switch(exception) { + case EXCP_FP_SIMD: + case EXCP_TRAP_FP: +#ifdef VFP + vfp_restore_state(); +#else + panic("VFP exception in userland"); +#endif + break; + case EXCP_SVC: + svc_handler(frame); + break; + case EXCP_INSN_ABORT_L: + case EXCP_DATA_ABORT_L: + data_abort(frame, esr, 1); + break; + default: + panic("Unknown userland exception %x esr_el1 %lx\n", exception, + esr); + } +} + +void +do_el0_error(struct trapframe *frame) +{ + + panic("do_el0_error"); +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/trap.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/uio_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/uio_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/uio_machdep.c (revision 281504) @@ -0,0 +1,134 @@ +/*- + * Copyright (c) 2004 Alan L. Cox + * Copyright (c) 1982, 1986, 1991, 1993 + * The Regents of the University of California. All rights reserved. + * (c) UNIX System Laboratories, Inc. + * All or some portions of this file are derived from material licensed + * to the University of California by American Telephone and Telegraph + * Co. or Unix System Laboratories, Inc. and are reproduced herein with + * the permission of UNIX System Laboratories, Inc. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * @(#)kern_subr.c 8.3 (Berkeley) 1/21/94 + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include + +/* + * Implement uiomove(9) from physical memory using the direct map to + * avoid the creation and destruction of ephemeral mappings. + */ +int +uiomove_fromphys(vm_page_t ma[], vm_offset_t offset, int n, struct uio *uio) +{ + struct thread *td = curthread; + struct iovec *iov; + void *cp; + vm_offset_t page_offset, vaddr; + size_t cnt; + int error = 0; + int save = 0; + boolean_t mapped; + + KASSERT(uio->uio_rw == UIO_READ || uio->uio_rw == UIO_WRITE, + ("uiomove_fromphys: mode")); + KASSERT(uio->uio_segflg != UIO_USERSPACE || uio->uio_td == curthread, + ("uiomove_fromphys proc")); + save = td->td_pflags & TDP_DEADLKTREAT; + td->td_pflags |= TDP_DEADLKTREAT; + mapped = FALSE; + while (n > 0 && uio->uio_resid) { + iov = uio->uio_iov; + cnt = iov->iov_len; + if (cnt == 0) { + uio->uio_iov++; + uio->uio_iovcnt--; + continue; + } + if (cnt > n) + cnt = n; + page_offset = offset & PAGE_MASK; + cnt = min(cnt, PAGE_SIZE - page_offset); + if (uio->uio_segflg != UIO_NOCOPY) { + mapped = pmap_map_io_transient( + &ma[offset >> PAGE_SHIFT], &vaddr, 1, TRUE); + cp = (char *)vaddr + page_offset; + } + switch (uio->uio_segflg) { + case UIO_USERSPACE: + maybe_yield(); + if (uio->uio_rw == UIO_READ) + error = copyout(cp, iov->iov_base, cnt); + else + error = copyin(iov->iov_base, cp, cnt); + if (error) + goto out; + break; + case UIO_SYSSPACE: + if (uio->uio_rw == UIO_READ) + bcopy(cp, iov->iov_base, cnt); + else + bcopy(iov->iov_base, cp, cnt); + break; + case UIO_NOCOPY: + break; + } + if (__predict_false(mapped)) { + pmap_unmap_io_transient(&ma[offset >> PAGE_SHIFT], + &vaddr, 1, TRUE); + mapped = FALSE; + } + iov->iov_base = (char *)iov->iov_base + cnt; + iov->iov_len -= cnt; + uio->uio_resid -= cnt; + uio->uio_offset += cnt; + offset += cnt; + n -= cnt; + } +out: + if (__predict_false(mapped)) { + panic("TODO 3"); + pmap_unmap_io_transient(&ma[offset >> PAGE_SHIFT], &vaddr, 1, + TRUE); + } + if (save == 0) + td->td_pflags &= ~TDP_DEADLKTREAT; + return (error); +} Property changes on: user/ngie/more-tests/sys/arm64/arm64/uio_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/vfp.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/vfp.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/vfp.c (revision 281504) @@ -0,0 +1,194 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#ifdef VFP +#include +#include +#include +#include +#include + +#include +#include +#include + +/* Sanity check we can store all the VFP registers */ +CTASSERT(sizeof(((struct pcb *)0)->pcb_vfp) == 16 * 32); + +static void +vfp_enable(void) +{ + uint32_t cpacr; + + cpacr = READ_SPECIALREG(cpacr_el1); + cpacr = (cpacr & ~CPACR_FPEN_MASK) | CPACR_FPEN_TRAP_NONE; + WRITE_SPECIALREG(cpacr_el1, cpacr); + isb(); +} + +static void +vfp_disable(void) +{ + uint32_t cpacr; + + cpacr = READ_SPECIALREG(cpacr_el1); + cpacr = (cpacr & ~CPACR_FPEN_MASK) | CPACR_FPEN_TRAP_ALL1; + WRITE_SPECIALREG(cpacr_el1, cpacr); + isb(); +} + +/* + * Called when the thread is dying. If the thread was the last to use the + * VFP unit mark it as unused to tell the kernel the fp state is unowned. + * Ensure the VFP unit is off so we get an exception on the next access. + */ +void +vfp_discard(struct thread *td) +{ + + if (PCPU_GET(fpcurthread) == td) + PCPU_SET(fpcurthread, NULL); + + vfp_disable(); +} + +void +vfp_save_state(struct thread *td) +{ + __int128_t *vfp_state; + uint64_t fpcr, fpsr; + uint32_t cpacr; + + /* + * Only store the registers if the VFP is enabled, + * i.e. return if we are trapping on FP access. + */ + cpacr = READ_SPECIALREG(cpacr_el1); + if ((cpacr & CPACR_FPEN_MASK) != CPACR_FPEN_TRAP_NONE) + return; + + vfp_state = td->td_pcb->pcb_vfp; + __asm __volatile( + "mrs %0, fpcr \n" + "mrs %1, fpsr \n" + "stp q0, q1, [%2, #16 * 0]\n" + "stp q2, q3, [%2, #16 * 2]\n" + "stp q4, q5, [%2, #16 * 4]\n" + "stp q6, q7, [%2, #16 * 6]\n" + "stp q8, q9, [%2, #16 * 8]\n" + "stp q10, q11, [%2, #16 * 10]\n" + "stp q12, q13, [%2, #16 * 12]\n" + "stp q14, q15, [%2, #16 * 14]\n" + "stp q16, q17, [%2, #16 * 16]\n" + "stp q18, q19, [%2, #16 * 18]\n" + "stp q20, q21, [%2, #16 * 20]\n" + "stp q22, q23, [%2, #16 * 22]\n" + "stp q24, q25, [%2, #16 * 24]\n" + "stp q26, q27, [%2, #16 * 26]\n" + "stp q28, q29, [%2, #16 * 28]\n" + "stp q30, q31, [%2, #16 * 30]\n" + : "=&r"(fpcr), "=&r"(fpsr) : "r"(vfp_state)); + + td->td_pcb->pcb_fpcr = fpcr; + td->td_pcb->pcb_fpsr = fpsr; + + dsb(); + vfp_disable(); +} + +void +vfp_restore_state(void) +{ + __int128_t *vfp_state; + uint64_t fpcr, fpsr; + struct pcb *curpcb; + u_int cpu; + + critical_enter(); + + cpu = PCPU_GET(cpuid); + curpcb = curthread->td_pcb; + curpcb->pcb_fpflags |= PCB_FP_STARTED; + + vfp_enable(); + + if (PCPU_GET(fpcurthread) != curthread && cpu != curpcb->pcb_vfpcpu) { + + vfp_state = curthread->td_pcb->pcb_vfp; + fpcr = curthread->td_pcb->pcb_fpcr; + fpsr = curthread->td_pcb->pcb_fpsr; + + __asm __volatile( + "ldp q0, q1, [%2, #16 * 0]\n" + "ldp q2, q3, [%2, #16 * 2]\n" + "ldp q4, q5, [%2, #16 * 4]\n" + "ldp q6, q7, [%2, #16 * 6]\n" + "ldp q8, q9, [%2, #16 * 8]\n" + "ldp q10, q11, [%2, #16 * 10]\n" + "ldp q12, q13, [%2, #16 * 12]\n" + "ldp q14, q15, [%2, #16 * 14]\n" + "ldp q16, q17, [%2, #16 * 16]\n" + "ldp q18, q19, [%2, #16 * 18]\n" + "ldp q20, q21, [%2, #16 * 20]\n" + "ldp q22, q23, [%2, #16 * 22]\n" + "ldp q24, q25, [%2, #16 * 24]\n" + "ldp q26, q27, [%2, #16 * 26]\n" + "ldp q28, q29, [%2, #16 * 28]\n" + "ldp q30, q31, [%2, #16 * 30]\n" + "msr fpcr, %0 \n" + "msr fpsr, %1 \n" + : : "r"(fpcr), "r"(fpsr), "r"(vfp_state)); + + PCPU_SET(fpcurthread, curthread); + curpcb->pcb_vfpcpu = cpu; + } + + critical_exit(); +} + +void +vfp_init(void) +{ + uint64_t pfr; + + /* Check if there is a vfp unit present */ + pfr = READ_SPECIALREG(id_aa64pfr0_el1); + if ((pfr & ID_AA64PFR0_FP_MASK) == ID_AA64PFR0_FP_NONE) + return; + + /* Disable to be enabled when it's used */ + vfp_disable(); +} + +SYSINIT(vfp, SI_SUB_CPU, SI_ORDER_ANY, vfp_init, NULL); + +#endif Property changes on: user/ngie/more-tests/sys/arm64/arm64/vfp.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/arm64/vm_machdep.c =================================================================== --- user/ngie/more-tests/sys/arm64/arm64/vm_machdep.c (nonexistent) +++ user/ngie/more-tests/sys/arm64/arm64/vm_machdep.c (revision 281504) @@ -0,0 +1,248 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include + +/* + * Finish a fork operation, with process p2 nearly set up. + * Copy and update the pcb, set up the stack so that the child + * ready to run and return to user mode. + */ +void +cpu_fork(struct thread *td1, struct proc *p2, struct thread *td2, int flags) +{ + struct pcb *pcb2; + struct trapframe *tf; + + if ((flags & RFPROC) == 0) + return; + + pcb2 = (struct pcb *)(td2->td_kstack + + td2->td_kstack_pages * PAGE_SIZE) - 1; + + td2->td_pcb = pcb2; + bcopy(td1->td_pcb, pcb2, sizeof(*pcb2)); + + td2->td_pcb->pcb_l1addr = + vtophys(vmspace_pmap(td2->td_proc->p_vmspace)->pm_l1); + + tf = (struct trapframe *)STACKALIGN((struct trapframe *)pcb2 - 1); + bcopy(td1->td_frame, tf, sizeof(*tf)); + tf->tf_x[0] = 0; + tf->tf_x[1] = 0; + tf->tf_spsr = 0; + + td2->td_frame = tf; + + /* Set the return value registers for fork() */ + td2->td_pcb->pcb_x[8] = (uintptr_t)fork_return; + td2->td_pcb->pcb_x[9] = (uintptr_t)td2; + td2->td_pcb->pcb_x[PCB_LR] = (uintptr_t)fork_trampoline; + td2->td_pcb->pcb_sp = (uintptr_t)td2->td_frame; + td2->td_pcb->pcb_vfpcpu = UINT_MAX; + + /* Setup to release spin count in fork_exit(). */ + td2->td_md.md_spinlock_count = 1; + td2->td_md.md_saved_daif = 0; +} + +void +cpu_reset(void) +{ + + printf("cpu_reset"); + while(1) + __asm volatile("wfi" ::: "memory"); +} + +void +cpu_thread_swapin(struct thread *td) +{ +} + +void +cpu_thread_swapout(struct thread *td) +{ +} + +void +cpu_set_syscall_retval(struct thread *td, int error) +{ + struct trapframe *frame; + + frame = td->td_frame; + + switch (error) { + case 0: + frame->tf_x[0] = td->td_retval[0]; + frame->tf_x[1] = td->td_retval[1]; + frame->tf_spsr &= ~PSR_C; /* carry bit */ + break; + case ERESTART: + frame->tf_elr -= 4; + break; + case EJUSTRETURN: + break; + default: + frame->tf_spsr |= PSR_C; /* carry bit */ + frame->tf_x[0] = error; + break; + } +} + +/* + * Initialize machine state (pcb and trap frame) for a new thread about to + * upcall. Put enough state in the new thread's PCB to get it to go back + * userret(), where we can intercept it again to set the return (upcall) + * Address and stack, along with those from upcals that are from other sources + * such as those generated in thread_userret() itself. + */ +void +cpu_set_upcall(struct thread *td, struct thread *td0) +{ + bcopy(td0->td_frame, td->td_frame, sizeof(struct trapframe)); + bcopy(td0->td_pcb, td->td_pcb, sizeof(struct pcb)); + + td->td_pcb->pcb_x[8] = (uintptr_t)fork_return; + td->td_pcb->pcb_x[9] = (uintptr_t)td; + td->td_pcb->pcb_x[PCB_LR] = (uintptr_t)fork_trampoline; + td->td_pcb->pcb_sp = (uintptr_t)td->td_frame; + td->td_pcb->pcb_vfpcpu = UINT_MAX; + + /* Setup to release spin count in fork_exit(). */ + td->td_md.md_spinlock_count = 1; + td->td_md.md_saved_daif = 0; +} + +/* + * Set that machine state for performing an upcall that has to + * be done in thread_userret() so that those upcalls generated + * in thread_userret() itself can be done as well. + */ +void +cpu_set_upcall_kse(struct thread *td, void (*entry)(void *), void *arg, + stack_t *stack) +{ + + panic("cpu_set_upcall_kse"); +} + +int +cpu_set_user_tls(struct thread *td, void *tls_base) +{ + + panic("cpu_set_user_tls"); +} + +void +cpu_thread_exit(struct thread *td) +{ +} + +void +cpu_thread_alloc(struct thread *td) +{ + + td->td_pcb = (struct pcb *)(td->td_kstack + + td->td_kstack_pages * PAGE_SIZE) - 1; + td->td_frame = (struct trapframe *)STACKALIGN( + td->td_pcb - 1); +} + +void +cpu_thread_free(struct thread *td) +{ +} + +void +cpu_thread_clean(struct thread *td) +{ +} + +/* + * Intercept the return address from a freshly forked process that has NOT + * been scheduled yet. + * + * This is needed to make kernel threads stay in kernel mode. + */ +void +cpu_set_fork_handler(struct thread *td, void (*func)(void *), void *arg) +{ + + td->td_pcb->pcb_x[8] = (uintptr_t)func; + td->td_pcb->pcb_x[9] = (uintptr_t)arg; + td->td_pcb->pcb_x[PCB_LR] = (uintptr_t)fork_trampoline; + td->td_pcb->pcb_sp = (uintptr_t)td->td_frame; + td->td_pcb->pcb_vfpcpu = UINT_MAX; +} + +void +cpu_exit(struct thread *td) +{ +} + +void +swi_vm(void *v) +{ + + /* Nothing to do here - busdma bounce buffers are not implemented. */ +} + +void * +uma_small_alloc(uma_zone_t zone, vm_size_t bytes, u_int8_t *flags, int wait) +{ + + panic("uma_small_alloc"); +} + +void +uma_small_free(void *mem, vm_size_t size, u_int8_t flags) +{ + + panic("uma_small_free"); +} + Property changes on: user/ngie/more-tests/sys/arm64/arm64/vm_machdep.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/conf/GENERIC =================================================================== --- user/ngie/more-tests/sys/arm64/conf/GENERIC (nonexistent) +++ user/ngie/more-tests/sys/arm64/conf/GENERIC (revision 281504) @@ -0,0 +1,94 @@ +# +# GENERIC -- Generic kernel configuration file for FreeBSD/arm64 +# +# For more information on this file, please read the config(5) manual page, +# and/or the handbook section on Kernel Configuration Files: +# +# http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html +# +# The handbook is also available locally in /usr/share/doc/handbook +# if you've installed the doc distribution, otherwise always see the +# FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the +# latest information. +# +# An exhaustive list of options and more detailed explanations of the +# device lines is also present in the ../../conf/NOTES and NOTES files. +# If you are in doubt as to the purpose or necessity of a line, check first +# in NOTES. +# +# $FreeBSD$ + +cpu ARM64 +ident GENERIC + +makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols +makeoptions NO_MODULES=1 # We don't yet support modules on arm64 + +options SCHED_ULE # ULE scheduler +options PREEMPTION # Enable kernel thread preemption +options INET # InterNETworking +options INET6 # IPv6 communications protocols +options TCP_OFFLOAD # TCP offload +options SCTP # Stream Control Transmission Protocol +options FFS # Berkeley Fast Filesystem +options SOFTUPDATES # Enable FFS soft updates support +options UFS_ACL # Support for access control lists +options UFS_DIRHASH # Improve performance on big directories +options UFS_GJOURNAL # Enable gjournal-based UFS journaling +options QUOTA # Enable disk quotas for UFS +options MD_ROOT # MD is a potential root device +options NFSCL # New Network Filesystem Client +options NFSD # New Network Filesystem Server +options NFSLOCKD # Network Lock Manager +options NFS_ROOT # NFS usable as /, requires NFSCL +options MSDOSFS # MSDOS Filesystem +options CD9660 # ISO 9660 Filesystem +options PROCFS # Process filesystem (requires PSEUDOFS) +options PSEUDOFS # Pseudo-filesystem framework +options GEOM_PART_GPT # GUID Partition Tables. +options GEOM_RAID # Soft RAID functionality. +options GEOM_LABEL # Provides labelization +options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI +options KTRACE # ktrace(1) support +options STACK # stack(9) support +options SYSVSHM # SYSV-style shared memory +options SYSVMSG # SYSV-style message queues +options SYSVSEM # SYSV-style semaphores +options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions +options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. +options KBD_INSTALL_CDEV # install a CDEV entry in /dev +options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) +options AUDIT # Security event auditing +options CAPABILITY_MODE # Capsicum capability mode +options CAPABILITIES # Capsicum capabilities +options MAC # TrustedBSD MAC Framework +options KDTRACE_FRAME # Ensure frames are compiled in +options KDTRACE_HOOKS # Kernel DTrace hooks +options VFP # Floating-point support + +device virtio +device virtio_mmio +device virtio_blk +device vtnet + +# Serial (COM) ports +device uart # Generic UART driver +device pl011 + +# Pseudo devices. +device loop # Network loopback +device random # Entropy device +device ether # Ethernet support +device vlan # 802.1Q VLAN support +device tun # Packet tunnel. +device md # Memory "disks" +device gif # IPv6 and IPv4 tunneling +device firmware # firmware assist module +device psci # Support for ARM PSCI + +# The `bpf' device enables the Berkeley Packet Filter. +# Be aware of the administrative consequences of enabling this! +# Note that 'bpf' is required for DHCP. +device bpf # Berkeley packet filter + +options FDT Property changes on: user/ngie/more-tests/sys/arm64/conf/GENERIC ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/conf/DEFAULTS =================================================================== --- user/ngie/more-tests/sys/arm64/conf/DEFAULTS (nonexistent) +++ user/ngie/more-tests/sys/arm64/conf/DEFAULTS (revision 281504) @@ -0,0 +1,14 @@ +# +# DEFAULTS -- Default kernel configuration file for FreeBSD/arm64 +# +# $FreeBSD$ + +machine arm64 + +# Pseudo devices. +device mem # Memory and kernel memory devices + +# Default partitioning schemes +options GEOM_PART_BSD +options GEOM_PART_MBR + Property changes on: user/ngie/more-tests/sys/arm64/conf/DEFAULTS ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/_bus.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/_bus.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/_bus.h (revision 281504) @@ -0,0 +1,46 @@ +/*- + * Copyright (c) 2005 M. Warner Losh. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions, and the following disclaimer, + * without modification, immediately at the beginning of the file. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in + * the documentation and/or other materials provided with the + * distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE FOR + * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE__BUS_H_ +#define _MACHINE__BUS_H_ + +/* + * Addresses (in bus space). + */ +typedef u_long bus_addr_t; +typedef u_long bus_size_t; + +/* + * Access methods for bus space. + */ +typedef u_long bus_space_handle_t; +typedef struct bus_space *bus_space_tag_t; + +#endif /* !_MACHINE__BUS_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/_bus.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/armreg.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/armreg.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/armreg.h (revision 281504) @@ -0,0 +1,194 @@ +/*- + * Copyright (c) 2013, 2014 Andrew Turner + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_ARMREG_H_ +#define _MACHINE_ARMREG_H_ + +#define READ_SPECIALREG(reg) \ +({ uint64_t val; \ + __asm __volatile("mrs %0, " __STRING(reg) : "=&r" (val)); \ + val; \ +}) +#define WRITE_SPECIALREG(reg, val) \ + __asm __volatile("msr " __STRING(reg) ", %0" : : "r"((uint64_t)val)) + +/* CPACR_EL1 */ +#define CPACR_FPEN_MASK (0x3 << 20) +#define CPACR_FPEN_TRAP_ALL1 (0x0 << 20) /* Traps from EL0 and EL1 */ +#define CPACR_FPEN_TRAP_EL0 (0x1 << 20) /* Traps from EL0 */ +#define CPACR_FPEN_TRAP_ALL2 (0x2 << 20) /* Traps from EL0 and EL1 */ +#define CPACR_FPEN_TRAP_NONE (0x3 << 20) /* No traps */ +#define CPACR_TTA (0x1 << 28) + +/* CTR_EL0 - Cache Type Register */ +#define CTR_DLINE_SHIFT 16 +#define CTR_DLINE_MASK (0xf << CTR_DLINE_SHIFT) +#define CTR_DLINE_SIZE(reg) (((reg) & CTR_DLINE_MASK) >> CTR_DLINE_SHIFT) +#define CTR_ILINE_SHIFT 0 +#define CTR_ILINE_MASK (0xf << CTR_ILINE_SHIFT) +#define CTR_ILINE_SIZE(reg) (((reg) & CTR_ILINE_MASK) >> CTR_ILINE_SHIFT) + +/* ESR_ELx */ +#define ESR_ELx_ISS_MASK 0x00ffffff +#define ISS_INSN_FnV (0x01 << 10) +#define ISS_INSN_EA (0x01 << 9) +#define ISS_INSN_S1PTW (0x01 << 7) +#define ISS_INSN_IFSC_MASK (0x1f << 0) +#define ISS_DATA_ISV (0x01 << 24) +#define ISS_DATA_SAS_MASK (0x03 << 22) +#define ISS_DATA_SSE (0x01 << 21) +#define ISS_DATA_SRT_MASK (0x1f << 16) +#define ISS_DATA_SF (0x01 << 15) +#define ISS_DATA_AR (0x01 << 14) +#define ISS_DATA_FnV (0x01 << 10) +#define ISS_DATa_EA (0x01 << 9) +#define ISS_DATa_CM (0x01 << 8) +#define ISS_INSN_S1PTW (0x01 << 7) +#define ISS_DATa_WnR (0x01 << 6) +#define ISS_DATA_DFSC_MASK (0x1f << 0) +#define ESR_ELx_IL (0x01 << 25) +#define ESR_ELx_EC_SHIFT 26 +#define ESR_ELx_EC_MASK (0x3f << 26) +#define ESR_ELx_EXCEPTION(esr) (((esr) & ESR_ELx_EC_MASK) >> ESR_ELx_EC_SHIFT) +#define EXCP_UNKNOWN 0x00 /* Unkwn exception */ +#define EXCP_FP_SIMD 0x07 /* VFP/SIMD trap */ +#define EXCP_ILL_STATE 0x0e /* Illegal execution state */ +#define EXCP_SVC 0x15 /* SVC trap */ +#define EXCP_MSR 0x18 /* MSR/MRS trap */ +#define EXCP_INSN_ABORT_L 0x20 /* Instruction abort, from lower EL */ +#define EXCP_INSN_ABORT 0x21 /* Instruction abort, from same EL */ +#define EXCP_PC_ALIGN 0x22 /* PC alignment fault */ +#define EXCP_DATA_ABORT_L 0x24 /* Data abort, from lower EL */ +#define EXCP_DATA_ABORT 0x25 /* Data abort, from same EL */ +#define EXCP_SP_ALIGN 0x26 /* SP slignment fault */ +#define EXCP_TRAP_FP 0x2c /* Trapped FP exception */ +#define EXCP_SERROR 0x2f /* SError interrupt */ +#define EXCP_SOFTSTP_EL1 0x33 /* Software Step, from same EL */ +#define EXCP_WATCHPT_EL1 0x35 /* Watchpoint, from same EL */ +#define EXCP_BRK 0x3c /* Breakpoint */ + +/* ID_AA64PFR0_EL1 */ +#define ID_AA64PFR0_EL0_MASK (0xf << 0) +#define ID_AA64PFR0_EL1_MASK (0xf << 4) +#define ID_AA64PFR0_EL2_MASK (0xf << 8) +#define ID_AA64PFR0_EL3_MASK (0xf << 12) +#define ID_AA64PFR0_FP_MASK (0xf << 16) +#define ID_AA64PFR0_FP_IMPL (0x0 << 16) /* Floating-point implemented */ +#define ID_AA64PFR0_FP_NONE (0xf << 16) /* Floating-point not implemented */ +#define ID_AA64PFR0_ADV_SIMD_MASK (0xf << 20) +#define ID_AA64PFR0_GIC_MASK (0xf << 24) + +/* MAIR_EL1 - Memory Attribute Indirection Register */ +#define MAIR_ATTR_MASK(idx) (0xff << ((n)* 8)) +#define MAIR_ATTR(attr, idx) ((attr) << ((idx) * 8)) + +/* SCTLR_EL1 - System Control Register */ +#define SCTLR_RES0 0xc8222400 /* Reserved, write 0 */ +#define SCTLR_RES1 0x30d00800 /* Reserved, write 1 */ + +#define SCTLR_M 0x00000001 +#define SCTLR_A 0x00000002 +#define SCTLR_C 0x00000004 +#define SCTLR_SA 0x00000008 +#define SCTLR_SA0 0x00000010 +#define SCTLR_CP15BEN 0x00000020 +#define SCTLR_THEE 0x00000040 +#define SCTLR_ITD 0x00000080 +#define SCTLR_SED 0x00000100 +#define SCTLR_UMA 0x00000200 +#define SCTLR_I 0x00001000 +#define SCTLR_DZE 0x00004000 +#define SCTLR_UCT 0x00008000 +#define SCTLR_nTWI 0x00010000 +#define SCTLR_nTWE 0x00040000 +#define SCTLR_WXN 0x00080000 +#define SCTLR_EOE 0x01000000 +#define SCTLR_EE 0x02000000 +#define SCTLR_UCI 0x04000000 + +/* SPSR_EL1 */ +/* + * When the exception is taken in AArch64: + * M[4] is 0 for AArch64 mode + * M[3:2] is the exception level + * M[1] is unused + * M[0] is the SP select: + * 0: always SP0 + * 1: current ELs SP + */ +#define PSR_M_EL0t 0x00000000 +#define PSR_M_EL1t 0x00000004 +#define PSR_M_EL1h 0x00000005 +#define PSR_M_EL2t 0x00000008 +#define PSR_M_EL2h 0x00000009 +#define PSR_M_MASK 0x0000001f + +#define PSR_F 0x00000040 +#define PSR_I 0x00000080 +#define PSR_A 0x00000100 +#define PSR_D 0x00000200 +#define PSR_IL 0x00100000 +#define PSR_SS 0x00200000 +#define PSR_V 0x10000000 +#define PSR_C 0x20000000 +#define PSR_Z 0x40000000 +#define PSR_N 0x80000000 + +/* TCR_EL1 - Translation Control Register */ +#define TCR_ASID_16 (1 << 36) + +#define TCR_IPS_SHIFT 32 +#define TCR_IPS_32BIT (0 << TCR_IPS_SHIFT) +#define TCR_IPS_36BIT (1 << TCR_IPS_SHIFT) +#define TCR_IPS_40BIT (2 << TCR_IPS_SHIFT) +#define TCR_IPS_42BIT (3 << TCR_IPS_SHIFT) +#define TCR_IPS_44BIT (4 << TCR_IPS_SHIFT) +#define TCR_IPS_48BIT (5 << TCR_IPS_SHIFT) + +#define TCR_TG1_SHIFT 30 +#define TCR_TG1_16K (1 << TCR_TG1_SHIFT) +#define TCR_TG1_4K (2 << TCR_TG1_SHIFT) +#define TCR_TG1_64K (3 << TCR_TG1_SHIFT) + +#define TCR_T1SZ_SHIFT 16 +#define TCR_T0SZ_SHIFT 0 +#define TCR_TxSZ(x) (((x) << TCR_T1SZ_SHIFT) | ((x) << TCR_T0SZ_SHIFT)) + +/* Saved Program Status Register */ +#define DBG_SPSR_SS (0x1 << 21) + +/* Monitor Debug System Control Register */ +#define DBG_MDSCR_SS (0x1 << 0) +#define DBG_MDSCR_KDE (0x1 << 13) +#define DBG_MDSCR_MDE (0x1 << 15) + +#endif /* !_MACHINE_ARMREG_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/armreg.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/bus.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/bus.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/bus.h (revision 281504) @@ -0,0 +1,469 @@ +/* $NetBSD: bus.h,v 1.11 2003/07/28 17:35:54 thorpej Exp $ */ + +/*- + * Copyright (c) 1996, 1997, 1998, 2001 The NetBSD Foundation, Inc. + * All rights reserved. + * + * This code is derived from software contributed to The NetBSD Foundation + * by Jason R. Thorpe of the Numerical Aerospace Simulation Facility, + * NASA Ames Research Center. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED + * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR + * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS + * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +/*- + * Copyright (c) 1996 Charles M. Hannum. All rights reserved. + * Copyright (c) 1996 Christopher G. Demetriou. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by Christopher G. Demetriou + * for the NetBSD Project. + * 4. The name of the author may not be used to endorse or promote products + * derived from this software without specific prior written permission + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + * From: sys/arm/include/bus.h + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_BUS_H_ +#define _MACHINE_BUS_H_ + +#include + +#define BUS_SPACE_ALIGNED_POINTER(p, t) ALIGNED_POINTER(p, t) + +#define BUS_SPACE_MAXADDR_24BIT 0xFFFFFFUL +#define BUS_SPACE_MAXADDR_32BIT 0xFFFFFFFFUL +#define BUS_SPACE_MAXSIZE_24BIT 0xFFFFFFUL +#define BUS_SPACE_MAXSIZE_32BIT 0xFFFFFFFFUL + +#define BUS_SPACE_MAXADDR 0xFFFFFFFFFFFFFFFFUL +#define BUS_SPACE_MAXSIZE 0xFFFFFFFFFFFFFFFFUL + +#define BUS_SPACE_MAP_CACHEABLE 0x01 +#define BUS_SPACE_MAP_LINEAR 0x02 +#define BUS_SPACE_MAP_PREFETCHABLE 0x04 + +#define BUS_SPACE_UNRESTRICTED (~0) + +#define BUS_SPACE_BARRIER_READ 0x01 +#define BUS_SPACE_BARRIER_WRITE 0x02 + + +struct bus_space { + /* cookie */ + void *bs_cookie; + + /* mapping/unmapping */ + int (*bs_map) (void *, bus_addr_t, bus_size_t, + int, bus_space_handle_t *); + void (*bs_unmap) (void *, bus_space_handle_t, bus_size_t); + int (*bs_subregion) (void *, bus_space_handle_t, + bus_size_t, bus_size_t, bus_space_handle_t *); + + /* allocation/deallocation */ + int (*bs_alloc) (void *, bus_addr_t, bus_addr_t, + bus_size_t, bus_size_t, bus_size_t, int, + bus_addr_t *, bus_space_handle_t *); + void (*bs_free) (void *, bus_space_handle_t, + bus_size_t); + + /* get kernel virtual address */ + /* barrier */ + void (*bs_barrier) (void *, bus_space_handle_t, + bus_size_t, bus_size_t, int); + + /* read single */ + u_int8_t (*bs_r_1) (void *, bus_space_handle_t, bus_size_t); + u_int16_t (*bs_r_2) (void *, bus_space_handle_t, bus_size_t); + u_int32_t (*bs_r_4) (void *, bus_space_handle_t, bus_size_t); + u_int64_t (*bs_r_8) (void *, bus_space_handle_t, bus_size_t); + + /* read multiple */ + void (*bs_rm_1) (void *, bus_space_handle_t, bus_size_t, + u_int8_t *, bus_size_t); + void (*bs_rm_2) (void *, bus_space_handle_t, bus_size_t, + u_int16_t *, bus_size_t); + void (*bs_rm_4) (void *, bus_space_handle_t, + bus_size_t, u_int32_t *, bus_size_t); + void (*bs_rm_8) (void *, bus_space_handle_t, + bus_size_t, u_int64_t *, bus_size_t); + + /* read region */ + void (*bs_rr_1) (void *, bus_space_handle_t, + bus_size_t, u_int8_t *, bus_size_t); + void (*bs_rr_2) (void *, bus_space_handle_t, + bus_size_t, u_int16_t *, bus_size_t); + void (*bs_rr_4) (void *, bus_space_handle_t, + bus_size_t, u_int32_t *, bus_size_t); + void (*bs_rr_8) (void *, bus_space_handle_t, + bus_size_t, u_int64_t *, bus_size_t); + + /* write single */ + void (*bs_w_1) (void *, bus_space_handle_t, + bus_size_t, u_int8_t); + void (*bs_w_2) (void *, bus_space_handle_t, + bus_size_t, u_int16_t); + void (*bs_w_4) (void *, bus_space_handle_t, + bus_size_t, u_int32_t); + void (*bs_w_8) (void *, bus_space_handle_t, + bus_size_t, u_int64_t); + + /* write multiple */ + void (*bs_wm_1) (void *, bus_space_handle_t, + bus_size_t, const u_int8_t *, bus_size_t); + void (*bs_wm_2) (void *, bus_space_handle_t, + bus_size_t, const u_int16_t *, bus_size_t); + void (*bs_wm_4) (void *, bus_space_handle_t, + bus_size_t, const u_int32_t *, bus_size_t); + void (*bs_wm_8) (void *, bus_space_handle_t, + bus_size_t, const u_int64_t *, bus_size_t); + + /* write region */ + void (*bs_wr_1) (void *, bus_space_handle_t, + bus_size_t, const u_int8_t *, bus_size_t); + void (*bs_wr_2) (void *, bus_space_handle_t, + bus_size_t, const u_int16_t *, bus_size_t); + void (*bs_wr_4) (void *, bus_space_handle_t, + bus_size_t, const u_int32_t *, bus_size_t); + void (*bs_wr_8) (void *, bus_space_handle_t, + bus_size_t, const u_int64_t *, bus_size_t); + + /* set multiple */ + void (*bs_sm_1) (void *, bus_space_handle_t, + bus_size_t, u_int8_t, bus_size_t); + void (*bs_sm_2) (void *, bus_space_handle_t, + bus_size_t, u_int16_t, bus_size_t); + void (*bs_sm_4) (void *, bus_space_handle_t, + bus_size_t, u_int32_t, bus_size_t); + void (*bs_sm_8) (void *, bus_space_handle_t, + bus_size_t, u_int64_t, bus_size_t); + + /* set region */ + void (*bs_sr_1) (void *, bus_space_handle_t, + bus_size_t, u_int8_t, bus_size_t); + void (*bs_sr_2) (void *, bus_space_handle_t, + bus_size_t, u_int16_t, bus_size_t); + void (*bs_sr_4) (void *, bus_space_handle_t, + bus_size_t, u_int32_t, bus_size_t); + void (*bs_sr_8) (void *, bus_space_handle_t, + bus_size_t, u_int64_t, bus_size_t); + + /* copy */ + void (*bs_c_1) (void *, bus_space_handle_t, bus_size_t, + bus_space_handle_t, bus_size_t, bus_size_t); + void (*bs_c_2) (void *, bus_space_handle_t, bus_size_t, + bus_space_handle_t, bus_size_t, bus_size_t); + void (*bs_c_4) (void *, bus_space_handle_t, bus_size_t, + bus_space_handle_t, bus_size_t, bus_size_t); + void (*bs_c_8) (void *, bus_space_handle_t, bus_size_t, + bus_space_handle_t, bus_size_t, bus_size_t); + + /* read single stream */ + u_int8_t (*bs_r_1_s) (void *, bus_space_handle_t, bus_size_t); + u_int16_t (*bs_r_2_s) (void *, bus_space_handle_t, bus_size_t); + u_int32_t (*bs_r_4_s) (void *, bus_space_handle_t, bus_size_t); + u_int64_t (*bs_r_8_s) (void *, bus_space_handle_t, bus_size_t); + + /* read multiple stream */ + void (*bs_rm_1_s) (void *, bus_space_handle_t, bus_size_t, + u_int8_t *, bus_size_t); + void (*bs_rm_2_s) (void *, bus_space_handle_t, bus_size_t, + u_int16_t *, bus_size_t); + void (*bs_rm_4_s) (void *, bus_space_handle_t, + bus_size_t, u_int32_t *, bus_size_t); + void (*bs_rm_8_s) (void *, bus_space_handle_t, + bus_size_t, u_int64_t *, bus_size_t); + + /* read region stream */ + void (*bs_rr_1_s) (void *, bus_space_handle_t, + bus_size_t, u_int8_t *, bus_size_t); + void (*bs_rr_2_s) (void *, bus_space_handle_t, + bus_size_t, u_int16_t *, bus_size_t); + void (*bs_rr_4_s) (void *, bus_space_handle_t, + bus_size_t, u_int32_t *, bus_size_t); + void (*bs_rr_8_s) (void *, bus_space_handle_t, + bus_size_t, u_int64_t *, bus_size_t); + + /* write single stream */ + void (*bs_w_1_s) (void *, bus_space_handle_t, + bus_size_t, u_int8_t); + void (*bs_w_2_s) (void *, bus_space_handle_t, + bus_size_t, u_int16_t); + void (*bs_w_4_s) (void *, bus_space_handle_t, + bus_size_t, u_int32_t); + void (*bs_w_8_s) (void *, bus_space_handle_t, + bus_size_t, u_int64_t); + + /* write multiple stream */ + void (*bs_wm_1_s) (void *, bus_space_handle_t, + bus_size_t, const u_int8_t *, bus_size_t); + void (*bs_wm_2_s) (void *, bus_space_handle_t, + bus_size_t, const u_int16_t *, bus_size_t); + void (*bs_wm_4_s) (void *, bus_space_handle_t, + bus_size_t, const u_int32_t *, bus_size_t); + void (*bs_wm_8_s) (void *, bus_space_handle_t, + bus_size_t, const u_int64_t *, bus_size_t); + + /* write region stream */ + void (*bs_wr_1_s) (void *, bus_space_handle_t, + bus_size_t, const u_int8_t *, bus_size_t); + void (*bs_wr_2_s) (void *, bus_space_handle_t, + bus_size_t, const u_int16_t *, bus_size_t); + void (*bs_wr_4_s) (void *, bus_space_handle_t, + bus_size_t, const u_int32_t *, bus_size_t); + void (*bs_wr_8_s) (void *, bus_space_handle_t, + bus_size_t, const u_int64_t *, bus_size_t); +}; + + +/* + * Utility macros; INTERNAL USE ONLY. + */ +#define __bs_c(a,b) __CONCAT(a,b) +#define __bs_opname(op,size) __bs_c(__bs_c(__bs_c(bs_,op),_),size) + +#define __bs_rs(sz, t, h, o) \ + (*(t)->__bs_opname(r,sz))((t)->bs_cookie, h, o) +#define __bs_ws(sz, t, h, o, v) \ + (*(t)->__bs_opname(w,sz))((t)->bs_cookie, h, o, v) +#define __bs_nonsingle(type, sz, t, h, o, a, c) \ + (*(t)->__bs_opname(type,sz))((t)->bs_cookie, h, o, a, c) +#define __bs_set(type, sz, t, h, o, v, c) \ + (*(t)->__bs_opname(type,sz))((t)->bs_cookie, h, o, v, c) +#define __bs_copy(sz, t, h1, o1, h2, o2, cnt) \ + (*(t)->__bs_opname(c,sz))((t)->bs_cookie, h1, o1, h2, o2, cnt) + +#define __bs_opname_s(op,size) __bs_c(__bs_c(__bs_c(__bs_c(bs_,op),_),size),_s) +#define __bs_rs_s(sz, t, h, o) \ + (*(t)->__bs_opname_s(r,sz))((t)->bs_cookie, h, o) +#define __bs_ws_s(sz, t, h, o, v) \ + (*(t)->__bs_opname_s(w,sz))((t)->bs_cookie, h, o, v) +#define __bs_nonsingle_s(type, sz, t, h, o, a, c) \ + (*(t)->__bs_opname_s(type,sz))((t)->bs_cookie, h, o, a, c) + + +/* + * Mapping and unmapping operations. + */ +#define bus_space_map(t, a, s, c, hp) \ + (*(t)->bs_map)((t)->bs_cookie, (a), (s), (c), (hp)) +#define bus_space_unmap(t, h, s) \ + (*(t)->bs_unmap)((t)->bs_cookie, (h), (s)) +#define bus_space_subregion(t, h, o, s, hp) \ + (*(t)->bs_subregion)((t)->bs_cookie, (h), (o), (s), (hp)) + + +/* + * Allocation and deallocation operations. + */ +#define bus_space_alloc(t, rs, re, s, a, b, c, ap, hp) \ + (*(t)->bs_alloc)((t)->bs_cookie, (rs), (re), (s), (a), (b), \ + (c), (ap), (hp)) +#define bus_space_free(t, h, s) \ + (*(t)->bs_free)((t)->bs_cookie, (h), (s)) + +/* + * Bus barrier operations. + */ +#define bus_space_barrier(t, h, o, l, f) \ + (*(t)->bs_barrier)((t)->bs_cookie, (h), (o), (l), (f)) + + + +/* + * Bus read (single) operations. + */ +#define bus_space_read_1(t, h, o) __bs_rs(1,(t),(h),(o)) +#define bus_space_read_2(t, h, o) __bs_rs(2,(t),(h),(o)) +#define bus_space_read_4(t, h, o) __bs_rs(4,(t),(h),(o)) +#define bus_space_read_8(t, h, o) __bs_rs(8,(t),(h),(o)) + +#define bus_space_read_stream_1(t, h, o) __bs_rs_s(1,(t), (h), (o)) +#define bus_space_read_stream_2(t, h, o) __bs_rs_s(2,(t), (h), (o)) +#define bus_space_read_stream_4(t, h, o) __bs_rs_s(4,(t), (h), (o)) +#define bus_space_read_stream_8(t, h, o) __bs_rs_s(8,8,(t),(h),(o)) + +/* + * Bus read multiple operations. + */ +#define bus_space_read_multi_1(t, h, o, a, c) \ + __bs_nonsingle(rm,1,(t),(h),(o),(a),(c)) +#define bus_space_read_multi_2(t, h, o, a, c) \ + __bs_nonsingle(rm,2,(t),(h),(o),(a),(c)) +#define bus_space_read_multi_4(t, h, o, a, c) \ + __bs_nonsingle(rm,4,(t),(h),(o),(a),(c)) +#define bus_space_read_multi_8(t, h, o, a, c) \ + __bs_nonsingle(rm,8,(t),(h),(o),(a),(c)) + +#define bus_space_read_multi_stream_1(t, h, o, a, c) \ + __bs_nonsingle_s(rm,1,(t),(h),(o),(a),(c)) +#define bus_space_read_multi_stream_2(t, h, o, a, c) \ + __bs_nonsingle_s(rm,2,(t),(h),(o),(a),(c)) +#define bus_space_read_multi_stream_4(t, h, o, a, c) \ + __bs_nonsingle_s(rm,4,(t),(h),(o),(a),(c)) +#define bus_space_read_multi_stream_8(t, h, o, a, c) \ + __bs_nonsingle_s(rm,8,(t),(h),(o),(a),(c)) + + +/* + * Bus read region operations. + */ +#define bus_space_read_region_1(t, h, o, a, c) \ + __bs_nonsingle(rr,1,(t),(h),(o),(a),(c)) +#define bus_space_read_region_2(t, h, o, a, c) \ + __bs_nonsingle(rr,2,(t),(h),(o),(a),(c)) +#define bus_space_read_region_4(t, h, o, a, c) \ + __bs_nonsingle(rr,4,(t),(h),(o),(a),(c)) +#define bus_space_read_region_8(t, h, o, a, c) \ + __bs_nonsingle(rr,8,(t),(h),(o),(a),(c)) + +#define bus_space_read_region_stream_1(t, h, o, a, c) \ + __bs_nonsingle_s(rr,1,(t),(h),(o),(a),(c)) +#define bus_space_read_region_stream_2(t, h, o, a, c) \ + __bs_nonsingle_s(rr,2,(t),(h),(o),(a),(c)) +#define bus_space_read_region_stream_4(t, h, o, a, c) \ + __bs_nonsingle_s(rr,4,(t),(h),(o),(a),(c)) +#define bus_space_read_region_stream_8(t, h, o, a, c) \ + __bs_nonsingle_s(rr,8,(t),(h),(o),(a),(c)) + + +/* + * Bus write (single) operations. + */ +#define bus_space_write_1(t, h, o, v) __bs_ws(1,(t),(h),(o),(v)) +#define bus_space_write_2(t, h, o, v) __bs_ws(2,(t),(h),(o),(v)) +#define bus_space_write_4(t, h, o, v) __bs_ws(4,(t),(h),(o),(v)) +#define bus_space_write_8(t, h, o, v) __bs_ws(8,(t),(h),(o),(v)) + +#define bus_space_write_stream_1(t, h, o, v) __bs_ws_s(1,(t),(h),(o),(v)) +#define bus_space_write_stream_2(t, h, o, v) __bs_ws_s(2,(t),(h),(o),(v)) +#define bus_space_write_stream_4(t, h, o, v) __bs_ws_s(4,(t),(h),(o),(v)) +#define bus_space_write_stream_8(t, h, o, v) __bs_ws_s(8,(t),(h),(o),(v)) + + +/* + * Bus write multiple operations. + */ +#define bus_space_write_multi_1(t, h, o, a, c) \ + __bs_nonsingle(wm,1,(t),(h),(o),(a),(c)) +#define bus_space_write_multi_2(t, h, o, a, c) \ + __bs_nonsingle(wm,2,(t),(h),(o),(a),(c)) +#define bus_space_write_multi_4(t, h, o, a, c) \ + __bs_nonsingle(wm,4,(t),(h),(o),(a),(c)) +#define bus_space_write_multi_8(t, h, o, a, c) \ + __bs_nonsingle(wm,8,(t),(h),(o),(a),(c)) + +#define bus_space_write_multi_stream_1(t, h, o, a, c) \ + __bs_nonsingle_s(wm,1,(t),(h),(o),(a),(c)) +#define bus_space_write_multi_stream_2(t, h, o, a, c) \ + __bs_nonsingle_s(wm,2,(t),(h),(o),(a),(c)) +#define bus_space_write_multi_stream_4(t, h, o, a, c) \ + __bs_nonsingle_s(wm,4,(t),(h),(o),(a),(c)) +#define bus_space_write_multi_stream_8(t, h, o, a, c) \ + __bs_nonsingle_s(wm,8,(t),(h),(o),(a),(c)) + + +/* + * Bus write region operations. + */ +#define bus_space_write_region_1(t, h, o, a, c) \ + __bs_nonsingle(wr,1,(t),(h),(o),(a),(c)) +#define bus_space_write_region_2(t, h, o, a, c) \ + __bs_nonsingle(wr,2,(t),(h),(o),(a),(c)) +#define bus_space_write_region_4(t, h, o, a, c) \ + __bs_nonsingle(wr,4,(t),(h),(o),(a),(c)) +#define bus_space_write_region_8(t, h, o, a, c) \ + __bs_nonsingle(wr,8,(t),(h),(o),(a),(c)) + +#define bus_space_write_region_stream_1(t, h, o, a, c) \ + __bs_nonsingle_s(wr,1,(t),(h),(o),(a),(c)) +#define bus_space_write_region_stream_2(t, h, o, a, c) \ + __bs_nonsingle_s(wr,2,(t),(h),(o),(a),(c)) +#define bus_space_write_region_stream_4(t, h, o, a, c) \ + __bs_nonsingle_s(wr,4,(t),(h),(o),(a),(c)) +#define bus_space_write_region_stream_8(t, h, o, a, c) \ + __bs_nonsingle_s(wr,8,(t),(h),(o),(a),(c)) + + +/* + * Set multiple operations. + */ +#define bus_space_set_multi_1(t, h, o, v, c) \ + __bs_set(sm,1,(t),(h),(o),(v),(c)) +#define bus_space_set_multi_2(t, h, o, v, c) \ + __bs_set(sm,2,(t),(h),(o),(v),(c)) +#define bus_space_set_multi_4(t, h, o, v, c) \ + __bs_set(sm,4,(t),(h),(o),(v),(c)) +#define bus_space_set_multi_8(t, h, o, v, c) \ + __bs_set(sm,8,(t),(h),(o),(v),(c)) + + +/* + * Set region operations. + */ +#define bus_space_set_region_1(t, h, o, v, c) \ + __bs_set(sr,1,(t),(h),(o),(v),(c)) +#define bus_space_set_region_2(t, h, o, v, c) \ + __bs_set(sr,2,(t),(h),(o),(v),(c)) +#define bus_space_set_region_4(t, h, o, v, c) \ + __bs_set(sr,4,(t),(h),(o),(v),(c)) +#define bus_space_set_region_8(t, h, o, v, c) \ + __bs_set(sr,8,(t),(h),(o),(v),(c)) + + +/* + * Copy operations. + */ +#define bus_space_copy_region_1(t, h1, o1, h2, o2, c) \ + __bs_copy(1, t, h1, o1, h2, o2, c) +#define bus_space_copy_region_2(t, h1, o1, h2, o2, c) \ + __bs_copy(2, t, h1, o1, h2, o2, c) +#define bus_space_copy_region_4(t, h1, o1, h2, o2, c) \ + __bs_copy(4, t, h1, o1, h2, o2, c) +#define bus_space_copy_region_8(t, h1, o1, h2, o2, c) \ + __bs_copy(8, t, h1, o1, h2, o2, c) + +#include + +#endif /* _MACHINE_BUS_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/bus.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/bus_dma.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/bus_dma.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/bus_dma.h (revision 281504) @@ -0,0 +1,8 @@ +/* $FreeBSD$ */ + +#ifndef _MACHINE_BUS_DMA_H_ +#define _MACHINE_BUS_DMA_H_ + +#include + +#endif /* !_MACHINE_BUS_DMA_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/bus_dma.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/clock.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/clock.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/clock.h (revision 281504) @@ -0,0 +1 @@ +/* $FreeBSD$ */ Property changes on: user/ngie/more-tests/sys/arm64/include/clock.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/counter.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/counter.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/counter.h (revision 281504) @@ -0,0 +1,93 @@ +/*- + * Copyright (c) 2012 Konstantin Belousov + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_COUNTER_H_ +#define _MACHINE_COUNTER_H_ + +#include +#ifdef INVARIANTS +#include +#endif + +#define counter_enter() critical_enter() +#define counter_exit() critical_exit() + +#ifdef IN_SUBR_COUNTER_C +static inline uint64_t +counter_u64_read_one(uint64_t *p, int cpu) +{ + + return (*(uint64_t *)((char *)p + sizeof(struct pcpu) * cpu)); +} + +static inline uint64_t +counter_u64_fetch_inline(uint64_t *p) +{ + uint64_t r; + int i; + + r = 0; + for (i = 0; i < mp_ncpus; i++) + r += counter_u64_read_one((uint64_t *)p, i); + + return (r); +} + +/* XXXKIB might interrupt increment */ +static void +counter_u64_zero_one_cpu(void *arg) +{ + + *((uint64_t *)((char *)arg + sizeof(struct pcpu) * + PCPU_GET(cpuid))) = 0; +} + +static inline void +counter_u64_zero_inline(counter_u64_t c) +{ + + smp_rendezvous(smp_no_rendevous_barrier, counter_u64_zero_one_cpu, + smp_no_rendevous_barrier, c); +} +#endif + +#define counter_u64_add_protected(c, inc) do { \ + CRITICAL_ASSERT(curthread); \ + *(uint64_t *)zpcpu_get(c) += (inc); \ +} while (0) + +static inline void +counter_u64_add(counter_u64_t c, int64_t inc) +{ + + counter_enter(); + counter_u64_add_protected(c, inc); + counter_exit(); +} + +#endif /* ! _MACHINE_COUNTER_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/counter.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/cpufunc.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/cpufunc.h (revision 281503) +++ user/ngie/more-tests/sys/arm64/include/cpufunc.h (revision 281504) @@ -1,135 +1,136 @@ /*- * Copyright (c) 2014 Andrew Turner * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _MACHINE_CPUFUNC_H_ #define _MACHINE_CPUFUNC_H_ #ifdef _KERNEL #include static __inline void breakpoint(void) { __asm("brk #0"); } static __inline register_t dbg_disable(void) { uint32_t ret; __asm __volatile( "mrs %x0, daif \n" "msr daifset, #8 \n" : "=&r" (ret)); return (ret); } static __inline void dbg_enable(void) { __asm __volatile("msr daifclr, #8"); } static __inline register_t intr_disable(void) { /* DAIF is a 32-bit register */ uint32_t ret; __asm __volatile( "mrs %x0, daif \n" "msr daifset, #2 \n" : "=&r" (ret)); return (ret); } static __inline void intr_restore(register_t s) { WRITE_SPECIALREG(daif, s); } static __inline void intr_enable(void) { __asm __volatile("msr daifclr, #2"); } static __inline register_t get_midr(void) { uint64_t midr; midr = READ_SPECIALREG(midr_el1); return (midr); } static __inline register_t get_mpidr(void) { uint64_t mpidr; mpidr = READ_SPECIALREG(mpidr_el1); return (mpidr); } #define cpu_nullop() arm64_nullop() +#define cpufunc_nullop() arm64_nullop() #define cpu_setttb(a) arm64_setttb(a) #define cpu_tlb_flushID() arm64_tlb_flushID() #define cpu_tlb_flushID_SE(e) arm64_tlb_flushID_SE(e) #define cpu_dcache_wbinv_range(a, s) arm64_dcache_wbinv_range((a), (s)) #define cpu_dcache_inv_range(a, s) arm64_dcache_inv_range((a), (s)) #define cpu_dcache_wb_range(a, s) arm64_dcache_wb_range((a), (s)) #define cpu_idcache_wbinv_range(a, s) arm64_idcache_wbinv_range((a), (s)) #define cpu_icache_sync_range(a, s) arm64_icache_sync_range((a), (s)) void arm64_nullop(void); void arm64_setttb(vm_offset_t); void arm64_tlb_flushID(void); void arm64_tlb_flushID_SE(vm_offset_t); void arm64_icache_sync_range(vm_offset_t, vm_size_t); void arm64_idcache_wbinv_range(vm_offset_t, vm_size_t); void arm64_dcache_wbinv_range(vm_offset_t, vm_size_t); void arm64_dcache_inv_range(vm_offset_t, vm_size_t); void arm64_dcache_wb_range(vm_offset_t, vm_size_t); #endif /* _KERNEL */ #endif /* _MACHINE_CPUFUNC_H_ */ Index: user/ngie/more-tests/sys/arm64/include/db_machdep.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/db_machdep.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/db_machdep.h (revision 281504) @@ -0,0 +1,126 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * Copyright (c) 2014-2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Semihalf under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_DB_MACHDEP_H_ +#define _MACHINE_DB_MACHDEP_H_ + +#include +#include +#include + +#define T_BREAKPOINT (EXCP_BRK) +#define T_WATCHPOINT (EXCP_WATCHPT_EL1) + +typedef vm_offset_t db_addr_t; +typedef long db_expr_t; + +#define PC_REGS() ((db_addr_t)kdb_thrctx->pcb_pc) + +#define BKPT_INST (0xd4200000) +#define BKPT_SIZE (4) +#define BKPT_SET(inst) (BKPT_INST) + +#define BKPT_SKIP do { \ + kdb_frame->tf_elr += BKPT_SIZE; \ +} while (0) + +#define db_clear_single_step kdb_cpu_clear_singlestep +#define db_set_single_step kdb_cpu_set_singlestep + +#define IS_BREAKPOINT_TRAP(type, code) (type == T_BREAKPOINT) +#define IS_WATCHPOINT_TRAP(type, code) (type == T_WATCHPOINT) + +#define inst_trap_return(ins) (0) +/* ret */ +#define inst_return(ins) (((ins) & 0xfffffc1fu) == 0xd65f0000) +#define inst_call(ins) (((ins) & 0xfc000000u) == 0x94000000u || /* BL */ \ + ((ins) & 0xfffffc1fu) == 0xd63f0000u) /* BLR */ + +#define inst_load(ins) ({ \ + uint32_t tmp_instr = db_get_value(PC_REGS(), sizeof(uint32_t), FALSE); \ + is_load_instr(tmp_instr); \ +}) + +#define inst_store(ins) ({ \ + uint32_t tmp_instr = db_get_value(PC_REGS(), sizeof(uint32_t), FALSE); \ + is_store_instr(tmp_instr); \ +}) + +#define is_load_instr(ins) ((((ins) & 0x3b000000u) == 0x18000000u) || /* literal */ \ + (((ins) & 0x3f400000u) == 0x08400000u) || /* exclusive */ \ + (((ins) & 0x3bc00000u) == 0x28400000u) || /* no-allocate pair */ \ + ((((ins) & 0x3b200c00u) == 0x38000400u) && \ + (((ins) & 0x3be00c00u) != 0x38000400u) && \ + (((ins) & 0xffe00c00u) != 0x3c800400u)) || /* immediate post-indexed */ \ + ((((ins) & 0x3b200c00u) == 0x38000c00u) && \ + (((ins) & 0x3be00c00u) != 0x38000c00u) && \ + (((ins) & 0xffe00c00u) != 0x3c800c00u)) || /* immediate pre-indexed */ \ + ((((ins) & 0x3b200c00u) == 0x38200800u) && \ + (((ins) & 0x3be00c00u) != 0x38200800u) && \ + (((ins) & 0xffe00c00u) != 0x3ca00c80u)) || /* register offset */ \ + ((((ins) & 0x3b200c00u) == 0x38000800u) && \ + (((ins) & 0x3be00c00u) != 0x38000800u)) || /* unprivileged */ \ + ((((ins) & 0x3b200c00u) == 0x38000000u) && \ + (((ins) & 0x3be00c00u) != 0x38000000u) && \ + (((ins) & 0xffe00c00u) != 0x3c800000u)) || /* unscaled immediate */ \ + ((((ins) & 0x3b000000u) == 0x39000000u) && \ + (((ins) & 0x3bc00000u) != 0x39000000u) && \ + (((ins) & 0xffc00000u) != 0x3d800000u)) && /* unsigned immediate */ \ + (((ins) & 0x3bc00000u) == 0x28400000u) || /* pair (offset) */ \ + (((ins) & 0x3bc00000u) == 0x28c00000u) || /* pair (post-indexed) */ \ + (((ins) & 0x3bc00000u) == 0x29800000u)) /* pair (pre-indexed) */ + +#define is_store_instr(ins) ((((ins) & 0x3f400000u) == 0x08000000u) || /* exclusive */ \ + (((ins) & 0x3bc00000u) == 0x28000000u) || /* no-allocate pair */ \ + ((((ins) & 0x3be00c00u) == 0x38000400u) || \ + (((ins) & 0xffe00c00u) == 0x3c800400u)) || /* immediate post-indexed */ \ + ((((ins) & 0x3be00c00u) == 0x38000c00u) || \ + (((ins) & 0xffe00c00u) == 0x3c800c00u)) || /* immediate pre-indexed */ \ + ((((ins) & 0x3be00c00u) == 0x38200800u) || \ + (((ins) & 0xffe00c00u) == 0x3ca00800u)) || /* register offset */ \ + (((ins) & 0x3be00c00u) == 0x38000800u) || /* unprivileged */ \ + ((((ins) & 0x3be00c00u) == 0x38000000u) || \ + (((ins) & 0xffe00c00u) == 0x3c800000u)) || /* unscaled immediate */ \ + ((((ins) & 0x3bc00000u) == 0x39000000u) || \ + (((ins) & 0xffc00000u) == 0x3d800000u)) || /* unsigned immediate */ \ + (((ins) & 0x3bc00000u) == 0x28000000u) || /* pair (offset) */ \ + (((ins) & 0x3bc00000u) == 0x28800000u) || /* pair (post-indexed) */ \ + (((ins) & 0x3bc00000u) == 0x29800000u)) /* pair (pre-indexed) */ + +#define next_instr_address(pc, bd) ((bd) ? (pc) : ((pc) + 4)) + +#define DB_SMALL_VALUE_MAX (0x7fffffff) +#define DB_SMALL_VALUE_MIN (-0x40001) + +#define DB_ELFSIZE 64 + +#endif /* !_MACHINE_DB_MACHDEP_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/db_machdep.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/debug_monitor.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/debug_monitor.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/debug_monitor.h (revision 281504) @@ -0,0 +1,63 @@ +/*- + * Copyright (c) 2014 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Semihalf under + * the sponsorship of the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_DEBUG_MONITOR_H_ +#define _MACHINE_DEBUG_MONITOR_H_ + +#ifdef KDB + +#include + +enum dbg_el_t { + DBG_FROM_EL0 = 0, + DBG_FROM_EL1 = 1, +}; + +enum dbg_access_t { + HW_BREAKPOINT_X = 0, + HW_BREAKPOINT_R = 1, + HW_BREAKPOINT_W = 2, + HW_BREAKPOINT_RW = HW_BREAKPOINT_R | HW_BREAKPOINT_W, +}; + +void dbg_monitor_init(void); +void dbg_show_watchpoint(void); +int dbg_setup_watchpoint(db_expr_t addr, db_expr_t size, enum dbg_el_t el, + enum dbg_access_t access); +int dbg_remove_watchpoint(db_expr_t addr, db_expr_t size, enum dbg_el_t el); +#else +static __inline void +dbg_monitor_init(void) +{ +} +#endif + +#endif /* _MACHINE_DEBUG_MONITOR_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/debug_monitor.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/devmap.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/devmap.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/devmap.h (revision 281504) @@ -0,0 +1,93 @@ +/*- + * Copyright (c) 2013 Ian Lepore + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_DEVMAP_H_ +#define _MACHINE_DEVMAP_H_ + +/* + * This structure is used by MD code to describe static mappings of devices + * which are established as part of bringing up the MMU early in the boot. + */ +struct arm_devmap_entry { + vm_offset_t pd_va; /* virtual address */ + vm_paddr_t pd_pa; /* physical address */ + vm_size_t pd_size; /* size of region */ + vm_prot_t pd_prot; /* protection code */ + int pd_cache; /* cache attributes */ +}; + +/* + * Return the lowest KVA address used in any entry in the registered devmap + * table. This works with whatever table is registered, including the internal + * table used by arm_devmap_add_entry() if that routine was used. Platforms can + * implement initarm_lastaddr() by calling this if static device mappings are + * their only use of high KVA space. + */ +vm_offset_t arm_devmap_lastaddr(void); + +/* + * Automatically allocate KVA (from the top of the address space downwards) and + * make static device mapping entries in an internal table. The internal table + * is automatically registered on the first call to this. + */ +void arm_devmap_add_entry(vm_paddr_t pa, vm_size_t sz); + +/* + * Register a platform-local table to be bootstrapped by the generic + * initarm() in arm/machdep.c. This is used by newer code that allocates and + * fills in its own local table but does not have its own initarm() routine. + */ +void arm_devmap_register_table(const struct arm_devmap_entry * _table); + +/* + * Establish mappings for all the entries in the table. This is called + * automatically from the common initarm() in arm/machdep.c, and also from the + * custom initarm() routines in older code. If the table pointer is NULL, this + * will use the table installed previously by arm_devmap_register_table(). + */ +void arm_devmap_bootstrap(vm_offset_t _l1pt, + const struct arm_devmap_entry *_table); + +/* + * Translate between virtual and physical addresses within a region that is + * static-mapped by the devmap code. If the given address range isn't + * static-mapped, then ptov returns NULL and vtop returns DEVMAP_PADDR_NOTFOUND. + * The latter implies that you can't vtop just the last byte of physical address + * space. This is not as limiting as it might sound, because even if a device + * occupies the end of the physical address space, you're only prevented from + * doing vtop for that single byte. If you vtop a size bigger than 1 it works. + */ +#define DEVMAP_PADDR_NOTFOUND ((vm_paddr_t)(-1)) + +void * arm_devmap_ptov(vm_paddr_t _pa, vm_size_t _sz); +vm_paddr_t arm_devmap_vtop(void * _va, vm_size_t _sz); + +/* Print the static mapping table; used for bootverbose output. */ +void arm_devmap_print_table(void); + +#endif Property changes on: user/ngie/more-tests/sys/arm64/include/devmap.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/dump.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/dump.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/dump.h (revision 281504) @@ -0,0 +1,74 @@ +/*- + * Copyright (c) 2014 EMC Corp. + * Author: Conrad Meyer + * Copyright (c) 2015 The FreeBSD Foundation. + * All rights reserved. + * + * Portions of this software were developed by Andrew Turner + * under sponsorship from the FreeBSD Foundation + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_DUMP_H_ +#define _MACHINE_DUMP_H_ + +#define KERNELDUMP_ARCH_VERSION KERNELDUMP_AARCH64_VERSION +#define EM_VALUE EM_AARCH64 +/* XXX: I suppose 20 should be enough. */ +#define DUMPSYS_MD_PA_NPAIRS 20 +#define DUMPSYS_NUM_AUX_HDRS 1 + +void dumpsys_wbinv_all(void); +int dumpsys_write_aux_headers(struct dumperinfo *di); + +static inline void +dumpsys_pa_init(void) +{ + + dumpsys_gen_pa_init(); +} + +static inline struct dump_pa * +dumpsys_pa_next(struct dump_pa *p) +{ + + return (dumpsys_gen_pa_next(p)); +} + +static inline void +dumpsys_unmap_chunk(vm_paddr_t pa, size_t s, void *va) +{ + + dumpsys_gen_unmap_chunk(pa, s, va); +} + +static inline int +dumpsys(struct dumperinfo *di) +{ + + return (dumpsys_generic(di)); +} + +#endif /* !_MACHINE_DUMP_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/dump.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/hypervisor.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/hypervisor.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/hypervisor.h (revision 281504) @@ -0,0 +1,85 @@ +/*- + * Copyright (c) 2013, 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_HYPERVISOR_H_ +#define _MACHINE_HYPERVISOR_H_ + +/* + * These registers are only useful when in hypervisor context, + * e.g. specific to EL2, or controlling the hypervisor. + */ + +/* + * Architecture feature trap register + */ +#define CPTR_RES0 0x7fefc800 +#define CPTR_RES1 0x000033ff +#define CPTR_TFP 0x00000400 +#define CPTR_TTA 0x00100000 +#define CPTR_TCPAC 0x80000000 + +/* + * Hypervisor Config Register + */ + +#define HCR_VM 0x0000000000000001 +#define HCR_SWIO 0x0000000000000002 +#define HCR_PTW 0x0000000000000004 +#define HCR_FMO 0x0000000000000008 +#define HCR_IMO 0x0000000000000010 +#define HCR_AMO 0x0000000000000020 +#define HCR_VF 0x0000000000000040 +#define HCR_VI 0x0000000000000080 +#define HCR_VSE 0x0000000000000100 +#define HCR_FB 0x0000000000000200 +#define HCR_BSU_MASK 0x0000000000000c00 +#define HCR_DC 0x0000000000001000 +#define HCR_TWI 0x0000000000002000 +#define HCR_TWE 0x0000000000004000 +#define HCR_TID0 0x0000000000008000 +#define HCR_TID1 0x0000000000010000 +#define HCR_TID2 0x0000000000020000 +#define HCR_TID3 0x0000000000040000 +#define HCR_TSC 0x0000000000080000 +#define HCR_TIDCP 0x0000000000100000 +#define HCR_TACR 0x0000000000200000 +#define HCR_TSW 0x0000000000400000 +#define HCR_TPC 0x0000000000800000 +#define HCR_TPU 0x0000000001000000 +#define HCR_TTLB 0x0000000002000000 +#define HCR_TVM 0x0000000004000000 +#define HCR_TGE 0x0000000008000000 +#define HCR_TDZ 0x0000000010000000 +#define HCR_HCD 0x0000000020000000 +#define HCR_TRVM 0x0000000040000000 +#define HCR_RW 0x0000000080000000 +#define HCR_CD 0x0000000100000000 +#define HCR_ID 0x0000000200000000 + +#endif + Property changes on: user/ngie/more-tests/sys/arm64/include/hypervisor.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/in_cksum.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/in_cksum.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/in_cksum.h (revision 281504) @@ -0,0 +1,56 @@ +/*- + * Copyright (c) 1990 The Regents of the University of California. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. All advertising materials mentioning features or use of this software + * must display the following acknowledgement: + * This product includes software developed by the University of + * California, Berkeley and its contributors. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * from tahoe: in_cksum.c 1.2 86/01/05 + * from: @(#)in_cksum.c 1.3 (Berkeley) 1/19/91 + * from: Id: in_cksum.c,v 1.8 1995/12/03 18:35:19 bde Exp + * $FreeBSD$ + */ + +#ifndef _MACHINE_IN_CKSUM_H_ +#define _MACHINE_IN_CKSUM_H_ 1 + +#include + +#ifdef _KERNEL +#define in_cksum(m, len) in_cksum_skip(m, len, 0) +u_short in_addword(u_short sum, u_short b); +u_short in_cksum_skip(struct mbuf *m, int len, int skip); +u_int do_cksum(const void *, int); +#if defined(IPVERSION) && (IPVERSION == 4) +u_int in_cksum_hdr(const struct ip *); +#endif + +u_short in_pseudo(u_int sum, u_int b, u_int c); + +#endif /* _KERNEL */ +#endif /* _MACHINE_IN_CKSUM_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/in_cksum.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/intr.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/intr.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/intr.h (revision 281504) @@ -0,0 +1,56 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_INTR_H_ +#define _MACHINE_INTR_H_ + +int arm_config_intr(u_int, enum intr_trigger, enum intr_polarity); +void arm_cpu_intr(struct trapframe *); +void arm_dispatch_intr(u_int, struct trapframe *); +int arm_enable_intr(void); +void arm_mask_irq(u_int); +void arm_register_root_pic(device_t, u_int); +void arm_register_msi_pic(device_t); +int arm_alloc_msi(device_t, int, int *); +int arm_release_msi(device_t, int, int *); +int arm_alloc_msix(device_t, int *); +int arm_release_msix(device_t, int); +int arm_map_msi(device_t, int, uint64_t *, uint32_t *); +int arm_map_msix(device_t, int, uint64_t *, uint32_t *); +int arm_setup_intr(const char *, driver_filter_t *, driver_intr_t, + void *, u_int, enum intr_type, void **); +int arm_teardown_intr(void *); +void arm_unmask_irq(u_int); + +#ifdef SMP +void arm_init_secondary(void); +void arm_setup_ipihandler(driver_filter_t *, u_int); +void arm_unmask_ipi(u_int); +#endif + +#endif /* _MACHINE_INTR_H */ Property changes on: user/ngie/more-tests/sys/arm64/include/intr.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/kdb.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/kdb.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/kdb.h (revision 281504) @@ -0,0 +1,53 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Semihalf under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_KDB_H_ +#define _MACHINE_KDB_H_ + +#include + +#define KDB_STOPPEDPCB(pc) &stoppcbs[pc->pc_cpuid] + +void kdb_cpu_clear_singlestep(void); +void kdb_cpu_set_singlestep(void); + +static __inline void +kdb_cpu_sync_icache(unsigned char *addr, size_t size) +{ +} + +static __inline void +kdb_cpu_trap(int type, int code) +{ +} + +#endif /* _MACHINE_KDB_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/kdb.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/machdep.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/machdep.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/machdep.h (revision 281504) @@ -0,0 +1,44 @@ +/*- + * Copyright (c) 2013 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_MACHDEP_H_ +#define _MACHINE_MACHDEP_H_ + +struct arm64_bootparams { + vm_offset_t modulep; + vm_offset_t kern_l1pt; /* L1 page table for the kernel */ + uint64_t kern_delta; + vm_offset_t kern_stack; +}; + +extern vm_paddr_t physmap[]; +extern u_int physmap_idx; + +void initarm(struct arm64_bootparams *); + +#endif /* _MACHINE_MACHDEP_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/machdep.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/md_var.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/md_var.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/md_var.h (revision 281504) @@ -0,0 +1,44 @@ +/*- + * Copyright (c) 1995 Bruce D. Evans. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 3. Neither the name of the author nor the names of contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * from: FreeBSD: src/sys/i386/include/md_var.h,v 1.40 2001/07/12 + * $FreeBSD$ + */ + +#ifndef _MACHINE_MD_VAR_H_ +#define _MACHINE_MD_VAR_H_ + +extern long Maxmem; +extern char sigcode[]; +extern int szsigcode; + +struct dumperinfo; +int minidumpsys(struct dumperinfo *); +void busdma_swi(void); + +#endif /* !_MACHINE_MD_VAR_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/md_var.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/memdev.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/memdev.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/memdev.h (revision 281504) @@ -0,0 +1,40 @@ +/*- + * Copyright (c) 2004 Mark R V Murray + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer + * in this position and unchanged. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHORS ``AS IS'' AND ANY EXPRESS OR + * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES + * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. + * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, + * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT + * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF + * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_MEMDEV_H_ +#define _MACHINE_MEMDEV_H_ + +#define CDEV_MINOR_MEM 0 +#define CDEV_MINOR_KMEM 1 + +d_open_t memopen; +d_read_t memrw; +#define memioctl (d_ioctl_t *)NULL +#define memmmap (d_mmap_t *)NULL + +#endif /* _MACHINE_MEMDEV_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/memdev.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/metadata.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/metadata.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/metadata.h (revision 281504) @@ -0,0 +1,41 @@ +/*- + * Copyright (c) 2014 Andrew Turner + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_METADATA_H_ +#define _MACHINE_METADATA_H_ + +#define MODINFOMD_EFI_MAP 0x1001 +#define MODINFOMD_DTBP 0x1002 + +struct efi_map_header { + size_t memory_size; + size_t descriptor_size; + uint32_t descriptor_version; +}; + +#endif /* !_MACHINE_METADATA_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/metadata.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/ofw_machdep.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/ofw_machdep.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/ofw_machdep.h (revision 281504) @@ -0,0 +1,44 @@ +/*- + * Copyright (c) 2009 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Semihalf under sponsorship from + * the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_OFW_MACHDEP_H_ +#define _MACHINE_OFW_MACHDEP_H_ + +#include + +typedef uint32_t cell_t; + +struct mem_region { + vm_offset_t mr_start; + vm_size_t mr_size; +}; + +#endif /* _MACHINE_OFW_MACHDEP_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/ofw_machdep.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/pmap.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/pmap.h (revision 281503) +++ user/ngie/more-tests/sys/arm64/include/pmap.h (revision 281504) @@ -1,155 +1,156 @@ /*- * Copyright (c) 1991 Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department and William Jolitz of UUNET Technologies Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _MACHINE_PMAP_H_ #define _MACHINE_PMAP_H_ #include #ifndef LOCORE #include #include #include #ifdef _KERNEL #define vtophys(va) pmap_kextract((vm_offset_t)(va)) #endif #define pmap_page_get_memattr(m) ((m)->md.pv_memattr) #define pmap_page_is_write_mapped(m) (((m)->aflags & PGA_WRITEABLE) != 0) void pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma); /* * Pmap stuff */ struct md_page { TAILQ_HEAD(,pv_entry) pv_list; int pv_gen; vm_memattr_t pv_memattr; }; /* * This structure is used to hold a virtual<->physical address * association and is used mostly by bootstrap code */ struct pv_addr { SLIST_ENTRY(pv_addr) pv_list; vm_offset_t pv_va; vm_paddr_t pv_pa; }; struct pmap { struct mtx pm_mtx; struct pmap_statistics pm_stats; /* pmap statictics */ pd_entry_t *pm_l1; TAILQ_HEAD(,pv_chunk) pm_pvchunk; /* list of mappings in pmap */ }; typedef struct pv_entry { vm_offset_t pv_va; /* virtual address for mapping */ TAILQ_ENTRY(pv_entry) pv_next; } *pv_entry_t; /* * pv_entries are allocated in chunks per-process. This avoids the * need to track per-pmap assignments. */ #define _NPCM 3 #define _NPCPV 168 struct pv_chunk { struct pmap * pc_pmap; TAILQ_ENTRY(pv_chunk) pc_list; uint64_t pc_map[_NPCM]; /* bitmap; 1 = free */ TAILQ_ENTRY(pv_chunk) pc_lru; struct pv_entry pc_pventry[_NPCPV]; }; typedef struct pmap *pmap_t; #ifdef _KERNEL extern struct pmap kernel_pmap_store; #define kernel_pmap (&kernel_pmap_store) #define pmap_kernel() kernel_pmap #define PMAP_ASSERT_LOCKED(pmap) \ mtx_assert(&(pmap)->pm_mtx, MA_OWNED) #define PMAP_LOCK(pmap) mtx_lock(&(pmap)->pm_mtx) #define PMAP_LOCK_ASSERT(pmap, type) \ mtx_assert(&(pmap)->pm_mtx, (type)) #define PMAP_LOCK_DESTROY(pmap) mtx_destroy(&(pmap)->pm_mtx) #define PMAP_LOCK_INIT(pmap) mtx_init(&(pmap)->pm_mtx, "pmap", \ NULL, MTX_DEF | MTX_DUPOK) #define PMAP_OWNED(pmap) mtx_owned(&(pmap)->pm_mtx) #define PMAP_MTX(pmap) (&(pmap)->pm_mtx) #define PMAP_TRYLOCK(pmap) mtx_trylock(&(pmap)->pm_mtx) #define PMAP_UNLOCK(pmap) mtx_unlock(&(pmap)->pm_mtx) #define PHYS_AVAIL_SIZE 10 extern vm_paddr_t phys_avail[]; extern vm_paddr_t dump_avail[]; extern vm_offset_t virtual_avail; extern vm_offset_t virtual_end; /* * Macros to test if a mapping is mappable with an L1 Section mapping * or an L2 Large Page mapping. */ #define L1_MAPPABLE_P(va, pa, size) \ ((((va) | (pa)) & L1_OFFSET) == 0 && (size) >= L1_SIZE) void pmap_bootstrap(vm_offset_t, vm_paddr_t, vm_size_t); void pmap_kenter(vm_offset_t, vm_paddr_t); -void pmap_kenter_device(vm_offset_t, vm_paddr_t); +void pmap_kenter_device(vm_offset_t, vm_size_t, vm_paddr_t); vm_paddr_t pmap_kextract(vm_offset_t va); void pmap_kremove(vm_offset_t); +void pmap_kremove_device(vm_offset_t, vm_size_t); void *pmap_mapdev(vm_offset_t, vm_size_t); void pmap_unmapdev(vm_offset_t, vm_size_t); boolean_t pmap_map_io_transient(vm_page_t *, vm_offset_t *, int, boolean_t); void pmap_unmap_io_transient(vm_page_t *, vm_offset_t *, int, boolean_t); #define pmap_page_is_mapped(m) (!TAILQ_EMPTY(&(m)->md.pv_list)) #endif /* _KERNEL */ #endif /* !LOCORE */ #endif /* !_MACHINE_PMAP_H_ */ Index: user/ngie/more-tests/sys/arm64/include/resource.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/resource.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/resource.h (revision 281504) @@ -0,0 +1,46 @@ +/*- + * Copyright 1998 Massachusetts Institute of Technology + * + * Permission to use, copy, modify, and distribute this software and + * its documentation for any purpose and without fee is hereby + * granted, provided that both the above copyright notice and this + * permission notice appear in all copies, that both the above + * copyright notice and this permission notice appear in all + * supporting documentation, and that the name of M.I.T. not be used + * in advertising or publicity pertaining to distribution of the + * software without specific, written prior permission. M.I.T. makes + * no representations about the suitability of this software for any + * purpose. It is provided "as is" without express or implied + * warranty. + * + * THIS SOFTWARE IS PROVIDED BY M.I.T. ``AS IS''. M.I.T. DISCLAIMS + * ALL EXPRESS OR IMPLIED WARRANTIES WITH REGARD TO THIS SOFTWARE, + * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF + * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. IN NO EVENT + * SHALL M.I.T. BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_RESOURCE_H_ +#define _MACHINE_RESOURCE_H_ 1 + +/* + * Definitions of resource types for Intel Architecture machines + * with support for legacy ISA devices and drivers. + */ + +#define SYS_RES_IRQ 1 /* interrupt lines */ +#define SYS_RES_DRQ 2 /* isa dma lines */ +#define SYS_RES_MEMORY 3 /* i/o memory */ +#define SYS_RES_IOPORT 4 /* i/o ports */ +#define SYS_RES_GPIO 5 /* general purpose i/o */ + +#endif /* !_MACHINE_RESOURCE_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/resource.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/sf_buf.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/sf_buf.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/sf_buf.h (revision 281504) @@ -0,0 +1,51 @@ +/*- + * Copyright (c) 2003, 2005 Alan L. Cox + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_SF_BUF_H_ +#define _MACHINE_SF_BUF_H_ + +/* + * On this machine, the only purpose for which sf_buf is used is to implement + * an opaque pointer required by the machine-independent parts of the kernel. + * That pointer references the vm_page that is "mapped" by the sf_buf. The + * actual mapping is provided by the direct virtual-to-physical mapping. + */ +static inline vm_offset_t +sf_buf_kva(struct sf_buf *sf) +{ + + return (PHYS_TO_DMAP(VM_PAGE_TO_PHYS((vm_page_t)sf))); +} + +static inline vm_page_t +sf_buf_page(struct sf_buf *sf) +{ + + return ((vm_page_t)sf); +} +#endif /* !_MACHINE_SF_BUF_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/sf_buf.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/smp.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/smp.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/smp.h (revision 281504) @@ -0,0 +1 @@ +/* $FreeBSD$ */ Property changes on: user/ngie/more-tests/sys/arm64/include/smp.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/stack.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/stack.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/stack.h (revision 281504) @@ -0,0 +1,35 @@ +/*- + * Copyright (c) 2001 Jake Burkholder. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_STACK_H_ +#define _MACHINE_STACK_H_ + +#define INKERNEL(va) \ + ((va) >= VM_MIN_KERNEL_ADDRESS && (va) <= VM_MAX_KERNEL_ADDRESS) + +#endif /* !_MACHINE_STACK_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/stack.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/trap.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/trap.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/trap.h (revision 281504) @@ -0,0 +1 @@ +/* $FreeBSD$ */ Property changes on: user/ngie/more-tests/sys/arm64/include/trap.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/arm64/include/vfp.h =================================================================== --- user/ngie/more-tests/sys/arm64/include/vfp.h (nonexistent) +++ user/ngie/more-tests/sys/arm64/include/vfp.h (revision 281504) @@ -0,0 +1,46 @@ +/*- + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +#ifndef _MACHINE_VFP_H_ +#define _MACHINE_VFP_H_ + +#ifdef _KERNEL + +#ifndef LOCORE +void vfp_init(void); +void vfp_discard(struct thread *); +void vfp_restore_state(void); +void vfp_save_state(struct thread *); +#endif + +#endif + +#endif /* !_MACHINE_VFP_H_ */ Property changes on: user/ngie/more-tests/sys/arm64/include/vfp.h ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/boot/efi/boot1/Makefile =================================================================== --- user/ngie/more-tests/sys/boot/efi/boot1/Makefile (revision 281503) +++ user/ngie/more-tests/sys/boot/efi/boot1/Makefile (revision 281504) @@ -1,103 +1,106 @@ # $FreeBSD$ MAN= .include # In-tree GCC does not support __attribute__((ms_abi)). .if ${COMPILER_TYPE} != "gcc" MK_SSP= no PROG= loader.sym INTERNALPROG= # architecture-specific loader code SRCS= boot1.c reloc.c start.S CFLAGS+= -I. CFLAGS+= -I${.CURDIR}/../include CFLAGS+= -I${.CURDIR}/../include/${MACHINE_CPUARCH} CFLAGS+= -I${.CURDIR}/../../../contrib/dev/acpica/include CFLAGS+= -I${.CURDIR}/../../.. # Always add MI sources and REGULAR efi loader bits .PATH: ${.CURDIR}/../loader/arch/${MACHINE_CPUARCH} .PATH: ${.CURDIR}/../loader .PATH: ${.CURDIR}/../../common CFLAGS+= -I${.CURDIR}/../../common FILES= boot1.efi boot1.efifat FILESMODE_boot1.efi= ${BINMODE} LDSCRIPT= ${.CURDIR}/../loader/arch/${MACHINE_CPUARCH}/ldscript.${MACHINE_CPUARCH} LDFLAGS= -Wl,-T${LDSCRIPT} -Wl,-Bsymbolic -shared .if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386" CFLAGS+= -fPIC LDFLAGS+= -Wl,-znocombreloc .endif .if ${MACHINE_CPUARCH} == "arm" || ${MACHINE_CPUARCH} == "i386" # # Add libstand for the runtime functions used by the compiler - for example # __aeabi_* (arm) or __divdi3 (i386). # DPADD+= ${LIBSTAND} LDADD+= -lstand .endif ${PROG}: ${LDSCRIPT} OBJCOPY?= objcopy OBJDUMP?= objdump .if ${MACHINE_CPUARCH} == "amd64" EFI_TARGET= efi-app-x86_64 .elif ${MACHINE_CPUARCH} == "i386" EFI_TARGET= efi-app-ia32 .else EFI_TARGET= binary .endif boot1.efi: loader.sym if [ `${OBJDUMP} -t ${.ALLSRC} | fgrep '*UND*' | wc -l` != 0 ]; then \ ${OBJDUMP} -t ${.ALLSRC} | fgrep '*UND*'; \ exit 1; \ fi ${OBJCOPY} -j .peheader -j .text -j .sdata -j .data \ -j .dynamic -j .dynsym -j .rel.dyn \ -j .rela.dyn -j .reloc -j .eh_frame -j set_Xcommand_set \ --output-target=${EFI_TARGET} ${.ALLSRC} ${.TARGET} boot1.o: ${.CURDIR}/../../common/ufsread.c # The following inserts out objects into a template FAT file system # created by generate-fat.sh .include "${.CURDIR}/Makefile.fat" boot1.efifat: boot1.efi echo ${.OBJDIR} uudecode ${.CURDIR}/fat-${MACHINE_CPUARCH}.tmpl.bz2.uu mv fat-${MACHINE_CPUARCH}.tmpl.bz2 ${.TARGET}.bz2 bzip2 -f -d ${.TARGET}.bz2 dd if=boot1.efi of=${.TARGET} seek=${BOOT1_OFFSET} conv=notrunc CLEANFILES= boot1.efi boot1.efifat .endif # ${COMPILER_TYPE} != "gcc" .include -.if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386" -beforedepend ${OBJS}: machine x86 +beforedepend ${OBJS}: machine -CLEANFILES+= machine x86 +CLEANFILES+= machine machine: - ln -sf ${.CURDIR}/../../../amd64/include machine + ln -sf ${.CURDIR}/../../../${MACHINE}/include machine + +.if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386" +beforedepend ${OBJS}: x86 +CLEANFILES+= x86 x86: ln -sf ${.CURDIR}/../../../x86/include x86 .endif Index: user/ngie/more-tests/sys/boot/efi/loader/Makefile =================================================================== --- user/ngie/more-tests/sys/boot/efi/loader/Makefile (revision 281503) +++ user/ngie/more-tests/sys/boot/efi/loader/Makefile (revision 281504) @@ -1,122 +1,127 @@ # $FreeBSD$ MAN= .include # In-tree GCC does not support __attribute__((ms_abi)). .if ${COMPILER_TYPE} != "gcc" MK_SSP= no PROG= loader.sym INTERNALPROG= .PATH: ${.CURDIR}/../../efi/loader # architecture-specific loader code SRCS= autoload.c \ bootinfo.c \ conf.c \ copy.c \ devicename.c \ main.c \ reloc.c \ smbios.c \ vers.c .PATH: ${.CURDIR}/arch/${MACHINE_CPUARCH} # For smbios.c .PATH: ${.CURDIR}/../../i386/libi386 .include "${.CURDIR}/arch/${MACHINE_CPUARCH}/Makefile.inc" CFLAGS+= -I${.CURDIR} CFLAGS+= -I${.CURDIR}/arch/${MACHINE_CPUARCH} CFLAGS+= -I${.CURDIR}/../include CFLAGS+= -I${.CURDIR}/../include/${MACHINE_CPUARCH} CFLAGS+= -I${.CURDIR}/../../../contrib/dev/acpica/include CFLAGS+= -I${.CURDIR}/../../.. CFLAGS+= -I${.CURDIR}/../../i386/libi386 CFLAGS+= -DNO_PCI -DEFI .if ${MK_FORTH} != "no" BOOT_FORTH= yes CFLAGS+= -DBOOT_FORTH CFLAGS+= -I${.CURDIR}/../../ficl CFLAGS+= -I${.CURDIR}/../../ficl/${MACHINE_CPUARCH} LIBFICL= ${.OBJDIR}/../../ficl/libficl.a .endif LOADER_FDT_SUPPORT?= no .if ${MK_FDT} != "no" && ${LOADER_FDT_SUPPORT} != "no" CFLAGS+= -I${.CURDIR}/../../fdt CFLAGS+= -I${.OBJDIR}/../../fdt CFLAGS+= -DLOADER_FDT_SUPPORT LIBEFI_FDT= ${.OBJDIR}/../../efi/fdt/libefi_fdt.a LIBFDT= ${.OBJDIR}/../../fdt/libfdt.a .endif # Include bcache code. HAVE_BCACHE= yes .if defined(EFI_STAGING_SIZE) CFLAGS+= -DEFI_STAGING_SIZE=${EFI_STAGING_SIZE} .endif # Always add MI sources .PATH: ${.CURDIR}/../../common .include "${.CURDIR}/../../common/Makefile.inc" CFLAGS+= -I${.CURDIR}/../../common FILES= loader.efi FILESMODE_loader.efi= ${BINMODE} LDSCRIPT= ${.CURDIR}/arch/${MACHINE_CPUARCH}/ldscript.${MACHINE_CPUARCH} LDFLAGS+= -Wl,-T${LDSCRIPT} -Wl,-Bsymbolic -shared CLEANFILES= vers.c loader.efi NEWVERSWHAT= "EFI loader" ${MACHINE_CPUARCH} vers.c: ${.CURDIR}/../../common/newvers.sh ${.CURDIR}/../../efi/loader/version sh ${.CURDIR}/../../common/newvers.sh ${.CURDIR}/version ${NEWVERSWHAT} OBJCOPY?= objcopy OBJDUMP?= objdump .if ${MACHINE_CPUARCH} == "amd64" EFI_TARGET= efi-app-x86_64 .elif ${MACHINE_CPUARCH} == "i386" EFI_TARGET= efi-app-ia32 .else EFI_TARGET= binary .endif loader.efi: loader.sym if [ `${OBJDUMP} -t ${.ALLSRC} | fgrep '*UND*' | wc -l` != 0 ]; then \ ${OBJDUMP} -t ${.ALLSRC} | fgrep '*UND*'; \ exit 1; \ fi ${OBJCOPY} -j .peheader -j .text -j .sdata -j .data \ -j .dynamic -j .dynsym -j .rel.dyn \ -j .rela.dyn -j .reloc -j .eh_frame -j set_Xcommand_set \ --output-target=${EFI_TARGET} ${.ALLSRC} ${.TARGET} LIBEFI= ${.OBJDIR}/../libefi/libefi.a DPADD= ${LIBFICL} ${LIBEFI} ${LIBFDT} ${LIBEFI_FDT} ${LIBSTAND} \ ${LDSCRIPT} LDADD= ${LIBFICL} ${LIBEFI} ${LIBFDT} ${LIBEFI_FDT} ${LIBSTAND} .endif # ${COMPILER_TYPE} != "gcc" .include -beforedepend ${OBJS}: machine x86 +beforedepend ${OBJS}: machine -CLEANFILES+= machine x86 +CLEANFILES+= machine machine: - ln -sf ${.CURDIR}/../../../amd64/include machine + ln -sf ${.CURDIR}/../../../${MACHINE}/include machine +.if ${MACHINE_CPUARCH} == "amd64" || ${MACHINE_CPUARCH} == "i386" +beforedepend ${OBJS}: x86 +CLEANFILES+= x86 + x86: ln -sf ${.CURDIR}/../../../x86/include x86 +.endif Index: user/ngie/more-tests/sys/boot =================================================================== --- user/ngie/more-tests/sys/boot (revision 281503) +++ user/ngie/more-tests/sys/boot (revision 281504) Property changes on: user/ngie/more-tests/sys/boot ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/boot:r281477-281503 Index: user/ngie/more-tests/sys/conf/Makefile.arm64 =================================================================== --- user/ngie/more-tests/sys/conf/Makefile.arm64 (nonexistent) +++ user/ngie/more-tests/sys/conf/Makefile.arm64 (revision 281504) @@ -0,0 +1,54 @@ +# Makefile.arm64 -- with config changes. +# Copyright 1990 W. Jolitz +# from: @(#)Makefile.i386 7.1 5/10/91 +# from FreeBSD: src/sys/conf/Makefile.i386,v 1.255 2002/02/20 23:35:49 +# $FreeBSD$ +# +# Makefile for FreeBSD +# +# This makefile is constructed from a machine description: +# config machineid +# Most changes should be made in the machine description +# /sys/arm64/conf/``machineid'' +# after which you should do +# config machineid +# Generic makefile changes should be made in +# /sys/conf/Makefile.arm64 +# after which config should be rerun for all machines. +# + +# Which version of config(8) is required. +%VERSREQ= 600012 + +.if !defined(S) +S= ../../.. +.endif +.include "$S/conf/kern.pre.mk" + +INCLUDES+= -I$S/contrib/libfdt + +# We generally don't want fpu instructions in the kernel. +CFLAGS += -mgeneral-regs-only + +# Reserve x18 for pcpu data +CFLAGS += -ffixed-x18 + +.if !empty(DDB_ENABLED) +CFLAGS += -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer +.endif + +%BEFORE_DEPEND + +%OBJS + +%FILES.c + +%FILES.s + +%FILES.m + +%CLEAN + +%RULES + +.include "$S/conf/kern.post.mk" Property changes on: user/ngie/more-tests/sys/conf/Makefile.arm64 ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/conf/files.arm64 =================================================================== --- user/ngie/more-tests/sys/conf/files.arm64 (nonexistent) +++ user/ngie/more-tests/sys/conf/files.arm64 (revision 281504) @@ -0,0 +1,50 @@ +# $FreeBSD$ +arm/arm/devmap.c standard +arm/arm/generic_timer.c standard +arm64/arm64/autoconf.c standard +arm64/arm64/bcopy.c standard +arm64/arm64/bus_machdep.c standard +arm64/arm64/bus_space_asm.S standard +arm64/arm64/busdma_machdep.c standard +arm64/arm64/clock.c standard +arm64/arm64/copyinout.S standard +arm64/arm64/copystr.c standard +arm64/arm64/cpufunc_asm.S standard +arm64/arm64/dump_machdep.c standard +arm64/arm64/elf_machdep.c standard +arm64/arm64/exception.S standard +arm64/arm64/gic.c standard +arm64/arm64/identcpu.c standard +arm64/arm64/intr_machdep.c standard +arm64/arm64/in_cksum.c optional inet | inet6 +arm64/arm64/locore.S standard no-obj +arm64/arm64/machdep.c standard +arm64/arm64/mem.c standard +arm64/arm64/minidump_machdep.c standard +arm64/arm64/nexus.c standard +arm64/arm64/pic_if.m standard +arm64/arm64/pmap.c standard +arm64/arm64/stack_machdep.c standard +arm64/arm64/support.S standard +arm64/arm64/swtch.S standard +arm64/arm64/sys_machdep.c standard +arm64/arm64/trap.c standard +arm64/arm64/uio_machdep.c standard +arm64/arm64/vfp.c standard +arm64/arm64/vm_machdep.c standard +dev/fdt/fdt_arm64.c optional fdt +dev/ofw/ofw_cpu.c optional fdt +dev/psci/psci.c optional psci +dev/psci/psci_arm64.S optional psci +dev/uart/uart_cpu_fdt.c optional uart fdt +dev/uart/uart_dev_pl011.c optional uart pl011 +kern/kern_clocksource.c standard +kern/subr_dummy_vdso_tc.c standard +libkern/bcmp.c standard +libkern/ffs.c standard +libkern/ffsl.c standard +libkern/fls.c standard +libkern/flsl.c standard +libkern/flsll.c standard +libkern/memmove.c standard +libkern/memset.c standard Property changes on: user/ngie/more-tests/sys/conf/files.arm64 ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/more-tests/sys/conf/ldscript.arm64 =================================================================== --- user/ngie/more-tests/sys/conf/ldscript.arm64 (nonexistent) +++ user/ngie/more-tests/sys/conf/ldscript.arm64 (revision 281504) @@ -0,0 +1,149 @@ +/* $FreeBSD$ */ +OUTPUT_ARCH(aarch64) +ENTRY(_start) + +SEARCH_DIR(/usr/lib); +SECTIONS +{ + /* Read-only sections, merged into text segment: */ + . = kernbase + SIZEOF_HEADERS; + .text : + { + *(.text) + *(.stub) + /* .gnu.warning sections are handled specially by elf32.em. */ + *(.gnu.warning) + *(.gnu.linkonce.t*) + } =0x9090 + _etext = .; + PROVIDE (etext = .); + .fini : { *(.fini) } =0x9090 + .rodata : { *(.rodata) *(.gnu.linkonce.r*) } + .rodata1 : { *(.rodata1) } + .interp : { *(.interp) } + .hash : { *(.hash) } + .dynsym : { *(.dynsym) } + .dynstr : { *(.dynstr) } + .gnu.version : { *(.gnu.version) } + .gnu.version_d : { *(.gnu.version_d) } + .gnu.version_r : { *(.gnu.version_r) } + .rel.text : + { *(.rel.text) *(.rel.gnu.linkonce.t*) } + .rela.text : + { *(.rela.text) *(.rela.gnu.linkonce.t*) } + .rel.data : + { *(.rel.data) *(.rel.gnu.linkonce.d*) } + .rela.data : + { *(.rela.data) *(.rela.gnu.linkonce.d*) } + .rel.rodata : + { *(.rel.rodata) *(.rel.gnu.linkonce.r*) } + .rela.rodata : + { *(.rela.rodata) *(.rela.gnu.linkonce.r*) } + .rel.got : { *(.rel.got) } + .rela.got : { *(.rela.got) } + .rel.ctors : { *(.rel.ctors) } + .rela.ctors : { *(.rela.ctors) } + .rel.dtors : { *(.rel.dtors) } + .rela.dtors : { *(.rela.dtors) } + .rel.init : { *(.rel.init) } + .rela.init : { *(.rela.init) } + .rel.fini : { *(.rel.fini) } + .rela.fini : { *(.rela.fini) } + .rel.bss : { *(.rel.bss) } + .rela.bss : { *(.rela.bss) } + .rel.plt : { *(.rel.plt) } + .rela.plt : { *(.rela.plt) } + .init : { *(.init) } =0x9090 + .plt : { *(.plt) } + + . = ALIGN(4); + _extab_start = .; + PROVIDE(extab_start = .); + .ARM.extab : { *(.ARM.extab) } + _extab.end = .; + PROVIDE(extab_end = .); + + _exidx_start = .; + PROVIDE(exidx_start = .); + .ARM.exidx : { *(.ARM.exidx) } + _exidx_end = .; + PROVIDE(exidx_end = .); + + /* Adjust the address for the data segment. We want to adjust up to + the same address within the page on the next page up. */ + . = ALIGN(0x1000) + (. & (0x1000 - 1)) ; + .data : + { + *(.data) + *(.gnu.linkonce.d*) + CONSTRUCTORS + } + .data1 : { *(.data1) } + . = ALIGN(32 / 8); + _start_ctors = .; + PROVIDE (start_ctors = .); + .ctors : + { + *(.ctors) + } + _stop_ctors = .; + PROVIDE (stop_ctors = .); + .dtors : + { + *(.dtors) + } + .got : { *(.got.plt) *(.got) } + .dynamic : { *(.dynamic) } + /* We want the small data sections together, so single-instruction offsets + can access them all, and initialized data all before uninitialized, so + we can shorten the on-disk segment size. */ + . = ALIGN(8); + .sdata : { *(.sdata) } + _edata = .; + PROVIDE (edata = .); + __bss_start = .; + .sbss : { *(.sbss) *(.scommon) } + .bss : + { + *(.dynbss) + *(.bss) + *(COMMON) + } + . = ALIGN(8); + _end = . ; + PROVIDE (end = .); + /* Stabs debugging sections. */ + .stab 0 : { *(.stab) } + .stabstr 0 : { *(.stabstr) } + .stab.excl 0 : { *(.stab.excl) } + .stab.exclstr 0 : { *(.stab.exclstr) } + .stab.index 0 : { *(.stab.index) } + .stab.indexstr 0 : { *(.stab.indexstr) } + .comment 0 : { *(.comment) } + /* DWARF debug sections. + Symbols in the DWARF debugging sections are relative to the beginning + of the section so we begin them at 0. */ + /* DWARF 1 */ + .debug 0 : { *(.debug) } + .line 0 : { *(.line) } + /* GNU DWARF 1 extensions */ + .debug_srcinfo 0 : { *(.debug_srcinfo) } + .debug_sfnames 0 : { *(.debug_sfnames) } + /* DWARF 1.1 and DWARF 2 */ + .debug_aranges 0 : { *(.debug_aranges) } + .debug_pubnames 0 : { *(.debug_pubnames) } + /* DWARF 2 */ + .debug_info 0 : { *(.debug_info) } + .debug_abbrev 0 : { *(.debug_abbrev) } + .debug_line 0 : { *(.debug_line) } + .debug_frame 0 : { *(.debug_frame) } + .debug_str 0 : { *(.debug_str) } + .debug_loc 0 : { *(.debug_loc) } + .debug_macinfo 0 : { *(.debug_macinfo) } + /* SGI/MIPS DWARF 2 extensions */ + .debug_weaknames 0 : { *(.debug_weaknames) } + .debug_funcnames 0 : { *(.debug_funcnames) } + .debug_typenames 0 : { *(.debug_typenames) } + .debug_varnames 0 : { *(.debug_varnames) } + /* These must appear regardless of . */ +} Property changes on: user/ngie/more-tests/sys/conf/ldscript.arm64 ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/more-tests/sys/conf/options.arm64 =================================================================== --- user/ngie/more-tests/sys/conf/options.arm64 (nonexistent) +++ user/ngie/more-tests/sys/conf/options.arm64 (revision 281504) @@ -0,0 +1,4 @@ +# $FreeBSD$ + +ARM64 opt_global.h +VFP opt_global.h Property changes on: user/ngie/more-tests/sys/conf/options.arm64 ___________________________________________________________________ Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Index: user/ngie/more-tests/sys/conf/options.i386 =================================================================== --- user/ngie/more-tests/sys/conf/options.i386 (revision 281503) +++ user/ngie/more-tests/sys/conf/options.i386 (revision 281504) @@ -1,124 +1,129 @@ # $FreeBSD$ # Options specific to the i386 platform kernels AUTO_EOI_1 opt_auto_eoi.h AUTO_EOI_2 opt_auto_eoi.h BROKEN_KEYBOARD_RESET opt_reset.h COUNT_XINVLTLB_HITS opt_smp.h COUNT_IPIS opt_smp.h DISABLE_PG_G opt_pmap.h DISABLE_PSE opt_pmap.h I586_PMC_GUPROF opt_i586_guprof.h MAXMEM MPTABLE_FORCE_HTT MP_WATCHDOG NKPT opt_pmap.h PERFMON PMAP_SHPGPERPROC opt_pmap.h POWERFAIL_NMI opt_trap.h PV_STATS opt_pmap.h # Options for emulators. These should only be used at config time, so # they are handled like options for static filesystems # (see src/sys/conf/options), except for broken debugging options. COMPAT_AOUT opt_dontuse.h IBCS2 opt_dontuse.h COMPAT_LINUX opt_dontuse.h COMPAT_SVR4 opt_dontuse.h DEBUG_SVR4 opt_svr4.h LINPROCFS opt_dontuse.h LINSYSFS opt_dontuse.h NDISAPI opt_dontuse.h # Change KVM size. Changes things all over the kernel. KVA_PAGES opt_global.h # Physical address extensions and support for >4G ram. As above. PAE opt_global.h +# Use PAE page tables, but limit memory support to 4GB. +# This keeps the i386 non-PAE KBI, in particular, drivers see +# 32bit vm_paddr_t. +PAE_TABLES opt_global.h + TIMER_FREQ opt_clock.h CPU_ATHLON_SSE_HACK opt_cpu.h CPU_BLUELIGHTNING_3X opt_cpu.h CPU_BLUELIGHTNING_FPU_OP_CACHE opt_cpu.h CPU_BTB_EN opt_cpu.h CPU_CYRIX_NO_LOCK opt_cpu.h CPU_DIRECT_MAPPED_CACHE opt_cpu.h CPU_DISABLE_5X86_LSSER opt_cpu.h CPU_DISABLE_CMPXCHG opt_global.h # XXX global, unlike other CPU_* CPU_DISABLE_SSE opt_cpu.h CPU_ELAN opt_cpu.h CPU_ELAN_PPS opt_cpu.h CPU_ELAN_XTAL opt_cpu.h CPU_ENABLE_LONGRUN opt_cpu.h CPU_FASTER_5X86_FPU opt_cpu.h CPU_GEODE opt_cpu.h CPU_I486_ON_386 opt_cpu.h CPU_IORT opt_cpu.h CPU_L2_LATENCY opt_cpu.h CPU_LOOP_EN opt_cpu.h CPU_PPRO2CELERON opt_cpu.h CPU_RSTK_EN opt_cpu.h CPU_SOEKRIS opt_cpu.h CPU_SUSP_HLT opt_cpu.h CPU_UPGRADE_HW_CACHE opt_cpu.h CPU_WT_ALLOC opt_cpu.h CYRIX_CACHE_REALLY_WORKS opt_cpu.h CYRIX_CACHE_WORKS opt_cpu.h NO_F00F_HACK opt_cpu.h NO_MEMORY_HOLE opt_cpu.h # The CPU type affects the endian conversion functions all over the kernel. I486_CPU opt_global.h I586_CPU opt_global.h I686_CPU opt_global.h # options for serial support COM_ESP opt_sio.h COM_MULTIPORT opt_sio.h CONSPEED opt_sio.h GDBSPEED opt_sio.h COM_NO_ACPI opt_sio.h VGA_ALT_SEQACCESS opt_vga.h VGA_DEBUG opt_vga.h VGA_NO_FONT_LOADING opt_vga.h VGA_NO_MODE_CHANGE opt_vga.h VGA_SLOW_IOACCESS opt_vga.h VGA_WIDTH90 opt_vga.h VESA VESA_DEBUG opt_vesa.h # AGP debugging support AGP_DEBUG opt_agp.h PSM_DEBUG opt_psm.h PSM_HOOKRESUME opt_psm.h PSM_RESETAFTERSUSPEND opt_psm.h ATKBD_DFLT_KEYMAP opt_atkbd.h # Video spigot SPIGOT_UNSECURE opt_spigot.h # Enables NETGRAPH support for Cronyx adapters NETGRAPH_CRONYX opt_ng_cronyx.h # Device options DEV_APIC opt_apic.h DEV_ATPIC opt_atpic.h DEV_NPX opt_npx.h # Debugging NPX_DEBUG opt_npx.h # BPF just-in-time compiler BPF_JITTER opt_bpf.h NATIVE opt_global.h XEN opt_global.h XENHVM opt_global.h # options for the Intel C600 SAS driver (isci) ISCI_LOGGING opt_isci.h Index: user/ngie/more-tests/sys/conf =================================================================== --- user/ngie/more-tests/sys/conf (revision 281503) +++ user/ngie/more-tests/sys/conf (revision 281504) Property changes on: user/ngie/more-tests/sys/conf ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/conf:r281477-281503 Index: user/ngie/more-tests/sys/contrib/x86emu/x86emu.c =================================================================== --- user/ngie/more-tests/sys/contrib/x86emu/x86emu.c (revision 281503) +++ user/ngie/more-tests/sys/contrib/x86emu/x86emu.c (revision 281504) @@ -1,8408 +1,8408 @@ -/* $OpenBSD: x86emu.c,v 1.5 2010/02/17 15:09:47 pirofti Exp $ */ +/* $OpenBSD: x86emu.c,v 1.9 2014/06/15 11:04:49 pirofti Exp $ */ /* $NetBSD: x86emu.c,v 1.7 2009/02/03 19:26:29 joerg Exp $ */ /* * * Realmode X86 Emulator Library * * Copyright (C) 1996-1999 SciTech Software, Inc. * Copyright (C) David Mosberger-Tang * Copyright (C) 1999 Egbert Eich * Copyright (C) 2007 Joerg Sonnenberger * * ======================================================================== * * Permission to use, copy, modify, distribute, and sell this software and * its documentation for any purpose is hereby granted without fee, * provided that the above copyright notice appear in all copies and that * both that copyright notice and this permission notice appear in * supporting documentation, and that the name of the authors not be used * in advertising or publicity pertaining to distribution of the software * without specific, written prior permission. The authors makes no * representations about the suitability of this software for any purpose. * It is provided "as is" without express or implied warranty. * * THE AUTHORS DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, * INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO * EVENT SHALL THE AUTHORS BE LIABLE FOR ANY SPECIAL, INDIRECT OR * CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF * USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR * OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR * PERFORMANCE OF THIS SOFTWARE. * */ #include __FBSDID("$FreeBSD$"); #include #include static void x86emu_intr_raise (struct x86emu *, uint8_t type); static void x86emu_exec_one_byte(struct x86emu *); static void x86emu_exec_two_byte(struct x86emu *); static void fetch_decode_modrm (struct x86emu *); static uint8_t fetch_byte_imm (struct x86emu *); static uint16_t fetch_word_imm (struct x86emu *); static uint32_t fetch_long_imm (struct x86emu *); static uint8_t fetch_data_byte (struct x86emu *, uint32_t offset); static uint8_t fetch_byte (struct x86emu *, u_int segment, uint32_t offset); static uint16_t fetch_data_word (struct x86emu *, uint32_t offset); static uint16_t fetch_word (struct x86emu *, uint32_t segment, uint32_t offset); static uint32_t fetch_data_long (struct x86emu *, uint32_t offset); static uint32_t fetch_long (struct x86emu *, uint32_t segment, uint32_t offset); static void store_data_byte (struct x86emu *, uint32_t offset, uint8_t val); static void store_byte (struct x86emu *, uint32_t segment, uint32_t offset, uint8_t val); static void store_data_word (struct x86emu *, uint32_t offset, uint16_t val); static void store_word (struct x86emu *, uint32_t segment, uint32_t offset, uint16_t val); static void store_data_long (struct x86emu *, uint32_t offset, uint32_t val); static void store_long (struct x86emu *, uint32_t segment, uint32_t offset, uint32_t val); static uint8_t* decode_rl_byte_register(struct x86emu *); static uint16_t* decode_rl_word_register(struct x86emu *); static uint32_t* decode_rl_long_register(struct x86emu *); static uint8_t* decode_rh_byte_register(struct x86emu *); static uint16_t* decode_rh_word_register(struct x86emu *); static uint32_t* decode_rh_long_register(struct x86emu *); static uint16_t* decode_rh_seg_register(struct x86emu *); static uint32_t decode_rl_address(struct x86emu *); static uint8_t decode_and_fetch_byte(struct x86emu *); static uint16_t decode_and_fetch_word(struct x86emu *); static uint32_t decode_and_fetch_long(struct x86emu *); static uint8_t decode_and_fetch_byte_imm8(struct x86emu *, uint8_t *); static uint16_t decode_and_fetch_word_imm8(struct x86emu *, uint8_t *); static uint32_t decode_and_fetch_long_imm8(struct x86emu *, uint8_t *); static uint16_t decode_and_fetch_word_disp(struct x86emu *, int16_t); static uint32_t decode_and_fetch_long_disp(struct x86emu *, int16_t); static void write_back_byte(struct x86emu *, uint8_t); static void write_back_word(struct x86emu *, uint16_t); static void write_back_long(struct x86emu *, uint32_t); static uint16_t aaa_word (struct x86emu *, uint16_t d); static uint16_t aas_word (struct x86emu *, uint16_t d); static uint16_t aad_word (struct x86emu *, uint16_t d); static uint16_t aam_word (struct x86emu *, uint8_t d); static uint8_t adc_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t adc_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t adc_long (struct x86emu *, uint32_t d, uint32_t s); static uint8_t add_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t add_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t add_long (struct x86emu *, uint32_t d, uint32_t s); static uint8_t and_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t and_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t and_long (struct x86emu *, uint32_t d, uint32_t s); static uint8_t cmp_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t cmp_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t cmp_long (struct x86emu *, uint32_t d, uint32_t s); static void cmp_byte_no_return (struct x86emu *, uint8_t d, uint8_t s); static void cmp_word_no_return (struct x86emu *, uint16_t d, uint16_t s); static void cmp_long_no_return (struct x86emu *, uint32_t d, uint32_t s); static uint8_t daa_byte (struct x86emu *, uint8_t d); static uint8_t das_byte (struct x86emu *, uint8_t d); static uint8_t dec_byte (struct x86emu *, uint8_t d); static uint16_t dec_word (struct x86emu *, uint16_t d); static uint32_t dec_long (struct x86emu *, uint32_t d); static uint8_t inc_byte (struct x86emu *, uint8_t d); static uint16_t inc_word (struct x86emu *, uint16_t d); static uint32_t inc_long (struct x86emu *, uint32_t d); static uint8_t or_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t or_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t or_long (struct x86emu *, uint32_t d, uint32_t s); static uint8_t neg_byte (struct x86emu *, uint8_t s); static uint16_t neg_word (struct x86emu *, uint16_t s); static uint32_t neg_long (struct x86emu *, uint32_t s); static uint8_t rcl_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t rcl_word (struct x86emu *, uint16_t d, uint8_t s); static uint32_t rcl_long (struct x86emu *, uint32_t d, uint8_t s); static uint8_t rcr_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t rcr_word (struct x86emu *, uint16_t d, uint8_t s); static uint32_t rcr_long (struct x86emu *, uint32_t d, uint8_t s); static uint8_t rol_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t rol_word (struct x86emu *, uint16_t d, uint8_t s); static uint32_t rol_long (struct x86emu *, uint32_t d, uint8_t s); static uint8_t ror_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t ror_word (struct x86emu *, uint16_t d, uint8_t s); static uint32_t ror_long (struct x86emu *, uint32_t d, uint8_t s); static uint8_t shl_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t shl_word (struct x86emu *, uint16_t d, uint8_t s); static uint32_t shl_long (struct x86emu *, uint32_t d, uint8_t s); static uint8_t shr_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t shr_word (struct x86emu *, uint16_t d, uint8_t s); static uint32_t shr_long (struct x86emu *, uint32_t d, uint8_t s); static uint8_t sar_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t sar_word (struct x86emu *, uint16_t d, uint8_t s); static uint32_t sar_long (struct x86emu *, uint32_t d, uint8_t s); static uint16_t shld_word (struct x86emu *, uint16_t d, uint16_t fill, uint8_t s); static uint32_t shld_long (struct x86emu *, uint32_t d, uint32_t fill, uint8_t s); static uint16_t shrd_word (struct x86emu *, uint16_t d, uint16_t fill, uint8_t s); static uint32_t shrd_long (struct x86emu *, uint32_t d, uint32_t fill, uint8_t s); static uint8_t sbb_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t sbb_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t sbb_long (struct x86emu *, uint32_t d, uint32_t s); static uint8_t sub_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t sub_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t sub_long (struct x86emu *, uint32_t d, uint32_t s); static void test_byte (struct x86emu *, uint8_t d, uint8_t s); static void test_word (struct x86emu *, uint16_t d, uint16_t s); static void test_long (struct x86emu *, uint32_t d, uint32_t s); static uint8_t xor_byte (struct x86emu *, uint8_t d, uint8_t s); static uint16_t xor_word (struct x86emu *, uint16_t d, uint16_t s); static uint32_t xor_long (struct x86emu *, uint32_t d, uint32_t s); static void imul_byte (struct x86emu *, uint8_t s); static void imul_word (struct x86emu *, uint16_t s); static void imul_long (struct x86emu *, uint32_t s); static void mul_byte (struct x86emu *, uint8_t s); static void mul_word (struct x86emu *, uint16_t s); static void mul_long (struct x86emu *, uint32_t s); static void idiv_byte (struct x86emu *, uint8_t s); static void idiv_word (struct x86emu *, uint16_t s); static void idiv_long (struct x86emu *, uint32_t s); static void div_byte (struct x86emu *, uint8_t s); static void div_word (struct x86emu *, uint16_t s); static void div_long (struct x86emu *, uint32_t s); static void ins (struct x86emu *, int size); static void outs (struct x86emu *, int size); static void push_word (struct x86emu *, uint16_t w); static void push_long (struct x86emu *, uint32_t w); static uint16_t pop_word (struct x86emu *); static uint32_t pop_long (struct x86emu *); /* * REMARKS: * Handles any pending asychronous interrupts. */ static void x86emu_intr_dispatch(struct x86emu *emu, uint8_t intno) { if (emu->_x86emu_intrTab[intno]) { (*emu->_x86emu_intrTab[intno]) (emu, intno); } else { push_word(emu, (uint16_t) emu->x86.R_FLG); CLEAR_FLAG(F_IF); CLEAR_FLAG(F_TF); push_word(emu, emu->x86.R_CS); emu->x86.R_CS = fetch_word(emu, 0, intno * 4 + 2); push_word(emu, emu->x86.R_IP); emu->x86.R_IP = fetch_word(emu, 0, intno * 4); } } static void x86emu_intr_handle(struct x86emu *emu) { uint8_t intno; if (emu->x86.intr & INTR_SYNCH) { intno = emu->x86.intno; emu->x86.intr = 0; x86emu_intr_dispatch(emu, intno); } } /* * PARAMETERS: * intrnum - Interrupt number to raise * * REMARKS: * Raise the specified interrupt to be handled before the execution of the * next instruction. */ void x86emu_intr_raise(struct x86emu *emu, uint8_t intrnum) { emu->x86.intno = intrnum; emu->x86.intr |= INTR_SYNCH; } /* * REMARKS: * Main execution loop for the emulator. We return from here when the system * halts, which is normally caused by a stack fault when we return from the * original real mode call. */ void x86emu_exec(struct x86emu *emu) { emu->x86.intr = 0; if (setjmp(emu->exec_state)) return; for (;;) { if (emu->x86.intr) { if (((emu->x86.intr & INTR_SYNCH) && (emu->x86.intno == 0 || emu->x86.intno == 2)) || !ACCESS_FLAG(F_IF)) { x86emu_intr_handle(emu); } } if (emu->x86.R_CS == 0 && emu->x86.R_IP == 0) return; x86emu_exec_one_byte(emu); ++emu->cur_cycles; } } void x86emu_exec_call(struct x86emu *emu, uint16_t seg, uint16_t off) { push_word(emu, 0); push_word(emu, 0); emu->x86.R_CS = seg; emu->x86.R_IP = off; x86emu_exec(emu); } void x86emu_exec_intr(struct x86emu *emu, uint8_t intr) { push_word(emu, emu->x86.R_FLG); CLEAR_FLAG(F_IF); CLEAR_FLAG(F_TF); push_word(emu, 0); push_word(emu, 0); emu->x86.R_CS = (*emu->emu_rdw)(emu, intr * 4 + 2); emu->x86.R_IP = (*emu->emu_rdw)(emu, intr * 4); emu->x86.intr = 0; x86emu_exec(emu); } /* * REMARKS: * Halts the system by setting the halted system flag. */ void x86emu_halt_sys(struct x86emu *emu) { longjmp(emu->exec_state, 1); } /* * PARAMETERS: * mod - Mod value from decoded byte * regh - Reg h value from decoded byte * regl - Reg l value from decoded byte * * REMARKS: * Raise the specified interrupt to be handled before the execution of the * next instruction. * * NOTE: Do not inline this function, as (*emu->emu_rdb) is already inline! */ static void fetch_decode_modrm(struct x86emu *emu) { int fetched; fetched = fetch_byte_imm(emu); emu->cur_mod = (fetched >> 6) & 0x03; emu->cur_rh = (fetched >> 3) & 0x07; emu->cur_rl = (fetched >> 0) & 0x07; } /* * RETURNS: * Immediate byte value read from instruction queue * * REMARKS: * This function returns the immediate byte from the instruction queue, and * moves the instruction pointer to the next value. * * NOTE: Do not inline this function, as (*emu->emu_rdb) is already inline! */ static uint8_t fetch_byte_imm(struct x86emu *emu) { uint8_t fetched; fetched = fetch_byte(emu, emu->x86.R_CS, emu->x86.R_IP); emu->x86.R_IP++; return fetched; } /* * RETURNS: * Immediate word value read from instruction queue * * REMARKS: * This function returns the immediate byte from the instruction queue, and * moves the instruction pointer to the next value. * * NOTE: Do not inline this function, as (*emu->emu_rdw) is already inline! */ static uint16_t fetch_word_imm(struct x86emu *emu) { uint16_t fetched; fetched = fetch_word(emu, emu->x86.R_CS, emu->x86.R_IP); emu->x86.R_IP += 2; return fetched; } /* * RETURNS: * Immediate lone value read from instruction queue * * REMARKS: * This function returns the immediate byte from the instruction queue, and * moves the instruction pointer to the next value. * * NOTE: Do not inline this function, as (*emu->emu_rdw) is already inline! */ static uint32_t fetch_long_imm(struct x86emu *emu) { uint32_t fetched; fetched = fetch_long(emu, emu->x86.R_CS, emu->x86.R_IP); emu->x86.R_IP += 4; return fetched; } /* * RETURNS: * Value of the default data segment * * REMARKS: * Inline function that returns the default data segment for the current * instruction. * * On the x86 processor, the default segment is not always DS if there is * no segment override. Address modes such as -3[BP] or 10[BP+SI] all refer to * addresses relative to SS (ie: on the stack). So, at the minimum, all * decodings of addressing modes would have to set/clear a bit describing * whether the access is relative to DS or SS. That is the function of the * cpu-state-varible emu->x86.mode. There are several potential states: * * repe prefix seen (handled elsewhere) * repne prefix seen (ditto) * * cs segment override * ds segment override * es segment override * fs segment override * gs segment override * ss segment override * * ds/ss select (in absense of override) * * Each of the above 7 items are handled with a bit in the mode field. */ static uint32_t get_data_segment(struct x86emu *emu) { switch (emu->x86.mode & SYSMODE_SEGMASK) { case 0: /* default case: use ds register */ case SYSMODE_SEGOVR_DS: case SYSMODE_SEGOVR_DS | SYSMODE_SEG_DS_SS: return emu->x86.R_DS; case SYSMODE_SEG_DS_SS:/* non-overridden, use ss register */ return emu->x86.R_SS; case SYSMODE_SEGOVR_CS: case SYSMODE_SEGOVR_CS | SYSMODE_SEG_DS_SS: return emu->x86.R_CS; case SYSMODE_SEGOVR_ES: case SYSMODE_SEGOVR_ES | SYSMODE_SEG_DS_SS: return emu->x86.R_ES; case SYSMODE_SEGOVR_FS: case SYSMODE_SEGOVR_FS | SYSMODE_SEG_DS_SS: return emu->x86.R_FS; case SYSMODE_SEGOVR_GS: case SYSMODE_SEGOVR_GS | SYSMODE_SEG_DS_SS: return emu->x86.R_GS; case SYSMODE_SEGOVR_SS: case SYSMODE_SEGOVR_SS | SYSMODE_SEG_DS_SS: return emu->x86.R_SS; } x86emu_halt_sys(emu); } /* * PARAMETERS: * offset - Offset to load data from * * RETURNS: * Byte value read from the absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_rdX) is already inline! */ static uint8_t fetch_data_byte(struct x86emu *emu, uint32_t offset) { return fetch_byte(emu, get_data_segment(emu), offset); } /* * PARAMETERS: * offset - Offset to load data from * * RETURNS: * Word value read from the absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_rdX) is already inline! */ static uint16_t fetch_data_word(struct x86emu *emu, uint32_t offset) { return fetch_word(emu, get_data_segment(emu), offset); } /* * PARAMETERS: * offset - Offset to load data from * * RETURNS: * Long value read from the absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_rdX) is already inline! */ static uint32_t fetch_data_long(struct x86emu *emu, uint32_t offset) { return fetch_long(emu, get_data_segment(emu), offset); } /* * PARAMETERS: * segment - Segment to load data from * offset - Offset to load data from * * RETURNS: * Byte value read from the absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_rdX) is already inline! */ static uint8_t fetch_byte(struct x86emu *emu, uint32_t segment, uint32_t offset) { return (*emu->emu_rdb) (emu, ((uint32_t) segment << 4) + offset); } /* * PARAMETERS: * segment - Segment to load data from * offset - Offset to load data from * * RETURNS: * Word value read from the absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_rdX) is already inline! */ static uint16_t fetch_word(struct x86emu *emu, uint32_t segment, uint32_t offset) { return (*emu->emu_rdw) (emu, ((uint32_t) segment << 4) + offset); } /* * PARAMETERS: * segment - Segment to load data from * offset - Offset to load data from * * RETURNS: * Long value read from the absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_rdX) is already inline! */ static uint32_t fetch_long(struct x86emu *emu, uint32_t segment, uint32_t offset) { return (*emu->emu_rdl) (emu, ((uint32_t) segment << 4) + offset); } /* * PARAMETERS: * offset - Offset to store data at * val - Value to store * * REMARKS: * Writes a word value to an segmented memory location. The segment used is * the current 'default' segment, which may have been overridden. * * NOTE: Do not inline this function as (*emu->emu_wrX) is already inline! */ static void store_data_byte(struct x86emu *emu, uint32_t offset, uint8_t val) { store_byte(emu, get_data_segment(emu), offset, val); } /* * PARAMETERS: * offset - Offset to store data at * val - Value to store * * REMARKS: * Writes a word value to an segmented memory location. The segment used is * the current 'default' segment, which may have been overridden. * * NOTE: Do not inline this function as (*emu->emu_wrX) is already inline! */ static void store_data_word(struct x86emu *emu, uint32_t offset, uint16_t val) { store_word(emu, get_data_segment(emu), offset, val); } /* * PARAMETERS: * offset - Offset to store data at * val - Value to store * * REMARKS: * Writes a long value to an segmented memory location. The segment used is * the current 'default' segment, which may have been overridden. * * NOTE: Do not inline this function as (*emu->emu_wrX) is already inline! */ static void store_data_long(struct x86emu *emu, uint32_t offset, uint32_t val) { store_long(emu, get_data_segment(emu), offset, val); } /* * PARAMETERS: * segment - Segment to store data at * offset - Offset to store data at * val - Value to store * * REMARKS: * Writes a byte value to an absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_wrX) is already inline! */ static void store_byte(struct x86emu *emu, uint32_t segment, uint32_t offset, uint8_t val) { (*emu->emu_wrb) (emu, ((uint32_t) segment << 4) + offset, val); } /* * PARAMETERS: * segment - Segment to store data at * offset - Offset to store data at * val - Value to store * * REMARKS: * Writes a word value to an absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_wrX) is already inline! */ static void store_word(struct x86emu *emu, uint32_t segment, uint32_t offset, uint16_t val) { (*emu->emu_wrw) (emu, ((uint32_t) segment << 4) + offset, val); } /* * PARAMETERS: * segment - Segment to store data at * offset - Offset to store data at * val - Value to store * * REMARKS: * Writes a long value to an absolute memory location. * * NOTE: Do not inline this function as (*emu->emu_wrX) is already inline! */ static void store_long(struct x86emu *emu, uint32_t segment, uint32_t offset, uint32_t val) { (*emu->emu_wrl) (emu, ((uint32_t) segment << 4) + offset, val); } /* * PARAMETERS: * reg - Register to decode * * RETURNS: * Pointer to the appropriate register * * REMARKS: * Return a pointer to the register given by the R/RM field of the * modrm byte, for byte operands. Also enables the decoding of instructions. */ static uint8_t * decode_rm_byte_register(struct x86emu *emu, int reg) { switch (reg) { case 0: return &emu->x86.R_AL; case 1: return &emu->x86.R_CL; case 2: return &emu->x86.R_DL; case 3: return &emu->x86.R_BL; case 4: return &emu->x86.R_AH; case 5: return &emu->x86.R_CH; case 6: return &emu->x86.R_DH; case 7: return &emu->x86.R_BH; default: x86emu_halt_sys(emu); } } static uint8_t * decode_rl_byte_register(struct x86emu *emu) { return decode_rm_byte_register(emu, emu->cur_rl); } static uint8_t * decode_rh_byte_register(struct x86emu *emu) { return decode_rm_byte_register(emu, emu->cur_rh); } /* * PARAMETERS: * reg - Register to decode * * RETURNS: * Pointer to the appropriate register * * REMARKS: * Return a pointer to the register given by the R/RM field of the * modrm byte, for word operands. Also enables the decoding of instructions. */ static uint16_t * decode_rm_word_register(struct x86emu *emu, int reg) { switch (reg) { case 0: return &emu->x86.R_AX; case 1: return &emu->x86.R_CX; case 2: return &emu->x86.R_DX; case 3: return &emu->x86.R_BX; case 4: return &emu->x86.R_SP; case 5: return &emu->x86.R_BP; case 6: return &emu->x86.R_SI; case 7: return &emu->x86.R_DI; default: x86emu_halt_sys(emu); } } static uint16_t * decode_rl_word_register(struct x86emu *emu) { return decode_rm_word_register(emu, emu->cur_rl); } static uint16_t * decode_rh_word_register(struct x86emu *emu) { return decode_rm_word_register(emu, emu->cur_rh); } /* * PARAMETERS: * reg - Register to decode * * RETURNS: * Pointer to the appropriate register * * REMARKS: * Return a pointer to the register given by the R/RM field of the * modrm byte, for dword operands. Also enables the decoding of instructions. */ static uint32_t * decode_rm_long_register(struct x86emu *emu, int reg) { switch (reg) { case 0: return &emu->x86.R_EAX; case 1: return &emu->x86.R_ECX; case 2: return &emu->x86.R_EDX; case 3: return &emu->x86.R_EBX; case 4: return &emu->x86.R_ESP; case 5: return &emu->x86.R_EBP; case 6: return &emu->x86.R_ESI; case 7: return &emu->x86.R_EDI; default: x86emu_halt_sys(emu); } } static uint32_t * decode_rl_long_register(struct x86emu *emu) { return decode_rm_long_register(emu, emu->cur_rl); } static uint32_t * decode_rh_long_register(struct x86emu *emu) { return decode_rm_long_register(emu, emu->cur_rh); } /* * PARAMETERS: * reg - Register to decode * * RETURNS: * Pointer to the appropriate register * * REMARKS: * Return a pointer to the register given by the R/RM field of the * modrm byte, for word operands, modified from above for the weirdo * special case of segreg operands. Also enables the decoding of instructions. */ static uint16_t * decode_rh_seg_register(struct x86emu *emu) { switch (emu->cur_rh) { case 0: return &emu->x86.R_ES; case 1: return &emu->x86.R_CS; case 2: return &emu->x86.R_SS; case 3: return &emu->x86.R_DS; case 4: return &emu->x86.R_FS; case 5: return &emu->x86.R_GS; default: x86emu_halt_sys(emu); } } /* * Return offset from the SIB Byte. */ static uint32_t decode_sib_address(struct x86emu *emu, int sib, int mod) { uint32_t base = 0, i = 0, scale = 1; switch (sib & 0x07) { case 0: base = emu->x86.R_EAX; break; case 1: base = emu->x86.R_ECX; break; case 2: base = emu->x86.R_EDX; break; case 3: base = emu->x86.R_EBX; break; case 4: base = emu->x86.R_ESP; emu->x86.mode |= SYSMODE_SEG_DS_SS; break; case 5: if (mod == 0) { base = fetch_long_imm(emu); } else { base = emu->x86.R_EBP; emu->x86.mode |= SYSMODE_SEG_DS_SS; } break; case 6: base = emu->x86.R_ESI; break; case 7: base = emu->x86.R_EDI; break; } switch ((sib >> 3) & 0x07) { case 0: i = emu->x86.R_EAX; break; case 1: i = emu->x86.R_ECX; break; case 2: i = emu->x86.R_EDX; break; case 3: i = emu->x86.R_EBX; break; case 4: i = 0; break; case 5: i = emu->x86.R_EBP; break; case 6: i = emu->x86.R_ESI; break; case 7: i = emu->x86.R_EDI; break; } scale = 1 << ((sib >> 6) & 0x03); return base + (i * scale); } /* * PARAMETERS: * rm - RM value to decode * * RETURNS: * Offset in memory for the address decoding * * REMARKS: * Return the offset given by mod=00, mod=01 or mod=10 addressing. * Also enables the decoding of instructions. */ static uint32_t decode_rl_address(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_ADDR) { uint32_t offset, sib; /* 32-bit addressing */ switch (emu->cur_rl) { case 0: offset = emu->x86.R_EAX; break; case 1: offset = emu->x86.R_ECX; break; case 2: offset = emu->x86.R_EDX; break; case 3: offset = emu->x86.R_EBX; break; case 4: sib = fetch_byte_imm(emu); offset = decode_sib_address(emu, sib, 0); break; case 5: if (emu->cur_mod == 0) { offset = fetch_long_imm(emu); } else { emu->x86.mode |= SYSMODE_SEG_DS_SS; offset = emu->x86.R_EBP; } break; case 6: offset = emu->x86.R_ESI; break; case 7: offset = emu->x86.R_EDI; break; default: x86emu_halt_sys(emu); } if (emu->cur_mod == 1) offset += (int8_t)fetch_byte_imm(emu); else if (emu->cur_mod == 2) offset += fetch_long_imm(emu); return offset; } else { uint16_t offset; /* 16-bit addressing */ switch (emu->cur_rl) { case 0: offset = emu->x86.R_BX + emu->x86.R_SI; break; case 1: offset = emu->x86.R_BX + emu->x86.R_DI; break; case 2: emu->x86.mode |= SYSMODE_SEG_DS_SS; offset = emu->x86.R_BP + emu->x86.R_SI; break; case 3: emu->x86.mode |= SYSMODE_SEG_DS_SS; offset = emu->x86.R_BP + emu->x86.R_DI; break; case 4: offset = emu->x86.R_SI; break; case 5: offset = emu->x86.R_DI; break; case 6: if (emu->cur_mod == 0) { offset = fetch_word_imm(emu); } else { emu->x86.mode |= SYSMODE_SEG_DS_SS; offset = emu->x86.R_BP; } break; case 7: offset = emu->x86.R_BX; break; default: x86emu_halt_sys(emu); } if (emu->cur_mod == 1) offset += (int8_t)fetch_byte_imm(emu); else if (emu->cur_mod == 2) offset += fetch_word_imm(emu); return offset; } } static uint8_t decode_and_fetch_byte(struct x86emu *emu) { if (emu->cur_mod != 3) { emu->cur_offset = decode_rl_address(emu); return fetch_data_byte(emu, emu->cur_offset); } else { return *decode_rl_byte_register(emu); } } static uint16_t decode_and_fetch_word_disp(struct x86emu *emu, int16_t disp) { if (emu->cur_mod != 3) { /* TODO: A20 gate emulation */ emu->cur_offset = decode_rl_address(emu) + disp; if ((emu->x86.mode & SYSMODE_PREFIX_ADDR) == 0) emu->cur_offset &= 0xffff; return fetch_data_word(emu, emu->cur_offset); } else { return *decode_rl_word_register(emu); } } static uint32_t decode_and_fetch_long_disp(struct x86emu *emu, int16_t disp) { if (emu->cur_mod != 3) { /* TODO: A20 gate emulation */ emu->cur_offset = decode_rl_address(emu) + disp; if ((emu->x86.mode & SYSMODE_PREFIX_ADDR) == 0) emu->cur_offset &= 0xffff; return fetch_data_long(emu, emu->cur_offset); } else { return *decode_rl_long_register(emu); } } uint16_t decode_and_fetch_word(struct x86emu *emu) { return decode_and_fetch_word_disp(emu, 0); } uint32_t decode_and_fetch_long(struct x86emu *emu) { return decode_and_fetch_long_disp(emu, 0); } uint8_t decode_and_fetch_byte_imm8(struct x86emu *emu, uint8_t *imm) { if (emu->cur_mod != 3) { emu->cur_offset = decode_rl_address(emu); *imm = fetch_byte_imm(emu); return fetch_data_byte(emu, emu->cur_offset); } else { *imm = fetch_byte_imm(emu); return *decode_rl_byte_register(emu); } } static uint16_t decode_and_fetch_word_imm8(struct x86emu *emu, uint8_t *imm) { if (emu->cur_mod != 3) { emu->cur_offset = decode_rl_address(emu); *imm = fetch_byte_imm(emu); return fetch_data_word(emu, emu->cur_offset); } else { *imm = fetch_byte_imm(emu); return *decode_rl_word_register(emu); } } static uint32_t decode_and_fetch_long_imm8(struct x86emu *emu, uint8_t *imm) { if (emu->cur_mod != 3) { emu->cur_offset = decode_rl_address(emu); *imm = fetch_byte_imm(emu); return fetch_data_long(emu, emu->cur_offset); } else { *imm = fetch_byte_imm(emu); return *decode_rl_long_register(emu); } } static void write_back_byte(struct x86emu *emu, uint8_t val) { if (emu->cur_mod != 3) store_data_byte(emu, emu->cur_offset, val); else *decode_rl_byte_register(emu) = val; } static void write_back_word(struct x86emu *emu, uint16_t val) { if (emu->cur_mod != 3) store_data_word(emu, emu->cur_offset, val); else *decode_rl_word_register(emu) = val; } static void write_back_long(struct x86emu *emu, uint32_t val) { if (emu->cur_mod != 3) store_data_long(emu, emu->cur_offset, val); else *decode_rl_long_register(emu) = val; } static void common_inc_word_long(struct x86emu *emu, union x86emu_register *reg) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) reg->I32_reg.e_reg = inc_long(emu, reg->I32_reg.e_reg); else reg->I16_reg.x_reg = inc_word(emu, reg->I16_reg.x_reg); } static void common_dec_word_long(struct x86emu *emu, union x86emu_register *reg) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) reg->I32_reg.e_reg = dec_long(emu, reg->I32_reg.e_reg); else reg->I16_reg.x_reg = dec_word(emu, reg->I16_reg.x_reg); } static void common_binop_byte_rm_r(struct x86emu *emu, uint8_t (*binop)(struct x86emu *, uint8_t, uint8_t)) { uint32_t destoffset; uint8_t *destreg, srcval; uint8_t destval; fetch_decode_modrm(emu); srcval = *decode_rh_byte_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_byte(emu, destoffset); destval = (*binop)(emu, destval, srcval); store_data_byte(emu, destoffset, destval); } else { destreg = decode_rl_byte_register(emu); *destreg = (*binop)(emu, *destreg, srcval); } } static void common_binop_ns_byte_rm_r(struct x86emu *emu, void (*binop)(struct x86emu *, uint8_t, uint8_t)) { uint32_t destoffset; uint8_t destval, srcval; fetch_decode_modrm(emu); srcval = *decode_rh_byte_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_byte(emu, destoffset); } else { destval = *decode_rl_byte_register(emu); } (*binop)(emu, destval, srcval); } static void common_binop_word_rm_r(struct x86emu *emu, uint16_t (*binop)(struct x86emu *, uint16_t, uint16_t)) { uint32_t destoffset; uint16_t destval, *destreg, srcval; fetch_decode_modrm(emu); srcval = *decode_rh_word_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_word(emu, destoffset); destval = (*binop)(emu, destval, srcval); store_data_word(emu, destoffset, destval); } else { destreg = decode_rl_word_register(emu); *destreg = (*binop)(emu, *destreg, srcval); } } static void common_binop_byte_r_rm(struct x86emu *emu, uint8_t (*binop)(struct x86emu *, uint8_t, uint8_t)) { uint8_t *destreg, srcval; uint32_t srcoffset; fetch_decode_modrm(emu); destreg = decode_rh_byte_register(emu); if (emu->cur_mod != 3) { srcoffset = decode_rl_address(emu); srcval = fetch_data_byte(emu, srcoffset); } else { srcval = *decode_rl_byte_register(emu); } *destreg = (*binop)(emu, *destreg, srcval); } static void common_binop_long_rm_r(struct x86emu *emu, uint32_t (*binop)(struct x86emu *, uint32_t, uint32_t)) { uint32_t destoffset; uint32_t destval, *destreg, srcval; fetch_decode_modrm(emu); srcval = *decode_rh_long_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_long(emu, destoffset); destval = (*binop)(emu, destval, srcval); store_data_long(emu, destoffset, destval); } else { destreg = decode_rl_long_register(emu); *destreg = (*binop)(emu, *destreg, srcval); } } static void common_binop_word_long_rm_r(struct x86emu *emu, uint16_t (*binop16)(struct x86emu *, uint16_t, uint16_t), uint32_t (*binop32)(struct x86emu *, uint32_t, uint32_t)) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) common_binop_long_rm_r(emu, binop32); else common_binop_word_rm_r(emu, binop16); } static void common_binop_ns_word_rm_r(struct x86emu *emu, void (*binop)(struct x86emu *, uint16_t, uint16_t)) { uint32_t destoffset; uint16_t destval, srcval; fetch_decode_modrm(emu); srcval = *decode_rh_word_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_word(emu, destoffset); } else { destval = *decode_rl_word_register(emu); } (*binop)(emu, destval, srcval); } static void common_binop_ns_long_rm_r(struct x86emu *emu, void (*binop)(struct x86emu *, uint32_t, uint32_t)) { uint32_t destoffset; uint32_t destval, srcval; fetch_decode_modrm(emu); srcval = *decode_rh_long_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_long(emu, destoffset); } else { destval = *decode_rl_long_register(emu); } (*binop)(emu, destval, srcval); } static void common_binop_ns_word_long_rm_r(struct x86emu *emu, void (*binop16)(struct x86emu *, uint16_t, uint16_t), void (*binop32)(struct x86emu *, uint32_t, uint32_t)) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) common_binop_ns_long_rm_r(emu, binop32); else common_binop_ns_word_rm_r(emu, binop16); } static void common_binop_long_r_rm(struct x86emu *emu, uint32_t (*binop)(struct x86emu *, uint32_t, uint32_t)) { uint32_t srcoffset; uint32_t *destreg, srcval; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); if (emu->cur_mod != 3) { srcoffset = decode_rl_address(emu); srcval = fetch_data_long(emu, srcoffset); } else { srcval = *decode_rl_long_register(emu); } *destreg = (*binop)(emu, *destreg, srcval); } static void common_binop_word_r_rm(struct x86emu *emu, uint16_t (*binop)(struct x86emu *, uint16_t, uint16_t)) { uint32_t srcoffset; uint16_t *destreg, srcval; fetch_decode_modrm(emu); destreg = decode_rh_word_register(emu); if (emu->cur_mod != 3) { srcoffset = decode_rl_address(emu); srcval = fetch_data_word(emu, srcoffset); } else { srcval = *decode_rl_word_register(emu); } *destreg = (*binop)(emu, *destreg, srcval); } static void common_binop_word_long_r_rm(struct x86emu *emu, uint16_t (*binop16)(struct x86emu *, uint16_t, uint16_t), uint32_t (*binop32)(struct x86emu *, uint32_t, uint32_t)) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) common_binop_long_r_rm(emu, binop32); else common_binop_word_r_rm(emu, binop16); } static void common_binop_byte_imm(struct x86emu *emu, uint8_t (*binop)(struct x86emu *, uint8_t, uint8_t)) { uint8_t srcval; srcval = fetch_byte_imm(emu); emu->x86.R_AL = (*binop)(emu, emu->x86.R_AL, srcval); } static void common_binop_word_long_imm(struct x86emu *emu, uint16_t (*binop16)(struct x86emu *, uint16_t, uint16_t), uint32_t (*binop32)(struct x86emu *, uint32_t, uint32_t)) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { uint32_t srcval; srcval = fetch_long_imm(emu); emu->x86.R_EAX = (*binop32)(emu, emu->x86.R_EAX, srcval); } else { uint16_t srcval; srcval = fetch_word_imm(emu); emu->x86.R_AX = (*binop16)(emu, emu->x86.R_AX, srcval); } } static void common_push_word_long(struct x86emu *emu, union x86emu_register *reg) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) push_long(emu, reg->I32_reg.e_reg); else push_word(emu, reg->I16_reg.x_reg); } static void common_pop_word_long(struct x86emu *emu, union x86emu_register *reg) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) reg->I32_reg.e_reg = pop_long(emu); else reg->I16_reg.x_reg = pop_word(emu); } static void common_imul_long_IMM(struct x86emu *emu, int byte_imm) { uint32_t srcoffset; uint32_t *destreg, srcval; int32_t imm; uint64_t res; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); if (emu->cur_mod != 3) { srcoffset = decode_rl_address(emu); srcval = fetch_data_long(emu, srcoffset); } else { srcval = *decode_rl_long_register(emu); } if (byte_imm) imm = (int8_t)fetch_byte_imm(emu); else imm = fetch_long_imm(emu); res = (int32_t)srcval * imm; if (res > 0xffffffff) { SET_FLAG(F_CF); SET_FLAG(F_OF); } else { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } *destreg = (uint32_t)res; } static void common_imul_word_IMM(struct x86emu *emu, int byte_imm) { uint32_t srcoffset; uint16_t *destreg, srcval; int16_t imm; uint32_t res; fetch_decode_modrm(emu); destreg = decode_rh_word_register(emu); if (emu->cur_mod != 3) { srcoffset = decode_rl_address(emu); srcval = fetch_data_word(emu, srcoffset); } else { srcval = *decode_rl_word_register(emu); } if (byte_imm) imm = (int8_t)fetch_byte_imm(emu); else imm = fetch_word_imm(emu); res = (int16_t)srcval * imm; if (res > 0xffff) { SET_FLAG(F_CF); SET_FLAG(F_OF); } else { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } *destreg = (uint16_t) res; } static void common_imul_imm(struct x86emu *emu, int byte_imm) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) common_imul_long_IMM(emu, byte_imm); else common_imul_word_IMM(emu, byte_imm); } static void common_jmp_near(struct x86emu *emu, int cond) { int8_t offset; uint16_t target; offset = (int8_t) fetch_byte_imm(emu); target = (uint16_t) (emu->x86.R_IP + (int16_t) offset); if (cond) emu->x86.R_IP = target; } static void common_load_far_pointer(struct x86emu *emu, uint16_t *seg) { uint16_t *dstreg; uint32_t srcoffset; fetch_decode_modrm(emu); if (emu->cur_mod == 3) x86emu_halt_sys(emu); dstreg = decode_rh_word_register(emu); srcoffset = decode_rl_address(emu); *dstreg = fetch_data_word(emu, srcoffset); *seg = fetch_data_word(emu, srcoffset + 2); } /* Implementation */ /* * REMARKS: * Handles opcode 0x3a */ static void x86emuOp_cmp_byte_R_RM(struct x86emu *emu) { uint8_t *destreg, srcval; fetch_decode_modrm(emu); destreg = decode_rh_byte_register(emu); srcval = decode_and_fetch_byte(emu); cmp_byte(emu, *destreg, srcval); } /* * REMARKS: * * Handles opcode 0x3b */ static void x86emuOp32_cmp_word_R_RM(struct x86emu *emu) { uint32_t srcval, *destreg; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); srcval = decode_and_fetch_long(emu); cmp_long(emu, *destreg, srcval); } static void x86emuOp16_cmp_word_R_RM(struct x86emu *emu) { uint16_t srcval, *destreg; fetch_decode_modrm(emu); destreg = decode_rh_word_register(emu); srcval = decode_and_fetch_word(emu); cmp_word(emu, *destreg, srcval); } static void x86emuOp_cmp_word_R_RM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_cmp_word_R_RM(emu); else x86emuOp16_cmp_word_R_RM(emu); } /* * REMARKS: * Handles opcode 0x3c */ static void x86emuOp_cmp_byte_AL_IMM(struct x86emu *emu) { uint8_t srcval; srcval = fetch_byte_imm(emu); cmp_byte(emu, emu->x86.R_AL, srcval); } /* * REMARKS: * Handles opcode 0x3d */ static void x86emuOp32_cmp_word_AX_IMM(struct x86emu *emu) { uint32_t srcval; srcval = fetch_long_imm(emu); cmp_long(emu, emu->x86.R_EAX, srcval); } static void x86emuOp16_cmp_word_AX_IMM(struct x86emu *emu) { uint16_t srcval; srcval = fetch_word_imm(emu); cmp_word(emu, emu->x86.R_AX, srcval); } static void x86emuOp_cmp_word_AX_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_cmp_word_AX_IMM(emu); else x86emuOp16_cmp_word_AX_IMM(emu); } /* * REMARKS: * Handles opcode 0x60 */ static void x86emuOp_push_all(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { uint32_t old_sp = emu->x86.R_ESP; push_long(emu, emu->x86.R_EAX); push_long(emu, emu->x86.R_ECX); push_long(emu, emu->x86.R_EDX); push_long(emu, emu->x86.R_EBX); push_long(emu, old_sp); push_long(emu, emu->x86.R_EBP); push_long(emu, emu->x86.R_ESI); push_long(emu, emu->x86.R_EDI); } else { uint16_t old_sp = emu->x86.R_SP; push_word(emu, emu->x86.R_AX); push_word(emu, emu->x86.R_CX); push_word(emu, emu->x86.R_DX); push_word(emu, emu->x86.R_BX); push_word(emu, old_sp); push_word(emu, emu->x86.R_BP); push_word(emu, emu->x86.R_SI); push_word(emu, emu->x86.R_DI); } } /* * REMARKS: * Handles opcode 0x61 */ static void x86emuOp_pop_all(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_EDI = pop_long(emu); emu->x86.R_ESI = pop_long(emu); emu->x86.R_EBP = pop_long(emu); emu->x86.R_ESP += 4; /* skip ESP */ emu->x86.R_EBX = pop_long(emu); emu->x86.R_EDX = pop_long(emu); emu->x86.R_ECX = pop_long(emu); emu->x86.R_EAX = pop_long(emu); } else { emu->x86.R_DI = pop_word(emu); emu->x86.R_SI = pop_word(emu); emu->x86.R_BP = pop_word(emu); emu->x86.R_SP += 2;/* skip SP */ emu->x86.R_BX = pop_word(emu); emu->x86.R_DX = pop_word(emu); emu->x86.R_CX = pop_word(emu); emu->x86.R_AX = pop_word(emu); } } /*opcode 0x62 ILLEGAL OP, calls x86emuOp_illegal_op() */ /*opcode 0x63 ILLEGAL OP, calls x86emuOp_illegal_op() */ /* * REMARKS: * Handles opcode 0x68 */ static void x86emuOp_push_word_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { uint32_t imm; imm = fetch_long_imm(emu); push_long(emu, imm); } else { uint16_t imm; imm = fetch_word_imm(emu); push_word(emu, imm); } } /* * REMARKS: * Handles opcode 0x6a */ static void x86emuOp_push_byte_IMM(struct x86emu *emu) { int16_t imm; imm = (int8_t) fetch_byte_imm(emu); if (emu->x86.mode & SYSMODE_PREFIX_DATA) { push_long(emu, (int32_t) imm); } else { push_word(emu, imm); } } /* * REMARKS: * Handles opcode 0x6c and 0x6d */ static void x86emuOp_ins_word(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { ins(emu, 4); } else { ins(emu, 2); } } /* * REMARKS: * Handles opcode 0x6f */ static void x86emuOp_outs_word(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { outs(emu, 4); } else { outs(emu, 2); } } /* * REMARKS: * Handles opcode 0x7c */ static void x86emuOp_jump_near_L(struct x86emu *emu) { int sf, of; sf = ACCESS_FLAG(F_SF) != 0; of = ACCESS_FLAG(F_OF) != 0; common_jmp_near(emu, sf != of); } /* * REMARKS: * Handles opcode 0x7d */ static void x86emuOp_jump_near_NL(struct x86emu *emu) { int sf, of; sf = ACCESS_FLAG(F_SF) != 0; of = ACCESS_FLAG(F_OF) != 0; common_jmp_near(emu, sf == of); } /* * REMARKS: * Handles opcode 0x7e */ static void x86emuOp_jump_near_LE(struct x86emu *emu) { int sf, of; sf = ACCESS_FLAG(F_SF) != 0; of = ACCESS_FLAG(F_OF) != 0; common_jmp_near(emu, sf != of || ACCESS_FLAG(F_ZF)); } /* * REMARKS: * Handles opcode 0x7f */ static void x86emuOp_jump_near_NLE(struct x86emu *emu) { int sf, of; sf = ACCESS_FLAG(F_SF) != 0; of = ACCESS_FLAG(F_OF) != 0; common_jmp_near(emu, sf == of && !ACCESS_FLAG(F_ZF)); } static uint8_t(*const opc80_byte_operation[]) (struct x86emu *, uint8_t d, uint8_t s) = { add_byte, /* 00 */ or_byte, /* 01 */ adc_byte, /* 02 */ sbb_byte, /* 03 */ and_byte, /* 04 */ sub_byte, /* 05 */ xor_byte, /* 06 */ cmp_byte, /* 07 */ }; /* * REMARKS: * Handles opcode 0x80 */ static void x86emuOp_opc80_byte_RM_IMM(struct x86emu *emu) { uint8_t imm, destval; /* * Weirdo special case instruction format. Part of the opcode * held below in "RH". Doubly nested case would result, except * that the decoded instruction */ fetch_decode_modrm(emu); destval = decode_and_fetch_byte(emu); imm = fetch_byte_imm(emu); destval = (*opc80_byte_operation[emu->cur_rh]) (emu, destval, imm); if (emu->cur_rh != 7) write_back_byte(emu, destval); } static uint16_t(* const opc81_word_operation[]) (struct x86emu *, uint16_t d, uint16_t s) = { add_word, /* 00 */ or_word, /* 01 */ adc_word, /* 02 */ sbb_word, /* 03 */ and_word, /* 04 */ sub_word, /* 05 */ xor_word, /* 06 */ cmp_word, /* 07 */ }; static uint32_t(* const opc81_long_operation[]) (struct x86emu *, uint32_t d, uint32_t s) = { add_long, /* 00 */ or_long, /* 01 */ adc_long, /* 02 */ sbb_long, /* 03 */ and_long, /* 04 */ sub_long, /* 05 */ xor_long, /* 06 */ cmp_long, /* 07 */ }; /* * REMARKS: * Handles opcode 0x81 */ static void x86emuOp32_opc81_word_RM_IMM(struct x86emu *emu) { uint32_t destval, imm; /* * Weirdo special case instruction format. Part of the opcode * held below in "RH". Doubly nested case would result, except * that the decoded instruction */ fetch_decode_modrm(emu); destval = decode_and_fetch_long(emu); imm = fetch_long_imm(emu); destval = (*opc81_long_operation[emu->cur_rh]) (emu, destval, imm); if (emu->cur_rh != 7) write_back_long(emu, destval); } static void x86emuOp16_opc81_word_RM_IMM(struct x86emu *emu) { uint16_t destval, imm; /* * Weirdo special case instruction format. Part of the opcode * held below in "RH". Doubly nested case would result, except * that the decoded instruction */ fetch_decode_modrm(emu); destval = decode_and_fetch_word(emu); imm = fetch_word_imm(emu); destval = (*opc81_word_operation[emu->cur_rh]) (emu, destval, imm); if (emu->cur_rh != 7) write_back_word(emu, destval); } static void x86emuOp_opc81_word_RM_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_opc81_word_RM_IMM(emu); else x86emuOp16_opc81_word_RM_IMM(emu); } static uint8_t(* const opc82_byte_operation[]) (struct x86emu *, uint8_t s, uint8_t d) = { add_byte, /* 00 */ or_byte, /* 01 *//* YYY UNUSED ???? */ adc_byte, /* 02 */ sbb_byte, /* 03 */ and_byte, /* 04 *//* YYY UNUSED ???? */ sub_byte, /* 05 */ xor_byte, /* 06 *//* YYY UNUSED ???? */ cmp_byte, /* 07 */ }; /* * REMARKS: * Handles opcode 0x82 */ static void x86emuOp_opc82_byte_RM_IMM(struct x86emu *emu) { uint8_t imm, destval; /* * Weirdo special case instruction format. Part of the opcode * held below in "RH". Doubly nested case would result, except * that the decoded instruction Similar to opcode 81, except that * the immediate byte is sign extended to a word length. */ fetch_decode_modrm(emu); destval = decode_and_fetch_byte(emu); imm = fetch_byte_imm(emu); destval = (*opc82_byte_operation[emu->cur_rh]) (emu, destval, imm); if (emu->cur_rh != 7) write_back_byte(emu, destval); } static uint16_t(* const opc83_word_operation[]) (struct x86emu *, uint16_t s, uint16_t d) = { add_word, /* 00 */ or_word, /* 01 *//* YYY UNUSED ???? */ adc_word, /* 02 */ sbb_word, /* 03 */ and_word, /* 04 *//* YYY UNUSED ???? */ sub_word, /* 05 */ xor_word, /* 06 *//* YYY UNUSED ???? */ cmp_word, /* 07 */ }; static uint32_t(* const opc83_long_operation[]) (struct x86emu *, uint32_t s, uint32_t d) = { add_long, /* 00 */ or_long, /* 01 *//* YYY UNUSED ???? */ adc_long, /* 02 */ sbb_long, /* 03 */ and_long, /* 04 *//* YYY UNUSED ???? */ sub_long, /* 05 */ xor_long, /* 06 *//* YYY UNUSED ???? */ cmp_long, /* 07 */ }; /* * REMARKS: * Handles opcode 0x83 */ static void x86emuOp32_opc83_word_RM_IMM(struct x86emu *emu) { uint32_t destval, imm; fetch_decode_modrm(emu); destval = decode_and_fetch_long(emu); imm = (int8_t) fetch_byte_imm(emu); destval = (*opc83_long_operation[emu->cur_rh]) (emu, destval, imm); if (emu->cur_rh != 7) write_back_long(emu, destval); } static void x86emuOp16_opc83_word_RM_IMM(struct x86emu *emu) { uint16_t destval, imm; fetch_decode_modrm(emu); destval = decode_and_fetch_word(emu); imm = (int8_t) fetch_byte_imm(emu); destval = (*opc83_word_operation[emu->cur_rh]) (emu, destval, imm); if (emu->cur_rh != 7) write_back_word(emu, destval); } static void x86emuOp_opc83_word_RM_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_opc83_word_RM_IMM(emu); else x86emuOp16_opc83_word_RM_IMM(emu); } /* * REMARKS: * Handles opcode 0x86 */ static void x86emuOp_xchg_byte_RM_R(struct x86emu *emu) { uint8_t *srcreg, destval, tmp; fetch_decode_modrm(emu); destval = decode_and_fetch_byte(emu); srcreg = decode_rh_byte_register(emu); tmp = destval; destval = *srcreg; *srcreg = tmp; write_back_byte(emu, destval); } /* * REMARKS: * Handles opcode 0x87 */ static void x86emuOp32_xchg_word_RM_R(struct x86emu *emu) { uint32_t *srcreg, destval, tmp; fetch_decode_modrm(emu); destval = decode_and_fetch_long(emu); srcreg = decode_rh_long_register(emu); tmp = destval; destval = *srcreg; *srcreg = tmp; write_back_long(emu, destval); } static void x86emuOp16_xchg_word_RM_R(struct x86emu *emu) { uint16_t *srcreg, destval, tmp; fetch_decode_modrm(emu); destval = decode_and_fetch_word(emu); srcreg = decode_rh_word_register(emu); tmp = destval; destval = *srcreg; *srcreg = tmp; write_back_word(emu, destval); } static void x86emuOp_xchg_word_RM_R(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_xchg_word_RM_R(emu); else x86emuOp16_xchg_word_RM_R(emu); } /* * REMARKS: * Handles opcode 0x88 */ static void x86emuOp_mov_byte_RM_R(struct x86emu *emu) { uint8_t *destreg, *srcreg; uint32_t destoffset; fetch_decode_modrm(emu); srcreg = decode_rh_byte_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); store_data_byte(emu, destoffset, *srcreg); } else { destreg = decode_rl_byte_register(emu); *destreg = *srcreg; } } /* * REMARKS: * Handles opcode 0x89 */ static void x86emuOp32_mov_word_RM_R(struct x86emu *emu) { uint32_t destoffset; uint32_t *destreg, srcval; fetch_decode_modrm(emu); srcval = *decode_rh_long_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); store_data_long(emu, destoffset, srcval); } else { destreg = decode_rl_long_register(emu); *destreg = srcval; } } static void x86emuOp16_mov_word_RM_R(struct x86emu *emu) { uint32_t destoffset; uint16_t *destreg, srcval; fetch_decode_modrm(emu); srcval = *decode_rh_word_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); store_data_word(emu, destoffset, srcval); } else { destreg = decode_rl_word_register(emu); *destreg = srcval; } } static void x86emuOp_mov_word_RM_R(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_mov_word_RM_R(emu); else x86emuOp16_mov_word_RM_R(emu); } /* * REMARKS: * Handles opcode 0x8a */ static void x86emuOp_mov_byte_R_RM(struct x86emu *emu) { uint8_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_byte_register(emu); *destreg = decode_and_fetch_byte(emu); } /* * REMARKS: * Handles opcode 0x8b */ static void x86emuOp_mov_word_R_RM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { uint32_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); *destreg = decode_and_fetch_long(emu); } else { uint16_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_word_register(emu); *destreg = decode_and_fetch_word(emu); } } /* * REMARKS: * Handles opcode 0x8c */ static void x86emuOp_mov_word_RM_SR(struct x86emu *emu) { uint16_t *destreg, srcval; uint32_t destoffset; fetch_decode_modrm(emu); srcval = *decode_rh_seg_register(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); store_data_word(emu, destoffset, srcval); } else { destreg = decode_rl_word_register(emu); *destreg = srcval; } } /* * REMARKS: * Handles opcode 0x8d */ static void x86emuOp_lea_word_R_M(struct x86emu *emu) { uint32_t destoffset; fetch_decode_modrm(emu); if (emu->cur_mod == 3) x86emu_halt_sys(emu); destoffset = decode_rl_address(emu); if (emu->x86.mode & SYSMODE_PREFIX_ADDR) { uint32_t *srcreg; srcreg = decode_rh_long_register(emu); *srcreg = (uint32_t) destoffset; } else { uint16_t *srcreg; srcreg = decode_rh_word_register(emu); *srcreg = (uint16_t) destoffset; } } /* * REMARKS: * Handles opcode 0x8e */ static void x86emuOp_mov_word_SR_RM(struct x86emu *emu) { uint16_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_seg_register(emu); *destreg = decode_and_fetch_word(emu); /* * Clean up, and reset all the R_xSP pointers to the correct * locations. This is about 3x too much overhead (doing all the * segreg ptrs when only one is needed, but this instruction * *cannot* be that common, and this isn't too much work anyway. */ } /* * REMARKS: * Handles opcode 0x8f */ static void x86emuOp32_pop_RM(struct x86emu *emu) { uint32_t destoffset; uint32_t destval, *destreg; fetch_decode_modrm(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = pop_long(emu); store_data_long(emu, destoffset, destval); } else { destreg = decode_rl_long_register(emu); *destreg = pop_long(emu); } } static void x86emuOp16_pop_RM(struct x86emu *emu) { uint32_t destoffset; uint16_t destval, *destreg; fetch_decode_modrm(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = pop_word(emu); store_data_word(emu, destoffset, destval); } else { destreg = decode_rl_word_register(emu); *destreg = pop_word(emu); } } static void x86emuOp_pop_RM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_pop_RM(emu); else x86emuOp16_pop_RM(emu); } /* * REMARKS: * Handles opcode 0x91 */ static void x86emuOp_xchg_word_AX_CX(struct x86emu *emu) { uint32_t tmp; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { tmp = emu->x86.R_EAX; emu->x86.R_EAX = emu->x86.R_ECX; emu->x86.R_ECX = tmp; } else { tmp = emu->x86.R_AX; emu->x86.R_AX = emu->x86.R_CX; emu->x86.R_CX = (uint16_t) tmp; } } /* * REMARKS: * Handles opcode 0x92 */ static void x86emuOp_xchg_word_AX_DX(struct x86emu *emu) { uint32_t tmp; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { tmp = emu->x86.R_EAX; emu->x86.R_EAX = emu->x86.R_EDX; emu->x86.R_EDX = tmp; } else { tmp = emu->x86.R_AX; emu->x86.R_AX = emu->x86.R_DX; emu->x86.R_DX = (uint16_t) tmp; } } /* * REMARKS: * Handles opcode 0x93 */ static void x86emuOp_xchg_word_AX_BX(struct x86emu *emu) { uint32_t tmp; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { tmp = emu->x86.R_EAX; emu->x86.R_EAX = emu->x86.R_EBX; emu->x86.R_EBX = tmp; } else { tmp = emu->x86.R_AX; emu->x86.R_AX = emu->x86.R_BX; emu->x86.R_BX = (uint16_t) tmp; } } /* * REMARKS: * Handles opcode 0x94 */ static void x86emuOp_xchg_word_AX_SP(struct x86emu *emu) { uint32_t tmp; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { tmp = emu->x86.R_EAX; emu->x86.R_EAX = emu->x86.R_ESP; emu->x86.R_ESP = tmp; } else { tmp = emu->x86.R_AX; emu->x86.R_AX = emu->x86.R_SP; emu->x86.R_SP = (uint16_t) tmp; } } /* * REMARKS: * Handles opcode 0x95 */ static void x86emuOp_xchg_word_AX_BP(struct x86emu *emu) { uint32_t tmp; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { tmp = emu->x86.R_EAX; emu->x86.R_EAX = emu->x86.R_EBP; emu->x86.R_EBP = tmp; } else { tmp = emu->x86.R_AX; emu->x86.R_AX = emu->x86.R_BP; emu->x86.R_BP = (uint16_t) tmp; } } /* * REMARKS: * Handles opcode 0x96 */ static void x86emuOp_xchg_word_AX_SI(struct x86emu *emu) { uint32_t tmp; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { tmp = emu->x86.R_EAX; emu->x86.R_EAX = emu->x86.R_ESI; emu->x86.R_ESI = tmp; } else { tmp = emu->x86.R_AX; emu->x86.R_AX = emu->x86.R_SI; emu->x86.R_SI = (uint16_t) tmp; } } /* * REMARKS: * Handles opcode 0x97 */ static void x86emuOp_xchg_word_AX_DI(struct x86emu *emu) { uint32_t tmp; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { tmp = emu->x86.R_EAX; emu->x86.R_EAX = emu->x86.R_EDI; emu->x86.R_EDI = tmp; } else { tmp = emu->x86.R_AX; emu->x86.R_AX = emu->x86.R_DI; emu->x86.R_DI = (uint16_t) tmp; } } /* * REMARKS: * Handles opcode 0x98 */ static void x86emuOp_cbw(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { if (emu->x86.R_AX & 0x8000) { emu->x86.R_EAX |= 0xffff0000; } else { emu->x86.R_EAX &= 0x0000ffff; } } else { if (emu->x86.R_AL & 0x80) { emu->x86.R_AH = 0xff; } else { emu->x86.R_AH = 0x0; } } } /* * REMARKS: * Handles opcode 0x99 */ static void x86emuOp_cwd(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { if (emu->x86.R_EAX & 0x80000000) { emu->x86.R_EDX = 0xffffffff; } else { emu->x86.R_EDX = 0x0; } } else { if (emu->x86.R_AX & 0x8000) { emu->x86.R_DX = 0xffff; } else { emu->x86.R_DX = 0x0; } } } /* * REMARKS: * Handles opcode 0x9a */ static void x86emuOp_call_far_IMM(struct x86emu *emu) { uint16_t farseg, faroff; faroff = fetch_word_imm(emu); farseg = fetch_word_imm(emu); /* XXX * * Hooked interrupt vectors calling into our "BIOS" will cause problems * unless all intersegment stuff is checked for BIOS access. Check * needed here. For moment, let it alone. */ push_word(emu, emu->x86.R_CS); emu->x86.R_CS = farseg; push_word(emu, emu->x86.R_IP); emu->x86.R_IP = faroff; } /* * REMARKS: * Handles opcode 0x9c */ static void x86emuOp_pushf_word(struct x86emu *emu) { uint32_t flags; /* clear out *all* bits not representing flags, and turn on real bits */ flags = (emu->x86.R_EFLG & F_MSK) | F_ALWAYS_ON; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { push_long(emu, flags); } else { push_word(emu, (uint16_t) flags); } } /* * REMARKS: * Handles opcode 0x9d */ static void x86emuOp_popf_word(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_EFLG = pop_long(emu); } else { emu->x86.R_FLG = pop_word(emu); } } /* * REMARKS: * Handles opcode 0x9e */ static void x86emuOp_sahf(struct x86emu *emu) { /* clear the lower bits of the flag register */ emu->x86.R_FLG &= 0xffffff00; /* or in the AH register into the flags register */ emu->x86.R_FLG |= emu->x86.R_AH; } /* * REMARKS: * Handles opcode 0x9f */ static void x86emuOp_lahf(struct x86emu *emu) { emu->x86.R_AH = (uint8_t) (emu->x86.R_FLG & 0xff); /* undocumented TC++ behavior??? Nope. It's documented, but you have * too look real hard to notice it. */ emu->x86.R_AH |= 0x2; } /* * REMARKS: * Handles opcode 0xa0 */ static void x86emuOp_mov_AL_M_IMM(struct x86emu *emu) { uint16_t offset; offset = fetch_word_imm(emu); emu->x86.R_AL = fetch_data_byte(emu, offset); } /* * REMARKS: * Handles opcode 0xa1 */ static void x86emuOp_mov_AX_M_IMM(struct x86emu *emu) { uint16_t offset; offset = fetch_word_imm(emu); if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_EAX = fetch_data_long(emu, offset); } else { emu->x86.R_AX = fetch_data_word(emu, offset); } } /* * REMARKS: * Handles opcode 0xa2 */ static void x86emuOp_mov_M_AL_IMM(struct x86emu *emu) { uint16_t offset; offset = fetch_word_imm(emu); store_data_byte(emu, offset, emu->x86.R_AL); } /* * REMARKS: * Handles opcode 0xa3 */ static void x86emuOp_mov_M_AX_IMM(struct x86emu *emu) { uint16_t offset; offset = fetch_word_imm(emu); if (emu->x86.mode & SYSMODE_PREFIX_DATA) { store_data_long(emu, offset, emu->x86.R_EAX); } else { store_data_word(emu, offset, emu->x86.R_AX); } } /* * REMARKS: * Handles opcode 0xa4 */ static void x86emuOp_movs_byte(struct x86emu *emu) { uint8_t val; uint32_t count; int inc; if (ACCESS_FLAG(F_DF)) /* down */ inc = -1; else inc = 1; count = 1; if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* move them until CX is ZERO. */ count = emu->x86.R_CX; emu->x86.R_CX = 0; emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } while (count--) { val = fetch_data_byte(emu, emu->x86.R_SI); store_byte(emu, emu->x86.R_ES, emu->x86.R_DI, val); emu->x86.R_SI += inc; emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xa5 */ static void x86emuOp_movs_word(struct x86emu *emu) { uint32_t val; int inc; uint32_t count; if (emu->x86.mode & SYSMODE_PREFIX_DATA) inc = 4; else inc = 2; if (ACCESS_FLAG(F_DF)) /* down */ inc = -inc; count = 1; if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* move them until CX is ZERO. */ count = emu->x86.R_CX; emu->x86.R_CX = 0; emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } while (count--) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { val = fetch_data_long(emu, emu->x86.R_SI); store_long(emu, emu->x86.R_ES, emu->x86.R_DI, val); } else { val = fetch_data_word(emu, emu->x86.R_SI); store_word(emu, emu->x86.R_ES, emu->x86.R_DI, (uint16_t) val); } emu->x86.R_SI += inc; emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xa6 */ static void x86emuOp_cmps_byte(struct x86emu *emu) { int8_t val1, val2; int inc; if (ACCESS_FLAG(F_DF)) /* down */ inc = -1; else inc = 1; if (emu->x86.mode & SYSMODE_PREFIX_REPE) { /* REPE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { val1 = fetch_data_byte(emu, emu->x86.R_SI); val2 = fetch_byte(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_byte(emu, val1, val2); emu->x86.R_CX -= 1; emu->x86.R_SI += inc; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF) == 0) break; } emu->x86.mode &= ~SYSMODE_PREFIX_REPE; } else if (emu->x86.mode & SYSMODE_PREFIX_REPNE) { /* REPNE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { val1 = fetch_data_byte(emu, emu->x86.R_SI); val2 = fetch_byte(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_byte(emu, val1, val2); emu->x86.R_CX -= 1; emu->x86.R_SI += inc; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF)) break; /* zero flag set means equal */ } emu->x86.mode &= ~SYSMODE_PREFIX_REPNE; } else { val1 = fetch_data_byte(emu, emu->x86.R_SI); val2 = fetch_byte(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_byte(emu, val1, val2); emu->x86.R_SI += inc; emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xa7 */ static void x86emuOp_cmps_word(struct x86emu *emu) { uint32_t val1, val2; int inc; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { if (ACCESS_FLAG(F_DF)) /* down */ inc = -4; else inc = 4; } else { if (ACCESS_FLAG(F_DF)) /* down */ inc = -2; else inc = 2; } if (emu->x86.mode & SYSMODE_PREFIX_REPE) { /* REPE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { val1 = fetch_data_long(emu, emu->x86.R_SI); val2 = fetch_long(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_long(emu, val1, val2); } else { val1 = fetch_data_word(emu, emu->x86.R_SI); val2 = fetch_word(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_word(emu, (uint16_t) val1, (uint16_t) val2); } emu->x86.R_CX -= 1; emu->x86.R_SI += inc; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF) == 0) break; } emu->x86.mode &= ~SYSMODE_PREFIX_REPE; } else if (emu->x86.mode & SYSMODE_PREFIX_REPNE) { /* REPNE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { val1 = fetch_data_long(emu, emu->x86.R_SI); val2 = fetch_long(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_long(emu, val1, val2); } else { val1 = fetch_data_word(emu, emu->x86.R_SI); val2 = fetch_word(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_word(emu, (uint16_t) val1, (uint16_t) val2); } emu->x86.R_CX -= 1; emu->x86.R_SI += inc; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF)) break; /* zero flag set means equal */ } emu->x86.mode &= ~SYSMODE_PREFIX_REPNE; } else { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { val1 = fetch_data_long(emu, emu->x86.R_SI); val2 = fetch_long(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_long(emu, val1, val2); } else { val1 = fetch_data_word(emu, emu->x86.R_SI); val2 = fetch_word(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_word(emu, (uint16_t) val1, (uint16_t) val2); } emu->x86.R_SI += inc; emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xa9 */ static void x86emuOp_test_AX_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { test_long(emu, emu->x86.R_EAX, fetch_long_imm(emu)); } else { test_word(emu, emu->x86.R_AX, fetch_word_imm(emu)); } } /* * REMARKS: * Handles opcode 0xaa */ static void x86emuOp_stos_byte(struct x86emu *emu) { int inc; if (ACCESS_FLAG(F_DF)) /* down */ inc = -1; else inc = 1; if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { store_byte(emu, emu->x86.R_ES, emu->x86.R_DI, emu->x86.R_AL); emu->x86.R_CX -= 1; emu->x86.R_DI += inc; } emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } else { store_byte(emu, emu->x86.R_ES, emu->x86.R_DI, emu->x86.R_AL); emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xab */ static void x86emuOp_stos_word(struct x86emu *emu) { int inc; uint32_t count; if (emu->x86.mode & SYSMODE_PREFIX_DATA) inc = 4; else inc = 2; if (ACCESS_FLAG(F_DF)) /* down */ inc = -inc; count = 1; if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* move them until CX is ZERO. */ count = emu->x86.R_CX; emu->x86.R_CX = 0; emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } while (count--) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { store_long(emu, emu->x86.R_ES, emu->x86.R_DI, emu->x86.R_EAX); } else { store_word(emu, emu->x86.R_ES, emu->x86.R_DI, emu->x86.R_AX); } emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xac */ static void x86emuOp_lods_byte(struct x86emu *emu) { int inc; if (ACCESS_FLAG(F_DF)) /* down */ inc = -1; else inc = 1; if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { emu->x86.R_AL = fetch_data_byte(emu, emu->x86.R_SI); emu->x86.R_CX -= 1; emu->x86.R_SI += inc; } emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } else { emu->x86.R_AL = fetch_data_byte(emu, emu->x86.R_SI); emu->x86.R_SI += inc; } } /* * REMARKS: * Handles opcode 0xad */ static void x86emuOp_lods_word(struct x86emu *emu) { int inc; uint32_t count; if (emu->x86.mode & SYSMODE_PREFIX_DATA) inc = 4; else inc = 2; if (ACCESS_FLAG(F_DF)) /* down */ inc = -inc; count = 1; if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* move them until CX is ZERO. */ count = emu->x86.R_CX; emu->x86.R_CX = 0; emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } while (count--) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_EAX = fetch_data_long(emu, emu->x86.R_SI); } else { emu->x86.R_AX = fetch_data_word(emu, emu->x86.R_SI); } emu->x86.R_SI += inc; } } /* * REMARKS: * Handles opcode 0xae */ static void x86emuOp_scas_byte(struct x86emu *emu) { int8_t val2; int inc; if (ACCESS_FLAG(F_DF)) /* down */ inc = -1; else inc = 1; if (emu->x86.mode & SYSMODE_PREFIX_REPE) { /* REPE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { val2 = fetch_byte(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_byte(emu, emu->x86.R_AL, val2); emu->x86.R_CX -= 1; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF) == 0) break; } emu->x86.mode &= ~SYSMODE_PREFIX_REPE; } else if (emu->x86.mode & SYSMODE_PREFIX_REPNE) { /* REPNE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { val2 = fetch_byte(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_byte(emu, emu->x86.R_AL, val2); emu->x86.R_CX -= 1; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF)) break; /* zero flag set means equal */ } emu->x86.mode &= ~SYSMODE_PREFIX_REPNE; } else { val2 = fetch_byte(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_byte(emu, emu->x86.R_AL, val2); emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xaf */ static void x86emuOp_scas_word(struct x86emu *emu) { int inc; uint32_t val; if (emu->x86.mode & SYSMODE_PREFIX_DATA) inc = 4; else inc = 2; if (ACCESS_FLAG(F_DF)) /* down */ inc = -inc; if (emu->x86.mode & SYSMODE_PREFIX_REPE) { /* REPE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { val = fetch_long(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_long(emu, emu->x86.R_EAX, val); } else { val = fetch_word(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_word(emu, emu->x86.R_AX, (uint16_t) val); } emu->x86.R_CX -= 1; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF) == 0) break; } emu->x86.mode &= ~SYSMODE_PREFIX_REPE; } else if (emu->x86.mode & SYSMODE_PREFIX_REPNE) { /* REPNE */ /* move them until CX is ZERO. */ while (emu->x86.R_CX != 0) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { val = fetch_long(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_long(emu, emu->x86.R_EAX, val); } else { val = fetch_word(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_word(emu, emu->x86.R_AX, (uint16_t) val); } emu->x86.R_CX -= 1; emu->x86.R_DI += inc; if (ACCESS_FLAG(F_ZF)) break; /* zero flag set means equal */ } emu->x86.mode &= ~SYSMODE_PREFIX_REPNE; } else { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { val = fetch_long(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_long(emu, emu->x86.R_EAX, val); } else { val = fetch_word(emu, emu->x86.R_ES, emu->x86.R_DI); cmp_word(emu, emu->x86.R_AX, (uint16_t) val); } emu->x86.R_DI += inc; } } /* * REMARKS: * Handles opcode 0xb8 */ static void x86emuOp_mov_word_AX_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_EAX = fetch_long_imm(emu); else emu->x86.R_AX = fetch_word_imm(emu); } /* * REMARKS: * Handles opcode 0xb9 */ static void x86emuOp_mov_word_CX_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_ECX = fetch_long_imm(emu); else emu->x86.R_CX = fetch_word_imm(emu); } /* * REMARKS: * Handles opcode 0xba */ static void x86emuOp_mov_word_DX_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_EDX = fetch_long_imm(emu); else emu->x86.R_DX = fetch_word_imm(emu); } /* * REMARKS: * Handles opcode 0xbb */ static void x86emuOp_mov_word_BX_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_EBX = fetch_long_imm(emu); else emu->x86.R_BX = fetch_word_imm(emu); } /* * REMARKS: * Handles opcode 0xbc */ static void x86emuOp_mov_word_SP_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_ESP = fetch_long_imm(emu); else emu->x86.R_SP = fetch_word_imm(emu); } /* * REMARKS: * Handles opcode 0xbd */ static void x86emuOp_mov_word_BP_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_EBP = fetch_long_imm(emu); else emu->x86.R_BP = fetch_word_imm(emu); } /* * REMARKS: * Handles opcode 0xbe */ static void x86emuOp_mov_word_SI_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_ESI = fetch_long_imm(emu); else emu->x86.R_SI = fetch_word_imm(emu); } /* * REMARKS: * Handles opcode 0xbf */ static void x86emuOp_mov_word_DI_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) emu->x86.R_EDI = fetch_long_imm(emu); else emu->x86.R_DI = fetch_word_imm(emu); } /* used by opcodes c0, d0, and d2. */ static uint8_t(* const opcD0_byte_operation[]) (struct x86emu *, uint8_t d, uint8_t s) = { rol_byte, ror_byte, rcl_byte, rcr_byte, shl_byte, shr_byte, shl_byte, /* sal_byte === shl_byte by definition */ sar_byte, }; /* * REMARKS: * Handles opcode 0xc0 */ static void x86emuOp_opcC0_byte_RM_MEM(struct x86emu *emu) { uint8_t destval, amt; /* * Yet another weirdo special case instruction format. Part of * the opcode held below in "RH". Doubly nested case would * result, except that the decoded instruction */ fetch_decode_modrm(emu); /* know operation, decode the mod byte to find the addressing mode. */ destval = decode_and_fetch_byte_imm8(emu, &amt); destval = (*opcD0_byte_operation[emu->cur_rh]) (emu, destval, amt); write_back_byte(emu, destval); } /* used by opcodes c1, d1, and d3. */ static uint16_t(* const opcD1_word_operation[]) (struct x86emu *, uint16_t s, uint8_t d) = { rol_word, ror_word, rcl_word, rcr_word, shl_word, shr_word, shl_word, /* sal_byte === shl_byte by definition */ sar_word, }; /* used by opcodes c1, d1, and d3. */ static uint32_t(* const opcD1_long_operation[]) (struct x86emu *, uint32_t s, uint8_t d) = { rol_long, ror_long, rcl_long, rcr_long, shl_long, shr_long, shl_long, /* sal_byte === shl_byte by definition */ sar_long, }; /* * REMARKS: * Handles opcode 0xc1 */ static void x86emuOp_opcC1_word_RM_MEM(struct x86emu *emu) { uint8_t amt; /* * Yet another weirdo special case instruction format. Part of * the opcode held below in "RH". Doubly nested case would * result, except that the decoded instruction */ fetch_decode_modrm(emu); if (emu->x86.mode & SYSMODE_PREFIX_DATA) { uint32_t destval; destval = decode_and_fetch_long_imm8(emu, &amt); destval = (*opcD1_long_operation[emu->cur_rh]) (emu, destval, amt); write_back_long(emu, destval); } else { uint16_t destval; destval = decode_and_fetch_word_imm8(emu, &amt); destval = (*opcD1_word_operation[emu->cur_rh]) (emu, destval, amt); write_back_word(emu, destval); } } /* * REMARKS: * Handles opcode 0xc2 */ static void x86emuOp_ret_near_IMM(struct x86emu *emu) { uint16_t imm; imm = fetch_word_imm(emu); emu->x86.R_IP = pop_word(emu); emu->x86.R_SP += imm; } /* * REMARKS: * Handles opcode 0xc6 */ static void x86emuOp_mov_byte_RM_IMM(struct x86emu *emu) { uint8_t *destreg; uint32_t destoffset; uint8_t imm; fetch_decode_modrm(emu); if (emu->cur_rh != 0) x86emu_halt_sys(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); imm = fetch_byte_imm(emu); store_data_byte(emu, destoffset, imm); } else { destreg = decode_rl_byte_register(emu); imm = fetch_byte_imm(emu); *destreg = imm; } } /* * REMARKS: * Handles opcode 0xc7 */ static void x86emuOp32_mov_word_RM_IMM(struct x86emu *emu) { uint32_t destoffset; uint32_t imm, *destreg; fetch_decode_modrm(emu); if (emu->cur_rh != 0) x86emu_halt_sys(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); imm = fetch_long_imm(emu); store_data_long(emu, destoffset, imm); } else { destreg = decode_rl_long_register(emu); imm = fetch_long_imm(emu); *destreg = imm; } } static void x86emuOp16_mov_word_RM_IMM(struct x86emu *emu) { uint32_t destoffset; uint16_t imm, *destreg; fetch_decode_modrm(emu); if (emu->cur_rh != 0) x86emu_halt_sys(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); imm = fetch_word_imm(emu); store_data_word(emu, destoffset, imm); } else { destreg = decode_rl_word_register(emu); imm = fetch_word_imm(emu); *destreg = imm; } } static void x86emuOp_mov_word_RM_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_mov_word_RM_IMM(emu); else x86emuOp16_mov_word_RM_IMM(emu); } /* * REMARKS: * Handles opcode 0xc8 */ static void x86emuOp_enter(struct x86emu *emu) { uint16_t local, frame_pointer; uint8_t nesting; int i; local = fetch_word_imm(emu); nesting = fetch_byte_imm(emu); push_word(emu, emu->x86.R_BP); frame_pointer = emu->x86.R_SP; if (nesting > 0) { for (i = 1; i < nesting; i++) { emu->x86.R_BP -= 2; push_word(emu, fetch_word(emu, emu->x86.R_SS, emu->x86.R_BP)); } push_word(emu, frame_pointer); } emu->x86.R_BP = frame_pointer; emu->x86.R_SP = (uint16_t) (emu->x86.R_SP - local); } /* * REMARKS: * Handles opcode 0xc9 */ static void x86emuOp_leave(struct x86emu *emu) { emu->x86.R_SP = emu->x86.R_BP; emu->x86.R_BP = pop_word(emu); } /* * REMARKS: * Handles opcode 0xca */ static void x86emuOp_ret_far_IMM(struct x86emu *emu) { uint16_t imm; imm = fetch_word_imm(emu); emu->x86.R_IP = pop_word(emu); emu->x86.R_CS = pop_word(emu); emu->x86.R_SP += imm; } /* * REMARKS: * Handles opcode 0xcb */ static void x86emuOp_ret_far(struct x86emu *emu) { emu->x86.R_IP = pop_word(emu); emu->x86.R_CS = pop_word(emu); } /* * REMARKS: * Handles opcode 0xcc */ static void x86emuOp_int3(struct x86emu *emu) { x86emu_intr_dispatch(emu, 3); } /* * REMARKS: * Handles opcode 0xcd */ static void x86emuOp_int_IMM(struct x86emu *emu) { uint8_t intnum; intnum = fetch_byte_imm(emu); x86emu_intr_dispatch(emu, intnum); } /* * REMARKS: * Handles opcode 0xce */ static void x86emuOp_into(struct x86emu *emu) { if (ACCESS_FLAG(F_OF)) x86emu_intr_dispatch(emu, 4); } /* * REMARKS: * Handles opcode 0xcf */ static void x86emuOp_iret(struct x86emu *emu) { emu->x86.R_IP = pop_word(emu); emu->x86.R_CS = pop_word(emu); emu->x86.R_FLG = pop_word(emu); } /* * REMARKS: * Handles opcode 0xd0 */ static void x86emuOp_opcD0_byte_RM_1(struct x86emu *emu) { uint8_t destval; fetch_decode_modrm(emu); destval = decode_and_fetch_byte(emu); destval = (*opcD0_byte_operation[emu->cur_rh]) (emu, destval, 1); write_back_byte(emu, destval); } /* * REMARKS: * Handles opcode 0xd1 */ static void x86emuOp_opcD1_word_RM_1(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { uint32_t destval; fetch_decode_modrm(emu); destval = decode_and_fetch_long(emu); destval = (*opcD1_long_operation[emu->cur_rh])(emu, destval, 1); write_back_long(emu, destval); } else { uint16_t destval; fetch_decode_modrm(emu); destval = decode_and_fetch_word(emu); destval = (*opcD1_word_operation[emu->cur_rh])(emu, destval, 1); write_back_word(emu, destval); } } /* * REMARKS: * Handles opcode 0xd2 */ static void x86emuOp_opcD2_byte_RM_CL(struct x86emu *emu) { uint8_t destval; fetch_decode_modrm(emu); destval = decode_and_fetch_byte(emu); destval = (*opcD0_byte_operation[emu->cur_rh]) (emu, destval, emu->x86.R_CL); write_back_byte(emu, destval); } /* * REMARKS: * Handles opcode 0xd3 */ static void x86emuOp_opcD3_word_RM_CL(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { uint32_t destval; fetch_decode_modrm(emu); destval = decode_and_fetch_long(emu); destval = (*opcD1_long_operation[emu->cur_rh]) (emu, destval, emu->x86.R_CL); write_back_long(emu, destval); } else { uint16_t destval; fetch_decode_modrm(emu); destval = decode_and_fetch_word(emu); destval = (*opcD1_word_operation[emu->cur_rh]) (emu, destval, emu->x86.R_CL); write_back_word(emu, destval); } } /* * REMARKS: * Handles opcode 0xd4 */ static void x86emuOp_aam(struct x86emu *emu) { uint8_t a; a = fetch_byte_imm(emu); /* this is a stupid encoding. */ if (a != 10) { /* fix: add base decoding aam_word(uint8_t val, int base a) */ x86emu_halt_sys(emu); } /* note the type change here --- returning AL and AH in AX. */ emu->x86.R_AX = aam_word(emu, emu->x86.R_AL); } /* * REMARKS: * Handles opcode 0xd5 */ static void x86emuOp_aad(struct x86emu *emu) { uint8_t a; a = fetch_byte_imm(emu); if (a != 10) { /* fix: add base decoding aad_word(uint16_t val, int base a) */ x86emu_halt_sys(emu); } emu->x86.R_AX = aad_word(emu, emu->x86.R_AX); } /* opcode 0xd6 ILLEGAL OPCODE */ /* * REMARKS: * Handles opcode 0xd7 */ static void x86emuOp_xlat(struct x86emu *emu) { uint16_t addr; addr = (uint16_t) (emu->x86.R_BX + (uint8_t) emu->x86.R_AL); emu->x86.R_AL = fetch_data_byte(emu, addr); } /* opcode=0xd8 */ static void x86emuOp_esc_coprocess_d8(struct x86emu *emu) { } /* opcode=0xd9 */ static void x86emuOp_esc_coprocess_d9(struct x86emu *emu) { fetch_decode_modrm(emu); if (emu->cur_mod != 3) decode_rl_address(emu); } /* opcode=0xda */ static void x86emuOp_esc_coprocess_da(struct x86emu *emu) { fetch_decode_modrm(emu); if (emu->cur_mod != 3) decode_rl_address(emu); } /* opcode=0xdb */ static void x86emuOp_esc_coprocess_db(struct x86emu *emu) { fetch_decode_modrm(emu); if (emu->cur_mod != 3) decode_rl_address(emu); } /* opcode=0xdc */ static void x86emuOp_esc_coprocess_dc(struct x86emu *emu) { fetch_decode_modrm(emu); if (emu->cur_mod != 3) decode_rl_address(emu); } /* opcode=0xdd */ static void x86emuOp_esc_coprocess_dd(struct x86emu *emu) { fetch_decode_modrm(emu); if (emu->cur_mod != 3) decode_rl_address(emu); } /* opcode=0xde */ static void x86emuOp_esc_coprocess_de(struct x86emu *emu) { fetch_decode_modrm(emu); if (emu->cur_mod != 3) decode_rl_address(emu); } /* opcode=0xdf */ static void x86emuOp_esc_coprocess_df(struct x86emu *emu) { fetch_decode_modrm(emu); if (emu->cur_mod != 3) decode_rl_address(emu); } /* * REMARKS: * Handles opcode 0xe0 */ static void x86emuOp_loopne(struct x86emu *emu) { int16_t ip; ip = (int8_t) fetch_byte_imm(emu); ip += (int16_t) emu->x86.R_IP; emu->x86.R_CX -= 1; if (emu->x86.R_CX != 0 && !ACCESS_FLAG(F_ZF)) /* CX != 0 and !ZF */ emu->x86.R_IP = ip; } /* * REMARKS: * Handles opcode 0xe1 */ static void x86emuOp_loope(struct x86emu *emu) { int16_t ip; ip = (int8_t) fetch_byte_imm(emu); ip += (int16_t) emu->x86.R_IP; emu->x86.R_CX -= 1; if (emu->x86.R_CX != 0 && ACCESS_FLAG(F_ZF)) /* CX != 0 and ZF */ emu->x86.R_IP = ip; } /* * REMARKS: * Handles opcode 0xe2 */ static void x86emuOp_loop(struct x86emu *emu) { int16_t ip; ip = (int8_t) fetch_byte_imm(emu); ip += (int16_t) emu->x86.R_IP; emu->x86.R_CX -= 1; if (emu->x86.R_CX != 0) emu->x86.R_IP = ip; } /* * REMARKS: * Handles opcode 0xe3 */ static void x86emuOp_jcxz(struct x86emu *emu) { uint16_t target; int8_t offset; /* jump to byte offset if overflow flag is set */ offset = (int8_t) fetch_byte_imm(emu); target = (uint16_t) (emu->x86.R_IP + offset); if (emu->x86.R_CX == 0) emu->x86.R_IP = target; } /* * REMARKS: * Handles opcode 0xe4 */ static void x86emuOp_in_byte_AL_IMM(struct x86emu *emu) { uint8_t port; port = (uint8_t) fetch_byte_imm(emu); emu->x86.R_AL = (*emu->emu_inb) (emu, port); } /* * REMARKS: * Handles opcode 0xe5 */ static void x86emuOp_in_word_AX_IMM(struct x86emu *emu) { uint8_t port; port = (uint8_t) fetch_byte_imm(emu); if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_EAX = (*emu->emu_inl) (emu, port); } else { emu->x86.R_AX = (*emu->emu_inw) (emu, port); } } /* * REMARKS: * Handles opcode 0xe6 */ static void x86emuOp_out_byte_IMM_AL(struct x86emu *emu) { uint8_t port; port = (uint8_t) fetch_byte_imm(emu); (*emu->emu_outb) (emu, port, emu->x86.R_AL); } /* * REMARKS: * Handles opcode 0xe7 */ static void x86emuOp_out_word_IMM_AX(struct x86emu *emu) { uint8_t port; port = (uint8_t) fetch_byte_imm(emu); if (emu->x86.mode & SYSMODE_PREFIX_DATA) { (*emu->emu_outl) (emu, port, emu->x86.R_EAX); } else { (*emu->emu_outw) (emu, port, emu->x86.R_AX); } } /* * REMARKS: * Handles opcode 0xe8 */ static void x86emuOp_call_near_IMM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { int32_t ip; ip = (int32_t) fetch_long_imm(emu); ip += (int32_t) emu->x86.R_EIP; push_long(emu, emu->x86.R_EIP); emu->x86.R_EIP = ip; } else { int16_t ip; ip = (int16_t) fetch_word_imm(emu); ip += (int16_t) emu->x86.R_IP; /* CHECK SIGN */ push_word(emu, emu->x86.R_IP); emu->x86.R_IP = ip; } } /* * REMARKS: * Handles opcode 0xe9 */ static void x86emuOp_jump_near_IMM(struct x86emu *emu) { int ip; ip = (int16_t) fetch_word_imm(emu); ip += (int16_t) emu->x86.R_IP; emu->x86.R_IP = (uint16_t) ip; } /* * REMARKS: * Handles opcode 0xea */ static void x86emuOp_jump_far_IMM(struct x86emu *emu) { uint16_t cs, ip; ip = fetch_word_imm(emu); cs = fetch_word_imm(emu); emu->x86.R_IP = ip; emu->x86.R_CS = cs; } /* * REMARKS: * Handles opcode 0xeb */ static void x86emuOp_jump_byte_IMM(struct x86emu *emu) { uint16_t target; int8_t offset; offset = (int8_t) fetch_byte_imm(emu); target = (uint16_t) (emu->x86.R_IP + offset); emu->x86.R_IP = target; } /* * REMARKS: * Handles opcode 0xec */ static void x86emuOp_in_byte_AL_DX(struct x86emu *emu) { emu->x86.R_AL = (*emu->emu_inb) (emu, emu->x86.R_DX); } /* * REMARKS: * Handles opcode 0xed */ static void x86emuOp_in_word_AX_DX(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_EAX = (*emu->emu_inl) (emu, emu->x86.R_DX); } else { emu->x86.R_AX = (*emu->emu_inw) (emu, emu->x86.R_DX); } } /* * REMARKS: * Handles opcode 0xee */ static void x86emuOp_out_byte_DX_AL(struct x86emu *emu) { (*emu->emu_outb) (emu, emu->x86.R_DX, emu->x86.R_AL); } /* * REMARKS: * Handles opcode 0xef */ static void x86emuOp_out_word_DX_AX(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) { (*emu->emu_outl) (emu, emu->x86.R_DX, emu->x86.R_EAX); } else { (*emu->emu_outw) (emu, emu->x86.R_DX, emu->x86.R_AX); } } /* * REMARKS: * Handles opcode 0xf0 */ static void x86emuOp_lock(struct x86emu *emu) { } /*opcode 0xf1 ILLEGAL OPERATION */ /* * REMARKS: * Handles opcode 0xf5 */ static void x86emuOp_cmc(struct x86emu *emu) { if (ACCESS_FLAG(F_CF)) CLEAR_FLAG(F_CF); else SET_FLAG(F_CF); } /* * REMARKS: * Handles opcode 0xf6 */ static void x86emuOp_opcF6_byte_RM(struct x86emu *emu) { uint8_t destval, srcval; /* long, drawn out code follows. Double switch for a total of 32 * cases. */ fetch_decode_modrm(emu); if (emu->cur_rh == 1) x86emu_halt_sys(emu); if (emu->cur_rh == 0) { destval = decode_and_fetch_byte_imm8(emu, &srcval); test_byte(emu, destval, srcval); return; } destval = decode_and_fetch_byte(emu); switch (emu->cur_rh) { case 2: destval = ~destval; write_back_byte(emu, destval); break; case 3: destval = neg_byte(emu, destval); write_back_byte(emu, destval); break; case 4: mul_byte(emu, destval); break; case 5: imul_byte(emu, destval); break; case 6: div_byte(emu, destval); break; case 7: idiv_byte(emu, destval); break; } } /* * REMARKS: * Handles opcode 0xf7 */ static void x86emuOp32_opcF7_word_RM(struct x86emu *emu) { uint32_t destval, srcval; /* long, drawn out code follows. Double switch for a total of 32 * cases. */ fetch_decode_modrm(emu); if (emu->cur_rh == 1) x86emu_halt_sys(emu); if (emu->cur_rh == 0) { if (emu->cur_mod != 3) { uint32_t destoffset; destoffset = decode_rl_address(emu); srcval = fetch_long_imm(emu); destval = fetch_data_long(emu, destoffset); } else { srcval = fetch_long_imm(emu); destval = *decode_rl_long_register(emu); } test_long(emu, destval, srcval); return; } destval = decode_and_fetch_long(emu); switch (emu->cur_rh) { case 2: destval = ~destval; write_back_long(emu, destval); break; case 3: destval = neg_long(emu, destval); write_back_long(emu, destval); break; case 4: mul_long(emu, destval); break; case 5: imul_long(emu, destval); break; case 6: div_long(emu, destval); break; case 7: idiv_long(emu, destval); break; } } static void x86emuOp16_opcF7_word_RM(struct x86emu *emu) { uint16_t destval, srcval; /* long, drawn out code follows. Double switch for a total of 32 * cases. */ fetch_decode_modrm(emu); if (emu->cur_rh == 1) x86emu_halt_sys(emu); if (emu->cur_rh == 0) { if (emu->cur_mod != 3) { uint32_t destoffset; destoffset = decode_rl_address(emu); srcval = fetch_word_imm(emu); destval = fetch_data_word(emu, destoffset); } else { srcval = fetch_word_imm(emu); destval = *decode_rl_word_register(emu); } test_word(emu, destval, srcval); return; } destval = decode_and_fetch_word(emu); switch (emu->cur_rh) { case 2: destval = ~destval; write_back_word(emu, destval); break; case 3: destval = neg_word(emu, destval); write_back_word(emu, destval); break; case 4: mul_word(emu, destval); break; case 5: imul_word(emu, destval); break; case 6: div_word(emu, destval); break; case 7: idiv_word(emu, destval); break; } } static void x86emuOp_opcF7_word_RM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_opcF7_word_RM(emu); else x86emuOp16_opcF7_word_RM(emu); } /* * REMARKS: * Handles opcode 0xfe */ static void x86emuOp_opcFE_byte_RM(struct x86emu *emu) { uint8_t destval; uint32_t destoffset; uint8_t *destreg; /* Yet another special case instruction. */ fetch_decode_modrm(emu); if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); switch (emu->cur_rh) { case 0: /* inc word ptr ... */ destval = fetch_data_byte(emu, destoffset); destval = inc_byte(emu, destval); store_data_byte(emu, destoffset, destval); break; case 1: /* dec word ptr ... */ destval = fetch_data_byte(emu, destoffset); destval = dec_byte(emu, destval); store_data_byte(emu, destoffset, destval); break; } } else { destreg = decode_rl_byte_register(emu); switch (emu->cur_rh) { case 0: *destreg = inc_byte(emu, *destreg); break; case 1: *destreg = dec_byte(emu, *destreg); break; } } } /* * REMARKS: * Handles opcode 0xff */ static void x86emuOp32_opcFF_word_RM(struct x86emu *emu) { uint32_t destoffset = 0; uint32_t destval, *destreg; if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_long(emu, destoffset); switch (emu->cur_rh) { case 0: /* inc word ptr ... */ destval = inc_long(emu, destval); store_data_long(emu, destoffset, destval); break; case 1: /* dec word ptr ... */ destval = dec_long(emu, destval); store_data_long(emu, destoffset, destval); break; case 6: /* push word ptr ... */ push_long(emu, destval); break; } } else { destreg = decode_rl_long_register(emu); switch (emu->cur_rh) { case 0: *destreg = inc_long(emu, *destreg); break; case 1: *destreg = dec_long(emu, *destreg); break; case 6: push_long(emu, *destreg); break; } } } static void x86emuOp16_opcFF_word_RM(struct x86emu *emu) { uint32_t destoffset = 0; uint16_t *destreg; uint16_t destval; if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_word(emu, destoffset); switch (emu->cur_rh) { case 0: destval = inc_word(emu, destval); store_data_word(emu, destoffset, destval); break; case 1: /* dec word ptr ... */ destval = dec_word(emu, destval); store_data_word(emu, destoffset, destval); break; case 6: /* push word ptr ... */ push_word(emu, destval); break; } } else { destreg = decode_rl_word_register(emu); switch (emu->cur_rh) { case 0: *destreg = inc_word(emu, *destreg); break; case 1: *destreg = dec_word(emu, *destreg); break; case 6: push_word(emu, *destreg); break; } } } static void x86emuOp_opcFF_word_RM(struct x86emu *emu) { uint32_t destoffset = 0; uint16_t destval, destval2; /* Yet another special case instruction. */ fetch_decode_modrm(emu); if ((emu->cur_mod == 3 && (emu->cur_rh == 3 || emu->cur_rh == 5)) || emu->cur_rh == 7) x86emu_halt_sys(emu); if (emu->cur_rh == 0 || emu->cur_rh == 1 || emu->cur_rh == 6) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp32_opcFF_word_RM(emu); else x86emuOp16_opcFF_word_RM(emu); return; } if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); destval = fetch_data_word(emu, destoffset); switch (emu->cur_rh) { case 3: /* call far ptr ... */ destval2 = fetch_data_word(emu, destoffset + 2); push_word(emu, emu->x86.R_CS); emu->x86.R_CS = destval2; push_word(emu, emu->x86.R_IP); emu->x86.R_IP = destval; break; case 5: /* jmp far ptr ... */ destval2 = fetch_data_word(emu, destoffset + 2); emu->x86.R_IP = destval; emu->x86.R_CS = destval2; break; } } else { destval = *decode_rl_word_register(emu); } switch (emu->cur_rh) { case 2: /* call word ptr */ push_word(emu, emu->x86.R_IP); emu->x86.R_IP = destval; break; case 4: /* jmp */ emu->x86.R_IP = destval; break; } } /* * * Single byte operation code table: */ static void x86emu_exec_one_byte(struct x86emu * emu) { uint8_t op1; op1 = fetch_byte_imm(emu); switch (op1) { case 0x00: common_binop_byte_rm_r(emu, add_byte); break; case 0x01: common_binop_word_long_rm_r(emu, add_word, add_long); break; case 0x02: common_binop_byte_r_rm(emu, add_byte); break; case 0x03: common_binop_word_long_r_rm(emu, add_word, add_long); break; case 0x04: common_binop_byte_imm(emu, add_byte); break; case 0x05: common_binop_word_long_imm(emu, add_word, add_long); break; case 0x06: push_word(emu, emu->x86.R_ES); break; case 0x07: emu->x86.R_ES = pop_word(emu); break; case 0x08: common_binop_byte_rm_r(emu, or_byte); break; case 0x09: common_binop_word_long_rm_r(emu, or_word, or_long); break; case 0x0a: common_binop_byte_r_rm(emu, or_byte); break; case 0x0b: common_binop_word_long_r_rm(emu, or_word, or_long); break; case 0x0c: common_binop_byte_imm(emu, or_byte); break; case 0x0d: common_binop_word_long_imm(emu, or_word, or_long); break; case 0x0e: push_word(emu, emu->x86.R_CS); break; case 0x0f: x86emu_exec_two_byte(emu); break; case 0x10: common_binop_byte_rm_r(emu, adc_byte); break; case 0x11: common_binop_word_long_rm_r(emu, adc_word, adc_long); break; case 0x12: common_binop_byte_r_rm(emu, adc_byte); break; case 0x13: common_binop_word_long_r_rm(emu, adc_word, adc_long); break; case 0x14: common_binop_byte_imm(emu, adc_byte); break; case 0x15: common_binop_word_long_imm(emu, adc_word, adc_long); break; case 0x16: push_word(emu, emu->x86.R_SS); break; case 0x17: emu->x86.R_SS = pop_word(emu); break; case 0x18: common_binop_byte_rm_r(emu, sbb_byte); break; case 0x19: common_binop_word_long_rm_r(emu, sbb_word, sbb_long); break; case 0x1a: common_binop_byte_r_rm(emu, sbb_byte); break; case 0x1b: common_binop_word_long_r_rm(emu, sbb_word, sbb_long); break; case 0x1c: common_binop_byte_imm(emu, sbb_byte); break; case 0x1d: common_binop_word_long_imm(emu, sbb_word, sbb_long); break; case 0x1e: push_word(emu, emu->x86.R_DS); break; case 0x1f: emu->x86.R_DS = pop_word(emu); break; case 0x20: common_binop_byte_rm_r(emu, and_byte); break; case 0x21: common_binop_word_long_rm_r(emu, and_word, and_long); break; case 0x22: common_binop_byte_r_rm(emu, and_byte); break; case 0x23: common_binop_word_long_r_rm(emu, and_word, and_long); break; case 0x24: common_binop_byte_imm(emu, and_byte); break; case 0x25: common_binop_word_long_imm(emu, and_word, and_long); break; case 0x26: emu->x86.mode |= SYSMODE_SEGOVR_ES; break; case 0x27: emu->x86.R_AL = daa_byte(emu, emu->x86.R_AL); break; case 0x28: common_binop_byte_rm_r(emu, sub_byte); break; case 0x29: common_binop_word_long_rm_r(emu, sub_word, sub_long); break; case 0x2a: common_binop_byte_r_rm(emu, sub_byte); break; case 0x2b: common_binop_word_long_r_rm(emu, sub_word, sub_long); break; case 0x2c: common_binop_byte_imm(emu, sub_byte); break; case 0x2d: common_binop_word_long_imm(emu, sub_word, sub_long); break; case 0x2e: emu->x86.mode |= SYSMODE_SEGOVR_CS; break; case 0x2f: emu->x86.R_AL = das_byte(emu, emu->x86.R_AL); break; case 0x30: common_binop_byte_rm_r(emu, xor_byte); break; case 0x31: common_binop_word_long_rm_r(emu, xor_word, xor_long); break; case 0x32: common_binop_byte_r_rm(emu, xor_byte); break; case 0x33: common_binop_word_long_r_rm(emu, xor_word, xor_long); break; case 0x34: common_binop_byte_imm(emu, xor_byte); break; case 0x35: common_binop_word_long_imm(emu, xor_word, xor_long); break; case 0x36: emu->x86.mode |= SYSMODE_SEGOVR_SS; break; case 0x37: emu->x86.R_AX = aaa_word(emu, emu->x86.R_AX); break; case 0x38: common_binop_ns_byte_rm_r(emu, cmp_byte_no_return); break; case 0x39: common_binop_ns_word_long_rm_r(emu, cmp_word_no_return, cmp_long_no_return); break; case 0x3a: x86emuOp_cmp_byte_R_RM(emu); break; case 0x3b: x86emuOp_cmp_word_R_RM(emu); break; case 0x3c: x86emuOp_cmp_byte_AL_IMM(emu); break; case 0x3d: x86emuOp_cmp_word_AX_IMM(emu); break; case 0x3e: emu->x86.mode |= SYSMODE_SEGOVR_DS; break; case 0x3f: emu->x86.R_AX = aas_word(emu, emu->x86.R_AX); break; case 0x40: common_inc_word_long(emu, &emu->x86.register_a); break; case 0x41: common_inc_word_long(emu, &emu->x86.register_c); break; case 0x42: common_inc_word_long(emu, &emu->x86.register_d); break; case 0x43: common_inc_word_long(emu, &emu->x86.register_b); break; case 0x44: common_inc_word_long(emu, &emu->x86.register_sp); break; case 0x45: common_inc_word_long(emu, &emu->x86.register_bp); break; case 0x46: common_inc_word_long(emu, &emu->x86.register_si); break; case 0x47: common_inc_word_long(emu, &emu->x86.register_di); break; case 0x48: common_dec_word_long(emu, &emu->x86.register_a); break; case 0x49: common_dec_word_long(emu, &emu->x86.register_c); break; case 0x4a: common_dec_word_long(emu, &emu->x86.register_d); break; case 0x4b: common_dec_word_long(emu, &emu->x86.register_b); break; case 0x4c: common_dec_word_long(emu, &emu->x86.register_sp); break; case 0x4d: common_dec_word_long(emu, &emu->x86.register_bp); break; case 0x4e: common_dec_word_long(emu, &emu->x86.register_si); break; case 0x4f: common_dec_word_long(emu, &emu->x86.register_di); break; case 0x50: common_push_word_long(emu, &emu->x86.register_a); break; case 0x51: common_push_word_long(emu, &emu->x86.register_c); break; case 0x52: common_push_word_long(emu, &emu->x86.register_d); break; case 0x53: common_push_word_long(emu, &emu->x86.register_b); break; case 0x54: common_push_word_long(emu, &emu->x86.register_sp); break; case 0x55: common_push_word_long(emu, &emu->x86.register_bp); break; case 0x56: common_push_word_long(emu, &emu->x86.register_si); break; case 0x57: common_push_word_long(emu, &emu->x86.register_di); break; case 0x58: common_pop_word_long(emu, &emu->x86.register_a); break; case 0x59: common_pop_word_long(emu, &emu->x86.register_c); break; case 0x5a: common_pop_word_long(emu, &emu->x86.register_d); break; case 0x5b: common_pop_word_long(emu, &emu->x86.register_b); break; case 0x5c: common_pop_word_long(emu, &emu->x86.register_sp); break; case 0x5d: common_pop_word_long(emu, &emu->x86.register_bp); break; case 0x5e: common_pop_word_long(emu, &emu->x86.register_si); break; case 0x5f: common_pop_word_long(emu, &emu->x86.register_di); break; case 0x60: x86emuOp_push_all(emu); break; case 0x61: x86emuOp_pop_all(emu); break; /* 0x62 bound */ /* 0x63 arpl */ case 0x64: emu->x86.mode |= SYSMODE_SEGOVR_FS; break; case 0x65: emu->x86.mode |= SYSMODE_SEGOVR_GS; break; case 0x66: emu->x86.mode |= SYSMODE_PREFIX_DATA; break; case 0x67: emu->x86.mode |= SYSMODE_PREFIX_ADDR; break; case 0x68: x86emuOp_push_word_IMM(emu); break; case 0x69: common_imul_imm(emu, 0); break; case 0x6a: x86emuOp_push_byte_IMM(emu); break; case 0x6b: common_imul_imm(emu, 1); break; case 0x6c: ins(emu, 1); break; case 0x6d: x86emuOp_ins_word(emu); break; case 0x6e: outs(emu, 1); break; case 0x6f: x86emuOp_outs_word(emu); break; case 0x70: common_jmp_near(emu, ACCESS_FLAG(F_OF)); break; case 0x71: common_jmp_near(emu, !ACCESS_FLAG(F_OF)); break; case 0x72: common_jmp_near(emu, ACCESS_FLAG(F_CF)); break; case 0x73: common_jmp_near(emu, !ACCESS_FLAG(F_CF)); break; case 0x74: common_jmp_near(emu, ACCESS_FLAG(F_ZF)); break; case 0x75: common_jmp_near(emu, !ACCESS_FLAG(F_ZF)); break; case 0x76: common_jmp_near(emu, ACCESS_FLAG(F_CF) || ACCESS_FLAG(F_ZF)); break; case 0x77: common_jmp_near(emu, !ACCESS_FLAG(F_CF) && !ACCESS_FLAG(F_ZF)); break; case 0x78: common_jmp_near(emu, ACCESS_FLAG(F_SF)); break; case 0x79: common_jmp_near(emu, !ACCESS_FLAG(F_SF)); break; case 0x7a: common_jmp_near(emu, ACCESS_FLAG(F_PF)); break; case 0x7b: common_jmp_near(emu, !ACCESS_FLAG(F_PF)); break; case 0x7c: x86emuOp_jump_near_L(emu); break; case 0x7d: x86emuOp_jump_near_NL(emu); break; case 0x7e: x86emuOp_jump_near_LE(emu); break; case 0x7f: x86emuOp_jump_near_NLE(emu); break; case 0x80: x86emuOp_opc80_byte_RM_IMM(emu); break; case 0x81: x86emuOp_opc81_word_RM_IMM(emu); break; case 0x82: x86emuOp_opc82_byte_RM_IMM(emu); break; case 0x83: x86emuOp_opc83_word_RM_IMM(emu); break; case 0x84: common_binop_ns_byte_rm_r(emu, test_byte); break; case 0x85: common_binop_ns_word_long_rm_r(emu, test_word, test_long); break; case 0x86: x86emuOp_xchg_byte_RM_R(emu); break; case 0x87: x86emuOp_xchg_word_RM_R(emu); break; case 0x88: x86emuOp_mov_byte_RM_R(emu); break; case 0x89: x86emuOp_mov_word_RM_R(emu); break; case 0x8a: x86emuOp_mov_byte_R_RM(emu); break; case 0x8b: x86emuOp_mov_word_R_RM(emu); break; case 0x8c: x86emuOp_mov_word_RM_SR(emu); break; case 0x8d: x86emuOp_lea_word_R_M(emu); break; case 0x8e: x86emuOp_mov_word_SR_RM(emu); break; case 0x8f: x86emuOp_pop_RM(emu); break; case 0x90: /* nop */ break; case 0x91: x86emuOp_xchg_word_AX_CX(emu); break; case 0x92: x86emuOp_xchg_word_AX_DX(emu); break; case 0x93: x86emuOp_xchg_word_AX_BX(emu); break; case 0x94: x86emuOp_xchg_word_AX_SP(emu); break; case 0x95: x86emuOp_xchg_word_AX_BP(emu); break; case 0x96: x86emuOp_xchg_word_AX_SI(emu); break; case 0x97: x86emuOp_xchg_word_AX_DI(emu); break; case 0x98: x86emuOp_cbw(emu); break; case 0x99: x86emuOp_cwd(emu); break; case 0x9a: x86emuOp_call_far_IMM(emu); break; case 0x9b: /* wait */ break; case 0x9c: x86emuOp_pushf_word(emu); break; case 0x9d: x86emuOp_popf_word(emu); break; case 0x9e: x86emuOp_sahf(emu); break; case 0x9f: x86emuOp_lahf(emu); break; case 0xa0: x86emuOp_mov_AL_M_IMM(emu); break; case 0xa1: x86emuOp_mov_AX_M_IMM(emu); break; case 0xa2: x86emuOp_mov_M_AL_IMM(emu); break; case 0xa3: x86emuOp_mov_M_AX_IMM(emu); break; case 0xa4: x86emuOp_movs_byte(emu); break; case 0xa5: x86emuOp_movs_word(emu); break; case 0xa6: x86emuOp_cmps_byte(emu); break; case 0xa7: x86emuOp_cmps_word(emu); break; case 0xa8: test_byte(emu, emu->x86.R_AL, fetch_byte_imm(emu)); break; case 0xa9: x86emuOp_test_AX_IMM(emu); break; case 0xaa: x86emuOp_stos_byte(emu); break; case 0xab: x86emuOp_stos_word(emu); break; case 0xac: x86emuOp_lods_byte(emu); break; case 0xad: x86emuOp_lods_word(emu); break; case 0xae: x86emuOp_scas_byte(emu); break; case 0xaf: x86emuOp_scas_word(emu); break; case 0xb0: emu->x86.R_AL = fetch_byte_imm(emu); break; case 0xb1: emu->x86.R_CL = fetch_byte_imm(emu); break; case 0xb2: emu->x86.R_DL = fetch_byte_imm(emu); break; case 0xb3: emu->x86.R_BL = fetch_byte_imm(emu); break; case 0xb4: emu->x86.R_AH = fetch_byte_imm(emu); break; case 0xb5: emu->x86.R_CH = fetch_byte_imm(emu); break; case 0xb6: emu->x86.R_DH = fetch_byte_imm(emu); break; case 0xb7: emu->x86.R_BH = fetch_byte_imm(emu); break; case 0xb8: x86emuOp_mov_word_AX_IMM(emu); break; case 0xb9: x86emuOp_mov_word_CX_IMM(emu); break; case 0xba: x86emuOp_mov_word_DX_IMM(emu); break; case 0xbb: x86emuOp_mov_word_BX_IMM(emu); break; case 0xbc: x86emuOp_mov_word_SP_IMM(emu); break; case 0xbd: x86emuOp_mov_word_BP_IMM(emu); break; case 0xbe: x86emuOp_mov_word_SI_IMM(emu); break; case 0xbf: x86emuOp_mov_word_DI_IMM(emu); break; case 0xc0: x86emuOp_opcC0_byte_RM_MEM(emu); break; case 0xc1: x86emuOp_opcC1_word_RM_MEM(emu); break; case 0xc2: x86emuOp_ret_near_IMM(emu); break; case 0xc3: emu->x86.R_IP = pop_word(emu); break; case 0xc4: common_load_far_pointer(emu, &emu->x86.R_ES); break; case 0xc5: common_load_far_pointer(emu, &emu->x86.R_DS); break; case 0xc6: x86emuOp_mov_byte_RM_IMM(emu); break; case 0xc7: x86emuOp_mov_word_RM_IMM(emu); break; case 0xc8: x86emuOp_enter(emu); break; case 0xc9: x86emuOp_leave(emu); break; case 0xca: x86emuOp_ret_far_IMM(emu); break; case 0xcb: x86emuOp_ret_far(emu); break; case 0xcc: x86emuOp_int3(emu); break; case 0xcd: x86emuOp_int_IMM(emu); break; case 0xce: x86emuOp_into(emu); break; case 0xcf: x86emuOp_iret(emu); break; case 0xd0: x86emuOp_opcD0_byte_RM_1(emu); break; case 0xd1: x86emuOp_opcD1_word_RM_1(emu); break; case 0xd2: x86emuOp_opcD2_byte_RM_CL(emu); break; case 0xd3: x86emuOp_opcD3_word_RM_CL(emu); break; case 0xd4: x86emuOp_aam(emu); break; case 0xd5: x86emuOp_aad(emu); break; /* 0xd6 Undocumented SETALC instruction */ case 0xd7: x86emuOp_xlat(emu); break; case 0xd8: x86emuOp_esc_coprocess_d8(emu); break; case 0xd9: x86emuOp_esc_coprocess_d9(emu); break; case 0xda: x86emuOp_esc_coprocess_da(emu); break; case 0xdb: x86emuOp_esc_coprocess_db(emu); break; case 0xdc: x86emuOp_esc_coprocess_dc(emu); break; case 0xdd: x86emuOp_esc_coprocess_dd(emu); break; case 0xde: x86emuOp_esc_coprocess_de(emu); break; case 0xdf: x86emuOp_esc_coprocess_df(emu); break; case 0xe0: x86emuOp_loopne(emu); break; case 0xe1: x86emuOp_loope(emu); break; case 0xe2: x86emuOp_loop(emu); break; case 0xe3: x86emuOp_jcxz(emu); break; case 0xe4: x86emuOp_in_byte_AL_IMM(emu); break; case 0xe5: x86emuOp_in_word_AX_IMM(emu); break; case 0xe6: x86emuOp_out_byte_IMM_AL(emu); break; case 0xe7: x86emuOp_out_word_IMM_AX(emu); break; case 0xe8: x86emuOp_call_near_IMM(emu); break; case 0xe9: x86emuOp_jump_near_IMM(emu); break; case 0xea: x86emuOp_jump_far_IMM(emu); break; case 0xeb: x86emuOp_jump_byte_IMM(emu); break; case 0xec: x86emuOp_in_byte_AL_DX(emu); break; case 0xed: x86emuOp_in_word_AX_DX(emu); break; case 0xee: x86emuOp_out_byte_DX_AL(emu); break; case 0xef: x86emuOp_out_word_DX_AX(emu); break; case 0xf0: x86emuOp_lock(emu); break; case 0xf2: emu->x86.mode |= SYSMODE_PREFIX_REPNE; break; case 0xf3: emu->x86.mode |= SYSMODE_PREFIX_REPE; break; case 0xf4: x86emu_halt_sys(emu); break; case 0xf5: x86emuOp_cmc(emu); break; case 0xf6: x86emuOp_opcF6_byte_RM(emu); break; case 0xf7: x86emuOp_opcF7_word_RM(emu); break; case 0xf8: CLEAR_FLAG(F_CF); break; case 0xf9: SET_FLAG(F_CF); break; case 0xfa: CLEAR_FLAG(F_IF); break; case 0xfb: SET_FLAG(F_IF); break; case 0xfc: CLEAR_FLAG(F_DF); break; case 0xfd: SET_FLAG(F_DF); break; case 0xfe: x86emuOp_opcFE_byte_RM(emu); break; case 0xff: x86emuOp_opcFF_word_RM(emu); break; default: x86emu_halt_sys(emu); break; } if (op1 != 0x26 && op1 != 0x2e && op1 != 0x36 && op1 != 0x3e && (op1 | 3) != 0x67) emu->x86.mode &= ~SYSMODE_CLRMASK; } static void common_jmp_long(struct x86emu *emu, int cond) { int16_t target; target = (int16_t) fetch_word_imm(emu); target += (int16_t) emu->x86.R_IP; if (cond) emu->x86.R_IP = (uint16_t) target; } static void common_set_byte(struct x86emu *emu, int cond) { uint32_t destoffset; uint8_t *destreg, destval; fetch_decode_modrm(emu); destval = cond ? 0x01 : 0x00; if (emu->cur_mod != 3) { destoffset = decode_rl_address(emu); store_data_byte(emu, destoffset, destval); } else { destreg = decode_rl_byte_register(emu); *destreg = destval; } } static void common_bitstring32(struct x86emu *emu, int op) { int bit; uint32_t srcval, *shiftreg, mask; fetch_decode_modrm(emu); shiftreg = decode_rh_long_register(emu); srcval = decode_and_fetch_long_disp(emu, (int16_t) *shiftreg >> 5); bit = *shiftreg & 0x1F; mask = 0x1 << bit; CONDITIONAL_SET_FLAG(srcval & mask, F_CF); switch (op) { case 0: break; case 1: write_back_long(emu, srcval | mask); break; case 2: write_back_long(emu, srcval & ~mask); break; case 3: write_back_long(emu, srcval ^ mask); break; } } static void common_bitstring16(struct x86emu *emu, int op) { int bit; uint16_t srcval, *shiftreg, mask; fetch_decode_modrm(emu); shiftreg = decode_rh_word_register(emu); srcval = decode_and_fetch_word_disp(emu, (int16_t) *shiftreg >> 4); bit = *shiftreg & 0xF; mask = 0x1 << bit; CONDITIONAL_SET_FLAG(srcval & mask, F_CF); switch (op) { case 0: break; case 1: write_back_word(emu, srcval | mask); break; case 2: write_back_word(emu, srcval & ~mask); break; case 3: write_back_word(emu, srcval ^ mask); break; } } static void common_bitstring(struct x86emu *emu, int op) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) common_bitstring32(emu, op); else common_bitstring16(emu, op); } static void common_bitsearch32(struct x86emu *emu, int diff) { uint32_t srcval, *dstreg; fetch_decode_modrm(emu); dstreg = decode_rh_long_register(emu); srcval = decode_and_fetch_long(emu); CONDITIONAL_SET_FLAG(srcval == 0, F_ZF); for (*dstreg = 0; *dstreg < 32; *dstreg += diff) { if ((srcval >> *dstreg) & 1) break; } } static void common_bitsearch16(struct x86emu *emu, int diff) { uint16_t srcval, *dstreg; fetch_decode_modrm(emu); dstreg = decode_rh_word_register(emu); srcval = decode_and_fetch_word(emu); CONDITIONAL_SET_FLAG(srcval == 0, F_ZF); for (*dstreg = 0; *dstreg < 16; *dstreg += diff) { if ((srcval >> *dstreg) & 1) break; } } static void common_bitsearch(struct x86emu *emu, int diff) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) common_bitsearch32(emu, diff); else common_bitsearch16(emu, diff); } static void common_shift32(struct x86emu *emu, int shift_left, int use_cl) { uint8_t shift; uint32_t destval, *shiftreg; fetch_decode_modrm(emu); shiftreg = decode_rh_long_register(emu); if (use_cl) { destval = decode_and_fetch_long(emu); shift = emu->x86.R_CL; } else { destval = decode_and_fetch_long_imm8(emu, &shift); } if (shift_left) destval = shld_long(emu, destval, *shiftreg, shift); else destval = shrd_long(emu, destval, *shiftreg, shift); write_back_long(emu, destval); } static void common_shift16(struct x86emu *emu, int shift_left, int use_cl) { uint8_t shift; uint16_t destval, *shiftreg; fetch_decode_modrm(emu); shiftreg = decode_rh_word_register(emu); if (use_cl) { destval = decode_and_fetch_word(emu); shift = emu->x86.R_CL; } else { destval = decode_and_fetch_word_imm8(emu, &shift); } if (shift_left) destval = shld_word(emu, destval, *shiftreg, shift); else destval = shrd_word(emu, destval, *shiftreg, shift); write_back_word(emu, destval); } static void common_shift(struct x86emu *emu, int shift_left, int use_cl) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) common_shift32(emu, shift_left, use_cl); else common_shift16(emu, shift_left, use_cl); } /* * Implementation */ #define xorl(a,b) ((a) && !(b)) || (!(a) && (b)) /* * REMARKS: * Handles opcode 0x0f,0x31 */ static void x86emuOp2_rdtsc(struct x86emu *emu) { emu->x86.R_EAX = emu->cur_cycles & 0xffffffff; emu->x86.R_EDX = emu->cur_cycles >> 32; } /* * REMARKS: * Handles opcode 0x0f,0xa0 */ static void x86emuOp2_push_FS(struct x86emu *emu) { push_word(emu, emu->x86.R_FS); } /* * REMARKS: * Handles opcode 0x0f,0xa1 */ static void x86emuOp2_pop_FS(struct x86emu *emu) { emu->x86.R_FS = pop_word(emu); } /* * REMARKS: * Handles opcode 0x0f,0xa1 */ #if defined(__i386__) || defined(__amd64__) static void hw_cpuid(uint32_t *a, uint32_t *b, uint32_t *c, uint32_t *d) { - __asm__ __volatile__("cpuid" + __asm__ volatile("cpuid" : "=a" (*a), "=b" (*b), "=c" (*c), "=d" (*d) : "a" (*a), "c" (*c) : "cc"); } #endif static void x86emuOp2_cpuid(struct x86emu *emu) { #if defined(__i386__) || defined(__amd64__) hw_cpuid(&emu->x86.R_EAX, &emu->x86.R_EBX, &emu->x86.R_ECX, &emu->x86.R_EDX); #endif switch (emu->x86.R_EAX) { case 0: emu->x86.R_EAX = 1; #if !defined(__i386__) && !defined(__amd64__) /* "GenuineIntel" */ emu->x86.R_EBX = 0x756e6547; emu->x86.R_EDX = 0x49656e69; emu->x86.R_ECX = 0x6c65746e; #endif break; case 1: #if !defined(__i386__) && !defined(__amd64__) emu->x86.R_EAX = 0x00000480; emu->x86.R_EBX = emu->x86.R_ECX = 0; emu->x86.R_EDX = 0x00000002; #else emu->x86.R_EDX &= 0x00000012; #endif break; default: emu->x86.R_EAX = emu->x86.R_EBX = emu->x86.R_ECX = emu->x86.R_EDX = 0; break; } } /* * REMARKS: * Handles opcode 0x0f,0xa3 */ static void x86emuOp2_bt_R(struct x86emu *emu) { common_bitstring(emu, 0); } /* * REMARKS: * Handles opcode 0x0f,0xa4 */ static void x86emuOp2_shld_IMM(struct x86emu *emu) { common_shift(emu, 1, 0); } /* * REMARKS: * Handles opcode 0x0f,0xa5 */ static void x86emuOp2_shld_CL(struct x86emu *emu) { common_shift(emu, 1, 1); } /* * REMARKS: * Handles opcode 0x0f,0xa8 */ static void x86emuOp2_push_GS(struct x86emu *emu) { push_word(emu, emu->x86.R_GS); } /* * REMARKS: * Handles opcode 0x0f,0xa9 */ static void x86emuOp2_pop_GS(struct x86emu *emu) { emu->x86.R_GS = pop_word(emu); } /* * REMARKS: * Handles opcode 0x0f,0xab */ static void x86emuOp2_bts_R(struct x86emu *emu) { common_bitstring(emu, 1); } /* * REMARKS: * Handles opcode 0x0f,0xac */ static void x86emuOp2_shrd_IMM(struct x86emu *emu) { common_shift(emu, 0, 0); } /* * REMARKS: * Handles opcode 0x0f,0xad */ static void x86emuOp2_shrd_CL(struct x86emu *emu) { common_shift(emu, 0, 1); } /* * REMARKS: * Handles opcode 0x0f,0xaf */ static void x86emuOp2_32_imul_R_RM(struct x86emu *emu) { uint32_t *destreg, srcval; uint64_t res; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); srcval = decode_and_fetch_long(emu); res = (int32_t) *destreg * (int32_t)srcval; if (res > 0xffffffff) { SET_FLAG(F_CF); SET_FLAG(F_OF); } else { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } *destreg = (uint32_t) res; } static void x86emuOp2_16_imul_R_RM(struct x86emu *emu) { uint16_t *destreg, srcval; uint32_t res; fetch_decode_modrm(emu); destreg = decode_rh_word_register(emu); srcval = decode_and_fetch_word(emu); res = (int16_t) * destreg * (int16_t)srcval; if (res > 0xFFFF) { SET_FLAG(F_CF); SET_FLAG(F_OF); } else { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } *destreg = (uint16_t) res; } static void x86emuOp2_imul_R_RM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp2_32_imul_R_RM(emu); else x86emuOp2_16_imul_R_RM(emu); } /* * REMARKS: * Handles opcode 0x0f,0xb2 */ static void x86emuOp2_lss_R_IMM(struct x86emu *emu) { common_load_far_pointer(emu, &emu->x86.R_SS); } /* * REMARKS: * Handles opcode 0x0f,0xb3 */ static void x86emuOp2_btr_R(struct x86emu *emu) { common_bitstring(emu, 2); } /* * REMARKS: * Handles opcode 0x0f,0xb4 */ static void x86emuOp2_lfs_R_IMM(struct x86emu *emu) { common_load_far_pointer(emu, &emu->x86.R_FS); } /* * REMARKS: * Handles opcode 0x0f,0xb5 */ static void x86emuOp2_lgs_R_IMM(struct x86emu *emu) { common_load_far_pointer(emu, &emu->x86.R_GS); } /* * REMARKS: * Handles opcode 0x0f,0xb6 */ static void x86emuOp2_32_movzx_byte_R_RM(struct x86emu *emu) { uint32_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); *destreg = decode_and_fetch_byte(emu); } static void x86emuOp2_16_movzx_byte_R_RM(struct x86emu *emu) { uint16_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_word_register(emu); *destreg = decode_and_fetch_byte(emu); } static void x86emuOp2_movzx_byte_R_RM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp2_32_movzx_byte_R_RM(emu); else x86emuOp2_16_movzx_byte_R_RM(emu); } /* * REMARKS: * Handles opcode 0x0f,0xb7 */ static void x86emuOp2_movzx_word_R_RM(struct x86emu *emu) { uint32_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); *destreg = decode_and_fetch_word(emu); } /* * REMARKS: * Handles opcode 0x0f,0xba */ static void x86emuOp2_32_btX_I(struct x86emu *emu) { int bit; uint32_t srcval, mask; uint8_t shift; fetch_decode_modrm(emu); if (emu->cur_rh < 4) x86emu_halt_sys(emu); srcval = decode_and_fetch_long_imm8(emu, &shift); bit = shift & 0x1F; mask = (0x1 << bit); switch (emu->cur_rh) { case 5: write_back_long(emu, srcval | mask); break; case 6: write_back_long(emu, srcval & ~mask); break; case 7: write_back_long(emu, srcval ^ mask); break; } CONDITIONAL_SET_FLAG(srcval & mask, F_CF); } static void x86emuOp2_16_btX_I(struct x86emu *emu) { int bit; uint16_t srcval, mask; uint8_t shift; fetch_decode_modrm(emu); if (emu->cur_rh < 4) x86emu_halt_sys(emu); srcval = decode_and_fetch_word_imm8(emu, &shift); bit = shift & 0xF; mask = (0x1 << bit); switch (emu->cur_rh) { case 5: write_back_word(emu, srcval | mask); break; case 6: write_back_word(emu, srcval & ~mask); break; case 7: write_back_word(emu, srcval ^ mask); break; } CONDITIONAL_SET_FLAG(srcval & mask, F_CF); } static void x86emuOp2_btX_I(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp2_32_btX_I(emu); else x86emuOp2_16_btX_I(emu); } /* * REMARKS: * Handles opcode 0x0f,0xbb */ static void x86emuOp2_btc_R(struct x86emu *emu) { common_bitstring(emu, 3); } /* * REMARKS: * Handles opcode 0x0f,0xbc */ static void x86emuOp2_bsf(struct x86emu *emu) { common_bitsearch(emu, +1); } /* * REMARKS: * Handles opcode 0x0f,0xbd */ static void x86emuOp2_bsr(struct x86emu *emu) { common_bitsearch(emu, -1); } /* * REMARKS: * Handles opcode 0x0f,0xbe */ static void x86emuOp2_32_movsx_byte_R_RM(struct x86emu *emu) { uint32_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); *destreg = (int32_t)(int8_t)decode_and_fetch_byte(emu); } static void x86emuOp2_16_movsx_byte_R_RM(struct x86emu *emu) { uint16_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_word_register(emu); *destreg = (int16_t)(int8_t)decode_and_fetch_byte(emu); } static void x86emuOp2_movsx_byte_R_RM(struct x86emu *emu) { if (emu->x86.mode & SYSMODE_PREFIX_DATA) x86emuOp2_32_movsx_byte_R_RM(emu); else x86emuOp2_16_movsx_byte_R_RM(emu); } /* * REMARKS: * Handles opcode 0x0f,0xbf */ static void x86emuOp2_movsx_word_R_RM(struct x86emu *emu) { uint32_t *destreg; fetch_decode_modrm(emu); destreg = decode_rh_long_register(emu); *destreg = (int32_t)(int16_t)decode_and_fetch_word(emu); } static void x86emu_exec_two_byte(struct x86emu * emu) { uint8_t op2; op2 = fetch_byte_imm(emu); switch (op2) { /* 0x00 Group F (ring 0 PM) */ /* 0x01 Group G (ring 0 PM) */ /* 0x02 lar (ring 0 PM) */ /* 0x03 lsl (ring 0 PM) */ /* 0x05 loadall (undocumented) */ /* 0x06 clts (ring 0 PM) */ /* 0x07 loadall (undocumented) */ /* 0x08 invd (ring 0 PM) */ /* 0x09 wbinvd (ring 0 PM) */ /* 0x20 mov reg32(op2); break;creg (ring 0 PM) */ /* 0x21 mov reg32(op2); break;dreg (ring 0 PM) */ /* 0x22 mov creg(op2); break;reg32 (ring 0 PM) */ /* 0x23 mov dreg(op2); break;reg32 (ring 0 PM) */ /* 0x24 mov reg32(op2); break;treg (ring 0 PM) */ /* 0x26 mov treg(op2); break;reg32 (ring 0 PM) */ case 0x31: x86emuOp2_rdtsc(emu); break; case 0x80: common_jmp_long(emu, ACCESS_FLAG(F_OF)); break; case 0x81: common_jmp_long(emu, !ACCESS_FLAG(F_OF)); break; case 0x82: common_jmp_long(emu, ACCESS_FLAG(F_CF)); break; case 0x83: common_jmp_long(emu, !ACCESS_FLAG(F_CF)); break; case 0x84: common_jmp_long(emu, ACCESS_FLAG(F_ZF)); break; case 0x85: common_jmp_long(emu, !ACCESS_FLAG(F_ZF)); break; case 0x86: common_jmp_long(emu, ACCESS_FLAG(F_CF) || ACCESS_FLAG(F_ZF)); break; case 0x87: common_jmp_long(emu, !(ACCESS_FLAG(F_CF) || ACCESS_FLAG(F_ZF))); break; case 0x88: common_jmp_long(emu, ACCESS_FLAG(F_SF)); break; case 0x89: common_jmp_long(emu, !ACCESS_FLAG(F_SF)); break; case 0x8a: common_jmp_long(emu, ACCESS_FLAG(F_PF)); break; case 0x8b: common_jmp_long(emu, !ACCESS_FLAG(F_PF)); break; case 0x8c: common_jmp_long(emu, xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF))); break; case 0x8d: common_jmp_long(emu, !(xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF)))); break; case 0x8e: common_jmp_long(emu, (xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF)) || ACCESS_FLAG(F_ZF))); break; case 0x8f: common_jmp_long(emu, !(xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF)) || ACCESS_FLAG(F_ZF))); break; case 0x90: common_set_byte(emu, ACCESS_FLAG(F_OF)); break; case 0x91: common_set_byte(emu, !ACCESS_FLAG(F_OF)); break; case 0x92: common_set_byte(emu, ACCESS_FLAG(F_CF)); break; case 0x93: common_set_byte(emu, !ACCESS_FLAG(F_CF)); break; case 0x94: common_set_byte(emu, ACCESS_FLAG(F_ZF)); break; case 0x95: common_set_byte(emu, !ACCESS_FLAG(F_ZF)); break; case 0x96: common_set_byte(emu, ACCESS_FLAG(F_CF) || ACCESS_FLAG(F_ZF)); break; case 0x97: common_set_byte(emu, !(ACCESS_FLAG(F_CF) || ACCESS_FLAG(F_ZF))); break; case 0x98: common_set_byte(emu, ACCESS_FLAG(F_SF)); break; case 0x99: common_set_byte(emu, !ACCESS_FLAG(F_SF)); break; case 0x9a: common_set_byte(emu, ACCESS_FLAG(F_PF)); break; case 0x9b: common_set_byte(emu, !ACCESS_FLAG(F_PF)); break; case 0x9c: common_set_byte(emu, xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF))); break; case 0x9d: common_set_byte(emu, xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF))); break; case 0x9e: common_set_byte(emu, (xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF)) || ACCESS_FLAG(F_ZF))); break; case 0x9f: common_set_byte(emu, !(xorl(ACCESS_FLAG(F_SF), ACCESS_FLAG(F_OF)) || ACCESS_FLAG(F_ZF))); break; case 0xa0: x86emuOp2_push_FS(emu); break; case 0xa1: x86emuOp2_pop_FS(emu); break; case 0xa2: x86emuOp2_cpuid(emu); break; case 0xa3: x86emuOp2_bt_R(emu); break; case 0xa4: x86emuOp2_shld_IMM(emu); break; case 0xa5: x86emuOp2_shld_CL(emu); break; case 0xa8: x86emuOp2_push_GS(emu); break; case 0xa9: x86emuOp2_pop_GS(emu); break; case 0xab: x86emuOp2_bts_R(emu); break; case 0xac: x86emuOp2_shrd_IMM(emu); break; case 0xad: x86emuOp2_shrd_CL(emu); break; case 0xaf: x86emuOp2_imul_R_RM(emu); break; /* 0xb0 TODO: cmpxchg */ /* 0xb1 TODO: cmpxchg */ case 0xb2: x86emuOp2_lss_R_IMM(emu); break; case 0xb3: x86emuOp2_btr_R(emu); break; case 0xb4: x86emuOp2_lfs_R_IMM(emu); break; case 0xb5: x86emuOp2_lgs_R_IMM(emu); break; case 0xb6: x86emuOp2_movzx_byte_R_RM(emu); break; case 0xb7: x86emuOp2_movzx_word_R_RM(emu); break; case 0xba: x86emuOp2_btX_I(emu); break; case 0xbb: x86emuOp2_btc_R(emu); break; case 0xbc: x86emuOp2_bsf(emu); break; case 0xbd: x86emuOp2_bsr(emu); break; case 0xbe: x86emuOp2_movsx_byte_R_RM(emu); break; case 0xbf: x86emuOp2_movsx_word_R_RM(emu); break; /* 0xc0 TODO: xadd */ /* 0xc1 TODO: xadd */ /* 0xc8 TODO: bswap */ /* 0xc9 TODO: bswap */ /* 0xca TODO: bswap */ /* 0xcb TODO: bswap */ /* 0xcc TODO: bswap */ /* 0xcd TODO: bswap */ /* 0xce TODO: bswap */ /* 0xcf TODO: bswap */ default: x86emu_halt_sys(emu); break; } } /* * Carry Chain Calculation * * This represents a somewhat expensive calculation which is * apparently required to emulate the setting of the OF and AF flag. * The latter is not so important, but the former is. The overflow * flag is the XOR of the top two bits of the carry chain for an * addition (similar for subtraction). Since we do not want to * simulate the addition in a bitwise manner, we try to calculate the * carry chain given the two operands and the result. * * So, given the following table, which represents the addition of two * bits, we can derive a formula for the carry chain. * * a b cin r cout * 0 0 0 0 0 * 0 0 1 1 0 * 0 1 0 1 0 * 0 1 1 0 1 * 1 0 0 1 0 * 1 0 1 0 1 * 1 1 0 0 1 * 1 1 1 1 1 * * Construction of table for cout: * * ab * r \ 00 01 11 10 * |------------------ * 0 | 0 1 1 1 * 1 | 0 0 1 0 * * By inspection, one gets: cc = ab + r'(a + b) * * That represents alot of operations, but NO CHOICE.... * * Borrow Chain Calculation. * * The following table represents the subtraction of two bits, from * which we can derive a formula for the borrow chain. * * a b bin r bout * 0 0 0 0 0 * 0 0 1 1 1 * 0 1 0 1 1 * 0 1 1 0 1 * 1 0 0 1 0 * 1 0 1 0 0 * 1 1 0 0 0 * 1 1 1 1 1 * * Construction of table for cout: * * ab * r \ 00 01 11 10 * |------------------ * 0 | 0 1 0 0 * 1 | 1 1 1 0 * * By inspection, one gets: bc = a'b + r(a' + b) * */ /* * Global Variables */ static uint32_t x86emu_parity_tab[8] = { 0x96696996, 0x69969669, 0x69969669, 0x96696996, 0x69969669, 0x96696996, 0x96696996, 0x69969669, }; #define PARITY(x) (((x86emu_parity_tab[(x) / 32] >> ((x) % 32)) & 1) == 0) #define XOR2(x) (((x) ^ ((x)>>1)) & 0x1) /* * REMARKS: * Implements the AAA instruction and side effects. */ static uint16_t aaa_word(struct x86emu *emu, uint16_t d) { uint16_t res; if ((d & 0xf) > 0x9 || ACCESS_FLAG(F_AF)) { d += 0x6; d += 0x100; SET_FLAG(F_AF); SET_FLAG(F_CF); } else { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); } res = (uint16_t) (d & 0xFF0F); CLEAR_FLAG(F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); return res; } /* * REMARKS: * Implements the AAA instruction and side effects. */ static uint16_t aas_word(struct x86emu *emu, uint16_t d) { uint16_t res; if ((d & 0xf) > 0x9 || ACCESS_FLAG(F_AF)) { d -= 0x6; d -= 0x100; SET_FLAG(F_AF); SET_FLAG(F_CF); } else { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); } res = (uint16_t) (d & 0xFF0F); CLEAR_FLAG(F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); return res; } /* * REMARKS: * Implements the AAD instruction and side effects. */ static uint16_t aad_word(struct x86emu *emu, uint16_t d) { uint16_t l; uint8_t hb, lb; hb = (uint8_t) ((d >> 8) & 0xff); lb = (uint8_t) ((d & 0xff)); l = (uint16_t) ((lb + 10 * hb) & 0xFF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(l & 0x80, F_SF); CONDITIONAL_SET_FLAG(l == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(l & 0xff), F_PF); return l; } /* * REMARKS: * Implements the AAM instruction and side effects. */ static uint16_t aam_word(struct x86emu *emu, uint8_t d) { uint16_t h, l; h = (uint16_t) (d / 10); l = (uint16_t) (d % 10); l |= (uint16_t) (h << 8); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(l & 0x80, F_SF); CONDITIONAL_SET_FLAG(l == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(l & 0xff), F_PF); return l; } /* * REMARKS: * Implements the ADC instruction and side effects. */ static uint8_t adc_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint32_t res; /* all operands in native machine order */ uint32_t cc; if (ACCESS_FLAG(F_CF)) res = 1 + d + s; else res = d + s; CONDITIONAL_SET_FLAG(res & 0x100, F_CF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (s & d) | ((~res) & (s | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 6), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return (uint8_t) res; } /* * REMARKS: * Implements the ADC instruction and side effects. */ static uint16_t adc_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint32_t res; /* all operands in native machine order */ uint32_t cc; if (ACCESS_FLAG(F_CF)) res = 1 + d + s; else res = d + s; CONDITIONAL_SET_FLAG(res & 0x10000, F_CF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (s & d) | ((~res) & (s | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 14), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return (uint16_t) res; } /* * REMARKS: * Implements the ADC instruction and side effects. */ static uint32_t adc_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t lo; /* all operands in native machine order */ uint32_t hi; uint32_t res; uint32_t cc; if (ACCESS_FLAG(F_CF)) { lo = 1 + (d & 0xFFFF) + (s & 0xFFFF); res = 1 + d + s; } else { lo = (d & 0xFFFF) + (s & 0xFFFF); res = d + s; } hi = (lo >> 16) + (d >> 16) + (s >> 16); CONDITIONAL_SET_FLAG(hi & 0x10000, F_CF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (s & d) | ((~res) & (s | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 30), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the ADD instruction and side effects. */ static uint8_t add_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint32_t res; /* all operands in native machine order */ uint32_t cc; res = d + s; CONDITIONAL_SET_FLAG(res & 0x100, F_CF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (s & d) | ((~res) & (s | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 6), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return (uint8_t) res; } /* * REMARKS: * Implements the ADD instruction and side effects. */ static uint16_t add_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint32_t res; /* all operands in native machine order */ uint32_t cc; res = d + s; CONDITIONAL_SET_FLAG(res & 0x10000, F_CF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (s & d) | ((~res) & (s | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 14), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return (uint16_t) res; } /* * REMARKS: * Implements the ADD instruction and side effects. */ static uint32_t add_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t lo; /* all operands in native machine order */ uint32_t hi; uint32_t res; uint32_t cc; lo = (d & 0xFFFF) + (s & 0xFFFF); res = d + s; hi = (lo >> 16) + (d >> 16) + (s >> 16); CONDITIONAL_SET_FLAG(hi & 0x10000, F_CF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (s & d) | ((~res) & (s | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 30), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the AND instruction and side effects. */ static uint8_t and_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint8_t res; /* all operands in native machine order */ res = d & s; /* set the flags */ CLEAR_FLAG(F_OF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res), F_PF); return res; } /* * REMARKS: * Implements the AND instruction and side effects. */ static uint16_t and_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint16_t res; /* all operands in native machine order */ res = d & s; /* set the flags */ CLEAR_FLAG(F_OF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); return res; } /* * REMARKS: * Implements the AND instruction and side effects. */ static uint32_t and_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t res; /* all operands in native machine order */ res = d & s; /* set the flags */ CLEAR_FLAG(F_OF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); return res; } /* * REMARKS: * Implements the CMP instruction and side effects. */ static uint8_t cmp_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - s; CLEAR_FLAG(F_CF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x80, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 6), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return d; } static void cmp_byte_no_return(struct x86emu *emu, uint8_t d, uint8_t s) { cmp_byte(emu, d, s); } /* * REMARKS: * Implements the CMP instruction and side effects. */ static uint16_t cmp_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - s; CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x8000, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 14), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return d; } static void cmp_word_no_return(struct x86emu *emu, uint16_t d, uint16_t s) { cmp_word(emu, d, s); } /* * REMARKS: * Implements the CMP instruction and side effects. */ static uint32_t cmp_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - s; CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x80000000, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 30), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return d; } static void cmp_long_no_return(struct x86emu *emu, uint32_t d, uint32_t s) { cmp_long(emu, d, s); } /* * REMARKS: * Implements the DAA instruction and side effects. */ static uint8_t daa_byte(struct x86emu *emu, uint8_t d) { uint32_t res = d; if ((d & 0xf) > 9 || ACCESS_FLAG(F_AF)) { res += 6; SET_FLAG(F_AF); } if (res > 0x9F || ACCESS_FLAG(F_CF)) { res += 0x60; SET_FLAG(F_CF); } CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG((res & 0xFF) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); return (uint8_t) res; } /* * REMARKS: * Implements the DAS instruction and side effects. */ static uint8_t das_byte(struct x86emu *emu, uint8_t d) { if ((d & 0xf) > 9 || ACCESS_FLAG(F_AF)) { d -= 6; SET_FLAG(F_AF); } if (d > 0x9F || ACCESS_FLAG(F_CF)) { d -= 0x60; SET_FLAG(F_CF); } CONDITIONAL_SET_FLAG(d & 0x80, F_SF); CONDITIONAL_SET_FLAG(d == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(d & 0xff), F_PF); return d; } /* * REMARKS: * Implements the DEC instruction and side effects. */ static uint8_t dec_byte(struct x86emu *emu, uint8_t d) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - 1; CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ /* based on sub_byte, uses s==1. */ bc = (res & (~d | 1)) | (~d & 1); /* carry flag unchanged */ CONDITIONAL_SET_FLAG(XOR2(bc >> 6), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return (uint8_t) res; } /* * REMARKS: * Implements the DEC instruction and side effects. */ static uint16_t dec_word(struct x86emu *emu, uint16_t d) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - 1; CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ /* based on the sub_byte routine, with s==1 */ bc = (res & (~d | 1)) | (~d & 1); /* carry flag unchanged */ CONDITIONAL_SET_FLAG(XOR2(bc >> 14), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return (uint16_t) res; } /* * REMARKS: * Implements the DEC instruction and side effects. */ static uint32_t dec_long(struct x86emu *emu, uint32_t d) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - 1; CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | 1)) | (~d & 1); /* carry flag unchanged */ CONDITIONAL_SET_FLAG(XOR2(bc >> 30), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the INC instruction and side effects. */ static uint8_t inc_byte(struct x86emu *emu, uint8_t d) { uint32_t res; /* all operands in native machine order */ uint32_t cc; res = d + 1; CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = ((1 & d) | (~res)) & (1 | d); CONDITIONAL_SET_FLAG(XOR2(cc >> 6), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return (uint8_t) res; } /* * REMARKS: * Implements the INC instruction and side effects. */ static uint16_t inc_word(struct x86emu *emu, uint16_t d) { uint32_t res; /* all operands in native machine order */ uint32_t cc; res = d + 1; CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (1 & d) | ((~res) & (1 | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 14), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return (uint16_t) res; } /* * REMARKS: * Implements the INC instruction and side effects. */ static uint32_t inc_long(struct x86emu *emu, uint32_t d) { uint32_t res; /* all operands in native machine order */ uint32_t cc; res = d + 1; CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the carry chain SEE NOTE AT TOP. */ cc = (1 & d) | ((~res) & (1 | d)); CONDITIONAL_SET_FLAG(XOR2(cc >> 30), F_OF); CONDITIONAL_SET_FLAG(cc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the OR instruction and side effects. */ static uint8_t or_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint8_t res; /* all operands in native machine order */ res = d | s; CLEAR_FLAG(F_OF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res), F_PF); return res; } /* * REMARKS: * Implements the OR instruction and side effects. */ static uint16_t or_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint16_t res; /* all operands in native machine order */ res = d | s; /* set the carry flag to be bit 8 */ CLEAR_FLAG(F_OF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); return res; } /* * REMARKS: * Implements the OR instruction and side effects. */ static uint32_t or_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t res; /* all operands in native machine order */ res = d | s; /* set the carry flag to be bit 8 */ CLEAR_FLAG(F_OF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); return res; } /* * REMARKS: * Implements the OR instruction and side effects. */ static uint8_t neg_byte(struct x86emu *emu, uint8_t s) { uint8_t res; uint8_t bc; CONDITIONAL_SET_FLAG(s != 0, F_CF); res = (uint8_t) - s; CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(PARITY(res), F_PF); /* calculate the borrow chain --- modified such that d=0. * substitutiing d=0 into bc= res&(~d|s)|(~d&s); (the one used for * sub) and simplifying, since ~d=0xff..., ~d|s == 0xffff..., and * res&0xfff... == res. Similarly ~d&s == s. So the simplified * result is: */ bc = res | s; CONDITIONAL_SET_FLAG(XOR2(bc >> 6), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the OR instruction and side effects. */ static uint16_t neg_word(struct x86emu *emu, uint16_t s) { uint16_t res; uint16_t bc; CONDITIONAL_SET_FLAG(s != 0, F_CF); res = (uint16_t) - s; CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain --- modified such that d=0. * substitutiing d=0 into bc= res&(~d|s)|(~d&s); (the one used for * sub) and simplifying, since ~d=0xff..., ~d|s == 0xffff..., and * res&0xfff... == res. Similarly ~d&s == s. So the simplified * result is: */ bc = res | s; CONDITIONAL_SET_FLAG(XOR2(bc >> 14), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the OR instruction and side effects. */ static uint32_t neg_long(struct x86emu *emu, uint32_t s) { uint32_t res; uint32_t bc; CONDITIONAL_SET_FLAG(s != 0, F_CF); res = (uint32_t) - s; CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain --- modified such that d=0. * substitutiing d=0 into bc= res&(~d|s)|(~d&s); (the one used for * sub) and simplifying, since ~d=0xff..., ~d|s == 0xffff..., and * res&0xfff... == res. Similarly ~d&s == s. So the simplified * result is: */ bc = res | s; CONDITIONAL_SET_FLAG(XOR2(bc >> 30), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the RCL instruction and side effects. */ static uint8_t rcl_byte(struct x86emu *emu, uint8_t d, uint8_t s) { unsigned int res, cnt, mask, cf; /* s is the rotate distance. It varies from 0 - 8. */ /* have * * CF B_7 B_6 B_5 B_4 B_3 B_2 B_1 B_0 * * want to rotate through the carry by "s" bits. We could loop, but * that's inefficient. So the width is 9, and we split into three * parts: * * The new carry flag (was B_n) the stuff in B_n-1 .. B_0 the stuff * in B_7 .. B_n+1 * * The new rotate is done mod 9, and given this, for a rotation of n * bits (mod 9) the new carry flag is then located n bits from the MSB. * The low part is then shifted up cnt bits, and the high part is or'd * in. Using CAPS for new values, and lowercase for the original * values, this can be expressed as: * * IF n > 0 1) CF <- b_(8-n) 2) B_(7) .. B_(n) <- b_(8-(n+1)) .. b_0 * 3) B_(n-1) <- cf 4) B_(n-2) .. B_0 <- b_7 .. b_(8-(n-1)) */ res = d; if ((cnt = s % 9) != 0) { /* extract the new CARRY FLAG. */ /* CF <- b_(8-n) */ cf = (d >> (8 - cnt)) & 0x1; /* * Get the low stuff which rotated into the range B_7 .. B_cnt * B_(7) .. B_(n) <- b_(8-(n+1)) .. b_0 * note that the right hand side done by the mask. */ res = (d << cnt) & 0xff; /* * now the high stuff which rotated around into the positions * B_cnt-2 .. B_0 * B_(n-2) .. B_0 <- b_7 .. b_(8-(n-1)) * shift it downward, 7-(n-2) = 9-n positions. and mask off * the result before or'ing in. */ mask = (1 << (cnt - 1)) - 1; res |= (d >> (9 - cnt)) & mask; /* if the carry flag was set, or it in. */ if (ACCESS_FLAG(F_CF)) { /* carry flag is set */ /* B_(n-1) <- cf */ res |= 1 << (cnt - 1); } /* set the new carry flag, based on the variable "cf" */ CONDITIONAL_SET_FLAG(cf, F_CF); /* OVERFLOW is set *IFF* cnt==1, then it is the xor of CF and * the most significant bit. Blecck. */ /* parenthesized this expression since it appears to be * causing OF to be misset */ CONDITIONAL_SET_FLAG(cnt == 1 && XOR2(cf + ((res >> 6) & 0x2)), F_OF); } return (uint8_t) res; } /* * REMARKS: * Implements the RCL instruction and side effects. */ static uint16_t rcl_word(struct x86emu *emu, uint16_t d, uint8_t s) { unsigned int res, cnt, mask, cf; res = d; if ((cnt = s % 17) != 0) { cf = (d >> (16 - cnt)) & 0x1; res = (d << cnt) & 0xffff; mask = (1 << (cnt - 1)) - 1; res |= (d >> (17 - cnt)) & mask; if (ACCESS_FLAG(F_CF)) { res |= 1 << (cnt - 1); } CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG(cnt == 1 && XOR2(cf + ((res >> 14) & 0x2)), F_OF); } return (uint16_t) res; } /* * REMARKS: * Implements the RCL instruction and side effects. */ static uint32_t rcl_long(struct x86emu *emu, uint32_t d, uint8_t s) { uint32_t res, cnt, mask, cf; res = d; if ((cnt = s % 33) != 0) { cf = (d >> (32 - cnt)) & 0x1; res = (d << cnt) & 0xffffffff; mask = (1 << (cnt - 1)) - 1; res |= (d >> (33 - cnt)) & mask; if (ACCESS_FLAG(F_CF)) { /* carry flag is set */ res |= 1 << (cnt - 1); } CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG(cnt == 1 && XOR2(cf + ((res >> 30) & 0x2)), F_OF); } return res; } /* * REMARKS: * Implements the RCR instruction and side effects. */ static uint8_t rcr_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint32_t res, cnt; uint32_t mask, cf, ocf = 0; /* rotate right through carry */ /* s is the rotate distance. It varies from 0 - 8. d is the byte * object rotated. * * have * * CF B_7 B_6 B_5 B_4 B_3 B_2 B_1 B_0 * * The new rotate is done mod 9, and given this, for a rotation of n * bits (mod 9) the new carry flag is then located n bits from the LSB. * The low part is then shifted up cnt bits, and the high part is or'd * in. Using CAPS for new values, and lowercase for the original * values, this can be expressed as: * * IF n > 0 * 1) CF <- b_(n-1) * 2) B_(8-(n+1)) .. B_(0) <- b_(7) .. b_(n) * 3) B_(8-n) <- cf 4) B_(7) .. B_(8-(n-1)) <- b_(n-2) .. b_(0) */ res = d; if ((cnt = s % 9) != 0) { /* extract the new CARRY FLAG. */ /* CF <- b_(n-1) */ if (cnt == 1) { cf = d & 0x1; /* note hackery here. Access_flag(..) evaluates to * either 0 if flag not set non-zero if flag is set. * doing access_flag(..) != 0 casts that into either * 0..1 in any representation of the flags register * (i.e. packed bit array or unpacked.) */ ocf = ACCESS_FLAG(F_CF) != 0; } else cf = (d >> (cnt - 1)) & 0x1; /* B_(8-(n+1)) .. B_(0) <- b_(7) .. b_n */ /* note that the right hand side done by the mask This is * effectively done by shifting the object to the right. The * result must be masked, in case the object came in and was * treated as a negative number. Needed??? */ mask = (1 << (8 - cnt)) - 1; res = (d >> cnt) & mask; /* now the high stuff which rotated around into the positions * B_cnt-2 .. B_0 */ /* B_(7) .. B_(8-(n-1)) <- b_(n-2) .. b_(0) */ /* shift it downward, 7-(n-2) = 9-n positions. and mask off * the result before or'ing in. */ res |= (d << (9 - cnt)); /* if the carry flag was set, or it in. */ if (ACCESS_FLAG(F_CF)) { /* carry flag is set */ /* B_(8-n) <- cf */ res |= 1 << (8 - cnt); } /* set the new carry flag, based on the variable "cf" */ CONDITIONAL_SET_FLAG(cf, F_CF); /* OVERFLOW is set *IFF* cnt==1, then it is the xor of CF and * the most significant bit. Blecck. */ /* parenthesized... */ if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(ocf + ((d >> 6) & 0x2)), F_OF); } } return (uint8_t) res; } /* * REMARKS: * Implements the RCR instruction and side effects. */ static uint16_t rcr_word(struct x86emu *emu, uint16_t d, uint8_t s) { uint32_t res, cnt; uint32_t mask, cf, ocf = 0; /* rotate right through carry */ res = d; if ((cnt = s % 17) != 0) { if (cnt == 1) { cf = d & 0x1; ocf = ACCESS_FLAG(F_CF) != 0; } else cf = (d >> (cnt - 1)) & 0x1; mask = (1 << (16 - cnt)) - 1; res = (d >> cnt) & mask; res |= (d << (17 - cnt)); if (ACCESS_FLAG(F_CF)) { res |= 1 << (16 - cnt); } CONDITIONAL_SET_FLAG(cf, F_CF); if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(ocf + ((d >> 14) & 0x2)), F_OF); } } return (uint16_t) res; } /* * REMARKS: * Implements the RCR instruction and side effects. */ static uint32_t rcr_long(struct x86emu *emu, uint32_t d, uint8_t s) { uint32_t res, cnt; uint32_t mask, cf, ocf = 0; /* rotate right through carry */ res = d; if ((cnt = s % 33) != 0) { if (cnt == 1) { cf = d & 0x1; ocf = ACCESS_FLAG(F_CF) != 0; } else cf = (d >> (cnt - 1)) & 0x1; mask = (1 << (32 - cnt)) - 1; res = (d >> cnt) & mask; if (cnt != 1) res |= (d << (33 - cnt)); if (ACCESS_FLAG(F_CF)) { /* carry flag is set */ res |= 1 << (32 - cnt); } CONDITIONAL_SET_FLAG(cf, F_CF); if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(ocf + ((d >> 30) & 0x2)), F_OF); } } return res; } /* * REMARKS: * Implements the ROL instruction and side effects. */ static uint8_t rol_byte(struct x86emu *emu, uint8_t d, uint8_t s) { unsigned int res, cnt, mask; /* rotate left */ /* s is the rotate distance. It varies from 0 - 8. d is the byte * object rotated. * * have * * CF B_7 ... B_0 * * The new rotate is done mod 8. Much simpler than the "rcl" or "rcr" * operations. * * IF n > 0 1) B_(7) .. B_(n) <- b_(8-(n+1)) .. b_(0) 2) B_(n-1) .. * B_(0) <- b_(7) .. b_(8-n) */ res = d; if ((cnt = s % 8) != 0) { /* B_(7) .. B_(n) <- b_(8-(n+1)) .. b_(0) */ res = (d << cnt); /* B_(n-1) .. B_(0) <- b_(7) .. b_(8-n) */ mask = (1 << cnt) - 1; res |= (d >> (8 - cnt)) & mask; /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x1, F_CF); /* OVERFLOW is set *IFF* s==1, then it is the xor of CF and * the most significant bit. Blecck. */ CONDITIONAL_SET_FLAG(s == 1 && XOR2((res & 0x1) + ((res >> 6) & 0x2)), F_OF); } if (s != 0) { /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x1, F_CF); } return (uint8_t) res; } /* * REMARKS: * Implements the ROL instruction and side effects. */ static uint16_t rol_word(struct x86emu *emu, uint16_t d, uint8_t s) { unsigned int res, cnt, mask; res = d; if ((cnt = s % 16) != 0) { res = (d << cnt); mask = (1 << cnt) - 1; res |= (d >> (16 - cnt)) & mask; CONDITIONAL_SET_FLAG(res & 0x1, F_CF); CONDITIONAL_SET_FLAG(s == 1 && XOR2((res & 0x1) + ((res >> 14) & 0x2)), F_OF); } if (s != 0) { /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x1, F_CF); } return (uint16_t) res; } /* * REMARKS: * Implements the ROL instruction and side effects. */ static uint32_t rol_long(struct x86emu *emu, uint32_t d, uint8_t s) { uint32_t res, cnt, mask; res = d; if ((cnt = s % 32) != 0) { res = (d << cnt); mask = (1 << cnt) - 1; res |= (d >> (32 - cnt)) & mask; CONDITIONAL_SET_FLAG(res & 0x1, F_CF); CONDITIONAL_SET_FLAG(s == 1 && XOR2((res & 0x1) + ((res >> 30) & 0x2)), F_OF); } if (s != 0) { /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x1, F_CF); } return res; } /* * REMARKS: * Implements the ROR instruction and side effects. */ static uint8_t ror_byte(struct x86emu *emu, uint8_t d, uint8_t s) { unsigned int res, cnt, mask; /* rotate right */ /* s is the rotate distance. It varies from 0 - 8. d is the byte * object rotated. * * have * * B_7 ... B_0 * * The rotate is done mod 8. * * IF n > 0 1) B_(8-(n+1)) .. B_(0) <- b_(7) .. b_(n) 2) B_(7) .. * B_(8-n) <- b_(n-1) .. b_(0) */ res = d; if ((cnt = s % 8) != 0) { /* not a typo, do nada if cnt==0 */ /* B_(7) .. B_(8-n) <- b_(n-1) .. b_(0) */ res = (d << (8 - cnt)); /* B_(8-(n+1)) .. B_(0) <- b_(7) .. b_(n) */ mask = (1 << (8 - cnt)) - 1; res |= (d >> (cnt)) & mask; /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x80, F_CF); /* OVERFLOW is set *IFF* s==1, then it is the xor of the two * most significant bits. Blecck. */ CONDITIONAL_SET_FLAG(s == 1 && XOR2(res >> 6), F_OF); } else if (s != 0) { /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x80, F_CF); } return (uint8_t) res; } /* * REMARKS: * Implements the ROR instruction and side effects. */ static uint16_t ror_word(struct x86emu *emu, uint16_t d, uint8_t s) { unsigned int res, cnt, mask; res = d; if ((cnt = s % 16) != 0) { res = (d << (16 - cnt)); mask = (1 << (16 - cnt)) - 1; res |= (d >> (cnt)) & mask; CONDITIONAL_SET_FLAG(res & 0x8000, F_CF); CONDITIONAL_SET_FLAG(s == 1 && XOR2(res >> 14), F_OF); } else if (s != 0) { /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x8000, F_CF); } return (uint16_t) res; } /* * REMARKS: * Implements the ROR instruction and side effects. */ static uint32_t ror_long(struct x86emu *emu, uint32_t d, uint8_t s) { uint32_t res, cnt, mask; res = d; if ((cnt = s % 32) != 0) { res = (d << (32 - cnt)); mask = (1 << (32 - cnt)) - 1; res |= (d >> (cnt)) & mask; CONDITIONAL_SET_FLAG(res & 0x80000000, F_CF); CONDITIONAL_SET_FLAG(s == 1 && XOR2(res >> 30), F_OF); } else if (s != 0) { /* set the new carry flag, Note that it is the low order bit * of the result!!! */ CONDITIONAL_SET_FLAG(res & 0x80000000, F_CF); } return res; } /* * REMARKS: * Implements the SHL instruction and side effects. */ static uint8_t shl_byte(struct x86emu *emu, uint8_t d, uint8_t s) { unsigned int cnt, res, cf; if (s < 8) { cnt = s % 8; /* last bit shifted out goes into carry flag */ if (cnt > 0) { res = d << cnt; cf = d & (1 << (8 - cnt)); CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = (uint8_t) d; } if (cnt == 1) { /* Needs simplification. */ CONDITIONAL_SET_FLAG( (((res & 0x80) == 0x80) ^ (ACCESS_FLAG(F_CF) != 0)), /* was (emu->x86.R_FLG&F_CF)==F_CF)), */ F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CONDITIONAL_SET_FLAG((d << (s - 1)) & 0x80, F_CF); CLEAR_FLAG(F_OF); CLEAR_FLAG(F_SF); SET_FLAG(F_PF); SET_FLAG(F_ZF); } return (uint8_t) res; } /* * REMARKS: * Implements the SHL instruction and side effects. */ static uint16_t shl_word(struct x86emu *emu, uint16_t d, uint8_t s) { unsigned int cnt, res, cf; if (s < 16) { cnt = s % 16; if (cnt > 0) { res = d << cnt; cf = d & (1 << (16 - cnt)); CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = (uint16_t) d; } if (cnt == 1) { CONDITIONAL_SET_FLAG( (((res & 0x8000) == 0x8000) ^ (ACCESS_FLAG(F_CF) != 0)), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CONDITIONAL_SET_FLAG((d << (s - 1)) & 0x8000, F_CF); CLEAR_FLAG(F_OF); CLEAR_FLAG(F_SF); SET_FLAG(F_PF); SET_FLAG(F_ZF); } return (uint16_t) res; } /* * REMARKS: * Implements the SHL instruction and side effects. */ static uint32_t shl_long(struct x86emu *emu, uint32_t d, uint8_t s) { unsigned int cnt, res, cf; if (s < 32) { cnt = s % 32; if (cnt > 0) { res = d << cnt; cf = d & (1 << (32 - cnt)); CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = d; } if (cnt == 1) { CONDITIONAL_SET_FLAG((((res & 0x80000000) == 0x80000000) ^ (ACCESS_FLAG(F_CF) != 0)), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CONDITIONAL_SET_FLAG((d << (s - 1)) & 0x80000000, F_CF); CLEAR_FLAG(F_OF); CLEAR_FLAG(F_SF); SET_FLAG(F_PF); SET_FLAG(F_ZF); } return res; } /* * REMARKS: * Implements the SHR instruction and side effects. */ static uint8_t shr_byte(struct x86emu *emu, uint8_t d, uint8_t s) { unsigned int cnt, res, cf; if (s < 8) { cnt = s % 8; if (cnt > 0) { cf = d & (1 << (cnt - 1)); res = d >> cnt; CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = (uint8_t) d; } if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(res >> 6), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CONDITIONAL_SET_FLAG((d >> (s - 1)) & 0x1, F_CF); CLEAR_FLAG(F_OF); CLEAR_FLAG(F_SF); SET_FLAG(F_PF); SET_FLAG(F_ZF); } return (uint8_t) res; } /* * REMARKS: * Implements the SHR instruction and side effects. */ static uint16_t shr_word(struct x86emu *emu, uint16_t d, uint8_t s) { unsigned int cnt, res, cf; if (s < 16) { cnt = s % 16; if (cnt > 0) { cf = d & (1 << (cnt - 1)); res = d >> cnt; CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = d; } if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(res >> 14), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); SET_FLAG(F_ZF); CLEAR_FLAG(F_SF); CLEAR_FLAG(F_PF); } return (uint16_t) res; } /* * REMARKS: * Implements the SHR instruction and side effects. */ static uint32_t shr_long(struct x86emu *emu, uint32_t d, uint8_t s) { unsigned int cnt, res, cf; if (s < 32) { cnt = s % 32; if (cnt > 0) { cf = d & (1 << (cnt - 1)); res = d >> cnt; CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = d; } if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(res >> 30), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); SET_FLAG(F_ZF); CLEAR_FLAG(F_SF); CLEAR_FLAG(F_PF); } return res; } /* * REMARKS: * Implements the SAR instruction and side effects. */ static uint8_t sar_byte(struct x86emu *emu, uint8_t d, uint8_t s) { unsigned int cnt, res, cf, mask, sf; res = d; sf = d & 0x80; cnt = s % 8; if (cnt > 0 && cnt < 8) { mask = (1 << (8 - cnt)) - 1; cf = d & (1 << (cnt - 1)); res = (d >> cnt) & mask; CONDITIONAL_SET_FLAG(cf, F_CF); if (sf) { res |= ~mask; } CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); } else if (cnt >= 8) { if (sf) { res = 0xff; SET_FLAG(F_CF); CLEAR_FLAG(F_ZF); SET_FLAG(F_SF); SET_FLAG(F_PF); } else { res = 0; CLEAR_FLAG(F_CF); SET_FLAG(F_ZF); CLEAR_FLAG(F_SF); CLEAR_FLAG(F_PF); } } return (uint8_t) res; } /* * REMARKS: * Implements the SAR instruction and side effects. */ static uint16_t sar_word(struct x86emu *emu, uint16_t d, uint8_t s) { unsigned int cnt, res, cf, mask, sf; sf = d & 0x8000; cnt = s % 16; res = d; if (cnt > 0 && cnt < 16) { mask = (1 << (16 - cnt)) - 1; cf = d & (1 << (cnt - 1)); res = (d >> cnt) & mask; CONDITIONAL_SET_FLAG(cf, F_CF); if (sf) { res |= ~mask; } CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else if (cnt >= 16) { if (sf) { res = 0xffff; SET_FLAG(F_CF); CLEAR_FLAG(F_ZF); SET_FLAG(F_SF); SET_FLAG(F_PF); } else { res = 0; CLEAR_FLAG(F_CF); SET_FLAG(F_ZF); CLEAR_FLAG(F_SF); CLEAR_FLAG(F_PF); } } return (uint16_t) res; } /* * REMARKS: * Implements the SAR instruction and side effects. */ static uint32_t sar_long(struct x86emu *emu, uint32_t d, uint8_t s) { uint32_t cnt, res, cf, mask, sf; sf = d & 0x80000000; cnt = s % 32; res = d; if (cnt > 0 && cnt < 32) { mask = (1 << (32 - cnt)) - 1; cf = d & (1 << (cnt - 1)); res = (d >> cnt) & mask; CONDITIONAL_SET_FLAG(cf, F_CF); if (sf) { res |= ~mask; } CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else if (cnt >= 32) { if (sf) { res = 0xffffffff; SET_FLAG(F_CF); CLEAR_FLAG(F_ZF); SET_FLAG(F_SF); SET_FLAG(F_PF); } else { res = 0; CLEAR_FLAG(F_CF); SET_FLAG(F_ZF); CLEAR_FLAG(F_SF); CLEAR_FLAG(F_PF); } } return res; } /* * REMARKS: * Implements the SHLD instruction and side effects. */ static uint16_t shld_word(struct x86emu *emu, uint16_t d, uint16_t fill, uint8_t s) { unsigned int cnt, res, cf; if (s < 16) { cnt = s % 16; if (cnt > 0) { res = (d << cnt) | (fill >> (16 - cnt)); cf = d & (1 << (16 - cnt)); CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = d; } if (cnt == 1) { CONDITIONAL_SET_FLAG((((res & 0x8000) == 0x8000) ^ (ACCESS_FLAG(F_CF) != 0)), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CONDITIONAL_SET_FLAG((d << (s - 1)) & 0x8000, F_CF); CLEAR_FLAG(F_OF); CLEAR_FLAG(F_SF); SET_FLAG(F_PF); SET_FLAG(F_ZF); } return (uint16_t) res; } /* * REMARKS: * Implements the SHLD instruction and side effects. */ static uint32_t shld_long(struct x86emu *emu, uint32_t d, uint32_t fill, uint8_t s) { unsigned int cnt, res, cf; if (s < 32) { cnt = s % 32; if (cnt > 0) { res = (d << cnt) | (fill >> (32 - cnt)); cf = d & (1 << (32 - cnt)); CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = d; } if (cnt == 1) { CONDITIONAL_SET_FLAG((((res & 0x80000000) == 0x80000000) ^ (ACCESS_FLAG(F_CF) != 0)), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CONDITIONAL_SET_FLAG((d << (s - 1)) & 0x80000000, F_CF); CLEAR_FLAG(F_OF); CLEAR_FLAG(F_SF); SET_FLAG(F_PF); SET_FLAG(F_ZF); } return res; } /* * REMARKS: * Implements the SHRD instruction and side effects. */ static uint16_t shrd_word(struct x86emu *emu, uint16_t d, uint16_t fill, uint8_t s) { unsigned int cnt, res, cf; if (s < 16) { cnt = s % 16; if (cnt > 0) { cf = d & (1 << (cnt - 1)); res = (d >> cnt) | (fill << (16 - cnt)); CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = d; } if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(res >> 14), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); SET_FLAG(F_ZF); CLEAR_FLAG(F_SF); CLEAR_FLAG(F_PF); } return (uint16_t) res; } /* * REMARKS: * Implements the SHRD instruction and side effects. */ static uint32_t shrd_long(struct x86emu *emu, uint32_t d, uint32_t fill, uint8_t s) { unsigned int cnt, res, cf; if (s < 32) { cnt = s % 32; if (cnt > 0) { cf = d & (1 << (cnt - 1)); res = (d >> cnt) | (fill << (32 - cnt)); CONDITIONAL_SET_FLAG(cf, F_CF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); } else { res = d; } if (cnt == 1) { CONDITIONAL_SET_FLAG(XOR2(res >> 30), F_OF); } else { CLEAR_FLAG(F_OF); } } else { res = 0; CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); SET_FLAG(F_ZF); CLEAR_FLAG(F_SF); CLEAR_FLAG(F_PF); } return res; } /* * REMARKS: * Implements the SBB instruction and side effects. */ static uint8_t sbb_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; if (ACCESS_FLAG(F_CF)) res = d - s - 1; else res = d - s; CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x80, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 6), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return (uint8_t) res; } /* * REMARKS: * Implements the SBB instruction and side effects. */ static uint16_t sbb_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; if (ACCESS_FLAG(F_CF)) res = d - s - 1; else res = d - s; CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x8000, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 14), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return (uint16_t) res; } /* * REMARKS: * Implements the SBB instruction and side effects. */ static uint32_t sbb_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; if (ACCESS_FLAG(F_CF)) res = d - s - 1; else res = d - s; CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x80000000, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 30), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the SUB instruction and side effects. */ static uint8_t sub_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - s; CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG((res & 0xff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x80, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 6), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return (uint8_t) res; } /* * REMARKS: * Implements the SUB instruction and side effects. */ static uint16_t sub_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - s; CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x8000, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 14), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return (uint16_t) res; } /* * REMARKS: * Implements the SUB instruction and side effects. */ static uint32_t sub_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t res; /* all operands in native machine order */ uint32_t bc; res = d - s; CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG((res & 0xffffffff) == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* calculate the borrow chain. See note at top */ bc = (res & (~d | s)) | (~d & s); CONDITIONAL_SET_FLAG(bc & 0x80000000, F_CF); CONDITIONAL_SET_FLAG(XOR2(bc >> 30), F_OF); CONDITIONAL_SET_FLAG(bc & 0x8, F_AF); return res; } /* * REMARKS: * Implements the TEST instruction and side effects. */ static void test_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint32_t res; /* all operands in native machine order */ res = d & s; CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* AF == dont care */ CLEAR_FLAG(F_CF); } /* * REMARKS: * Implements the TEST instruction and side effects. */ static void test_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint32_t res; /* all operands in native machine order */ res = d & s; CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* AF == dont care */ CLEAR_FLAG(F_CF); } /* * REMARKS: * Implements the TEST instruction and side effects. */ static void test_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t res; /* all operands in native machine order */ res = d & s; CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); /* AF == dont care */ CLEAR_FLAG(F_CF); } /* * REMARKS: * Implements the XOR instruction and side effects. */ static uint8_t xor_byte(struct x86emu *emu, uint8_t d, uint8_t s) { uint8_t res; /* all operands in native machine order */ res = d ^ s; CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(res & 0x80, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res), F_PF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); return res; } /* * REMARKS: * Implements the XOR instruction and side effects. */ static uint16_t xor_word(struct x86emu *emu, uint16_t d, uint16_t s) { uint16_t res; /* all operands in native machine order */ res = d ^ s; CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(res & 0x8000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); return res; } /* * REMARKS: * Implements the XOR instruction and side effects. */ static uint32_t xor_long(struct x86emu *emu, uint32_t d, uint32_t s) { uint32_t res; /* all operands in native machine order */ res = d ^ s; CLEAR_FLAG(F_OF); CONDITIONAL_SET_FLAG(res & 0x80000000, F_SF); CONDITIONAL_SET_FLAG(res == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(res & 0xff), F_PF); CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); return res; } /* * REMARKS: * Implements the IMUL instruction and side effects. */ static void imul_byte(struct x86emu *emu, uint8_t s) { int16_t res = (int16_t) ((int8_t) emu->x86.R_AL * (int8_t) s); emu->x86.R_AX = res; if (((emu->x86.R_AL & 0x80) == 0 && emu->x86.R_AH == 0x00) || ((emu->x86.R_AL & 0x80) != 0 && emu->x86.R_AH == 0xFF)) { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } else { SET_FLAG(F_CF); SET_FLAG(F_OF); } } /* * REMARKS: * Implements the IMUL instruction and side effects. */ static void imul_word(struct x86emu *emu, uint16_t s) { int32_t res = (int16_t) emu->x86.R_AX * (int16_t) s; emu->x86.R_AX = (uint16_t) res; emu->x86.R_DX = (uint16_t) (res >> 16); if (((emu->x86.R_AX & 0x8000) == 0 && emu->x86.R_DX == 0x00) || ((emu->x86.R_AX & 0x8000) != 0 && emu->x86.R_DX == 0xFF)) { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } else { SET_FLAG(F_CF); SET_FLAG(F_OF); } } /* * REMARKS: * Implements the IMUL instruction and side effects. */ static void imul_long(struct x86emu *emu, uint32_t s) { int64_t res; res = (int64_t)(int32_t)emu->x86.R_EAX * (int32_t)s; emu->x86.R_EAX = (uint32_t)res; emu->x86.R_EDX = ((uint64_t)res) >> 32; if (((emu->x86.R_EAX & 0x80000000) == 0 && emu->x86.R_EDX == 0x00) || ((emu->x86.R_EAX & 0x80000000) != 0 && emu->x86.R_EDX == 0xFF)) { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } else { SET_FLAG(F_CF); SET_FLAG(F_OF); } } /* * REMARKS: * Implements the MUL instruction and side effects. */ static void mul_byte(struct x86emu *emu, uint8_t s) { uint16_t res = (uint16_t) (emu->x86.R_AL * s); emu->x86.R_AX = res; if (emu->x86.R_AH == 0) { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } else { SET_FLAG(F_CF); SET_FLAG(F_OF); } } /* * REMARKS: * Implements the MUL instruction and side effects. */ static void mul_word(struct x86emu *emu, uint16_t s) { uint32_t res = emu->x86.R_AX * s; emu->x86.R_AX = (uint16_t) res; emu->x86.R_DX = (uint16_t) (res >> 16); if (emu->x86.R_DX == 0) { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } else { SET_FLAG(F_CF); SET_FLAG(F_OF); } } /* * REMARKS: * Implements the MUL instruction and side effects. */ static void mul_long(struct x86emu *emu, uint32_t s) { uint64_t res = (uint64_t) emu->x86.R_EAX * s; emu->x86.R_EAX = (uint32_t) res; emu->x86.R_EDX = (uint32_t) (res >> 32); if (emu->x86.R_EDX == 0) { CLEAR_FLAG(F_CF); CLEAR_FLAG(F_OF); } else { SET_FLAG(F_CF); SET_FLAG(F_OF); } } /* * REMARKS: * Implements the IDIV instruction and side effects. */ static void idiv_byte(struct x86emu *emu, uint8_t s) { int32_t dvd, div, mod; dvd = (int16_t) emu->x86.R_AX; if (s == 0) { x86emu_intr_raise(emu, 8); return; } div = dvd / (int8_t) s; mod = dvd % (int8_t) s; if (div > 0x7f || div < -0x7f) { x86emu_intr_raise(emu, 8); return; } emu->x86.R_AL = (int8_t) div; emu->x86.R_AH = (int8_t) mod; } /* * REMARKS: * Implements the IDIV instruction and side effects. */ static void idiv_word(struct x86emu *emu, uint16_t s) { int32_t dvd, div, mod; dvd = (((int32_t) emu->x86.R_DX) << 16) | emu->x86.R_AX; if (s == 0) { x86emu_intr_raise(emu, 8); return; } div = dvd / (int16_t) s; mod = dvd % (int16_t) s; if (div > 0x7fff || div < -0x7fff) { x86emu_intr_raise(emu, 8); return; } CLEAR_FLAG(F_CF); CLEAR_FLAG(F_SF); CONDITIONAL_SET_FLAG(div == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(mod & 0xff), F_PF); emu->x86.R_AX = (uint16_t) div; emu->x86.R_DX = (uint16_t) mod; } /* * REMARKS: * Implements the IDIV instruction and side effects. */ static void idiv_long(struct x86emu *emu, uint32_t s) { int64_t dvd, div, mod; dvd = (((int64_t) emu->x86.R_EDX) << 32) | emu->x86.R_EAX; if (s == 0) { x86emu_intr_raise(emu, 8); return; } div = dvd / (int32_t) s; mod = dvd % (int32_t) s; if (div > 0x7fffffff || div < -0x7fffffff) { x86emu_intr_raise(emu, 8); return; } CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CLEAR_FLAG(F_SF); SET_FLAG(F_ZF); CONDITIONAL_SET_FLAG(PARITY(mod & 0xff), F_PF); emu->x86.R_EAX = (uint32_t) div; emu->x86.R_EDX = (uint32_t) mod; } /* * REMARKS: * Implements the DIV instruction and side effects. */ static void div_byte(struct x86emu *emu, uint8_t s) { uint32_t dvd, div, mod; dvd = emu->x86.R_AX; if (s == 0) { x86emu_intr_raise(emu, 8); return; } div = dvd / (uint8_t) s; mod = dvd % (uint8_t) s; if (div > 0xff) { x86emu_intr_raise(emu, 8); return; } emu->x86.R_AL = (uint8_t) div; emu->x86.R_AH = (uint8_t) mod; } /* * REMARKS: * Implements the DIV instruction and side effects. */ static void div_word(struct x86emu *emu, uint16_t s) { uint32_t dvd, div, mod; dvd = (((uint32_t) emu->x86.R_DX) << 16) | emu->x86.R_AX; if (s == 0) { x86emu_intr_raise(emu, 8); return; } div = dvd / (uint16_t) s; mod = dvd % (uint16_t) s; if (div > 0xffff) { x86emu_intr_raise(emu, 8); return; } CLEAR_FLAG(F_CF); CLEAR_FLAG(F_SF); CONDITIONAL_SET_FLAG(div == 0, F_ZF); CONDITIONAL_SET_FLAG(PARITY(mod & 0xff), F_PF); emu->x86.R_AX = (uint16_t) div; emu->x86.R_DX = (uint16_t) mod; } /* * REMARKS: * Implements the DIV instruction and side effects. */ static void div_long(struct x86emu *emu, uint32_t s) { uint64_t dvd, div, mod; dvd = (((uint64_t) emu->x86.R_EDX) << 32) | emu->x86.R_EAX; if (s == 0) { x86emu_intr_raise(emu, 8); return; } div = dvd / (uint32_t) s; mod = dvd % (uint32_t) s; if (div > 0xffffffff) { x86emu_intr_raise(emu, 8); return; } CLEAR_FLAG(F_CF); CLEAR_FLAG(F_AF); CLEAR_FLAG(F_SF); SET_FLAG(F_ZF); CONDITIONAL_SET_FLAG(PARITY(mod & 0xff), F_PF); emu->x86.R_EAX = (uint32_t) div; emu->x86.R_EDX = (uint32_t) mod; } /* * REMARKS: * Implements the IN string instruction and side effects. */ static void ins(struct x86emu *emu, int size) { int inc = size; if (ACCESS_FLAG(F_DF)) { inc = -size; } if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* in until CX is ZERO. */ uint32_t count = ((emu->x86.mode & SYSMODE_PREFIX_DATA) ? emu->x86.R_ECX : emu->x86.R_CX); switch (size) { case 1: while (count--) { store_byte(emu, emu->x86.R_ES, emu->x86.R_DI, (*emu->emu_inb) (emu, emu->x86.R_DX)); emu->x86.R_DI += inc; } break; case 2: while (count--) { store_word(emu, emu->x86.R_ES, emu->x86.R_DI, (*emu->emu_inw) (emu, emu->x86.R_DX)); emu->x86.R_DI += inc; } break; case 4: while (count--) { store_long(emu, emu->x86.R_ES, emu->x86.R_DI, (*emu->emu_inl) (emu, emu->x86.R_DX)); emu->x86.R_DI += inc; break; } } emu->x86.R_CX = 0; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_ECX = 0; } emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } else { switch (size) { case 1: store_byte(emu, emu->x86.R_ES, emu->x86.R_DI, (*emu->emu_inb) (emu, emu->x86.R_DX)); break; case 2: store_word(emu, emu->x86.R_ES, emu->x86.R_DI, (*emu->emu_inw) (emu, emu->x86.R_DX)); break; case 4: store_long(emu, emu->x86.R_ES, emu->x86.R_DI, (*emu->emu_inl) (emu, emu->x86.R_DX)); break; } emu->x86.R_DI += inc; } } /* * REMARKS: * Implements the OUT string instruction and side effects. */ static void outs(struct x86emu *emu, int size) { int inc = size; if (ACCESS_FLAG(F_DF)) { inc = -size; } if (emu->x86.mode & (SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE)) { /* dont care whether REPE or REPNE */ /* out until CX is ZERO. */ uint32_t count = ((emu->x86.mode & SYSMODE_PREFIX_DATA) ? emu->x86.R_ECX : emu->x86.R_CX); switch (size) { case 1: while (count--) { (*emu->emu_outb) (emu, emu->x86.R_DX, fetch_byte(emu, emu->x86.R_ES, emu->x86.R_SI)); emu->x86.R_SI += inc; } break; case 2: while (count--) { (*emu->emu_outw) (emu, emu->x86.R_DX, fetch_word(emu, emu->x86.R_ES, emu->x86.R_SI)); emu->x86.R_SI += inc; } break; case 4: while (count--) { (*emu->emu_outl) (emu, emu->x86.R_DX, fetch_long(emu, emu->x86.R_ES, emu->x86.R_SI)); emu->x86.R_SI += inc; break; } } emu->x86.R_CX = 0; if (emu->x86.mode & SYSMODE_PREFIX_DATA) { emu->x86.R_ECX = 0; } emu->x86.mode &= ~(SYSMODE_PREFIX_REPE | SYSMODE_PREFIX_REPNE); } else { switch (size) { case 1: (*emu->emu_outb) (emu, emu->x86.R_DX, fetch_byte(emu, emu->x86.R_ES, emu->x86.R_SI)); break; case 2: (*emu->emu_outw) (emu, emu->x86.R_DX, fetch_word(emu, emu->x86.R_ES, emu->x86.R_SI)); break; case 4: (*emu->emu_outl) (emu, emu->x86.R_DX, fetch_long(emu, emu->x86.R_ES, emu->x86.R_SI)); break; } emu->x86.R_SI += inc; } } /* * REMARKS: * Pushes a word onto the stack. * * NOTE: Do not inline this, as (*emu->emu_wrX) is already inline! */ static void push_word(struct x86emu *emu, uint16_t w) { emu->x86.R_SP -= 2; store_word(emu, emu->x86.R_SS, emu->x86.R_SP, w); } /* * REMARKS: * Pushes a long onto the stack. * * NOTE: Do not inline this, as (*emu->emu_wrX) is already inline! */ static void push_long(struct x86emu *emu, uint32_t w) { emu->x86.R_SP -= 4; store_long(emu, emu->x86.R_SS, emu->x86.R_SP, w); } /* * REMARKS: * Pops a word from the stack. * * NOTE: Do not inline this, as (*emu->emu_rdX) is already inline! */ static uint16_t pop_word(struct x86emu *emu) { uint16_t res; res = fetch_word(emu, emu->x86.R_SS, emu->x86.R_SP); emu->x86.R_SP += 2; return res; } /* * REMARKS: * Pops a long from the stack. * * NOTE: Do not inline this, as (*emu->emu_rdX) is already inline! */ static uint32_t pop_long(struct x86emu *emu) { uint32_t res; res = fetch_long(emu, emu->x86.R_SS, emu->x86.R_SP); emu->x86.R_SP += 4; return res; } Index: user/ngie/more-tests/sys/contrib/x86emu =================================================================== --- user/ngie/more-tests/sys/contrib/x86emu (revision 281503) +++ user/ngie/more-tests/sys/contrib/x86emu (revision 281504) Property changes on: user/ngie/more-tests/sys/contrib/x86emu ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys/contrib/x86emu:r281414-281503 Index: user/ngie/more-tests/sys/dev/fdt/fdt_arm64.c =================================================================== --- user/ngie/more-tests/sys/dev/fdt/fdt_arm64.c (nonexistent) +++ user/ngie/more-tests/sys/dev/fdt/fdt_arm64.c (revision 281504) @@ -0,0 +1,49 @@ +/*- + * Copyright (c) 2009-2010 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Semihalf under sponsorship from + * the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#include +__FBSDID("$FreeBSD$"); + +#include +#include +#include +#include +#include + +#include +#include +#include + +#include "ofw_bus_if.h" +#include "fdt_common.h" + +struct fdt_fixup_entry fdt_fixup_table[] = { + { NULL, NULL } +}; + Property changes on: user/ngie/more-tests/sys/dev/fdt/fdt_arm64.c ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/dev/psci/psci_arm64.S =================================================================== --- user/ngie/more-tests/sys/dev/psci/psci_arm64.S (nonexistent) +++ user/ngie/more-tests/sys/dev/psci/psci_arm64.S (revision 281504) @@ -0,0 +1,49 @@ +/*- + * Copyright (c) 2013, 2014 Robin Randhawa + * Copyright (c) 2015 The FreeBSD Foundation + * All rights reserved. + * + * This software was developed by Andrew Turner under + * sponsorship from the FreeBSD Foundation. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#include +__FBSDID("$FreeBSD$"); + +/* + * uint64_t psci_hvc_despatch(uint64_t psci_fnid, uint64_t, uint64_t, uint64_t) + */ +ENTRY(psci_hvc_despatch) + hvc #0 + ret +END(psci_hvc_despatch) + +/* + * uint64_t psci_smc_despatch(uint64_t psci_fnid, uint64_t, uint64_t, uint64_t) + */ +ENTRY(psci_smc_despatch) + smc #0 + ret +END(psci_hvc_despatch) Property changes on: user/ngie/more-tests/sys/dev/psci/psci_arm64.S ___________________________________________________________________ Added: svn:eol-style ## -0,0 +1 ## +native \ No newline at end of property Added: svn:keywords ## -0,0 +1 ## +FreeBSD=%H \ No newline at end of property Added: svn:mime-type ## -0,0 +1 ## +text/plain \ No newline at end of property Index: user/ngie/more-tests/sys/dev/usb/controller/ehci_pci.c =================================================================== --- user/ngie/more-tests/sys/dev/usb/controller/ehci_pci.c (revision 281503) +++ user/ngie/more-tests/sys/dev/usb/controller/ehci_pci.c (revision 281504) @@ -1,582 +1,582 @@ /*- * Copyright (c) 1998 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (augustss@carlstedt.se) at * Carlstedt Research & Technology. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* * USB Enhanced Host Controller Driver, a.k.a. USB 2.0 controller. * * The EHCI 1.0 spec can be found at * http://developer.intel.com/technology/usb/download/ehci-r10.pdf * and the USB 2.0 spec at * http://www.usb.org/developers/docs/usb_20.zip */ /* The low level controller code for EHCI has been split into * PCI probes and EHCI specific code. This was done to facilitate the * sharing of code between *BSD's */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "usb_if.h" #define PCI_EHCI_VENDORID_ACERLABS 0x10b9 #define PCI_EHCI_VENDORID_AMD 0x1022 #define PCI_EHCI_VENDORID_APPLE 0x106b #define PCI_EHCI_VENDORID_ATI 0x1002 #define PCI_EHCI_VENDORID_CMDTECH 0x1095 #define PCI_EHCI_VENDORID_INTEL 0x8086 #define PCI_EHCI_VENDORID_NEC 0x1033 #define PCI_EHCI_VENDORID_OPTI 0x1045 #define PCI_EHCI_VENDORID_PHILIPS 0x1131 #define PCI_EHCI_VENDORID_SIS 0x1039 #define PCI_EHCI_VENDORID_NVIDIA 0x12D2 #define PCI_EHCI_VENDORID_NVIDIA2 0x10DE #define PCI_EHCI_VENDORID_VIA 0x1106 static device_probe_t ehci_pci_probe; static device_attach_t ehci_pci_attach; static device_detach_t ehci_pci_detach; static usb_take_controller_t ehci_pci_take_controller; static const char * ehci_pci_match(device_t self) { uint32_t device_id = pci_get_devid(self); switch (device_id) { case 0x523910b9: return "ALi M5239 USB 2.0 controller"; case 0x10227463: return "AMD 8111 USB 2.0 controller"; case 0x20951022: return ("AMD CS5536 (Geode) USB 2.0 controller"); case 0x43451002: return "ATI SB200 USB 2.0 controller"; case 0x43731002: return "ATI SB400 USB 2.0 controller"; case 0x43961002: return ("AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller"); case 0x0f348086: return ("Intel BayTrail USB 2.0 controller"); case 0x1d268086: return ("Intel Patsburg USB 2.0 controller"); case 0x1d2d8086: return ("Intel Patsburg USB 2.0 controller"); case 0x1e268086: return ("Intel Panther Point USB 2.0 controller"); case 0x1e2d8086: return ("Intel Panther Point USB 2.0 controller"); case 0x1f2c8086: return ("Intel Avoton USB 2.0 controller"); case 0x25ad8086: return "Intel 6300ESB USB 2.0 controller"; case 0x24cd8086: return "Intel 82801DB/L/M (ICH4) USB 2.0 controller"; case 0x24dd8086: return "Intel 82801EB/R (ICH5) USB 2.0 controller"; case 0x265c8086: return "Intel 82801FB (ICH6) USB 2.0 controller"; case 0x268c8086: return ("Intel 63XXESB USB 2.0 controller"); case 0x27cc8086: return "Intel 82801GB/R (ICH7) USB 2.0 controller"; case 0x28368086: return "Intel 82801H (ICH8) USB 2.0 controller USB2-A"; case 0x283a8086: return "Intel 82801H (ICH8) USB 2.0 controller USB2-B"; case 0x293a8086: return "Intel 82801I (ICH9) USB 2.0 controller"; case 0x293c8086: return "Intel 82801I (ICH9) USB 2.0 controller"; case 0x3a3a8086: return "Intel 82801JI (ICH10) USB 2.0 controller USB-A"; case 0x3a3c8086: return "Intel 82801JI (ICH10) USB 2.0 controller USB-B"; case 0x3b348086: return ("Intel PCH USB 2.0 controller USB-A"); case 0x3b3c8086: return ("Intel PCH USB 2.0 controller USB-B"); case 0x8c268086: return ("Intel Lynx Point USB 2.0 controller USB-A"); case 0x8c2d8086: return ("Intel Lynx Point USB 2.0 controller USB-B"); case 0x8ca68086: return ("Intel Wildcat Point USB 2.0 controller USB-A"); case 0x8cad8086: return ("Intel Wildcat Point USB 2.0 controller USB-B"); case 0x9c268086: return ("Intel Lynx Point LP USB 2.0 controller USB"); case 0x00e01033: return ("NEC uPD 720100 USB 2.0 controller"); case 0x006810de: return "NVIDIA nForce2 USB 2.0 controller"; case 0x008810de: return "NVIDIA nForce2 Ultra 400 USB 2.0 controller"; case 0x00d810de: return "NVIDIA nForce3 USB 2.0 controller"; case 0x00e810de: return "NVIDIA nForce3 250 USB 2.0 controller"; case 0x005b10de: return "NVIDIA nForce CK804 USB 2.0 controller"; case 0x036d10de: return "NVIDIA nForce MCP55 USB 2.0 controller"; case 0x03f210de: return "NVIDIA nForce MCP61 USB 2.0 controller"; case 0x0aa610de: return "NVIDIA nForce MCP79 USB 2.0 controller"; case 0x0aa910de: return "NVIDIA nForce MCP79 USB 2.0 controller"; case 0x0aaa10de: return "NVIDIA nForce MCP79 USB 2.0 controller"; case 0x15621131: return "Philips ISP156x USB 2.0 controller"; case 0x31041106: return ("VIA VT6202 USB 2.0 controller"); default: break; } if ((pci_get_class(self) == PCIC_SERIALBUS) && (pci_get_subclass(self) == PCIS_SERIALBUS_USB) && (pci_get_progif(self) == PCI_INTERFACE_EHCI)) { return ("EHCI (generic) USB 2.0 controller"); } return (NULL); /* dunno */ } static int ehci_pci_probe(device_t self) { const char *desc = ehci_pci_match(self); if (desc) { device_set_desc(self, desc); - return (0); + return (BUS_PROBE_DEFAULT); } else { return (ENXIO); } } static void ehci_pci_ati_quirk(device_t self, uint8_t is_sb700) { device_t smbdev; uint32_t val; if (is_sb700) { /* Lookup SMBUS PCI device */ smbdev = pci_find_device(PCI_EHCI_VENDORID_ATI, 0x4385); if (smbdev == NULL) return; val = pci_get_revid(smbdev); if (val != 0x3a && val != 0x3b) return; } /* * Note: this bit is described as reserved in SB700 * Register Reference Guide. */ val = pci_read_config(self, 0x53, 1); if (!(val & 0x8)) { val |= 0x8; pci_write_config(self, 0x53, val, 1); device_printf(self, "AMD SB600/700 quirk applied\n"); } } static void ehci_pci_via_quirk(device_t self) { uint32_t val; if ((pci_get_device(self) == 0x3104) && ((pci_get_revid(self) & 0xf0) == 0x60)) { /* Correct schedule sleep time to 10us */ val = pci_read_config(self, 0x4b, 1); if (val & 0x20) return; val |= 0x20; pci_write_config(self, 0x4b, val, 1); device_printf(self, "VIA-quirk applied\n"); } } static int ehci_pci_attach(device_t self) { ehci_softc_t *sc = device_get_softc(self); int err; int rid; /* initialise some bus fields */ sc->sc_bus.parent = self; sc->sc_bus.devices = sc->sc_devices; sc->sc_bus.devices_max = EHCI_MAX_DEVICES; sc->sc_bus.dma_bits = 32; /* get all DMA memory */ if (usb_bus_mem_alloc_all(&sc->sc_bus, USB_GET_DMA_TAG(self), &ehci_iterate_hw_softc)) { return (ENOMEM); } pci_enable_busmaster(self); switch (pci_read_config(self, PCI_USBREV, 1) & PCI_USB_REV_MASK) { case PCI_USB_REV_PRE_1_0: case PCI_USB_REV_1_0: case PCI_USB_REV_1_1: /* * NOTE: some EHCI USB controllers have the wrong USB * revision number. It appears those controllers are * fully compliant so we just ignore this value in * some common cases. */ device_printf(self, "pre-2.0 USB revision (ignored)\n"); /* fallthrough */ case PCI_USB_REV_2_0: break; default: /* Quirk for Parallels Desktop 4.0 */ device_printf(self, "USB revision is unknown. Assuming v2.0.\n"); break; } rid = PCI_CBMEM; sc->sc_io_res = bus_alloc_resource_any(self, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (!sc->sc_io_res) { device_printf(self, "Could not map memory\n"); goto error; } sc->sc_io_tag = rman_get_bustag(sc->sc_io_res); sc->sc_io_hdl = rman_get_bushandle(sc->sc_io_res); sc->sc_io_size = rman_get_size(sc->sc_io_res); rid = 0; sc->sc_irq_res = bus_alloc_resource_any(self, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (sc->sc_irq_res == NULL) { device_printf(self, "Could not allocate irq\n"); goto error; } sc->sc_bus.bdev = device_add_child(self, "usbus", -1); if (!sc->sc_bus.bdev) { device_printf(self, "Could not add USB device\n"); goto error; } device_set_ivars(sc->sc_bus.bdev, &sc->sc_bus); /* * ehci_pci_match will never return NULL if ehci_pci_probe * succeeded */ device_set_desc(sc->sc_bus.bdev, ehci_pci_match(self)); switch (pci_get_vendor(self)) { case PCI_EHCI_VENDORID_ACERLABS: sprintf(sc->sc_vendor, "AcerLabs"); break; case PCI_EHCI_VENDORID_AMD: sprintf(sc->sc_vendor, "AMD"); break; case PCI_EHCI_VENDORID_APPLE: sprintf(sc->sc_vendor, "Apple"); break; case PCI_EHCI_VENDORID_ATI: sprintf(sc->sc_vendor, "ATI"); break; case PCI_EHCI_VENDORID_CMDTECH: sprintf(sc->sc_vendor, "CMDTECH"); break; case PCI_EHCI_VENDORID_INTEL: sprintf(sc->sc_vendor, "Intel"); break; case PCI_EHCI_VENDORID_NEC: sprintf(sc->sc_vendor, "NEC"); break; case PCI_EHCI_VENDORID_OPTI: sprintf(sc->sc_vendor, "OPTi"); break; case PCI_EHCI_VENDORID_PHILIPS: sprintf(sc->sc_vendor, "Philips"); break; case PCI_EHCI_VENDORID_SIS: sprintf(sc->sc_vendor, "SiS"); break; case PCI_EHCI_VENDORID_NVIDIA: case PCI_EHCI_VENDORID_NVIDIA2: sprintf(sc->sc_vendor, "nVidia"); break; case PCI_EHCI_VENDORID_VIA: sprintf(sc->sc_vendor, "VIA"); break; default: if (bootverbose) device_printf(self, "(New EHCI DeviceId=0x%08x)\n", pci_get_devid(self)); sprintf(sc->sc_vendor, "(0x%04x)", pci_get_vendor(self)); } #if (__FreeBSD_version >= 700031) err = bus_setup_intr(self, sc->sc_irq_res, INTR_TYPE_BIO | INTR_MPSAFE, NULL, (driver_intr_t *)ehci_interrupt, sc, &sc->sc_intr_hdl); #else err = bus_setup_intr(self, sc->sc_irq_res, INTR_TYPE_BIO | INTR_MPSAFE, (driver_intr_t *)ehci_interrupt, sc, &sc->sc_intr_hdl); #endif if (err) { device_printf(self, "Could not setup irq, %d\n", err); sc->sc_intr_hdl = NULL; goto error; } ehci_pci_take_controller(self); /* Undocumented quirks taken from Linux */ switch (pci_get_vendor(self)) { case PCI_EHCI_VENDORID_ATI: /* SB600 and SB700 EHCI quirk */ switch (pci_get_device(self)) { case 0x4386: ehci_pci_ati_quirk(self, 0); break; case 0x4396: ehci_pci_ati_quirk(self, 1); break; default: break; } break; case PCI_EHCI_VENDORID_VIA: ehci_pci_via_quirk(self); break; default: break; } /* Dropped interrupts workaround */ switch (pci_get_vendor(self)) { case PCI_EHCI_VENDORID_ATI: case PCI_EHCI_VENDORID_VIA: sc->sc_flags |= EHCI_SCFLG_LOSTINTRBUG; if (bootverbose) device_printf(self, "Dropped interrupts workaround enabled\n"); break; default: break; } /* Doorbell feature workaround */ switch (pci_get_vendor(self)) { case PCI_EHCI_VENDORID_NVIDIA: case PCI_EHCI_VENDORID_NVIDIA2: sc->sc_flags |= EHCI_SCFLG_IAADBUG; if (bootverbose) device_printf(self, "Doorbell workaround enabled\n"); break; default: break; } err = ehci_init(sc); if (!err) { err = device_probe_and_attach(sc->sc_bus.bdev); } if (err) { device_printf(self, "USB init failed err=%d\n", err); goto error; } return (0); error: ehci_pci_detach(self); return (ENXIO); } static int ehci_pci_detach(device_t self) { ehci_softc_t *sc = device_get_softc(self); device_t bdev; if (sc->sc_bus.bdev) { bdev = sc->sc_bus.bdev; device_detach(bdev); device_delete_child(self, bdev); } /* during module unload there are lots of children leftover */ device_delete_children(self); pci_disable_busmaster(self); if (sc->sc_irq_res && sc->sc_intr_hdl) { /* * only call ehci_detach() after ehci_init() */ ehci_detach(sc); int err = bus_teardown_intr(self, sc->sc_irq_res, sc->sc_intr_hdl); if (err) /* XXX or should we panic? */ device_printf(self, "Could not tear down irq, %d\n", err); sc->sc_intr_hdl = NULL; } if (sc->sc_irq_res) { bus_release_resource(self, SYS_RES_IRQ, 0, sc->sc_irq_res); sc->sc_irq_res = NULL; } if (sc->sc_io_res) { bus_release_resource(self, SYS_RES_MEMORY, PCI_CBMEM, sc->sc_io_res); sc->sc_io_res = NULL; } usb_bus_mem_free_all(&sc->sc_bus, &ehci_iterate_hw_softc); return (0); } static int ehci_pci_take_controller(device_t self) { ehci_softc_t *sc = device_get_softc(self); uint32_t cparams; uint32_t eec; uint16_t to; uint8_t eecp; uint8_t bios_sem; cparams = EREAD4(sc, EHCI_HCCPARAMS); /* Synchronise with the BIOS if it owns the controller. */ for (eecp = EHCI_HCC_EECP(cparams); eecp != 0; eecp = EHCI_EECP_NEXT(eec)) { eec = pci_read_config(self, eecp, 4); if (EHCI_EECP_ID(eec) != EHCI_EC_LEGSUP) { continue; } bios_sem = pci_read_config(self, eecp + EHCI_LEGSUP_BIOS_SEM, 1); if (bios_sem == 0) { continue; } device_printf(sc->sc_bus.bdev, "waiting for BIOS " "to give up control\n"); pci_write_config(self, eecp + EHCI_LEGSUP_OS_SEM, 1, 1); to = 500; while (1) { bios_sem = pci_read_config(self, eecp + EHCI_LEGSUP_BIOS_SEM, 1); if (bios_sem == 0) break; if (--to == 0) { device_printf(sc->sc_bus.bdev, "timed out waiting for BIOS\n"); break; } usb_pause_mtx(NULL, hz / 100); /* wait 10ms */ } } return (0); } static device_method_t ehci_pci_methods[] = { /* Device interface */ DEVMETHOD(device_probe, ehci_pci_probe), DEVMETHOD(device_attach, ehci_pci_attach), DEVMETHOD(device_detach, ehci_pci_detach), DEVMETHOD(device_suspend, bus_generic_suspend), DEVMETHOD(device_resume, bus_generic_resume), DEVMETHOD(device_shutdown, bus_generic_shutdown), DEVMETHOD(usb_take_controller, ehci_pci_take_controller), DEVMETHOD_END }; static driver_t ehci_driver = { .name = "ehci", .methods = ehci_pci_methods, .size = sizeof(struct ehci_softc), }; static devclass_t ehci_devclass; DRIVER_MODULE(ehci, pci, ehci_driver, ehci_devclass, 0, 0); MODULE_DEPEND(ehci, usb, 1, 1, 1); Index: user/ngie/more-tests/sys/dev/usb/controller/uhci_pci.c =================================================================== --- user/ngie/more-tests/sys/dev/usb/controller/uhci_pci.c (revision 281503) +++ user/ngie/more-tests/sys/dev/usb/controller/uhci_pci.c (revision 281504) @@ -1,453 +1,453 @@ /*- * Copyright (c) 1998 The NetBSD Foundation, Inc. * All rights reserved. * * This code is derived from software contributed to The NetBSD Foundation * by Lennart Augustsson (augustss@carlstedt.se) at * Carlstedt Research & Technology. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE NETBSD FOUNDATION, INC. AND CONTRIBUTORS * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED * TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR * PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION OR CONTRIBUTORS * BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE * POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* Universal Host Controller Interface * * UHCI spec: http://www.intel.com/ */ /* The low level controller code for UHCI has been split into * PCI probes and UHCI specific code. This was done to facilitate the * sharing of code between *BSD's */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "usb_if.h" #define PCI_UHCI_VENDORID_INTEL 0x8086 #define PCI_UHCI_VENDORID_VIA 0x1106 /* PIIX4E has no separate stepping */ static device_probe_t uhci_pci_probe; static device_attach_t uhci_pci_attach; static device_detach_t uhci_pci_detach; static usb_take_controller_t uhci_pci_take_controller; static int uhci_pci_take_controller(device_t self) { pci_write_config(self, PCI_LEGSUP, PCI_LEGSUP_USBPIRQDEN, 2); return (0); } static const char * uhci_pci_match(device_t self) { uint32_t device_id = pci_get_devid(self); switch (device_id) { case 0x26888086: return ("Intel 631XESB/632XESB/3100 USB controller USB-1"); case 0x26898086: return ("Intel 631XESB/632XESB/3100 USB controller USB-2"); case 0x268a8086: return ("Intel 631XESB/632XESB/3100 USB controller USB-3"); case 0x268b8086: return ("Intel 631XESB/632XESB/3100 USB controller USB-4"); case 0x70208086: return ("Intel 82371SB (PIIX3) USB controller"); case 0x71128086: return ("Intel 82371AB/EB (PIIX4) USB controller"); case 0x24128086: return ("Intel 82801AA (ICH) USB controller"); case 0x24228086: return ("Intel 82801AB (ICH0) USB controller"); case 0x24428086: return ("Intel 82801BA/BAM (ICH2) USB controller USB-A"); case 0x24448086: return ("Intel 82801BA/BAM (ICH2) USB controller USB-B"); case 0x24828086: return ("Intel 82801CA/CAM (ICH3) USB controller USB-A"); case 0x24848086: return ("Intel 82801CA/CAM (ICH3) USB controller USB-B"); case 0x24878086: return ("Intel 82801CA/CAM (ICH3) USB controller USB-C"); case 0x24c28086: return ("Intel 82801DB (ICH4) USB controller USB-A"); case 0x24c48086: return ("Intel 82801DB (ICH4) USB controller USB-B"); case 0x24c78086: return ("Intel 82801DB (ICH4) USB controller USB-C"); case 0x24d28086: return ("Intel 82801EB (ICH5) USB controller USB-A"); case 0x24d48086: return ("Intel 82801EB (ICH5) USB controller USB-B"); case 0x24d78086: return ("Intel 82801EB (ICH5) USB controller USB-C"); case 0x24de8086: return ("Intel 82801EB (ICH5) USB controller USB-D"); case 0x26588086: return ("Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-A"); case 0x26598086: return ("Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-B"); case 0x265a8086: return ("Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-C"); case 0x265b8086: return ("Intel 82801FB/FR/FW/FRW (ICH6) USB controller USB-D"); case 0x27c88086: return ("Intel 82801G (ICH7) USB controller USB-A"); case 0x27c98086: return ("Intel 82801G (ICH7) USB controller USB-B"); case 0x27ca8086: return ("Intel 82801G (ICH7) USB controller USB-C"); case 0x27cb8086: return ("Intel 82801G (ICH7) USB controller USB-D"); case 0x28308086: return ("Intel 82801H (ICH8) USB controller USB-A"); case 0x28318086: return ("Intel 82801H (ICH8) USB controller USB-B"); case 0x28328086: return ("Intel 82801H (ICH8) USB controller USB-C"); case 0x28348086: return ("Intel 82801H (ICH8) USB controller USB-D"); case 0x28358086: return ("Intel 82801H (ICH8) USB controller USB-E"); case 0x29348086: return ("Intel 82801I (ICH9) USB controller"); case 0x29358086: return ("Intel 82801I (ICH9) USB controller"); case 0x29368086: return ("Intel 82801I (ICH9) USB controller"); case 0x29378086: return ("Intel 82801I (ICH9) USB controller"); case 0x29388086: return ("Intel 82801I (ICH9) USB controller"); case 0x29398086: return ("Intel 82801I (ICH9) USB controller"); case 0x3a348086: return ("Intel 82801JI (ICH10) USB controller USB-A"); case 0x3a358086: return ("Intel 82801JI (ICH10) USB controller USB-B"); case 0x3a368086: return ("Intel 82801JI (ICH10) USB controller USB-C"); case 0x3a378086: return ("Intel 82801JI (ICH10) USB controller USB-D"); case 0x3a388086: return ("Intel 82801JI (ICH10) USB controller USB-E"); case 0x3a398086: return ("Intel 82801JI (ICH10) USB controller USB-F"); case 0x719a8086: return ("Intel 82443MX USB controller"); case 0x76028086: return ("Intel 82372FB/82468GX USB controller"); case 0x30381106: return ("VIA 83C572 USB controller"); default: break; } if ((pci_get_class(self) == PCIC_SERIALBUS) && (pci_get_subclass(self) == PCIS_SERIALBUS_USB) && (pci_get_progif(self) == PCI_INTERFACE_UHCI)) { return ("UHCI (generic) USB controller"); } return (NULL); } static int uhci_pci_probe(device_t self) { const char *desc = uhci_pci_match(self); if (desc) { device_set_desc(self, desc); - return (0); + return (BUS_PROBE_DEFAULT); } else { return (ENXIO); } } static int uhci_pci_attach(device_t self) { uhci_softc_t *sc = device_get_softc(self); int rid; int err; /* initialise some bus fields */ sc->sc_bus.parent = self; sc->sc_bus.devices = sc->sc_devices; sc->sc_bus.devices_max = UHCI_MAX_DEVICES; sc->sc_bus.dma_bits = 32; /* get all DMA memory */ if (usb_bus_mem_alloc_all(&sc->sc_bus, USB_GET_DMA_TAG(self), &uhci_iterate_hw_softc)) { return ENOMEM; } sc->sc_dev = self; pci_enable_busmaster(self); rid = PCI_UHCI_BASE_REG; sc->sc_io_res = bus_alloc_resource_any(self, SYS_RES_IOPORT, &rid, RF_ACTIVE); if (!sc->sc_io_res) { device_printf(self, "Could not map ports\n"); goto error; } sc->sc_io_tag = rman_get_bustag(sc->sc_io_res); sc->sc_io_hdl = rman_get_bushandle(sc->sc_io_res); sc->sc_io_size = rman_get_size(sc->sc_io_res); /* disable interrupts */ bus_space_write_2(sc->sc_io_tag, sc->sc_io_hdl, UHCI_INTR, 0); rid = 0; sc->sc_irq_res = bus_alloc_resource_any(self, SYS_RES_IRQ, &rid, RF_SHAREABLE | RF_ACTIVE); if (sc->sc_irq_res == NULL) { device_printf(self, "Could not allocate irq\n"); goto error; } sc->sc_bus.bdev = device_add_child(self, "usbus", -1); if (!sc->sc_bus.bdev) { device_printf(self, "Could not add USB device\n"); goto error; } device_set_ivars(sc->sc_bus.bdev, &sc->sc_bus); /* * uhci_pci_match must never return NULL if uhci_pci_probe * succeeded */ device_set_desc(sc->sc_bus.bdev, uhci_pci_match(self)); switch (pci_get_vendor(self)) { case PCI_UHCI_VENDORID_INTEL: sprintf(sc->sc_vendor, "Intel"); break; case PCI_UHCI_VENDORID_VIA: sprintf(sc->sc_vendor, "VIA"); break; default: if (bootverbose) { device_printf(self, "(New UHCI DeviceId=0x%08x)\n", pci_get_devid(self)); } sprintf(sc->sc_vendor, "(0x%04x)", pci_get_vendor(self)); } switch (pci_read_config(self, PCI_USBREV, 1) & PCI_USB_REV_MASK) { case PCI_USB_REV_PRE_1_0: sc->sc_bus.usbrev = USB_REV_PRE_1_0; break; case PCI_USB_REV_1_0: sc->sc_bus.usbrev = USB_REV_1_0; break; default: /* Quirk for Parallels Desktop 4.0 */ device_printf(self, "USB revision is unknown. Assuming v1.1.\n"); sc->sc_bus.usbrev = USB_REV_1_1; break; } #if (__FreeBSD_version >= 700031) err = bus_setup_intr(self, sc->sc_irq_res, INTR_TYPE_BIO | INTR_MPSAFE, NULL, (driver_intr_t *)uhci_interrupt, sc, &sc->sc_intr_hdl); #else err = bus_setup_intr(self, sc->sc_irq_res, INTR_TYPE_BIO | INTR_MPSAFE, (driver_intr_t *)uhci_interrupt, sc, &sc->sc_intr_hdl); #endif if (err) { device_printf(self, "Could not setup irq, %d\n", err); sc->sc_intr_hdl = NULL; goto error; } /* * Set the PIRQD enable bit and switch off all the others. We don't * want legacy support to interfere with us XXX Does this also mean * that the BIOS won't touch the keyboard anymore if it is connected * to the ports of the root hub? */ #ifdef USB_DEBUG if (pci_read_config(self, PCI_LEGSUP, 2) != PCI_LEGSUP_USBPIRQDEN) { device_printf(self, "LegSup = 0x%04x\n", pci_read_config(self, PCI_LEGSUP, 2)); } #endif pci_write_config(self, PCI_LEGSUP, PCI_LEGSUP_USBPIRQDEN, 2); err = uhci_init(sc); if (!err) { err = device_probe_and_attach(sc->sc_bus.bdev); } if (err) { device_printf(self, "USB init failed\n"); goto error; } return (0); error: uhci_pci_detach(self); return (ENXIO); } int uhci_pci_detach(device_t self) { uhci_softc_t *sc = device_get_softc(self); device_t bdev; if (sc->sc_bus.bdev) { bdev = sc->sc_bus.bdev; device_detach(bdev); device_delete_child(self, bdev); } /* during module unload there are lots of children leftover */ device_delete_children(self); /* * disable interrupts that might have been switched on in * uhci_init. */ if (sc->sc_io_res) { USB_BUS_LOCK(&sc->sc_bus); /* stop the controller */ uhci_reset(sc); USB_BUS_UNLOCK(&sc->sc_bus); } pci_disable_busmaster(self); if (sc->sc_irq_res && sc->sc_intr_hdl) { int err = bus_teardown_intr(self, sc->sc_irq_res, sc->sc_intr_hdl); if (err) { /* XXX or should we panic? */ device_printf(self, "Could not tear down irq, %d\n", err); } sc->sc_intr_hdl = NULL; } if (sc->sc_irq_res) { bus_release_resource(self, SYS_RES_IRQ, 0, sc->sc_irq_res); sc->sc_irq_res = NULL; } if (sc->sc_io_res) { bus_release_resource(self, SYS_RES_IOPORT, PCI_UHCI_BASE_REG, sc->sc_io_res); sc->sc_io_res = NULL; } usb_bus_mem_free_all(&sc->sc_bus, &uhci_iterate_hw_softc); return (0); } static device_method_t uhci_pci_methods[] = { /* Device interface */ DEVMETHOD(device_probe, uhci_pci_probe), DEVMETHOD(device_attach, uhci_pci_attach), DEVMETHOD(device_detach, uhci_pci_detach), DEVMETHOD(device_suspend, bus_generic_suspend), DEVMETHOD(device_resume, bus_generic_resume), DEVMETHOD(device_shutdown, bus_generic_shutdown), DEVMETHOD(usb_take_controller, uhci_pci_take_controller), DEVMETHOD_END }; static driver_t uhci_driver = { .name = "uhci", .methods = uhci_pci_methods, .size = sizeof(struct uhci_softc), }; static devclass_t uhci_devclass; DRIVER_MODULE(uhci, pci, uhci_driver, uhci_devclass, 0, 0); MODULE_DEPEND(uhci, usb, 1, 1, 1); Index: user/ngie/more-tests/sys/dev/usb/controller/xhci_pci.c =================================================================== --- user/ngie/more-tests/sys/dev/usb/controller/xhci_pci.c (revision 281503) +++ user/ngie/more-tests/sys/dev/usb/controller/xhci_pci.c (revision 281504) @@ -1,384 +1,384 @@ /*- * Copyright (c) 2010 Hans Petter Selasky. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "usb_if.h" static device_probe_t xhci_pci_probe; static device_attach_t xhci_pci_attach; static device_detach_t xhci_pci_detach; static usb_take_controller_t xhci_pci_take_controller; static device_method_t xhci_device_methods[] = { /* device interface */ DEVMETHOD(device_probe, xhci_pci_probe), DEVMETHOD(device_attach, xhci_pci_attach), DEVMETHOD(device_detach, xhci_pci_detach), DEVMETHOD(device_suspend, bus_generic_suspend), DEVMETHOD(device_resume, bus_generic_resume), DEVMETHOD(device_shutdown, bus_generic_shutdown), DEVMETHOD(usb_take_controller, xhci_pci_take_controller), DEVMETHOD_END }; static driver_t xhci_driver = { .name = "xhci", .methods = xhci_device_methods, .size = sizeof(struct xhci_softc), }; static devclass_t xhci_devclass; DRIVER_MODULE(xhci, pci, xhci_driver, xhci_devclass, 0, 0); MODULE_DEPEND(xhci, usb, 1, 1, 1); static const char * xhci_pci_match(device_t self) { uint32_t device_id = pci_get_devid(self); switch (device_id) { case 0x01941033: return ("NEC uPD720200 USB 3.0 controller"); case 0x10421b21: return ("ASMedia ASM1042 USB 3.0 controller"); case 0x0f358086: return ("Intel Intel BayTrail USB 3.0 controller"); case 0x9c318086: case 0x1e318086: return ("Intel Panther Point USB 3.0 controller"); case 0x8c318086: return ("Intel Lynx Point USB 3.0 controller"); case 0x8cb18086: return ("Intel Wildcat Point USB 3.0 controller"); default: break; } if ((pci_get_class(self) == PCIC_SERIALBUS) && (pci_get_subclass(self) == PCIS_SERIALBUS_USB) && (pci_get_progif(self) == PCIP_SERIALBUS_USB_XHCI)) { return ("XHCI (generic) USB 3.0 controller"); } return (NULL); /* dunno */ } static int xhci_pci_probe(device_t self) { const char *desc = xhci_pci_match(self); if (desc) { device_set_desc(self, desc); - return (0); + return (BUS_PROBE_DEFAULT); } else { return (ENXIO); } } static int xhci_use_msi = 1; TUNABLE_INT("hw.usb.xhci.msi", &xhci_use_msi); static void xhci_interrupt_poll(void *_sc) { struct xhci_softc *sc = _sc; USB_BUS_UNLOCK(&sc->sc_bus); xhci_interrupt(sc); USB_BUS_LOCK(&sc->sc_bus); usb_callout_reset(&sc->sc_callout, 1, (void *)&xhci_interrupt_poll, sc); } static int xhci_pci_port_route(device_t self, uint32_t set, uint32_t clear) { uint32_t temp; uint32_t usb3_mask; uint32_t usb2_mask; temp = pci_read_config(self, PCI_XHCI_INTEL_USB3_PSSEN, 4) | pci_read_config(self, PCI_XHCI_INTEL_XUSB2PR, 4); temp |= set; temp &= ~clear; /* Don't set bits which the hardware doesn't support */ usb3_mask = pci_read_config(self, PCI_XHCI_INTEL_USB3PRM, 4); usb2_mask = pci_read_config(self, PCI_XHCI_INTEL_USB2PRM, 4); pci_write_config(self, PCI_XHCI_INTEL_USB3_PSSEN, temp & usb3_mask, 4); pci_write_config(self, PCI_XHCI_INTEL_XUSB2PR, temp & usb2_mask, 4); device_printf(self, "Port routing mask set to 0x%08x\n", temp); return (0); } static int xhci_pci_attach(device_t self) { struct xhci_softc *sc = device_get_softc(self); int count, err, rid; uint8_t usedma32; rid = PCI_XHCI_CBMEM; sc->sc_io_res = bus_alloc_resource_any(self, SYS_RES_MEMORY, &rid, RF_ACTIVE); if (!sc->sc_io_res) { device_printf(self, "Could not map memory\n"); return (ENOMEM); } sc->sc_io_tag = rman_get_bustag(sc->sc_io_res); sc->sc_io_hdl = rman_get_bushandle(sc->sc_io_res); sc->sc_io_size = rman_get_size(sc->sc_io_res); /* check for USB 3.0 controllers which don't support 64-bit DMA */ switch (pci_get_devid(self)) { case 0x01941033: /* NEC uPD720200 USB 3.0 controller */ usedma32 = 1; break; default: usedma32 = 0; break; } if (xhci_init(sc, self, usedma32)) { device_printf(self, "Could not initialize softc\n"); bus_release_resource(self, SYS_RES_MEMORY, PCI_XHCI_CBMEM, sc->sc_io_res); return (ENXIO); } pci_enable_busmaster(self); usb_callout_init_mtx(&sc->sc_callout, &sc->sc_bus.bus_mtx, 0); rid = 0; if (xhci_use_msi) { count = 1; if (pci_alloc_msi(self, &count) == 0) { if (bootverbose) device_printf(self, "MSI enabled\n"); rid = 1; } } sc->sc_irq_res = bus_alloc_resource_any(self, SYS_RES_IRQ, &rid, RF_ACTIVE | (rid != 0 ? 0 : RF_SHAREABLE)); if (sc->sc_irq_res == NULL) { pci_release_msi(self); device_printf(self, "Could not allocate IRQ\n"); /* goto error; FALLTHROUGH - use polling */ } sc->sc_bus.bdev = device_add_child(self, "usbus", -1); if (sc->sc_bus.bdev == NULL) { device_printf(self, "Could not add USB device\n"); goto error; } device_set_ivars(sc->sc_bus.bdev, &sc->sc_bus); sprintf(sc->sc_vendor, "0x%04x", pci_get_vendor(self)); if (sc->sc_irq_res != NULL) { err = bus_setup_intr(self, sc->sc_irq_res, INTR_TYPE_BIO | INTR_MPSAFE, NULL, (driver_intr_t *)xhci_interrupt, sc, &sc->sc_intr_hdl); if (err != 0) { bus_release_resource(self, SYS_RES_IRQ, rman_get_rid(sc->sc_irq_res), sc->sc_irq_res); sc->sc_irq_res = NULL; pci_release_msi(self); device_printf(self, "Could not setup IRQ, err=%d\n", err); sc->sc_intr_hdl = NULL; } } if (sc->sc_irq_res == NULL || sc->sc_intr_hdl == NULL) { if (xhci_use_polling() != 0) { device_printf(self, "Interrupt polling at %dHz\n", hz); USB_BUS_LOCK(&sc->sc_bus); xhci_interrupt_poll(sc); USB_BUS_UNLOCK(&sc->sc_bus); } else goto error; } /* On Intel chipsets reroute ports from EHCI to XHCI controller. */ switch (pci_get_devid(self)) { case 0x0f358086: /* BayTrail */ case 0x9c318086: /* Panther Point */ case 0x1e318086: /* Panther Point */ case 0x8c318086: /* Lynx Point */ case 0x8cb18086: /* Wildcat Point */ sc->sc_port_route = &xhci_pci_port_route; sc->sc_imod_default = XHCI_IMOD_DEFAULT_LP; break; default: break; } xhci_pci_take_controller(self); err = xhci_halt_controller(sc); if (err == 0) err = xhci_start_controller(sc); if (err == 0) err = device_probe_and_attach(sc->sc_bus.bdev); if (err) { device_printf(self, "XHCI halt/start/probe failed err=%d\n", err); goto error; } return (0); error: xhci_pci_detach(self); return (ENXIO); } static int xhci_pci_detach(device_t self) { struct xhci_softc *sc = device_get_softc(self); device_t bdev; if (sc->sc_bus.bdev != NULL) { bdev = sc->sc_bus.bdev; device_detach(bdev); device_delete_child(self, bdev); } /* during module unload there are lots of children leftover */ device_delete_children(self); usb_callout_drain(&sc->sc_callout); xhci_halt_controller(sc); pci_disable_busmaster(self); if (sc->sc_irq_res && sc->sc_intr_hdl) { bus_teardown_intr(self, sc->sc_irq_res, sc->sc_intr_hdl); sc->sc_intr_hdl = NULL; } if (sc->sc_irq_res) { bus_release_resource(self, SYS_RES_IRQ, rman_get_rid(sc->sc_irq_res), sc->sc_irq_res); sc->sc_irq_res = NULL; pci_release_msi(self); } if (sc->sc_io_res) { bus_release_resource(self, SYS_RES_MEMORY, PCI_XHCI_CBMEM, sc->sc_io_res); sc->sc_io_res = NULL; } xhci_uninit(sc); return (0); } static int xhci_pci_take_controller(device_t self) { struct xhci_softc *sc = device_get_softc(self); uint32_t cparams; uint32_t eecp; uint32_t eec; uint16_t to; uint8_t bios_sem; cparams = XREAD4(sc, capa, XHCI_HCSPARAMS0); eec = -1; /* Synchronise with the BIOS if it owns the controller. */ for (eecp = XHCI_HCS0_XECP(cparams) << 2; eecp != 0 && XHCI_XECP_NEXT(eec); eecp += XHCI_XECP_NEXT(eec) << 2) { eec = XREAD4(sc, capa, eecp); if (XHCI_XECP_ID(eec) != XHCI_ID_USB_LEGACY) continue; bios_sem = XREAD1(sc, capa, eecp + XHCI_XECP_BIOS_SEM); if (bios_sem == 0) continue; device_printf(sc->sc_bus.bdev, "waiting for BIOS " "to give up control\n"); XWRITE1(sc, capa, eecp + XHCI_XECP_OS_SEM, 1); to = 500; while (1) { bios_sem = XREAD1(sc, capa, eecp + XHCI_XECP_BIOS_SEM); if (bios_sem == 0) break; if (--to == 0) { device_printf(sc->sc_bus.bdev, "timed out waiting for BIOS\n"); break; } usb_pause_mtx(NULL, hz / 100); /* wait 10ms */ } } return (0); } Index: user/ngie/more-tests/sys/i386/i386/bios.c =================================================================== --- user/ngie/more-tests/sys/i386/i386/bios.c (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/bios.c (revision 281504) @@ -1,768 +1,768 @@ /*- * Copyright (c) 1997 Michael Smith * Copyright (c) 1998 Jonathan Lemon * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* * Code for dealing with the BIOS in x86 PC systems. */ #include "opt_isa.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DEV_ISA #include #include #include #endif #define BIOS_START 0xe0000 #define BIOS_SIZE 0x20000 /* exported lookup results */ struct bios32_SDentry PCIbios; static struct PnPBIOS_table *PnPBIOStable; static u_int bios32_SDCI; /* start fairly early */ static void bios32_init(void *junk); SYSINIT(bios32, SI_SUB_CPU, SI_ORDER_ANY, bios32_init, NULL); /* * bios32_init * * Locate various bios32 entities. */ static void bios32_init(void *junk) { u_long sigaddr; struct bios32_SDheader *sdh; struct PnPBIOS_table *pt; u_int8_t ck, *cv; int i; char *p; /* * BIOS32 Service Directory, PCI BIOS */ /* look for the signature */ if ((sigaddr = bios_sigsearch(0, "_32_", 4, 16, 0)) != 0) { /* get a virtual pointer to the structure */ sdh = (struct bios32_SDheader *)(uintptr_t)BIOS_PADDRTOVADDR(sigaddr); for (cv = (u_int8_t *)sdh, ck = 0, i = 0; i < (sdh->len * 16); i++) { ck += cv[i]; } /* If checksum is OK, enable use of the entrypoint */ if ((ck == 0) && (BIOS_START <= sdh->entry ) && (sdh->entry < (BIOS_START + BIOS_SIZE))) { bios32_SDCI = BIOS_PADDRTOVADDR(sdh->entry); if (bootverbose) { printf("bios32: Found BIOS32 Service Directory header at %p\n", sdh); printf("bios32: Entry = 0x%x (%x) Rev = %d Len = %d\n", sdh->entry, bios32_SDCI, sdh->revision, sdh->len); } /* Allow user override of PCI BIOS search */ if (((p = kern_getenv("machdep.bios.pci")) == NULL) || strcmp(p, "disable")) { /* See if there's a PCI BIOS entrypoint here */ PCIbios.ident.id = 0x49435024; /* PCI systems should have this */ if (!bios32_SDlookup(&PCIbios) && bootverbose) printf("pcibios: PCI BIOS entry at 0x%x+0x%x\n", PCIbios.base, PCIbios.entry); } if (p != NULL) freeenv(p); } else { printf("bios32: Bad BIOS32 Service Directory\n"); } } /* * PnP BIOS * * Allow user override of PnP BIOS search */ if ((((p = kern_getenv("machdep.bios.pnp")) == NULL) || strcmp(p, "disable")) && ((sigaddr = bios_sigsearch(0, "$PnP", 4, 16, 0)) != 0)) { /* get a virtual pointer to the structure */ pt = (struct PnPBIOS_table *)(uintptr_t)BIOS_PADDRTOVADDR(sigaddr); for (cv = (u_int8_t *)pt, ck = 0, i = 0; i < pt->len; i++) { ck += cv[i]; } /* If checksum is OK, enable use of the entrypoint */ if (ck == 0) { PnPBIOStable = pt; if (bootverbose) { printf("pnpbios: Found PnP BIOS data at %p\n", pt); printf("pnpbios: Entry = %x:%x Rev = %d.%d\n", pt->pmentrybase, pt->pmentryoffset, pt->version >> 4, pt->version & 0xf); if ((pt->control & 0x3) == 0x01) printf("pnpbios: Event flag at %x\n", pt->evflagaddr); if (pt->oemdevid != 0) printf("pnpbios: OEM ID %x\n", pt->oemdevid); } } else { printf("pnpbios: Bad PnP BIOS data checksum\n"); } } if (p != NULL) freeenv(p); if (bootverbose) { /* look for other know signatures */ printf("Other BIOS signatures found:\n"); } } /* * bios32_SDlookup * * Query the BIOS32 Service Directory for the service named in (ent), * returns nonzero if the lookup fails. The caller must fill in * (ent->ident), the remainder are populated on a successful lookup. */ int bios32_SDlookup(struct bios32_SDentry *ent) { struct bios_regs args; if (bios32_SDCI == 0) return (1); args.eax = ent->ident.id; /* set up arguments */ args.ebx = args.ecx = args.edx = 0; bios32(&args, bios32_SDCI, GSEL(GCODE_SEL, SEL_KPL)); if ((args.eax & 0xff) == 0) { /* success? */ ent->base = args.ebx; ent->len = args.ecx; ent->entry = args.edx; ent->ventry = BIOS_PADDRTOVADDR(ent->base + ent->entry); return (0); /* all OK */ } return (1); /* failed */ } /* * bios_sigsearch * * Search some or all of the BIOS region for a signature string. * * (start) Optional offset returned from this function * (for searching for multiple matches), or NULL * to start the search from the base of the BIOS. * Note that this will be a _physical_ address in * the range 0xe0000 - 0xfffff. * (sig) is a pointer to the byte(s) of the signature. * (siglen) number of bytes in the signature. * (paralen) signature paragraph (alignment) size. * (sigofs) offset of the signature within the paragraph. * * Returns the _physical_ address of the found signature, 0 if the * signature was not found. */ u_int32_t bios_sigsearch(u_int32_t start, u_char *sig, int siglen, int paralen, int sigofs) { u_char *sp, *end; /* compute the starting address */ if ((start >= BIOS_START) && (start <= (BIOS_START + BIOS_SIZE))) { sp = (char *)BIOS_PADDRTOVADDR(start); } else if (start == 0) { sp = (char *)BIOS_PADDRTOVADDR(BIOS_START); } else { return 0; /* bogus start address */ } /* compute the end address */ end = (u_char *)BIOS_PADDRTOVADDR(BIOS_START + BIOS_SIZE); /* loop searching */ while ((sp + sigofs + siglen) < end) { /* compare here */ if (!bcmp(sp + sigofs, sig, siglen)) { /* convert back to physical address */ return((u_int32_t)BIOS_VADDRTOPADDR(sp)); } sp += paralen; } return(0); } /* * do not staticize, used by bioscall.s */ union { struct { u_short offset; u_short segment; } vec16; struct { u_int offset; u_short segment; } vec32; } bioscall_vector; /* bios jump vector */ void set_bios_selectors(struct bios_segments *seg, int flags) { struct soft_segment_descriptor ssd = { 0, /* segment base address (overwritten) */ 0, /* length (overwritten) */ SDT_MEMERA, /* segment type (overwritten) */ 0, /* priority level */ 1, /* descriptor present */ 0, 0, 1, /* descriptor size (overwritten) */ 0 /* granularity == byte units */ }; union descriptor *p_gdt; #ifdef SMP p_gdt = &gdt[PCPU_GET(cpuid) * NGDT]; #else p_gdt = gdt; #endif ssd.ssd_base = seg->code32.base; ssd.ssd_limit = seg->code32.limit; ssdtosd(&ssd, &p_gdt[GBIOSCODE32_SEL].sd); ssd.ssd_def32 = 0; if (flags & BIOSCODE_FLAG) { ssd.ssd_base = seg->code16.base; ssd.ssd_limit = seg->code16.limit; ssdtosd(&ssd, &p_gdt[GBIOSCODE16_SEL].sd); } ssd.ssd_type = SDT_MEMRWA; if (flags & BIOSDATA_FLAG) { ssd.ssd_base = seg->data.base; ssd.ssd_limit = seg->data.limit; ssdtosd(&ssd, &p_gdt[GBIOSDATA_SEL].sd); } if (flags & BIOSUTIL_FLAG) { ssd.ssd_base = seg->util.base; ssd.ssd_limit = seg->util.limit; ssdtosd(&ssd, &p_gdt[GBIOSUTIL_SEL].sd); } if (flags & BIOSARGS_FLAG) { ssd.ssd_base = seg->args.base; ssd.ssd_limit = seg->args.limit; ssdtosd(&ssd, &p_gdt[GBIOSARGS_SEL].sd); } } extern int vm86pa; extern void bios16_jmp(void); /* * this routine is really greedy with selectors, and uses 5: * * 32-bit code selector: to return to kernel * 16-bit code selector: for running code * data selector: for 16-bit data * util selector: extra utility selector * args selector: to handle pointers * * the util selector is set from the util16 entry in bios16_args, if a * "U" specifier is seen. * * See for description of format specifiers */ int bios16(struct bios_args *args, char *fmt, ...) { char *p, *stack, *stack_top; va_list ap; int flags = BIOSCODE_FLAG | BIOSDATA_FLAG; u_int i, arg_start, arg_end; pt_entry_t *pte; pd_entry_t *ptd; arg_start = 0xffffffff; arg_end = 0; /* * Some BIOS entrypoints attempt to copy the largest-case * argument frame (in order to generalise handling for * different entry types). If our argument frame is * smaller than this, the BIOS will reach off the top of * our constructed stack segment. Pad the top of the stack * with some garbage to avoid this. */ stack = (caddr_t)PAGE_SIZE - 32; va_start(ap, fmt); for (p = fmt; p && *p; p++) { switch (*p) { case 'p': /* 32-bit pointer */ i = va_arg(ap, u_int); arg_start = min(arg_start, i); arg_end = max(arg_end, i); flags |= BIOSARGS_FLAG; stack -= 4; break; case 'i': /* 32-bit integer */ i = va_arg(ap, u_int); stack -= 4; break; case 'U': /* 16-bit selector */ flags |= BIOSUTIL_FLAG; /* FALLTHROUGH */ case 'D': /* 16-bit selector */ case 'C': /* 16-bit selector */ stack -= 2; break; case 's': /* 16-bit integer passed as an int */ i = va_arg(ap, int); stack -= 2; break; default: va_end(ap); return (EINVAL); } } va_end(ap); if (flags & BIOSARGS_FLAG) { if (arg_end - arg_start > ctob(16)) return (EACCES); args->seg.args.base = arg_start; args->seg.args.limit = 0xffff; } args->seg.code32.base = (u_int)&bios16_jmp & PG_FRAME; args->seg.code32.limit = 0xffff; ptd = (pd_entry_t *)rcr3(); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if (ptd == IdlePDPT) #else if (ptd == IdlePTD) #endif { /* * no page table, so create one and install it. */ pte = (pt_entry_t *)malloc(PAGE_SIZE, M_TEMP, M_WAITOK); ptd = (pd_entry_t *)((u_int)IdlePTD + KERNBASE); *pte = (vm86pa - PAGE_SIZE) | PG_RW | PG_V; *ptd = vtophys(pte) | PG_RW | PG_V; } else { /* * this is a user-level page table */ pte = PTmap; *pte = (vm86pa - PAGE_SIZE) | PG_RW | PG_V; } pmap_invalidate_all(kernel_pmap); /* XXX insurance for now */ stack_top = stack; va_start(ap, fmt); for (p = fmt; p && *p; p++) { switch (*p) { case 'p': /* 32-bit pointer */ i = va_arg(ap, u_int); *(u_int *)stack = (i - arg_start) | (GSEL(GBIOSARGS_SEL, SEL_KPL) << 16); stack += 4; break; case 'i': /* 32-bit integer */ i = va_arg(ap, u_int); *(u_int *)stack = i; stack += 4; break; case 'U': /* 16-bit selector */ *(u_short *)stack = GSEL(GBIOSUTIL_SEL, SEL_KPL); stack += 2; break; case 'D': /* 16-bit selector */ *(u_short *)stack = GSEL(GBIOSDATA_SEL, SEL_KPL); stack += 2; break; case 'C': /* 16-bit selector */ *(u_short *)stack = GSEL(GBIOSCODE16_SEL, SEL_KPL); stack += 2; break; case 's': /* 16-bit integer passed as an int */ i = va_arg(ap, int); *(u_short *)stack = i; stack += 2; break; default: va_end(ap); return (EINVAL); } } va_end(ap); set_bios_selectors(&args->seg, flags); bioscall_vector.vec16.offset = (u_short)args->entry; bioscall_vector.vec16.segment = GSEL(GBIOSCODE16_SEL, SEL_KPL); i = bios16_call(&args->r, stack_top); if (pte == PTmap) { *pte = 0; /* remove entry */ /* * XXX only needs to be invlpg(0) but that doesn't work on the 386 */ pmap_invalidate_all(kernel_pmap); } else { *ptd = 0; /* remove page table */ /* * XXX only needs to be invlpg(0) but that doesn't work on the 386 */ pmap_invalidate_all(kernel_pmap); free(pte, M_TEMP); /* ... and free it */ } return (i); } int bios_oem_strings(struct bios_oem *oem, u_char *buffer, size_t maxlen) { size_t idx = 0; struct bios_oem_signature *sig; u_int from, to; u_char c, *s, *se, *str, *bios_str; size_t i, off, len, tot; if ( !oem || !buffer || maxlen<2 ) return(-1); sig = oem->signature; if (!sig) return(-2); from = oem->range.from; to = oem->range.to; if ( (to<=from) || (from(BIOS_START+BIOS_SIZE)) ) return(-3); while (sig->anchor != NULL) { str = sig->anchor; len = strlen(str); off = sig->offset; tot = sig->totlen; /* make sure offset doesn't go beyond bios area */ if ( (to+off)>(BIOS_START+BIOS_SIZE) || ((from+off) maxlen - 1) { printf("sys/i386/i386/bios.c: sig '%s' " "idx %d + tot %d = %d > maxlen-1 %d\n", str, idx, tot, idx+tot, maxlen-1); return(-5); } bios_str = NULL; s = (u_char *)BIOS_PADDRTOVADDR(from); se = (u_char *)BIOS_PADDRTOVADDR(to-len); for (; s 0x7E) ) c = ' '; if (idx == 0) { if (c != ' ') buffer[idx++] = c; } else if ( (c != ' ') || ((c == ' ') && (buffer[idx-1] != ' ')) ) buffer[idx++] = c; } } sig++; } /* remove a final trailing space */ if ( (idx > 1) && (buffer[idx-1] == ' ') ) idx--; buffer[idx] = '\0'; return (idx); } #ifdef DEV_ISA /* * PnP BIOS interface; enumerate devices only known to the system * BIOS and save information about them for later use. */ struct pnp_sysdev { u_int16_t size; u_int8_t handle; u_int32_t devid; u_int8_t type[3]; u_int16_t attrib; #define PNPATTR_NODISABLE (1<<0) /* can't be disabled */ #define PNPATTR_NOCONFIG (1<<1) /* can't be configured */ #define PNPATTR_OUTPUT (1<<2) /* can be primary output */ #define PNPATTR_INPUT (1<<3) /* can be primary input */ #define PNPATTR_BOOTABLE (1<<4) /* can be booted from */ #define PNPATTR_DOCK (1<<5) /* is a docking station */ #define PNPATTR_REMOVEABLE (1<<6) /* device is removeable */ #define PNPATTR_CONFIG_STATIC (0) #define PNPATTR_CONFIG_DYNAMIC (1) #define PNPATTR_CONFIG_DYNONLY (3) #define PNPATTR_CONFIG(a) (((a) >> 7) & 0x3) /* device-specific data comes here */ u_int8_t devdata[0]; } __packed; /* We have to cluster arguments within a 64k range for the bios16 call */ struct pnp_sysdevargs { u_int16_t next; struct pnp_sysdev node; }; /* * This function is called after the bus has assigned resource * locations for a logical device. */ static void pnpbios_set_config(void *arg, struct isa_config *config, int enable) { } /* * Quiz the PnP BIOS, build a list of PNP IDs and resource data. */ static void pnpbios_identify(driver_t *driver, device_t parent) { struct PnPBIOS_table *pt = PnPBIOStable; struct bios_args args; struct pnp_sysdev *pd; struct pnp_sysdevargs *pda; u_int16_t ndevs, bigdev; int error, currdev; u_int8_t *devnodebuf, tag; u_int32_t *devid, *compid; int idx, left; device_t dev; /* no PnP BIOS information */ if (pt == NULL) return; /* Check to see if ACPI is already active. */ dev = devclass_get_device(devclass_find("acpi"), 0); if (dev != NULL && device_is_attached(dev)) return; /* get count of PnP devices */ bzero(&args, sizeof(args)); args.seg.code16.base = BIOS_PADDRTOVADDR(pt->pmentrybase); args.seg.code16.limit = 0xffff; /* XXX ? */ args.seg.data.base = BIOS_PADDRTOVADDR(pt->pmdataseg); args.seg.data.limit = 0xffff; args.entry = pt->pmentryoffset; if ((error = bios16(&args, PNP_COUNT_DEVNODES, &ndevs, &bigdev)) || (args.r.eax & 0xff)) { printf("pnpbios: error %d/%x getting device count/size limit\n", error, args.r.eax); return; } ndevs &= 0xff; /* clear high byte garbage */ if (bootverbose) printf("pnpbios: %d devices, largest %d bytes\n", ndevs, bigdev); devnodebuf = malloc(bigdev + (sizeof(struct pnp_sysdevargs) - sizeof(struct pnp_sysdev)), M_DEVBUF, M_NOWAIT); if (devnodebuf == NULL) { printf("pnpbios: cannot allocate memory, bailing\n"); return; } pda = (struct pnp_sysdevargs *)devnodebuf; pd = &pda->node; for (currdev = 0, left = ndevs; (currdev != 0xff) && (left > 0); left--) { bzero(pd, bigdev); pda->next = currdev; /* get current configuration */ if ((error = bios16(&args, PNP_GET_DEVNODE, &pda->next, &pda->node, 1))) { printf("pnpbios: error %d making BIOS16 call\n", error); break; } if ((error = (args.r.eax & 0xff))) { if (bootverbose) printf("pnpbios: %s 0x%x fetching node %d\n", error & 0x80 ? "error" : "warning", error, currdev); if (error & 0x80) break; } currdev = pda->next; if (pd->size < sizeof(struct pnp_sysdev)) { printf("pnpbios: bogus system node data, aborting scan\n"); break; } /* * Ignore PICs so that we don't have to worry about the PICs * claiming IRQs to prevent their use. The PIC drivers * already ensure that invalid IRQs are not used. */ if (!strcmp(pnp_eisaformat(pd->devid), "PNP0000")) /* ISA PIC */ continue; if (!strcmp(pnp_eisaformat(pd->devid), "PNP0003")) /* APIC */ continue; /* Add the device and parse its resources */ dev = BUS_ADD_CHILD(parent, ISA_ORDER_PNPBIOS, NULL, -1); isa_set_vendorid(dev, pd->devid); isa_set_logicalid(dev, pd->devid); /* * It appears that some PnP BIOS doesn't allow us to re-enable * the embedded system device once it is disabled. We shall * mark all system device nodes as "cannot be disabled", regardless * of actual settings in the device attribute byte. * XXX isa_set_configattr(dev, ((pd->attrib & PNPATTR_NODISABLE) ? 0 : ISACFGATTR_CANDISABLE) | ((!(pd->attrib & PNPATTR_NOCONFIG) && PNPATTR_CONFIG(pd->attrib) != PNPATTR_CONFIG_STATIC) ? ISACFGATTR_DYNAMIC : 0)); */ isa_set_configattr(dev, (!(pd->attrib & PNPATTR_NOCONFIG) && PNPATTR_CONFIG(pd->attrib) != PNPATTR_CONFIG_STATIC) ? ISACFGATTR_DYNAMIC : 0); isa_set_pnpbios_handle(dev, pd->handle); ISA_SET_CONFIG_CALLBACK(parent, dev, pnpbios_set_config, 0); pnp_parse_resources(dev, &pd->devdata[0], pd->size - sizeof(struct pnp_sysdev), 0); if (!device_get_desc(dev)) device_set_desc_copy(dev, pnp_eisaformat(pd->devid)); /* Find device IDs */ devid = &pd->devid; compid = NULL; /* look for a compatible device ID too */ left = pd->size - sizeof(struct pnp_sysdev); idx = 0; while (idx < left) { tag = pd->devdata[idx++]; if (PNP_RES_TYPE(tag) == 0) { /* Small resource */ switch (PNP_SRES_NUM(tag)) { case PNP_TAG_COMPAT_DEVICE: compid = (u_int32_t *)(pd->devdata + idx); if (bootverbose) printf("pnpbios: node %d compat ID 0x%08x\n", pd->handle, *compid); /* FALLTHROUGH */ case PNP_TAG_END: idx = left; break; default: idx += PNP_SRES_LEN(tag); break; } } else /* Large resource, skip it */ idx += *(u_int16_t *)(pd->devdata + idx) + 2; } if (bootverbose) { printf("pnpbios: handle %d device ID %s (%08x)", pd->handle, pnp_eisaformat(*devid), *devid); if (compid != NULL) printf(" compat ID %s (%08x)", pnp_eisaformat(*compid), *compid); printf("\n"); } } } static device_method_t pnpbios_methods[] = { /* Device interface */ DEVMETHOD(device_identify, pnpbios_identify), { 0, 0 } }; static driver_t pnpbios_driver = { "pnpbios", pnpbios_methods, 1, /* no softc */ }; static devclass_t pnpbios_devclass; DRIVER_MODULE(pnpbios, isa, pnpbios_driver, pnpbios_devclass, 0, 0); #endif /* DEV_ISA */ Index: user/ngie/more-tests/sys/i386/i386/initcpu.c =================================================================== --- user/ngie/more-tests/sys/i386/i386/initcpu.c (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/initcpu.c (revision 281504) @@ -1,1104 +1,1104 @@ /*- * Copyright (c) KATO Takenori, 1997, 1998. * * All rights reserved. Unpublished rights reserved under the copyright * laws of Japan. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer as * the first lines of this file unmodified. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include "opt_cpu.h" #include #include #include #include #include #include #include #include #include #if !defined(CPU_DISABLE_SSE) && defined(I686_CPU) #define CPU_ENABLE_SSE #endif #ifdef I486_CPU static void init_5x86(void); static void init_bluelightning(void); static void init_486dlc(void); static void init_cy486dx(void); #ifdef CPU_I486_ON_386 static void init_i486_on_386(void); #endif static void init_6x86(void); #endif /* I486_CPU */ #if defined(I586_CPU) && defined(CPU_WT_ALLOC) static void enable_K5_wt_alloc(void); static void enable_K6_wt_alloc(void); static void enable_K6_2_wt_alloc(void); #endif #ifdef I686_CPU static void init_6x86MX(void); static void init_ppro(void); static void init_mendocino(void); #endif static int hw_instruction_sse; SYSCTL_INT(_hw, OID_AUTO, instruction_sse, CTLFLAG_RD, &hw_instruction_sse, 0, "SIMD/MMX2 instructions available in CPU"); /* * -1: automatic (default) * 0: keep enable CLFLUSH * 1: force disable CLFLUSH */ static int hw_clflush_disable = -1; int cpu; /* Are we 386, 386sx, 486, etc? */ u_int cpu_feature; /* Feature flags */ u_int cpu_feature2; /* Feature flags */ u_int amd_feature; /* AMD feature flags */ u_int amd_feature2; /* AMD feature flags */ u_int amd_pminfo; /* AMD advanced power management info */ u_int via_feature_rng; /* VIA RNG features */ u_int via_feature_xcrypt; /* VIA ACE features */ u_int cpu_high; /* Highest arg to CPUID */ u_int cpu_exthigh; /* Highest arg to extended CPUID */ u_int cpu_id; /* Stepping ID */ u_int cpu_procinfo; /* HyperThreading Info / Brand Index / CLFUSH */ u_int cpu_procinfo2; /* Multicore info */ char cpu_vendor[20]; /* CPU Origin code */ u_int cpu_vendor_id; /* CPU vendor ID */ #ifdef CPU_ENABLE_SSE u_int cpu_fxsr; /* SSE enabled */ u_int cpu_mxcsr_mask; /* Valid bits in mxcsr */ #endif u_int cpu_clflush_line_size = 32; u_int cpu_stdext_feature; u_int cpu_max_ext_state_size; u_int cpu_mon_mwait_flags; /* MONITOR/MWAIT flags (CPUID.05H.ECX) */ u_int cpu_mon_min_size; /* MONITOR minimum range size, bytes */ u_int cpu_mon_max_size; /* MONITOR minimum range size, bytes */ u_int cyrix_did; /* Device ID of Cyrix CPU */ u_int cpu_maxphyaddr; /* Max phys addr width in bits */ SYSCTL_UINT(_hw, OID_AUTO, via_feature_rng, CTLFLAG_RD, &via_feature_rng, 0, "VIA RNG feature available in CPU"); SYSCTL_UINT(_hw, OID_AUTO, via_feature_xcrypt, CTLFLAG_RD, &via_feature_xcrypt, 0, "VIA xcrypt feature available in CPU"); #ifdef I486_CPU /* * IBM Blue Lightning */ static void init_bluelightning(void) { register_t saveintr; #if defined(PC98) && !defined(CPU_UPGRADE_HW_CACHE) need_post_dma_flush = 1; #endif saveintr = intr_disable(); load_cr0(rcr0() | CR0_CD | CR0_NW); invd(); #ifdef CPU_BLUELIGHTNING_FPU_OP_CACHE wrmsr(0x1000, 0x9c92LL); /* FP operand can be cacheable on Cyrix FPU */ #else wrmsr(0x1000, 0x1c92LL); /* Intel FPU */ #endif /* Enables 13MB and 0-640KB cache. */ wrmsr(0x1001, (0xd0LL << 32) | 0x3ff); #ifdef CPU_BLUELIGHTNING_3X wrmsr(0x1002, 0x04000000LL); /* Enables triple-clock mode. */ #else wrmsr(0x1002, 0x03000000LL); /* Enables double-clock mode. */ #endif /* Enable caching in CR0. */ load_cr0(rcr0() & ~(CR0_CD | CR0_NW)); /* CD = 0 and NW = 0 */ invd(); intr_restore(saveintr); } /* * Cyrix 486SLC/DLC/SR/DR series */ static void init_486dlc(void) { register_t saveintr; u_char ccr0; saveintr = intr_disable(); invd(); ccr0 = read_cyrix_reg(CCR0); #ifndef CYRIX_CACHE_WORKS ccr0 |= CCR0_NC1 | CCR0_BARB; write_cyrix_reg(CCR0, ccr0); invd(); #else ccr0 &= ~CCR0_NC0; #ifndef CYRIX_CACHE_REALLY_WORKS ccr0 |= CCR0_NC1 | CCR0_BARB; #else ccr0 |= CCR0_NC1; #endif #ifdef CPU_DIRECT_MAPPED_CACHE ccr0 |= CCR0_CO; /* Direct mapped mode. */ #endif write_cyrix_reg(CCR0, ccr0); /* Clear non-cacheable region. */ write_cyrix_reg(NCR1+2, NCR_SIZE_0K); write_cyrix_reg(NCR2+2, NCR_SIZE_0K); write_cyrix_reg(NCR3+2, NCR_SIZE_0K); write_cyrix_reg(NCR4+2, NCR_SIZE_0K); write_cyrix_reg(0, 0); /* dummy write */ /* Enable caching in CR0. */ load_cr0(rcr0() & ~(CR0_CD | CR0_NW)); /* CD = 0 and NW = 0 */ invd(); #endif /* !CYRIX_CACHE_WORKS */ intr_restore(saveintr); } /* * Cyrix 486S/DX series */ static void init_cy486dx(void) { register_t saveintr; u_char ccr2; saveintr = intr_disable(); invd(); ccr2 = read_cyrix_reg(CCR2); #ifdef CPU_SUSP_HLT ccr2 |= CCR2_SUSP_HLT; #endif #ifdef PC98 /* Enables WB cache interface pin and Lock NW bit in CR0. */ ccr2 |= CCR2_WB | CCR2_LOCK_NW; /* Unlock NW bit in CR0. */ write_cyrix_reg(CCR2, ccr2 & ~CCR2_LOCK_NW); load_cr0((rcr0() & ~CR0_CD) | CR0_NW); /* CD = 0, NW = 1 */ #endif write_cyrix_reg(CCR2, ccr2); intr_restore(saveintr); } /* * Cyrix 5x86 */ static void init_5x86(void) { register_t saveintr; u_char ccr2, ccr3, ccr4, pcr0; saveintr = intr_disable(); load_cr0(rcr0() | CR0_CD | CR0_NW); wbinvd(); (void)read_cyrix_reg(CCR3); /* dummy */ /* Initialize CCR2. */ ccr2 = read_cyrix_reg(CCR2); ccr2 |= CCR2_WB; #ifdef CPU_SUSP_HLT ccr2 |= CCR2_SUSP_HLT; #else ccr2 &= ~CCR2_SUSP_HLT; #endif ccr2 |= CCR2_WT1; write_cyrix_reg(CCR2, ccr2); /* Initialize CCR4. */ ccr3 = read_cyrix_reg(CCR3); write_cyrix_reg(CCR3, CCR3_MAPEN0); ccr4 = read_cyrix_reg(CCR4); ccr4 |= CCR4_DTE; ccr4 |= CCR4_MEM; #ifdef CPU_FASTER_5X86_FPU ccr4 |= CCR4_FASTFPE; #else ccr4 &= ~CCR4_FASTFPE; #endif ccr4 &= ~CCR4_IOMASK; /******************************************************************** * WARNING: The "BIOS Writers Guide" mentions that I/O recovery time * should be 0 for errata fix. ********************************************************************/ #ifdef CPU_IORT ccr4 |= CPU_IORT & CCR4_IOMASK; #endif write_cyrix_reg(CCR4, ccr4); /* Initialize PCR0. */ /**************************************************************** * WARNING: RSTK_EN and LOOP_EN could make your system unstable. * BTB_EN might make your system unstable. ****************************************************************/ pcr0 = read_cyrix_reg(PCR0); #ifdef CPU_RSTK_EN pcr0 |= PCR0_RSTK; #else pcr0 &= ~PCR0_RSTK; #endif #ifdef CPU_BTB_EN pcr0 |= PCR0_BTB; #else pcr0 &= ~PCR0_BTB; #endif #ifdef CPU_LOOP_EN pcr0 |= PCR0_LOOP; #else pcr0 &= ~PCR0_LOOP; #endif /**************************************************************** * WARNING: if you use a memory mapped I/O device, don't use * DISABLE_5X86_LSSER option, which may reorder memory mapped * I/O access. * IF YOUR MOTHERBOARD HAS PCI BUS, DON'T DISABLE LSSER. ****************************************************************/ #ifdef CPU_DISABLE_5X86_LSSER pcr0 &= ~PCR0_LSSER; #else pcr0 |= PCR0_LSSER; #endif write_cyrix_reg(PCR0, pcr0); /* Restore CCR3. */ write_cyrix_reg(CCR3, ccr3); (void)read_cyrix_reg(0x80); /* dummy */ /* Unlock NW bit in CR0. */ write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) & ~CCR2_LOCK_NW); load_cr0((rcr0() & ~CR0_CD) | CR0_NW); /* CD = 0, NW = 1 */ /* Lock NW bit in CR0. */ write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) | CCR2_LOCK_NW); intr_restore(saveintr); } #ifdef CPU_I486_ON_386 /* * There are i486 based upgrade products for i386 machines. * In this case, BIOS doesn't enable CPU cache. */ static void init_i486_on_386(void) { register_t saveintr; #if defined(PC98) && !defined(CPU_UPGRADE_HW_CACHE) need_post_dma_flush = 1; #endif saveintr = intr_disable(); load_cr0(rcr0() & ~(CR0_CD | CR0_NW)); /* CD = 0, NW = 0 */ intr_restore(saveintr); } #endif /* * Cyrix 6x86 * * XXX - What should I do here? Please let me know. */ static void init_6x86(void) { register_t saveintr; u_char ccr3, ccr4; saveintr = intr_disable(); load_cr0(rcr0() | CR0_CD | CR0_NW); wbinvd(); /* Initialize CCR0. */ write_cyrix_reg(CCR0, read_cyrix_reg(CCR0) | CCR0_NC1); /* Initialize CCR1. */ #ifdef CPU_CYRIX_NO_LOCK write_cyrix_reg(CCR1, read_cyrix_reg(CCR1) | CCR1_NO_LOCK); #else write_cyrix_reg(CCR1, read_cyrix_reg(CCR1) & ~CCR1_NO_LOCK); #endif /* Initialize CCR2. */ #ifdef CPU_SUSP_HLT write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) | CCR2_SUSP_HLT); #else write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) & ~CCR2_SUSP_HLT); #endif ccr3 = read_cyrix_reg(CCR3); write_cyrix_reg(CCR3, CCR3_MAPEN0); /* Initialize CCR4. */ ccr4 = read_cyrix_reg(CCR4); ccr4 |= CCR4_DTE; ccr4 &= ~CCR4_IOMASK; #ifdef CPU_IORT write_cyrix_reg(CCR4, ccr4 | (CPU_IORT & CCR4_IOMASK)); #else write_cyrix_reg(CCR4, ccr4 | 7); #endif /* Initialize CCR5. */ #ifdef CPU_WT_ALLOC write_cyrix_reg(CCR5, read_cyrix_reg(CCR5) | CCR5_WT_ALLOC); #endif /* Restore CCR3. */ write_cyrix_reg(CCR3, ccr3); /* Unlock NW bit in CR0. */ write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) & ~CCR2_LOCK_NW); /* * Earlier revision of the 6x86 CPU could crash the system if * L1 cache is in write-back mode. */ if ((cyrix_did & 0xff00) > 0x1600) load_cr0(rcr0() & ~(CR0_CD | CR0_NW)); /* CD = 0 and NW = 0 */ else { /* Revision 2.6 and lower. */ #ifdef CYRIX_CACHE_REALLY_WORKS load_cr0(rcr0() & ~(CR0_CD | CR0_NW)); /* CD = 0 and NW = 0 */ #else load_cr0((rcr0() & ~CR0_CD) | CR0_NW); /* CD = 0 and NW = 1 */ #endif } /* Lock NW bit in CR0. */ write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) | CCR2_LOCK_NW); intr_restore(saveintr); } #endif /* I486_CPU */ #ifdef I586_CPU /* * Rise mP6 */ static void init_rise(void) { /* * The CMPXCHG8B instruction is always available but hidden. */ cpu_feature |= CPUID_CX8; } /* * IDT WinChip C6/2/2A/2B/3 * * http://www.centtech.com/winchip_bios_writers_guide_v4_0.pdf */ static void init_winchip(void) { u_int regs[4]; uint64_t fcr; fcr = rdmsr(0x0107); /* * Set ECX8, DSMC, DTLOCK/EDCTLB, EMMX, and ERETSTK and clear DPDC. */ fcr |= (1 << 1) | (1 << 7) | (1 << 8) | (1 << 9) | (1 << 16); fcr &= ~(1ULL << 11); /* * Additionally, set EBRPRED, E2MMX and EAMD3D for WinChip 2 and 3. */ if (CPUID_TO_MODEL(cpu_id) >= 8) fcr |= (1 << 12) | (1 << 19) | (1 << 20); wrmsr(0x0107, fcr); do_cpuid(1, regs); cpu_feature = regs[3]; } #endif #ifdef I686_CPU /* * Cyrix 6x86MX (code-named M2) * * XXX - What should I do here? Please let me know. */ static void init_6x86MX(void) { register_t saveintr; u_char ccr3, ccr4; saveintr = intr_disable(); load_cr0(rcr0() | CR0_CD | CR0_NW); wbinvd(); /* Initialize CCR0. */ write_cyrix_reg(CCR0, read_cyrix_reg(CCR0) | CCR0_NC1); /* Initialize CCR1. */ #ifdef CPU_CYRIX_NO_LOCK write_cyrix_reg(CCR1, read_cyrix_reg(CCR1) | CCR1_NO_LOCK); #else write_cyrix_reg(CCR1, read_cyrix_reg(CCR1) & ~CCR1_NO_LOCK); #endif /* Initialize CCR2. */ #ifdef CPU_SUSP_HLT write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) | CCR2_SUSP_HLT); #else write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) & ~CCR2_SUSP_HLT); #endif ccr3 = read_cyrix_reg(CCR3); write_cyrix_reg(CCR3, CCR3_MAPEN0); /* Initialize CCR4. */ ccr4 = read_cyrix_reg(CCR4); ccr4 &= ~CCR4_IOMASK; #ifdef CPU_IORT write_cyrix_reg(CCR4, ccr4 | (CPU_IORT & CCR4_IOMASK)); #else write_cyrix_reg(CCR4, ccr4 | 7); #endif /* Initialize CCR5. */ #ifdef CPU_WT_ALLOC write_cyrix_reg(CCR5, read_cyrix_reg(CCR5) | CCR5_WT_ALLOC); #endif /* Restore CCR3. */ write_cyrix_reg(CCR3, ccr3); /* Unlock NW bit in CR0. */ write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) & ~CCR2_LOCK_NW); load_cr0(rcr0() & ~(CR0_CD | CR0_NW)); /* CD = 0 and NW = 0 */ /* Lock NW bit in CR0. */ write_cyrix_reg(CCR2, read_cyrix_reg(CCR2) | CCR2_LOCK_NW); intr_restore(saveintr); } static int ppro_apic_used = -1; static void init_ppro(void) { u_int64_t apicbase; /* * Local APIC should be disabled if it is not going to be used. */ if (ppro_apic_used != 1) { apicbase = rdmsr(MSR_APICBASE); apicbase &= ~APICBASE_ENABLED; wrmsr(MSR_APICBASE, apicbase); ppro_apic_used = 0; } } /* * If the local APIC is going to be used after being disabled above, * re-enable it and don't disable it in the future. */ void ppro_reenable_apic(void) { u_int64_t apicbase; if (ppro_apic_used == 0) { apicbase = rdmsr(MSR_APICBASE); apicbase |= APICBASE_ENABLED; wrmsr(MSR_APICBASE, apicbase); ppro_apic_used = 1; } } /* * Initialize BBL_CR_CTL3 (Control register 3: used to configure the * L2 cache). */ static void init_mendocino(void) { #ifdef CPU_PPRO2CELERON register_t saveintr; u_int64_t bbl_cr_ctl3; saveintr = intr_disable(); load_cr0(rcr0() | CR0_CD | CR0_NW); wbinvd(); bbl_cr_ctl3 = rdmsr(MSR_BBL_CR_CTL3); /* If the L2 cache is configured, do nothing. */ if (!(bbl_cr_ctl3 & 1)) { bbl_cr_ctl3 = 0x134052bLL; /* Set L2 Cache Latency (Default: 5). */ #ifdef CPU_CELERON_L2_LATENCY #if CPU_L2_LATENCY > 15 #error invalid CPU_L2_LATENCY. #endif bbl_cr_ctl3 |= CPU_L2_LATENCY << 1; #else bbl_cr_ctl3 |= 5 << 1; #endif wrmsr(MSR_BBL_CR_CTL3, bbl_cr_ctl3); } load_cr0(rcr0() & ~(CR0_CD | CR0_NW)); intr_restore(saveintr); #endif /* CPU_PPRO2CELERON */ } /* * Initialize special VIA features */ static void init_via(void) { u_int regs[4], val; uint64_t fcr; /* * Explicitly enable CX8 and PGE on C3. * * http://www.via.com.tw/download/mainboards/6/13/VIA_C3_EBGA%20datasheet110.pdf */ if (CPUID_TO_MODEL(cpu_id) <= 9) fcr = (1 << 1) | (1 << 7); else fcr = 0; /* * Check extended CPUID for PadLock features. * * http://www.via.com.tw/en/downloads/whitepapers/initiatives/padlock/programming_guide.pdf */ do_cpuid(0xc0000000, regs); if (regs[0] >= 0xc0000001) { do_cpuid(0xc0000001, regs); val = regs[3]; } else val = 0; /* Enable RNG if present. */ if ((val & VIA_CPUID_HAS_RNG) != 0) { via_feature_rng = VIA_HAS_RNG; wrmsr(0x110B, rdmsr(0x110B) | VIA_CPUID_DO_RNG); } /* Enable PadLock if present. */ if ((val & VIA_CPUID_HAS_ACE) != 0) via_feature_xcrypt |= VIA_HAS_AES; if ((val & VIA_CPUID_HAS_ACE2) != 0) via_feature_xcrypt |= VIA_HAS_AESCTR; if ((val & VIA_CPUID_HAS_PHE) != 0) via_feature_xcrypt |= VIA_HAS_SHA; if ((val & VIA_CPUID_HAS_PMM) != 0) via_feature_xcrypt |= VIA_HAS_MM; if (via_feature_xcrypt != 0) fcr |= 1 << 28; wrmsr(0x1107, rdmsr(0x1107) | fcr); } #endif /* I686_CPU */ #if defined(I586_CPU) || defined(I686_CPU) static void init_transmeta(void) { u_int regs[0]; /* Expose all hidden features. */ wrmsr(0x80860004, rdmsr(0x80860004) | ~0UL); do_cpuid(1, regs); cpu_feature = regs[3]; } #endif extern int elf32_nxstack; void initializecpu(void) { switch (cpu) { #ifdef I486_CPU case CPU_BLUE: init_bluelightning(); break; case CPU_486DLC: init_486dlc(); break; case CPU_CY486DX: init_cy486dx(); break; case CPU_M1SC: init_5x86(); break; #ifdef CPU_I486_ON_386 case CPU_486: init_i486_on_386(); break; #endif case CPU_M1: init_6x86(); break; #endif /* I486_CPU */ #ifdef I586_CPU case CPU_586: switch (cpu_vendor_id) { case CPU_VENDOR_AMD: #ifdef CPU_WT_ALLOC if (((cpu_id & 0x0f0) > 0) && ((cpu_id & 0x0f0) < 0x60) && ((cpu_id & 0x00f) > 3)) enable_K5_wt_alloc(); else if (((cpu_id & 0x0f0) > 0x80) || (((cpu_id & 0x0f0) == 0x80) && (cpu_id & 0x00f) > 0x07)) enable_K6_2_wt_alloc(); else if ((cpu_id & 0x0f0) > 0x50) enable_K6_wt_alloc(); #endif if ((cpu_id & 0xf0) == 0xa0) /* * Make sure the TSC runs through * suspension, otherwise we can't use * it as timecounter */ wrmsr(0x1900, rdmsr(0x1900) | 0x20ULL); break; case CPU_VENDOR_CENTAUR: init_winchip(); break; case CPU_VENDOR_TRANSMETA: init_transmeta(); break; case CPU_VENDOR_RISE: init_rise(); break; } break; #endif #ifdef I686_CPU case CPU_M2: init_6x86MX(); break; case CPU_686: switch (cpu_vendor_id) { case CPU_VENDOR_INTEL: switch (cpu_id & 0xff0) { case 0x610: init_ppro(); break; case 0x660: init_mendocino(); break; } break; #ifdef CPU_ATHLON_SSE_HACK case CPU_VENDOR_AMD: /* * Sometimes the BIOS doesn't enable SSE instructions. * According to AMD document 20734, the mobile * Duron, the (mobile) Athlon 4 and the Athlon MP * support SSE. These correspond to cpu_id 0x66X * or 0x67X. */ if ((cpu_feature & CPUID_XMM) == 0 && ((cpu_id & ~0xf) == 0x660 || (cpu_id & ~0xf) == 0x670 || (cpu_id & ~0xf) == 0x680)) { u_int regs[4]; wrmsr(MSR_HWCR, rdmsr(MSR_HWCR) & ~0x08000); do_cpuid(1, regs); cpu_feature = regs[3]; } break; #endif case CPU_VENDOR_CENTAUR: init_via(); break; case CPU_VENDOR_TRANSMETA: init_transmeta(); break; } -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((amd_feature & AMDID_NX) != 0) { uint64_t msr; msr = rdmsr(MSR_EFER) | EFER_NXE; wrmsr(MSR_EFER, msr); pg_nx = PG_NX; elf32_nxstack = 1; } #endif break; #endif default: break; } #if defined(CPU_ENABLE_SSE) if ((cpu_feature & CPUID_XMM) && (cpu_feature & CPUID_FXSR)) { load_cr4(rcr4() | CR4_FXSR | CR4_XMM); cpu_fxsr = hw_instruction_sse = 1; } #endif } void initializecpucache(void) { /* * CPUID with %eax = 1, %ebx returns * Bits 15-8: CLFLUSH line size * (Value * 8 = cache line size in bytes) */ if ((cpu_feature & CPUID_CLFSH) != 0) cpu_clflush_line_size = ((cpu_procinfo >> 8) & 0xff) * 8; /* * XXXKIB: (temporary) hack to work around traps generated * when CLFLUSHing APIC register window under virtualization * environments. These environments tend to disable the * CPUID_SS feature even though the native CPU supports it. */ TUNABLE_INT_FETCH("hw.clflush_disable", &hw_clflush_disable); if (vm_guest != VM_GUEST_NO && hw_clflush_disable == -1) cpu_feature &= ~CPUID_CLFSH; /* * Allow to disable CLFLUSH feature manually by * hw.clflush_disable tunable. */ if (hw_clflush_disable == 1) cpu_feature &= ~CPUID_CLFSH; #if defined(PC98) && !defined(CPU_UPGRADE_HW_CACHE) /* * OS should flush L1 cache by itself because no PC-98 supports * non-Intel CPUs. Use wbinvd instruction before DMA transfer * when need_pre_dma_flush = 1, use invd instruction after DMA * transfer when need_post_dma_flush = 1. If your CPU upgrade * product supports hardware cache control, you can add the * CPU_UPGRADE_HW_CACHE option in your kernel configuration file. * This option eliminates unneeded cache flush instruction(s). */ if (cpu_vendor_id == CPU_VENDOR_CYRIX) { switch (cpu) { #ifdef I486_CPU case CPU_486DLC: need_post_dma_flush = 1; break; case CPU_M1SC: need_pre_dma_flush = 1; break; case CPU_CY486DX: need_pre_dma_flush = 1; #ifdef CPU_I486_ON_386 need_post_dma_flush = 1; #endif break; #endif default: break; } } else if (cpu_vendor_id == CPU_VENDOR_AMD) { switch (cpu_id & 0xFF0) { case 0x470: /* Enhanced Am486DX2 WB */ case 0x490: /* Enhanced Am486DX4 WB */ case 0x4F0: /* Am5x86 WB */ need_pre_dma_flush = 1; break; } } else if (cpu_vendor_id == CPU_VENDOR_IBM) { need_post_dma_flush = 1; } else { #ifdef CPU_I486_ON_386 need_pre_dma_flush = 1; #endif } #endif /* PC98 && !CPU_UPGRADE_HW_CACHE */ } #if defined(I586_CPU) && defined(CPU_WT_ALLOC) /* * Enable write allocate feature of AMD processors. * Following two functions require the Maxmem variable being set. */ static void enable_K5_wt_alloc(void) { u_int64_t msr; register_t saveintr; /* * Write allocate is supported only on models 1, 2, and 3, with * a stepping of 4 or greater. */ if (((cpu_id & 0xf0) > 0) && ((cpu_id & 0x0f) > 3)) { saveintr = intr_disable(); msr = rdmsr(0x83); /* HWCR */ wrmsr(0x83, msr & !(0x10)); /* * We have to tell the chip where the top of memory is, * since video cards could have frame bufferes there, * memory-mapped I/O could be there, etc. */ if(Maxmem > 0) msr = Maxmem / 16; else msr = 0; msr |= AMD_WT_ALLOC_TME | AMD_WT_ALLOC_FRE; #ifdef PC98 if (!(inb(0x43b) & 4)) { wrmsr(0x86, 0x0ff00f0); msr |= AMD_WT_ALLOC_PRE; } #else /* * There is no way to know wheter 15-16M hole exists or not. * Therefore, we disable write allocate for this range. */ wrmsr(0x86, 0x0ff00f0); msr |= AMD_WT_ALLOC_PRE; #endif wrmsr(0x85, msr); msr=rdmsr(0x83); wrmsr(0x83, msr|0x10); /* enable write allocate */ intr_restore(saveintr); } } static void enable_K6_wt_alloc(void) { quad_t size; u_int64_t whcr; register_t saveintr; saveintr = intr_disable(); wbinvd(); #ifdef CPU_DISABLE_CACHE /* * Certain K6-2 box becomes unstable when write allocation is * enabled. */ /* * The AMD-K6 processer provides the 64-bit Test Register 12(TR12), * but only the Cache Inhibit(CI) (bit 3 of TR12) is suppported. * All other bits in TR12 have no effect on the processer's operation. * The I/O Trap Restart function (bit 9 of TR12) is always enabled * on the AMD-K6. */ wrmsr(0x0000000e, (u_int64_t)0x0008); #endif /* Don't assume that memory size is aligned with 4M. */ if (Maxmem > 0) size = ((Maxmem >> 8) + 3) >> 2; else size = 0; /* Limit is 508M bytes. */ if (size > 0x7f) size = 0x7f; whcr = (rdmsr(0xc0000082) & ~(0x7fLL << 1)) | (size << 1); #if defined(PC98) || defined(NO_MEMORY_HOLE) if (whcr & (0x7fLL << 1)) { #ifdef PC98 /* * If bit 2 of port 0x43b is 0, disable wrte allocate for the * 15-16M range. */ if (!(inb(0x43b) & 4)) whcr &= ~0x0001LL; else #endif whcr |= 0x0001LL; } #else /* * There is no way to know wheter 15-16M hole exists or not. * Therefore, we disable write allocate for this range. */ whcr &= ~0x0001LL; #endif wrmsr(0x0c0000082, whcr); intr_restore(saveintr); } static void enable_K6_2_wt_alloc(void) { quad_t size; u_int64_t whcr; register_t saveintr; saveintr = intr_disable(); wbinvd(); #ifdef CPU_DISABLE_CACHE /* * Certain K6-2 box becomes unstable when write allocation is * enabled. */ /* * The AMD-K6 processer provides the 64-bit Test Register 12(TR12), * but only the Cache Inhibit(CI) (bit 3 of TR12) is suppported. * All other bits in TR12 have no effect on the processer's operation. * The I/O Trap Restart function (bit 9 of TR12) is always enabled * on the AMD-K6. */ wrmsr(0x0000000e, (u_int64_t)0x0008); #endif /* Don't assume that memory size is aligned with 4M. */ if (Maxmem > 0) size = ((Maxmem >> 8) + 3) >> 2; else size = 0; /* Limit is 4092M bytes. */ if (size > 0x3fff) size = 0x3ff; whcr = (rdmsr(0xc0000082) & ~(0x3ffLL << 22)) | (size << 22); #if defined(PC98) || defined(NO_MEMORY_HOLE) if (whcr & (0x3ffLL << 22)) { #ifdef PC98 /* * If bit 2 of port 0x43b is 0, disable wrte allocate for the * 15-16M range. */ if (!(inb(0x43b) & 4)) whcr &= ~(1LL << 16); else #endif whcr |= 1LL << 16; } #else /* * There is no way to know wheter 15-16M hole exists or not. * Therefore, we disable write allocate for this range. */ whcr &= ~(1LL << 16); #endif wrmsr(0x0c0000082, whcr); intr_restore(saveintr); } #endif /* I585_CPU && CPU_WT_ALLOC */ #include "opt_ddb.h" #ifdef DDB #include DB_SHOW_COMMAND(cyrixreg, cyrixreg) { register_t saveintr; u_int cr0; u_char ccr1, ccr2, ccr3; u_char ccr0 = 0, ccr4 = 0, ccr5 = 0, pcr0 = 0; cr0 = rcr0(); if (cpu_vendor_id == CPU_VENDOR_CYRIX) { saveintr = intr_disable(); if ((cpu != CPU_M1SC) && (cpu != CPU_CY486DX)) { ccr0 = read_cyrix_reg(CCR0); } ccr1 = read_cyrix_reg(CCR1); ccr2 = read_cyrix_reg(CCR2); ccr3 = read_cyrix_reg(CCR3); if ((cpu == CPU_M1SC) || (cpu == CPU_M1) || (cpu == CPU_M2)) { write_cyrix_reg(CCR3, CCR3_MAPEN0); ccr4 = read_cyrix_reg(CCR4); if ((cpu == CPU_M1) || (cpu == CPU_M2)) ccr5 = read_cyrix_reg(CCR5); else pcr0 = read_cyrix_reg(PCR0); write_cyrix_reg(CCR3, ccr3); /* Restore CCR3. */ } intr_restore(saveintr); if ((cpu != CPU_M1SC) && (cpu != CPU_CY486DX)) printf("CCR0=%x, ", (u_int)ccr0); printf("CCR1=%x, CCR2=%x, CCR3=%x", (u_int)ccr1, (u_int)ccr2, (u_int)ccr3); if ((cpu == CPU_M1SC) || (cpu == CPU_M1) || (cpu == CPU_M2)) { printf(", CCR4=%x, ", (u_int)ccr4); if (cpu == CPU_M1SC) printf("PCR0=%x\n", pcr0); else printf("CCR5=%x\n", ccr5); } } printf("CR0=%x\n", cr0); } #endif /* DDB */ Index: user/ngie/more-tests/sys/i386/i386/locore.s =================================================================== --- user/ngie/more-tests/sys/i386/i386/locore.s (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/locore.s (revision 281504) @@ -1,909 +1,909 @@ /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)locore.s 7.3 (Berkeley) 5/13/91 * $FreeBSD$ * * originally from: locore.s, by William F. Jolitz * * Substantially rewritten by David Greenman, Rod Grimes, * Bruce Evans, Wolfgang Solfrank, Poul-Henning Kamp * and many others. */ #include "opt_bootp.h" #include "opt_compat.h" #include "opt_nfsroot.h" #include "opt_pmap.h" #include #include #include #include #include #include #include #include "assym.s" /* * XXX * * Note: This version greatly munged to avoid various assembler errors * that may be fixed in newer versions of gas. Perhaps newer versions * will have more pleasant appearance. */ /* * PTmap is recursive pagemap at top of virtual address space. * Within PTmap, the page directory can be found (third indirection). */ .globl PTmap,PTD,PTDpde .set PTmap,(PTDPTDI << PDRSHIFT) .set PTD,PTmap + (PTDPTDI * PAGE_SIZE) .set PTDpde,PTD + (PTDPTDI * PDESIZE) /* * Compiled KERNBASE location and the kernel load address */ .globl kernbase .set kernbase,KERNBASE .globl kernload .set kernload,KERNLOAD /* * Globals */ .data ALIGN_DATA /* just to be sure */ .space 0x2000 /* space for tmpstk - temporary stack */ tmpstk: .globl bootinfo bootinfo: .space BOOTINFO_SIZE /* bootinfo that we can handle */ .globl KERNend KERNend: .long 0 /* phys addr end of kernel (just after bss) */ physfree: .long 0 /* phys addr of next free page */ .globl IdlePTD IdlePTD: .long 0 /* phys addr of kernel PTD */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) .globl IdlePDPT IdlePDPT: .long 0 /* phys addr of kernel PDPT */ #endif .globl KPTmap KPTmap: .long 0 /* address of kernel page tables */ .globl KPTphys KPTphys: .long 0 /* phys addr of kernel page tables */ .globl proc0kstack proc0kstack: .long 0 /* address of proc 0 kstack space */ p0kpa: .long 0 /* phys addr of proc0's STACK */ vm86phystk: .long 0 /* PA of vm86/bios stack */ .globl vm86paddr, vm86pa vm86paddr: .long 0 /* address of vm86 region */ vm86pa: .long 0 /* phys addr of vm86 region */ #ifdef PC98 .globl pc98_system_parameter pc98_system_parameter: .space 0x240 #endif /********************************************************************** * * Some handy macros * */ #define R(foo) ((foo)-KERNBASE) #define ALLOCPAGES(foo) \ movl R(physfree), %esi ; \ movl $((foo)*PAGE_SIZE), %eax ; \ addl %esi, %eax ; \ movl %eax, R(physfree) ; \ movl %esi, %edi ; \ movl $((foo)*PAGE_SIZE),%ecx ; \ xorl %eax,%eax ; \ cld ; \ rep ; \ stosb /* * fillkpt * eax = page frame address * ebx = index into page table * ecx = how many pages to map * base = base address of page dir/table * prot = protection bits */ #define fillkpt(base, prot) \ shll $PTESHIFT,%ebx ; \ addl base,%ebx ; \ orl $PG_V,%eax ; \ orl prot,%eax ; \ 1: movl %eax,(%ebx) ; \ addl $PAGE_SIZE,%eax ; /* increment physical address */ \ addl $PTESIZE,%ebx ; /* next pte */ \ loop 1b /* * fillkptphys(prot) * eax = physical address * ecx = how many pages to map * prot = protection bits */ #define fillkptphys(prot) \ movl %eax, %ebx ; \ shrl $PAGE_SHIFT, %ebx ; \ fillkpt(R(KPTphys), prot) .text /********************************************************************** * * This is where the bootblocks start us, set the ball rolling... * */ NON_GPROF_ENTRY(btext) #ifdef PC98 /* save SYSTEM PARAMETER for resume (NS/T or other) */ movl $0xa1400,%esi movl $R(pc98_system_parameter),%edi movl $0x0240,%ecx cld rep movsb #else /* IBM-PC */ /* Tell the bios to warmboot next time */ movw $0x1234,0x472 #endif /* PC98 */ /* Set up a real frame in case the double return in newboot is executed. */ pushl %ebp movl %esp, %ebp /* Don't trust what the BIOS gives for eflags. */ pushl $PSL_KERNEL popfl /* * Don't trust what the BIOS gives for %fs and %gs. Trust the bootstrap * to set %cs, %ds, %es and %ss. */ mov %ds, %ax mov %ax, %fs mov %ax, %gs /* * Clear the bss. Not all boot programs do it, and it is our job anyway. * * XXX we don't check that there is memory for our bss and page tables * before using it. * * Note: we must be careful to not overwrite an active gdt or idt. They * inactive from now until we switch to new ones, since we don't load any * more segment registers or permit interrupts until after the switch. */ movl $R(end),%ecx movl $R(edata),%edi subl %edi,%ecx xorl %eax,%eax cld rep stosb call recover_bootinfo /* Get onto a stack that we can trust. */ /* * XXX this step is delayed in case recover_bootinfo needs to return via * the old stack, but it need not be, since recover_bootinfo actually * returns via the old frame. */ movl $R(tmpstk),%esp #ifdef PC98 /* pc98_machine_type & M_EPSON_PC98 */ testb $0x02,R(pc98_system_parameter)+220 jz 3f /* epson_machine_id <= 0x0b */ cmpb $0x0b,R(pc98_system_parameter)+224 ja 3f /* count up memory */ movl $0x100000,%eax /* next, talley remaining memory */ movl $0xFFF-0x100,%ecx 1: movl 0(%eax),%ebx /* save location to check */ movl $0xa55a5aa5,0(%eax) /* write test pattern */ cmpl $0xa55a5aa5,0(%eax) /* does not check yet for rollover */ jne 2f movl %ebx,0(%eax) /* restore memory */ addl $PAGE_SIZE,%eax loop 1b 2: subl $0x100000,%eax shrl $17,%eax movb %al,R(pc98_system_parameter)+1 3: movw R(pc98_system_parameter+0x86),%ax movw %ax,R(cpu_id) #endif call identify_cpu call create_pagetables /* * If the CPU has support for VME, turn it on. */ testl $CPUID_VME, R(cpu_feature) jz 1f movl %cr4, %eax orl $CR4_VME, %eax movl %eax, %cr4 1: /* Now enable paging */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) movl R(IdlePDPT), %eax movl %eax, %cr3 movl %cr4, %eax orl $CR4_PAE, %eax movl %eax, %cr4 #else movl R(IdlePTD), %eax movl %eax,%cr3 /* load ptd addr into mmu */ #endif movl %cr0,%eax /* get control word */ orl $CR0_PE|CR0_PG,%eax /* enable paging */ movl %eax,%cr0 /* and let's page NOW! */ pushl $begin /* jump to high virtualized address */ ret /* now running relocated at KERNBASE where the system is linked to run */ begin: /* set up bootstrap stack */ movl proc0kstack,%eax /* location of in-kernel stack */ /* * Only use bottom page for init386(). init386() calculates the * PCB + FPU save area size and returns the true top of stack. */ leal PAGE_SIZE(%eax),%esp xorl %ebp,%ebp /* mark end of frames */ pushl physfree /* value of first for init386(first) */ call init386 /* wire 386 chip for unix operation */ /* * Clean up the stack in a way that db_numargs() understands, so * that backtraces in ddb don't underrun the stack. Traps for * inaccessible memory are more fatal than usual this early. */ addl $4,%esp /* Switch to true top of stack. */ movl %eax,%esp call mi_startup /* autoconfiguration, mountroot etc */ /* NOTREACHED */ addl $0,%esp /* for db_numargs() again */ /* * Signal trampoline, copied to top of user stack */ NON_GPROF_ENTRY(sigcode) calll *SIGF_HANDLER(%esp) leal SIGF_UC(%esp),%eax /* get ucontext */ pushl %eax testl $PSL_VM,UC_EFLAGS(%eax) jne 1f mov UC_GS(%eax),%gs /* restore %gs */ 1: movl $SYS_sigreturn,%eax pushl %eax /* junk to fake return addr. */ int $0x80 /* enter kernel with args */ /* on stack */ 1: jmp 1b #ifdef COMPAT_FREEBSD4 ALIGN_TEXT freebsd4_sigcode: calll *SIGF_HANDLER(%esp) leal SIGF_UC4(%esp),%eax /* get ucontext */ pushl %eax testl $PSL_VM,UC4_EFLAGS(%eax) jne 1f mov UC4_GS(%eax),%gs /* restore %gs */ 1: movl $344,%eax /* 4.x SYS_sigreturn */ pushl %eax /* junk to fake return addr. */ int $0x80 /* enter kernel with args */ /* on stack */ 1: jmp 1b #endif #ifdef COMPAT_43 ALIGN_TEXT osigcode: call *SIGF_HANDLER(%esp) /* call signal handler */ lea SIGF_SC(%esp),%eax /* get sigcontext */ pushl %eax testl $PSL_VM,SC_PS(%eax) jne 9f mov SC_GS(%eax),%gs /* restore %gs */ 9: movl $103,%eax /* 3.x SYS_sigreturn */ pushl %eax /* junk to fake return addr. */ int $0x80 /* enter kernel with args */ 0: jmp 0b #endif /* COMPAT_43 */ ALIGN_TEXT esigcode: .data .globl szsigcode szsigcode: .long esigcode-sigcode #ifdef COMPAT_FREEBSD4 .globl szfreebsd4_sigcode szfreebsd4_sigcode: .long esigcode-freebsd4_sigcode #endif #ifdef COMPAT_43 .globl szosigcode szosigcode: .long esigcode-osigcode #endif .text /********************************************************************** * * Recover the bootinfo passed to us from the boot program * */ recover_bootinfo: /* * This code is called in different ways depending on what loaded * and started the kernel. This is used to detect how we get the * arguments from the other code and what we do with them. * * Old disk boot blocks: * (*btext)(howto, bootdev, cyloffset, esym); * [return address == 0, and can NOT be returned to] * [cyloffset was not supported by the FreeBSD boot code * and always passed in as 0] * [esym is also known as total in the boot code, and * was never properly supported by the FreeBSD boot code] * * Old diskless netboot code: * (*btext)(0,0,0,0,&nfsdiskless,0,0,0); * [return address != 0, and can NOT be returned to] * If we are being booted by this code it will NOT work, * so we are just going to halt if we find this case. * * New uniform boot code: * (*btext)(howto, bootdev, 0, 0, 0, &bootinfo) * [return address != 0, and can be returned to] * * There may seem to be a lot of wasted arguments in here, but * that is so the newer boot code can still load very old kernels * and old boot code can load new kernels. */ /* * The old style disk boot blocks fake a frame on the stack and * did an lret to get here. The frame on the stack has a return * address of 0. */ cmpl $0,4(%ebp) je olddiskboot /* * We have some form of return address, so this is either the * old diskless netboot code, or the new uniform code. That can * be detected by looking at the 5th argument, if it is 0 * we are being booted by the new uniform boot code. */ cmpl $0,24(%ebp) je newboot /* * Seems we have been loaded by the old diskless boot code, we * don't stand a chance of running as the diskless structure * changed considerably between the two, so just halt. */ hlt /* * We have been loaded by the new uniform boot code. * Let's check the bootinfo version, and if we do not understand * it we return to the loader with a status of 1 to indicate this error */ newboot: movl 28(%ebp),%ebx /* &bootinfo.version */ movl BI_VERSION(%ebx),%eax cmpl $1,%eax /* We only understand version 1 */ je 1f movl $1,%eax /* Return status */ leave /* * XXX this returns to our caller's caller (as is required) since * we didn't set up a frame and our caller did. */ ret 1: /* * If we have a kernelname copy it in */ movl BI_KERNELNAME(%ebx),%esi cmpl $0,%esi je 2f /* No kernelname */ movl $MAXPATHLEN,%ecx /* Brute force!!! */ movl $R(kernelname),%edi cmpb $'/',(%esi) /* Make sure it starts with a slash */ je 1f movb $'/',(%edi) incl %edi decl %ecx 1: cld rep movsb 2: /* * Determine the size of the boot loader's copy of the bootinfo * struct. This is impossible to do properly because old versions * of the struct don't contain a size field and there are 2 old * versions with the same version number. */ movl $BI_ENDCOMMON,%ecx /* prepare for sizeless version */ testl $RB_BOOTINFO,8(%ebp) /* bi_size (and bootinfo) valid? */ je got_bi_size /* no, sizeless version */ movl BI_SIZE(%ebx),%ecx got_bi_size: /* * Copy the common part of the bootinfo struct */ movl %ebx,%esi movl $R(bootinfo),%edi cmpl $BOOTINFO_SIZE,%ecx jbe got_common_bi_size movl $BOOTINFO_SIZE,%ecx got_common_bi_size: cld rep movsb #ifdef NFS_ROOT #ifndef BOOTP_NFSV3 /* * If we have a nfs_diskless structure copy it in */ movl BI_NFS_DISKLESS(%ebx),%esi cmpl $0,%esi je olddiskboot movl $R(nfs_diskless),%edi movl $NFSDISKLESS_SIZE,%ecx cld rep movsb movl $R(nfs_diskless_valid),%edi movl $1,(%edi) #endif #endif /* * The old style disk boot. * (*btext)(howto, bootdev, cyloffset, esym); * Note that the newer boot code just falls into here to pick * up howto and bootdev, cyloffset and esym are no longer used */ olddiskboot: movl 8(%ebp),%eax movl %eax,R(boothowto) movl 12(%ebp),%eax movl %eax,R(bootdev) ret /********************************************************************** * * Identify the CPU and initialize anything special about it * */ identify_cpu: /* Try to toggle alignment check flag; does not exist on 386. */ pushfl popl %eax movl %eax,%ecx orl $PSL_AC,%eax pushl %eax popfl pushfl popl %eax xorl %ecx,%eax andl $PSL_AC,%eax pushl %ecx popfl testl %eax,%eax jnz try486 /* NexGen CPU does not have aligment check flag. */ pushfl movl $0x5555, %eax xorl %edx, %edx movl $2, %ecx clc divl %ecx jz trynexgen popfl movl $CPU_386,R(cpu) jmp 3f trynexgen: popfl movl $CPU_NX586,R(cpu) movl $0x4778654e,R(cpu_vendor) # store vendor string movl $0x72446e65,R(cpu_vendor+4) movl $0x6e657669,R(cpu_vendor+8) movl $0,R(cpu_vendor+12) jmp 3f try486: /* Try to toggle identification flag; does not exist on early 486s. */ pushfl popl %eax movl %eax,%ecx xorl $PSL_ID,%eax pushl %eax popfl pushfl popl %eax xorl %ecx,%eax andl $PSL_ID,%eax pushl %ecx popfl testl %eax,%eax jnz trycpuid movl $CPU_486,R(cpu) /* * Check Cyrix CPU * Cyrix CPUs do not change the undefined flags following * execution of the divide instruction which divides 5 by 2. * * Note: CPUID is enabled on M2, so it passes another way. */ pushfl movl $0x5555, %eax xorl %edx, %edx movl $2, %ecx clc divl %ecx jnc trycyrix popfl jmp 3f /* You may use Intel CPU. */ trycyrix: popfl /* * IBM Bluelighting CPU also doesn't change the undefined flags. * Because IBM doesn't disclose the information for Bluelighting * CPU, we couldn't distinguish it from Cyrix's (including IBM * brand of Cyrix CPUs). */ movl $0x69727943,R(cpu_vendor) # store vendor string movl $0x736e4978,R(cpu_vendor+4) movl $0x64616574,R(cpu_vendor+8) jmp 3f trycpuid: /* Use the `cpuid' instruction. */ xorl %eax,%eax cpuid # cpuid 0 movl %eax,R(cpu_high) # highest capability movl %ebx,R(cpu_vendor) # store vendor string movl %edx,R(cpu_vendor+4) movl %ecx,R(cpu_vendor+8) movb $0,R(cpu_vendor+12) movl $1,%eax cpuid # cpuid 1 movl %eax,R(cpu_id) # store cpu_id movl %ebx,R(cpu_procinfo) # store cpu_procinfo movl %edx,R(cpu_feature) # store cpu_feature movl %ecx,R(cpu_feature2) # store cpu_feature2 rorl $8,%eax # extract family type andl $15,%eax cmpl $5,%eax jae 1f /* less than Pentium; must be 486 */ movl $CPU_486,R(cpu) jmp 3f 1: /* a Pentium? */ cmpl $5,%eax jne 2f movl $CPU_586,R(cpu) jmp 3f 2: /* Greater than Pentium...call it a Pentium Pro */ movl $CPU_686,R(cpu) 3: ret /********************************************************************** * * Create the first page directory and its page tables. * */ create_pagetables: /* Find end of kernel image (rounded up to a page boundary). */ movl $R(_end),%esi /* Include symbols, if any. */ movl R(bootinfo+BI_ESYMTAB),%edi testl %edi,%edi je over_symalloc movl %edi,%esi movl $KERNBASE,%edi addl %edi,R(bootinfo+BI_SYMTAB) addl %edi,R(bootinfo+BI_ESYMTAB) over_symalloc: /* If we are told where the end of the kernel space is, believe it. */ movl R(bootinfo+BI_KERNEND),%edi testl %edi,%edi je no_kernend movl %edi,%esi no_kernend: addl $PDRMASK,%esi /* Play conservative for now, and */ andl $~PDRMASK,%esi /* ... wrap to next 4M. */ movl %esi,R(KERNend) /* save end of kernel */ movl %esi,R(physfree) /* next free page is at end of kernel */ /* Allocate Kernel Page Tables */ ALLOCPAGES(NKPT) movl %esi,R(KPTphys) addl $(KERNBASE-(KPTDI<<(PDRSHIFT-PAGE_SHIFT+PTESHIFT))),%esi movl %esi,R(KPTmap) /* Allocate Page Table Directory */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) /* XXX only need 32 bytes (easier for now) */ ALLOCPAGES(1) movl %esi,R(IdlePDPT) #endif ALLOCPAGES(NPGPTD) movl %esi,R(IdlePTD) /* Allocate KSTACK */ ALLOCPAGES(KSTACK_PAGES) movl %esi,R(p0kpa) addl $KERNBASE, %esi movl %esi, R(proc0kstack) ALLOCPAGES(1) /* vm86/bios stack */ movl %esi,R(vm86phystk) ALLOCPAGES(3) /* pgtable + ext + IOPAGES */ movl %esi,R(vm86pa) addl $KERNBASE, %esi movl %esi, R(vm86paddr) /* * Enable PSE and PGE. */ #ifndef DISABLE_PSE testl $CPUID_PSE, R(cpu_feature) jz 1f movl $PG_PS, R(pseflag) movl %cr4, %eax orl $CR4_PSE, %eax movl %eax, %cr4 1: #endif #ifndef DISABLE_PG_G testl $CPUID_PGE, R(cpu_feature) jz 2f movl $PG_G, R(pgeflag) movl %cr4, %eax orl $CR4_PGE, %eax movl %eax, %cr4 2: #endif /* * Initialize page table pages mapping physical address zero through the * end of the kernel. All of the page table entries allow read and write * access. Write access to the first physical page is required by bios32 * calls, and write access to the first 1 MB of physical memory is required * by ACPI for implementing suspend and resume. We do this even * if we've enabled PSE above, we'll just switch the corresponding kernel * PDEs before we turn on paging. * * XXX: We waste some pages here in the PSE case! */ xorl %eax, %eax movl R(KERNend),%ecx shrl $PAGE_SHIFT,%ecx fillkptphys($PG_RW) /* Map page table pages. */ movl R(KPTphys),%eax movl $NKPT,%ecx fillkptphys($PG_RW) /* Map page directory. */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) movl R(IdlePDPT), %eax movl $1, %ecx fillkptphys($PG_RW) #endif movl R(IdlePTD), %eax movl $NPGPTD, %ecx fillkptphys($PG_RW) /* Map proc0's KSTACK in the physical way ... */ movl R(p0kpa), %eax movl $(KSTACK_PAGES), %ecx fillkptphys($PG_RW) /* Map ISA hole */ movl $ISA_HOLE_START, %eax movl $ISA_HOLE_LENGTH>>PAGE_SHIFT, %ecx fillkptphys($PG_RW) /* Map space for the vm86 region */ movl R(vm86phystk), %eax movl $4, %ecx fillkptphys($PG_RW) /* Map page 0 into the vm86 page table */ movl $0, %eax movl $0, %ebx movl $1, %ecx fillkpt(R(vm86pa), $PG_RW|PG_U) /* ...likewise for the ISA hole */ movl $ISA_HOLE_START, %eax movl $ISA_HOLE_START>>PAGE_SHIFT, %ebx movl $ISA_HOLE_LENGTH>>PAGE_SHIFT, %ecx fillkpt(R(vm86pa), $PG_RW|PG_U) /* * Create an identity mapping for low physical memory, including the kernel. * The part of this mapping that covers the first 1 MB of physical memory * becomes a permanent part of the kernel's address space. The rest of this * mapping is destroyed in pmap_bootstrap(). Ordinarily, the same page table * pages are shared by the identity mapping and the kernel's native mapping. * However, the permanent identity mapping cannot contain PG_G mappings. * Thus, if the kernel is loaded within the permanent identity mapping, that * page table page must be duplicated and not shared. * * N.B. Due to errata concerning large pages and physical address zero, * a PG_PS mapping is not used. */ movl R(KPTphys), %eax xorl %ebx, %ebx movl $NKPT, %ecx fillkpt(R(IdlePTD), $PG_RW) #if KERNLOAD < (1 << PDRSHIFT) testl $PG_G, R(pgeflag) jz 1f ALLOCPAGES(1) movl %esi, %edi movl R(IdlePTD), %eax movl (%eax), %esi movl %edi, (%eax) movl $PAGE_SIZE, %ecx cld rep movsb 1: #endif /* * For the non-PSE case, install PDEs for PTs covering the KVA. * For the PSE case, do the same, but clobber the ones corresponding * to the kernel (from btext to KERNend) with 4M (2M for PAE) ('PS') * PDEs immediately after. */ movl R(KPTphys), %eax movl $KPTDI, %ebx movl $NKPT, %ecx fillkpt(R(IdlePTD), $PG_RW) cmpl $0,R(pseflag) je done_pde movl R(KERNend), %ecx movl $KERNLOAD, %eax subl %eax, %ecx shrl $PDRSHIFT, %ecx movl $(KPTDI+(KERNLOAD/(1 << PDRSHIFT))), %ebx shll $PDESHIFT, %ebx addl R(IdlePTD), %ebx orl $(PG_V|PG_RW|PG_PS), %eax 1: movl %eax, (%ebx) addl $(1 << PDRSHIFT), %eax addl $PDESIZE, %ebx loop 1b done_pde: /* install a pde recursively mapping page directory as a page table */ movl R(IdlePTD), %eax movl $PTDPTDI, %ebx movl $NPGPTD,%ecx fillkpt(R(IdlePTD), $PG_RW) -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) movl R(IdlePTD), %eax xorl %ebx, %ebx movl $NPGPTD, %ecx fillkpt(R(IdlePDPT), $0x0) #endif ret #ifdef XENHVM /* Xen Hypercall page */ .text .p2align PAGE_SHIFT, 0x90 /* Hypercall_page needs to be PAGE aligned */ NON_GPROF_ENTRY(hypercall_page) .skip 0x1000, 0x90 /* Fill with "nop"s */ #endif Index: user/ngie/more-tests/sys/i386/i386/machdep.c =================================================================== --- user/ngie/more-tests/sys/i386/i386/machdep.c (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/machdep.c (revision 281504) @@ -1,4195 +1,4195 @@ /*- * Copyright (c) 1992 Terrence R. Lambert. * Copyright (c) 1982, 1987, 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)machdep.c 7.4 (Berkeley) 6/3/91 */ #include __FBSDID("$FreeBSD$"); #include "opt_apic.h" #include "opt_atpic.h" #include "opt_compat.h" #include "opt_cpu.h" #include "opt_ddb.h" #include "opt_inet.h" #include "opt_isa.h" #include "opt_kstack_pages.h" #include "opt_maxmem.h" #include "opt_mp_watchdog.h" #include "opt_npx.h" #include "opt_perfmon.h" #include "opt_platform.h" #include "opt_xbox.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef SMP #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DDB #ifndef KDB #error KDB must be enabled in order for DDB to work! #endif #include #include #endif #ifdef PC98 #include #else #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef PERFMON #include #endif #ifdef SMP #include #endif #ifdef FDT #include #endif #ifdef DEV_APIC #include #endif #ifdef DEV_ISA #include #endif #ifdef XBOX #include int arch_i386_is_xbox = 0; uint32_t arch_i386_xbox_memsize = 0; #endif #ifdef XEN /* XEN includes */ #include #include #include #include #include void Xhypervisor_callback(void); void failsafe_callback(void); extern trap_info_t trap_table[]; struct proc_ldt default_proc_ldt; extern int init_first; int running_xen = 1; extern unsigned long physfree; #endif /* XEN */ /* Sanity check for __curthread() */ CTASSERT(offsetof(struct pcpu, pc_curthread) == 0); extern register_t init386(int first); extern void dblfault_handler(void); #define CS_SECURE(cs) (ISPL(cs) == SEL_UPL) #define EFL_SECURE(ef, oef) ((((ef) ^ (oef)) & ~PSL_USERCHANGE) == 0) #if !defined(CPU_DISABLE_SSE) && defined(I686_CPU) #define CPU_ENABLE_SSE #endif static void cpu_startup(void *); static void fpstate_drop(struct thread *td); static void get_fpcontext(struct thread *td, mcontext_t *mcp, char *xfpusave, size_t xfpusave_len); static int set_fpcontext(struct thread *td, mcontext_t *mcp, char *xfpustate, size_t xfpustate_len); #ifdef CPU_ENABLE_SSE static void set_fpregs_xmm(struct save87 *, struct savexmm *); static void fill_fpregs_xmm(struct savexmm *, struct save87 *); #endif /* CPU_ENABLE_SSE */ SYSINIT(cpu, SI_SUB_CPU, SI_ORDER_FIRST, cpu_startup, NULL); /* Intel ICH registers */ #define ICH_PMBASE 0x400 #define ICH_SMI_EN ICH_PMBASE + 0x30 int _udatasel, _ucodesel; u_int basemem; #ifdef PC98 int need_pre_dma_flush; /* If 1, use wbinvd befor DMA transfer. */ int need_post_dma_flush; /* If 1, use invd after DMA transfer. */ static int ispc98 = 1; SYSCTL_INT(_machdep, OID_AUTO, ispc98, CTLFLAG_RD, &ispc98, 0, ""); #endif int cold = 1; #ifdef COMPAT_43 static void osendsig(sig_t catcher, ksiginfo_t *, sigset_t *mask); #endif #ifdef COMPAT_FREEBSD4 static void freebsd4_sendsig(sig_t catcher, ksiginfo_t *, sigset_t *mask); #endif long Maxmem = 0; long realmem = 0; #ifdef PAE FEATURE(pae, "Physical Address Extensions"); #endif /* * The number of PHYSMAP entries must be one less than the number of * PHYSSEG entries because the PHYSMAP entry that spans the largest * physical address that is accessible by ISA DMA is split into two * PHYSSEG entries. */ #define PHYSMAP_SIZE (2 * (VM_PHYSSEG_MAX - 1)) vm_paddr_t phys_avail[PHYSMAP_SIZE + 2]; vm_paddr_t dump_avail[PHYSMAP_SIZE + 2]; /* must be 2 less so 0 0 can signal end of chunks */ #define PHYS_AVAIL_ARRAY_END ((sizeof(phys_avail) / sizeof(phys_avail[0])) - 2) #define DUMP_AVAIL_ARRAY_END ((sizeof(dump_avail) / sizeof(dump_avail[0])) - 2) struct kva_md_info kmi; static struct trapframe proc0_tf; struct pcpu __pcpu[MAXCPU]; struct mtx icu_lock; struct mem_range_softc mem_range_softc; /* Default init_ops implementation. */ struct init_ops init_ops = { .early_clock_source_init = i8254_init, .early_delay = i8254_delay, #ifdef DEV_APIC .msi_init = msi_init, #endif }; static void cpu_startup(dummy) void *dummy; { uintmax_t memsize; char *sysenv; #ifndef PC98 /* * On MacBooks, we need to disallow the legacy USB circuit to * generate an SMI# because this can cause several problems, * namely: incorrect CPU frequency detection and failure to * start the APs. * We do this by disabling a bit in the SMI_EN (SMI Control and * Enable register) of the Intel ICH LPC Interface Bridge. */ sysenv = kern_getenv("smbios.system.product"); if (sysenv != NULL) { if (strncmp(sysenv, "MacBook1,1", 10) == 0 || strncmp(sysenv, "MacBook3,1", 10) == 0 || strncmp(sysenv, "MacBook4,1", 10) == 0 || strncmp(sysenv, "MacBookPro1,1", 13) == 0 || strncmp(sysenv, "MacBookPro1,2", 13) == 0 || strncmp(sysenv, "MacBookPro3,1", 13) == 0 || strncmp(sysenv, "MacBookPro4,1", 13) == 0 || strncmp(sysenv, "Macmini1,1", 10) == 0) { if (bootverbose) printf("Disabling LEGACY_USB_EN bit on " "Intel ICH.\n"); outl(ICH_SMI_EN, inl(ICH_SMI_EN) & ~0x8); } freeenv(sysenv); } #endif /* !PC98 */ /* * Good {morning,afternoon,evening,night}. */ startrtclock(); printcpuinfo(); panicifcpuunsupported(); #ifdef PERFMON perfmon_init(); #endif /* * Display physical memory if SMBIOS reports reasonable amount. */ memsize = 0; sysenv = kern_getenv("smbios.memory.enabled"); if (sysenv != NULL) { memsize = (uintmax_t)strtoul(sysenv, (char **)NULL, 10) << 10; freeenv(sysenv); } if (memsize < ptoa((uintmax_t)vm_cnt.v_free_count)) memsize = ptoa((uintmax_t)Maxmem); printf("real memory = %ju (%ju MB)\n", memsize, memsize >> 20); realmem = atop(memsize); /* * Display any holes after the first chunk of extended memory. */ if (bootverbose) { int indx; printf("Physical memory chunk(s):\n"); for (indx = 0; phys_avail[indx + 1] != 0; indx += 2) { vm_paddr_t size; size = phys_avail[indx + 1] - phys_avail[indx]; printf( "0x%016jx - 0x%016jx, %ju bytes (%ju pages)\n", (uintmax_t)phys_avail[indx], (uintmax_t)phys_avail[indx + 1] - 1, (uintmax_t)size, (uintmax_t)size / PAGE_SIZE); } } vm_ksubmap_init(&kmi); printf("avail memory = %ju (%ju MB)\n", ptoa((uintmax_t)vm_cnt.v_free_count), ptoa((uintmax_t)vm_cnt.v_free_count) / 1048576); /* * Set up buffers, so they can be used to read disk labels. */ bufinit(); vm_pager_bufferinit(); #ifndef XEN cpu_setregs(); #endif } /* * Send an interrupt to process. * * Stack is set up to allow sigcode stored * at top to call routine, followed by call * to sigreturn routine below. After sigreturn * resets the signal mask, the stack, and the * frame pointer, it returns to the user * specified pc, psl. */ #ifdef COMPAT_43 static void osendsig(sig_t catcher, ksiginfo_t *ksi, sigset_t *mask) { struct osigframe sf, *fp; struct proc *p; struct thread *td; struct sigacts *psp; struct trapframe *regs; int sig; int oonstack; td = curthread; p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); sig = ksi->ksi_signo; psp = p->p_sigacts; mtx_assert(&psp->ps_mtx, MA_OWNED); regs = td->td_frame; oonstack = sigonstack(regs->tf_esp); /* Allocate space for the signal handler context. */ if ((td->td_pflags & TDP_ALTSTACK) && !oonstack && SIGISMEMBER(psp->ps_sigonstack, sig)) { fp = (struct osigframe *)(td->td_sigstk.ss_sp + td->td_sigstk.ss_size - sizeof(struct osigframe)); #if defined(COMPAT_43) td->td_sigstk.ss_flags |= SS_ONSTACK; #endif } else fp = (struct osigframe *)regs->tf_esp - 1; /* Translate the signal if appropriate. */ if (p->p_sysent->sv_sigtbl && sig <= p->p_sysent->sv_sigsize) sig = p->p_sysent->sv_sigtbl[_SIG_IDX(sig)]; /* Build the argument list for the signal handler. */ sf.sf_signum = sig; sf.sf_scp = (register_t)&fp->sf_siginfo.si_sc; bzero(&sf.sf_siginfo, sizeof(sf.sf_siginfo)); if (SIGISMEMBER(psp->ps_siginfo, sig)) { /* Signal handler installed with SA_SIGINFO. */ sf.sf_arg2 = (register_t)&fp->sf_siginfo; sf.sf_siginfo.si_signo = sig; sf.sf_siginfo.si_code = ksi->ksi_code; sf.sf_ahu.sf_action = (__osiginfohandler_t *)catcher; sf.sf_addr = 0; } else { /* Old FreeBSD-style arguments. */ sf.sf_arg2 = ksi->ksi_code; sf.sf_addr = (register_t)ksi->ksi_addr; sf.sf_ahu.sf_handler = catcher; } mtx_unlock(&psp->ps_mtx); PROC_UNLOCK(p); /* Save most if not all of trap frame. */ sf.sf_siginfo.si_sc.sc_eax = regs->tf_eax; sf.sf_siginfo.si_sc.sc_ebx = regs->tf_ebx; sf.sf_siginfo.si_sc.sc_ecx = regs->tf_ecx; sf.sf_siginfo.si_sc.sc_edx = regs->tf_edx; sf.sf_siginfo.si_sc.sc_esi = regs->tf_esi; sf.sf_siginfo.si_sc.sc_edi = regs->tf_edi; sf.sf_siginfo.si_sc.sc_cs = regs->tf_cs; sf.sf_siginfo.si_sc.sc_ds = regs->tf_ds; sf.sf_siginfo.si_sc.sc_ss = regs->tf_ss; sf.sf_siginfo.si_sc.sc_es = regs->tf_es; sf.sf_siginfo.si_sc.sc_fs = regs->tf_fs; sf.sf_siginfo.si_sc.sc_gs = rgs(); sf.sf_siginfo.si_sc.sc_isp = regs->tf_isp; /* Build the signal context to be used by osigreturn(). */ sf.sf_siginfo.si_sc.sc_onstack = (oonstack) ? 1 : 0; SIG2OSIG(*mask, sf.sf_siginfo.si_sc.sc_mask); sf.sf_siginfo.si_sc.sc_sp = regs->tf_esp; sf.sf_siginfo.si_sc.sc_fp = regs->tf_ebp; sf.sf_siginfo.si_sc.sc_pc = regs->tf_eip; sf.sf_siginfo.si_sc.sc_ps = regs->tf_eflags; sf.sf_siginfo.si_sc.sc_trapno = regs->tf_trapno; sf.sf_siginfo.si_sc.sc_err = regs->tf_err; /* * If we're a vm86 process, we want to save the segment registers. * We also change eflags to be our emulated eflags, not the actual * eflags. */ if (regs->tf_eflags & PSL_VM) { /* XXX confusing names: `tf' isn't a trapframe; `regs' is. */ struct trapframe_vm86 *tf = (struct trapframe_vm86 *)regs; struct vm86_kernel *vm86 = &td->td_pcb->pcb_ext->ext_vm86; sf.sf_siginfo.si_sc.sc_gs = tf->tf_vm86_gs; sf.sf_siginfo.si_sc.sc_fs = tf->tf_vm86_fs; sf.sf_siginfo.si_sc.sc_es = tf->tf_vm86_es; sf.sf_siginfo.si_sc.sc_ds = tf->tf_vm86_ds; if (vm86->vm86_has_vme == 0) sf.sf_siginfo.si_sc.sc_ps = (tf->tf_eflags & ~(PSL_VIF | PSL_VIP)) | (vm86->vm86_eflags & (PSL_VIF | PSL_VIP)); /* See sendsig() for comments. */ tf->tf_eflags &= ~(PSL_VM | PSL_NT | PSL_VIF | PSL_VIP); } /* * Copy the sigframe out to the user's stack. */ if (copyout(&sf, fp, sizeof(*fp)) != 0) { #ifdef DEBUG printf("process %ld has trashed its stack\n", (long)p->p_pid); #endif PROC_LOCK(p); sigexit(td, SIGILL); } regs->tf_esp = (int)fp; if (p->p_sysent->sv_sigcode_base != 0) { regs->tf_eip = p->p_sysent->sv_sigcode_base + szsigcode - szosigcode; } else { /* a.out sysentvec does not use shared page */ regs->tf_eip = p->p_sysent->sv_psstrings - szosigcode; } regs->tf_eflags &= ~(PSL_T | PSL_D); regs->tf_cs = _ucodesel; regs->tf_ds = _udatasel; regs->tf_es = _udatasel; regs->tf_fs = _udatasel; load_gs(_udatasel); regs->tf_ss = _udatasel; PROC_LOCK(p); mtx_lock(&psp->ps_mtx); } #endif /* COMPAT_43 */ #ifdef COMPAT_FREEBSD4 static void freebsd4_sendsig(sig_t catcher, ksiginfo_t *ksi, sigset_t *mask) { struct sigframe4 sf, *sfp; struct proc *p; struct thread *td; struct sigacts *psp; struct trapframe *regs; int sig; int oonstack; td = curthread; p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); sig = ksi->ksi_signo; psp = p->p_sigacts; mtx_assert(&psp->ps_mtx, MA_OWNED); regs = td->td_frame; oonstack = sigonstack(regs->tf_esp); /* Save user context. */ bzero(&sf, sizeof(sf)); sf.sf_uc.uc_sigmask = *mask; sf.sf_uc.uc_stack = td->td_sigstk; sf.sf_uc.uc_stack.ss_flags = (td->td_pflags & TDP_ALTSTACK) ? ((oonstack) ? SS_ONSTACK : 0) : SS_DISABLE; sf.sf_uc.uc_mcontext.mc_onstack = (oonstack) ? 1 : 0; sf.sf_uc.uc_mcontext.mc_gs = rgs(); bcopy(regs, &sf.sf_uc.uc_mcontext.mc_fs, sizeof(*regs)); bzero(sf.sf_uc.uc_mcontext.mc_fpregs, sizeof(sf.sf_uc.uc_mcontext.mc_fpregs)); bzero(sf.sf_uc.uc_mcontext.__spare__, sizeof(sf.sf_uc.uc_mcontext.__spare__)); bzero(sf.sf_uc.__spare__, sizeof(sf.sf_uc.__spare__)); /* Allocate space for the signal handler context. */ if ((td->td_pflags & TDP_ALTSTACK) != 0 && !oonstack && SIGISMEMBER(psp->ps_sigonstack, sig)) { sfp = (struct sigframe4 *)(td->td_sigstk.ss_sp + td->td_sigstk.ss_size - sizeof(struct sigframe4)); #if defined(COMPAT_43) td->td_sigstk.ss_flags |= SS_ONSTACK; #endif } else sfp = (struct sigframe4 *)regs->tf_esp - 1; /* Translate the signal if appropriate. */ if (p->p_sysent->sv_sigtbl && sig <= p->p_sysent->sv_sigsize) sig = p->p_sysent->sv_sigtbl[_SIG_IDX(sig)]; /* Build the argument list for the signal handler. */ sf.sf_signum = sig; sf.sf_ucontext = (register_t)&sfp->sf_uc; bzero(&sf.sf_si, sizeof(sf.sf_si)); if (SIGISMEMBER(psp->ps_siginfo, sig)) { /* Signal handler installed with SA_SIGINFO. */ sf.sf_siginfo = (register_t)&sfp->sf_si; sf.sf_ahu.sf_action = (__siginfohandler_t *)catcher; /* Fill in POSIX parts */ sf.sf_si.si_signo = sig; sf.sf_si.si_code = ksi->ksi_code; sf.sf_si.si_addr = ksi->ksi_addr; } else { /* Old FreeBSD-style arguments. */ sf.sf_siginfo = ksi->ksi_code; sf.sf_addr = (register_t)ksi->ksi_addr; sf.sf_ahu.sf_handler = catcher; } mtx_unlock(&psp->ps_mtx); PROC_UNLOCK(p); /* * If we're a vm86 process, we want to save the segment registers. * We also change eflags to be our emulated eflags, not the actual * eflags. */ if (regs->tf_eflags & PSL_VM) { struct trapframe_vm86 *tf = (struct trapframe_vm86 *)regs; struct vm86_kernel *vm86 = &td->td_pcb->pcb_ext->ext_vm86; sf.sf_uc.uc_mcontext.mc_gs = tf->tf_vm86_gs; sf.sf_uc.uc_mcontext.mc_fs = tf->tf_vm86_fs; sf.sf_uc.uc_mcontext.mc_es = tf->tf_vm86_es; sf.sf_uc.uc_mcontext.mc_ds = tf->tf_vm86_ds; if (vm86->vm86_has_vme == 0) sf.sf_uc.uc_mcontext.mc_eflags = (tf->tf_eflags & ~(PSL_VIF | PSL_VIP)) | (vm86->vm86_eflags & (PSL_VIF | PSL_VIP)); /* * Clear PSL_NT to inhibit T_TSSFLT faults on return from * syscalls made by the signal handler. This just avoids * wasting time for our lazy fixup of such faults. PSL_NT * does nothing in vm86 mode, but vm86 programs can set it * almost legitimately in probes for old cpu types. */ tf->tf_eflags &= ~(PSL_VM | PSL_NT | PSL_VIF | PSL_VIP); } /* * Copy the sigframe out to the user's stack. */ if (copyout(&sf, sfp, sizeof(*sfp)) != 0) { #ifdef DEBUG printf("process %ld has trashed its stack\n", (long)p->p_pid); #endif PROC_LOCK(p); sigexit(td, SIGILL); } regs->tf_esp = (int)sfp; regs->tf_eip = p->p_sysent->sv_sigcode_base + szsigcode - szfreebsd4_sigcode; regs->tf_eflags &= ~(PSL_T | PSL_D); regs->tf_cs = _ucodesel; regs->tf_ds = _udatasel; regs->tf_es = _udatasel; regs->tf_fs = _udatasel; regs->tf_ss = _udatasel; PROC_LOCK(p); mtx_lock(&psp->ps_mtx); } #endif /* COMPAT_FREEBSD4 */ void sendsig(sig_t catcher, ksiginfo_t *ksi, sigset_t *mask) { struct sigframe sf, *sfp; struct proc *p; struct thread *td; struct sigacts *psp; char *sp; struct trapframe *regs; struct segment_descriptor *sdp; char *xfpusave; size_t xfpusave_len; int sig; int oonstack; td = curthread; p = td->td_proc; PROC_LOCK_ASSERT(p, MA_OWNED); sig = ksi->ksi_signo; psp = p->p_sigacts; mtx_assert(&psp->ps_mtx, MA_OWNED); #ifdef COMPAT_FREEBSD4 if (SIGISMEMBER(psp->ps_freebsd4, sig)) { freebsd4_sendsig(catcher, ksi, mask); return; } #endif #ifdef COMPAT_43 if (SIGISMEMBER(psp->ps_osigset, sig)) { osendsig(catcher, ksi, mask); return; } #endif regs = td->td_frame; oonstack = sigonstack(regs->tf_esp); #ifdef CPU_ENABLE_SSE if (cpu_max_ext_state_size > sizeof(union savefpu) && use_xsave) { xfpusave_len = cpu_max_ext_state_size - sizeof(union savefpu); xfpusave = __builtin_alloca(xfpusave_len); } else { #else { #endif xfpusave_len = 0; xfpusave = NULL; } /* Save user context. */ bzero(&sf, sizeof(sf)); sf.sf_uc.uc_sigmask = *mask; sf.sf_uc.uc_stack = td->td_sigstk; sf.sf_uc.uc_stack.ss_flags = (td->td_pflags & TDP_ALTSTACK) ? ((oonstack) ? SS_ONSTACK : 0) : SS_DISABLE; sf.sf_uc.uc_mcontext.mc_onstack = (oonstack) ? 1 : 0; sf.sf_uc.uc_mcontext.mc_gs = rgs(); bcopy(regs, &sf.sf_uc.uc_mcontext.mc_fs, sizeof(*regs)); sf.sf_uc.uc_mcontext.mc_len = sizeof(sf.sf_uc.uc_mcontext); /* magic */ get_fpcontext(td, &sf.sf_uc.uc_mcontext, xfpusave, xfpusave_len); fpstate_drop(td); /* * Unconditionally fill the fsbase and gsbase into the mcontext. */ sdp = &td->td_pcb->pcb_fsd; sf.sf_uc.uc_mcontext.mc_fsbase = sdp->sd_hibase << 24 | sdp->sd_lobase; sdp = &td->td_pcb->pcb_gsd; sf.sf_uc.uc_mcontext.mc_gsbase = sdp->sd_hibase << 24 | sdp->sd_lobase; bzero(sf.sf_uc.uc_mcontext.mc_spare2, sizeof(sf.sf_uc.uc_mcontext.mc_spare2)); bzero(sf.sf_uc.__spare__, sizeof(sf.sf_uc.__spare__)); /* Allocate space for the signal handler context. */ if ((td->td_pflags & TDP_ALTSTACK) != 0 && !oonstack && SIGISMEMBER(psp->ps_sigonstack, sig)) { sp = td->td_sigstk.ss_sp + td->td_sigstk.ss_size; #if defined(COMPAT_43) td->td_sigstk.ss_flags |= SS_ONSTACK; #endif } else sp = (char *)regs->tf_esp - 128; if (xfpusave != NULL) { sp -= xfpusave_len; sp = (char *)((unsigned int)sp & ~0x3F); sf.sf_uc.uc_mcontext.mc_xfpustate = (register_t)sp; } sp -= sizeof(struct sigframe); /* Align to 16 bytes. */ sfp = (struct sigframe *)((unsigned int)sp & ~0xF); /* Translate the signal if appropriate. */ if (p->p_sysent->sv_sigtbl && sig <= p->p_sysent->sv_sigsize) sig = p->p_sysent->sv_sigtbl[_SIG_IDX(sig)]; /* Build the argument list for the signal handler. */ sf.sf_signum = sig; sf.sf_ucontext = (register_t)&sfp->sf_uc; bzero(&sf.sf_si, sizeof(sf.sf_si)); if (SIGISMEMBER(psp->ps_siginfo, sig)) { /* Signal handler installed with SA_SIGINFO. */ sf.sf_siginfo = (register_t)&sfp->sf_si; sf.sf_ahu.sf_action = (__siginfohandler_t *)catcher; /* Fill in POSIX parts */ sf.sf_si = ksi->ksi_info; sf.sf_si.si_signo = sig; /* maybe a translated signal */ } else { /* Old FreeBSD-style arguments. */ sf.sf_siginfo = ksi->ksi_code; sf.sf_addr = (register_t)ksi->ksi_addr; sf.sf_ahu.sf_handler = catcher; } mtx_unlock(&psp->ps_mtx); PROC_UNLOCK(p); /* * If we're a vm86 process, we want to save the segment registers. * We also change eflags to be our emulated eflags, not the actual * eflags. */ if (regs->tf_eflags & PSL_VM) { struct trapframe_vm86 *tf = (struct trapframe_vm86 *)regs; struct vm86_kernel *vm86 = &td->td_pcb->pcb_ext->ext_vm86; sf.sf_uc.uc_mcontext.mc_gs = tf->tf_vm86_gs; sf.sf_uc.uc_mcontext.mc_fs = tf->tf_vm86_fs; sf.sf_uc.uc_mcontext.mc_es = tf->tf_vm86_es; sf.sf_uc.uc_mcontext.mc_ds = tf->tf_vm86_ds; if (vm86->vm86_has_vme == 0) sf.sf_uc.uc_mcontext.mc_eflags = (tf->tf_eflags & ~(PSL_VIF | PSL_VIP)) | (vm86->vm86_eflags & (PSL_VIF | PSL_VIP)); /* * Clear PSL_NT to inhibit T_TSSFLT faults on return from * syscalls made by the signal handler. This just avoids * wasting time for our lazy fixup of such faults. PSL_NT * does nothing in vm86 mode, but vm86 programs can set it * almost legitimately in probes for old cpu types. */ tf->tf_eflags &= ~(PSL_VM | PSL_NT | PSL_VIF | PSL_VIP); } /* * Copy the sigframe out to the user's stack. */ if (copyout(&sf, sfp, sizeof(*sfp)) != 0 || (xfpusave != NULL && copyout(xfpusave, (void *)sf.sf_uc.uc_mcontext.mc_xfpustate, xfpusave_len) != 0)) { #ifdef DEBUG printf("process %ld has trashed its stack\n", (long)p->p_pid); #endif PROC_LOCK(p); sigexit(td, SIGILL); } regs->tf_esp = (int)sfp; regs->tf_eip = p->p_sysent->sv_sigcode_base; if (regs->tf_eip == 0) regs->tf_eip = p->p_sysent->sv_psstrings - szsigcode; regs->tf_eflags &= ~(PSL_T | PSL_D); regs->tf_cs = _ucodesel; regs->tf_ds = _udatasel; regs->tf_es = _udatasel; regs->tf_fs = _udatasel; regs->tf_ss = _udatasel; PROC_LOCK(p); mtx_lock(&psp->ps_mtx); } /* * System call to cleanup state after a signal * has been taken. Reset signal mask and * stack state from context left by sendsig (above). * Return to previous pc and psl as specified by * context left by sendsig. Check carefully to * make sure that the user has not modified the * state to gain improper privileges. * * MPSAFE */ #ifdef COMPAT_43 int osigreturn(td, uap) struct thread *td; struct osigreturn_args /* { struct osigcontext *sigcntxp; } */ *uap; { struct osigcontext sc; struct trapframe *regs; struct osigcontext *scp; int eflags, error; ksiginfo_t ksi; regs = td->td_frame; error = copyin(uap->sigcntxp, &sc, sizeof(sc)); if (error != 0) return (error); scp = ≻ eflags = scp->sc_ps; if (eflags & PSL_VM) { struct trapframe_vm86 *tf = (struct trapframe_vm86 *)regs; struct vm86_kernel *vm86; /* * if pcb_ext == 0 or vm86_inited == 0, the user hasn't * set up the vm86 area, and we can't enter vm86 mode. */ if (td->td_pcb->pcb_ext == 0) return (EINVAL); vm86 = &td->td_pcb->pcb_ext->ext_vm86; if (vm86->vm86_inited == 0) return (EINVAL); /* Go back to user mode if both flags are set. */ if ((eflags & PSL_VIP) && (eflags & PSL_VIF)) { ksiginfo_init_trap(&ksi); ksi.ksi_signo = SIGBUS; ksi.ksi_code = BUS_OBJERR; ksi.ksi_addr = (void *)regs->tf_eip; trapsignal(td, &ksi); } if (vm86->vm86_has_vme) { eflags = (tf->tf_eflags & ~VME_USERCHANGE) | (eflags & VME_USERCHANGE) | PSL_VM; } else { vm86->vm86_eflags = eflags; /* save VIF, VIP */ eflags = (tf->tf_eflags & ~VM_USERCHANGE) | (eflags & VM_USERCHANGE) | PSL_VM; } tf->tf_vm86_ds = scp->sc_ds; tf->tf_vm86_es = scp->sc_es; tf->tf_vm86_fs = scp->sc_fs; tf->tf_vm86_gs = scp->sc_gs; tf->tf_ds = _udatasel; tf->tf_es = _udatasel; tf->tf_fs = _udatasel; } else { /* * Don't allow users to change privileged or reserved flags. */ if (!EFL_SECURE(eflags, regs->tf_eflags)) { return (EINVAL); } /* * Don't allow users to load a valid privileged %cs. Let the * hardware check for invalid selectors, excess privilege in * other selectors, invalid %eip's and invalid %esp's. */ if (!CS_SECURE(scp->sc_cs)) { ksiginfo_init_trap(&ksi); ksi.ksi_signo = SIGBUS; ksi.ksi_code = BUS_OBJERR; ksi.ksi_trapno = T_PROTFLT; ksi.ksi_addr = (void *)regs->tf_eip; trapsignal(td, &ksi); return (EINVAL); } regs->tf_ds = scp->sc_ds; regs->tf_es = scp->sc_es; regs->tf_fs = scp->sc_fs; } /* Restore remaining registers. */ regs->tf_eax = scp->sc_eax; regs->tf_ebx = scp->sc_ebx; regs->tf_ecx = scp->sc_ecx; regs->tf_edx = scp->sc_edx; regs->tf_esi = scp->sc_esi; regs->tf_edi = scp->sc_edi; regs->tf_cs = scp->sc_cs; regs->tf_ss = scp->sc_ss; regs->tf_isp = scp->sc_isp; regs->tf_ebp = scp->sc_fp; regs->tf_esp = scp->sc_sp; regs->tf_eip = scp->sc_pc; regs->tf_eflags = eflags; #if defined(COMPAT_43) if (scp->sc_onstack & 1) td->td_sigstk.ss_flags |= SS_ONSTACK; else td->td_sigstk.ss_flags &= ~SS_ONSTACK; #endif kern_sigprocmask(td, SIG_SETMASK, (sigset_t *)&scp->sc_mask, NULL, SIGPROCMASK_OLD); return (EJUSTRETURN); } #endif /* COMPAT_43 */ #ifdef COMPAT_FREEBSD4 /* * MPSAFE */ int freebsd4_sigreturn(td, uap) struct thread *td; struct freebsd4_sigreturn_args /* { const ucontext4 *sigcntxp; } */ *uap; { struct ucontext4 uc; struct trapframe *regs; struct ucontext4 *ucp; int cs, eflags, error; ksiginfo_t ksi; error = copyin(uap->sigcntxp, &uc, sizeof(uc)); if (error != 0) return (error); ucp = &uc; regs = td->td_frame; eflags = ucp->uc_mcontext.mc_eflags; if (eflags & PSL_VM) { struct trapframe_vm86 *tf = (struct trapframe_vm86 *)regs; struct vm86_kernel *vm86; /* * if pcb_ext == 0 or vm86_inited == 0, the user hasn't * set up the vm86 area, and we can't enter vm86 mode. */ if (td->td_pcb->pcb_ext == 0) return (EINVAL); vm86 = &td->td_pcb->pcb_ext->ext_vm86; if (vm86->vm86_inited == 0) return (EINVAL); /* Go back to user mode if both flags are set. */ if ((eflags & PSL_VIP) && (eflags & PSL_VIF)) { ksiginfo_init_trap(&ksi); ksi.ksi_signo = SIGBUS; ksi.ksi_code = BUS_OBJERR; ksi.ksi_addr = (void *)regs->tf_eip; trapsignal(td, &ksi); } if (vm86->vm86_has_vme) { eflags = (tf->tf_eflags & ~VME_USERCHANGE) | (eflags & VME_USERCHANGE) | PSL_VM; } else { vm86->vm86_eflags = eflags; /* save VIF, VIP */ eflags = (tf->tf_eflags & ~VM_USERCHANGE) | (eflags & VM_USERCHANGE) | PSL_VM; } bcopy(&ucp->uc_mcontext.mc_fs, tf, sizeof(struct trapframe)); tf->tf_eflags = eflags; tf->tf_vm86_ds = tf->tf_ds; tf->tf_vm86_es = tf->tf_es; tf->tf_vm86_fs = tf->tf_fs; tf->tf_vm86_gs = ucp->uc_mcontext.mc_gs; tf->tf_ds = _udatasel; tf->tf_es = _udatasel; tf->tf_fs = _udatasel; } else { /* * Don't allow users to change privileged or reserved flags. */ if (!EFL_SECURE(eflags, regs->tf_eflags)) { uprintf("pid %d (%s): freebsd4_sigreturn eflags = 0x%x\n", td->td_proc->p_pid, td->td_name, eflags); return (EINVAL); } /* * Don't allow users to load a valid privileged %cs. Let the * hardware check for invalid selectors, excess privilege in * other selectors, invalid %eip's and invalid %esp's. */ cs = ucp->uc_mcontext.mc_cs; if (!CS_SECURE(cs)) { uprintf("pid %d (%s): freebsd4_sigreturn cs = 0x%x\n", td->td_proc->p_pid, td->td_name, cs); ksiginfo_init_trap(&ksi); ksi.ksi_signo = SIGBUS; ksi.ksi_code = BUS_OBJERR; ksi.ksi_trapno = T_PROTFLT; ksi.ksi_addr = (void *)regs->tf_eip; trapsignal(td, &ksi); return (EINVAL); } bcopy(&ucp->uc_mcontext.mc_fs, regs, sizeof(*regs)); } #if defined(COMPAT_43) if (ucp->uc_mcontext.mc_onstack & 1) td->td_sigstk.ss_flags |= SS_ONSTACK; else td->td_sigstk.ss_flags &= ~SS_ONSTACK; #endif kern_sigprocmask(td, SIG_SETMASK, &ucp->uc_sigmask, NULL, 0); return (EJUSTRETURN); } #endif /* COMPAT_FREEBSD4 */ /* * MPSAFE */ int sys_sigreturn(td, uap) struct thread *td; struct sigreturn_args /* { const struct __ucontext *sigcntxp; } */ *uap; { ucontext_t uc; struct proc *p; struct trapframe *regs; ucontext_t *ucp; char *xfpustate; size_t xfpustate_len; int cs, eflags, error, ret; ksiginfo_t ksi; p = td->td_proc; error = copyin(uap->sigcntxp, &uc, sizeof(uc)); if (error != 0) return (error); ucp = &uc; if ((ucp->uc_mcontext.mc_flags & ~_MC_FLAG_MASK) != 0) { uprintf("pid %d (%s): sigreturn mc_flags %x\n", p->p_pid, td->td_name, ucp->uc_mcontext.mc_flags); return (EINVAL); } regs = td->td_frame; eflags = ucp->uc_mcontext.mc_eflags; if (eflags & PSL_VM) { struct trapframe_vm86 *tf = (struct trapframe_vm86 *)regs; struct vm86_kernel *vm86; /* * if pcb_ext == 0 or vm86_inited == 0, the user hasn't * set up the vm86 area, and we can't enter vm86 mode. */ if (td->td_pcb->pcb_ext == 0) return (EINVAL); vm86 = &td->td_pcb->pcb_ext->ext_vm86; if (vm86->vm86_inited == 0) return (EINVAL); /* Go back to user mode if both flags are set. */ if ((eflags & PSL_VIP) && (eflags & PSL_VIF)) { ksiginfo_init_trap(&ksi); ksi.ksi_signo = SIGBUS; ksi.ksi_code = BUS_OBJERR; ksi.ksi_addr = (void *)regs->tf_eip; trapsignal(td, &ksi); } if (vm86->vm86_has_vme) { eflags = (tf->tf_eflags & ~VME_USERCHANGE) | (eflags & VME_USERCHANGE) | PSL_VM; } else { vm86->vm86_eflags = eflags; /* save VIF, VIP */ eflags = (tf->tf_eflags & ~VM_USERCHANGE) | (eflags & VM_USERCHANGE) | PSL_VM; } bcopy(&ucp->uc_mcontext.mc_fs, tf, sizeof(struct trapframe)); tf->tf_eflags = eflags; tf->tf_vm86_ds = tf->tf_ds; tf->tf_vm86_es = tf->tf_es; tf->tf_vm86_fs = tf->tf_fs; tf->tf_vm86_gs = ucp->uc_mcontext.mc_gs; tf->tf_ds = _udatasel; tf->tf_es = _udatasel; tf->tf_fs = _udatasel; } else { /* * Don't allow users to change privileged or reserved flags. */ if (!EFL_SECURE(eflags, regs->tf_eflags)) { uprintf("pid %d (%s): sigreturn eflags = 0x%x\n", td->td_proc->p_pid, td->td_name, eflags); return (EINVAL); } /* * Don't allow users to load a valid privileged %cs. Let the * hardware check for invalid selectors, excess privilege in * other selectors, invalid %eip's and invalid %esp's. */ cs = ucp->uc_mcontext.mc_cs; if (!CS_SECURE(cs)) { uprintf("pid %d (%s): sigreturn cs = 0x%x\n", td->td_proc->p_pid, td->td_name, cs); ksiginfo_init_trap(&ksi); ksi.ksi_signo = SIGBUS; ksi.ksi_code = BUS_OBJERR; ksi.ksi_trapno = T_PROTFLT; ksi.ksi_addr = (void *)regs->tf_eip; trapsignal(td, &ksi); return (EINVAL); } if ((uc.uc_mcontext.mc_flags & _MC_HASFPXSTATE) != 0) { xfpustate_len = uc.uc_mcontext.mc_xfpustate_len; if (xfpustate_len > cpu_max_ext_state_size - sizeof(union savefpu)) { uprintf( "pid %d (%s): sigreturn xfpusave_len = 0x%zx\n", p->p_pid, td->td_name, xfpustate_len); return (EINVAL); } xfpustate = __builtin_alloca(xfpustate_len); error = copyin((const void *)uc.uc_mcontext.mc_xfpustate, xfpustate, xfpustate_len); if (error != 0) { uprintf( "pid %d (%s): sigreturn copying xfpustate failed\n", p->p_pid, td->td_name); return (error); } } else { xfpustate = NULL; xfpustate_len = 0; } ret = set_fpcontext(td, &ucp->uc_mcontext, xfpustate, xfpustate_len); if (ret != 0) return (ret); bcopy(&ucp->uc_mcontext.mc_fs, regs, sizeof(*regs)); } #if defined(COMPAT_43) if (ucp->uc_mcontext.mc_onstack & 1) td->td_sigstk.ss_flags |= SS_ONSTACK; else td->td_sigstk.ss_flags &= ~SS_ONSTACK; #endif kern_sigprocmask(td, SIG_SETMASK, &ucp->uc_sigmask, NULL, 0); return (EJUSTRETURN); } /* * Machine dependent boot() routine * * I haven't seen anything to put here yet * Possibly some stuff might be grafted back here from boot() */ void cpu_boot(int howto) { } /* * Flush the D-cache for non-DMA I/O so that the I-cache can * be made coherent later. */ void cpu_flush_dcache(void *ptr, size_t len) { /* Not applicable */ } /* Get current clock frequency for the given cpu id. */ int cpu_est_clockrate(int cpu_id, uint64_t *rate) { uint64_t tsc1, tsc2; uint64_t acnt, mcnt, perf; register_t reg; if (pcpu_find(cpu_id) == NULL || rate == NULL) return (EINVAL); if ((cpu_feature & CPUID_TSC) == 0) return (EOPNOTSUPP); /* * If TSC is P-state invariant and APERF/MPERF MSRs do not exist, * DELAY(9) based logic fails. */ if (tsc_is_invariant && !tsc_perf_stat) return (EOPNOTSUPP); #ifdef SMP if (smp_cpus > 1) { /* Schedule ourselves on the indicated cpu. */ thread_lock(curthread); sched_bind(curthread, cpu_id); thread_unlock(curthread); } #endif /* Calibrate by measuring a short delay. */ reg = intr_disable(); if (tsc_is_invariant) { wrmsr(MSR_MPERF, 0); wrmsr(MSR_APERF, 0); tsc1 = rdtsc(); DELAY(1000); mcnt = rdmsr(MSR_MPERF); acnt = rdmsr(MSR_APERF); tsc2 = rdtsc(); intr_restore(reg); perf = 1000 * acnt / mcnt; *rate = (tsc2 - tsc1) * perf; } else { tsc1 = rdtsc(); DELAY(1000); tsc2 = rdtsc(); intr_restore(reg); *rate = (tsc2 - tsc1) * 1000; } #ifdef SMP if (smp_cpus > 1) { thread_lock(curthread); sched_unbind(curthread); thread_unlock(curthread); } #endif return (0); } #ifdef XEN static void idle_block(void) { HYPERVISOR_sched_op(SCHEDOP_block, 0); } void cpu_halt(void) { HYPERVISOR_shutdown(SHUTDOWN_poweroff); } int scheduler_running; static void cpu_idle_hlt(sbintime_t sbt) { scheduler_running = 1; enable_intr(); idle_block(); } #else /* * Shutdown the CPU as much as possible */ void cpu_halt(void) { for (;;) halt(); } #endif void (*cpu_idle_hook)(sbintime_t) = NULL; /* ACPI idle hook. */ static int cpu_ident_amdc1e = 0; /* AMD C1E supported. */ static int idle_mwait = 1; /* Use MONITOR/MWAIT for short idle. */ SYSCTL_INT(_machdep, OID_AUTO, idle_mwait, CTLFLAG_RWTUN, &idle_mwait, 0, "Use MONITOR/MWAIT for short idle"); #define STATE_RUNNING 0x0 #define STATE_MWAIT 0x1 #define STATE_SLEEPING 0x2 #ifndef PC98 static void cpu_idle_acpi(sbintime_t sbt) { int *state; state = (int *)PCPU_PTR(monitorbuf); *state = STATE_SLEEPING; /* See comments in cpu_idle_hlt(). */ disable_intr(); if (sched_runnable()) enable_intr(); else if (cpu_idle_hook) cpu_idle_hook(sbt); else __asm __volatile("sti; hlt"); *state = STATE_RUNNING; } #endif /* !PC98 */ #ifndef XEN static void cpu_idle_hlt(sbintime_t sbt) { int *state; state = (int *)PCPU_PTR(monitorbuf); *state = STATE_SLEEPING; /* * Since we may be in a critical section from cpu_idle(), if * an interrupt fires during that critical section we may have * a pending preemption. If the CPU halts, then that thread * may not execute until a later interrupt awakens the CPU. * To handle this race, check for a runnable thread after * disabling interrupts and immediately return if one is * found. Also, we must absolutely guarentee that hlt is * the next instruction after sti. This ensures that any * interrupt that fires after the call to disable_intr() will * immediately awaken the CPU from hlt. Finally, please note * that on x86 this works fine because of interrupts enabled only * after the instruction following sti takes place, while IF is set * to 1 immediately, allowing hlt instruction to acknowledge the * interrupt. */ disable_intr(); if (sched_runnable()) enable_intr(); else __asm __volatile("sti; hlt"); *state = STATE_RUNNING; } #endif /* * MWAIT cpu power states. Lower 4 bits are sub-states. */ #define MWAIT_C0 0xf0 #define MWAIT_C1 0x00 #define MWAIT_C2 0x10 #define MWAIT_C3 0x20 #define MWAIT_C4 0x30 static void cpu_idle_mwait(sbintime_t sbt) { int *state; state = (int *)PCPU_PTR(monitorbuf); *state = STATE_MWAIT; /* See comments in cpu_idle_hlt(). */ disable_intr(); if (sched_runnable()) { enable_intr(); *state = STATE_RUNNING; return; } cpu_monitor(state, 0, 0); if (*state == STATE_MWAIT) __asm __volatile("sti; mwait" : : "a" (MWAIT_C1), "c" (0)); else enable_intr(); *state = STATE_RUNNING; } static void cpu_idle_spin(sbintime_t sbt) { int *state; int i; state = (int *)PCPU_PTR(monitorbuf); *state = STATE_RUNNING; /* * The sched_runnable() call is racy but as long as there is * a loop missing it one time will have just a little impact if any * (and it is much better than missing the check at all). */ for (i = 0; i < 1000; i++) { if (sched_runnable()) return; cpu_spinwait(); } } /* * C1E renders the local APIC timer dead, so we disable it by * reading the Interrupt Pending Message register and clearing * both C1eOnCmpHalt (bit 28) and SmiOnCmpHalt (bit 27). * * Reference: * "BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh Processors" * #32559 revision 3.00+ */ #define MSR_AMDK8_IPM 0xc0010055 #define AMDK8_SMIONCMPHALT (1ULL << 27) #define AMDK8_C1EONCMPHALT (1ULL << 28) #define AMDK8_CMPHALT (AMDK8_SMIONCMPHALT | AMDK8_C1EONCMPHALT) static void cpu_probe_amdc1e(void) { /* * Detect the presence of C1E capability mostly on latest * dual-cores (or future) k8 family. */ if (cpu_vendor_id == CPU_VENDOR_AMD && (cpu_id & 0x00000f00) == 0x00000f00 && (cpu_id & 0x0fff0000) >= 0x00040000) { cpu_ident_amdc1e = 1; } } #if defined(PC98) || defined(XEN) void (*cpu_idle_fn)(sbintime_t) = cpu_idle_hlt; #else void (*cpu_idle_fn)(sbintime_t) = cpu_idle_acpi; #endif void cpu_idle(int busy) { #ifndef XEN uint64_t msr; #endif sbintime_t sbt = -1; CTR2(KTR_SPARE2, "cpu_idle(%d) at %d", busy, curcpu); #if defined(MP_WATCHDOG) && !defined(XEN) ap_watchdog(PCPU_GET(cpuid)); #endif #ifndef XEN /* If we are busy - try to use fast methods. */ if (busy) { if ((cpu_feature2 & CPUID2_MON) && idle_mwait) { cpu_idle_mwait(busy); goto out; } } #endif /* If we have time - switch timers into idle mode. */ if (!busy) { critical_enter(); sbt = cpu_idleclock(); } #ifndef XEN /* Apply AMD APIC timer C1E workaround. */ if (cpu_ident_amdc1e && cpu_disable_c3_sleep) { msr = rdmsr(MSR_AMDK8_IPM); if (msr & AMDK8_CMPHALT) wrmsr(MSR_AMDK8_IPM, msr & ~AMDK8_CMPHALT); } #endif /* Call main idle method. */ cpu_idle_fn(sbt); /* Switch timers back into active mode. */ if (!busy) { cpu_activeclock(); critical_exit(); } #ifndef XEN out: #endif CTR2(KTR_SPARE2, "cpu_idle(%d) at %d done", busy, curcpu); } int cpu_idle_wakeup(int cpu) { struct pcpu *pcpu; int *state; pcpu = pcpu_find(cpu); state = (int *)pcpu->pc_monitorbuf; /* * This doesn't need to be atomic since missing the race will * simply result in unnecessary IPIs. */ if (*state == STATE_SLEEPING) return (0); if (*state == STATE_MWAIT) *state = STATE_RUNNING; return (1); } /* * Ordered by speed/power consumption. */ struct { void *id_fn; char *id_name; } idle_tbl[] = { { cpu_idle_spin, "spin" }, { cpu_idle_mwait, "mwait" }, { cpu_idle_hlt, "hlt" }, #ifndef PC98 { cpu_idle_acpi, "acpi" }, #endif { NULL, NULL } }; static int idle_sysctl_available(SYSCTL_HANDLER_ARGS) { char *avail, *p; int error; int i; avail = malloc(256, M_TEMP, M_WAITOK); p = avail; for (i = 0; idle_tbl[i].id_name != NULL; i++) { if (strstr(idle_tbl[i].id_name, "mwait") && (cpu_feature2 & CPUID2_MON) == 0) continue; #ifndef PC98 if (strcmp(idle_tbl[i].id_name, "acpi") == 0 && cpu_idle_hook == NULL) continue; #endif p += sprintf(p, "%s%s", p != avail ? ", " : "", idle_tbl[i].id_name); } error = sysctl_handle_string(oidp, avail, 0, req); free(avail, M_TEMP); return (error); } SYSCTL_PROC(_machdep, OID_AUTO, idle_available, CTLTYPE_STRING | CTLFLAG_RD, 0, 0, idle_sysctl_available, "A", "list of available idle functions"); static int idle_sysctl(SYSCTL_HANDLER_ARGS) { char buf[16]; int error; char *p; int i; p = "unknown"; for (i = 0; idle_tbl[i].id_name != NULL; i++) { if (idle_tbl[i].id_fn == cpu_idle_fn) { p = idle_tbl[i].id_name; break; } } strncpy(buf, p, sizeof(buf)); error = sysctl_handle_string(oidp, buf, sizeof(buf), req); if (error != 0 || req->newptr == NULL) return (error); for (i = 0; idle_tbl[i].id_name != NULL; i++) { if (strstr(idle_tbl[i].id_name, "mwait") && (cpu_feature2 & CPUID2_MON) == 0) continue; #ifndef PC98 if (strcmp(idle_tbl[i].id_name, "acpi") == 0 && cpu_idle_hook == NULL) continue; #endif if (strcmp(idle_tbl[i].id_name, buf)) continue; cpu_idle_fn = idle_tbl[i].id_fn; return (0); } return (EINVAL); } SYSCTL_PROC(_machdep, OID_AUTO, idle, CTLTYPE_STRING | CTLFLAG_RW, 0, 0, idle_sysctl, "A", "currently selected idle function"); /* * Reset registers to default values on exec. */ void exec_setregs(struct thread *td, struct image_params *imgp, u_long stack) { struct trapframe *regs = td->td_frame; struct pcb *pcb = td->td_pcb; /* Reset pc->pcb_gs and %gs before possibly invalidating it. */ pcb->pcb_gs = _udatasel; load_gs(_udatasel); mtx_lock_spin(&dt_lock); if (td->td_proc->p_md.md_ldt) user_ldt_free(td); else mtx_unlock_spin(&dt_lock); bzero((char *)regs, sizeof(struct trapframe)); regs->tf_eip = imgp->entry_addr; regs->tf_esp = stack; regs->tf_eflags = PSL_USER | (regs->tf_eflags & PSL_T); regs->tf_ss = _udatasel; regs->tf_ds = _udatasel; regs->tf_es = _udatasel; regs->tf_fs = _udatasel; regs->tf_cs = _ucodesel; /* PS_STRINGS value for BSD/OS binaries. It is 0 for non-BSD/OS. */ regs->tf_ebx = imgp->ps_strings; /* * Reset the hardware debug registers if they were in use. * They won't have any meaning for the newly exec'd process. */ if (pcb->pcb_flags & PCB_DBREGS) { pcb->pcb_dr0 = 0; pcb->pcb_dr1 = 0; pcb->pcb_dr2 = 0; pcb->pcb_dr3 = 0; pcb->pcb_dr6 = 0; pcb->pcb_dr7 = 0; if (pcb == curpcb) { /* * Clear the debug registers on the running * CPU, otherwise they will end up affecting * the next process we switch to. */ reset_dbregs(); } pcb->pcb_flags &= ~PCB_DBREGS; } pcb->pcb_initial_npxcw = __INITIAL_NPXCW__; /* * Drop the FP state if we hold it, so that the process gets a * clean FP state if it uses the FPU again. */ fpstate_drop(td); /* * XXX - Linux emulator * Make sure sure edx is 0x0 on entry. Linux binaries depend * on it. */ td->td_retval[1] = 0; } void cpu_setregs(void) { unsigned int cr0; cr0 = rcr0(); /* * CR0_MP, CR0_NE and CR0_TS are set for NPX (FPU) support: * * Prepare to trap all ESC (i.e., NPX) instructions and all WAIT * instructions. We must set the CR0_MP bit and use the CR0_TS * bit to control the trap, because setting the CR0_EM bit does * not cause WAIT instructions to trap. It's important to trap * WAIT instructions - otherwise the "wait" variants of no-wait * control instructions would degenerate to the "no-wait" variants * after FP context switches but work correctly otherwise. It's * particularly important to trap WAITs when there is no NPX - * otherwise the "wait" variants would always degenerate. * * Try setting CR0_NE to get correct error reporting on 486DX's. * Setting it should fail or do nothing on lesser processors. */ cr0 |= CR0_MP | CR0_NE | CR0_TS | CR0_WP | CR0_AM; load_cr0(cr0); load_gs(_udatasel); } u_long bootdev; /* not a struct cdev *- encoding is different */ SYSCTL_ULONG(_machdep, OID_AUTO, guessed_bootdev, CTLFLAG_RD, &bootdev, 0, "Maybe the Boot device (not in struct cdev *format)"); static char bootmethod[16] = "BIOS"; SYSCTL_STRING(_machdep, OID_AUTO, bootmethod, CTLFLAG_RD, bootmethod, 0, "System firmware boot method"); /* * Initialize 386 and configure to run kernel */ /* * Initialize segments & interrupt table */ int _default_ldt; #ifdef XEN union descriptor *gdt; union descriptor *ldt; #else union descriptor gdt[NGDT * MAXCPU]; /* global descriptor table */ union descriptor ldt[NLDT]; /* local descriptor table */ #endif static struct gate_descriptor idt0[NIDT]; struct gate_descriptor *idt = &idt0[0]; /* interrupt descriptor table */ struct region_descriptor r_gdt, r_idt; /* table descriptors */ struct mtx dt_lock; /* lock for GDT and LDT */ static struct i386tss dblfault_tss; static char dblfault_stack[PAGE_SIZE]; extern vm_offset_t proc0kstack; /* * software prototypes -- in more palatable form. * * GCODE_SEL through GUDATA_SEL must be in this order for syscall/sysret * GUFS_SEL and GUGS_SEL must be in this order (swtch.s knows it) */ struct soft_segment_descriptor gdt_segs[] = { /* GNULL_SEL 0 Null Descriptor */ { .ssd_base = 0x0, .ssd_limit = 0x0, .ssd_type = 0, .ssd_dpl = SEL_KPL, .ssd_p = 0, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* GPRIV_SEL 1 SMP Per-Processor Private Data Descriptor */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = SEL_KPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GUFS_SEL 2 %fs Descriptor for user */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = SEL_UPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GUGS_SEL 3 %gs Descriptor for user */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = SEL_UPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GCODE_SEL 4 Code Descriptor for kernel */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMERA, .ssd_dpl = SEL_KPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GDATA_SEL 5 Data Descriptor for kernel */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = SEL_KPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GUCODE_SEL 6 Code Descriptor for user */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMERA, .ssd_dpl = SEL_UPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GUDATA_SEL 7 Data Descriptor for user */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = SEL_UPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GBIOSLOWMEM_SEL 8 BIOS access to realmode segment 0x40, must be #8 in GDT */ { .ssd_base = 0x400, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = SEL_KPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, #ifndef XEN /* GPROC0_SEL 9 Proc 0 Tss Descriptor */ { .ssd_base = 0x0, .ssd_limit = sizeof(struct i386tss)-1, .ssd_type = SDT_SYS386TSS, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* GLDT_SEL 10 LDT Descriptor */ { .ssd_base = (int) ldt, .ssd_limit = sizeof(ldt)-1, .ssd_type = SDT_SYSLDT, .ssd_dpl = SEL_UPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* GUSERLDT_SEL 11 User LDT Descriptor per process */ { .ssd_base = (int) ldt, .ssd_limit = (512 * sizeof(union descriptor)-1), .ssd_type = SDT_SYSLDT, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* GPANIC_SEL 12 Panic Tss Descriptor */ { .ssd_base = (int) &dblfault_tss, .ssd_limit = sizeof(struct i386tss)-1, .ssd_type = SDT_SYS386TSS, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* GBIOSCODE32_SEL 13 BIOS 32-bit interface (32bit Code) */ { .ssd_base = 0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMERA, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 1 }, /* GBIOSCODE16_SEL 14 BIOS 32-bit interface (16bit Code) */ { .ssd_base = 0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMERA, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 1 }, /* GBIOSDATA_SEL 15 BIOS 32-bit interface (Data) */ { .ssd_base = 0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* GBIOSUTIL_SEL 16 BIOS 16-bit interface (Utility) */ { .ssd_base = 0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 1 }, /* GBIOSARGS_SEL 17 BIOS 16-bit interface (Arguments) */ { .ssd_base = 0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = 0, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 1 }, /* GNDIS_SEL 18 NDIS Descriptor */ { .ssd_base = 0x0, .ssd_limit = 0x0, .ssd_type = 0, .ssd_dpl = 0, .ssd_p = 0, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, #endif /* !XEN */ }; static struct soft_segment_descriptor ldt_segs[] = { /* Null Descriptor - overwritten by call gate */ { .ssd_base = 0x0, .ssd_limit = 0x0, .ssd_type = 0, .ssd_dpl = 0, .ssd_p = 0, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* Null Descriptor - overwritten by call gate */ { .ssd_base = 0x0, .ssd_limit = 0x0, .ssd_type = 0, .ssd_dpl = 0, .ssd_p = 0, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* Null Descriptor - overwritten by call gate */ { .ssd_base = 0x0, .ssd_limit = 0x0, .ssd_type = 0, .ssd_dpl = 0, .ssd_p = 0, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* Code Descriptor for user */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMERA, .ssd_dpl = SEL_UPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, /* Null Descriptor - overwritten by call gate */ { .ssd_base = 0x0, .ssd_limit = 0x0, .ssd_type = 0, .ssd_dpl = 0, .ssd_p = 0, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 0, .ssd_gran = 0 }, /* Data Descriptor for user */ { .ssd_base = 0x0, .ssd_limit = 0xfffff, .ssd_type = SDT_MEMRWA, .ssd_dpl = SEL_UPL, .ssd_p = 1, .ssd_xx = 0, .ssd_xx1 = 0, .ssd_def32 = 1, .ssd_gran = 1 }, }; void setidt(idx, func, typ, dpl, selec) int idx; inthand_t *func; int typ; int dpl; int selec; { struct gate_descriptor *ip; ip = idt + idx; ip->gd_looffset = (int)func; ip->gd_selector = selec; ip->gd_stkcpy = 0; ip->gd_xx = 0; ip->gd_type = typ; ip->gd_dpl = dpl; ip->gd_p = 1; ip->gd_hioffset = ((int)func)>>16 ; } extern inthand_t IDTVEC(div), IDTVEC(dbg), IDTVEC(nmi), IDTVEC(bpt), IDTVEC(ofl), IDTVEC(bnd), IDTVEC(ill), IDTVEC(dna), IDTVEC(fpusegm), IDTVEC(tss), IDTVEC(missing), IDTVEC(stk), IDTVEC(prot), IDTVEC(page), IDTVEC(mchk), IDTVEC(rsvd), IDTVEC(fpu), IDTVEC(align), IDTVEC(xmm), #ifdef KDTRACE_HOOKS IDTVEC(dtrace_ret), #endif #ifdef XENHVM IDTVEC(xen_intr_upcall), #endif IDTVEC(lcall_syscall), IDTVEC(int0x80_syscall); #ifdef DDB /* * Display the index and function name of any IDT entries that don't use * the default 'rsvd' entry point. */ DB_SHOW_COMMAND(idt, db_show_idt) { struct gate_descriptor *ip; int idx; uintptr_t func; ip = idt; for (idx = 0; idx < NIDT && !db_pager_quit; idx++) { func = (ip->gd_hioffset << 16 | ip->gd_looffset); if (func != (uintptr_t)&IDTVEC(rsvd)) { db_printf("%3d\t", idx); db_printsym(func, DB_STGY_PROC); db_printf("\n"); } ip++; } } /* Show privileged registers. */ DB_SHOW_COMMAND(sysregs, db_show_sysregs) { uint64_t idtr, gdtr; idtr = ridt(); db_printf("idtr\t0x%08x/%04x\n", (u_int)(idtr >> 16), (u_int)idtr & 0xffff); gdtr = rgdt(); db_printf("gdtr\t0x%08x/%04x\n", (u_int)(gdtr >> 16), (u_int)gdtr & 0xffff); db_printf("ldtr\t0x%04x\n", rldt()); db_printf("tr\t0x%04x\n", rtr()); db_printf("cr0\t0x%08x\n", rcr0()); db_printf("cr2\t0x%08x\n", rcr2()); db_printf("cr3\t0x%08x\n", rcr3()); db_printf("cr4\t0x%08x\n", rcr4()); } #endif void sdtossd(sd, ssd) struct segment_descriptor *sd; struct soft_segment_descriptor *ssd; { ssd->ssd_base = (sd->sd_hibase << 24) | sd->sd_lobase; ssd->ssd_limit = (sd->sd_hilimit << 16) | sd->sd_lolimit; ssd->ssd_type = sd->sd_type; ssd->ssd_dpl = sd->sd_dpl; ssd->ssd_p = sd->sd_p; ssd->ssd_def32 = sd->sd_def32; ssd->ssd_gran = sd->sd_gran; } #if !defined(PC98) && !defined(XEN) static int add_physmap_entry(uint64_t base, uint64_t length, vm_paddr_t *physmap, int *physmap_idxp) { int i, insert_idx, physmap_idx; physmap_idx = *physmap_idxp; if (length == 0) return (1); #ifndef PAE if (base > 0xffffffff) { printf("%uK of memory above 4GB ignored\n", (u_int)(length / 1024)); return (1); } #endif /* * Find insertion point while checking for overlap. Start off by * assuming the new entry will be added to the end. */ insert_idx = physmap_idx + 2; for (i = 0; i <= physmap_idx; i += 2) { if (base < physmap[i + 1]) { if (base + length <= physmap[i]) { insert_idx = i; break; } if (boothowto & RB_VERBOSE) printf( "Overlapping memory regions, ignoring second region\n"); return (1); } } /* See if we can prepend to the next entry. */ if (insert_idx <= physmap_idx && base + length == physmap[insert_idx]) { physmap[insert_idx] = base; return (1); } /* See if we can append to the previous entry. */ if (insert_idx > 0 && base == physmap[insert_idx - 1]) { physmap[insert_idx - 1] += length; return (1); } physmap_idx += 2; *physmap_idxp = physmap_idx; if (physmap_idx == PHYSMAP_SIZE) { printf( "Too many segments in the physical address map, giving up\n"); return (0); } /* * Move the last 'N' entries down to make room for the new * entry if needed. */ for (i = physmap_idx; i > insert_idx; i -= 2) { physmap[i] = physmap[i - 2]; physmap[i + 1] = physmap[i - 1]; } /* Insert the new entry. */ physmap[insert_idx] = base; physmap[insert_idx + 1] = base + length; return (1); } static int add_smap_entry(struct bios_smap *smap, vm_paddr_t *physmap, int *physmap_idxp) { if (boothowto & RB_VERBOSE) printf("SMAP type=%02x base=%016llx len=%016llx\n", smap->type, smap->base, smap->length); if (smap->type != SMAP_TYPE_MEMORY) return (1); return (add_physmap_entry(smap->base, smap->length, physmap, physmap_idxp)); } static void add_smap_entries(struct bios_smap *smapbase, vm_paddr_t *physmap, int *physmap_idxp) { struct bios_smap *smap, *smapend; u_int32_t smapsize; /* * Memory map from INT 15:E820. * * subr_module.c says: * "Consumer may safely assume that size value precedes data." * ie: an int32_t immediately precedes SMAP. */ smapsize = *((u_int32_t *)smapbase - 1); smapend = (struct bios_smap *)((uintptr_t)smapbase + smapsize); for (smap = smapbase; smap < smapend; smap++) if (!add_smap_entry(smap, physmap, physmap_idxp)) break; } #endif /* !PC98 && !XEN */ #ifndef XEN static void basemem_setup(void) { vm_paddr_t pa; pt_entry_t *pte; int i; if (basemem > 640) { printf("Preposterous BIOS basemem of %uK, truncating to 640K\n", basemem); basemem = 640; } /* * XXX if biosbasemem is now < 640, there is a `hole' * between the end of base memory and the start of * ISA memory. The hole may be empty or it may * contain BIOS code or data. Map it read/write so * that the BIOS can write to it. (Memory from 0 to * the physical end of the kernel is mapped read-only * to begin with and then parts of it are remapped. * The parts that aren't remapped form holes that * remain read-only and are unused by the kernel. * The base memory area is below the physical end of * the kernel and right now forms a read-only hole. * The part of it from PAGE_SIZE to * (trunc_page(biosbasemem * 1024) - 1) will be * remapped and used by the kernel later.) * * This code is similar to the code used in * pmap_mapdev, but since no memory needs to be * allocated we simply change the mapping. */ for (pa = trunc_page(basemem * 1024); pa < ISA_HOLE_START; pa += PAGE_SIZE) pmap_kenter(KERNBASE + pa, pa); /* * Map pages between basemem and ISA_HOLE_START, if any, r/w into * the vm86 page table so that vm86 can scribble on them using * the vm86 map too. XXX: why 2 ways for this and only 1 way for * page 0, at least as initialized here? */ pte = (pt_entry_t *)vm86paddr; for (i = basemem / 4; i < 160; i++) pte[i] = (i << PAGE_SHIFT) | PG_V | PG_RW | PG_U; } #endif /* !XEN */ /* * Populate the (physmap) array with base/bound pairs describing the * available physical memory in the system, then test this memory and * build the phys_avail array describing the actually-available memory. * * If we cannot accurately determine the physical memory map, then use * value from the 0xE801 call, and failing that, the RTC. * * Total memory size may be set by the kernel environment variable * hw.physmem or the compile-time define MAXMEM. * * XXX first should be vm_paddr_t. */ #ifdef PC98 static void getmemsize(int first) { int off, physmap_idx, pa_indx, da_indx; u_long physmem_tunable, memtest; vm_paddr_t physmap[PHYSMAP_SIZE]; pt_entry_t *pte; quad_t dcons_addr, dcons_size; int i; int pg_n; u_int extmem; u_int under16; vm_paddr_t pa; bzero(physmap, sizeof(physmap)); /* XXX - some of EPSON machines can't use PG_N */ pg_n = PG_N; if (pc98_machine_type & M_EPSON_PC98) { switch (epson_machine_id) { #ifdef WB_CACHE default: #endif case EPSON_PC486_HX: case EPSON_PC486_HG: case EPSON_PC486_HA: pg_n = 0; break; } } under16 = pc98_getmemsize(&basemem, &extmem); basemem_setup(); physmap[0] = 0; physmap[1] = basemem * 1024; physmap_idx = 2; physmap[physmap_idx] = 0x100000; physmap[physmap_idx + 1] = physmap[physmap_idx] + extmem * 1024; /* * Now, physmap contains a map of physical memory. */ #ifdef SMP /* make hole for AP bootstrap code */ physmap[1] = mp_bootaddress(physmap[1]); #endif /* * Maxmem isn't the "maximum memory", it's one larger than the * highest page of the physical address space. It should be * called something like "Maxphyspage". We may adjust this * based on ``hw.physmem'' and the results of the memory test. */ Maxmem = atop(physmap[physmap_idx + 1]); #ifdef MAXMEM Maxmem = MAXMEM / 4; #endif if (TUNABLE_ULONG_FETCH("hw.physmem", &physmem_tunable)) Maxmem = atop(physmem_tunable); /* * By default keep the memtest enabled. Use a general name so that * one could eventually do more with the code than just disable it. */ memtest = 1; TUNABLE_ULONG_FETCH("hw.memtest.tests", &memtest); if (atop(physmap[physmap_idx + 1]) != Maxmem && (boothowto & RB_VERBOSE)) printf("Physical memory use set to %ldK\n", Maxmem * 4); /* * If Maxmem has been increased beyond what the system has detected, * extend the last memory segment to the new limit. */ if (atop(physmap[physmap_idx + 1]) < Maxmem) physmap[physmap_idx + 1] = ptoa((vm_paddr_t)Maxmem); /* * We need to divide chunk if Maxmem is larger than 16MB and * under 16MB area is not full of memory. * (1) system area (15-16MB region) is cut off * (2) extended memory is only over 16MB area (ex. Melco "HYPERMEMORY") */ if ((under16 != 16 * 1024) && (extmem > 15 * 1024)) { /* 15M - 16M region is cut off, so need to divide chunk */ physmap[physmap_idx + 1] = under16 * 1024; physmap_idx += 2; physmap[physmap_idx] = 0x1000000; physmap[physmap_idx + 1] = physmap[2] + extmem * 1024; } /* call pmap initialization to make new kernel address space */ pmap_bootstrap(first); /* * Size up each available chunk of physical memory. */ physmap[0] = PAGE_SIZE; /* mask off page 0 */ pa_indx = 0; da_indx = 1; phys_avail[pa_indx++] = physmap[0]; phys_avail[pa_indx] = physmap[0]; dump_avail[da_indx] = physmap[0]; pte = CMAP3; /* * Get dcons buffer address */ if (getenv_quad("dcons.addr", &dcons_addr) == 0 || getenv_quad("dcons.size", &dcons_size) == 0) dcons_addr = 0; /* * physmap is in bytes, so when converting to page boundaries, * round up the start address and round down the end address. */ for (i = 0; i <= physmap_idx; i += 2) { vm_paddr_t end; end = ptoa((vm_paddr_t)Maxmem); if (physmap[i + 1] < end) end = trunc_page(physmap[i + 1]); for (pa = round_page(physmap[i]); pa < end; pa += PAGE_SIZE) { int tmp, page_bad, full; int *ptr = (int *)CADDR3; full = FALSE; /* * block out kernel memory as not available. */ if (pa >= KERNLOAD && pa < first) goto do_dump_avail; /* * block out dcons buffer */ if (dcons_addr > 0 && pa >= trunc_page(dcons_addr) && pa < dcons_addr + dcons_size) goto do_dump_avail; page_bad = FALSE; if (memtest == 0) goto skip_memtest; /* * map page into kernel: valid, read/write,non-cacheable */ *pte = pa | PG_V | PG_RW | pg_n; invltlb(); tmp = *(int *)ptr; /* * Test for alternating 1's and 0's */ *(volatile int *)ptr = 0xaaaaaaaa; if (*(volatile int *)ptr != 0xaaaaaaaa) page_bad = TRUE; /* * Test for alternating 0's and 1's */ *(volatile int *)ptr = 0x55555555; if (*(volatile int *)ptr != 0x55555555) page_bad = TRUE; /* * Test for all 1's */ *(volatile int *)ptr = 0xffffffff; if (*(volatile int *)ptr != 0xffffffff) page_bad = TRUE; /* * Test for all 0's */ *(volatile int *)ptr = 0x0; if (*(volatile int *)ptr != 0x0) page_bad = TRUE; /* * Restore original value. */ *(int *)ptr = tmp; skip_memtest: /* * Adjust array of valid/good pages. */ if (page_bad == TRUE) continue; /* * If this good page is a continuation of the * previous set of good pages, then just increase * the end pointer. Otherwise start a new chunk. * Note that "end" points one higher than end, * making the range >= start and < end. * If we're also doing a speculative memory * test and we at or past the end, bump up Maxmem * so that we keep going. The first bad page * will terminate the loop. */ if (phys_avail[pa_indx] == pa) { phys_avail[pa_indx] += PAGE_SIZE; } else { pa_indx++; if (pa_indx == PHYS_AVAIL_ARRAY_END) { printf( "Too many holes in the physical address space, giving up\n"); pa_indx--; full = TRUE; goto do_dump_avail; } phys_avail[pa_indx++] = pa; /* start */ phys_avail[pa_indx] = pa + PAGE_SIZE; /* end */ } physmem++; do_dump_avail: if (dump_avail[da_indx] == pa) { dump_avail[da_indx] += PAGE_SIZE; } else { da_indx++; if (da_indx == DUMP_AVAIL_ARRAY_END) { da_indx--; goto do_next; } dump_avail[da_indx++] = pa; /* start */ dump_avail[da_indx] = pa + PAGE_SIZE; /* end */ } do_next: if (full) break; } } *pte = 0; invltlb(); /* * XXX * The last chunk must contain at least one page plus the message * buffer to avoid complicating other code (message buffer address * calculation, etc.). */ while (phys_avail[pa_indx - 1] + PAGE_SIZE + round_page(msgbufsize) >= phys_avail[pa_indx]) { physmem -= atop(phys_avail[pa_indx] - phys_avail[pa_indx - 1]); phys_avail[pa_indx--] = 0; phys_avail[pa_indx--] = 0; } Maxmem = atop(phys_avail[pa_indx]); /* Trim off space for the message buffer. */ phys_avail[pa_indx] -= round_page(msgbufsize); /* Map the message buffer. */ for (off = 0; off < round_page(msgbufsize); off += PAGE_SIZE) pmap_kenter((vm_offset_t)msgbufp + off, phys_avail[pa_indx] + off); PT_UPDATES_FLUSH(); } #else /* PC98 */ static void getmemsize(int first) { int has_smap, off, physmap_idx, pa_indx, da_indx; u_long memtest; vm_paddr_t physmap[PHYSMAP_SIZE]; pt_entry_t *pte; quad_t dcons_addr, dcons_size, physmem_tunable; #ifndef XEN int hasbrokenint12, i, res; u_int extmem; struct vm86frame vmf; struct vm86context vmc; vm_paddr_t pa; struct bios_smap *smap, *smapbase; caddr_t kmdp; #endif has_smap = 0; #if defined(XEN) Maxmem = xen_start_info->nr_pages - init_first; physmem = Maxmem; basemem = 0; physmap[0] = init_first << PAGE_SHIFT; physmap[1] = ptoa(Maxmem) - round_page(msgbufsize); physmap_idx = 0; #else #ifdef XBOX if (arch_i386_is_xbox) { /* * We queried the memory size before, so chop off 4MB for * the framebuffer and inform the OS of this. */ physmap[0] = 0; physmap[1] = (arch_i386_xbox_memsize * 1024 * 1024) - XBOX_FB_SIZE; physmap_idx = 0; goto physmap_done; } #endif bzero(&vmf, sizeof(vmf)); bzero(physmap, sizeof(physmap)); basemem = 0; /* * Check if the loader supplied an SMAP memory map. If so, * use that and do not make any VM86 calls. */ physmap_idx = 0; smapbase = NULL; kmdp = preload_search_by_type("elf kernel"); if (kmdp == NULL) kmdp = preload_search_by_type("elf32 kernel"); if (kmdp != NULL) smapbase = (struct bios_smap *)preload_search_info(kmdp, MODINFO_METADATA | MODINFOMD_SMAP); if (smapbase != NULL) { add_smap_entries(smapbase, physmap, &physmap_idx); has_smap = 1; goto have_smap; } /* * Some newer BIOSes have a broken INT 12H implementation * which causes a kernel panic immediately. In this case, we * need use the SMAP to determine the base memory size. */ hasbrokenint12 = 0; TUNABLE_INT_FETCH("hw.hasbrokenint12", &hasbrokenint12); if (hasbrokenint12 == 0) { /* Use INT12 to determine base memory size. */ vm86_intcall(0x12, &vmf); basemem = vmf.vmf_ax; basemem_setup(); } /* * Fetch the memory map with INT 15:E820. Map page 1 R/W into * the kernel page table so we can use it as a buffer. The * kernel will unmap this page later. */ pmap_kenter(KERNBASE + (1 << PAGE_SHIFT), 1 << PAGE_SHIFT); vmc.npages = 0; smap = (void *)vm86_addpage(&vmc, 1, KERNBASE + (1 << PAGE_SHIFT)); res = vm86_getptr(&vmc, (vm_offset_t)smap, &vmf.vmf_es, &vmf.vmf_di); KASSERT(res != 0, ("vm86_getptr() failed: address not found")); vmf.vmf_ebx = 0; do { vmf.vmf_eax = 0xE820; vmf.vmf_edx = SMAP_SIG; vmf.vmf_ecx = sizeof(struct bios_smap); i = vm86_datacall(0x15, &vmf, &vmc); if (i || vmf.vmf_eax != SMAP_SIG) break; has_smap = 1; if (!add_smap_entry(smap, physmap, &physmap_idx)) break; } while (vmf.vmf_ebx != 0); have_smap: /* * If we didn't fetch the "base memory" size from INT12, * figure it out from the SMAP (or just guess). */ if (basemem == 0) { for (i = 0; i <= physmap_idx; i += 2) { if (physmap[i] == 0x00000000) { basemem = physmap[i + 1] / 1024; break; } } /* XXX: If we couldn't find basemem from SMAP, just guess. */ if (basemem == 0) basemem = 640; basemem_setup(); } if (physmap[1] != 0) goto physmap_done; /* * If we failed to find an SMAP, figure out the extended * memory size. We will then build a simple memory map with * two segments, one for "base memory" and the second for * "extended memory". Note that "extended memory" starts at a * physical address of 1MB and that both basemem and extmem * are in units of 1KB. * * First, try to fetch the extended memory size via INT 15:E801. */ vmf.vmf_ax = 0xE801; if (vm86_intcall(0x15, &vmf) == 0) { extmem = vmf.vmf_cx + vmf.vmf_dx * 64; } else { /* * If INT15:E801 fails, this is our last ditch effort * to determine the extended memory size. Currently * we prefer the RTC value over INT15:88. */ #if 0 vmf.vmf_ah = 0x88; vm86_intcall(0x15, &vmf); extmem = vmf.vmf_ax; #else extmem = rtcin(RTC_EXTLO) + (rtcin(RTC_EXTHI) << 8); #endif } /* * Special hack for chipsets that still remap the 384k hole when * there's 16MB of memory - this really confuses people that * are trying to use bus mastering ISA controllers with the * "16MB limit"; they only have 16MB, but the remapping puts * them beyond the limit. * * If extended memory is between 15-16MB (16-17MB phys address range), * chop it to 15MB. */ if ((extmem > 15 * 1024) && (extmem < 16 * 1024)) extmem = 15 * 1024; physmap[0] = 0; physmap[1] = basemem * 1024; physmap_idx = 2; physmap[physmap_idx] = 0x100000; physmap[physmap_idx + 1] = physmap[physmap_idx] + extmem * 1024; physmap_done: #endif /* * Now, physmap contains a map of physical memory. */ #ifdef SMP /* make hole for AP bootstrap code */ physmap[1] = mp_bootaddress(physmap[1]); #endif /* * Maxmem isn't the "maximum memory", it's one larger than the * highest page of the physical address space. It should be * called something like "Maxphyspage". We may adjust this * based on ``hw.physmem'' and the results of the memory test. */ Maxmem = atop(physmap[physmap_idx + 1]); #ifdef MAXMEM Maxmem = MAXMEM / 4; #endif if (TUNABLE_QUAD_FETCH("hw.physmem", &physmem_tunable)) Maxmem = atop(physmem_tunable); /* * If we have an SMAP, don't allow MAXMEM or hw.physmem to extend * the amount of memory in the system. */ if (has_smap && Maxmem > atop(physmap[physmap_idx + 1])) Maxmem = atop(physmap[physmap_idx + 1]); /* * By default enable the memory test on real hardware, and disable * it if we appear to be running in a VM. This avoids touching all * pages unnecessarily, which doesn't matter on real hardware but is * bad for shared VM hosts. Use a general name so that * one could eventually do more with the code than just disable it. */ memtest = (vm_guest > VM_GUEST_NO) ? 0 : 1; TUNABLE_ULONG_FETCH("hw.memtest.tests", &memtest); if (atop(physmap[physmap_idx + 1]) != Maxmem && (boothowto & RB_VERBOSE)) printf("Physical memory use set to %ldK\n", Maxmem * 4); /* * If Maxmem has been increased beyond what the system has detected, * extend the last memory segment to the new limit. */ if (atop(physmap[physmap_idx + 1]) < Maxmem) physmap[physmap_idx + 1] = ptoa((vm_paddr_t)Maxmem); /* call pmap initialization to make new kernel address space */ pmap_bootstrap(first); /* * Size up each available chunk of physical memory. */ physmap[0] = PAGE_SIZE; /* mask off page 0 */ pa_indx = 0; da_indx = 1; phys_avail[pa_indx++] = physmap[0]; phys_avail[pa_indx] = physmap[0]; dump_avail[da_indx] = physmap[0]; pte = CMAP3; /* * Get dcons buffer address */ if (getenv_quad("dcons.addr", &dcons_addr) == 0 || getenv_quad("dcons.size", &dcons_size) == 0) dcons_addr = 0; #ifndef XEN /* * physmap is in bytes, so when converting to page boundaries, * round up the start address and round down the end address. */ for (i = 0; i <= physmap_idx; i += 2) { vm_paddr_t end; end = ptoa((vm_paddr_t)Maxmem); if (physmap[i + 1] < end) end = trunc_page(physmap[i + 1]); for (pa = round_page(physmap[i]); pa < end; pa += PAGE_SIZE) { int tmp, page_bad, full; int *ptr = (int *)CADDR3; full = FALSE; /* * block out kernel memory as not available. */ if (pa >= KERNLOAD && pa < first) goto do_dump_avail; /* * block out dcons buffer */ if (dcons_addr > 0 && pa >= trunc_page(dcons_addr) && pa < dcons_addr + dcons_size) goto do_dump_avail; page_bad = FALSE; if (memtest == 0) goto skip_memtest; /* * map page into kernel: valid, read/write,non-cacheable */ *pte = pa | PG_V | PG_RW | PG_N; invltlb(); tmp = *(int *)ptr; /* * Test for alternating 1's and 0's */ *(volatile int *)ptr = 0xaaaaaaaa; if (*(volatile int *)ptr != 0xaaaaaaaa) page_bad = TRUE; /* * Test for alternating 0's and 1's */ *(volatile int *)ptr = 0x55555555; if (*(volatile int *)ptr != 0x55555555) page_bad = TRUE; /* * Test for all 1's */ *(volatile int *)ptr = 0xffffffff; if (*(volatile int *)ptr != 0xffffffff) page_bad = TRUE; /* * Test for all 0's */ *(volatile int *)ptr = 0x0; if (*(volatile int *)ptr != 0x0) page_bad = TRUE; /* * Restore original value. */ *(int *)ptr = tmp; skip_memtest: /* * Adjust array of valid/good pages. */ if (page_bad == TRUE) continue; /* * If this good page is a continuation of the * previous set of good pages, then just increase * the end pointer. Otherwise start a new chunk. * Note that "end" points one higher than end, * making the range >= start and < end. * If we're also doing a speculative memory * test and we at or past the end, bump up Maxmem * so that we keep going. The first bad page * will terminate the loop. */ if (phys_avail[pa_indx] == pa) { phys_avail[pa_indx] += PAGE_SIZE; } else { pa_indx++; if (pa_indx == PHYS_AVAIL_ARRAY_END) { printf( "Too many holes in the physical address space, giving up\n"); pa_indx--; full = TRUE; goto do_dump_avail; } phys_avail[pa_indx++] = pa; /* start */ phys_avail[pa_indx] = pa + PAGE_SIZE; /* end */ } physmem++; do_dump_avail: if (dump_avail[da_indx] == pa) { dump_avail[da_indx] += PAGE_SIZE; } else { da_indx++; if (da_indx == DUMP_AVAIL_ARRAY_END) { da_indx--; goto do_next; } dump_avail[da_indx++] = pa; /* start */ dump_avail[da_indx] = pa + PAGE_SIZE; /* end */ } do_next: if (full) break; } } *pte = 0; invltlb(); #else phys_avail[0] = physfree; phys_avail[1] = xen_start_info->nr_pages*PAGE_SIZE; dump_avail[0] = 0; dump_avail[1] = xen_start_info->nr_pages*PAGE_SIZE; #endif /* * XXX * The last chunk must contain at least one page plus the message * buffer to avoid complicating other code (message buffer address * calculation, etc.). */ while (phys_avail[pa_indx - 1] + PAGE_SIZE + round_page(msgbufsize) >= phys_avail[pa_indx]) { physmem -= atop(phys_avail[pa_indx] - phys_avail[pa_indx - 1]); phys_avail[pa_indx--] = 0; phys_avail[pa_indx--] = 0; } Maxmem = atop(phys_avail[pa_indx]); /* Trim off space for the message buffer. */ phys_avail[pa_indx] -= round_page(msgbufsize); /* Map the message buffer. */ for (off = 0; off < round_page(msgbufsize); off += PAGE_SIZE) pmap_kenter((vm_offset_t)msgbufp + off, phys_avail[pa_indx] + off); PT_UPDATES_FLUSH(); } #endif /* PC98 */ #ifdef XEN #define MTOPSIZE (1<<(14 + PAGE_SHIFT)) register_t init386(first) int first; { unsigned long gdtmachpfn; int error, gsel_tss, metadata_missing, x, pa; struct pcpu *pc; #ifdef CPU_ENABLE_SSE struct xstate_hdr *xhdr; #endif struct callback_register event = { .type = CALLBACKTYPE_event, .address = {GSEL(GCODE_SEL, SEL_KPL), (unsigned long)Xhypervisor_callback }, }; struct callback_register failsafe = { .type = CALLBACKTYPE_failsafe, .address = {GSEL(GCODE_SEL, SEL_KPL), (unsigned long)failsafe_callback }, }; thread0.td_kstack = proc0kstack; thread0.td_kstack_pages = KSTACK_PAGES; /* * This may be done better later if it gets more high level * components in it. If so just link td->td_proc here. */ proc_linkup0(&proc0, &thread0); metadata_missing = 0; if (xen_start_info->mod_start) { preload_metadata = (caddr_t)xen_start_info->mod_start; preload_bootstrap_relocate(KERNBASE); } else { metadata_missing = 1; } if (envmode == 1) kern_envp = static_env; else if ((caddr_t)xen_start_info->cmd_line) kern_envp = xen_setbootenv((caddr_t)xen_start_info->cmd_line); boothowto |= xen_boothowto(kern_envp); /* Init basic tunables, hz etc */ init_param1(); /* * XEN occupies a portion of the upper virtual address space * At its base it manages an array mapping machine page frames * to physical page frames - hence we need to be able to * access 4GB - (64MB - 4MB + 64k) */ gdt_segs[GPRIV_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); gdt_segs[GUFS_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); gdt_segs[GUGS_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); gdt_segs[GCODE_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); gdt_segs[GDATA_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); gdt_segs[GUCODE_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); gdt_segs[GUDATA_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); gdt_segs[GBIOSLOWMEM_SEL].ssd_limit = atop(HYPERVISOR_VIRT_START + MTOPSIZE); pc = &__pcpu[0]; gdt_segs[GPRIV_SEL].ssd_base = (int) pc; gdt_segs[GPROC0_SEL].ssd_base = (int) &pc->pc_common_tss; PT_SET_MA(gdt, xpmap_ptom(VTOP(gdt)) | PG_V | PG_RW); bzero(gdt, PAGE_SIZE); for (x = 0; x < NGDT; x++) ssdtosd(&gdt_segs[x], &gdt[x].sd); mtx_init(&dt_lock, "descriptor tables", NULL, MTX_SPIN); gdtmachpfn = vtomach(gdt) >> PAGE_SHIFT; PT_SET_MA(gdt, xpmap_ptom(VTOP(gdt)) | PG_V); PANIC_IF(HYPERVISOR_set_gdt(&gdtmachpfn, 512) != 0); lgdt(&r_gdt); gdtset = 1; if ((error = HYPERVISOR_set_trap_table(trap_table)) != 0) { panic("set_trap_table failed - error %d\n", error); } error = HYPERVISOR_callback_op(CALLBACKOP_register, &event); if (error == 0) error = HYPERVISOR_callback_op(CALLBACKOP_register, &failsafe); #if CONFIG_XEN_COMPAT <= 0x030002 if (error == -ENOXENSYS) HYPERVISOR_set_callbacks(GSEL(GCODE_SEL, SEL_KPL), (unsigned long)Xhypervisor_callback, GSEL(GCODE_SEL, SEL_KPL), (unsigned long)failsafe_callback); #endif pcpu_init(pc, 0, sizeof(struct pcpu)); for (pa = first; pa < first + DPCPU_SIZE; pa += PAGE_SIZE) pmap_kenter(pa + KERNBASE, pa); dpcpu_init((void *)(first + KERNBASE), 0); first += DPCPU_SIZE; physfree += DPCPU_SIZE; init_first += DPCPU_SIZE / PAGE_SIZE; PCPU_SET(prvspace, pc); PCPU_SET(curthread, &thread0); /* * Initialize mutexes. * * icu_lock: in order to allow an interrupt to occur in a critical * section, to set pcpu->ipending (etc...) properly, we * must be able to get the icu lock, so it can't be * under witness. */ mutex_init(); mtx_init(&icu_lock, "icu", NULL, MTX_SPIN | MTX_NOWITNESS | MTX_NOPROFILE); /* make ldt memory segments */ PT_SET_MA(ldt, xpmap_ptom(VTOP(ldt)) | PG_V | PG_RW); bzero(ldt, PAGE_SIZE); ldt_segs[LUCODE_SEL].ssd_limit = atop(0 - 1); ldt_segs[LUDATA_SEL].ssd_limit = atop(0 - 1); for (x = 0; x < sizeof ldt_segs / sizeof ldt_segs[0]; x++) ssdtosd(&ldt_segs[x], &ldt[x].sd); default_proc_ldt.ldt_base = (caddr_t)ldt; default_proc_ldt.ldt_len = 6; _default_ldt = (int)&default_proc_ldt; PCPU_SET(currentldt, _default_ldt); PT_SET_MA(ldt, *vtopte((unsigned long)ldt) & ~PG_RW); xen_set_ldt((unsigned long) ldt, (sizeof ldt_segs / sizeof ldt_segs[0])); #if defined(XEN_PRIVILEGED) /* * Initialize the i8254 before the console so that console * initialization can use DELAY(). */ i8254_init(); #endif /* * Initialize the console before we print anything out. */ cninit(); if (metadata_missing) printf("WARNING: loader(8) metadata is missing!\n"); #ifdef DEV_ISA #ifdef DEV_ATPIC elcr_probe(); atpic_startup(); #else /* Reset and mask the atpics and leave them shut down. */ atpic_reset(); /* * Point the ICU spurious interrupt vectors at the APIC spurious * interrupt handler. */ setidt(IDT_IO_INTS + 7, IDTVEC(spuriousint), SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_IO_INTS + 15, IDTVEC(spuriousint), SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); #endif #endif #ifdef DDB db_fetch_ksymtab(bootinfo.bi_symtab, bootinfo.bi_esymtab); #endif kdb_init(); #ifdef KDB if (boothowto & RB_KDB) kdb_enter(KDB_WHY_BOOTFLAGS, "Boot flags requested debugger"); #endif finishidentcpu(); /* Final stage of CPU initialization */ setidt(IDT_UD, &IDTVEC(ill), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_GP, &IDTVEC(prot), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); initializecpu(); /* Initialize CPU registers */ initializecpucache(); /* pointer to selector slot for %fs/%gs */ PCPU_SET(fsgs_gdt, &gdt[GUFS_SEL].sd); dblfault_tss.tss_esp = dblfault_tss.tss_esp0 = dblfault_tss.tss_esp1 = dblfault_tss.tss_esp2 = (int)&dblfault_stack[sizeof(dblfault_stack)]; dblfault_tss.tss_ss = dblfault_tss.tss_ss0 = dblfault_tss.tss_ss1 = dblfault_tss.tss_ss2 = GSEL(GDATA_SEL, SEL_KPL); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) dblfault_tss.tss_cr3 = (int)IdlePDPT; #else dblfault_tss.tss_cr3 = (int)IdlePTD; #endif dblfault_tss.tss_eip = (int)dblfault_handler; dblfault_tss.tss_eflags = PSL_KERNEL; dblfault_tss.tss_ds = dblfault_tss.tss_es = dblfault_tss.tss_gs = GSEL(GDATA_SEL, SEL_KPL); dblfault_tss.tss_fs = GSEL(GPRIV_SEL, SEL_KPL); dblfault_tss.tss_cs = GSEL(GCODE_SEL, SEL_KPL); dblfault_tss.tss_ldt = GSEL(GLDT_SEL, SEL_KPL); vm86_initialize(); getmemsize(first); init_param2(physmem); /* now running on new page tables, configured,and u/iom is accessible */ msgbufinit(msgbufp, msgbufsize); #ifdef DEV_NPX npxinit(true); #endif /* * Set up thread0 pcb after npxinit calculated pcb + fpu save * area size. Zero out the extended state header in fpu save * area. */ thread0.td_pcb = get_pcb_td(&thread0); bzero(get_pcb_user_save_td(&thread0), cpu_max_ext_state_size); #ifdef CPU_ENABLE_SSE if (use_xsave) { xhdr = (struct xstate_hdr *)(get_pcb_user_save_td(&thread0) + 1); xhdr->xstate_bv = xsave_mask; } #endif PCPU_SET(curpcb, thread0.td_pcb); /* make an initial tss so cpu can get interrupt stack on syscall! */ /* Note: -16 is so we can grow the trapframe if we came from vm86 */ PCPU_SET(common_tss.tss_esp0, (vm_offset_t)thread0.td_pcb - 16); PCPU_SET(common_tss.tss_ss0, GSEL(GDATA_SEL, SEL_KPL)); gsel_tss = GSEL(GPROC0_SEL, SEL_KPL); HYPERVISOR_stack_switch(GSEL(GDATA_SEL, SEL_KPL), PCPU_GET(common_tss.tss_esp0)); /* transfer to user mode */ _ucodesel = GSEL(GUCODE_SEL, SEL_UPL); _udatasel = GSEL(GUDATA_SEL, SEL_UPL); /* setup proc 0's pcb */ thread0.td_pcb->pcb_flags = 0; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) thread0.td_pcb->pcb_cr3 = (int)IdlePDPT; #else thread0.td_pcb->pcb_cr3 = (int)IdlePTD; #endif thread0.td_pcb->pcb_ext = 0; thread0.td_frame = &proc0_tf; thread0.td_pcb->pcb_fsd = PCPU_GET(fsgs_gdt)[0]; thread0.td_pcb->pcb_gsd = PCPU_GET(fsgs_gdt)[1]; cpu_probe_amdc1e(); /* Location of kernel stack for locore */ return ((register_t)thread0.td_pcb); } #else register_t init386(first) int first; { struct gate_descriptor *gdp; int gsel_tss, metadata_missing, x, pa; struct pcpu *pc; #ifdef CPU_ENABLE_SSE struct xstate_hdr *xhdr; #endif thread0.td_kstack = proc0kstack; thread0.td_kstack_pages = KSTACK_PAGES; /* * This may be done better later if it gets more high level * components in it. If so just link td->td_proc here. */ proc_linkup0(&proc0, &thread0); #ifdef PC98 /* * Initialize DMAC */ pc98_init_dmac(); #endif metadata_missing = 0; if (bootinfo.bi_modulep) { preload_metadata = (caddr_t)bootinfo.bi_modulep + KERNBASE; preload_bootstrap_relocate(KERNBASE); } else { metadata_missing = 1; } if (envmode == 1) kern_envp = static_env; else if (bootinfo.bi_envp) kern_envp = (caddr_t)bootinfo.bi_envp + KERNBASE; /* Init basic tunables, hz etc */ init_param1(); /* * Make gdt memory segments. All segments cover the full 4GB * of address space and permissions are enforced at page level. */ gdt_segs[GCODE_SEL].ssd_limit = atop(0 - 1); gdt_segs[GDATA_SEL].ssd_limit = atop(0 - 1); gdt_segs[GUCODE_SEL].ssd_limit = atop(0 - 1); gdt_segs[GUDATA_SEL].ssd_limit = atop(0 - 1); gdt_segs[GUFS_SEL].ssd_limit = atop(0 - 1); gdt_segs[GUGS_SEL].ssd_limit = atop(0 - 1); pc = &__pcpu[0]; gdt_segs[GPRIV_SEL].ssd_limit = atop(0 - 1); gdt_segs[GPRIV_SEL].ssd_base = (int) pc; gdt_segs[GPROC0_SEL].ssd_base = (int) &pc->pc_common_tss; for (x = 0; x < NGDT; x++) ssdtosd(&gdt_segs[x], &gdt[x].sd); r_gdt.rd_limit = NGDT * sizeof(gdt[0]) - 1; r_gdt.rd_base = (int) gdt; mtx_init(&dt_lock, "descriptor tables", NULL, MTX_SPIN); lgdt(&r_gdt); pcpu_init(pc, 0, sizeof(struct pcpu)); for (pa = first; pa < first + DPCPU_SIZE; pa += PAGE_SIZE) pmap_kenter(pa + KERNBASE, pa); dpcpu_init((void *)(first + KERNBASE), 0); first += DPCPU_SIZE; PCPU_SET(prvspace, pc); PCPU_SET(curthread, &thread0); /* * Initialize mutexes. * * icu_lock: in order to allow an interrupt to occur in a critical * section, to set pcpu->ipending (etc...) properly, we * must be able to get the icu lock, so it can't be * under witness. */ mutex_init(); mtx_init(&icu_lock, "icu", NULL, MTX_SPIN | MTX_NOWITNESS | MTX_NOPROFILE); /* make ldt memory segments */ ldt_segs[LUCODE_SEL].ssd_limit = atop(0 - 1); ldt_segs[LUDATA_SEL].ssd_limit = atop(0 - 1); for (x = 0; x < sizeof ldt_segs / sizeof ldt_segs[0]; x++) ssdtosd(&ldt_segs[x], &ldt[x].sd); _default_ldt = GSEL(GLDT_SEL, SEL_KPL); lldt(_default_ldt); PCPU_SET(currentldt, _default_ldt); /* exceptions */ for (x = 0; x < NIDT; x++) setidt(x, &IDTVEC(rsvd), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_DE, &IDTVEC(div), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_DB, &IDTVEC(dbg), SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_NMI, &IDTVEC(nmi), SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_BP, &IDTVEC(bpt), SDT_SYS386IGT, SEL_UPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_OF, &IDTVEC(ofl), SDT_SYS386TGT, SEL_UPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_BR, &IDTVEC(bnd), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_UD, &IDTVEC(ill), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_NM, &IDTVEC(dna), SDT_SYS386TGT, SEL_KPL , GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_DF, 0, SDT_SYSTASKGT, SEL_KPL, GSEL(GPANIC_SEL, SEL_KPL)); setidt(IDT_FPUGP, &IDTVEC(fpusegm), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_TS, &IDTVEC(tss), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_NP, &IDTVEC(missing), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_SS, &IDTVEC(stk), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_GP, &IDTVEC(prot), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_PF, &IDTVEC(page), SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_MF, &IDTVEC(fpu), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_AC, &IDTVEC(align), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_MC, &IDTVEC(mchk), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_XF, &IDTVEC(xmm), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_SYSCALL, &IDTVEC(int0x80_syscall), SDT_SYS386TGT, SEL_UPL, GSEL(GCODE_SEL, SEL_KPL)); #ifdef KDTRACE_HOOKS setidt(IDT_DTRACE_RET, &IDTVEC(dtrace_ret), SDT_SYS386TGT, SEL_UPL, GSEL(GCODE_SEL, SEL_KPL)); #endif #ifdef XENHVM setidt(IDT_EVTCHN, &IDTVEC(xen_intr_upcall), SDT_SYS386IGT, SEL_UPL, GSEL(GCODE_SEL, SEL_KPL)); #endif r_idt.rd_limit = sizeof(idt0) - 1; r_idt.rd_base = (int) idt; lidt(&r_idt); #ifdef XBOX /* * The following code queries the PCI ID of 0:0:0. For the XBOX, * This should be 0x10de / 0x02a5. * * This is exactly what Linux does. */ outl(0xcf8, 0x80000000); if (inl(0xcfc) == 0x02a510de) { arch_i386_is_xbox = 1; pic16l_setled(XBOX_LED_GREEN); /* * We are an XBOX, but we may have either 64MB or 128MB of * memory. The PCI host bridge should be programmed for this, * so we just query it. */ outl(0xcf8, 0x80000084); arch_i386_xbox_memsize = (inl(0xcfc) == 0x7FFFFFF) ? 128 : 64; } #endif /* XBOX */ /* * Initialize the clock before the console so that console * initialization can use DELAY(). */ clock_init(); /* * Initialize the console before we print anything out. */ cninit(); if (metadata_missing) printf("WARNING: loader(8) metadata is missing!\n"); #ifdef DEV_ISA #ifdef DEV_ATPIC #ifndef PC98 elcr_probe(); #endif atpic_startup(); #else /* Reset and mask the atpics and leave them shut down. */ atpic_reset(); /* * Point the ICU spurious interrupt vectors at the APIC spurious * interrupt handler. */ setidt(IDT_IO_INTS + 7, IDTVEC(spuriousint), SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_IO_INTS + 15, IDTVEC(spuriousint), SDT_SYS386IGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); #endif #endif #ifdef DDB db_fetch_ksymtab(bootinfo.bi_symtab, bootinfo.bi_esymtab); #endif kdb_init(); #ifdef KDB if (boothowto & RB_KDB) kdb_enter(KDB_WHY_BOOTFLAGS, "Boot flags requested debugger"); #endif finishidentcpu(); /* Final stage of CPU initialization */ setidt(IDT_UD, &IDTVEC(ill), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); setidt(IDT_GP, &IDTVEC(prot), SDT_SYS386TGT, SEL_KPL, GSEL(GCODE_SEL, SEL_KPL)); initializecpu(); /* Initialize CPU registers */ initializecpucache(); /* pointer to selector slot for %fs/%gs */ PCPU_SET(fsgs_gdt, &gdt[GUFS_SEL].sd); dblfault_tss.tss_esp = dblfault_tss.tss_esp0 = dblfault_tss.tss_esp1 = dblfault_tss.tss_esp2 = (int)&dblfault_stack[sizeof(dblfault_stack)]; dblfault_tss.tss_ss = dblfault_tss.tss_ss0 = dblfault_tss.tss_ss1 = dblfault_tss.tss_ss2 = GSEL(GDATA_SEL, SEL_KPL); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) dblfault_tss.tss_cr3 = (int)IdlePDPT; #else dblfault_tss.tss_cr3 = (int)IdlePTD; #endif dblfault_tss.tss_eip = (int)dblfault_handler; dblfault_tss.tss_eflags = PSL_KERNEL; dblfault_tss.tss_ds = dblfault_tss.tss_es = dblfault_tss.tss_gs = GSEL(GDATA_SEL, SEL_KPL); dblfault_tss.tss_fs = GSEL(GPRIV_SEL, SEL_KPL); dblfault_tss.tss_cs = GSEL(GCODE_SEL, SEL_KPL); dblfault_tss.tss_ldt = GSEL(GLDT_SEL, SEL_KPL); vm86_initialize(); getmemsize(first); init_param2(physmem); /* now running on new page tables, configured,and u/iom is accessible */ msgbufinit(msgbufp, msgbufsize); #ifdef DEV_NPX npxinit(true); #endif /* * Set up thread0 pcb after npxinit calculated pcb + fpu save * area size. Zero out the extended state header in fpu save * area. */ thread0.td_pcb = get_pcb_td(&thread0); bzero(get_pcb_user_save_td(&thread0), cpu_max_ext_state_size); #ifdef CPU_ENABLE_SSE if (use_xsave) { xhdr = (struct xstate_hdr *)(get_pcb_user_save_td(&thread0) + 1); xhdr->xstate_bv = xsave_mask; } #endif PCPU_SET(curpcb, thread0.td_pcb); /* make an initial tss so cpu can get interrupt stack on syscall! */ /* Note: -16 is so we can grow the trapframe if we came from vm86 */ PCPU_SET(common_tss.tss_esp0, (vm_offset_t)thread0.td_pcb - 16); PCPU_SET(common_tss.tss_ss0, GSEL(GDATA_SEL, SEL_KPL)); gsel_tss = GSEL(GPROC0_SEL, SEL_KPL); PCPU_SET(tss_gdt, &gdt[GPROC0_SEL].sd); PCPU_SET(common_tssd, *PCPU_GET(tss_gdt)); PCPU_SET(common_tss.tss_ioopt, (sizeof (struct i386tss)) << 16); ltr(gsel_tss); /* make a call gate to reenter kernel with */ gdp = &ldt[LSYS5CALLS_SEL].gd; x = (int) &IDTVEC(lcall_syscall); gdp->gd_looffset = x; gdp->gd_selector = GSEL(GCODE_SEL,SEL_KPL); gdp->gd_stkcpy = 1; gdp->gd_type = SDT_SYS386CGT; gdp->gd_dpl = SEL_UPL; gdp->gd_p = 1; gdp->gd_hioffset = x >> 16; /* XXX does this work? */ /* XXX yes! */ ldt[LBSDICALLS_SEL] = ldt[LSYS5CALLS_SEL]; ldt[LSOL26CALLS_SEL] = ldt[LSYS5CALLS_SEL]; /* transfer to user mode */ _ucodesel = GSEL(GUCODE_SEL, SEL_UPL); _udatasel = GSEL(GUDATA_SEL, SEL_UPL); /* setup proc 0's pcb */ thread0.td_pcb->pcb_flags = 0; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) thread0.td_pcb->pcb_cr3 = (int)IdlePDPT; #else thread0.td_pcb->pcb_cr3 = (int)IdlePTD; #endif thread0.td_pcb->pcb_ext = 0; thread0.td_frame = &proc0_tf; cpu_probe_amdc1e(); #ifdef FDT x86_init_fdt(); #endif /* Location of kernel stack for locore */ return ((register_t)thread0.td_pcb); } #endif void cpu_pcpu_init(struct pcpu *pcpu, int cpuid, size_t size) { pcpu->pc_acpi_id = 0xffffffff; } #ifndef PC98 static int smap_sysctl_handler(SYSCTL_HANDLER_ARGS) { struct bios_smap *smapbase; struct bios_smap_xattr smap; caddr_t kmdp; uint32_t *smapattr; int count, error, i; /* Retrieve the system memory map from the loader. */ kmdp = preload_search_by_type("elf kernel"); if (kmdp == NULL) kmdp = preload_search_by_type("elf32 kernel"); if (kmdp == NULL) return (0); smapbase = (struct bios_smap *)preload_search_info(kmdp, MODINFO_METADATA | MODINFOMD_SMAP); if (smapbase == NULL) return (0); smapattr = (uint32_t *)preload_search_info(kmdp, MODINFO_METADATA | MODINFOMD_SMAP_XATTR); count = *((u_int32_t *)smapbase - 1) / sizeof(*smapbase); error = 0; for (i = 0; i < count; i++) { smap.base = smapbase[i].base; smap.length = smapbase[i].length; smap.type = smapbase[i].type; if (smapattr != NULL) smap.xattr = smapattr[i]; else smap.xattr = 0; error = SYSCTL_OUT(req, &smap, sizeof(smap)); } return (error); } SYSCTL_PROC(_machdep, OID_AUTO, smap, CTLTYPE_OPAQUE|CTLFLAG_RD, NULL, 0, smap_sysctl_handler, "S,bios_smap_xattr", "Raw BIOS SMAP data"); #endif /* !PC98 */ void spinlock_enter(void) { struct thread *td; register_t flags; td = curthread; if (td->td_md.md_spinlock_count == 0) { flags = intr_disable(); td->td_md.md_spinlock_count = 1; td->td_md.md_saved_flags = flags; } else td->td_md.md_spinlock_count++; critical_enter(); } void spinlock_exit(void) { struct thread *td; register_t flags; td = curthread; critical_exit(); flags = td->td_md.md_saved_flags; td->td_md.md_spinlock_count--; if (td->td_md.md_spinlock_count == 0) intr_restore(flags); } #if defined(I586_CPU) && !defined(NO_F00F_HACK) static void f00f_hack(void *unused); SYSINIT(f00f_hack, SI_SUB_INTRINSIC, SI_ORDER_FIRST, f00f_hack, NULL); static void f00f_hack(void *unused) { struct gate_descriptor *new_idt; vm_offset_t tmp; if (!has_f00f_bug) return; GIANT_REQUIRED; printf("Intel Pentium detected, installing workaround for F00F bug\n"); tmp = kmem_malloc(kernel_arena, PAGE_SIZE * 2, M_WAITOK | M_ZERO); if (tmp == 0) panic("kmem_malloc returned 0"); /* Put the problematic entry (#6) at the end of the lower page. */ new_idt = (struct gate_descriptor*) (tmp + PAGE_SIZE - 7 * sizeof(struct gate_descriptor)); bcopy(idt, new_idt, sizeof(idt0)); r_idt.rd_base = (u_int)new_idt; lidt(&r_idt); idt = new_idt; pmap_protect(kernel_pmap, tmp, tmp + PAGE_SIZE, VM_PROT_READ); } #endif /* defined(I586_CPU) && !NO_F00F_HACK */ /* * Construct a PCB from a trapframe. This is called from kdb_trap() where * we want to start a backtrace from the function that caused us to enter * the debugger. We have the context in the trapframe, but base the trace * on the PCB. The PCB doesn't have to be perfect, as long as it contains * enough for a backtrace. */ void makectx(struct trapframe *tf, struct pcb *pcb) { pcb->pcb_edi = tf->tf_edi; pcb->pcb_esi = tf->tf_esi; pcb->pcb_ebp = tf->tf_ebp; pcb->pcb_ebx = tf->tf_ebx; pcb->pcb_eip = tf->tf_eip; pcb->pcb_esp = (ISPL(tf->tf_cs)) ? tf->tf_esp : (int)(tf + 1) - 8; } int ptrace_set_pc(struct thread *td, u_long addr) { td->td_frame->tf_eip = addr; return (0); } int ptrace_single_step(struct thread *td) { td->td_frame->tf_eflags |= PSL_T; return (0); } int ptrace_clear_single_step(struct thread *td) { td->td_frame->tf_eflags &= ~PSL_T; return (0); } int fill_regs(struct thread *td, struct reg *regs) { struct pcb *pcb; struct trapframe *tp; tp = td->td_frame; pcb = td->td_pcb; regs->r_gs = pcb->pcb_gs; return (fill_frame_regs(tp, regs)); } int fill_frame_regs(struct trapframe *tp, struct reg *regs) { regs->r_fs = tp->tf_fs; regs->r_es = tp->tf_es; regs->r_ds = tp->tf_ds; regs->r_edi = tp->tf_edi; regs->r_esi = tp->tf_esi; regs->r_ebp = tp->tf_ebp; regs->r_ebx = tp->tf_ebx; regs->r_edx = tp->tf_edx; regs->r_ecx = tp->tf_ecx; regs->r_eax = tp->tf_eax; regs->r_eip = tp->tf_eip; regs->r_cs = tp->tf_cs; regs->r_eflags = tp->tf_eflags; regs->r_esp = tp->tf_esp; regs->r_ss = tp->tf_ss; return (0); } int set_regs(struct thread *td, struct reg *regs) { struct pcb *pcb; struct trapframe *tp; tp = td->td_frame; if (!EFL_SECURE(regs->r_eflags, tp->tf_eflags) || !CS_SECURE(regs->r_cs)) return (EINVAL); pcb = td->td_pcb; tp->tf_fs = regs->r_fs; tp->tf_es = regs->r_es; tp->tf_ds = regs->r_ds; tp->tf_edi = regs->r_edi; tp->tf_esi = regs->r_esi; tp->tf_ebp = regs->r_ebp; tp->tf_ebx = regs->r_ebx; tp->tf_edx = regs->r_edx; tp->tf_ecx = regs->r_ecx; tp->tf_eax = regs->r_eax; tp->tf_eip = regs->r_eip; tp->tf_cs = regs->r_cs; tp->tf_eflags = regs->r_eflags; tp->tf_esp = regs->r_esp; tp->tf_ss = regs->r_ss; pcb->pcb_gs = regs->r_gs; return (0); } #ifdef CPU_ENABLE_SSE static void fill_fpregs_xmm(sv_xmm, sv_87) struct savexmm *sv_xmm; struct save87 *sv_87; { register struct env87 *penv_87 = &sv_87->sv_env; register struct envxmm *penv_xmm = &sv_xmm->sv_env; int i; bzero(sv_87, sizeof(*sv_87)); /* FPU control/status */ penv_87->en_cw = penv_xmm->en_cw; penv_87->en_sw = penv_xmm->en_sw; penv_87->en_tw = penv_xmm->en_tw; penv_87->en_fip = penv_xmm->en_fip; penv_87->en_fcs = penv_xmm->en_fcs; penv_87->en_opcode = penv_xmm->en_opcode; penv_87->en_foo = penv_xmm->en_foo; penv_87->en_fos = penv_xmm->en_fos; /* FPU registers */ for (i = 0; i < 8; ++i) sv_87->sv_ac[i] = sv_xmm->sv_fp[i].fp_acc; } static void set_fpregs_xmm(sv_87, sv_xmm) struct save87 *sv_87; struct savexmm *sv_xmm; { register struct env87 *penv_87 = &sv_87->sv_env; register struct envxmm *penv_xmm = &sv_xmm->sv_env; int i; /* FPU control/status */ penv_xmm->en_cw = penv_87->en_cw; penv_xmm->en_sw = penv_87->en_sw; penv_xmm->en_tw = penv_87->en_tw; penv_xmm->en_fip = penv_87->en_fip; penv_xmm->en_fcs = penv_87->en_fcs; penv_xmm->en_opcode = penv_87->en_opcode; penv_xmm->en_foo = penv_87->en_foo; penv_xmm->en_fos = penv_87->en_fos; /* FPU registers */ for (i = 0; i < 8; ++i) sv_xmm->sv_fp[i].fp_acc = sv_87->sv_ac[i]; } #endif /* CPU_ENABLE_SSE */ int fill_fpregs(struct thread *td, struct fpreg *fpregs) { KASSERT(td == curthread || TD_IS_SUSPENDED(td) || P_SHOULDSTOP(td->td_proc), ("not suspended thread %p", td)); #ifdef DEV_NPX npxgetregs(td); #else bzero(fpregs, sizeof(*fpregs)); #endif #ifdef CPU_ENABLE_SSE if (cpu_fxsr) fill_fpregs_xmm(&get_pcb_user_save_td(td)->sv_xmm, (struct save87 *)fpregs); else #endif /* CPU_ENABLE_SSE */ bcopy(&get_pcb_user_save_td(td)->sv_87, fpregs, sizeof(*fpregs)); return (0); } int set_fpregs(struct thread *td, struct fpreg *fpregs) { #ifdef CPU_ENABLE_SSE if (cpu_fxsr) set_fpregs_xmm((struct save87 *)fpregs, &get_pcb_user_save_td(td)->sv_xmm); else #endif /* CPU_ENABLE_SSE */ bcopy(fpregs, &get_pcb_user_save_td(td)->sv_87, sizeof(*fpregs)); #ifdef DEV_NPX npxuserinited(td); #endif return (0); } /* * Get machine context. */ int get_mcontext(struct thread *td, mcontext_t *mcp, int flags) { struct trapframe *tp; struct segment_descriptor *sdp; tp = td->td_frame; PROC_LOCK(curthread->td_proc); mcp->mc_onstack = sigonstack(tp->tf_esp); PROC_UNLOCK(curthread->td_proc); mcp->mc_gs = td->td_pcb->pcb_gs; mcp->mc_fs = tp->tf_fs; mcp->mc_es = tp->tf_es; mcp->mc_ds = tp->tf_ds; mcp->mc_edi = tp->tf_edi; mcp->mc_esi = tp->tf_esi; mcp->mc_ebp = tp->tf_ebp; mcp->mc_isp = tp->tf_isp; mcp->mc_eflags = tp->tf_eflags; if (flags & GET_MC_CLEAR_RET) { mcp->mc_eax = 0; mcp->mc_edx = 0; mcp->mc_eflags &= ~PSL_C; } else { mcp->mc_eax = tp->tf_eax; mcp->mc_edx = tp->tf_edx; } mcp->mc_ebx = tp->tf_ebx; mcp->mc_ecx = tp->tf_ecx; mcp->mc_eip = tp->tf_eip; mcp->mc_cs = tp->tf_cs; mcp->mc_esp = tp->tf_esp; mcp->mc_ss = tp->tf_ss; mcp->mc_len = sizeof(*mcp); get_fpcontext(td, mcp, NULL, 0); sdp = &td->td_pcb->pcb_fsd; mcp->mc_fsbase = sdp->sd_hibase << 24 | sdp->sd_lobase; sdp = &td->td_pcb->pcb_gsd; mcp->mc_gsbase = sdp->sd_hibase << 24 | sdp->sd_lobase; mcp->mc_flags = 0; mcp->mc_xfpustate = 0; mcp->mc_xfpustate_len = 0; bzero(mcp->mc_spare2, sizeof(mcp->mc_spare2)); return (0); } /* * Set machine context. * * However, we don't set any but the user modifiable flags, and we won't * touch the cs selector. */ int set_mcontext(struct thread *td, mcontext_t *mcp) { struct trapframe *tp; char *xfpustate; int eflags, ret; tp = td->td_frame; if (mcp->mc_len != sizeof(*mcp) || (mcp->mc_flags & ~_MC_FLAG_MASK) != 0) return (EINVAL); eflags = (mcp->mc_eflags & PSL_USERCHANGE) | (tp->tf_eflags & ~PSL_USERCHANGE); if (mcp->mc_flags & _MC_HASFPXSTATE) { if (mcp->mc_xfpustate_len > cpu_max_ext_state_size - sizeof(union savefpu)) return (EINVAL); xfpustate = __builtin_alloca(mcp->mc_xfpustate_len); ret = copyin((void *)mcp->mc_xfpustate, xfpustate, mcp->mc_xfpustate_len); if (ret != 0) return (ret); } else xfpustate = NULL; ret = set_fpcontext(td, mcp, xfpustate, mcp->mc_xfpustate_len); if (ret != 0) return (ret); tp->tf_fs = mcp->mc_fs; tp->tf_es = mcp->mc_es; tp->tf_ds = mcp->mc_ds; tp->tf_edi = mcp->mc_edi; tp->tf_esi = mcp->mc_esi; tp->tf_ebp = mcp->mc_ebp; tp->tf_ebx = mcp->mc_ebx; tp->tf_edx = mcp->mc_edx; tp->tf_ecx = mcp->mc_ecx; tp->tf_eax = mcp->mc_eax; tp->tf_eip = mcp->mc_eip; tp->tf_eflags = eflags; tp->tf_esp = mcp->mc_esp; tp->tf_ss = mcp->mc_ss; td->td_pcb->pcb_gs = mcp->mc_gs; return (0); } static void get_fpcontext(struct thread *td, mcontext_t *mcp, char *xfpusave, size_t xfpusave_len) { #ifdef CPU_ENABLE_SSE size_t max_len, len; #endif #ifndef DEV_NPX mcp->mc_fpformat = _MC_FPFMT_NODEV; mcp->mc_ownedfp = _MC_FPOWNED_NONE; bzero(mcp->mc_fpstate, sizeof(mcp->mc_fpstate)); #else mcp->mc_ownedfp = npxgetregs(td); bcopy(get_pcb_user_save_td(td), &mcp->mc_fpstate[0], sizeof(mcp->mc_fpstate)); mcp->mc_fpformat = npxformat(); #ifdef CPU_ENABLE_SSE if (!use_xsave || xfpusave_len == 0) return; max_len = cpu_max_ext_state_size - sizeof(union savefpu); len = xfpusave_len; if (len > max_len) { len = max_len; bzero(xfpusave + max_len, len - max_len); } mcp->mc_flags |= _MC_HASFPXSTATE; mcp->mc_xfpustate_len = len; bcopy(get_pcb_user_save_td(td) + 1, xfpusave, len); #endif #endif } static int set_fpcontext(struct thread *td, mcontext_t *mcp, char *xfpustate, size_t xfpustate_len) { union savefpu *fpstate; int error; if (mcp->mc_fpformat == _MC_FPFMT_NODEV) return (0); else if (mcp->mc_fpformat != _MC_FPFMT_387 && mcp->mc_fpformat != _MC_FPFMT_XMM) return (EINVAL); else if (mcp->mc_ownedfp == _MC_FPOWNED_NONE) { /* We don't care what state is left in the FPU or PCB. */ fpstate_drop(td); error = 0; } else if (mcp->mc_ownedfp == _MC_FPOWNED_FPU || mcp->mc_ownedfp == _MC_FPOWNED_PCB) { #ifdef DEV_NPX fpstate = (union savefpu *)&mcp->mc_fpstate; #ifdef CPU_ENABLE_SSE if (cpu_fxsr) fpstate->sv_xmm.sv_env.en_mxcsr &= cpu_mxcsr_mask; #endif error = npxsetregs(td, fpstate, xfpustate, xfpustate_len); #else error = EINVAL; #endif } else return (EINVAL); return (error); } static void fpstate_drop(struct thread *td) { KASSERT(PCB_USER_FPU(td->td_pcb), ("fpstate_drop: kernel-owned fpu")); critical_enter(); #ifdef DEV_NPX if (PCPU_GET(fpcurthread) == td) npxdrop(); #endif /* * XXX force a full drop of the npx. The above only drops it if we * owned it. npxgetregs() has the same bug in the !cpu_fxsr case. * * XXX I don't much like npxgetregs()'s semantics of doing a full * drop. Dropping only to the pcb matches fnsave's behaviour. * We only need to drop to !PCB_INITDONE in sendsig(). But * sendsig() is the only caller of npxgetregs()... perhaps we just * have too many layers. */ curthread->td_pcb->pcb_flags &= ~(PCB_NPXINITDONE | PCB_NPXUSERINITDONE); critical_exit(); } int fill_dbregs(struct thread *td, struct dbreg *dbregs) { struct pcb *pcb; if (td == NULL) { dbregs->dr[0] = rdr0(); dbregs->dr[1] = rdr1(); dbregs->dr[2] = rdr2(); dbregs->dr[3] = rdr3(); dbregs->dr[4] = rdr4(); dbregs->dr[5] = rdr5(); dbregs->dr[6] = rdr6(); dbregs->dr[7] = rdr7(); } else { pcb = td->td_pcb; dbregs->dr[0] = pcb->pcb_dr0; dbregs->dr[1] = pcb->pcb_dr1; dbregs->dr[2] = pcb->pcb_dr2; dbregs->dr[3] = pcb->pcb_dr3; dbregs->dr[4] = 0; dbregs->dr[5] = 0; dbregs->dr[6] = pcb->pcb_dr6; dbregs->dr[7] = pcb->pcb_dr7; } return (0); } int set_dbregs(struct thread *td, struct dbreg *dbregs) { struct pcb *pcb; int i; if (td == NULL) { load_dr0(dbregs->dr[0]); load_dr1(dbregs->dr[1]); load_dr2(dbregs->dr[2]); load_dr3(dbregs->dr[3]); load_dr4(dbregs->dr[4]); load_dr5(dbregs->dr[5]); load_dr6(dbregs->dr[6]); load_dr7(dbregs->dr[7]); } else { /* * Don't let an illegal value for dr7 get set. Specifically, * check for undefined settings. Setting these bit patterns * result in undefined behaviour and can lead to an unexpected * TRCTRAP. */ for (i = 0; i < 4; i++) { if (DBREG_DR7_ACCESS(dbregs->dr[7], i) == 0x02) return (EINVAL); if (DBREG_DR7_LEN(dbregs->dr[7], i) == 0x02) return (EINVAL); } pcb = td->td_pcb; /* * Don't let a process set a breakpoint that is not within the * process's address space. If a process could do this, it * could halt the system by setting a breakpoint in the kernel * (if ddb was enabled). Thus, we need to check to make sure * that no breakpoints are being enabled for addresses outside * process's address space. * * XXX - what about when the watched area of the user's * address space is written into from within the kernel * ... wouldn't that still cause a breakpoint to be generated * from within kernel mode? */ if (DBREG_DR7_ENABLED(dbregs->dr[7], 0)) { /* dr0 is enabled */ if (dbregs->dr[0] >= VM_MAXUSER_ADDRESS) return (EINVAL); } if (DBREG_DR7_ENABLED(dbregs->dr[7], 1)) { /* dr1 is enabled */ if (dbregs->dr[1] >= VM_MAXUSER_ADDRESS) return (EINVAL); } if (DBREG_DR7_ENABLED(dbregs->dr[7], 2)) { /* dr2 is enabled */ if (dbregs->dr[2] >= VM_MAXUSER_ADDRESS) return (EINVAL); } if (DBREG_DR7_ENABLED(dbregs->dr[7], 3)) { /* dr3 is enabled */ if (dbregs->dr[3] >= VM_MAXUSER_ADDRESS) return (EINVAL); } pcb->pcb_dr0 = dbregs->dr[0]; pcb->pcb_dr1 = dbregs->dr[1]; pcb->pcb_dr2 = dbregs->dr[2]; pcb->pcb_dr3 = dbregs->dr[3]; pcb->pcb_dr6 = dbregs->dr[6]; pcb->pcb_dr7 = dbregs->dr[7]; pcb->pcb_flags |= PCB_DBREGS; } return (0); } /* * Return > 0 if a hardware breakpoint has been hit, and the * breakpoint was in user space. Return 0, otherwise. */ int user_dbreg_trap(void) { u_int32_t dr7, dr6; /* debug registers dr6 and dr7 */ u_int32_t bp; /* breakpoint bits extracted from dr6 */ int nbp; /* number of breakpoints that triggered */ caddr_t addr[4]; /* breakpoint addresses */ int i; dr7 = rdr7(); if ((dr7 & 0x000000ff) == 0) { /* * all GE and LE bits in the dr7 register are zero, * thus the trap couldn't have been caused by the * hardware debug registers */ return 0; } nbp = 0; dr6 = rdr6(); bp = dr6 & 0x0000000f; if (!bp) { /* * None of the breakpoint bits are set meaning this * trap was not caused by any of the debug registers */ return 0; } /* * at least one of the breakpoints were hit, check to see * which ones and if any of them are user space addresses */ if (bp & 0x01) { addr[nbp++] = (caddr_t)rdr0(); } if (bp & 0x02) { addr[nbp++] = (caddr_t)rdr1(); } if (bp & 0x04) { addr[nbp++] = (caddr_t)rdr2(); } if (bp & 0x08) { addr[nbp++] = (caddr_t)rdr3(); } for (i = 0; i < nbp; i++) { if (addr[i] < (caddr_t)VM_MAXUSER_ADDRESS) { /* * addr[i] is in user space */ return nbp; } } /* * None of the breakpoints are in user space. */ return 0; } #ifdef KDB /* * Provide inb() and outb() as functions. They are normally only available as * inline functions, thus cannot be called from the debugger. */ /* silence compiler warnings */ u_char inb_(u_short); void outb_(u_short, u_char); u_char inb_(u_short port) { return inb(port); } void outb_(u_short port, u_char data) { outb(port, data); } #endif /* KDB */ Index: user/ngie/more-tests/sys/i386/i386/minidump_machdep.c =================================================================== --- user/ngie/more-tests/sys/i386/i386/minidump_machdep.c (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/minidump_machdep.c (revision 281504) @@ -1,416 +1,416 @@ /*- * Copyright (c) 2006 Peter Wemm * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include "opt_watchdog.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include CTASSERT(sizeof(struct kerneldumpheader) == 512); /* * Don't touch the first SIZEOF_METADATA bytes on the dump device. This * is to protect us from metadata and to protect metadata from us. */ #define SIZEOF_METADATA (64*1024) #define MD_ALIGN(x) (((off_t)(x) + PAGE_MASK) & ~PAGE_MASK) #define DEV_ALIGN(x) (((off_t)(x) + (DEV_BSIZE-1)) & ~(DEV_BSIZE-1)) uint32_t *vm_page_dump; int vm_page_dump_size; static struct kerneldumpheader kdh; static off_t dumplo; /* Handle chunked writes. */ static size_t fragsz; static void *dump_va; static uint64_t counter, progress; CTASSERT(sizeof(*vm_page_dump) == 4); #ifndef XEN #define xpmap_mtop(x) (x) #define xpmap_ptom(x) (x) #endif static int is_dumpable(vm_paddr_t pa) { int i; for (i = 0; dump_avail[i] != 0 || dump_avail[i + 1] != 0; i += 2) { if (pa >= dump_avail[i] && pa < dump_avail[i + 1]) return (1); } return (0); } #define PG2MB(pgs) (((pgs) + (1 << 8) - 1) >> 8) static int blk_flush(struct dumperinfo *di) { int error; if (fragsz == 0) return (0); error = dump_write(di, dump_va, 0, dumplo, fragsz); dumplo += fragsz; fragsz = 0; return (error); } static int blk_write(struct dumperinfo *di, char *ptr, vm_paddr_t pa, size_t sz) { size_t len; int error, i, c; u_int maxdumpsz; maxdumpsz = min(di->maxiosize, MAXDUMPPGS * PAGE_SIZE); if (maxdumpsz == 0) /* seatbelt */ maxdumpsz = PAGE_SIZE; error = 0; if ((sz % PAGE_SIZE) != 0) { printf("size not page aligned\n"); return (EINVAL); } if (ptr != NULL && pa != 0) { printf("cant have both va and pa!\n"); return (EINVAL); } if (pa != 0 && (((uintptr_t)ptr) % PAGE_SIZE) != 0) { printf("address not page aligned\n"); return (EINVAL); } if (ptr != NULL) { /* If we're doing a virtual dump, flush any pre-existing pa pages */ error = blk_flush(di); if (error) return (error); } while (sz) { len = maxdumpsz - fragsz; if (len > sz) len = sz; counter += len; progress -= len; if (counter >> 24) { printf(" %lld", PG2MB(progress >> PAGE_SHIFT)); counter &= (1<<24) - 1; } wdog_kern_pat(WD_LASTVAL); if (ptr) { error = dump_write(di, ptr, 0, dumplo, len); if (error) return (error); dumplo += len; ptr += len; sz -= len; } else { for (i = 0; i < len; i += PAGE_SIZE) dump_va = pmap_kenter_temporary(pa + i, (i + fragsz) >> PAGE_SHIFT); fragsz += len; pa += len; sz -= len; if (fragsz == maxdumpsz) { error = blk_flush(di); if (error) return (error); } } /* Check for user abort. */ c = cncheckc(); if (c == 0x03) return (ECANCELED); if (c != -1) printf(" (CTRL-C to abort) "); } return (0); } /* A fake page table page, to avoid having to handle both 4K and 2M pages */ static pt_entry_t fakept[NPTEPG]; int minidumpsys(struct dumperinfo *di) { uint64_t dumpsize; uint32_t ptesize; vm_offset_t va; int error; uint32_t bits; uint64_t pa; pd_entry_t *pd; pt_entry_t *pt; int i, j, k, bit; struct minidumphdr mdhdr; counter = 0; /* Walk page table pages, set bits in vm_page_dump */ ptesize = 0; for (va = KERNBASE; va < kernel_vm_end; va += NBPDR) { /* * We always write a page, even if it is zero. Each * page written corresponds to 2MB of space */ ptesize += PAGE_SIZE; pd = (pd_entry_t *)((uintptr_t)IdlePTD + KERNBASE); /* always mapped! */ j = va >> PDRSHIFT; if ((pd[j] & (PG_PS | PG_V)) == (PG_PS | PG_V)) { /* This is an entire 2M page. */ pa = xpmap_mtop(pd[j] & PG_PS_FRAME); for (k = 0; k < NPTEPG; k++) { if (is_dumpable(pa)) dump_add_page(pa); pa += PAGE_SIZE; } continue; } if ((pd[j] & PG_V) == PG_V) { /* set bit for each valid page in this 2MB block */ pt = pmap_kenter_temporary(xpmap_mtop(pd[j] & PG_FRAME), 0); for (k = 0; k < NPTEPG; k++) { if ((pt[k] & PG_V) == PG_V) { pa = xpmap_mtop(pt[k] & PG_FRAME); if (is_dumpable(pa)) dump_add_page(pa); } } } else { /* nothing, we're going to dump a null page */ } } /* Calculate dump size. */ dumpsize = ptesize; dumpsize += round_page(msgbufp->msg_size); dumpsize += round_page(vm_page_dump_size); for (i = 0; i < vm_page_dump_size / sizeof(*vm_page_dump); i++) { bits = vm_page_dump[i]; while (bits) { bit = bsfl(bits); pa = (((uint64_t)i * sizeof(*vm_page_dump) * NBBY) + bit) * PAGE_SIZE; /* Clear out undumpable pages now if needed */ if (is_dumpable(pa)) { dumpsize += PAGE_SIZE; } else { dump_drop_page(pa); } bits &= ~(1ul << bit); } } dumpsize += PAGE_SIZE; /* Determine dump offset on device. */ if (di->mediasize < SIZEOF_METADATA + dumpsize + sizeof(kdh) * 2) { error = ENOSPC; goto fail; } dumplo = di->mediaoffset + di->mediasize - dumpsize; dumplo -= sizeof(kdh) * 2; progress = dumpsize; /* Initialize mdhdr */ bzero(&mdhdr, sizeof(mdhdr)); strcpy(mdhdr.magic, MINIDUMP_MAGIC); mdhdr.version = MINIDUMP_VERSION; mdhdr.msgbufsize = msgbufp->msg_size; mdhdr.bitmapsize = vm_page_dump_size; mdhdr.ptesize = ptesize; mdhdr.kernbase = KERNBASE; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) mdhdr.paemode = 1; #endif mkdumpheader(&kdh, KERNELDUMPMAGIC, KERNELDUMP_I386_VERSION, dumpsize, di->blocksize); printf("Physical memory: %ju MB\n", ptoa((uintmax_t)physmem) / 1048576); printf("Dumping %llu MB:", (long long)dumpsize >> 20); /* Dump leader */ error = dump_write(di, &kdh, 0, dumplo, sizeof(kdh)); if (error) goto fail; dumplo += sizeof(kdh); /* Dump my header */ bzero(&fakept, sizeof(fakept)); bcopy(&mdhdr, &fakept, sizeof(mdhdr)); error = blk_write(di, (char *)&fakept, 0, PAGE_SIZE); if (error) goto fail; /* Dump msgbuf up front */ error = blk_write(di, (char *)msgbufp->msg_ptr, 0, round_page(msgbufp->msg_size)); if (error) goto fail; /* Dump bitmap */ error = blk_write(di, (char *)vm_page_dump, 0, round_page(vm_page_dump_size)); if (error) goto fail; /* Dump kernel page table pages */ for (va = KERNBASE; va < kernel_vm_end; va += NBPDR) { /* We always write a page, even if it is zero */ pd = (pd_entry_t *)((uintptr_t)IdlePTD + KERNBASE); /* always mapped! */ j = va >> PDRSHIFT; if ((pd[j] & (PG_PS | PG_V)) == (PG_PS | PG_V)) { /* This is a single 2M block. Generate a fake PTP */ pa = pd[j] & PG_PS_FRAME; for (k = 0; k < NPTEPG; k++) { fakept[k] = (pa + (k * PAGE_SIZE)) | PG_V | PG_RW | PG_A | PG_M; } error = blk_write(di, (char *)&fakept, 0, PAGE_SIZE); if (error) goto fail; /* flush, in case we reuse fakept in the same block */ error = blk_flush(di); if (error) goto fail; continue; } if ((pd[j] & PG_V) == PG_V) { pa = xpmap_mtop(pd[j] & PG_FRAME); #ifndef XEN error = blk_write(di, 0, pa, PAGE_SIZE); #else pt = pmap_kenter_temporary(pa, 0); memcpy(fakept, pt, PAGE_SIZE); for (i = 0; i < NPTEPG; i++) fakept[i] = xpmap_mtop(fakept[i]); error = blk_write(di, (char *)&fakept, 0, PAGE_SIZE); if (error) goto fail; /* flush, in case we reuse fakept in the same block */ error = blk_flush(di); if (error) goto fail; bzero(fakept, sizeof(fakept)); #endif if (error) goto fail; } else { bzero(fakept, sizeof(fakept)); error = blk_write(di, (char *)&fakept, 0, PAGE_SIZE); if (error) goto fail; /* flush, in case we reuse fakept in the same block */ error = blk_flush(di); if (error) goto fail; } } /* Dump memory chunks */ /* XXX cluster it up and use blk_dump() */ for (i = 0; i < vm_page_dump_size / sizeof(*vm_page_dump); i++) { bits = vm_page_dump[i]; while (bits) { bit = bsfl(bits); pa = (((uint64_t)i * sizeof(*vm_page_dump) * NBBY) + bit) * PAGE_SIZE; error = blk_write(di, 0, pa, PAGE_SIZE); if (error) goto fail; bits &= ~(1ul << bit); } } error = blk_flush(di); if (error) goto fail; /* Dump trailer */ error = dump_write(di, &kdh, 0, dumplo, sizeof(kdh)); if (error) goto fail; dumplo += sizeof(kdh); /* Signal completion, signoff and exit stage left. */ dump_write(di, NULL, 0, 0, 0); printf("\nDump complete\n"); return (0); fail: if (error < 0) error = -error; if (error == ECANCELED) printf("\nDump aborted\n"); else if (error == ENOSPC) printf("\nDump failed. Partition too small.\n"); else printf("\n** DUMP FAILED (ERROR %d) **\n", error); return (error); } void dump_add_page(vm_paddr_t pa) { int idx, bit; pa >>= PAGE_SHIFT; idx = pa >> 5; /* 2^5 = 32 */ bit = pa & 31; atomic_set_int(&vm_page_dump[idx], 1ul << bit); } void dump_drop_page(vm_paddr_t pa) { int idx, bit; pa >>= PAGE_SHIFT; idx = pa >> 5; /* 2^5 = 32 */ bit = pa & 31; atomic_clear_int(&vm_page_dump[idx], 1ul << bit); } Index: user/ngie/more-tests/sys/i386/i386/mpboot.s =================================================================== --- user/ngie/more-tests/sys/i386/i386/mpboot.s (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/mpboot.s (revision 281504) @@ -1,279 +1,279 @@ /*- * Copyright (c) 1995 Jack F. Vogel * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * mpboot.s: FreeBSD machine support for the Intel MP Spec * multiprocessor systems. * * $FreeBSD$ */ #include "opt_pmap.h" #include /* miscellaneous asm macros */ #include #include #include "assym.s" #define R(x) ((x)-KERNBASE) /* * this code MUST be enabled here and in mp_machdep.c * it follows the very early stages of AP boot by placing values in CMOS ram. * it NORMALLY will never be needed and thus the primitive method for enabling. * #define CHECK_POINTS */ #if defined(CHECK_POINTS) && !defined(PC98) #define CMOS_REG (0x70) #define CMOS_DATA (0x71) #define CHECKPOINT(A,D) \ movb $(A),%al ; \ outb %al,$CMOS_REG ; \ movb $(D),%al ; \ outb %al,$CMOS_DATA #else #define CHECKPOINT(A,D) #endif /* CHECK_POINTS */ /* * the APs enter here from their trampoline code (bootMP, below) */ .p2align 4 NON_GPROF_ENTRY(MPentry) CHECKPOINT(0x36, 3) /* * Enable features on this processor. We don't support SMP on * CPUs older than a Pentium, so we know that we can use the cpuid * instruction. */ movl $1,%eax cpuid /* Retrieve features */ movl %cr4,%eax #ifndef DISABLE_PSE testl $CPUID_PSE,%edx jz 1f orl $CR4_PSE,%eax /* Enable PSE */ 1: #endif #ifndef DISABLE_PG_G testl $CPUID_PGE,%edx jz 1f orl $CR4_PGE,%eax /* Enable PGE */ 1: #endif testl $CPUID_VME,%edx jz 1f orl $CR4_VME,%eax /* Enable VME */ 1: movl %eax,%cr4 /* Now enable paging mode */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) movl R(IdlePDPT), %eax movl %eax, %cr3 movl %cr4, %eax orl $CR4_PAE, %eax movl %eax, %cr4 #else movl R(IdlePTD), %eax movl %eax,%cr3 #endif movl %cr0,%eax orl $CR0_PE|CR0_PG,%eax /* enable paging */ movl %eax,%cr0 /* let the games begin! */ movl bootSTK,%esp /* boot stack end loc. */ pushl $mp_begin /* jump to high mem */ ret /* * Wait for the booting CPU to signal startup */ mp_begin: /* now running relocated at KERNBASE */ CHECKPOINT(0x37, 4) call init_secondary /* load i386 tables */ /* * This is the embedded trampoline or bootstrap that is * copied into 'real-mode' low memory, it is where the * secondary processor "wakes up". When it is executed * the processor will eventually jump into the routine * MPentry, which resides in normal kernel text above * 1Meg. -jackv */ .data ALIGN_DATA /* just to be sure */ BOOTMP1: NON_GPROF_ENTRY(bootMP) .code16 cli CHECKPOINT(0x34, 1) /* First guarantee a 'clean slate' */ xorl %eax, %eax movl %eax, %ebx movl %eax, %ecx movl %eax, %edx movl %eax, %esi movl %eax, %edi /* set up data segments */ mov %cs, %ax mov %ax, %ds mov %ax, %es mov %ax, %fs mov %ax, %gs mov %ax, %ss mov $(boot_stk-bootMP), %esp /* Now load the global descriptor table */ lgdt MP_GDTptr-bootMP /* Enable protected mode */ movl %cr0, %eax orl $CR0_PE, %eax movl %eax, %cr0 /* * make intrasegment jump to flush the processor pipeline and * reload CS register */ pushl $0x18 pushl $(protmode-bootMP) lretl .code32 protmode: CHECKPOINT(0x35, 2) /* * we are NOW running for the first time with %eip * having the full physical address, BUT we still * are using a segment descriptor with the origin * not matching the booting kernel. * * SO NOW... for the BIG Jump into kernel's segment * and physical text above 1 Meg. */ mov $0x10, %ebx movw %bx, %ds movw %bx, %es movw %bx, %fs movw %bx, %gs movw %bx, %ss .globl bigJump bigJump: /* this will be modified by mpInstallTramp() */ ljmp $0x08, $0 /* far jmp to MPentry() */ dead: hlt /* We should never get here */ jmp dead /* * MP boot strap Global Descriptor Table */ .p2align 4 .globl MP_GDT .globl bootCodeSeg .globl bootDataSeg MP_GDT: nulldesc: /* offset = 0x0 */ .word 0x0 .word 0x0 .byte 0x0 .byte 0x0 .byte 0x0 .byte 0x0 kernelcode: /* offset = 0x08 */ .word 0xffff /* segment limit 0..15 */ .word 0x0000 /* segment base 0..15 */ .byte 0x0 /* segment base 16..23; set for 0K */ .byte 0x9f /* flags; Type */ .byte 0xcf /* flags; Limit */ .byte 0x0 /* segment base 24..32 */ kerneldata: /* offset = 0x10 */ .word 0xffff /* segment limit 0..15 */ .word 0x0000 /* segment base 0..15 */ .byte 0x0 /* segment base 16..23; set for 0k */ .byte 0x93 /* flags; Type */ .byte 0xcf /* flags; Limit */ .byte 0x0 /* segment base 24..32 */ bootcode: /* offset = 0x18 */ .word 0xffff /* segment limit 0..15 */ bootCodeSeg: /* this will be modified by mpInstallTramp() */ .word 0x0000 /* segment base 0..15 */ .byte 0x00 /* segment base 16...23; set for 0x000xx000 */ .byte 0x9e /* flags; Type */ .byte 0xcf /* flags; Limit */ .byte 0x0 /*segment base 24..32 */ bootdata: /* offset = 0x20 */ .word 0xffff bootDataSeg: /* this will be modified by mpInstallTramp() */ .word 0x0000 /* segment base 0..15 */ .byte 0x00 /* segment base 16...23; set for 0x000xx000 */ .byte 0x92 .byte 0xcf .byte 0x0 /* * GDT pointer for the lgdt call */ .globl mp_gdtbase MP_GDTptr: mp_gdtlimit: .word 0x0028 mp_gdtbase: /* this will be modified by mpInstallTramp() */ .long 0 .space 0x100 /* space for boot_stk - 1st temporary stack */ boot_stk: BOOTMP2: .globl bootMP_size bootMP_size: .long BOOTMP2 - BOOTMP1 Index: user/ngie/more-tests/sys/i386/i386/pmap.c =================================================================== --- user/ngie/more-tests/sys/i386/i386/pmap.c (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/pmap.c (revision 281504) @@ -1,5615 +1,5615 @@ /*- * Copyright (c) 1991 Regents of the University of California. * All rights reserved. * Copyright (c) 1994 John S. Dyson * All rights reserved. * Copyright (c) 1994 David Greenman * All rights reserved. * Copyright (c) 2005-2010 Alan L. Cox * All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department and William Jolitz of UUNET Technologies Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)pmap.c 7.7 (Berkeley) 5/12/91 */ /*- * Copyright (c) 2003 Networks Associates Technology, Inc. * All rights reserved. * * This software was developed for the FreeBSD Project by Jake Burkholder, * Safeport Network Services, and Network Associates Laboratories, the * Security Research Division of Network Associates, Inc. under * DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the DARPA * CHATS research program. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); /* * Manages physical address maps. * * Since the information managed by this module is * also stored by the logical address mapping module, * this module may throw away valid virtual-to-physical * mappings at almost any time. However, invalidations * of virtual-to-physical mappings must be done as * requested. * * In order to cope with hardware architectures which * make virtual-to-physical map invalidates expensive, * this module may delay invalidate or reduced protection * operations until such time as they are actually * necessary. This module is given full information as * to which processors are currently using which maps, * and to when physical maps must be made correct. */ #include "opt_apic.h" #include "opt_cpu.h" #include "opt_pmap.h" #include "opt_smp.h" #include "opt_xbox.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef SMP #include #else #include #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DEV_APIC #include #include #include #endif #include #include #include #include #include #ifdef SMP #include #endif #ifdef XBOX #include #endif #if !defined(CPU_DISABLE_SSE) && defined(I686_CPU) #define CPU_ENABLE_SSE #endif #ifndef PMAP_SHPGPERPROC #define PMAP_SHPGPERPROC 200 #endif #if !defined(DIAGNOSTIC) #ifdef __GNUC_GNU_INLINE__ #define PMAP_INLINE __attribute__((__gnu_inline__)) inline #else #define PMAP_INLINE extern inline #endif #else #define PMAP_INLINE #endif #ifdef PV_STATS #define PV_STAT(x) do { x ; } while (0) #else #define PV_STAT(x) do { } while (0) #endif #define pa_index(pa) ((pa) >> PDRSHIFT) #define pa_to_pvh(pa) (&pv_table[pa_index(pa)]) /* * Get PDEs and PTEs for user/kernel address space */ #define pmap_pde(m, v) (&((m)->pm_pdir[(vm_offset_t)(v) >> PDRSHIFT])) #define pdir_pde(m, v) (m[(vm_offset_t)(v) >> PDRSHIFT]) #define pmap_pde_v(pte) ((*(int *)pte & PG_V) != 0) #define pmap_pte_w(pte) ((*(int *)pte & PG_W) != 0) #define pmap_pte_m(pte) ((*(int *)pte & PG_M) != 0) #define pmap_pte_u(pte) ((*(int *)pte & PG_A) != 0) #define pmap_pte_v(pte) ((*(int *)pte & PG_V) != 0) #define pmap_pte_set_w(pte, v) ((v) ? atomic_set_int((u_int *)(pte), PG_W) : \ atomic_clear_int((u_int *)(pte), PG_W)) #define pmap_pte_set_prot(pte, v) ((*(int *)pte &= ~PG_PROT), (*(int *)pte |= (v))) struct pmap kernel_pmap_store; LIST_HEAD(pmaplist, pmap); static struct pmaplist allpmaps; static struct mtx allpmaps_lock; vm_offset_t virtual_avail; /* VA of first avail page (after kernel bss) */ vm_offset_t virtual_end; /* VA of last avail page (end of kernel AS) */ int pgeflag = 0; /* PG_G or-in */ int pseflag = 0; /* PG_PS or-in */ static int nkpt = NKPT; vm_offset_t kernel_vm_end = KERNBASE + NKPT * NBPDR; extern u_int32_t KERNend; extern u_int32_t KPTphys; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) pt_entry_t pg_nx; static uma_zone_t pdptzone; #endif static SYSCTL_NODE(_vm, OID_AUTO, pmap, CTLFLAG_RD, 0, "VM/pmap parameters"); static int pat_works = 1; SYSCTL_INT(_vm_pmap, OID_AUTO, pat_works, CTLFLAG_RD, &pat_works, 1, "Is page attribute table fully functional?"); static int pg_ps_enabled = 1; SYSCTL_INT(_vm_pmap, OID_AUTO, pg_ps_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &pg_ps_enabled, 0, "Are large page mappings enabled?"); #define PAT_INDEX_SIZE 8 static int pat_index[PAT_INDEX_SIZE]; /* cache mode to PAT index conversion */ static struct rwlock_padalign pvh_global_lock; /* * Data for the pv entry allocation mechanism */ static TAILQ_HEAD(pch, pv_chunk) pv_chunks = TAILQ_HEAD_INITIALIZER(pv_chunks); static int pv_entry_count = 0, pv_entry_max = 0, pv_entry_high_water = 0; static struct md_page *pv_table; static int shpgperproc = PMAP_SHPGPERPROC; struct pv_chunk *pv_chunkbase; /* KVA block for pv_chunks */ int pv_maxchunks; /* How many chunks we have KVA for */ vm_offset_t pv_vafree; /* freelist stored in the PTE */ /* * All those kernel PT submaps that BSD is so fond of */ struct sysmaps { struct mtx lock; pt_entry_t *CMAP1; pt_entry_t *CMAP2; caddr_t CADDR1; caddr_t CADDR2; }; static struct sysmaps sysmaps_pcpu[MAXCPU]; pt_entry_t *CMAP3; static pd_entry_t *KPTD; caddr_t ptvmmap = 0; caddr_t CADDR3; struct msgbuf *msgbufp = 0; /* * Crashdump maps. */ static caddr_t crashdumpmap; static pt_entry_t *PMAP1 = 0, *PMAP2; static pt_entry_t *PADDR1 = 0, *PADDR2; #ifdef SMP static int PMAP1cpu; static int PMAP1changedcpu; SYSCTL_INT(_debug, OID_AUTO, PMAP1changedcpu, CTLFLAG_RD, &PMAP1changedcpu, 0, "Number of times pmap_pte_quick changed CPU with same PMAP1"); #endif static int PMAP1changed; SYSCTL_INT(_debug, OID_AUTO, PMAP1changed, CTLFLAG_RD, &PMAP1changed, 0, "Number of times pmap_pte_quick changed PMAP1"); static int PMAP1unchanged; SYSCTL_INT(_debug, OID_AUTO, PMAP1unchanged, CTLFLAG_RD, &PMAP1unchanged, 0, "Number of times pmap_pte_quick didn't change PMAP1"); static struct mtx PMAP2mutex; static void free_pv_chunk(struct pv_chunk *pc); static void free_pv_entry(pmap_t pmap, pv_entry_t pv); static pv_entry_t get_pv_entry(pmap_t pmap, boolean_t try); static void pmap_pv_demote_pde(pmap_t pmap, vm_offset_t va, vm_paddr_t pa); static boolean_t pmap_pv_insert_pde(pmap_t pmap, vm_offset_t va, vm_paddr_t pa); static void pmap_pv_promote_pde(pmap_t pmap, vm_offset_t va, vm_paddr_t pa); static void pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va); static pv_entry_t pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, vm_offset_t va); static int pmap_pvh_wired_mappings(struct md_page *pvh, int count); static boolean_t pmap_demote_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t va); static boolean_t pmap_enter_pde(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot); static vm_page_t pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, vm_page_t mpte); static void pmap_flush_page(vm_page_t m); static int pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte); static void pmap_fill_ptp(pt_entry_t *firstpte, pt_entry_t newpte); static boolean_t pmap_is_modified_pvh(struct md_page *pvh); static boolean_t pmap_is_referenced_pvh(struct md_page *pvh); static void pmap_kenter_attr(vm_offset_t va, vm_paddr_t pa, int mode); static void pmap_kenter_pde(vm_offset_t va, pd_entry_t newpde); static vm_page_t pmap_lookup_pt_page(pmap_t pmap, vm_offset_t va); static void pmap_pde_attr(pd_entry_t *pde, int cache_bits); static void pmap_promote_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t va); static boolean_t pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t sva, vm_prot_t prot); static void pmap_pte_attr(pt_entry_t *pte, int cache_bits); static void pmap_remove_pde(pmap_t pmap, pd_entry_t *pdq, vm_offset_t sva, struct spglist *free); static int pmap_remove_pte(pmap_t pmap, pt_entry_t *ptq, vm_offset_t sva, struct spglist *free); static void pmap_remove_pt_page(pmap_t pmap, vm_page_t mpte); static void pmap_remove_page(struct pmap *pmap, vm_offset_t va, struct spglist *free); static void pmap_remove_entry(struct pmap *pmap, vm_page_t m, vm_offset_t va); static void pmap_insert_entry(pmap_t pmap, vm_offset_t va, vm_page_t m); static boolean_t pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, vm_page_t m); static void pmap_update_pde(pmap_t pmap, vm_offset_t va, pd_entry_t *pde, pd_entry_t newpde); static void pmap_update_pde_invalidate(vm_offset_t va, pd_entry_t newpde); static vm_page_t pmap_allocpte(pmap_t pmap, vm_offset_t va, u_int flags); static vm_page_t _pmap_allocpte(pmap_t pmap, u_int ptepindex, u_int flags); static void _pmap_unwire_ptp(pmap_t pmap, vm_page_t m, struct spglist *free); static pt_entry_t *pmap_pte_quick(pmap_t pmap, vm_offset_t va); static void pmap_pte_release(pt_entry_t *pte); static int pmap_unuse_pt(pmap_t, vm_offset_t, struct spglist *); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) static void *pmap_pdpt_allocf(uma_zone_t zone, vm_size_t bytes, uint8_t *flags, int wait); #endif static void pmap_set_pg(void); static __inline void pagezero(void *page); CTASSERT(1 << PDESHIFT == sizeof(pd_entry_t)); CTASSERT(1 << PTESHIFT == sizeof(pt_entry_t)); /* * If you get an error here, then you set KVA_PAGES wrong! See the * description of KVA_PAGES in sys/i386/include/pmap.h. It must be * multiple of 4 for a normal kernel, or a multiple of 8 for a PAE. */ CTASSERT(KERNBASE % (1 << 24) == 0); /* * Bootstrap the system enough to run with virtual memory. * * On the i386 this is called after mapping has already been enabled * and just syncs the pmap module with what has already been done. * [We can't call it easily with mapping off since the kernel is not * mapped with PA == VA, hence we would have to relocate every address * from the linked base (virtual) address "KERNBASE" to the actual * (physical) address starting relative to 0] */ void pmap_bootstrap(vm_paddr_t firstaddr) { vm_offset_t va; pt_entry_t *pte, *unused; struct sysmaps *sysmaps; int i; /* * Add a physical memory segment (vm_phys_seg) corresponding to the * preallocated kernel page table pages so that vm_page structures * representing these pages will be created. The vm_page structures * are required for promotion of the corresponding kernel virtual * addresses to superpage mappings. */ vm_phys_add_seg(KPTphys, KPTphys + ptoa(nkpt)); /* * Initialize the first available kernel virtual address. However, * using "firstaddr" may waste a few pages of the kernel virtual * address space, because locore may not have mapped every physical * page that it allocated. Preferably, locore would provide a first * unused virtual address in addition to "firstaddr". */ virtual_avail = (vm_offset_t) KERNBASE + firstaddr; virtual_end = VM_MAX_KERNEL_ADDRESS; /* * Initialize the kernel pmap (which is statically allocated). */ PMAP_LOCK_INIT(kernel_pmap); kernel_pmap->pm_pdir = (pd_entry_t *) (KERNBASE + (u_int)IdlePTD); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) kernel_pmap->pm_pdpt = (pdpt_entry_t *) (KERNBASE + (u_int)IdlePDPT); #endif CPU_FILL(&kernel_pmap->pm_active); /* don't allow deactivation */ TAILQ_INIT(&kernel_pmap->pm_pvchunk); /* * Initialize the global pv list lock. */ rw_init(&pvh_global_lock, "pmap pv global"); LIST_INIT(&allpmaps); /* * Request a spin mutex so that changes to allpmaps cannot be * preempted by smp_rendezvous_cpus(). Otherwise, * pmap_update_pde_kernel() could access allpmaps while it is * being changed. */ mtx_init(&allpmaps_lock, "allpmaps", NULL, MTX_SPIN); mtx_lock_spin(&allpmaps_lock); LIST_INSERT_HEAD(&allpmaps, kernel_pmap, pm_list); mtx_unlock_spin(&allpmaps_lock); /* * Reserve some special page table entries/VA space for temporary * mapping of pages. */ #define SYSMAP(c, p, v, n) \ v = (c)va; va += ((n)*PAGE_SIZE); p = pte; pte += (n); va = virtual_avail; pte = vtopte(va); /* * CMAP1/CMAP2 are used for zeroing and copying pages. * CMAP3 is used for the idle process page zeroing. */ for (i = 0; i < MAXCPU; i++) { sysmaps = &sysmaps_pcpu[i]; mtx_init(&sysmaps->lock, "SYSMAPS", NULL, MTX_DEF); SYSMAP(caddr_t, sysmaps->CMAP1, sysmaps->CADDR1, 1) SYSMAP(caddr_t, sysmaps->CMAP2, sysmaps->CADDR2, 1) } SYSMAP(caddr_t, CMAP3, CADDR3, 1) /* * Crashdump maps. */ SYSMAP(caddr_t, unused, crashdumpmap, MAXDUMPPGS) /* * ptvmmap is used for reading arbitrary physical pages via /dev/mem. */ SYSMAP(caddr_t, unused, ptvmmap, 1) /* * msgbufp is used to map the system message buffer. */ SYSMAP(struct msgbuf *, unused, msgbufp, atop(round_page(msgbufsize))) /* * KPTmap is used by pmap_kextract(). * * KPTmap is first initialized by locore. However, that initial * KPTmap can only support NKPT page table pages. Here, a larger * KPTmap is created that can support KVA_PAGES page table pages. */ SYSMAP(pt_entry_t *, KPTD, KPTmap, KVA_PAGES) for (i = 0; i < NKPT; i++) KPTD[i] = (KPTphys + (i << PAGE_SHIFT)) | pgeflag | PG_RW | PG_V; /* * Adjust the start of the KPTD and KPTmap so that the implementation * of pmap_kextract() and pmap_growkernel() can be made simpler. */ KPTD -= KPTDI; KPTmap -= i386_btop(KPTDI << PDRSHIFT); /* * PADDR1 and PADDR2 are used by pmap_pte_quick() and pmap_pte(), * respectively. */ SYSMAP(pt_entry_t *, PMAP1, PADDR1, 1) SYSMAP(pt_entry_t *, PMAP2, PADDR2, 1) mtx_init(&PMAP2mutex, "PMAP2", NULL, MTX_DEF); virtual_avail = va; /* * Leave in place an identity mapping (virt == phys) for the low 1 MB * physical memory region that is used by the ACPI wakeup code. This * mapping must not have PG_G set. */ #ifdef XBOX /* FIXME: This is gross, but needed for the XBOX. Since we are in such * an early stadium, we cannot yet neatly map video memory ... :-( * Better fixes are very welcome! */ if (!arch_i386_is_xbox) #endif for (i = 1; i < NKPT; i++) PTD[i] = 0; /* Initialize the PAT MSR if present. */ pmap_init_pat(); /* Turn on PG_G on kernel page(s) */ pmap_set_pg(); } /* * Setup the PAT MSR. */ void pmap_init_pat(void) { int pat_table[PAT_INDEX_SIZE]; uint64_t pat_msr; u_long cr0, cr4; int i; /* Set default PAT index table. */ for (i = 0; i < PAT_INDEX_SIZE; i++) pat_table[i] = -1; pat_table[PAT_WRITE_BACK] = 0; pat_table[PAT_WRITE_THROUGH] = 1; pat_table[PAT_UNCACHEABLE] = 3; pat_table[PAT_WRITE_COMBINING] = 3; pat_table[PAT_WRITE_PROTECTED] = 3; pat_table[PAT_UNCACHED] = 3; /* Bail if this CPU doesn't implement PAT. */ if ((cpu_feature & CPUID_PAT) == 0) { for (i = 0; i < PAT_INDEX_SIZE; i++) pat_index[i] = pat_table[i]; pat_works = 0; return; } /* * Due to some Intel errata, we can only safely use the lower 4 * PAT entries. * * Intel Pentium III Processor Specification Update * Errata E.27 (Upper Four PAT Entries Not Usable With Mode B * or Mode C Paging) * * Intel Pentium IV Processor Specification Update * Errata N46 (PAT Index MSB May Be Calculated Incorrectly) */ if (cpu_vendor_id == CPU_VENDOR_INTEL && !(CPUID_TO_FAMILY(cpu_id) == 6 && CPUID_TO_MODEL(cpu_id) >= 0xe)) pat_works = 0; /* Initialize default PAT entries. */ pat_msr = PAT_VALUE(0, PAT_WRITE_BACK) | PAT_VALUE(1, PAT_WRITE_THROUGH) | PAT_VALUE(2, PAT_UNCACHED) | PAT_VALUE(3, PAT_UNCACHEABLE) | PAT_VALUE(4, PAT_WRITE_BACK) | PAT_VALUE(5, PAT_WRITE_THROUGH) | PAT_VALUE(6, PAT_UNCACHED) | PAT_VALUE(7, PAT_UNCACHEABLE); if (pat_works) { /* * Leave the indices 0-3 at the default of WB, WT, UC-, and UC. * Program 5 and 6 as WP and WC. * Leave 4 and 7 as WB and UC. */ pat_msr &= ~(PAT_MASK(5) | PAT_MASK(6)); pat_msr |= PAT_VALUE(5, PAT_WRITE_PROTECTED) | PAT_VALUE(6, PAT_WRITE_COMBINING); pat_table[PAT_UNCACHED] = 2; pat_table[PAT_WRITE_PROTECTED] = 5; pat_table[PAT_WRITE_COMBINING] = 6; } else { /* * Just replace PAT Index 2 with WC instead of UC-. */ pat_msr &= ~PAT_MASK(2); pat_msr |= PAT_VALUE(2, PAT_WRITE_COMBINING); pat_table[PAT_WRITE_COMBINING] = 2; } /* Disable PGE. */ cr4 = rcr4(); load_cr4(cr4 & ~CR4_PGE); /* Disable caches (CD = 1, NW = 0). */ cr0 = rcr0(); load_cr0((cr0 & ~CR0_NW) | CR0_CD); /* Flushes caches and TLBs. */ wbinvd(); invltlb(); /* Update PAT and index table. */ wrmsr(MSR_PAT, pat_msr); for (i = 0; i < PAT_INDEX_SIZE; i++) pat_index[i] = pat_table[i]; /* Flush caches and TLBs again. */ wbinvd(); invltlb(); /* Restore caches and PGE. */ load_cr0(cr0); load_cr4(cr4); } /* * Set PG_G on kernel pages. Only the BSP calls this when SMP is turned on. */ static void pmap_set_pg(void) { pt_entry_t *pte; vm_offset_t va, endva; if (pgeflag == 0) return; endva = KERNBASE + KERNend; if (pseflag) { va = KERNBASE + KERNLOAD; while (va < endva) { pdir_pde(PTD, va) |= pgeflag; invltlb(); /* Play it safe, invltlb() every time */ va += NBPDR; } } else { va = (vm_offset_t)btext; while (va < endva) { pte = vtopte(va); if (*pte) *pte |= pgeflag; invltlb(); /* Play it safe, invltlb() every time */ va += PAGE_SIZE; } } } /* * Initialize a vm_page's machine-dependent fields. */ void pmap_page_init(vm_page_t m) { TAILQ_INIT(&m->md.pv_list); m->md.pat_mode = PAT_WRITE_BACK; } -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) static void * pmap_pdpt_allocf(uma_zone_t zone, vm_size_t bytes, uint8_t *flags, int wait) { /* Inform UMA that this allocator uses kernel_map/object. */ *flags = UMA_SLAB_KERNEL; return ((void *)kmem_alloc_contig(kernel_arena, bytes, wait, 0x0ULL, 0xffffffffULL, 1, 0, VM_MEMATTR_DEFAULT)); } #endif /* - * ABuse the pte nodes for unmapped kva to thread a kva freelist through. + * Abuse the pte nodes for unmapped kva to thread a kva freelist through. * Requirements: * - Must deal with pages in order to ensure that none of the PG_* bits * are ever set, PG_V in particular. * - Assumes we can write to ptes without pte_store() atomic ops, even * on PAE systems. This should be ok. * - Assumes nothing will ever test these addresses for 0 to indicate * no mapping instead of correctly checking PG_V. * - Assumes a vm_offset_t will fit in a pte (true for i386). * Because PG_V is never set, there can be no mappings to invalidate. */ static vm_offset_t pmap_ptelist_alloc(vm_offset_t *head) { pt_entry_t *pte; vm_offset_t va; va = *head; if (va == 0) panic("pmap_ptelist_alloc: exhausted ptelist KVA"); pte = vtopte(va); *head = *pte; if (*head & PG_V) panic("pmap_ptelist_alloc: va with PG_V set!"); *pte = 0; return (va); } static void pmap_ptelist_free(vm_offset_t *head, vm_offset_t va) { pt_entry_t *pte; if (va & PG_V) panic("pmap_ptelist_free: freeing va with PG_V set!"); pte = vtopte(va); *pte = *head; /* virtual! PG_V is 0 though */ *head = va; } static void pmap_ptelist_init(vm_offset_t *head, void *base, int npages) { int i; vm_offset_t va; *head = 0; for (i = npages - 1; i >= 0; i--) { va = (vm_offset_t)base + i * PAGE_SIZE; pmap_ptelist_free(head, va); } } /* * Initialize the pmap module. * Called by vm_init, to initialize any structures that the pmap * system needs to map virtual memory. */ void pmap_init(void) { vm_page_t mpte; vm_size_t s; int i, pv_npg; /* * Initialize the vm page array entries for the kernel pmap's * page table pages. */ for (i = 0; i < NKPT; i++) { mpte = PHYS_TO_VM_PAGE(KPTphys + (i << PAGE_SHIFT)); KASSERT(mpte >= vm_page_array && mpte < &vm_page_array[vm_page_array_size], ("pmap_init: page table page is out of range")); mpte->pindex = i + KPTDI; mpte->phys_addr = KPTphys + (i << PAGE_SHIFT); } /* * Initialize the address space (zone) for the pv entries. Set a * high water mark so that the system can recover from excessive * numbers of pv entries. */ TUNABLE_INT_FETCH("vm.pmap.shpgperproc", &shpgperproc); pv_entry_max = shpgperproc * maxproc + vm_cnt.v_page_count; TUNABLE_INT_FETCH("vm.pmap.pv_entries", &pv_entry_max); pv_entry_max = roundup(pv_entry_max, _NPCPV); pv_entry_high_water = 9 * (pv_entry_max / 10); /* * If the kernel is running on a virtual machine, then it must assume * that MCA is enabled by the hypervisor. Moreover, the kernel must * be prepared for the hypervisor changing the vendor and family that * are reported by CPUID. Consequently, the workaround for AMD Family * 10h Erratum 383 is enabled if the processor's feature set does not * include at least one feature that is only supported by older Intel * or newer AMD processors. */ if (vm_guest == VM_GUEST_VM && (cpu_feature & CPUID_SS) == 0 && (cpu_feature2 & (CPUID2_SSSE3 | CPUID2_SSE41 | CPUID2_AESNI | CPUID2_AVX | CPUID2_XSAVE)) == 0 && (amd_feature2 & (AMDID2_XOP | AMDID2_FMA4)) == 0) workaround_erratum383 = 1; /* * Are large page mappings supported and enabled? */ TUNABLE_INT_FETCH("vm.pmap.pg_ps_enabled", &pg_ps_enabled); if (pseflag == 0) pg_ps_enabled = 0; else if (pg_ps_enabled) { KASSERT(MAXPAGESIZES > 1 && pagesizes[1] == 0, ("pmap_init: can't assign to pagesizes[1]")); pagesizes[1] = NBPDR; } /* * Calculate the size of the pv head table for superpages. * Handle the possibility that "vm_phys_segs[...].end" is zero. */ pv_npg = trunc_4mpage(vm_phys_segs[vm_phys_nsegs - 1].end - PAGE_SIZE) / NBPDR + 1; /* * Allocate memory for the pv head table for superpages. */ s = (vm_size_t)(pv_npg * sizeof(struct md_page)); s = round_page(s); pv_table = (struct md_page *)kmem_malloc(kernel_arena, s, M_WAITOK | M_ZERO); for (i = 0; i < pv_npg; i++) TAILQ_INIT(&pv_table[i].pv_list); pv_maxchunks = MAX(pv_entry_max / _NPCPV, maxproc); pv_chunkbase = (struct pv_chunk *)kva_alloc(PAGE_SIZE * pv_maxchunks); if (pv_chunkbase == NULL) panic("pmap_init: not enough kvm for pv chunks"); pmap_ptelist_init(&pv_vafree, pv_chunkbase, pv_maxchunks); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) pdptzone = uma_zcreate("PDPT", NPGPTD * sizeof(pdpt_entry_t), NULL, NULL, NULL, NULL, (NPGPTD * sizeof(pdpt_entry_t)) - 1, UMA_ZONE_VM | UMA_ZONE_NOFREE); uma_zone_set_allocf(pdptzone, pmap_pdpt_allocf); #endif } SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_max, CTLFLAG_RD, &pv_entry_max, 0, "Max number of PV entries"); SYSCTL_INT(_vm_pmap, OID_AUTO, shpgperproc, CTLFLAG_RD, &shpgperproc, 0, "Page share factor per proc"); static SYSCTL_NODE(_vm_pmap, OID_AUTO, pde, CTLFLAG_RD, 0, "2/4MB page mapping counters"); static u_long pmap_pde_demotions; SYSCTL_ULONG(_vm_pmap_pde, OID_AUTO, demotions, CTLFLAG_RD, &pmap_pde_demotions, 0, "2/4MB page demotions"); static u_long pmap_pde_mappings; SYSCTL_ULONG(_vm_pmap_pde, OID_AUTO, mappings, CTLFLAG_RD, &pmap_pde_mappings, 0, "2/4MB page mappings"); static u_long pmap_pde_p_failures; SYSCTL_ULONG(_vm_pmap_pde, OID_AUTO, p_failures, CTLFLAG_RD, &pmap_pde_p_failures, 0, "2/4MB page promotion failures"); static u_long pmap_pde_promotions; SYSCTL_ULONG(_vm_pmap_pde, OID_AUTO, promotions, CTLFLAG_RD, &pmap_pde_promotions, 0, "2/4MB page promotions"); /*************************************************** * Low level helper routines..... ***************************************************/ /* * Determine the appropriate bits to set in a PTE or PDE for a specified * caching mode. */ int pmap_cache_bits(int mode, boolean_t is_pde) { int cache_bits, pat_flag, pat_idx; if (mode < 0 || mode >= PAT_INDEX_SIZE || pat_index[mode] < 0) panic("Unknown caching mode %d\n", mode); /* The PAT bit is different for PTE's and PDE's. */ pat_flag = is_pde ? PG_PDE_PAT : PG_PTE_PAT; /* Map the caching mode to a PAT index. */ pat_idx = pat_index[mode]; /* Map the 3-bit index value into the PAT, PCD, and PWT bits. */ cache_bits = 0; if (pat_idx & 0x4) cache_bits |= pat_flag; if (pat_idx & 0x2) cache_bits |= PG_NC_PCD; if (pat_idx & 0x1) cache_bits |= PG_NC_PWT; return (cache_bits); } /* * The caller is responsible for maintaining TLB consistency. */ static void pmap_kenter_pde(vm_offset_t va, pd_entry_t newpde) { pd_entry_t *pde; pmap_t pmap; boolean_t PTD_updated; PTD_updated = FALSE; mtx_lock_spin(&allpmaps_lock); LIST_FOREACH(pmap, &allpmaps, pm_list) { if ((pmap->pm_pdir[PTDPTDI] & PG_FRAME) == (PTDpde[0] & PG_FRAME)) PTD_updated = TRUE; pde = pmap_pde(pmap, va); pde_store(pde, newpde); } mtx_unlock_spin(&allpmaps_lock); KASSERT(PTD_updated, ("pmap_kenter_pde: current page table is not in allpmaps")); } /* * After changing the page size for the specified virtual address in the page * table, flush the corresponding entries from the processor's TLB. Only the * calling processor's TLB is affected. * * The calling thread must be pinned to a processor. */ static void pmap_update_pde_invalidate(vm_offset_t va, pd_entry_t newpde) { u_long cr4; if ((newpde & PG_PS) == 0) /* Demotion: flush a specific 2MB page mapping. */ invlpg(va); else if ((newpde & PG_G) == 0) /* * Promotion: flush every 4KB page mapping from the TLB * because there are too many to flush individually. */ invltlb(); else { /* * Promotion: flush every 4KB page mapping from the TLB, * including any global (PG_G) mappings. */ cr4 = rcr4(); load_cr4(cr4 & ~CR4_PGE); /* * Although preemption at this point could be detrimental to * performance, it would not lead to an error. PG_G is simply * ignored if CR4.PGE is clear. Moreover, in case this block * is re-entered, the load_cr4() either above or below will * modify CR4.PGE flushing the TLB. */ load_cr4(cr4 | CR4_PGE); } } #ifdef SMP /* * For SMP, these functions have to use the IPI mechanism for coherence. * * N.B.: Before calling any of the following TLB invalidation functions, * the calling processor must ensure that all stores updating a non- * kernel page table are globally performed. Otherwise, another * processor could cache an old, pre-update entry without being * invalidated. This can happen one of two ways: (1) The pmap becomes * active on another processor after its pm_active field is checked by * one of the following functions but before a store updating the page * table is globally performed. (2) The pmap becomes active on another * processor before its pm_active field is checked but due to * speculative loads one of the following functions stills reads the * pmap as inactive on the other processor. * * The kernel page table is exempt because its pm_active field is * immutable. The kernel page table is always active on every * processor. */ void pmap_invalidate_page(pmap_t pmap, vm_offset_t va) { cpuset_t other_cpus; u_int cpuid; sched_pin(); if (pmap == kernel_pmap || !CPU_CMP(&pmap->pm_active, &all_cpus)) { invlpg(va); smp_invlpg(va); } else { cpuid = PCPU_GET(cpuid); other_cpus = all_cpus; CPU_CLR(cpuid, &other_cpus); if (CPU_ISSET(cpuid, &pmap->pm_active)) invlpg(va); CPU_AND(&other_cpus, &pmap->pm_active); if (!CPU_EMPTY(&other_cpus)) smp_masked_invlpg(other_cpus, va); } sched_unpin(); } void pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { cpuset_t other_cpus; vm_offset_t addr; u_int cpuid; sched_pin(); if (pmap == kernel_pmap || !CPU_CMP(&pmap->pm_active, &all_cpus)) { for (addr = sva; addr < eva; addr += PAGE_SIZE) invlpg(addr); smp_invlpg_range(sva, eva); } else { cpuid = PCPU_GET(cpuid); other_cpus = all_cpus; CPU_CLR(cpuid, &other_cpus); if (CPU_ISSET(cpuid, &pmap->pm_active)) for (addr = sva; addr < eva; addr += PAGE_SIZE) invlpg(addr); CPU_AND(&other_cpus, &pmap->pm_active); if (!CPU_EMPTY(&other_cpus)) smp_masked_invlpg_range(other_cpus, sva, eva); } sched_unpin(); } void pmap_invalidate_all(pmap_t pmap) { cpuset_t other_cpus; u_int cpuid; sched_pin(); if (pmap == kernel_pmap || !CPU_CMP(&pmap->pm_active, &all_cpus)) { invltlb(); smp_invltlb(); } else { cpuid = PCPU_GET(cpuid); other_cpus = all_cpus; CPU_CLR(cpuid, &other_cpus); if (CPU_ISSET(cpuid, &pmap->pm_active)) invltlb(); CPU_AND(&other_cpus, &pmap->pm_active); if (!CPU_EMPTY(&other_cpus)) smp_masked_invltlb(other_cpus); } sched_unpin(); } void pmap_invalidate_cache(void) { sched_pin(); wbinvd(); smp_cache_flush(); sched_unpin(); } struct pde_action { cpuset_t invalidate; /* processors that invalidate their TLB */ vm_offset_t va; pd_entry_t *pde; pd_entry_t newpde; u_int store; /* processor that updates the PDE */ }; static void pmap_update_pde_kernel(void *arg) { struct pde_action *act = arg; pd_entry_t *pde; pmap_t pmap; if (act->store == PCPU_GET(cpuid)) { /* * Elsewhere, this operation requires allpmaps_lock for * synchronization. Here, it does not because it is being * performed in the context of an all_cpus rendezvous. */ LIST_FOREACH(pmap, &allpmaps, pm_list) { pde = pmap_pde(pmap, act->va); pde_store(pde, act->newpde); } } } static void pmap_update_pde_user(void *arg) { struct pde_action *act = arg; if (act->store == PCPU_GET(cpuid)) pde_store(act->pde, act->newpde); } static void pmap_update_pde_teardown(void *arg) { struct pde_action *act = arg; if (CPU_ISSET(PCPU_GET(cpuid), &act->invalidate)) pmap_update_pde_invalidate(act->va, act->newpde); } /* * Change the page size for the specified virtual address in a way that * prevents any possibility of the TLB ever having two entries that map the * same virtual address using different page sizes. This is the recommended * workaround for Erratum 383 on AMD Family 10h processors. It prevents a * machine check exception for a TLB state that is improperly diagnosed as a * hardware error. */ static void pmap_update_pde(pmap_t pmap, vm_offset_t va, pd_entry_t *pde, pd_entry_t newpde) { struct pde_action act; cpuset_t active, other_cpus; u_int cpuid; sched_pin(); cpuid = PCPU_GET(cpuid); other_cpus = all_cpus; CPU_CLR(cpuid, &other_cpus); if (pmap == kernel_pmap) active = all_cpus; else active = pmap->pm_active; if (CPU_OVERLAP(&active, &other_cpus)) { act.store = cpuid; act.invalidate = active; act.va = va; act.pde = pde; act.newpde = newpde; CPU_SET(cpuid, &active); smp_rendezvous_cpus(active, smp_no_rendevous_barrier, pmap == kernel_pmap ? pmap_update_pde_kernel : pmap_update_pde_user, pmap_update_pde_teardown, &act); } else { if (pmap == kernel_pmap) pmap_kenter_pde(va, newpde); else pde_store(pde, newpde); if (CPU_ISSET(cpuid, &active)) pmap_update_pde_invalidate(va, newpde); } sched_unpin(); } #else /* !SMP */ /* * Normal, non-SMP, 486+ invalidation functions. * We inline these within pmap.c for speed. */ PMAP_INLINE void pmap_invalidate_page(pmap_t pmap, vm_offset_t va) { if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) invlpg(va); } PMAP_INLINE void pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { vm_offset_t addr; if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) for (addr = sva; addr < eva; addr += PAGE_SIZE) invlpg(addr); } PMAP_INLINE void pmap_invalidate_all(pmap_t pmap) { if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) invltlb(); } PMAP_INLINE void pmap_invalidate_cache(void) { wbinvd(); } static void pmap_update_pde(pmap_t pmap, vm_offset_t va, pd_entry_t *pde, pd_entry_t newpde) { if (pmap == kernel_pmap) pmap_kenter_pde(va, newpde); else pde_store(pde, newpde); if (pmap == kernel_pmap || !CPU_EMPTY(&pmap->pm_active)) pmap_update_pde_invalidate(va, newpde); } #endif /* !SMP */ #define PMAP_CLFLUSH_THRESHOLD (2 * 1024 * 1024) void pmap_invalidate_cache_range(vm_offset_t sva, vm_offset_t eva, boolean_t force) { if (force) { sva &= ~(vm_offset_t)cpu_clflush_line_size; } else { KASSERT((sva & PAGE_MASK) == 0, ("pmap_invalidate_cache_range: sva not page-aligned")); KASSERT((eva & PAGE_MASK) == 0, ("pmap_invalidate_cache_range: eva not page-aligned")); } if ((cpu_feature & CPUID_SS) != 0 && !force) ; /* If "Self Snoop" is supported and allowed, do nothing. */ else if ((cpu_feature & CPUID_CLFSH) != 0 && eva - sva < PMAP_CLFLUSH_THRESHOLD) { #ifdef DEV_APIC /* * XXX: Some CPUs fault, hang, or trash the local APIC * registers if we use CLFLUSH on the local APIC * range. The local APIC is always uncached, so we * don't need to flush for that range anyway. */ if (pmap_kextract(sva) == lapic_paddr) return; #endif /* * Otherwise, do per-cache line flush. Use the mfence * instruction to insure that previous stores are * included in the write-back. The processor * propagates flush to other processors in the cache * coherence domain. */ mfence(); for (; sva < eva; sva += cpu_clflush_line_size) clflush(sva); mfence(); } else { /* * No targeted cache flush methods are supported by CPU, * or the supplied range is bigger than 2MB. * Globally invalidate cache. */ pmap_invalidate_cache(); } } void pmap_invalidate_cache_pages(vm_page_t *pages, int count) { int i; if (count >= PMAP_CLFLUSH_THRESHOLD / PAGE_SIZE || (cpu_feature & CPUID_CLFSH) == 0) { pmap_invalidate_cache(); } else { for (i = 0; i < count; i++) pmap_flush_page(pages[i]); } } /* * Are we current address space or kernel? N.B. We return FALSE when * a pmap's page table is in use because a kernel thread is borrowing * it. The borrowed page table can change spontaneously, making any * dependence on its continued use subject to a race condition. */ static __inline int pmap_is_current(pmap_t pmap) { return (pmap == kernel_pmap || (pmap == vmspace_pmap(curthread->td_proc->p_vmspace) && (pmap->pm_pdir[PTDPTDI] & PG_FRAME) == (PTDpde[0] & PG_FRAME))); } /* * If the given pmap is not the current or kernel pmap, the returned pte must * be released by passing it to pmap_pte_release(). */ pt_entry_t * pmap_pte(pmap_t pmap, vm_offset_t va) { pd_entry_t newpf; pd_entry_t *pde; pde = pmap_pde(pmap, va); if (*pde & PG_PS) return (pde); if (*pde != 0) { /* are we current address space or kernel? */ if (pmap_is_current(pmap)) return (vtopte(va)); mtx_lock(&PMAP2mutex); newpf = *pde & PG_FRAME; if ((*PMAP2 & PG_FRAME) != newpf) { *PMAP2 = newpf | PG_RW | PG_V | PG_A | PG_M; pmap_invalidate_page(kernel_pmap, (vm_offset_t)PADDR2); } return (PADDR2 + (i386_btop(va) & (NPTEPG - 1))); } return (NULL); } /* * Releases a pte that was obtained from pmap_pte(). Be prepared for the pte * being NULL. */ static __inline void pmap_pte_release(pt_entry_t *pte) { if ((pt_entry_t *)((vm_offset_t)pte & ~PAGE_MASK) == PADDR2) mtx_unlock(&PMAP2mutex); } /* * NB: The sequence of updating a page table followed by accesses to the * corresponding pages is subject to the situation described in the "AMD64 * Architecture Programmer's Manual Volume 2: System Programming" rev. 3.23, * "7.3.1 Special Coherency Considerations". Therefore, issuing the INVLPG * right after modifying the PTE bits is crucial. */ static __inline void invlcaddr(void *caddr) { invlpg((u_int)caddr); } /* * Super fast pmap_pte routine best used when scanning * the pv lists. This eliminates many coarse-grained * invltlb calls. Note that many of the pv list * scans are across different pmaps. It is very wasteful * to do an entire invltlb for checking a single mapping. * * If the given pmap is not the current pmap, pvh_global_lock * must be held and curthread pinned to a CPU. */ static pt_entry_t * pmap_pte_quick(pmap_t pmap, vm_offset_t va) { pd_entry_t newpf; pd_entry_t *pde; pde = pmap_pde(pmap, va); if (*pde & PG_PS) return (pde); if (*pde != 0) { /* are we current address space or kernel? */ if (pmap_is_current(pmap)) return (vtopte(va)); rw_assert(&pvh_global_lock, RA_WLOCKED); KASSERT(curthread->td_pinned > 0, ("curthread not pinned")); newpf = *pde & PG_FRAME; if ((*PMAP1 & PG_FRAME) != newpf) { *PMAP1 = newpf | PG_RW | PG_V | PG_A | PG_M; #ifdef SMP PMAP1cpu = PCPU_GET(cpuid); #endif invlcaddr(PADDR1); PMAP1changed++; } else #ifdef SMP if (PMAP1cpu != PCPU_GET(cpuid)) { PMAP1cpu = PCPU_GET(cpuid); invlcaddr(PADDR1); PMAP1changedcpu++; } else #endif PMAP1unchanged++; return (PADDR1 + (i386_btop(va) & (NPTEPG - 1))); } return (0); } /* * Routine: pmap_extract * Function: * Extract the physical page address associated * with the given map/virtual_address pair. */ vm_paddr_t pmap_extract(pmap_t pmap, vm_offset_t va) { vm_paddr_t rtval; pt_entry_t *pte; pd_entry_t pde; rtval = 0; PMAP_LOCK(pmap); pde = pmap->pm_pdir[va >> PDRSHIFT]; if (pde != 0) { if ((pde & PG_PS) != 0) rtval = (pde & PG_PS_FRAME) | (va & PDRMASK); else { pte = pmap_pte(pmap, va); rtval = (*pte & PG_FRAME) | (va & PAGE_MASK); pmap_pte_release(pte); } } PMAP_UNLOCK(pmap); return (rtval); } /* * Routine: pmap_extract_and_hold * Function: * Atomically extract and hold the physical page * with the given pmap and virtual address pair * if that mapping permits the given protection. */ vm_page_t pmap_extract_and_hold(pmap_t pmap, vm_offset_t va, vm_prot_t prot) { pd_entry_t pde; pt_entry_t pte, *ptep; vm_page_t m; vm_paddr_t pa; pa = 0; m = NULL; PMAP_LOCK(pmap); retry: pde = *pmap_pde(pmap, va); if (pde != 0) { if (pde & PG_PS) { if ((pde & PG_RW) || (prot & VM_PROT_WRITE) == 0) { if (vm_page_pa_tryrelock(pmap, (pde & PG_PS_FRAME) | (va & PDRMASK), &pa)) goto retry; m = PHYS_TO_VM_PAGE((pde & PG_PS_FRAME) | (va & PDRMASK)); vm_page_hold(m); } } else { ptep = pmap_pte(pmap, va); pte = *ptep; pmap_pte_release(ptep); if (pte != 0 && ((pte & PG_RW) || (prot & VM_PROT_WRITE) == 0)) { if (vm_page_pa_tryrelock(pmap, pte & PG_FRAME, &pa)) goto retry; m = PHYS_TO_VM_PAGE(pte & PG_FRAME); vm_page_hold(m); } } } PA_UNLOCK_COND(pa); PMAP_UNLOCK(pmap); return (m); } /*************************************************** * Low level mapping routines..... ***************************************************/ /* * Add a wired page to the kva. * Note: not SMP coherent. * * This function may be used before pmap_bootstrap() is called. */ PMAP_INLINE void pmap_kenter(vm_offset_t va, vm_paddr_t pa) { pt_entry_t *pte; pte = vtopte(va); pte_store(pte, pa | PG_RW | PG_V | pgeflag); } static __inline void pmap_kenter_attr(vm_offset_t va, vm_paddr_t pa, int mode) { pt_entry_t *pte; pte = vtopte(va); pte_store(pte, pa | PG_RW | PG_V | pgeflag | pmap_cache_bits(mode, 0)); } /* * Remove a page from the kernel pagetables. * Note: not SMP coherent. * * This function may be used before pmap_bootstrap() is called. */ PMAP_INLINE void pmap_kremove(vm_offset_t va) { pt_entry_t *pte; pte = vtopte(va); pte_clear(pte); } /* * Used to map a range of physical addresses into kernel * virtual address space. * * The value passed in '*virt' is a suggested virtual address for * the mapping. Architectures which can support a direct-mapped * physical to virtual region can return the appropriate address * within that region, leaving '*virt' unchanged. Other * architectures should map the pages starting at '*virt' and * update '*virt' with the first usable address after the mapped * region. */ vm_offset_t pmap_map(vm_offset_t *virt, vm_paddr_t start, vm_paddr_t end, int prot) { vm_offset_t va, sva; vm_paddr_t superpage_offset; pd_entry_t newpde; va = *virt; /* * Does the physical address range's size and alignment permit at * least one superpage mapping to be created? */ superpage_offset = start & PDRMASK; if ((end - start) - ((NBPDR - superpage_offset) & PDRMASK) >= NBPDR) { /* * Increase the starting virtual address so that its alignment * does not preclude the use of superpage mappings. */ if ((va & PDRMASK) < superpage_offset) va = (va & ~PDRMASK) + superpage_offset; else if ((va & PDRMASK) > superpage_offset) va = ((va + PDRMASK) & ~PDRMASK) + superpage_offset; } sva = va; while (start < end) { if ((start & PDRMASK) == 0 && end - start >= NBPDR && pseflag) { KASSERT((va & PDRMASK) == 0, ("pmap_map: misaligned va %#x", va)); newpde = start | PG_PS | pgeflag | PG_RW | PG_V; pmap_kenter_pde(va, newpde); va += NBPDR; start += NBPDR; } else { pmap_kenter(va, start); va += PAGE_SIZE; start += PAGE_SIZE; } } pmap_invalidate_range(kernel_pmap, sva, va); *virt = va; return (sva); } /* * Add a list of wired pages to the kva * this routine is only used for temporary * kernel mappings that do not need to have * page modification or references recorded. * Note that old mappings are simply written * over. The page *must* be wired. * Note: SMP coherent. Uses a ranged shootdown IPI. */ void pmap_qenter(vm_offset_t sva, vm_page_t *ma, int count) { pt_entry_t *endpte, oldpte, pa, *pte; vm_page_t m; oldpte = 0; pte = vtopte(sva); endpte = pte + count; while (pte < endpte) { m = *ma++; pa = VM_PAGE_TO_PHYS(m) | pmap_cache_bits(m->md.pat_mode, 0); if ((*pte & (PG_FRAME | PG_PTE_CACHE)) != pa) { oldpte |= *pte; pte_store(pte, pa | pgeflag | PG_RW | PG_V); } pte++; } if (__predict_false((oldpte & PG_V) != 0)) pmap_invalidate_range(kernel_pmap, sva, sva + count * PAGE_SIZE); } /* * This routine tears out page mappings from the * kernel -- it is meant only for temporary mappings. * Note: SMP coherent. Uses a ranged shootdown IPI. */ void pmap_qremove(vm_offset_t sva, int count) { vm_offset_t va; va = sva; while (count-- > 0) { pmap_kremove(va); va += PAGE_SIZE; } pmap_invalidate_range(kernel_pmap, sva, va); } /*************************************************** * Page table page management routines..... ***************************************************/ static __inline void pmap_free_zero_pages(struct spglist *free) { vm_page_t m; while ((m = SLIST_FIRST(free)) != NULL) { SLIST_REMOVE_HEAD(free, plinks.s.ss); /* Preserve the page's PG_ZERO setting. */ vm_page_free_toq(m); } } /* * Schedule the specified unused page table page to be freed. Specifically, * add the page to the specified list of pages that will be released to the * physical memory manager after the TLB has been updated. */ static __inline void pmap_add_delayed_free_list(vm_page_t m, struct spglist *free, boolean_t set_PG_ZERO) { if (set_PG_ZERO) m->flags |= PG_ZERO; else m->flags &= ~PG_ZERO; SLIST_INSERT_HEAD(free, m, plinks.s.ss); } /* * Inserts the specified page table page into the specified pmap's collection * of idle page table pages. Each of a pmap's page table pages is responsible * for mapping a distinct range of virtual addresses. The pmap's collection is * ordered by this virtual address range. */ static __inline int pmap_insert_pt_page(pmap_t pmap, vm_page_t mpte) { PMAP_LOCK_ASSERT(pmap, MA_OWNED); return (vm_radix_insert(&pmap->pm_root, mpte)); } /* * Looks for a page table page mapping the specified virtual address in the * specified pmap's collection of idle page table pages. Returns NULL if there * is no page table page corresponding to the specified virtual address. */ static __inline vm_page_t pmap_lookup_pt_page(pmap_t pmap, vm_offset_t va) { PMAP_LOCK_ASSERT(pmap, MA_OWNED); return (vm_radix_lookup(&pmap->pm_root, va >> PDRSHIFT)); } /* * Removes the specified page table page from the specified pmap's collection * of idle page table pages. The specified page table page must be a member of * the pmap's collection. */ static __inline void pmap_remove_pt_page(pmap_t pmap, vm_page_t mpte) { PMAP_LOCK_ASSERT(pmap, MA_OWNED); vm_radix_remove(&pmap->pm_root, mpte->pindex); } /* * Decrements a page table page's wire count, which is used to record the * number of valid page table entries within the page. If the wire count * drops to zero, then the page table page is unmapped. Returns TRUE if the * page table page was unmapped and FALSE otherwise. */ static inline boolean_t pmap_unwire_ptp(pmap_t pmap, vm_page_t m, struct spglist *free) { --m->wire_count; if (m->wire_count == 0) { _pmap_unwire_ptp(pmap, m, free); return (TRUE); } else return (FALSE); } static void _pmap_unwire_ptp(pmap_t pmap, vm_page_t m, struct spglist *free) { vm_offset_t pteva; /* * unmap the page table page */ pmap->pm_pdir[m->pindex] = 0; --pmap->pm_stats.resident_count; /* * This is a release store so that the ordinary store unmapping * the page table page is globally performed before TLB shoot- * down is begun. */ atomic_subtract_rel_int(&vm_cnt.v_wire_count, 1); /* * Do an invltlb to make the invalidated mapping * take effect immediately. */ pteva = VM_MAXUSER_ADDRESS + i386_ptob(m->pindex); pmap_invalidate_page(pmap, pteva); /* * Put page on a list so that it is released after * *ALL* TLB shootdown is done */ pmap_add_delayed_free_list(m, free, TRUE); } /* * After removing a page table entry, this routine is used to * conditionally free the page, and manage the hold/wire counts. */ static int pmap_unuse_pt(pmap_t pmap, vm_offset_t va, struct spglist *free) { pd_entry_t ptepde; vm_page_t mpte; if (va >= VM_MAXUSER_ADDRESS) return (0); ptepde = *pmap_pde(pmap, va); mpte = PHYS_TO_VM_PAGE(ptepde & PG_FRAME); return (pmap_unwire_ptp(pmap, mpte, free)); } /* * Initialize the pmap for the swapper process. */ void pmap_pinit0(pmap_t pmap) { PMAP_LOCK_INIT(pmap); /* * Since the page table directory is shared with the kernel pmap, * which is already included in the list "allpmaps", this pmap does * not need to be inserted into that list. */ pmap->pm_pdir = (pd_entry_t *)(KERNBASE + (vm_offset_t)IdlePTD); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) pmap->pm_pdpt = (pdpt_entry_t *)(KERNBASE + (vm_offset_t)IdlePDPT); #endif pmap->pm_root.rt_root = 0; CPU_ZERO(&pmap->pm_active); PCPU_SET(curpmap, pmap); TAILQ_INIT(&pmap->pm_pvchunk); bzero(&pmap->pm_stats, sizeof pmap->pm_stats); } /* * Initialize a preallocated and zeroed pmap structure, * such as one in a vmspace structure. */ int pmap_pinit(pmap_t pmap) { vm_page_t m, ptdpg[NPGPTD]; vm_paddr_t pa; int i; /* * No need to allocate page table space yet but we do need a valid * page directory table. */ if (pmap->pm_pdir == NULL) { pmap->pm_pdir = (pd_entry_t *)kva_alloc(NBPTD); if (pmap->pm_pdir == NULL) return (0); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) pmap->pm_pdpt = uma_zalloc(pdptzone, M_WAITOK | M_ZERO); KASSERT(((vm_offset_t)pmap->pm_pdpt & ((NPGPTD * sizeof(pdpt_entry_t)) - 1)) == 0, ("pmap_pinit: pdpt misaligned")); KASSERT(pmap_kextract((vm_offset_t)pmap->pm_pdpt) < (4ULL<<30), ("pmap_pinit: pdpt above 4g")); #endif pmap->pm_root.rt_root = 0; } KASSERT(vm_radix_is_empty(&pmap->pm_root), ("pmap_pinit: pmap has reserved page table page(s)")); /* * allocate the page directory page(s) */ for (i = 0; i < NPGPTD;) { m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO); if (m == NULL) VM_WAIT; else { ptdpg[i++] = m; } } pmap_qenter((vm_offset_t)pmap->pm_pdir, ptdpg, NPGPTD); for (i = 0; i < NPGPTD; i++) if ((ptdpg[i]->flags & PG_ZERO) == 0) pagezero(pmap->pm_pdir + (i * NPDEPG)); mtx_lock_spin(&allpmaps_lock); LIST_INSERT_HEAD(&allpmaps, pmap, pm_list); /* Copy the kernel page table directory entries. */ bcopy(PTD + KPTDI, pmap->pm_pdir + KPTDI, nkpt * sizeof(pd_entry_t)); mtx_unlock_spin(&allpmaps_lock); /* install self-referential address mapping entry(s) */ for (i = 0; i < NPGPTD; i++) { pa = VM_PAGE_TO_PHYS(ptdpg[i]); pmap->pm_pdir[PTDPTDI + i] = pa | PG_V | PG_RW | PG_A | PG_M; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) pmap->pm_pdpt[i] = pa | PG_V; #endif } CPU_ZERO(&pmap->pm_active); TAILQ_INIT(&pmap->pm_pvchunk); bzero(&pmap->pm_stats, sizeof pmap->pm_stats); return (1); } /* * this routine is called if the page table page is not * mapped correctly. */ static vm_page_t _pmap_allocpte(pmap_t pmap, u_int ptepindex, u_int flags) { vm_paddr_t ptepa; vm_page_t m; /* * Allocate a page table page. */ if ((m = vm_page_alloc(NULL, ptepindex, VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO)) == NULL) { if ((flags & PMAP_ENTER_NOSLEEP) == 0) { PMAP_UNLOCK(pmap); rw_wunlock(&pvh_global_lock); VM_WAIT; rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); } /* * Indicate the need to retry. While waiting, the page table * page may have been allocated. */ return (NULL); } if ((m->flags & PG_ZERO) == 0) pmap_zero_page(m); /* * Map the pagetable page into the process address space, if * it isn't already there. */ pmap->pm_stats.resident_count++; ptepa = VM_PAGE_TO_PHYS(m); pmap->pm_pdir[ptepindex] = (pd_entry_t) (ptepa | PG_U | PG_RW | PG_V | PG_A | PG_M); return (m); } static vm_page_t pmap_allocpte(pmap_t pmap, vm_offset_t va, u_int flags) { u_int ptepindex; pd_entry_t ptepa; vm_page_t m; /* * Calculate pagetable page index */ ptepindex = va >> PDRSHIFT; retry: /* * Get the page directory entry */ ptepa = pmap->pm_pdir[ptepindex]; /* * This supports switching from a 4MB page to a * normal 4K page. */ if (ptepa & PG_PS) { (void)pmap_demote_pde(pmap, &pmap->pm_pdir[ptepindex], va); ptepa = pmap->pm_pdir[ptepindex]; } /* * If the page table page is mapped, we just increment the * hold count, and activate it. */ if (ptepa) { m = PHYS_TO_VM_PAGE(ptepa & PG_FRAME); m->wire_count++; } else { /* * Here if the pte page isn't mapped, or if it has * been deallocated. */ m = _pmap_allocpte(pmap, ptepindex, flags); if (m == NULL && (flags & PMAP_ENTER_NOSLEEP) == 0) goto retry; } return (m); } /*************************************************** * Pmap allocation/deallocation routines. ***************************************************/ #ifdef SMP /* * Deal with a SMP shootdown of other users of the pmap that we are * trying to dispose of. This can be a bit hairy. */ static cpuset_t *lazymask; static u_int lazyptd; static volatile u_int lazywait; void pmap_lazyfix_action(void); void pmap_lazyfix_action(void) { #ifdef COUNT_IPIS (*ipi_lazypmap_counts[PCPU_GET(cpuid)])++; #endif if (rcr3() == lazyptd) load_cr3(curpcb->pcb_cr3); CPU_CLR_ATOMIC(PCPU_GET(cpuid), lazymask); atomic_store_rel_int(&lazywait, 1); } static void pmap_lazyfix_self(u_int cpuid) { if (rcr3() == lazyptd) load_cr3(curpcb->pcb_cr3); CPU_CLR_ATOMIC(cpuid, lazymask); } static void pmap_lazyfix(pmap_t pmap) { cpuset_t mymask, mask; u_int cpuid, spins; int lsb; mask = pmap->pm_active; while (!CPU_EMPTY(&mask)) { spins = 50000000; /* Find least significant set bit. */ lsb = CPU_FFS(&mask); MPASS(lsb != 0); lsb--; CPU_SETOF(lsb, &mask); mtx_lock_spin(&smp_ipi_mtx); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) lazyptd = vtophys(pmap->pm_pdpt); #else lazyptd = vtophys(pmap->pm_pdir); #endif cpuid = PCPU_GET(cpuid); /* Use a cpuset just for having an easy check. */ CPU_SETOF(cpuid, &mymask); if (!CPU_CMP(&mask, &mymask)) { lazymask = &pmap->pm_active; pmap_lazyfix_self(cpuid); } else { atomic_store_rel_int((u_int *)&lazymask, (u_int)&pmap->pm_active); atomic_store_rel_int(&lazywait, 0); ipi_selected(mask, IPI_LAZYPMAP); while (lazywait == 0) { ia32_pause(); if (--spins == 0) break; } } mtx_unlock_spin(&smp_ipi_mtx); if (spins == 0) printf("pmap_lazyfix: spun for 50000000\n"); mask = pmap->pm_active; } } #else /* SMP */ /* * Cleaning up on uniprocessor is easy. For various reasons, we're * unlikely to have to even execute this code, including the fact * that the cleanup is deferred until the parent does a wait(2), which * means that another userland process has run. */ static void pmap_lazyfix(pmap_t pmap) { u_int cr3; cr3 = vtophys(pmap->pm_pdir); if (cr3 == rcr3()) { load_cr3(curpcb->pcb_cr3); CPU_CLR(PCPU_GET(cpuid), &pmap->pm_active); } } #endif /* SMP */ /* * Release any resources held by the given physical map. * Called when a pmap initialized by pmap_pinit is being released. * Should only be called if the map contains no valid mappings. */ void pmap_release(pmap_t pmap) { vm_page_t m, ptdpg[NPGPTD]; int i; KASSERT(pmap->pm_stats.resident_count == 0, ("pmap_release: pmap resident count %ld != 0", pmap->pm_stats.resident_count)); KASSERT(vm_radix_is_empty(&pmap->pm_root), ("pmap_release: pmap has reserved page table page(s)")); pmap_lazyfix(pmap); mtx_lock_spin(&allpmaps_lock); LIST_REMOVE(pmap, pm_list); mtx_unlock_spin(&allpmaps_lock); for (i = 0; i < NPGPTD; i++) ptdpg[i] = PHYS_TO_VM_PAGE(pmap->pm_pdir[PTDPTDI + i] & PG_FRAME); bzero(pmap->pm_pdir + PTDPTDI, (nkpt + NPGPTD) * sizeof(*pmap->pm_pdir)); pmap_qremove((vm_offset_t)pmap->pm_pdir, NPGPTD); for (i = 0; i < NPGPTD; i++) { m = ptdpg[i]; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) KASSERT(VM_PAGE_TO_PHYS(m) == (pmap->pm_pdpt[i] & PG_FRAME), ("pmap_release: got wrong ptd page")); #endif m->wire_count--; atomic_subtract_int(&vm_cnt.v_wire_count, 1); vm_page_free_zero(m); } } static int kvm_size(SYSCTL_HANDLER_ARGS) { unsigned long ksize = VM_MAX_KERNEL_ADDRESS - KERNBASE; return (sysctl_handle_long(oidp, &ksize, 0, req)); } SYSCTL_PROC(_vm, OID_AUTO, kvm_size, CTLTYPE_LONG|CTLFLAG_RD, 0, 0, kvm_size, "IU", "Size of KVM"); static int kvm_free(SYSCTL_HANDLER_ARGS) { unsigned long kfree = VM_MAX_KERNEL_ADDRESS - kernel_vm_end; return (sysctl_handle_long(oidp, &kfree, 0, req)); } SYSCTL_PROC(_vm, OID_AUTO, kvm_free, CTLTYPE_LONG|CTLFLAG_RD, 0, 0, kvm_free, "IU", "Amount of KVM free"); /* * grow the number of kernel page table entries, if needed */ void pmap_growkernel(vm_offset_t addr) { vm_paddr_t ptppaddr; vm_page_t nkpg; pd_entry_t newpdir; mtx_assert(&kernel_map->system_mtx, MA_OWNED); addr = roundup2(addr, NBPDR); if (addr - 1 >= kernel_map->max_offset) addr = kernel_map->max_offset; while (kernel_vm_end < addr) { if (pdir_pde(PTD, kernel_vm_end)) { kernel_vm_end = (kernel_vm_end + NBPDR) & ~PDRMASK; if (kernel_vm_end - 1 >= kernel_map->max_offset) { kernel_vm_end = kernel_map->max_offset; break; } continue; } nkpg = vm_page_alloc(NULL, kernel_vm_end >> PDRSHIFT, VM_ALLOC_INTERRUPT | VM_ALLOC_NOOBJ | VM_ALLOC_WIRED | VM_ALLOC_ZERO); if (nkpg == NULL) panic("pmap_growkernel: no memory to grow kernel"); nkpt++; if ((nkpg->flags & PG_ZERO) == 0) pmap_zero_page(nkpg); ptppaddr = VM_PAGE_TO_PHYS(nkpg); newpdir = (pd_entry_t) (ptppaddr | PG_V | PG_RW | PG_A | PG_M); pdir_pde(KPTD, kernel_vm_end) = pgeflag | newpdir; pmap_kenter_pde(kernel_vm_end, newpdir); kernel_vm_end = (kernel_vm_end + NBPDR) & ~PDRMASK; if (kernel_vm_end - 1 >= kernel_map->max_offset) { kernel_vm_end = kernel_map->max_offset; break; } } } /*************************************************** * page management routines. ***************************************************/ CTASSERT(sizeof(struct pv_chunk) == PAGE_SIZE); CTASSERT(_NPCM == 11); CTASSERT(_NPCPV == 336); static __inline struct pv_chunk * pv_to_chunk(pv_entry_t pv) { return ((struct pv_chunk *)((uintptr_t)pv & ~(uintptr_t)PAGE_MASK)); } #define PV_PMAP(pv) (pv_to_chunk(pv)->pc_pmap) #define PC_FREE0_9 0xfffffffful /* Free values for index 0 through 9 */ #define PC_FREE10 0x0000fffful /* Free values for index 10 */ static const uint32_t pc_freemask[_NPCM] = { PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE0_9, PC_FREE10 }; SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_count, CTLFLAG_RD, &pv_entry_count, 0, "Current number of pv entries"); #ifdef PV_STATS static int pc_chunk_count, pc_chunk_allocs, pc_chunk_frees, pc_chunk_tryfail; SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_count, CTLFLAG_RD, &pc_chunk_count, 0, "Current number of pv entry chunks"); SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_allocs, CTLFLAG_RD, &pc_chunk_allocs, 0, "Current number of pv entry chunks allocated"); SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_frees, CTLFLAG_RD, &pc_chunk_frees, 0, "Current number of pv entry chunks frees"); SYSCTL_INT(_vm_pmap, OID_AUTO, pc_chunk_tryfail, CTLFLAG_RD, &pc_chunk_tryfail, 0, "Number of times tried to get a chunk page but failed."); static long pv_entry_frees, pv_entry_allocs; static int pv_entry_spare; SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_frees, CTLFLAG_RD, &pv_entry_frees, 0, "Current number of pv entry frees"); SYSCTL_LONG(_vm_pmap, OID_AUTO, pv_entry_allocs, CTLFLAG_RD, &pv_entry_allocs, 0, "Current number of pv entry allocs"); SYSCTL_INT(_vm_pmap, OID_AUTO, pv_entry_spare, CTLFLAG_RD, &pv_entry_spare, 0, "Current number of spare pv entries"); #endif /* * We are in a serious low memory condition. Resort to * drastic measures to free some pages so we can allocate * another pv entry chunk. */ static vm_page_t pmap_pv_reclaim(pmap_t locked_pmap) { struct pch newtail; struct pv_chunk *pc; struct md_page *pvh; pd_entry_t *pde; pmap_t pmap; pt_entry_t *pte, tpte; pv_entry_t pv; vm_offset_t va; vm_page_t m, m_pc; struct spglist free; uint32_t inuse; int bit, field, freed; PMAP_LOCK_ASSERT(locked_pmap, MA_OWNED); pmap = NULL; m_pc = NULL; SLIST_INIT(&free); TAILQ_INIT(&newtail); while ((pc = TAILQ_FIRST(&pv_chunks)) != NULL && (pv_vafree == 0 || SLIST_EMPTY(&free))) { TAILQ_REMOVE(&pv_chunks, pc, pc_lru); if (pmap != pc->pc_pmap) { if (pmap != NULL) { pmap_invalidate_all(pmap); if (pmap != locked_pmap) PMAP_UNLOCK(pmap); } pmap = pc->pc_pmap; /* Avoid deadlock and lock recursion. */ if (pmap > locked_pmap) PMAP_LOCK(pmap); else if (pmap != locked_pmap && !PMAP_TRYLOCK(pmap)) { pmap = NULL; TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); continue; } } /* * Destroy every non-wired, 4 KB page mapping in the chunk. */ freed = 0; for (field = 0; field < _NPCM; field++) { for (inuse = ~pc->pc_map[field] & pc_freemask[field]; inuse != 0; inuse &= ~(1UL << bit)) { bit = bsfl(inuse); pv = &pc->pc_pventry[field * 32 + bit]; va = pv->pv_va; pde = pmap_pde(pmap, va); if ((*pde & PG_PS) != 0) continue; pte = pmap_pte(pmap, va); tpte = *pte; if ((tpte & PG_W) == 0) tpte = pte_load_clear(pte); pmap_pte_release(pte); if ((tpte & PG_W) != 0) continue; KASSERT(tpte != 0, ("pmap_pv_reclaim: pmap %p va %x zero pte", pmap, va)); if ((tpte & PG_G) != 0) pmap_invalidate_page(pmap, va); m = PHYS_TO_VM_PAGE(tpte & PG_FRAME); if ((tpte & (PG_M | PG_RW)) == (PG_M | PG_RW)) vm_page_dirty(m); if ((tpte & PG_A) != 0) vm_page_aflag_set(m, PGA_REFERENCED); TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); if (TAILQ_EMPTY(&m->md.pv_list) && (m->flags & PG_FICTITIOUS) == 0) { pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); if (TAILQ_EMPTY(&pvh->pv_list)) { vm_page_aflag_clear(m, PGA_WRITEABLE); } } pc->pc_map[field] |= 1UL << bit; pmap_unuse_pt(pmap, va, &free); freed++; } } if (freed == 0) { TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); continue; } /* Every freed mapping is for a 4 KB page. */ pmap->pm_stats.resident_count -= freed; PV_STAT(pv_entry_frees += freed); PV_STAT(pv_entry_spare += freed); pv_entry_count -= freed; TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); for (field = 0; field < _NPCM; field++) if (pc->pc_map[field] != pc_freemask[field]) { TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); TAILQ_INSERT_TAIL(&newtail, pc, pc_lru); /* * One freed pv entry in locked_pmap is * sufficient. */ if (pmap == locked_pmap) goto out; break; } if (field == _NPCM) { PV_STAT(pv_entry_spare -= _NPCPV); PV_STAT(pc_chunk_count--); PV_STAT(pc_chunk_frees++); /* Entire chunk is free; return it. */ m_pc = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)pc)); pmap_qremove((vm_offset_t)pc, 1); pmap_ptelist_free(&pv_vafree, (vm_offset_t)pc); break; } } out: TAILQ_CONCAT(&pv_chunks, &newtail, pc_lru); if (pmap != NULL) { pmap_invalidate_all(pmap); if (pmap != locked_pmap) PMAP_UNLOCK(pmap); } if (m_pc == NULL && pv_vafree != 0 && SLIST_EMPTY(&free)) { m_pc = SLIST_FIRST(&free); SLIST_REMOVE_HEAD(&free, plinks.s.ss); /* Recycle a freed page table page. */ m_pc->wire_count = 1; atomic_add_int(&vm_cnt.v_wire_count, 1); } pmap_free_zero_pages(&free); return (m_pc); } /* * free the pv_entry back to the free list */ static void free_pv_entry(pmap_t pmap, pv_entry_t pv) { struct pv_chunk *pc; int idx, field, bit; rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_LOCK_ASSERT(pmap, MA_OWNED); PV_STAT(pv_entry_frees++); PV_STAT(pv_entry_spare++); pv_entry_count--; pc = pv_to_chunk(pv); idx = pv - &pc->pc_pventry[0]; field = idx / 32; bit = idx % 32; pc->pc_map[field] |= 1ul << bit; for (idx = 0; idx < _NPCM; idx++) if (pc->pc_map[idx] != pc_freemask[idx]) { /* * 98% of the time, pc is already at the head of the * list. If it isn't already, move it to the head. */ if (__predict_false(TAILQ_FIRST(&pmap->pm_pvchunk) != pc)) { TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); } return; } TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); free_pv_chunk(pc); } static void free_pv_chunk(struct pv_chunk *pc) { vm_page_t m; TAILQ_REMOVE(&pv_chunks, pc, pc_lru); PV_STAT(pv_entry_spare -= _NPCPV); PV_STAT(pc_chunk_count--); PV_STAT(pc_chunk_frees++); /* entire chunk is free, return it */ m = PHYS_TO_VM_PAGE(pmap_kextract((vm_offset_t)pc)); pmap_qremove((vm_offset_t)pc, 1); vm_page_unwire(m, PQ_INACTIVE); vm_page_free(m); pmap_ptelist_free(&pv_vafree, (vm_offset_t)pc); } /* * get a new pv_entry, allocating a block from the system * when needed. */ static pv_entry_t get_pv_entry(pmap_t pmap, boolean_t try) { static const struct timeval printinterval = { 60, 0 }; static struct timeval lastprint; int bit, field; pv_entry_t pv; struct pv_chunk *pc; vm_page_t m; rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_LOCK_ASSERT(pmap, MA_OWNED); PV_STAT(pv_entry_allocs++); pv_entry_count++; if (pv_entry_count > pv_entry_high_water) if (ratecheck(&lastprint, &printinterval)) printf("Approaching the limit on PV entries, consider " "increasing either the vm.pmap.shpgperproc or the " "vm.pmap.pv_entry_max tunable.\n"); retry: pc = TAILQ_FIRST(&pmap->pm_pvchunk); if (pc != NULL) { for (field = 0; field < _NPCM; field++) { if (pc->pc_map[field]) { bit = bsfl(pc->pc_map[field]); break; } } if (field < _NPCM) { pv = &pc->pc_pventry[field * 32 + bit]; pc->pc_map[field] &= ~(1ul << bit); /* If this was the last item, move it to tail */ for (field = 0; field < _NPCM; field++) if (pc->pc_map[field] != 0) { PV_STAT(pv_entry_spare--); return (pv); /* not full, return */ } TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); TAILQ_INSERT_TAIL(&pmap->pm_pvchunk, pc, pc_list); PV_STAT(pv_entry_spare--); return (pv); } } /* * Access to the ptelist "pv_vafree" is synchronized by the pvh * global lock. If "pv_vafree" is currently non-empty, it will * remain non-empty until pmap_ptelist_alloc() completes. */ if (pv_vafree == 0 || (m = vm_page_alloc(NULL, 0, VM_ALLOC_NORMAL | VM_ALLOC_NOOBJ | VM_ALLOC_WIRED)) == NULL) { if (try) { pv_entry_count--; PV_STAT(pc_chunk_tryfail++); return (NULL); } m = pmap_pv_reclaim(pmap); if (m == NULL) goto retry; } PV_STAT(pc_chunk_count++); PV_STAT(pc_chunk_allocs++); pc = (struct pv_chunk *)pmap_ptelist_alloc(&pv_vafree); pmap_qenter((vm_offset_t)pc, &m, 1); pc->pc_pmap = pmap; pc->pc_map[0] = pc_freemask[0] & ~1ul; /* preallocated bit 0 */ for (field = 1; field < _NPCM; field++) pc->pc_map[field] = pc_freemask[field]; TAILQ_INSERT_TAIL(&pv_chunks, pc, pc_lru); pv = &pc->pc_pventry[0]; TAILQ_INSERT_HEAD(&pmap->pm_pvchunk, pc, pc_list); PV_STAT(pv_entry_spare += _NPCPV - 1); return (pv); } static __inline pv_entry_t pmap_pvh_remove(struct md_page *pvh, pmap_t pmap, vm_offset_t va) { pv_entry_t pv; rw_assert(&pvh_global_lock, RA_WLOCKED); TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { if (pmap == PV_PMAP(pv) && va == pv->pv_va) { TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); break; } } return (pv); } static void pmap_pv_demote_pde(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) { struct md_page *pvh; pv_entry_t pv; vm_offset_t va_last; vm_page_t m; rw_assert(&pvh_global_lock, RA_WLOCKED); KASSERT((pa & PDRMASK) == 0, ("pmap_pv_demote_pde: pa is not 4mpage aligned")); /* * Transfer the 4mpage's pv entry for this mapping to the first * page's pv list. */ pvh = pa_to_pvh(pa); va = trunc_4mpage(va); pv = pmap_pvh_remove(pvh, pmap, va); KASSERT(pv != NULL, ("pmap_pv_demote_pde: pv not found")); m = PHYS_TO_VM_PAGE(pa); TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); /* Instantiate the remaining NPTEPG - 1 pv entries. */ va_last = va + NBPDR - PAGE_SIZE; do { m++; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_pv_demote_pde: page %p is not managed", m)); va += PAGE_SIZE; pmap_insert_entry(pmap, va, m); } while (va < va_last); } static void pmap_pv_promote_pde(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) { struct md_page *pvh; pv_entry_t pv; vm_offset_t va_last; vm_page_t m; rw_assert(&pvh_global_lock, RA_WLOCKED); KASSERT((pa & PDRMASK) == 0, ("pmap_pv_promote_pde: pa is not 4mpage aligned")); /* * Transfer the first page's pv entry for this mapping to the * 4mpage's pv list. Aside from avoiding the cost of a call * to get_pv_entry(), a transfer avoids the possibility that * get_pv_entry() calls pmap_collect() and that pmap_collect() * removes one of the mappings that is being promoted. */ m = PHYS_TO_VM_PAGE(pa); va = trunc_4mpage(va); pv = pmap_pvh_remove(&m->md, pmap, va); KASSERT(pv != NULL, ("pmap_pv_promote_pde: pv not found")); pvh = pa_to_pvh(pa); TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); /* Free the remaining NPTEPG - 1 pv entries. */ va_last = va + NBPDR - PAGE_SIZE; do { m++; va += PAGE_SIZE; pmap_pvh_free(&m->md, pmap, va); } while (va < va_last); } static void pmap_pvh_free(struct md_page *pvh, pmap_t pmap, vm_offset_t va) { pv_entry_t pv; pv = pmap_pvh_remove(pvh, pmap, va); KASSERT(pv != NULL, ("pmap_pvh_free: pv not found")); free_pv_entry(pmap, pv); } static void pmap_remove_entry(pmap_t pmap, vm_page_t m, vm_offset_t va) { struct md_page *pvh; rw_assert(&pvh_global_lock, RA_WLOCKED); pmap_pvh_free(&m->md, pmap, va); if (TAILQ_EMPTY(&m->md.pv_list) && (m->flags & PG_FICTITIOUS) == 0) { pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); if (TAILQ_EMPTY(&pvh->pv_list)) vm_page_aflag_clear(m, PGA_WRITEABLE); } } /* * Create a pv entry for page at pa for * (pmap, va). */ static void pmap_insert_entry(pmap_t pmap, vm_offset_t va, vm_page_t m) { pv_entry_t pv; rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_LOCK_ASSERT(pmap, MA_OWNED); pv = get_pv_entry(pmap, FALSE); pv->pv_va = va; TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); } /* * Conditionally create a pv entry. */ static boolean_t pmap_try_insert_pv_entry(pmap_t pmap, vm_offset_t va, vm_page_t m) { pv_entry_t pv; rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_LOCK_ASSERT(pmap, MA_OWNED); if (pv_entry_count < pv_entry_high_water && (pv = get_pv_entry(pmap, TRUE)) != NULL) { pv->pv_va = va; TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); return (TRUE); } else return (FALSE); } /* * Create the pv entries for each of the pages within a superpage. */ static boolean_t pmap_pv_insert_pde(pmap_t pmap, vm_offset_t va, vm_paddr_t pa) { struct md_page *pvh; pv_entry_t pv; rw_assert(&pvh_global_lock, RA_WLOCKED); if (pv_entry_count < pv_entry_high_water && (pv = get_pv_entry(pmap, TRUE)) != NULL) { pv->pv_va = va; pvh = pa_to_pvh(pa); TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); return (TRUE); } else return (FALSE); } /* * Fills a page table page with mappings to consecutive physical pages. */ static void pmap_fill_ptp(pt_entry_t *firstpte, pt_entry_t newpte) { pt_entry_t *pte; for (pte = firstpte; pte < firstpte + NPTEPG; pte++) { *pte = newpte; newpte += PAGE_SIZE; } } /* * Tries to demote a 2- or 4MB page mapping. If demotion fails, the * 2- or 4MB page mapping is invalidated. */ static boolean_t pmap_demote_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t va) { pd_entry_t newpde, oldpde; pt_entry_t *firstpte, newpte; vm_paddr_t mptepa; vm_page_t mpte; struct spglist free; PMAP_LOCK_ASSERT(pmap, MA_OWNED); oldpde = *pde; KASSERT((oldpde & (PG_PS | PG_V)) == (PG_PS | PG_V), ("pmap_demote_pde: oldpde is missing PG_PS and/or PG_V")); if ((oldpde & PG_A) != 0 && (mpte = pmap_lookup_pt_page(pmap, va)) != NULL) pmap_remove_pt_page(pmap, mpte); else { KASSERT((oldpde & PG_W) == 0, ("pmap_demote_pde: page table page for a wired mapping" " is missing")); /* * Invalidate the 2- or 4MB page mapping and return * "failure" if the mapping was never accessed or the * allocation of the new page table page fails. */ if ((oldpde & PG_A) == 0 || (mpte = vm_page_alloc(NULL, va >> PDRSHIFT, VM_ALLOC_NOOBJ | VM_ALLOC_NORMAL | VM_ALLOC_WIRED)) == NULL) { SLIST_INIT(&free); pmap_remove_pde(pmap, pde, trunc_4mpage(va), &free); pmap_invalidate_page(pmap, trunc_4mpage(va)); pmap_free_zero_pages(&free); CTR2(KTR_PMAP, "pmap_demote_pde: failure for va %#x" " in pmap %p", va, pmap); return (FALSE); } if (va < VM_MAXUSER_ADDRESS) pmap->pm_stats.resident_count++; } mptepa = VM_PAGE_TO_PHYS(mpte); /* * If the page mapping is in the kernel's address space, then the * KPTmap can provide access to the page table page. Otherwise, * temporarily map the page table page (mpte) into the kernel's * address space at either PADDR1 or PADDR2. */ if (va >= KERNBASE) firstpte = &KPTmap[i386_btop(trunc_4mpage(va))]; else if (curthread->td_pinned > 0 && rw_wowned(&pvh_global_lock)) { if ((*PMAP1 & PG_FRAME) != mptepa) { *PMAP1 = mptepa | PG_RW | PG_V | PG_A | PG_M; #ifdef SMP PMAP1cpu = PCPU_GET(cpuid); #endif invlcaddr(PADDR1); PMAP1changed++; } else #ifdef SMP if (PMAP1cpu != PCPU_GET(cpuid)) { PMAP1cpu = PCPU_GET(cpuid); invlcaddr(PADDR1); PMAP1changedcpu++; } else #endif PMAP1unchanged++; firstpte = PADDR1; } else { mtx_lock(&PMAP2mutex); if ((*PMAP2 & PG_FRAME) != mptepa) { *PMAP2 = mptepa | PG_RW | PG_V | PG_A | PG_M; pmap_invalidate_page(kernel_pmap, (vm_offset_t)PADDR2); } firstpte = PADDR2; } newpde = mptepa | PG_M | PG_A | (oldpde & PG_U) | PG_RW | PG_V; KASSERT((oldpde & PG_A) != 0, ("pmap_demote_pde: oldpde is missing PG_A")); KASSERT((oldpde & (PG_M | PG_RW)) != PG_RW, ("pmap_demote_pde: oldpde is missing PG_M")); newpte = oldpde & ~PG_PS; if ((newpte & PG_PDE_PAT) != 0) newpte ^= PG_PDE_PAT | PG_PTE_PAT; /* * If the page table page is new, initialize it. */ if (mpte->wire_count == 1) { mpte->wire_count = NPTEPG; pmap_fill_ptp(firstpte, newpte); } KASSERT((*firstpte & PG_FRAME) == (newpte & PG_FRAME), ("pmap_demote_pde: firstpte and newpte map different physical" " addresses")); /* * If the mapping has changed attributes, update the page table * entries. */ if ((*firstpte & PG_PTE_PROMOTE) != (newpte & PG_PTE_PROMOTE)) pmap_fill_ptp(firstpte, newpte); /* * Demote the mapping. This pmap is locked. The old PDE has * PG_A set. If the old PDE has PG_RW set, it also has PG_M * set. Thus, there is no danger of a race with another * processor changing the setting of PG_A and/or PG_M between * the read above and the store below. */ if (workaround_erratum383) pmap_update_pde(pmap, va, pde, newpde); else if (pmap == kernel_pmap) pmap_kenter_pde(va, newpde); else pde_store(pde, newpde); if (firstpte == PADDR2) mtx_unlock(&PMAP2mutex); /* * Invalidate the recursive mapping of the page table page. */ pmap_invalidate_page(pmap, (vm_offset_t)vtopte(va)); /* * Demote the pv entry. This depends on the earlier demotion * of the mapping. Specifically, the (re)creation of a per- * page pv entry might trigger the execution of pmap_collect(), * which might reclaim a newly (re)created per-page pv entry * and destroy the associated mapping. In order to destroy * the mapping, the PDE must have already changed from mapping * the 2mpage to referencing the page table page. */ if ((oldpde & PG_MANAGED) != 0) pmap_pv_demote_pde(pmap, va, oldpde & PG_PS_FRAME); pmap_pde_demotions++; CTR2(KTR_PMAP, "pmap_demote_pde: success for va %#x" " in pmap %p", va, pmap); return (TRUE); } /* * Removes a 2- or 4MB page mapping from the kernel pmap. */ static void pmap_remove_kernel_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t va) { pd_entry_t newpde; vm_paddr_t mptepa; vm_page_t mpte; PMAP_LOCK_ASSERT(pmap, MA_OWNED); mpte = pmap_lookup_pt_page(pmap, va); if (mpte == NULL) panic("pmap_remove_kernel_pde: Missing pt page."); pmap_remove_pt_page(pmap, mpte); mptepa = VM_PAGE_TO_PHYS(mpte); newpde = mptepa | PG_M | PG_A | PG_RW | PG_V; /* * Initialize the page table page. */ pagezero((void *)&KPTmap[i386_btop(trunc_4mpage(va))]); /* * Remove the mapping. */ if (workaround_erratum383) pmap_update_pde(pmap, va, pde, newpde); else pmap_kenter_pde(va, newpde); /* * Invalidate the recursive mapping of the page table page. */ pmap_invalidate_page(pmap, (vm_offset_t)vtopte(va)); } /* * pmap_remove_pde: do the things to unmap a superpage in a process */ static void pmap_remove_pde(pmap_t pmap, pd_entry_t *pdq, vm_offset_t sva, struct spglist *free) { struct md_page *pvh; pd_entry_t oldpde; vm_offset_t eva, va; vm_page_t m, mpte; PMAP_LOCK_ASSERT(pmap, MA_OWNED); KASSERT((sva & PDRMASK) == 0, ("pmap_remove_pde: sva is not 4mpage aligned")); oldpde = pte_load_clear(pdq); if (oldpde & PG_W) pmap->pm_stats.wired_count -= NBPDR / PAGE_SIZE; /* * Machines that don't support invlpg, also don't support * PG_G. */ if (oldpde & PG_G) pmap_invalidate_page(kernel_pmap, sva); pmap->pm_stats.resident_count -= NBPDR / PAGE_SIZE; if (oldpde & PG_MANAGED) { pvh = pa_to_pvh(oldpde & PG_PS_FRAME); pmap_pvh_free(pvh, pmap, sva); eva = sva + NBPDR; for (va = sva, m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME); va < eva; va += PAGE_SIZE, m++) { if ((oldpde & (PG_M | PG_RW)) == (PG_M | PG_RW)) vm_page_dirty(m); if (oldpde & PG_A) vm_page_aflag_set(m, PGA_REFERENCED); if (TAILQ_EMPTY(&m->md.pv_list) && TAILQ_EMPTY(&pvh->pv_list)) vm_page_aflag_clear(m, PGA_WRITEABLE); } } if (pmap == kernel_pmap) { pmap_remove_kernel_pde(pmap, pdq, sva); } else { mpte = pmap_lookup_pt_page(pmap, sva); if (mpte != NULL) { pmap_remove_pt_page(pmap, mpte); pmap->pm_stats.resident_count--; KASSERT(mpte->wire_count == NPTEPG, ("pmap_remove_pde: pte page wire count error")); mpte->wire_count = 0; pmap_add_delayed_free_list(mpte, free, FALSE); atomic_subtract_int(&vm_cnt.v_wire_count, 1); } } } /* * pmap_remove_pte: do the things to unmap a page in a process */ static int pmap_remove_pte(pmap_t pmap, pt_entry_t *ptq, vm_offset_t va, struct spglist *free) { pt_entry_t oldpte; vm_page_t m; rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_LOCK_ASSERT(pmap, MA_OWNED); oldpte = pte_load_clear(ptq); KASSERT(oldpte != 0, ("pmap_remove_pte: pmap %p va %x zero pte", pmap, va)); if (oldpte & PG_W) pmap->pm_stats.wired_count -= 1; /* * Machines that don't support invlpg, also don't support * PG_G. */ if (oldpte & PG_G) pmap_invalidate_page(kernel_pmap, va); pmap->pm_stats.resident_count -= 1; if (oldpte & PG_MANAGED) { m = PHYS_TO_VM_PAGE(oldpte & PG_FRAME); if ((oldpte & (PG_M | PG_RW)) == (PG_M | PG_RW)) vm_page_dirty(m); if (oldpte & PG_A) vm_page_aflag_set(m, PGA_REFERENCED); pmap_remove_entry(pmap, m, va); } return (pmap_unuse_pt(pmap, va, free)); } /* * Remove a single page from a process address space */ static void pmap_remove_page(pmap_t pmap, vm_offset_t va, struct spglist *free) { pt_entry_t *pte; rw_assert(&pvh_global_lock, RA_WLOCKED); KASSERT(curthread->td_pinned > 0, ("curthread not pinned")); PMAP_LOCK_ASSERT(pmap, MA_OWNED); if ((pte = pmap_pte_quick(pmap, va)) == NULL || *pte == 0) return; pmap_remove_pte(pmap, pte, va, free); pmap_invalidate_page(pmap, va); } /* * Remove the given range of addresses from the specified map. * * It is assumed that the start and end are properly * rounded to the page size. */ void pmap_remove(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { vm_offset_t pdnxt; pd_entry_t ptpaddr; pt_entry_t *pte; struct spglist free; int anyvalid; /* * Perform an unsynchronized read. This is, however, safe. */ if (pmap->pm_stats.resident_count == 0) return; anyvalid = 0; SLIST_INIT(&free); rw_wlock(&pvh_global_lock); sched_pin(); PMAP_LOCK(pmap); /* * special handling of removing one page. a very * common operation and easy to short circuit some * code. */ if ((sva + PAGE_SIZE == eva) && ((pmap->pm_pdir[(sva >> PDRSHIFT)] & PG_PS) == 0)) { pmap_remove_page(pmap, sva, &free); goto out; } for (; sva < eva; sva = pdnxt) { u_int pdirindex; /* * Calculate index for next page table. */ pdnxt = (sva + NBPDR) & ~PDRMASK; if (pdnxt < sva) pdnxt = eva; if (pmap->pm_stats.resident_count == 0) break; pdirindex = sva >> PDRSHIFT; ptpaddr = pmap->pm_pdir[pdirindex]; /* * Weed out invalid mappings. Note: we assume that the page * directory table is always allocated, and in kernel virtual. */ if (ptpaddr == 0) continue; /* * Check for large page. */ if ((ptpaddr & PG_PS) != 0) { /* * Are we removing the entire large page? If not, * demote the mapping and fall through. */ if (sva + NBPDR == pdnxt && eva >= pdnxt) { /* * The TLB entry for a PG_G mapping is * invalidated by pmap_remove_pde(). */ if ((ptpaddr & PG_G) == 0) anyvalid = 1; pmap_remove_pde(pmap, &pmap->pm_pdir[pdirindex], sva, &free); continue; } else if (!pmap_demote_pde(pmap, &pmap->pm_pdir[pdirindex], sva)) { /* The large page mapping was destroyed. */ continue; } } /* * Limit our scan to either the end of the va represented * by the current page table page, or to the end of the * range being removed. */ if (pdnxt > eva) pdnxt = eva; for (pte = pmap_pte_quick(pmap, sva); sva != pdnxt; pte++, sva += PAGE_SIZE) { if (*pte == 0) continue; /* * The TLB entry for a PG_G mapping is invalidated * by pmap_remove_pte(). */ if ((*pte & PG_G) == 0) anyvalid = 1; if (pmap_remove_pte(pmap, pte, sva, &free)) break; } } out: sched_unpin(); if (anyvalid) pmap_invalidate_all(pmap); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); pmap_free_zero_pages(&free); } /* * Routine: pmap_remove_all * Function: * Removes this physical page from * all physical maps in which it resides. * Reflects back modify bits to the pager. * * Notes: * Original versions of this routine were very * inefficient because they iteratively called * pmap_remove (slow...) */ void pmap_remove_all(vm_page_t m) { struct md_page *pvh; pv_entry_t pv; pmap_t pmap; pt_entry_t *pte, tpte; pd_entry_t *pde; vm_offset_t va; struct spglist free; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_remove_all: page %p is not managed", m)); SLIST_INIT(&free); rw_wlock(&pvh_global_lock); sched_pin(); if ((m->flags & PG_FICTITIOUS) != 0) goto small_mappings; pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); while ((pv = TAILQ_FIRST(&pvh->pv_list)) != NULL) { va = pv->pv_va; pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pde = pmap_pde(pmap, va); (void)pmap_demote_pde(pmap, pde, va); PMAP_UNLOCK(pmap); } small_mappings: while ((pv = TAILQ_FIRST(&m->md.pv_list)) != NULL) { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pmap->pm_stats.resident_count--; pde = pmap_pde(pmap, pv->pv_va); KASSERT((*pde & PG_PS) == 0, ("pmap_remove_all: found" " a 4mpage in page %p's pv list", m)); pte = pmap_pte_quick(pmap, pv->pv_va); tpte = pte_load_clear(pte); KASSERT(tpte != 0, ("pmap_remove_all: pmap %p va %x zero pte", pmap, pv->pv_va)); if (tpte & PG_W) pmap->pm_stats.wired_count--; if (tpte & PG_A) vm_page_aflag_set(m, PGA_REFERENCED); /* * Update the vm_page_t clean and reference bits. */ if ((tpte & (PG_M | PG_RW)) == (PG_M | PG_RW)) vm_page_dirty(m); pmap_unuse_pt(pmap, pv->pv_va, &free); pmap_invalidate_page(pmap, pv->pv_va); TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); free_pv_entry(pmap, pv); PMAP_UNLOCK(pmap); } vm_page_aflag_clear(m, PGA_WRITEABLE); sched_unpin(); rw_wunlock(&pvh_global_lock); pmap_free_zero_pages(&free); } /* * pmap_protect_pde: do the things to protect a 4mpage in a process */ static boolean_t pmap_protect_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t sva, vm_prot_t prot) { pd_entry_t newpde, oldpde; vm_offset_t eva, va; vm_page_t m; boolean_t anychanged; PMAP_LOCK_ASSERT(pmap, MA_OWNED); KASSERT((sva & PDRMASK) == 0, ("pmap_protect_pde: sva is not 4mpage aligned")); anychanged = FALSE; retry: oldpde = newpde = *pde; if (oldpde & PG_MANAGED) { eva = sva + NBPDR; for (va = sva, m = PHYS_TO_VM_PAGE(oldpde & PG_PS_FRAME); va < eva; va += PAGE_SIZE, m++) if ((oldpde & (PG_M | PG_RW)) == (PG_M | PG_RW)) vm_page_dirty(m); } if ((prot & VM_PROT_WRITE) == 0) newpde &= ~(PG_RW | PG_M); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((prot & VM_PROT_EXECUTE) == 0) newpde |= pg_nx; #endif if (newpde != oldpde) { if (!pde_cmpset(pde, oldpde, newpde)) goto retry; if (oldpde & PG_G) pmap_invalidate_page(pmap, sva); else anychanged = TRUE; } return (anychanged); } /* * Set the physical protection on the * specified range of this map as requested. */ void pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot) { vm_offset_t pdnxt; pd_entry_t ptpaddr; pt_entry_t *pte; boolean_t anychanged, pv_lists_locked; KASSERT((prot & ~VM_PROT_ALL) == 0, ("invalid prot %x", prot)); if (prot == VM_PROT_NONE) { pmap_remove(pmap, sva, eva); return; } -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((prot & (VM_PROT_WRITE|VM_PROT_EXECUTE)) == (VM_PROT_WRITE|VM_PROT_EXECUTE)) return; #else if (prot & VM_PROT_WRITE) return; #endif if (pmap_is_current(pmap)) pv_lists_locked = FALSE; else { pv_lists_locked = TRUE; resume: rw_wlock(&pvh_global_lock); sched_pin(); } anychanged = FALSE; PMAP_LOCK(pmap); for (; sva < eva; sva = pdnxt) { pt_entry_t obits, pbits; u_int pdirindex; pdnxt = (sva + NBPDR) & ~PDRMASK; if (pdnxt < sva) pdnxt = eva; pdirindex = sva >> PDRSHIFT; ptpaddr = pmap->pm_pdir[pdirindex]; /* * Weed out invalid mappings. Note: we assume that the page * directory table is always allocated, and in kernel virtual. */ if (ptpaddr == 0) continue; /* * Check for large page. */ if ((ptpaddr & PG_PS) != 0) { /* * Are we protecting the entire large page? If not, * demote the mapping and fall through. */ if (sva + NBPDR == pdnxt && eva >= pdnxt) { /* * The TLB entry for a PG_G mapping is * invalidated by pmap_protect_pde(). */ if (pmap_protect_pde(pmap, &pmap->pm_pdir[pdirindex], sva, prot)) anychanged = TRUE; continue; } else { if (!pv_lists_locked) { pv_lists_locked = TRUE; if (!rw_try_wlock(&pvh_global_lock)) { if (anychanged) pmap_invalidate_all( pmap); PMAP_UNLOCK(pmap); goto resume; } sched_pin(); } if (!pmap_demote_pde(pmap, &pmap->pm_pdir[pdirindex], sva)) { /* * The large page mapping was * destroyed. */ continue; } } } if (pdnxt > eva) pdnxt = eva; for (pte = pmap_pte_quick(pmap, sva); sva != pdnxt; pte++, sva += PAGE_SIZE) { vm_page_t m; retry: /* * Regardless of whether a pte is 32 or 64 bits in * size, PG_RW, PG_A, and PG_M are among the least * significant 32 bits. */ obits = pbits = *pte; if ((pbits & PG_V) == 0) continue; if ((prot & VM_PROT_WRITE) == 0) { if ((pbits & (PG_MANAGED | PG_M | PG_RW)) == (PG_MANAGED | PG_M | PG_RW)) { m = PHYS_TO_VM_PAGE(pbits & PG_FRAME); vm_page_dirty(m); } pbits &= ~(PG_RW | PG_M); } -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((prot & VM_PROT_EXECUTE) == 0) pbits |= pg_nx; #endif if (pbits != obits) { -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if (!atomic_cmpset_64(pte, obits, pbits)) goto retry; #else if (!atomic_cmpset_int((u_int *)pte, obits, pbits)) goto retry; #endif if (obits & PG_G) pmap_invalidate_page(pmap, sva); else anychanged = TRUE; } } } if (anychanged) pmap_invalidate_all(pmap); if (pv_lists_locked) { sched_unpin(); rw_wunlock(&pvh_global_lock); } PMAP_UNLOCK(pmap); } /* * Tries to promote the 512 or 1024, contiguous 4KB page mappings that are * within a single page table page (PTP) to a single 2- or 4MB page mapping. * For promotion to occur, two conditions must be met: (1) the 4KB page * mappings must map aligned, contiguous physical memory and (2) the 4KB page * mappings must have identical characteristics. * * Managed (PG_MANAGED) mappings within the kernel address space are not * promoted. The reason is that kernel PDEs are replicated in each pmap but * pmap_clear_ptes() and pmap_ts_referenced() only read the PDE from the kernel * pmap. */ static void pmap_promote_pde(pmap_t pmap, pd_entry_t *pde, vm_offset_t va) { pd_entry_t newpde; pt_entry_t *firstpte, oldpte, pa, *pte; vm_offset_t oldpteva; vm_page_t mpte; PMAP_LOCK_ASSERT(pmap, MA_OWNED); /* * Examine the first PTE in the specified PTP. Abort if this PTE is * either invalid, unused, or does not map the first 4KB physical page * within a 2- or 4MB page. */ firstpte = pmap_pte_quick(pmap, trunc_4mpage(va)); setpde: newpde = *firstpte; if ((newpde & ((PG_FRAME & PDRMASK) | PG_A | PG_V)) != (PG_A | PG_V)) { pmap_pde_p_failures++; CTR2(KTR_PMAP, "pmap_promote_pde: failure for va %#x" " in pmap %p", va, pmap); return; } if ((*firstpte & PG_MANAGED) != 0 && pmap == kernel_pmap) { pmap_pde_p_failures++; CTR2(KTR_PMAP, "pmap_promote_pde: failure for va %#x" " in pmap %p", va, pmap); return; } if ((newpde & (PG_M | PG_RW)) == PG_RW) { /* * When PG_M is already clear, PG_RW can be cleared without * a TLB invalidation. */ if (!atomic_cmpset_int((u_int *)firstpte, newpde, newpde & ~PG_RW)) goto setpde; newpde &= ~PG_RW; } /* * Examine each of the other PTEs in the specified PTP. Abort if this * PTE maps an unexpected 4KB physical page or does not have identical * characteristics to the first PTE. */ pa = (newpde & (PG_PS_FRAME | PG_A | PG_V)) + NBPDR - PAGE_SIZE; for (pte = firstpte + NPTEPG - 1; pte > firstpte; pte--) { setpte: oldpte = *pte; if ((oldpte & (PG_FRAME | PG_A | PG_V)) != pa) { pmap_pde_p_failures++; CTR2(KTR_PMAP, "pmap_promote_pde: failure for va %#x" " in pmap %p", va, pmap); return; } if ((oldpte & (PG_M | PG_RW)) == PG_RW) { /* * When PG_M is already clear, PG_RW can be cleared * without a TLB invalidation. */ if (!atomic_cmpset_int((u_int *)pte, oldpte, oldpte & ~PG_RW)) goto setpte; oldpte &= ~PG_RW; oldpteva = (oldpte & PG_FRAME & PDRMASK) | (va & ~PDRMASK); CTR2(KTR_PMAP, "pmap_promote_pde: protect for va %#x" " in pmap %p", oldpteva, pmap); } if ((oldpte & PG_PTE_PROMOTE) != (newpde & PG_PTE_PROMOTE)) { pmap_pde_p_failures++; CTR2(KTR_PMAP, "pmap_promote_pde: failure for va %#x" " in pmap %p", va, pmap); return; } pa -= PAGE_SIZE; } /* * Save the page table page in its current state until the PDE * mapping the superpage is demoted by pmap_demote_pde() or * destroyed by pmap_remove_pde(). */ mpte = PHYS_TO_VM_PAGE(*pde & PG_FRAME); KASSERT(mpte >= vm_page_array && mpte < &vm_page_array[vm_page_array_size], ("pmap_promote_pde: page table page is out of range")); KASSERT(mpte->pindex == va >> PDRSHIFT, ("pmap_promote_pde: page table page's pindex is wrong")); if (pmap_insert_pt_page(pmap, mpte)) { pmap_pde_p_failures++; CTR2(KTR_PMAP, "pmap_promote_pde: failure for va %#x in pmap %p", va, pmap); return; } /* * Promote the pv entries. */ if ((newpde & PG_MANAGED) != 0) pmap_pv_promote_pde(pmap, va, newpde & PG_PS_FRAME); /* * Propagate the PAT index to its proper position. */ if ((newpde & PG_PTE_PAT) != 0) newpde ^= PG_PDE_PAT | PG_PTE_PAT; /* * Map the superpage. */ if (workaround_erratum383) pmap_update_pde(pmap, va, pde, PG_PS | newpde); else if (pmap == kernel_pmap) pmap_kenter_pde(va, PG_PS | newpde); else pde_store(pde, PG_PS | newpde); pmap_pde_promotions++; CTR2(KTR_PMAP, "pmap_promote_pde: success for va %#x" " in pmap %p", va, pmap); } /* * Insert the given physical page (p) at * the specified virtual address (v) in the * target physical map with the protection requested. * * If specified, the page will be wired down, meaning * that the related pte can not be reclaimed. * * NB: This is the only routine which MAY NOT lazy-evaluate * or lose information. That is, this routine must actually * insert this page into the given map NOW. */ int pmap_enter(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, u_int flags, int8_t psind) { pd_entry_t *pde; pt_entry_t *pte; pt_entry_t newpte, origpte; pv_entry_t pv; vm_paddr_t opa, pa; vm_page_t mpte, om; boolean_t invlva, wired; va = trunc_page(va); mpte = NULL; wired = (flags & PMAP_ENTER_WIRED) != 0; KASSERT(va <= VM_MAX_KERNEL_ADDRESS, ("pmap_enter: toobig")); KASSERT(va < UPT_MIN_ADDRESS || va >= UPT_MAX_ADDRESS, ("pmap_enter: invalid to pmap_enter page table pages (va: 0x%x)", va)); if ((m->oflags & VPO_UNMANAGED) == 0 && !vm_page_xbusied(m)) VM_OBJECT_ASSERT_LOCKED(m->object); rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); sched_pin(); /* * In the case that a page table page is not * resident, we are creating it here. */ if (va < VM_MAXUSER_ADDRESS) { mpte = pmap_allocpte(pmap, va, flags); if (mpte == NULL) { KASSERT((flags & PMAP_ENTER_NOSLEEP) != 0, ("pmap_allocpte failed with sleep allowed")); sched_unpin(); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); return (KERN_RESOURCE_SHORTAGE); } } pde = pmap_pde(pmap, va); if ((*pde & PG_PS) != 0) panic("pmap_enter: attempted pmap_enter on 4MB page"); pte = pmap_pte_quick(pmap, va); /* * Page Directory table entry not valid, we need a new PT page */ if (pte == NULL) { panic("pmap_enter: invalid page directory pdir=%#jx, va=%#x", (uintmax_t)pmap->pm_pdir[PTDPTDI], va); } pa = VM_PAGE_TO_PHYS(m); om = NULL; origpte = *pte; opa = origpte & PG_FRAME; /* * Mapping has not changed, must be protection or wiring change. */ if (origpte && (opa == pa)) { /* * Wiring change, just update stats. We don't worry about * wiring PT pages as they remain resident as long as there * are valid mappings in them. Hence, if a user page is wired, * the PT page will be also. */ if (wired && ((origpte & PG_W) == 0)) pmap->pm_stats.wired_count++; else if (!wired && (origpte & PG_W)) pmap->pm_stats.wired_count--; /* * Remove extra pte reference */ if (mpte) mpte->wire_count--; if (origpte & PG_MANAGED) { om = m; pa |= PG_MANAGED; } goto validate; } pv = NULL; /* * Mapping has changed, invalidate old range and fall through to * handle validating new mapping. */ if (opa) { if (origpte & PG_W) pmap->pm_stats.wired_count--; if (origpte & PG_MANAGED) { om = PHYS_TO_VM_PAGE(opa); pv = pmap_pvh_remove(&om->md, pmap, va); } if (mpte != NULL) { mpte->wire_count--; KASSERT(mpte->wire_count > 0, ("pmap_enter: missing reference to page table page," " va: 0x%x", va)); } } else pmap->pm_stats.resident_count++; /* * Enter on the PV list if part of our managed memory. */ if ((m->oflags & VPO_UNMANAGED) == 0) { KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva, ("pmap_enter: managed mapping within the clean submap")); if (pv == NULL) pv = get_pv_entry(pmap, FALSE); pv->pv_va = va; TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); pa |= PG_MANAGED; } else if (pv != NULL) free_pv_entry(pmap, pv); /* * Increment counters */ if (wired) pmap->pm_stats.wired_count++; validate: /* * Now validate mapping with desired protection/wiring. */ newpte = (pt_entry_t)(pa | pmap_cache_bits(m->md.pat_mode, 0) | PG_V); if ((prot & VM_PROT_WRITE) != 0) { newpte |= PG_RW; if ((newpte & PG_MANAGED) != 0) vm_page_aflag_set(m, PGA_WRITEABLE); } -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((prot & VM_PROT_EXECUTE) == 0) newpte |= pg_nx; #endif if (wired) newpte |= PG_W; if (va < VM_MAXUSER_ADDRESS) newpte |= PG_U; if (pmap == kernel_pmap) newpte |= pgeflag; /* * if the mapping or permission bits are different, we need * to update the pte. */ if ((origpte & ~(PG_M|PG_A)) != newpte) { newpte |= PG_A; if ((flags & VM_PROT_WRITE) != 0) newpte |= PG_M; if (origpte & PG_V) { invlva = FALSE; origpte = pte_load_store(pte, newpte); if (origpte & PG_A) { if (origpte & PG_MANAGED) vm_page_aflag_set(om, PGA_REFERENCED); if (opa != VM_PAGE_TO_PHYS(m)) invlva = TRUE; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((origpte & PG_NX) == 0 && (newpte & PG_NX) != 0) invlva = TRUE; #endif } if ((origpte & (PG_M | PG_RW)) == (PG_M | PG_RW)) { if ((origpte & PG_MANAGED) != 0) vm_page_dirty(om); if ((prot & VM_PROT_WRITE) == 0) invlva = TRUE; } if ((origpte & PG_MANAGED) != 0 && TAILQ_EMPTY(&om->md.pv_list) && ((om->flags & PG_FICTITIOUS) != 0 || TAILQ_EMPTY(&pa_to_pvh(opa)->pv_list))) vm_page_aflag_clear(om, PGA_WRITEABLE); if (invlva) pmap_invalidate_page(pmap, va); } else pte_store(pte, newpte); } /* * If both the page table page and the reservation are fully * populated, then attempt promotion. */ if ((mpte == NULL || mpte->wire_count == NPTEPG) && pg_ps_enabled && (m->flags & PG_FICTITIOUS) == 0 && vm_reserv_level_iffullpop(m) == 0) pmap_promote_pde(pmap, pde, va); sched_unpin(); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); return (KERN_SUCCESS); } /* * Tries to create a 2- or 4MB page mapping. Returns TRUE if successful and * FALSE otherwise. Fails if (1) a page table page cannot be allocated without * blocking, (2) a mapping already exists at the specified virtual address, or * (3) a pv entry cannot be allocated without reclaiming another pv entry. */ static boolean_t pmap_enter_pde(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) { pd_entry_t *pde, newpde; rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_LOCK_ASSERT(pmap, MA_OWNED); pde = pmap_pde(pmap, va); if (*pde != 0) { CTR2(KTR_PMAP, "pmap_enter_pde: failure for va %#lx" " in pmap %p", va, pmap); return (FALSE); } newpde = VM_PAGE_TO_PHYS(m) | pmap_cache_bits(m->md.pat_mode, 1) | PG_PS | PG_V; if ((m->oflags & VPO_UNMANAGED) == 0) { newpde |= PG_MANAGED; /* * Abort this mapping if its PV entry could not be created. */ if (!pmap_pv_insert_pde(pmap, va, VM_PAGE_TO_PHYS(m))) { CTR2(KTR_PMAP, "pmap_enter_pde: failure for va %#lx" " in pmap %p", va, pmap); return (FALSE); } } -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((prot & VM_PROT_EXECUTE) == 0) newpde |= pg_nx; #endif if (va < VM_MAXUSER_ADDRESS) newpde |= PG_U; /* * Increment counters. */ pmap->pm_stats.resident_count += NBPDR / PAGE_SIZE; /* * Map the superpage. */ pde_store(pde, newpde); pmap_pde_mappings++; CTR2(KTR_PMAP, "pmap_enter_pde: success for va %#lx" " in pmap %p", va, pmap); return (TRUE); } /* * Maps a sequence of resident pages belonging to the same object. * The sequence begins with the given page m_start. This page is * mapped at the given virtual address start. Each subsequent page is * mapped at a virtual address that is offset from start by the same * amount as the page is offset from m_start within the object. The * last page in the sequence is the page with the largest offset from * m_start that can be mapped at a virtual address less than the given * virtual address end. Not every virtual page between start and end * is mapped; only those for which a resident page exists with the * corresponding offset from m_start are mapped. */ void pmap_enter_object(pmap_t pmap, vm_offset_t start, vm_offset_t end, vm_page_t m_start, vm_prot_t prot) { vm_offset_t va; vm_page_t m, mpte; vm_pindex_t diff, psize; VM_OBJECT_ASSERT_LOCKED(m_start->object); psize = atop(end - start); mpte = NULL; m = m_start; rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); while (m != NULL && (diff = m->pindex - m_start->pindex) < psize) { va = start + ptoa(diff); if ((va & PDRMASK) == 0 && va + NBPDR <= end && m->psind == 1 && pg_ps_enabled && pmap_enter_pde(pmap, va, m, prot)) m = &m[NBPDR / PAGE_SIZE - 1]; else mpte = pmap_enter_quick_locked(pmap, va, m, prot, mpte); m = TAILQ_NEXT(m, listq); } rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } /* * this code makes some *MAJOR* assumptions: * 1. Current pmap & pmap exists. * 2. Not wired. * 3. Read access. * 4. No page table pages. * but is *MUCH* faster than pmap_enter... */ void pmap_enter_quick(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot) { rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); (void)pmap_enter_quick_locked(pmap, va, m, prot, NULL); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); } static vm_page_t pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, vm_prot_t prot, vm_page_t mpte) { pt_entry_t *pte; vm_paddr_t pa; struct spglist free; KASSERT(va < kmi.clean_sva || va >= kmi.clean_eva || (m->oflags & VPO_UNMANAGED) != 0, ("pmap_enter_quick_locked: managed mapping within the clean submap")); rw_assert(&pvh_global_lock, RA_WLOCKED); PMAP_LOCK_ASSERT(pmap, MA_OWNED); /* * In the case that a page table page is not * resident, we are creating it here. */ if (va < VM_MAXUSER_ADDRESS) { u_int ptepindex; pd_entry_t ptepa; /* * Calculate pagetable page index */ ptepindex = va >> PDRSHIFT; if (mpte && (mpte->pindex == ptepindex)) { mpte->wire_count++; } else { /* * Get the page directory entry */ ptepa = pmap->pm_pdir[ptepindex]; /* * If the page table page is mapped, we just increment * the hold count, and activate it. */ if (ptepa) { if (ptepa & PG_PS) return (NULL); mpte = PHYS_TO_VM_PAGE(ptepa & PG_FRAME); mpte->wire_count++; } else { mpte = _pmap_allocpte(pmap, ptepindex, PMAP_ENTER_NOSLEEP); if (mpte == NULL) return (mpte); } } } else { mpte = NULL; } /* * This call to vtopte makes the assumption that we are * entering the page into the current pmap. In order to support * quick entry into any pmap, one would likely use pmap_pte_quick. * But that isn't as quick as vtopte. */ pte = vtopte(va); if (*pte) { if (mpte != NULL) { mpte->wire_count--; mpte = NULL; } return (mpte); } /* * Enter on the PV list if part of our managed memory. */ if ((m->oflags & VPO_UNMANAGED) == 0 && !pmap_try_insert_pv_entry(pmap, va, m)) { if (mpte != NULL) { SLIST_INIT(&free); if (pmap_unwire_ptp(pmap, mpte, &free)) { pmap_invalidate_page(pmap, va); pmap_free_zero_pages(&free); } mpte = NULL; } return (mpte); } /* * Increment counters */ pmap->pm_stats.resident_count++; pa = VM_PAGE_TO_PHYS(m) | pmap_cache_bits(m->md.pat_mode, 0); -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) if ((prot & VM_PROT_EXECUTE) == 0) pa |= pg_nx; #endif /* * Now validate mapping with RO protection */ if ((m->oflags & VPO_UNMANAGED) != 0) pte_store(pte, pa | PG_V | PG_U); else pte_store(pte, pa | PG_V | PG_U | PG_MANAGED); return (mpte); } /* * Make a temporary mapping for a physical address. This is only intended * to be used for panic dumps. */ void * pmap_kenter_temporary(vm_paddr_t pa, int i) { vm_offset_t va; va = (vm_offset_t)crashdumpmap + (i * PAGE_SIZE); pmap_kenter(va, pa); invlpg(va); return ((void *)crashdumpmap); } /* * This code maps large physical mmap regions into the * processor address space. Note that some shortcuts * are taken, but the code works. */ void pmap_object_init_pt(pmap_t pmap, vm_offset_t addr, vm_object_t object, vm_pindex_t pindex, vm_size_t size) { pd_entry_t *pde; vm_paddr_t pa, ptepa; vm_page_t p; int pat_mode; VM_OBJECT_ASSERT_WLOCKED(object); KASSERT(object->type == OBJT_DEVICE || object->type == OBJT_SG, ("pmap_object_init_pt: non-device object")); if (pseflag && (addr & (NBPDR - 1)) == 0 && (size & (NBPDR - 1)) == 0) { if (!vm_object_populate(object, pindex, pindex + atop(size))) return; p = vm_page_lookup(object, pindex); KASSERT(p->valid == VM_PAGE_BITS_ALL, ("pmap_object_init_pt: invalid page %p", p)); pat_mode = p->md.pat_mode; /* * Abort the mapping if the first page is not physically * aligned to a 2/4MB page boundary. */ ptepa = VM_PAGE_TO_PHYS(p); if (ptepa & (NBPDR - 1)) return; /* * Skip the first page. Abort the mapping if the rest of * the pages are not physically contiguous or have differing * memory attributes. */ p = TAILQ_NEXT(p, listq); for (pa = ptepa + PAGE_SIZE; pa < ptepa + size; pa += PAGE_SIZE) { KASSERT(p->valid == VM_PAGE_BITS_ALL, ("pmap_object_init_pt: invalid page %p", p)); if (pa != VM_PAGE_TO_PHYS(p) || pat_mode != p->md.pat_mode) return; p = TAILQ_NEXT(p, listq); } /* * Map using 2/4MB pages. Since "ptepa" is 2/4M aligned and * "size" is a multiple of 2/4M, adding the PAT setting to * "pa" will not affect the termination of this loop. */ PMAP_LOCK(pmap); for (pa = ptepa | pmap_cache_bits(pat_mode, 1); pa < ptepa + size; pa += NBPDR) { pde = pmap_pde(pmap, addr); if (*pde == 0) { pde_store(pde, pa | PG_PS | PG_M | PG_A | PG_U | PG_RW | PG_V); pmap->pm_stats.resident_count += NBPDR / PAGE_SIZE; pmap_pde_mappings++; } /* Else continue on if the PDE is already valid. */ addr += NBPDR; } PMAP_UNLOCK(pmap); } } /* * Clear the wired attribute from the mappings for the specified range of * addresses in the given pmap. Every valid mapping within that range * must have the wired attribute set. In contrast, invalid mappings * cannot have the wired attribute set, so they are ignored. * * The wired attribute of the page table entry is not a hardware feature, * so there is no need to invalidate any TLB entries. */ void pmap_unwire(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { vm_offset_t pdnxt; pd_entry_t *pde; pt_entry_t *pte; boolean_t pv_lists_locked; if (pmap_is_current(pmap)) pv_lists_locked = FALSE; else { pv_lists_locked = TRUE; resume: rw_wlock(&pvh_global_lock); sched_pin(); } PMAP_LOCK(pmap); for (; sva < eva; sva = pdnxt) { pdnxt = (sva + NBPDR) & ~PDRMASK; if (pdnxt < sva) pdnxt = eva; pde = pmap_pde(pmap, sva); if ((*pde & PG_V) == 0) continue; if ((*pde & PG_PS) != 0) { if ((*pde & PG_W) == 0) panic("pmap_unwire: pde %#jx is missing PG_W", (uintmax_t)*pde); /* * Are we unwiring the entire large page? If not, * demote the mapping and fall through. */ if (sva + NBPDR == pdnxt && eva >= pdnxt) { /* * Regardless of whether a pde (or pte) is 32 * or 64 bits in size, PG_W is among the least * significant 32 bits. */ atomic_clear_int((u_int *)pde, PG_W); pmap->pm_stats.wired_count -= NBPDR / PAGE_SIZE; continue; } else { if (!pv_lists_locked) { pv_lists_locked = TRUE; if (!rw_try_wlock(&pvh_global_lock)) { PMAP_UNLOCK(pmap); /* Repeat sva. */ goto resume; } sched_pin(); } if (!pmap_demote_pde(pmap, pde, sva)) panic("pmap_unwire: demotion failed"); } } if (pdnxt > eva) pdnxt = eva; for (pte = pmap_pte_quick(pmap, sva); sva != pdnxt; pte++, sva += PAGE_SIZE) { if ((*pte & PG_V) == 0) continue; if ((*pte & PG_W) == 0) panic("pmap_unwire: pte %#jx is missing PG_W", (uintmax_t)*pte); /* * PG_W must be cleared atomically. Although the pmap * lock synchronizes access to PG_W, another processor * could be setting PG_M and/or PG_A concurrently. * * PG_W is among the least significant 32 bits. */ atomic_clear_int((u_int *)pte, PG_W); pmap->pm_stats.wired_count--; } } if (pv_lists_locked) { sched_unpin(); rw_wunlock(&pvh_global_lock); } PMAP_UNLOCK(pmap); } /* * Copy the range specified by src_addr/len * from the source map to the range dst_addr/len * in the destination map. * * This routine is only advisory and need not do anything. */ void pmap_copy(pmap_t dst_pmap, pmap_t src_pmap, vm_offset_t dst_addr, vm_size_t len, vm_offset_t src_addr) { struct spglist free; vm_offset_t addr; vm_offset_t end_addr = src_addr + len; vm_offset_t pdnxt; if (dst_addr != src_addr) return; if (!pmap_is_current(src_pmap)) return; rw_wlock(&pvh_global_lock); if (dst_pmap < src_pmap) { PMAP_LOCK(dst_pmap); PMAP_LOCK(src_pmap); } else { PMAP_LOCK(src_pmap); PMAP_LOCK(dst_pmap); } sched_pin(); for (addr = src_addr; addr < end_addr; addr = pdnxt) { pt_entry_t *src_pte, *dst_pte; vm_page_t dstmpte, srcmpte; pd_entry_t srcptepaddr; u_int ptepindex; KASSERT(addr < UPT_MIN_ADDRESS, ("pmap_copy: invalid to pmap_copy page tables")); pdnxt = (addr + NBPDR) & ~PDRMASK; if (pdnxt < addr) pdnxt = end_addr; ptepindex = addr >> PDRSHIFT; srcptepaddr = src_pmap->pm_pdir[ptepindex]; if (srcptepaddr == 0) continue; if (srcptepaddr & PG_PS) { if ((addr & PDRMASK) != 0 || addr + NBPDR > end_addr) continue; if (dst_pmap->pm_pdir[ptepindex] == 0 && ((srcptepaddr & PG_MANAGED) == 0 || pmap_pv_insert_pde(dst_pmap, addr, srcptepaddr & PG_PS_FRAME))) { dst_pmap->pm_pdir[ptepindex] = srcptepaddr & ~PG_W; dst_pmap->pm_stats.resident_count += NBPDR / PAGE_SIZE; } continue; } srcmpte = PHYS_TO_VM_PAGE(srcptepaddr & PG_FRAME); KASSERT(srcmpte->wire_count > 0, ("pmap_copy: source page table page is unused")); if (pdnxt > end_addr) pdnxt = end_addr; src_pte = vtopte(addr); while (addr < pdnxt) { pt_entry_t ptetemp; ptetemp = *src_pte; /* * we only virtual copy managed pages */ if ((ptetemp & PG_MANAGED) != 0) { dstmpte = pmap_allocpte(dst_pmap, addr, PMAP_ENTER_NOSLEEP); if (dstmpte == NULL) goto out; dst_pte = pmap_pte_quick(dst_pmap, addr); if (*dst_pte == 0 && pmap_try_insert_pv_entry(dst_pmap, addr, PHYS_TO_VM_PAGE(ptetemp & PG_FRAME))) { /* * Clear the wired, modified, and * accessed (referenced) bits * during the copy. */ *dst_pte = ptetemp & ~(PG_W | PG_M | PG_A); dst_pmap->pm_stats.resident_count++; } else { SLIST_INIT(&free); if (pmap_unwire_ptp(dst_pmap, dstmpte, &free)) { pmap_invalidate_page(dst_pmap, addr); pmap_free_zero_pages(&free); } goto out; } if (dstmpte->wire_count >= srcmpte->wire_count) break; } addr += PAGE_SIZE; src_pte++; } } out: sched_unpin(); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(src_pmap); PMAP_UNLOCK(dst_pmap); } static __inline void pagezero(void *page) { #if defined(I686_CPU) if (cpu_class == CPUCLASS_686) { #if defined(CPU_ENABLE_SSE) if (cpu_feature & CPUID_SSE2) sse2_pagezero(page); else #endif i686_pagezero(page); } else #endif bzero(page, PAGE_SIZE); } /* * pmap_zero_page zeros the specified hardware page by mapping * the page into KVM and using bzero to clear its contents. */ void pmap_zero_page(vm_page_t m) { struct sysmaps *sysmaps; sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; mtx_lock(&sysmaps->lock); if (*sysmaps->CMAP2) panic("pmap_zero_page: CMAP2 busy"); sched_pin(); *sysmaps->CMAP2 = PG_V | PG_RW | VM_PAGE_TO_PHYS(m) | PG_A | PG_M | pmap_cache_bits(m->md.pat_mode, 0); invlcaddr(sysmaps->CADDR2); pagezero(sysmaps->CADDR2); *sysmaps->CMAP2 = 0; sched_unpin(); mtx_unlock(&sysmaps->lock); } /* * pmap_zero_page_area zeros the specified hardware page by mapping * the page into KVM and using bzero to clear its contents. * * off and size may not cover an area beyond a single hardware page. */ void pmap_zero_page_area(vm_page_t m, int off, int size) { struct sysmaps *sysmaps; sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; mtx_lock(&sysmaps->lock); if (*sysmaps->CMAP2) panic("pmap_zero_page_area: CMAP2 busy"); sched_pin(); *sysmaps->CMAP2 = PG_V | PG_RW | VM_PAGE_TO_PHYS(m) | PG_A | PG_M | pmap_cache_bits(m->md.pat_mode, 0); invlcaddr(sysmaps->CADDR2); if (off == 0 && size == PAGE_SIZE) pagezero(sysmaps->CADDR2); else bzero((char *)sysmaps->CADDR2 + off, size); *sysmaps->CMAP2 = 0; sched_unpin(); mtx_unlock(&sysmaps->lock); } /* * pmap_zero_page_idle zeros the specified hardware page by mapping * the page into KVM and using bzero to clear its contents. This * is intended to be called from the vm_pagezero process only and * outside of Giant. */ void pmap_zero_page_idle(vm_page_t m) { if (*CMAP3) panic("pmap_zero_page_idle: CMAP3 busy"); sched_pin(); *CMAP3 = PG_V | PG_RW | VM_PAGE_TO_PHYS(m) | PG_A | PG_M | pmap_cache_bits(m->md.pat_mode, 0); invlcaddr(CADDR3); pagezero(CADDR3); *CMAP3 = 0; sched_unpin(); } /* * pmap_copy_page copies the specified (machine independent) * page by mapping the page into virtual memory and using * bcopy to copy the page, one machine dependent page at a * time. */ void pmap_copy_page(vm_page_t src, vm_page_t dst) { struct sysmaps *sysmaps; sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; mtx_lock(&sysmaps->lock); if (*sysmaps->CMAP1) panic("pmap_copy_page: CMAP1 busy"); if (*sysmaps->CMAP2) panic("pmap_copy_page: CMAP2 busy"); sched_pin(); *sysmaps->CMAP1 = PG_V | VM_PAGE_TO_PHYS(src) | PG_A | pmap_cache_bits(src->md.pat_mode, 0); invlcaddr(sysmaps->CADDR1); *sysmaps->CMAP2 = PG_V | PG_RW | VM_PAGE_TO_PHYS(dst) | PG_A | PG_M | pmap_cache_bits(dst->md.pat_mode, 0); invlcaddr(sysmaps->CADDR2); bcopy(sysmaps->CADDR1, sysmaps->CADDR2, PAGE_SIZE); *sysmaps->CMAP1 = 0; *sysmaps->CMAP2 = 0; sched_unpin(); mtx_unlock(&sysmaps->lock); } int unmapped_buf_allowed = 1; void pmap_copy_pages(vm_page_t ma[], vm_offset_t a_offset, vm_page_t mb[], vm_offset_t b_offset, int xfersize) { struct sysmaps *sysmaps; vm_page_t a_pg, b_pg; char *a_cp, *b_cp; vm_offset_t a_pg_offset, b_pg_offset; int cnt; sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; mtx_lock(&sysmaps->lock); if (*sysmaps->CMAP1 != 0) panic("pmap_copy_pages: CMAP1 busy"); if (*sysmaps->CMAP2 != 0) panic("pmap_copy_pages: CMAP2 busy"); sched_pin(); while (xfersize > 0) { a_pg = ma[a_offset >> PAGE_SHIFT]; a_pg_offset = a_offset & PAGE_MASK; cnt = min(xfersize, PAGE_SIZE - a_pg_offset); b_pg = mb[b_offset >> PAGE_SHIFT]; b_pg_offset = b_offset & PAGE_MASK; cnt = min(cnt, PAGE_SIZE - b_pg_offset); *sysmaps->CMAP1 = PG_V | VM_PAGE_TO_PHYS(a_pg) | PG_A | pmap_cache_bits(a_pg->md.pat_mode, 0); invlcaddr(sysmaps->CADDR1); *sysmaps->CMAP2 = PG_V | PG_RW | VM_PAGE_TO_PHYS(b_pg) | PG_A | PG_M | pmap_cache_bits(b_pg->md.pat_mode, 0); invlcaddr(sysmaps->CADDR2); a_cp = sysmaps->CADDR1 + a_pg_offset; b_cp = sysmaps->CADDR2 + b_pg_offset; bcopy(a_cp, b_cp, cnt); a_offset += cnt; b_offset += cnt; xfersize -= cnt; } *sysmaps->CMAP1 = 0; *sysmaps->CMAP2 = 0; sched_unpin(); mtx_unlock(&sysmaps->lock); } /* * Returns true if the pmap's pv is one of the first * 16 pvs linked to from this page. This count may * be changed upwards or downwards in the future; it * is only necessary that true be returned for a small * subset of pmaps for proper page aging. */ boolean_t pmap_page_exists_quick(pmap_t pmap, vm_page_t m) { struct md_page *pvh; pv_entry_t pv; int loops = 0; boolean_t rv; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_page_exists_quick: page %p is not managed", m)); rv = FALSE; rw_wlock(&pvh_global_lock); TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { if (PV_PMAP(pv) == pmap) { rv = TRUE; break; } loops++; if (loops >= 16) break; } if (!rv && loops < 16 && (m->flags & PG_FICTITIOUS) == 0) { pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { if (PV_PMAP(pv) == pmap) { rv = TRUE; break; } loops++; if (loops >= 16) break; } } rw_wunlock(&pvh_global_lock); return (rv); } /* * pmap_page_wired_mappings: * * Return the number of managed mappings to the given physical page * that are wired. */ int pmap_page_wired_mappings(vm_page_t m) { int count; count = 0; if ((m->oflags & VPO_UNMANAGED) != 0) return (count); rw_wlock(&pvh_global_lock); count = pmap_pvh_wired_mappings(&m->md, count); if ((m->flags & PG_FICTITIOUS) == 0) { count = pmap_pvh_wired_mappings(pa_to_pvh(VM_PAGE_TO_PHYS(m)), count); } rw_wunlock(&pvh_global_lock); return (count); } /* * pmap_pvh_wired_mappings: * * Return the updated number "count" of managed mappings that are wired. */ static int pmap_pvh_wired_mappings(struct md_page *pvh, int count) { pmap_t pmap; pt_entry_t *pte; pv_entry_t pv; rw_assert(&pvh_global_lock, RA_WLOCKED); sched_pin(); TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pte = pmap_pte_quick(pmap, pv->pv_va); if ((*pte & PG_W) != 0) count++; PMAP_UNLOCK(pmap); } sched_unpin(); return (count); } /* * Returns TRUE if the given page is mapped individually or as part of * a 4mpage. Otherwise, returns FALSE. */ boolean_t pmap_page_is_mapped(vm_page_t m) { boolean_t rv; if ((m->oflags & VPO_UNMANAGED) != 0) return (FALSE); rw_wlock(&pvh_global_lock); rv = !TAILQ_EMPTY(&m->md.pv_list) || ((m->flags & PG_FICTITIOUS) == 0 && !TAILQ_EMPTY(&pa_to_pvh(VM_PAGE_TO_PHYS(m))->pv_list)); rw_wunlock(&pvh_global_lock); return (rv); } /* * Remove all pages from specified address space * this aids process exit speeds. Also, this code * is special cased for current process only, but * can have the more generic (and slightly slower) * mode enabled. This is much faster than pmap_remove * in the case of running down an entire address space. */ void pmap_remove_pages(pmap_t pmap) { pt_entry_t *pte, tpte; vm_page_t m, mpte, mt; pv_entry_t pv; struct md_page *pvh; struct pv_chunk *pc, *npc; struct spglist free; int field, idx; int32_t bit; uint32_t inuse, bitmask; int allfree; if (pmap != PCPU_GET(curpmap)) { printf("warning: pmap_remove_pages called with non-current pmap\n"); return; } SLIST_INIT(&free); rw_wlock(&pvh_global_lock); PMAP_LOCK(pmap); sched_pin(); TAILQ_FOREACH_SAFE(pc, &pmap->pm_pvchunk, pc_list, npc) { KASSERT(pc->pc_pmap == pmap, ("Wrong pmap %p %p", pmap, pc->pc_pmap)); allfree = 1; for (field = 0; field < _NPCM; field++) { inuse = ~pc->pc_map[field] & pc_freemask[field]; while (inuse != 0) { bit = bsfl(inuse); bitmask = 1UL << bit; idx = field * 32 + bit; pv = &pc->pc_pventry[idx]; inuse &= ~bitmask; pte = pmap_pde(pmap, pv->pv_va); tpte = *pte; if ((tpte & PG_PS) == 0) { pte = vtopte(pv->pv_va); tpte = *pte & ~PG_PTE_PAT; } if (tpte == 0) { printf( "TPTE at %p IS ZERO @ VA %08x\n", pte, pv->pv_va); panic("bad pte"); } /* * We cannot remove wired pages from a process' mapping at this time */ if (tpte & PG_W) { allfree = 0; continue; } m = PHYS_TO_VM_PAGE(tpte & PG_FRAME); KASSERT(m->phys_addr == (tpte & PG_FRAME), ("vm_page_t %p phys_addr mismatch %016jx %016jx", m, (uintmax_t)m->phys_addr, (uintmax_t)tpte)); KASSERT((m->flags & PG_FICTITIOUS) != 0 || m < &vm_page_array[vm_page_array_size], ("pmap_remove_pages: bad tpte %#jx", (uintmax_t)tpte)); pte_clear(pte); /* * Update the vm_page_t clean/reference bits. */ if ((tpte & (PG_M | PG_RW)) == (PG_M | PG_RW)) { if ((tpte & PG_PS) != 0) { for (mt = m; mt < &m[NBPDR / PAGE_SIZE]; mt++) vm_page_dirty(mt); } else vm_page_dirty(m); } /* Mark free */ PV_STAT(pv_entry_frees++); PV_STAT(pv_entry_spare++); pv_entry_count--; pc->pc_map[field] |= bitmask; if ((tpte & PG_PS) != 0) { pmap->pm_stats.resident_count -= NBPDR / PAGE_SIZE; pvh = pa_to_pvh(tpte & PG_PS_FRAME); TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); if (TAILQ_EMPTY(&pvh->pv_list)) { for (mt = m; mt < &m[NBPDR / PAGE_SIZE]; mt++) if (TAILQ_EMPTY(&mt->md.pv_list)) vm_page_aflag_clear(mt, PGA_WRITEABLE); } mpte = pmap_lookup_pt_page(pmap, pv->pv_va); if (mpte != NULL) { pmap_remove_pt_page(pmap, mpte); pmap->pm_stats.resident_count--; KASSERT(mpte->wire_count == NPTEPG, ("pmap_remove_pages: pte page wire count error")); mpte->wire_count = 0; pmap_add_delayed_free_list(mpte, &free, FALSE); atomic_subtract_int(&vm_cnt.v_wire_count, 1); } } else { pmap->pm_stats.resident_count--; TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); if (TAILQ_EMPTY(&m->md.pv_list) && (m->flags & PG_FICTITIOUS) == 0) { pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); if (TAILQ_EMPTY(&pvh->pv_list)) vm_page_aflag_clear(m, PGA_WRITEABLE); } pmap_unuse_pt(pmap, pv->pv_va, &free); } } } if (allfree) { TAILQ_REMOVE(&pmap->pm_pvchunk, pc, pc_list); free_pv_chunk(pc); } } sched_unpin(); pmap_invalidate_all(pmap); rw_wunlock(&pvh_global_lock); PMAP_UNLOCK(pmap); pmap_free_zero_pages(&free); } /* * pmap_is_modified: * * Return whether or not the specified physical page was modified * in any physical maps. */ boolean_t pmap_is_modified(vm_page_t m) { boolean_t rv; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_is_modified: page %p is not managed", m)); /* * If the page is not exclusive busied, then PGA_WRITEABLE cannot be * concurrently set while the object is locked. Thus, if PGA_WRITEABLE * is clear, no PTEs can have PG_M set. */ VM_OBJECT_ASSERT_WLOCKED(m->object); if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) return (FALSE); rw_wlock(&pvh_global_lock); rv = pmap_is_modified_pvh(&m->md) || ((m->flags & PG_FICTITIOUS) == 0 && pmap_is_modified_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m)))); rw_wunlock(&pvh_global_lock); return (rv); } /* * Returns TRUE if any of the given mappings were used to modify * physical memory. Otherwise, returns FALSE. Both page and 2mpage * mappings are supported. */ static boolean_t pmap_is_modified_pvh(struct md_page *pvh) { pv_entry_t pv; pt_entry_t *pte; pmap_t pmap; boolean_t rv; rw_assert(&pvh_global_lock, RA_WLOCKED); rv = FALSE; sched_pin(); TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pte = pmap_pte_quick(pmap, pv->pv_va); rv = (*pte & (PG_M | PG_RW)) == (PG_M | PG_RW); PMAP_UNLOCK(pmap); if (rv) break; } sched_unpin(); return (rv); } /* * pmap_is_prefaultable: * * Return whether or not the specified virtual address is elgible * for prefault. */ boolean_t pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr) { pd_entry_t *pde; pt_entry_t *pte; boolean_t rv; rv = FALSE; PMAP_LOCK(pmap); pde = pmap_pde(pmap, addr); if (*pde != 0 && (*pde & PG_PS) == 0) { pte = vtopte(addr); rv = *pte == 0; } PMAP_UNLOCK(pmap); return (rv); } /* * pmap_is_referenced: * * Return whether or not the specified physical page was referenced * in any physical maps. */ boolean_t pmap_is_referenced(vm_page_t m) { boolean_t rv; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_is_referenced: page %p is not managed", m)); rw_wlock(&pvh_global_lock); rv = pmap_is_referenced_pvh(&m->md) || ((m->flags & PG_FICTITIOUS) == 0 && pmap_is_referenced_pvh(pa_to_pvh(VM_PAGE_TO_PHYS(m)))); rw_wunlock(&pvh_global_lock); return (rv); } /* * Returns TRUE if any of the given mappings were referenced and FALSE * otherwise. Both page and 4mpage mappings are supported. */ static boolean_t pmap_is_referenced_pvh(struct md_page *pvh) { pv_entry_t pv; pt_entry_t *pte; pmap_t pmap; boolean_t rv; rw_assert(&pvh_global_lock, RA_WLOCKED); rv = FALSE; sched_pin(); TAILQ_FOREACH(pv, &pvh->pv_list, pv_next) { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pte = pmap_pte_quick(pmap, pv->pv_va); rv = (*pte & (PG_A | PG_V)) == (PG_A | PG_V); PMAP_UNLOCK(pmap); if (rv) break; } sched_unpin(); return (rv); } /* * Clear the write and modified bits in each of the given page's mappings. */ void pmap_remove_write(vm_page_t m) { struct md_page *pvh; pv_entry_t next_pv, pv; pmap_t pmap; pd_entry_t *pde; pt_entry_t oldpte, *pte; vm_offset_t va; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_remove_write: page %p is not managed", m)); /* * If the page is not exclusive busied, then PGA_WRITEABLE cannot be * set by another thread while the object is locked. Thus, * if PGA_WRITEABLE is clear, no page table entries need updating. */ VM_OBJECT_ASSERT_WLOCKED(m->object); if (!vm_page_xbusied(m) && (m->aflags & PGA_WRITEABLE) == 0) return; rw_wlock(&pvh_global_lock); sched_pin(); if ((m->flags & PG_FICTITIOUS) != 0) goto small_mappings; pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_next, next_pv) { va = pv->pv_va; pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pde = pmap_pde(pmap, va); if ((*pde & PG_RW) != 0) (void)pmap_demote_pde(pmap, pde, va); PMAP_UNLOCK(pmap); } small_mappings: TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pde = pmap_pde(pmap, pv->pv_va); KASSERT((*pde & PG_PS) == 0, ("pmap_clear_write: found" " a 4mpage in page %p's pv list", m)); pte = pmap_pte_quick(pmap, pv->pv_va); retry: oldpte = *pte; if ((oldpte & PG_RW) != 0) { /* * Regardless of whether a pte is 32 or 64 bits * in size, PG_RW and PG_M are among the least * significant 32 bits. */ if (!atomic_cmpset_int((u_int *)pte, oldpte, oldpte & ~(PG_RW | PG_M))) goto retry; if ((oldpte & PG_M) != 0) vm_page_dirty(m); pmap_invalidate_page(pmap, pv->pv_va); } PMAP_UNLOCK(pmap); } vm_page_aflag_clear(m, PGA_WRITEABLE); sched_unpin(); rw_wunlock(&pvh_global_lock); } #define PMAP_TS_REFERENCED_MAX 5 /* * pmap_ts_referenced: * * Return a count of reference bits for a page, clearing those bits. * It is not necessary for every reference bit to be cleared, but it * is necessary that 0 only be returned when there are truly no * reference bits set. * * XXX: The exact number of bits to check and clear is a matter that * should be tested and standardized at some point in the future for * optimal aging of shared pages. */ int pmap_ts_referenced(vm_page_t m) { struct md_page *pvh; pv_entry_t pv, pvf; pmap_t pmap; pd_entry_t *pde; pt_entry_t *pte; vm_paddr_t pa; int rtval = 0; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_ts_referenced: page %p is not managed", m)); pa = VM_PAGE_TO_PHYS(m); pvh = pa_to_pvh(pa); rw_wlock(&pvh_global_lock); sched_pin(); if ((m->flags & PG_FICTITIOUS) != 0 || (pvf = TAILQ_FIRST(&pvh->pv_list)) == NULL) goto small_mappings; pv = pvf; do { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pde = pmap_pde(pmap, pv->pv_va); if ((*pde & PG_A) != 0) { /* * Since this reference bit is shared by either 1024 * or 512 4KB pages, it should not be cleared every * time it is tested. Apply a simple "hash" function * on the physical page number, the virtual superpage * number, and the pmap address to select one 4KB page * out of the 1024 or 512 on which testing the * reference bit will result in clearing that bit. * This function is designed to avoid the selection of * the same 4KB page for every 2- or 4MB page mapping. * * On demotion, a mapping that hasn't been referenced * is simply destroyed. To avoid the possibility of a * subsequent page fault on a demoted wired mapping, * always leave its reference bit set. Moreover, * since the superpage is wired, the current state of * its reference bit won't affect page replacement. */ if ((((pa >> PAGE_SHIFT) ^ (pv->pv_va >> PDRSHIFT) ^ (uintptr_t)pmap) & (NPTEPG - 1)) == 0 && (*pde & PG_W) == 0) { atomic_clear_int((u_int *)pde, PG_A); pmap_invalidate_page(pmap, pv->pv_va); } rtval++; } PMAP_UNLOCK(pmap); /* Rotate the PV list if it has more than one entry. */ if (TAILQ_NEXT(pv, pv_next) != NULL) { TAILQ_REMOVE(&pvh->pv_list, pv, pv_next); TAILQ_INSERT_TAIL(&pvh->pv_list, pv, pv_next); } if (rtval >= PMAP_TS_REFERENCED_MAX) goto out; } while ((pv = TAILQ_FIRST(&pvh->pv_list)) != pvf); small_mappings: if ((pvf = TAILQ_FIRST(&m->md.pv_list)) == NULL) goto out; pv = pvf; do { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pde = pmap_pde(pmap, pv->pv_va); KASSERT((*pde & PG_PS) == 0, ("pmap_ts_referenced: found a 4mpage in page %p's pv list", m)); pte = pmap_pte_quick(pmap, pv->pv_va); if ((*pte & PG_A) != 0) { atomic_clear_int((u_int *)pte, PG_A); pmap_invalidate_page(pmap, pv->pv_va); rtval++; } PMAP_UNLOCK(pmap); /* Rotate the PV list if it has more than one entry. */ if (TAILQ_NEXT(pv, pv_next) != NULL) { TAILQ_REMOVE(&m->md.pv_list, pv, pv_next); TAILQ_INSERT_TAIL(&m->md.pv_list, pv, pv_next); } } while ((pv = TAILQ_FIRST(&m->md.pv_list)) != pvf && rtval < PMAP_TS_REFERENCED_MAX); out: sched_unpin(); rw_wunlock(&pvh_global_lock); return (rtval); } /* * Apply the given advice to the specified range of addresses within the * given pmap. Depending on the advice, clear the referenced and/or * modified flags in each mapping and set the mapped page's dirty field. */ void pmap_advise(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, int advice) { pd_entry_t oldpde, *pde; pt_entry_t *pte; vm_offset_t pdnxt; vm_page_t m; boolean_t anychanged, pv_lists_locked; if (advice != MADV_DONTNEED && advice != MADV_FREE) return; if (pmap_is_current(pmap)) pv_lists_locked = FALSE; else { pv_lists_locked = TRUE; resume: rw_wlock(&pvh_global_lock); sched_pin(); } anychanged = FALSE; PMAP_LOCK(pmap); for (; sva < eva; sva = pdnxt) { pdnxt = (sva + NBPDR) & ~PDRMASK; if (pdnxt < sva) pdnxt = eva; pde = pmap_pde(pmap, sva); oldpde = *pde; if ((oldpde & PG_V) == 0) continue; else if ((oldpde & PG_PS) != 0) { if ((oldpde & PG_MANAGED) == 0) continue; if (!pv_lists_locked) { pv_lists_locked = TRUE; if (!rw_try_wlock(&pvh_global_lock)) { if (anychanged) pmap_invalidate_all(pmap); PMAP_UNLOCK(pmap); goto resume; } sched_pin(); } if (!pmap_demote_pde(pmap, pde, sva)) { /* * The large page mapping was destroyed. */ continue; } /* * Unless the page mappings are wired, remove the * mapping to a single page so that a subsequent * access may repromote. Since the underlying page * table page is fully populated, this removal never * frees a page table page. */ if ((oldpde & PG_W) == 0) { pte = pmap_pte_quick(pmap, sva); KASSERT((*pte & PG_V) != 0, ("pmap_advise: invalid PTE")); pmap_remove_pte(pmap, pte, sva, NULL); anychanged = TRUE; } } if (pdnxt > eva) pdnxt = eva; for (pte = pmap_pte_quick(pmap, sva); sva != pdnxt; pte++, sva += PAGE_SIZE) { if ((*pte & (PG_MANAGED | PG_V)) != (PG_MANAGED | PG_V)) continue; else if ((*pte & (PG_M | PG_RW)) == (PG_M | PG_RW)) { if (advice == MADV_DONTNEED) { /* * Future calls to pmap_is_modified() * can be avoided by making the page * dirty now. */ m = PHYS_TO_VM_PAGE(*pte & PG_FRAME); vm_page_dirty(m); } atomic_clear_int((u_int *)pte, PG_M | PG_A); } else if ((*pte & PG_A) != 0) atomic_clear_int((u_int *)pte, PG_A); else continue; if ((*pte & PG_G) != 0) pmap_invalidate_page(pmap, sva); else anychanged = TRUE; } } if (anychanged) pmap_invalidate_all(pmap); if (pv_lists_locked) { sched_unpin(); rw_wunlock(&pvh_global_lock); } PMAP_UNLOCK(pmap); } /* * Clear the modify bits on the specified physical page. */ void pmap_clear_modify(vm_page_t m) { struct md_page *pvh; pv_entry_t next_pv, pv; pmap_t pmap; pd_entry_t oldpde, *pde; pt_entry_t oldpte, *pte; vm_offset_t va; KASSERT((m->oflags & VPO_UNMANAGED) == 0, ("pmap_clear_modify: page %p is not managed", m)); VM_OBJECT_ASSERT_WLOCKED(m->object); KASSERT(!vm_page_xbusied(m), ("pmap_clear_modify: page %p is exclusive busied", m)); /* * If the page is not PGA_WRITEABLE, then no PTEs can have PG_M set. * If the object containing the page is locked and the page is not * exclusive busied, then PGA_WRITEABLE cannot be concurrently set. */ if ((m->aflags & PGA_WRITEABLE) == 0) return; rw_wlock(&pvh_global_lock); sched_pin(); if ((m->flags & PG_FICTITIOUS) != 0) goto small_mappings; pvh = pa_to_pvh(VM_PAGE_TO_PHYS(m)); TAILQ_FOREACH_SAFE(pv, &pvh->pv_list, pv_next, next_pv) { va = pv->pv_va; pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pde = pmap_pde(pmap, va); oldpde = *pde; if ((oldpde & PG_RW) != 0) { if (pmap_demote_pde(pmap, pde, va)) { if ((oldpde & PG_W) == 0) { /* * Write protect the mapping to a * single page so that a subsequent * write access may repromote. */ va += VM_PAGE_TO_PHYS(m) - (oldpde & PG_PS_FRAME); pte = pmap_pte_quick(pmap, va); oldpte = *pte; if ((oldpte & PG_V) != 0) { /* * Regardless of whether a pte is 32 or 64 bits * in size, PG_RW and PG_M are among the least * significant 32 bits. */ while (!atomic_cmpset_int((u_int *)pte, oldpte, oldpte & ~(PG_M | PG_RW))) oldpte = *pte; vm_page_dirty(m); pmap_invalidate_page(pmap, va); } } } } PMAP_UNLOCK(pmap); } small_mappings: TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { pmap = PV_PMAP(pv); PMAP_LOCK(pmap); pde = pmap_pde(pmap, pv->pv_va); KASSERT((*pde & PG_PS) == 0, ("pmap_clear_modify: found" " a 4mpage in page %p's pv list", m)); pte = pmap_pte_quick(pmap, pv->pv_va); if ((*pte & (PG_M | PG_RW)) == (PG_M | PG_RW)) { /* * Regardless of whether a pte is 32 or 64 bits * in size, PG_M is among the least significant * 32 bits. */ atomic_clear_int((u_int *)pte, PG_M); pmap_invalidate_page(pmap, pv->pv_va); } PMAP_UNLOCK(pmap); } sched_unpin(); rw_wunlock(&pvh_global_lock); } /* * Miscellaneous support routines follow */ /* Adjust the cache mode for a 4KB page mapped via a PTE. */ static __inline void pmap_pte_attr(pt_entry_t *pte, int cache_bits) { u_int opte, npte; /* * The cache mode bits are all in the low 32-bits of the * PTE, so we can just spin on updating the low 32-bits. */ do { opte = *(u_int *)pte; npte = opte & ~PG_PTE_CACHE; npte |= cache_bits; } while (npte != opte && !atomic_cmpset_int((u_int *)pte, opte, npte)); } /* Adjust the cache mode for a 2/4MB page mapped via a PDE. */ static __inline void pmap_pde_attr(pd_entry_t *pde, int cache_bits) { u_int opde, npde; /* * The cache mode bits are all in the low 32-bits of the * PDE, so we can just spin on updating the low 32-bits. */ do { opde = *(u_int *)pde; npde = opde & ~PG_PDE_CACHE; npde |= cache_bits; } while (npde != opde && !atomic_cmpset_int((u_int *)pde, opde, npde)); } /* * Map a set of physical memory pages into the kernel virtual * address space. Return a pointer to where it is mapped. This * routine is intended to be used for mapping device memory, * NOT real memory. */ void * pmap_mapdev_attr(vm_paddr_t pa, vm_size_t size, int mode) { vm_offset_t va, offset; vm_size_t tmpsize; offset = pa & PAGE_MASK; size = round_page(offset + size); pa = pa & PG_FRAME; if (pa < KERNLOAD && pa + size <= KERNLOAD) va = KERNBASE + pa; else va = kva_alloc(size); if (!va) panic("pmap_mapdev: Couldn't alloc kernel virtual memory"); for (tmpsize = 0; tmpsize < size; tmpsize += PAGE_SIZE) pmap_kenter_attr(va + tmpsize, pa + tmpsize, mode); pmap_invalidate_range(kernel_pmap, va, va + tmpsize); pmap_invalidate_cache_range(va, va + size, FALSE); return ((void *)(va + offset)); } void * pmap_mapdev(vm_paddr_t pa, vm_size_t size) { return (pmap_mapdev_attr(pa, size, PAT_UNCACHEABLE)); } void * pmap_mapbios(vm_paddr_t pa, vm_size_t size) { return (pmap_mapdev_attr(pa, size, PAT_WRITE_BACK)); } void pmap_unmapdev(vm_offset_t va, vm_size_t size) { vm_offset_t base, offset; if (va >= KERNBASE && va + size <= KERNBASE + KERNLOAD) return; base = trunc_page(va); offset = va & PAGE_MASK; size = round_page(offset + size); kva_free(base, size); } /* * Sets the memory attribute for the specified page. */ void pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma) { m->md.pat_mode = ma; if ((m->flags & PG_FICTITIOUS) != 0) return; /* * If "m" is a normal page, flush it from the cache. * See pmap_invalidate_cache_range(). * * First, try to find an existing mapping of the page by sf * buffer. sf_buf_invalidate_cache() modifies mapping and * flushes the cache. */ if (sf_buf_invalidate_cache(m)) return; /* * If page is not mapped by sf buffer, but CPU does not * support self snoop, map the page transient and do * invalidation. In the worst case, whole cache is flushed by * pmap_invalidate_cache_range(). */ if ((cpu_feature & CPUID_SS) == 0) pmap_flush_page(m); } static void pmap_flush_page(vm_page_t m) { struct sysmaps *sysmaps; vm_offset_t sva, eva; if ((cpu_feature & CPUID_CLFSH) != 0) { sysmaps = &sysmaps_pcpu[PCPU_GET(cpuid)]; mtx_lock(&sysmaps->lock); if (*sysmaps->CMAP2) panic("pmap_flush_page: CMAP2 busy"); sched_pin(); *sysmaps->CMAP2 = PG_V | PG_RW | VM_PAGE_TO_PHYS(m) | PG_A | PG_M | pmap_cache_bits(m->md.pat_mode, 0); invlcaddr(sysmaps->CADDR2); sva = (vm_offset_t)sysmaps->CADDR2; eva = sva + PAGE_SIZE; /* * Use mfence despite the ordering implied by * mtx_{un,}lock() because clflush is not guaranteed * to be ordered by any other instruction. */ mfence(); for (; sva < eva; sva += cpu_clflush_line_size) clflush(sva); mfence(); *sysmaps->CMAP2 = 0; sched_unpin(); mtx_unlock(&sysmaps->lock); } else pmap_invalidate_cache(); } /* * Changes the specified virtual address range's memory type to that given by * the parameter "mode". The specified virtual address range must be * completely contained within either the kernel map. * * Returns zero if the change completed successfully, and either EINVAL or * ENOMEM if the change failed. Specifically, EINVAL is returned if some part * of the virtual address range was not mapped, and ENOMEM is returned if * there was insufficient memory available to complete the change. */ int pmap_change_attr(vm_offset_t va, vm_size_t size, int mode) { vm_offset_t base, offset, tmpva; pd_entry_t *pde; pt_entry_t *pte; int cache_bits_pte, cache_bits_pde; boolean_t changed; base = trunc_page(va); offset = va & PAGE_MASK; size = round_page(offset + size); /* * Only supported on kernel virtual addresses above the recursive map. */ if (base < VM_MIN_KERNEL_ADDRESS) return (EINVAL); cache_bits_pde = pmap_cache_bits(mode, 1); cache_bits_pte = pmap_cache_bits(mode, 0); changed = FALSE; /* * Pages that aren't mapped aren't supported. Also break down * 2/4MB pages into 4KB pages if required. */ PMAP_LOCK(kernel_pmap); for (tmpva = base; tmpva < base + size; ) { pde = pmap_pde(kernel_pmap, tmpva); if (*pde == 0) { PMAP_UNLOCK(kernel_pmap); return (EINVAL); } if (*pde & PG_PS) { /* * If the current 2/4MB page already has * the required memory type, then we need not * demote this page. Just increment tmpva to * the next 2/4MB page frame. */ if ((*pde & PG_PDE_CACHE) == cache_bits_pde) { tmpva = trunc_4mpage(tmpva) + NBPDR; continue; } /* * If the current offset aligns with a 2/4MB * page frame and there is at least 2/4MB left * within the range, then we need not break * down this page into 4KB pages. */ if ((tmpva & PDRMASK) == 0 && tmpva + PDRMASK < base + size) { tmpva += NBPDR; continue; } if (!pmap_demote_pde(kernel_pmap, pde, tmpva)) { PMAP_UNLOCK(kernel_pmap); return (ENOMEM); } } pte = vtopte(tmpva); if (*pte == 0) { PMAP_UNLOCK(kernel_pmap); return (EINVAL); } tmpva += PAGE_SIZE; } PMAP_UNLOCK(kernel_pmap); /* * Ok, all the pages exist, so run through them updating their * cache mode if required. */ for (tmpva = base; tmpva < base + size; ) { pde = pmap_pde(kernel_pmap, tmpva); if (*pde & PG_PS) { if ((*pde & PG_PDE_CACHE) != cache_bits_pde) { pmap_pde_attr(pde, cache_bits_pde); changed = TRUE; } tmpva = trunc_4mpage(tmpva) + NBPDR; } else { pte = vtopte(tmpva); if ((*pte & PG_PTE_CACHE) != cache_bits_pte) { pmap_pte_attr(pte, cache_bits_pte); changed = TRUE; } tmpva += PAGE_SIZE; } } /* * Flush CPU caches to make sure any data isn't cached that * shouldn't be, etc. */ if (changed) { pmap_invalidate_range(kernel_pmap, base, tmpva); pmap_invalidate_cache_range(base, tmpva, FALSE); } return (0); } /* * perform the pmap work for mincore */ int pmap_mincore(pmap_t pmap, vm_offset_t addr, vm_paddr_t *locked_pa) { pd_entry_t *pdep; pt_entry_t *ptep, pte; vm_paddr_t pa; int val; PMAP_LOCK(pmap); retry: pdep = pmap_pde(pmap, addr); if (*pdep != 0) { if (*pdep & PG_PS) { pte = *pdep; /* Compute the physical address of the 4KB page. */ pa = ((*pdep & PG_PS_FRAME) | (addr & PDRMASK)) & PG_FRAME; val = MINCORE_SUPER; } else { ptep = pmap_pte(pmap, addr); pte = *ptep; pmap_pte_release(ptep); pa = pte & PG_FRAME; val = 0; } } else { pte = 0; pa = 0; val = 0; } if ((pte & PG_V) != 0) { val |= MINCORE_INCORE; if ((pte & (PG_M | PG_RW)) == (PG_M | PG_RW)) val |= MINCORE_MODIFIED | MINCORE_MODIFIED_OTHER; if ((pte & PG_A) != 0) val |= MINCORE_REFERENCED | MINCORE_REFERENCED_OTHER; } if ((val & (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER)) != (MINCORE_MODIFIED_OTHER | MINCORE_REFERENCED_OTHER) && (pte & (PG_MANAGED | PG_V)) == (PG_MANAGED | PG_V)) { /* Ensure that "PHYS_TO_VM_PAGE(pa)->object" doesn't change. */ if (vm_page_pa_tryrelock(pmap, pa, locked_pa)) goto retry; } else PA_UNLOCK_COND(*locked_pa); PMAP_UNLOCK(pmap); return (val); } void pmap_activate(struct thread *td) { pmap_t pmap, oldpmap; u_int cpuid; u_int32_t cr3; critical_enter(); pmap = vmspace_pmap(td->td_proc->p_vmspace); oldpmap = PCPU_GET(curpmap); cpuid = PCPU_GET(cpuid); #if defined(SMP) CPU_CLR_ATOMIC(cpuid, &oldpmap->pm_active); CPU_SET_ATOMIC(cpuid, &pmap->pm_active); #else CPU_CLR(cpuid, &oldpmap->pm_active); CPU_SET(cpuid, &pmap->pm_active); #endif -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) cr3 = vtophys(pmap->pm_pdpt); #else cr3 = vtophys(pmap->pm_pdir); #endif /* * pmap_activate is for the current thread on the current cpu */ td->td_pcb->pcb_cr3 = cr3; load_cr3(cr3); PCPU_SET(curpmap, pmap); critical_exit(); } void pmap_sync_icache(pmap_t pm, vm_offset_t va, vm_size_t sz) { } /* * Increase the starting virtual address of the given mapping if a * different alignment might result in more superpage mappings. */ void pmap_align_superpage(vm_object_t object, vm_ooffset_t offset, vm_offset_t *addr, vm_size_t size) { vm_offset_t superpage_offset; if (size < NBPDR) return; if (object != NULL && (object->flags & OBJ_COLORED) != 0) offset += ptoa(object->pg_color); superpage_offset = offset & PDRMASK; if (size - ((NBPDR - superpage_offset) & PDRMASK) < NBPDR || (*addr & PDRMASK) == superpage_offset) return; if ((*addr & PDRMASK) < superpage_offset) *addr = (*addr & ~PDRMASK) + superpage_offset; else *addr = ((*addr + PDRMASK) & ~PDRMASK) + superpage_offset; } #if defined(PMAP_DEBUG) pmap_pid_dump(int pid) { pmap_t pmap; struct proc *p; int npte = 0; int index; sx_slock(&allproc_lock); FOREACH_PROC_IN_SYSTEM(p) { if (p->p_pid != pid) continue; if (p->p_vmspace) { int i,j; index = 0; pmap = vmspace_pmap(p->p_vmspace); for (i = 0; i < NPDEPTD; i++) { pd_entry_t *pde; pt_entry_t *pte; vm_offset_t base = i << PDRSHIFT; pde = &pmap->pm_pdir[i]; if (pde && pmap_pde_v(pde)) { for (j = 0; j < NPTEPG; j++) { vm_offset_t va = base + (j << PAGE_SHIFT); if (va >= (vm_offset_t) VM_MIN_KERNEL_ADDRESS) { if (index) { index = 0; printf("\n"); } sx_sunlock(&allproc_lock); return (npte); } pte = pmap_pte(pmap, va); if (pte && pmap_pte_v(pte)) { pt_entry_t pa; vm_page_t m; pa = *pte; m = PHYS_TO_VM_PAGE(pa & PG_FRAME); printf("va: 0x%x, pt: 0x%x, h: %d, w: %d, f: 0x%x", va, pa, m->hold_count, m->wire_count, m->flags); npte++; index++; if (index >= 2) { index = 0; printf("\n"); } else { printf(" "); } } } } } } } sx_sunlock(&allproc_lock); return (npte); } #endif #if defined(DEBUG) static void pads(pmap_t pm); void pmap_pvdump(vm_paddr_t pa); /* print address space of pmap*/ static void pads(pmap_t pm) { int i, j; vm_paddr_t va; pt_entry_t *ptep; if (pm == kernel_pmap) return; for (i = 0; i < NPDEPTD; i++) if (pm->pm_pdir[i]) for (j = 0; j < NPTEPG; j++) { va = (i << PDRSHIFT) + (j << PAGE_SHIFT); if (pm == kernel_pmap && va < KERNBASE) continue; if (pm != kernel_pmap && va > UPT_MAX_ADDRESS) continue; ptep = pmap_pte(pm, va); if (pmap_pte_v(ptep)) printf("%x:%x ", va, *ptep); }; } void pmap_pvdump(vm_paddr_t pa) { pv_entry_t pv; pmap_t pmap; vm_page_t m; printf("pa %x", pa); m = PHYS_TO_VM_PAGE(pa); TAILQ_FOREACH(pv, &m->md.pv_list, pv_next) { pmap = PV_PMAP(pv); printf(" -> pmap %p, va %x", (void *)pmap, pv->pv_va); pads(pmap); } printf(" "); } #endif Index: user/ngie/more-tests/sys/i386/i386/swtch.s =================================================================== --- user/ngie/more-tests/sys/i386/i386/swtch.s (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/swtch.s (revision 281504) @@ -1,495 +1,495 @@ /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #include "opt_npx.h" #include "opt_sched.h" #include #include "assym.s" #if defined(SMP) && defined(SCHED_ULE) #define SETOP xchgl #define BLOCK_SPIN(reg) \ movl $blocked_lock,%eax ; \ 100: ; \ lock ; \ cmpxchgl %eax,TD_LOCK(reg) ; \ jne 101f ; \ pause ; \ jmp 100b ; \ 101: #else #define SETOP movl #define BLOCK_SPIN(reg) #endif /*****************************************************************************/ /* Scheduling */ /*****************************************************************************/ .text /* * cpu_throw() * * This is the second half of cpu_switch(). It is used when the current * thread is either a dummy or slated to die, and we no longer care * about its state. This is only a slight optimization and is probably * not worth it anymore. Note that we need to clear the pm_active bits so * we do need the old proc if it still exists. * 0(%esp) = ret * 4(%esp) = oldtd * 8(%esp) = newtd */ ENTRY(cpu_throw) movl PCPU(CPUID), %esi movl 4(%esp),%ecx /* Old thread */ testl %ecx,%ecx /* no thread? */ jz 1f /* release bit from old pm_active */ movl PCPU(CURPMAP), %ebx #ifdef SMP lock #endif btrl %esi, PM_ACTIVE(%ebx) /* clear old */ 1: movl 8(%esp),%ecx /* New thread */ movl TD_PCB(%ecx),%edx movl PCB_CR3(%edx),%eax LOAD_CR3(%eax) /* set bit in new pm_active */ movl TD_PROC(%ecx),%eax movl P_VMSPACE(%eax), %ebx addl $VM_PMAP, %ebx movl %ebx, PCPU(CURPMAP) #ifdef SMP lock #endif btsl %esi, PM_ACTIVE(%ebx) /* set new */ jmp sw1 END(cpu_throw) /* * cpu_switch(old, new) * * Save the current thread state, then select the next thread to run * and load its state. * 0(%esp) = ret * 4(%esp) = oldtd * 8(%esp) = newtd * 12(%esp) = newlock */ ENTRY(cpu_switch) /* Switch to new thread. First, save context. */ movl 4(%esp),%ecx #ifdef INVARIANTS testl %ecx,%ecx /* no thread? */ jz badsw2 /* no, panic */ #endif movl TD_PCB(%ecx),%edx movl (%esp),%eax /* Hardware registers */ movl %eax,PCB_EIP(%edx) movl %ebx,PCB_EBX(%edx) movl %esp,PCB_ESP(%edx) movl %ebp,PCB_EBP(%edx) movl %esi,PCB_ESI(%edx) movl %edi,PCB_EDI(%edx) mov %gs,PCB_GS(%edx) pushfl /* PSL */ popl PCB_PSL(%edx) /* Test if debug registers should be saved. */ testl $PCB_DBREGS,PCB_FLAGS(%edx) jz 1f /* no, skip over */ movl %dr7,%eax /* yes, do the save */ movl %eax,PCB_DR7(%edx) andl $0x0000fc00, %eax /* disable all watchpoints */ movl %eax,%dr7 movl %dr6,%eax movl %eax,PCB_DR6(%edx) movl %dr3,%eax movl %eax,PCB_DR3(%edx) movl %dr2,%eax movl %eax,PCB_DR2(%edx) movl %dr1,%eax movl %eax,PCB_DR1(%edx) movl %dr0,%eax movl %eax,PCB_DR0(%edx) 1: #ifdef DEV_NPX /* have we used fp, and need a save? */ cmpl %ecx,PCPU(FPCURTHREAD) jne 1f pushl PCB_SAVEFPU(%edx) /* h/w bugs make saving complicated */ call npxsave /* do it in a big C function */ popl %eax 1: #endif /* Save is done. Now fire up new thread. Leave old vmspace. */ movl 4(%esp),%edi movl 8(%esp),%ecx /* New thread */ movl 12(%esp),%esi /* New lock */ #ifdef INVARIANTS testl %ecx,%ecx /* no thread? */ jz badsw3 /* no, panic */ #endif movl TD_PCB(%ecx),%edx /* switch address space */ movl PCB_CR3(%edx),%eax -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) cmpl %eax,IdlePDPT /* Kernel address space? */ #else cmpl %eax,IdlePTD /* Kernel address space? */ #endif je sw0 READ_CR3(%ebx) /* The same address space? */ cmpl %ebx,%eax je sw0 LOAD_CR3(%eax) /* new address space */ movl %esi,%eax movl PCPU(CPUID),%esi SETOP %eax,TD_LOCK(%edi) /* Switchout td_lock */ /* Release bit from old pmap->pm_active */ movl PCPU(CURPMAP), %ebx #ifdef SMP lock #endif btrl %esi, PM_ACTIVE(%ebx) /* clear old */ /* Set bit in new pmap->pm_active */ movl TD_PROC(%ecx),%eax /* newproc */ movl P_VMSPACE(%eax), %ebx addl $VM_PMAP, %ebx movl %ebx, PCPU(CURPMAP) #ifdef SMP lock #endif btsl %esi, PM_ACTIVE(%ebx) /* set new */ jmp sw1 sw0: SETOP %esi,TD_LOCK(%edi) /* Switchout td_lock */ sw1: BLOCK_SPIN(%ecx) #ifdef XEN pushl %eax pushl %ecx pushl %edx call xen_handle_thread_switch popl %edx popl %ecx popl %eax /* * XXX set IOPL */ #else /* * At this point, we've switched address spaces and are ready * to load up the rest of the next context. */ cmpl $0, PCB_EXT(%edx) /* has pcb extension? */ je 1f /* If not, use the default */ movl $1, PCPU(PRIVATE_TSS) /* mark use of private tss */ movl PCB_EXT(%edx), %edi /* new tss descriptor */ jmp 2f /* Load it up */ 1: /* * Use the common default TSS instead of our own. * Set our stack pointer into the TSS, it's set to just * below the PCB. In C, common_tss.tss_esp0 = &pcb - 16; */ leal -16(%edx), %ebx /* leave space for vm86 */ movl %ebx, PCPU(COMMON_TSS) + TSS_ESP0 /* * Test this CPU's bit in the bitmap to see if this * CPU was using a private TSS. */ cmpl $0, PCPU(PRIVATE_TSS) /* Already using the common? */ je 3f /* if so, skip reloading */ movl $0, PCPU(PRIVATE_TSS) PCPU_ADDR(COMMON_TSSD, %edi) 2: /* Move correct tss descriptor into GDT slot, then reload tr. */ movl PCPU(TSS_GDT), %ebx /* entry in GDT */ movl 0(%edi), %eax movl 4(%edi), %esi movl %eax, 0(%ebx) movl %esi, 4(%ebx) movl $GPROC0_SEL*8, %esi /* GSEL(GPROC0_SEL, SEL_KPL) */ ltr %si 3: /* Copy the %fs and %gs selectors into this pcpu gdt */ leal PCB_FSD(%edx), %esi movl PCPU(FSGS_GDT), %edi movl 0(%esi), %eax /* %fs selector */ movl 4(%esi), %ebx movl %eax, 0(%edi) movl %ebx, 4(%edi) movl 8(%esi), %eax /* %gs selector, comes straight after */ movl 12(%esi), %ebx movl %eax, 8(%edi) movl %ebx, 12(%edi) #endif /* Restore context. */ movl PCB_EBX(%edx),%ebx movl PCB_ESP(%edx),%esp movl PCB_EBP(%edx),%ebp movl PCB_ESI(%edx),%esi movl PCB_EDI(%edx),%edi movl PCB_EIP(%edx),%eax movl %eax,(%esp) pushl PCB_PSL(%edx) popfl movl %edx, PCPU(CURPCB) movl TD_TID(%ecx),%eax movl %ecx, PCPU(CURTHREAD) /* into next thread */ /* * Determine the LDT to use and load it if is the default one and * that is not the current one. */ movl TD_PROC(%ecx),%eax cmpl $0,P_MD+MD_LDT(%eax) jnz 1f movl _default_ldt,%eax cmpl PCPU(CURRENTLDT),%eax je 2f LLDT(_default_ldt) movl %eax,PCPU(CURRENTLDT) jmp 2f 1: /* Load the LDT when it is not the default one. */ pushl %edx /* Preserve pointer to pcb. */ addl $P_MD,%eax /* Pointer to mdproc is arg. */ pushl %eax call set_user_ldt addl $4,%esp popl %edx 2: /* This must be done after loading the user LDT. */ .globl cpu_switch_load_gs cpu_switch_load_gs: mov PCB_GS(%edx),%gs /* Test if debug registers should be restored. */ testl $PCB_DBREGS,PCB_FLAGS(%edx) jz 1f /* * Restore debug registers. The special code for dr7 is to * preserve the current values of its reserved bits. */ movl PCB_DR6(%edx),%eax movl %eax,%dr6 movl PCB_DR3(%edx),%eax movl %eax,%dr3 movl PCB_DR2(%edx),%eax movl %eax,%dr2 movl PCB_DR1(%edx),%eax movl %eax,%dr1 movl PCB_DR0(%edx),%eax movl %eax,%dr0 movl %dr7,%eax andl $0x0000fc00,%eax movl PCB_DR7(%edx),%ecx andl $~0x0000fc00,%ecx orl %ecx,%eax movl %eax,%dr7 1: ret #ifdef INVARIANTS badsw1: pushal pushl $sw0_1 call panic sw0_1: .asciz "cpu_throw: no newthread supplied" badsw2: pushal pushl $sw0_2 call panic sw0_2: .asciz "cpu_switch: no curthread supplied" badsw3: pushal pushl $sw0_3 call panic sw0_3: .asciz "cpu_switch: no newthread supplied" #endif END(cpu_switch) /* * savectx(pcb) * Update pcb, saving current processor state. */ ENTRY(savectx) /* Fetch PCB. */ movl 4(%esp),%ecx /* Save caller's return address. Child won't execute this routine. */ movl (%esp),%eax movl %eax,PCB_EIP(%ecx) movl %cr3,%eax movl %eax,PCB_CR3(%ecx) movl %ebx,PCB_EBX(%ecx) movl %esp,PCB_ESP(%ecx) movl %ebp,PCB_EBP(%ecx) movl %esi,PCB_ESI(%ecx) movl %edi,PCB_EDI(%ecx) mov %gs,PCB_GS(%ecx) pushfl popl PCB_PSL(%ecx) movl %cr0,%eax movl %eax,PCB_CR0(%ecx) movl %cr2,%eax movl %eax,PCB_CR2(%ecx) movl %cr4,%eax movl %eax,PCB_CR4(%ecx) movl %dr0,%eax movl %eax,PCB_DR0(%ecx) movl %dr1,%eax movl %eax,PCB_DR1(%ecx) movl %dr2,%eax movl %eax,PCB_DR2(%ecx) movl %dr3,%eax movl %eax,PCB_DR3(%ecx) movl %dr6,%eax movl %eax,PCB_DR6(%ecx) movl %dr7,%eax movl %eax,PCB_DR7(%ecx) mov %ds,PCB_DS(%ecx) mov %es,PCB_ES(%ecx) mov %fs,PCB_FS(%ecx) mov %ss,PCB_SS(%ecx) sgdt PCB_GDT(%ecx) sidt PCB_IDT(%ecx) sldt PCB_LDT(%ecx) str PCB_TR(%ecx) movl $1,%eax ret END(savectx) /* * resumectx(pcb) __fastcall * Resuming processor state from pcb. */ ENTRY(resumectx) /* Restore GDT. */ lgdt PCB_GDT(%ecx) /* Restore segment registers */ movzwl PCB_DS(%ecx),%eax mov %ax,%ds movzwl PCB_ES(%ecx),%eax mov %ax,%es movzwl PCB_FS(%ecx),%eax mov %ax,%fs movzwl PCB_GS(%ecx),%eax movw %ax,%gs movzwl PCB_SS(%ecx),%eax mov %ax,%ss /* Restore CR2, CR4, CR3 and CR0 */ movl PCB_CR2(%ecx),%eax movl %eax,%cr2 movl PCB_CR4(%ecx),%eax movl %eax,%cr4 movl PCB_CR3(%ecx),%eax movl %eax,%cr3 movl PCB_CR0(%ecx),%eax movl %eax,%cr0 jmp 1f 1: /* Restore descriptor tables */ lidt PCB_IDT(%ecx) lldt PCB_LDT(%ecx) #define SDT_SYS386TSS 9 #define SDT_SYS386BSY 11 /* Clear "task busy" bit and reload TR */ movl PCPU(TSS_GDT),%eax andb $(~SDT_SYS386BSY | SDT_SYS386TSS),5(%eax) movzwl PCB_TR(%ecx),%eax ltr %ax #undef SDT_SYS386TSS #undef SDT_SYS386BSY /* Restore debug registers */ movl PCB_DR0(%ecx),%eax movl %eax,%dr0 movl PCB_DR1(%ecx),%eax movl %eax,%dr1 movl PCB_DR2(%ecx),%eax movl %eax,%dr2 movl PCB_DR3(%ecx),%eax movl %eax,%dr3 movl PCB_DR6(%ecx),%eax movl %eax,%dr6 movl PCB_DR7(%ecx),%eax movl %eax,%dr7 /* Restore other registers */ movl PCB_EDI(%ecx),%edi movl PCB_ESI(%ecx),%esi movl PCB_EBP(%ecx),%ebp movl PCB_ESP(%ecx),%esp movl PCB_EBX(%ecx),%ebx /* reload code selector by turning return into intersegmental return */ pushl PCB_EIP(%ecx) movl $KCSEL,4(%esp) xorl %eax,%eax lret END(resumectx) Index: user/ngie/more-tests/sys/i386/i386/trap.c =================================================================== --- user/ngie/more-tests/sys/i386/i386/trap.c (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/trap.c (revision 281504) @@ -1,1165 +1,1165 @@ /*- * Copyright (C) 1994, David Greenman * Copyright (c) 1990, 1993 * The Regents of the University of California. All rights reserved. * * This code is derived from software contributed to Berkeley by * the University of Utah, and William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)trap.c 7.4 (Berkeley) 5/13/91 */ #include __FBSDID("$FreeBSD$"); /* * 386 Trap and System call handling */ #include "opt_clock.h" #include "opt_cpu.h" #include "opt_hwpmc_hooks.h" #include "opt_isa.h" #include "opt_kdb.h" #include "opt_npx.h" #include "opt_trap.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef HWPMC_HOOKS #include PMC_SOFT_DEFINE( , , page_fault, all); PMC_SOFT_DEFINE( , , page_fault, read); PMC_SOFT_DEFINE( , , page_fault, write); #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef SMP #include #endif #include #include #ifdef POWERFAIL_NMI #include #include #endif #ifdef KDTRACE_HOOKS #include #endif extern void trap(struct trapframe *frame); extern void syscall(struct trapframe *frame); static int trap_pfault(struct trapframe *, int, vm_offset_t); static void trap_fatal(struct trapframe *, vm_offset_t); void dblfault_handler(void); extern inthand_t IDTVEC(lcall_syscall); #define MAX_TRAP_MSG 32 static char *trap_msg[] = { "", /* 0 unused */ "privileged instruction fault", /* 1 T_PRIVINFLT */ "", /* 2 unused */ "breakpoint instruction fault", /* 3 T_BPTFLT */ "", /* 4 unused */ "", /* 5 unused */ "arithmetic trap", /* 6 T_ARITHTRAP */ "", /* 7 unused */ "", /* 8 unused */ "general protection fault", /* 9 T_PROTFLT */ "trace trap", /* 10 T_TRCTRAP */ "", /* 11 unused */ "page fault", /* 12 T_PAGEFLT */ "", /* 13 unused */ "alignment fault", /* 14 T_ALIGNFLT */ "", /* 15 unused */ "", /* 16 unused */ "", /* 17 unused */ "integer divide fault", /* 18 T_DIVIDE */ "non-maskable interrupt trap", /* 19 T_NMI */ "overflow trap", /* 20 T_OFLOW */ "FPU bounds check fault", /* 21 T_BOUND */ "FPU device not available", /* 22 T_DNA */ "double fault", /* 23 T_DOUBLEFLT */ "FPU operand fetch fault", /* 24 T_FPOPFLT */ "invalid TSS fault", /* 25 T_TSSFLT */ "segment not present fault", /* 26 T_SEGNPFLT */ "stack fault", /* 27 T_STKFLT */ "machine check trap", /* 28 T_MCHK */ "SIMD floating-point exception", /* 29 T_XMMFLT */ "reserved (unknown) fault", /* 30 T_RESERVED */ "", /* 31 unused (reserved) */ "DTrace pid return trap", /* 32 T_DTRACE_RET */ }; #if defined(I586_CPU) && !defined(NO_F00F_HACK) int has_f00f_bug = 0; /* Initialized so that it can be patched. */ #endif #ifdef KDB static int kdb_on_nmi = 1; SYSCTL_INT(_machdep, OID_AUTO, kdb_on_nmi, CTLFLAG_RWTUN, &kdb_on_nmi, 0, "Go to KDB on NMI"); #endif static int panic_on_nmi = 1; SYSCTL_INT(_machdep, OID_AUTO, panic_on_nmi, CTLFLAG_RWTUN, &panic_on_nmi, 0, "Panic on NMI"); static int prot_fault_translation = 0; SYSCTL_INT(_machdep, OID_AUTO, prot_fault_translation, CTLFLAG_RW, &prot_fault_translation, 0, "Select signal to deliver on protection fault"); static int uprintf_signal; SYSCTL_INT(_machdep, OID_AUTO, uprintf_signal, CTLFLAG_RW, &uprintf_signal, 0, "Print debugging information on trap signal to ctty"); /* * Exception, fault, and trap interface to the FreeBSD kernel. * This common code is called from assembly language IDT gate entry * routines that prepare a suitable stack frame, and restore this * frame after the exception has been processed. */ void trap(struct trapframe *frame) { #ifdef KDTRACE_HOOKS struct reg regs; #endif struct thread *td = curthread; struct proc *p = td->td_proc; int i = 0, ucode = 0, code; u_int type; register_t addr = 0; vm_offset_t eva; ksiginfo_t ksi; #ifdef POWERFAIL_NMI static int lastalert = 0; #endif PCPU_INC(cnt.v_trap); type = frame->tf_trapno; #ifdef SMP /* Handler for NMI IPIs used for stopping CPUs. */ if (type == T_NMI) { if (ipi_nmi_handler() == 0) goto out; } #endif /* SMP */ #ifdef KDB if (kdb_active) { kdb_reenter(); goto out; } #endif if (type == T_RESERVED) { trap_fatal(frame, 0); goto out; } #ifdef HWPMC_HOOKS /* * CPU PMCs interrupt using an NMI so we check for that first. * If the HWPMC module is active, 'pmc_hook' will point to * the function to be called. A return value of '1' from the * hook means that the NMI was handled by it and that we can * return immediately. */ if (type == T_NMI && pmc_intr && (*pmc_intr)(PCPU_GET(cpuid), frame)) goto out; #endif if (type == T_MCHK) { mca_intr(); goto out; } #ifdef KDTRACE_HOOKS /* * A trap can occur while DTrace executes a probe. Before * executing the probe, DTrace blocks re-scheduling and sets * a flag in its per-cpu flags to indicate that it doesn't * want to fault. On returning from the probe, the no-fault * flag is cleared and finally re-scheduling is enabled. */ if ((type == T_PROTFLT || type == T_PAGEFLT) && dtrace_trap_func != NULL && (*dtrace_trap_func)(frame, type)) goto out; #endif if ((frame->tf_eflags & PSL_I) == 0) { /* * Buggy application or kernel code has disabled * interrupts and then trapped. Enabling interrupts * now is wrong, but it is better than running with * interrupts disabled until they are accidentally * enabled later. */ if (ISPL(frame->tf_cs) == SEL_UPL || (frame->tf_eflags & PSL_VM)) uprintf( "pid %ld (%s): trap %d with interrupts disabled\n", (long)curproc->p_pid, curthread->td_name, type); else if (type != T_NMI && type != T_BPTFLT && type != T_TRCTRAP && frame->tf_eip != (int)cpu_switch_load_gs) { /* * XXX not quite right, since this may be for a * multiple fault in user mode. */ printf("kernel trap %d with interrupts disabled\n", type); /* * Page faults need interrupts disabled until later, * and we shouldn't enable interrupts while holding * a spin lock. */ if (type != T_PAGEFLT && td->td_md.md_spinlock_count == 0) enable_intr(); } } eva = 0; code = frame->tf_err; if (type == T_PAGEFLT) { /* * For some Cyrix CPUs, %cr2 is clobbered by * interrupts. This problem is worked around by using * an interrupt gate for the pagefault handler. We * are finally ready to read %cr2 and conditionally * reenable interrupts. If we hold a spin lock, then * we must not reenable interrupts. This might be a * spurious page fault. */ eva = rcr2(); if (td->td_md.md_spinlock_count == 0) enable_intr(); } if ((ISPL(frame->tf_cs) == SEL_UPL) || ((frame->tf_eflags & PSL_VM) && !(curpcb->pcb_flags & PCB_VM86CALL))) { /* user trap */ td->td_pticks = 0; td->td_frame = frame; addr = frame->tf_eip; if (td->td_ucred != p->p_ucred) cred_update_thread(td); switch (type) { case T_PRIVINFLT: /* privileged instruction fault */ i = SIGILL; ucode = ILL_PRVOPC; break; case T_BPTFLT: /* bpt instruction fault */ case T_TRCTRAP: /* trace trap */ enable_intr(); #ifdef KDTRACE_HOOKS if (type == T_BPTFLT) { fill_frame_regs(frame, ®s); if (dtrace_pid_probe_ptr != NULL && dtrace_pid_probe_ptr(®s) == 0) goto out; } #endif frame->tf_eflags &= ~PSL_T; i = SIGTRAP; ucode = (type == T_TRCTRAP ? TRAP_TRACE : TRAP_BRKPT); break; case T_ARITHTRAP: /* arithmetic trap */ #ifdef DEV_NPX ucode = npxtrap_x87(); if (ucode == -1) goto userout; #else ucode = 0; #endif i = SIGFPE; break; /* * The following two traps can happen in * vm86 mode, and, if so, we want to handle * them specially. */ case T_PROTFLT: /* general protection fault */ case T_STKFLT: /* stack fault */ if (frame->tf_eflags & PSL_VM) { i = vm86_emulate((struct vm86frame *)frame); if (i == 0) goto user; break; } i = SIGBUS; ucode = (type == T_PROTFLT) ? BUS_OBJERR : BUS_ADRERR; break; case T_SEGNPFLT: /* segment not present fault */ i = SIGBUS; ucode = BUS_ADRERR; break; case T_TSSFLT: /* invalid TSS fault */ i = SIGBUS; ucode = BUS_OBJERR; break; case T_ALIGNFLT: i = SIGBUS; ucode = BUS_ADRALN; break; case T_DOUBLEFLT: /* double fault */ default: i = SIGBUS; ucode = BUS_OBJERR; break; case T_PAGEFLT: /* page fault */ i = trap_pfault(frame, TRUE, eva); #if defined(I586_CPU) && !defined(NO_F00F_HACK) if (i == -2) { /* * The f00f hack workaround has triggered, so * treat the fault as an illegal instruction * (T_PRIVINFLT) instead of a page fault. */ type = frame->tf_trapno = T_PRIVINFLT; /* Proceed as in that case. */ ucode = ILL_PRVOPC; i = SIGILL; break; } #endif if (i == -1) goto userout; if (i == 0) goto user; if (i == SIGSEGV) ucode = SEGV_MAPERR; else { if (prot_fault_translation == 0) { /* * Autodetect. * This check also covers the images * without the ABI-tag ELF note. */ if (SV_CURPROC_ABI() == SV_ABI_FREEBSD && p->p_osrel >= P_OSREL_SIGSEGV) { i = SIGSEGV; ucode = SEGV_ACCERR; } else { i = SIGBUS; ucode = BUS_PAGE_FAULT; } } else if (prot_fault_translation == 1) { /* * Always compat mode. */ i = SIGBUS; ucode = BUS_PAGE_FAULT; } else { /* * Always SIGSEGV mode. */ i = SIGSEGV; ucode = SEGV_ACCERR; } } addr = eva; break; case T_DIVIDE: /* integer divide fault */ ucode = FPE_INTDIV; i = SIGFPE; break; #ifdef DEV_ISA case T_NMI: #ifdef POWERFAIL_NMI #ifndef TIMER_FREQ # define TIMER_FREQ 1193182 #endif if (time_second - lastalert > 10) { log(LOG_WARNING, "NMI: power fail\n"); sysbeep(880, hz); lastalert = time_second; } goto userout; #else /* !POWERFAIL_NMI */ /* machine/parity/power fail/"kitchen sink" faults */ if (isa_nmi(code) == 0) { #ifdef KDB /* * NMI can be hooked up to a pushbutton * for debugging. */ if (kdb_on_nmi) { printf ("NMI ... going to debugger\n"); kdb_trap(type, 0, frame); } #endif /* KDB */ goto userout; } else if (panic_on_nmi) panic("NMI indicates hardware failure"); break; #endif /* POWERFAIL_NMI */ #endif /* DEV_ISA */ case T_OFLOW: /* integer overflow fault */ ucode = FPE_INTOVF; i = SIGFPE; break; case T_BOUND: /* bounds check fault */ ucode = FPE_FLTSUB; i = SIGFPE; break; case T_DNA: #ifdef DEV_NPX KASSERT(PCB_USER_FPU(td->td_pcb), ("kernel FPU ctx has leaked")); /* transparent fault (due to context switch "late") */ if (npxdna()) goto userout; #endif uprintf("pid %d killed due to lack of floating point\n", p->p_pid); i = SIGKILL; ucode = 0; break; case T_FPOPFLT: /* FPU operand fetch fault */ ucode = ILL_COPROC; i = SIGILL; break; case T_XMMFLT: /* SIMD floating-point exception */ #if defined(DEV_NPX) && !defined(CPU_DISABLE_SSE) && defined(I686_CPU) ucode = npxtrap_sse(); if (ucode == -1) goto userout; #else ucode = 0; #endif i = SIGFPE; break; #ifdef KDTRACE_HOOKS case T_DTRACE_RET: enable_intr(); fill_frame_regs(frame, ®s); if (dtrace_return_probe_ptr != NULL && dtrace_return_probe_ptr(®s) == 0) goto out; break; #endif } } else { /* kernel trap */ KASSERT(cold || td->td_ucred != NULL, ("kernel trap doesn't have ucred")); switch (type) { case T_PAGEFLT: /* page fault */ (void) trap_pfault(frame, FALSE, eva); goto out; case T_DNA: #ifdef DEV_NPX KASSERT(!PCB_USER_FPU(td->td_pcb), ("Unregistered use of FPU in kernel")); if (npxdna()) goto out; #endif break; case T_ARITHTRAP: /* arithmetic trap */ case T_XMMFLT: /* SIMD floating-point exception */ case T_FPOPFLT: /* FPU operand fetch fault */ /* * XXXKIB for now disable any FPU traps in kernel * handler registration seems to be overkill */ trap_fatal(frame, 0); goto out; /* * The following two traps can happen in * vm86 mode, and, if so, we want to handle * them specially. */ case T_PROTFLT: /* general protection fault */ case T_STKFLT: /* stack fault */ if (frame->tf_eflags & PSL_VM) { i = vm86_emulate((struct vm86frame *)frame); if (i != 0) /* * returns to original process */ vm86_trap((struct vm86frame *)frame); goto out; } if (type == T_STKFLT) break; /* FALL THROUGH */ case T_SEGNPFLT: /* segment not present fault */ if (curpcb->pcb_flags & PCB_VM86CALL) break; /* * Invalid %fs's and %gs's can be created using * procfs or PT_SETREGS or by invalidating the * underlying LDT entry. This causes a fault * in kernel mode when the kernel attempts to * switch contexts. Lose the bad context * (XXX) so that we can continue, and generate * a signal. */ if (frame->tf_eip == (int)cpu_switch_load_gs) { curpcb->pcb_gs = 0; #if 0 PROC_LOCK(p); kern_psignal(p, SIGBUS); PROC_UNLOCK(p); #endif goto out; } if (td->td_intr_nesting_level != 0) break; /* * Invalid segment selectors and out of bounds * %eip's and %esp's can be set up in user mode. * This causes a fault in kernel mode when the * kernel tries to return to user mode. We want * to get this fault so that we can fix the * problem here and not have to check all the * selectors and pointers when the user changes * them. */ if (frame->tf_eip == (int)doreti_iret) { frame->tf_eip = (int)doreti_iret_fault; goto out; } if (frame->tf_eip == (int)doreti_popl_ds) { frame->tf_eip = (int)doreti_popl_ds_fault; goto out; } if (frame->tf_eip == (int)doreti_popl_es) { frame->tf_eip = (int)doreti_popl_es_fault; goto out; } if (frame->tf_eip == (int)doreti_popl_fs) { frame->tf_eip = (int)doreti_popl_fs_fault; goto out; } if (curpcb->pcb_onfault != NULL) { frame->tf_eip = (int)curpcb->pcb_onfault; goto out; } break; case T_TSSFLT: /* * PSL_NT can be set in user mode and isn't cleared * automatically when the kernel is entered. This * causes a TSS fault when the kernel attempts to * `iret' because the TSS link is uninitialized. We * want to get this fault so that we can fix the * problem here and not every time the kernel is * entered. */ if (frame->tf_eflags & PSL_NT) { frame->tf_eflags &= ~PSL_NT; goto out; } break; case T_TRCTRAP: /* trace trap */ if (frame->tf_eip == (int)IDTVEC(lcall_syscall)) { /* * We've just entered system mode via the * syscall lcall. Continue single stepping * silently until the syscall handler has * saved the flags. */ goto out; } if (frame->tf_eip == (int)IDTVEC(lcall_syscall) + 1) { /* * The syscall handler has now saved the * flags. Stop single stepping it. */ frame->tf_eflags &= ~PSL_T; goto out; } /* * Ignore debug register trace traps due to * accesses in the user's address space, which * can happen under several conditions such as * if a user sets a watchpoint on a buffer and * then passes that buffer to a system call. * We still want to get TRCTRAPS for addresses * in kernel space because that is useful when * debugging the kernel. */ if (user_dbreg_trap() && !(curpcb->pcb_flags & PCB_VM86CALL)) { /* * Reset breakpoint bits because the * processor doesn't */ load_dr6(rdr6() & 0xfffffff0); goto out; } /* * FALLTHROUGH (TRCTRAP kernel mode, kernel address) */ case T_BPTFLT: /* * If KDB is enabled, let it handle the debugger trap. * Otherwise, debugger traps "can't happen". */ #ifdef KDB if (kdb_trap(type, 0, frame)) goto out; #endif break; #ifdef DEV_ISA case T_NMI: #ifdef POWERFAIL_NMI if (time_second - lastalert > 10) { log(LOG_WARNING, "NMI: power fail\n"); sysbeep(880, hz); lastalert = time_second; } goto out; #else /* !POWERFAIL_NMI */ /* machine/parity/power fail/"kitchen sink" faults */ if (isa_nmi(code) == 0) { #ifdef KDB /* * NMI can be hooked up to a pushbutton * for debugging. */ if (kdb_on_nmi) { printf ("NMI ... going to debugger\n"); kdb_trap(type, 0, frame); } #endif /* KDB */ goto out; } else if (panic_on_nmi == 0) goto out; /* FALLTHROUGH */ #endif /* POWERFAIL_NMI */ #endif /* DEV_ISA */ } trap_fatal(frame, eva); goto out; } /* Translate fault for emulators (e.g. Linux) */ if (*p->p_sysent->sv_transtrap) i = (*p->p_sysent->sv_transtrap)(i, type); ksiginfo_init_trap(&ksi); ksi.ksi_signo = i; ksi.ksi_code = ucode; ksi.ksi_addr = (void *)addr; ksi.ksi_trapno = type; if (uprintf_signal) { uprintf("pid %d comm %s: signal %d err %x code %d type %d " "addr 0x%x esp 0x%08x eip 0x%08x " "<%02x %02x %02x %02x %02x %02x %02x %02x>\n", p->p_pid, p->p_comm, i, frame->tf_err, ucode, type, addr, frame->tf_esp, frame->tf_eip, fubyte((void *)(frame->tf_eip + 0)), fubyte((void *)(frame->tf_eip + 1)), fubyte((void *)(frame->tf_eip + 2)), fubyte((void *)(frame->tf_eip + 3)), fubyte((void *)(frame->tf_eip + 4)), fubyte((void *)(frame->tf_eip + 5)), fubyte((void *)(frame->tf_eip + 6)), fubyte((void *)(frame->tf_eip + 7))); } KASSERT((read_eflags() & PSL_I) != 0, ("interrupts disabled")); trapsignal(td, &ksi); #ifdef DEBUG if (type <= MAX_TRAP_MSG) { uprintf("fatal process exception: %s", trap_msg[type]); if ((type == T_PAGEFLT) || (type == T_PROTFLT)) uprintf(", fault VA = 0x%lx", (u_long)eva); uprintf("\n"); } #endif user: userret(td, frame); KASSERT(PCB_USER_FPU(td->td_pcb), ("Return from trap with kernel FPU ctx leaked")); userout: out: return; } static int trap_pfault(frame, usermode, eva) struct trapframe *frame; int usermode; vm_offset_t eva; { vm_offset_t va; struct vmspace *vm; vm_map_t map; int rv = 0; vm_prot_t ftype; struct thread *td = curthread; struct proc *p = td->td_proc; if (__predict_false((td->td_pflags & TDP_NOFAULTING) != 0)) { /* * Due to both processor errata and lazy TLB invalidation when * access restrictions are removed from virtual pages, memory * accesses that are allowed by the physical mapping layer may * nonetheless cause one spurious page fault per virtual page. * When the thread is executing a "no faulting" section that * is bracketed by vm_fault_{disable,enable}_pagefaults(), * every page fault is treated as a spurious page fault, * unless it accesses the same virtual address as the most * recent page fault within the same "no faulting" section. */ if (td->td_md.md_spurflt_addr != eva || (td->td_pflags & TDP_RESETSPUR) != 0) { /* * Do nothing to the TLB. A stale TLB entry is * flushed automatically by a page fault. */ td->td_md.md_spurflt_addr = eva; td->td_pflags &= ~TDP_RESETSPUR; return (0); } } else { /* * If we get a page fault while in a critical section, then * it is most likely a fatal kernel page fault. The kernel * is already going to panic trying to get a sleep lock to * do the VM lookup, so just consider it a fatal trap so the * kernel can print out a useful trap message and even get * to the debugger. * * If we get a page fault while holding a non-sleepable * lock, then it is most likely a fatal kernel page fault. * If WITNESS is enabled, then it's going to whine about * bogus LORs with various VM locks, so just skip to the * fatal trap handling directly. */ if (td->td_critnest != 0 || WITNESS_CHECK(WARN_SLEEPOK | WARN_GIANTOK, NULL, "Kernel page fault") != 0) { trap_fatal(frame, eva); return (-1); } } va = trunc_page(eva); if (va >= KERNBASE) { /* * Don't allow user-mode faults in kernel address space. * An exception: if the faulting address is the invalid * instruction entry in the IDT, then the Intel Pentium * F00F bug workaround was triggered, and we need to * treat it is as an illegal instruction, and not a page * fault. */ #if defined(I586_CPU) && !defined(NO_F00F_HACK) if ((eva == (unsigned int)&idt[6]) && has_f00f_bug) return (-2); #endif if (usermode) goto nogo; map = kernel_map; } else { /* * This is a fault on non-kernel virtual memory. If either * p or p->p_vmspace is NULL, then the fault is fatal. */ if (p == NULL || (vm = p->p_vmspace) == NULL) goto nogo; map = &vm->vm_map; /* * When accessing a user-space address, kernel must be * ready to accept the page fault, and provide a * handling routine. Since accessing the address * without the handler is a bug, do not try to handle * it normally, and panic immediately. */ if (!usermode && (td->td_intr_nesting_level != 0 || curpcb->pcb_onfault == NULL)) { trap_fatal(frame, eva); return (-1); } } /* * PGEX_I is defined only if the execute disable bit capability is * supported and enabled. */ if (frame->tf_err & PGEX_W) ftype = VM_PROT_WRITE; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) else if ((frame->tf_err & PGEX_I) && pg_nx != 0) ftype = VM_PROT_EXECUTE; #endif else ftype = VM_PROT_READ; if (map != kernel_map) { /* * Keep swapout from messing with us during this * critical time. */ PROC_LOCK(p); ++p->p_lock; PROC_UNLOCK(p); /* Fault in the user page: */ rv = vm_fault(map, va, ftype, VM_FAULT_NORMAL); PROC_LOCK(p); --p->p_lock; PROC_UNLOCK(p); } else { /* * Don't have to worry about process locking or stacks in the * kernel. */ rv = vm_fault(map, va, ftype, VM_FAULT_NORMAL); } if (rv == KERN_SUCCESS) { #ifdef HWPMC_HOOKS if (ftype == VM_PROT_READ || ftype == VM_PROT_WRITE) { PMC_SOFT_CALL_TF( , , page_fault, all, frame); if (ftype == VM_PROT_READ) PMC_SOFT_CALL_TF( , , page_fault, read, frame); else PMC_SOFT_CALL_TF( , , page_fault, write, frame); } #endif return (0); } nogo: if (!usermode) { if (td->td_intr_nesting_level == 0 && curpcb->pcb_onfault != NULL) { frame->tf_eip = (int)curpcb->pcb_onfault; return (0); } trap_fatal(frame, eva); return (-1); } return ((rv == KERN_PROTECTION_FAILURE) ? SIGBUS : SIGSEGV); } static void trap_fatal(frame, eva) struct trapframe *frame; vm_offset_t eva; { int code, ss, esp; u_int type; struct soft_segment_descriptor softseg; char *msg; code = frame->tf_err; type = frame->tf_trapno; sdtossd(&gdt[IDXSEL(frame->tf_cs & 0xffff)].sd, &softseg); if (type <= MAX_TRAP_MSG) msg = trap_msg[type]; else msg = "UNKNOWN"; printf("\n\nFatal trap %d: %s while in %s mode\n", type, msg, frame->tf_eflags & PSL_VM ? "vm86" : ISPL(frame->tf_cs) == SEL_UPL ? "user" : "kernel"); #ifdef SMP /* two separate prints in case of a trap on an unmapped page */ printf("cpuid = %d; ", PCPU_GET(cpuid)); printf("apic id = %02x\n", PCPU_GET(apic_id)); #endif if (type == T_PAGEFLT) { printf("fault virtual address = 0x%x\n", eva); printf("fault code = %s %s, %s\n", code & PGEX_U ? "user" : "supervisor", code & PGEX_W ? "write" : "read", code & PGEX_P ? "protection violation" : "page not present"); } printf("instruction pointer = 0x%x:0x%x\n", frame->tf_cs & 0xffff, frame->tf_eip); if ((ISPL(frame->tf_cs) == SEL_UPL) || (frame->tf_eflags & PSL_VM)) { ss = frame->tf_ss & 0xffff; esp = frame->tf_esp; } else { ss = GSEL(GDATA_SEL, SEL_KPL); esp = (int)&frame->tf_esp; } printf("stack pointer = 0x%x:0x%x\n", ss, esp); printf("frame pointer = 0x%x:0x%x\n", ss, frame->tf_ebp); printf("code segment = base 0x%x, limit 0x%x, type 0x%x\n", softseg.ssd_base, softseg.ssd_limit, softseg.ssd_type); printf(" = DPL %d, pres %d, def32 %d, gran %d\n", softseg.ssd_dpl, softseg.ssd_p, softseg.ssd_def32, softseg.ssd_gran); printf("processor eflags = "); if (frame->tf_eflags & PSL_T) printf("trace trap, "); if (frame->tf_eflags & PSL_I) printf("interrupt enabled, "); if (frame->tf_eflags & PSL_NT) printf("nested task, "); if (frame->tf_eflags & PSL_RF) printf("resume, "); if (frame->tf_eflags & PSL_VM) printf("vm86, "); printf("IOPL = %d\n", (frame->tf_eflags & PSL_IOPL) >> 12); printf("current process = "); if (curproc) { printf("%lu (%s)\n", (u_long)curproc->p_pid, curthread->td_name); } else { printf("Idle\n"); } #ifdef KDB if (debugger_on_panic || kdb_active) { frame->tf_err = eva; /* smuggle fault address to ddb */ if (kdb_trap(type, 0, frame)) { frame->tf_err = code; /* restore error code */ return; } frame->tf_err = code; /* restore error code */ } #endif printf("trap number = %d\n", type); if (type <= MAX_TRAP_MSG) panic("%s", trap_msg[type]); else panic("unknown/reserved trap"); } /* * Double fault handler. Called when a fault occurs while writing * a frame for a trap/exception onto the stack. This usually occurs * when the stack overflows (such is the case with infinite recursion, * for example). * * XXX Note that the current PTD gets replaced by IdlePTD when the * task switch occurs. This means that the stack that was active at * the time of the double fault is not available at unless * the machine was idle when the double fault occurred. The downside * of this is that "trace " in ddb won't work. */ void dblfault_handler() { #ifdef KDTRACE_HOOKS if (dtrace_doubletrap_func != NULL) (*dtrace_doubletrap_func)(); #endif printf("\nFatal double fault:\n"); printf("eip = 0x%x\n", PCPU_GET(common_tss.tss_eip)); printf("esp = 0x%x\n", PCPU_GET(common_tss.tss_esp)); printf("ebp = 0x%x\n", PCPU_GET(common_tss.tss_ebp)); #ifdef SMP /* two separate prints in case of a trap on an unmapped page */ printf("cpuid = %d; ", PCPU_GET(cpuid)); printf("apic id = %02x\n", PCPU_GET(apic_id)); #endif panic("double fault"); } int cpu_fetch_syscall_args(struct thread *td, struct syscall_args *sa) { struct proc *p; struct trapframe *frame; caddr_t params; long tmp; int error; p = td->td_proc; frame = td->td_frame; params = (caddr_t)frame->tf_esp + sizeof(int); sa->code = frame->tf_eax; /* * Need to check if this is a 32 bit or 64 bit syscall. */ if (sa->code == SYS_syscall) { /* * Code is first argument, followed by actual args. */ error = fueword(params, &tmp); if (error == -1) return (EFAULT); sa->code = tmp; params += sizeof(int); } else if (sa->code == SYS___syscall) { /* * Like syscall, but code is a quad, so as to maintain * quad alignment for the rest of the arguments. */ error = fueword(params, &tmp); if (error == -1) return (EFAULT); sa->code = tmp; params += sizeof(quad_t); } if (p->p_sysent->sv_mask) sa->code &= p->p_sysent->sv_mask; if (sa->code >= p->p_sysent->sv_size) sa->callp = &p->p_sysent->sv_table[0]; else sa->callp = &p->p_sysent->sv_table[sa->code]; sa->narg = sa->callp->sy_narg; if (params != NULL && sa->narg != 0) error = copyin(params, (caddr_t)sa->args, (u_int)(sa->narg * sizeof(int))); else error = 0; if (error == 0) { td->td_retval[0] = 0; td->td_retval[1] = frame->tf_edx; } return (error); } #include "../../kern/subr_syscall.c" /* * syscall - system call request C handler. A system call is * essentially treated as a trap by reusing the frame layout. */ void syscall(struct trapframe *frame) { struct thread *td; struct syscall_args sa; register_t orig_tf_eflags; int error; ksiginfo_t ksi; #ifdef DIAGNOSTIC if (ISPL(frame->tf_cs) != SEL_UPL) { panic("syscall"); /* NOT REACHED */ } #endif orig_tf_eflags = frame->tf_eflags; td = curthread; td->td_frame = frame; error = syscallenter(td, &sa); /* * Traced syscall. */ if ((orig_tf_eflags & PSL_T) && !(orig_tf_eflags & PSL_VM)) { frame->tf_eflags &= ~PSL_T; ksiginfo_init_trap(&ksi); ksi.ksi_signo = SIGTRAP; ksi.ksi_code = TRAP_TRACE; ksi.ksi_addr = (void *)frame->tf_eip; trapsignal(td, &ksi); } KASSERT(PCB_USER_FPU(td->td_pcb), ("System call %s returning with kernel FPU ctx leaked", syscallname(td->td_proc, sa.code))); KASSERT(td->td_pcb->pcb_save == get_pcb_user_save_td(td), ("System call %s returning with mangled pcb_save", syscallname(td->td_proc, sa.code))); syscallret(td, error, &sa); } Index: user/ngie/more-tests/sys/i386/i386/vm86bios.s =================================================================== --- user/ngie/more-tests/sys/i386/i386/vm86bios.s (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/vm86bios.s (revision 281504) @@ -1,177 +1,177 @@ /*- * Copyright (c) 1998 Jonathan Lemon * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #include "opt_npx.h" #include /* miscellaneous asm macros */ #include #include "assym.s" #define SCR_NEWPTD PCB_ESI /* readability macros */ #define SCR_VMFRAME PCB_EBP /* see vm86.c for explanation */ #define SCR_STACK PCB_ESP #define SCR_PGTABLE PCB_EBX #define SCR_ARGFRAME PCB_EIP #define SCR_TSS0 PCB_VM86 #define SCR_TSS1 (PCB_VM86+4) .data ALIGN_DATA .globl vm86pcb vm86pcb: .long 0 .text /* * vm86_bioscall(struct trapframe_vm86 *vm86) */ ENTRY(vm86_bioscall) movl vm86pcb,%edx /* scratch data area */ movl 4(%esp),%eax movl %eax,SCR_ARGFRAME(%edx) /* save argument pointer */ pushl %ebx pushl %ebp pushl %esi pushl %edi pushl %gs #ifdef DEV_NPX pushfl cli movl PCPU(CURTHREAD),%ecx cmpl %ecx,PCPU(FPCURTHREAD) /* do we need to save fp? */ jne 1f pushl %edx movl TD_PCB(%ecx),%ecx pushl PCB_SAVEFPU(%ecx) call npxsave addl $4,%esp popl %edx /* recover our pcb */ 1: popfl #endif movl SCR_VMFRAME(%edx),%ebx /* target frame location */ movl %ebx,%edi /* destination */ movl SCR_ARGFRAME(%edx),%esi /* source (set on entry) */ movl $VM86_FRAMESIZE/4,%ecx /* sizeof(struct vm86frame)/4 */ cld rep movsl /* copy frame to new stack */ movl PCPU(CURPCB),%eax pushl %eax /* save curpcb */ movl %edx,PCPU(CURPCB) /* set curpcb to vm86pcb */ movl PCPU(TSS_GDT),%ebx /* entry in GDT */ movl 0(%ebx),%eax movl %eax,SCR_TSS0(%edx) /* save first word */ movl 4(%ebx),%eax andl $~0x200, %eax /* flip 386BSY -> 386TSS */ movl %eax,SCR_TSS1(%edx) /* save second word */ movl PCB_EXT(%edx),%edi /* vm86 tssd entry */ movl 0(%edi),%eax movl %eax,0(%ebx) movl 4(%edi),%eax movl %eax,4(%ebx) movl $GPROC0_SEL*8,%esi /* GSEL(entry, SEL_KPL) */ ltr %si movl %cr3,%eax pushl %eax /* save address space */ movl IdlePTD,%ecx movl %ecx,%ebx addl $KERNBASE,%ebx /* va of Idle PTD */ movl 0(%ebx),%eax pushl %eax /* old ptde != 0 when booting */ pushl %ebx /* keep for reuse */ movl %esp,SCR_STACK(%edx) /* save current stack location */ movl SCR_NEWPTD(%edx),%eax /* mapping for vm86 page table */ movl %eax,0(%ebx) /* ... install as PTD entry 0 */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) movl IdlePDPT,%ecx #endif movl %ecx,%cr3 /* new page tables */ movl SCR_VMFRAME(%edx),%esp /* switch to new stack */ pushl %esp call vm86_prepcall /* finish setup */ add $4, %esp /* * Return via doreti */ MEXITCOUNT jmp doreti /* * vm86_biosret(struct trapframe_vm86 *vm86) */ ENTRY(vm86_biosret) movl vm86pcb,%edx /* data area */ movl 4(%esp),%esi /* source */ movl SCR_ARGFRAME(%edx),%edi /* destination */ movl $VM86_FRAMESIZE/4,%ecx /* size */ cld rep movsl /* copy frame to original frame */ movl SCR_STACK(%edx),%esp /* back to old stack */ popl %ebx /* saved va of Idle PTD */ popl %eax movl %eax,0(%ebx) /* restore old pte */ popl %eax movl %eax,%cr3 /* install old page table */ movl PCPU(TSS_GDT),%ebx /* entry in GDT */ movl SCR_TSS0(%edx),%eax movl %eax,0(%ebx) /* restore first word */ movl SCR_TSS1(%edx),%eax movl %eax,4(%ebx) /* restore second word */ movl $GPROC0_SEL*8,%esi /* GSEL(entry, SEL_KPL) */ ltr %si popl PCPU(CURPCB) /* restore curpcb/curproc */ movl SCR_ARGFRAME(%edx),%edx /* original stack frame */ movl TF_TRAPNO(%edx),%eax /* return (trapno) */ popl %gs popl %edi popl %esi popl %ebp popl %ebx ret /* back to our normal program */ Index: user/ngie/more-tests/sys/i386/i386/vm_machdep.c =================================================================== --- user/ngie/more-tests/sys/i386/i386/vm_machdep.c (revision 281503) +++ user/ngie/more-tests/sys/i386/i386/vm_machdep.c (revision 281504) @@ -1,927 +1,927 @@ /*- * Copyright (c) 1982, 1986 The Regents of the University of California. * Copyright (c) 1989, 1990 William Jolitz * Copyright (c) 1994 John Dyson * All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department, and William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. All advertising materials mentioning features or use of this software * must display the following acknowledgement: * This product includes software developed by the University of * California, Berkeley and its contributors. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)vm_machdep.c 7.3 (Berkeley) 5/13/91 * Utah $Hdr: vm_machdep.c 1.16.1.1 89/06/23$ */ #include __FBSDID("$FreeBSD$"); #include "opt_isa.h" #include "opt_npx.h" #include "opt_reset.h" #include "opt_cpu.h" #include "opt_xbox.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef CPU_ELAN #include #endif #include #include #include #include #include #include #ifdef XEN #include #endif #ifdef PC98 #include #else #include #endif #ifdef XBOX #include #endif #ifndef NSFBUFS #define NSFBUFS (512 + maxusers * 16) #endif #if !defined(CPU_DISABLE_SSE) && defined(I686_CPU) #define CPU_ENABLE_SSE #endif _Static_assert(OFFSETOF_CURTHREAD == offsetof(struct pcpu, pc_curthread), "OFFSETOF_CURTHREAD does not correspond with offset of pc_curthread."); _Static_assert(OFFSETOF_CURPCB == offsetof(struct pcpu, pc_curpcb), "OFFSETOF_CURPCB does not correspond with offset of pc_curpcb."); static void cpu_reset_real(void); #ifdef SMP static void cpu_reset_proxy(void); static u_int cpu_reset_proxyid; static volatile u_int cpu_reset_proxy_active; #endif union savefpu * get_pcb_user_save_td(struct thread *td) { vm_offset_t p; p = td->td_kstack + td->td_kstack_pages * PAGE_SIZE - cpu_max_ext_state_size; KASSERT((p % 64) == 0, ("Unaligned pcb_user_save area")); return ((union savefpu *)p); } union savefpu * get_pcb_user_save_pcb(struct pcb *pcb) { vm_offset_t p; p = (vm_offset_t)(pcb + 1); return ((union savefpu *)p); } struct pcb * get_pcb_td(struct thread *td) { vm_offset_t p; p = td->td_kstack + td->td_kstack_pages * PAGE_SIZE - cpu_max_ext_state_size - sizeof(struct pcb); return ((struct pcb *)p); } void * alloc_fpusave(int flags) { void *res; #ifdef CPU_ENABLE_SSE struct savefpu_ymm *sf; #endif res = malloc(cpu_max_ext_state_size, M_DEVBUF, flags); #ifdef CPU_ENABLE_SSE if (use_xsave) { sf = (struct savefpu_ymm *)res; bzero(&sf->sv_xstate.sx_hd, sizeof(sf->sv_xstate.sx_hd)); sf->sv_xstate.sx_hd.xstate_bv = xsave_mask; } #endif return (res); } /* * Finish a fork operation, with process p2 nearly set up. * Copy and update the pcb, set up the stack so that the child * ready to run and return to user mode. */ void cpu_fork(td1, p2, td2, flags) register struct thread *td1; register struct proc *p2; struct thread *td2; int flags; { register struct proc *p1; struct pcb *pcb2; struct mdproc *mdp2; p1 = td1->td_proc; if ((flags & RFPROC) == 0) { if ((flags & RFMEM) == 0) { /* unshare user LDT */ struct mdproc *mdp1 = &p1->p_md; struct proc_ldt *pldt, *pldt1; mtx_lock_spin(&dt_lock); if ((pldt1 = mdp1->md_ldt) != NULL && pldt1->ldt_refcnt > 1) { pldt = user_ldt_alloc(mdp1, pldt1->ldt_len); if (pldt == NULL) panic("could not copy LDT"); mdp1->md_ldt = pldt; set_user_ldt(mdp1); user_ldt_deref(pldt1); } else mtx_unlock_spin(&dt_lock); } return; } /* Ensure that td1's pcb is up to date. */ if (td1 == curthread) td1->td_pcb->pcb_gs = rgs(); #ifdef DEV_NPX critical_enter(); if (PCPU_GET(fpcurthread) == td1) npxsave(td1->td_pcb->pcb_save); critical_exit(); #endif /* Point the pcb to the top of the stack */ pcb2 = get_pcb_td(td2); td2->td_pcb = pcb2; /* Copy td1's pcb */ bcopy(td1->td_pcb, pcb2, sizeof(*pcb2)); /* Properly initialize pcb_save */ pcb2->pcb_save = get_pcb_user_save_pcb(pcb2); bcopy(get_pcb_user_save_td(td1), get_pcb_user_save_pcb(pcb2), cpu_max_ext_state_size); /* Point mdproc and then copy over td1's contents */ mdp2 = &p2->p_md; bcopy(&p1->p_md, mdp2, sizeof(*mdp2)); /* * Create a new fresh stack for the new process. * Copy the trap frame for the return to user mode as if from a * syscall. This copies most of the user mode register values. * The -16 is so we can expand the trapframe if we go to vm86. */ td2->td_frame = (struct trapframe *)((caddr_t)td2->td_pcb - 16) - 1; bcopy(td1->td_frame, td2->td_frame, sizeof(struct trapframe)); td2->td_frame->tf_eax = 0; /* Child returns zero */ td2->td_frame->tf_eflags &= ~PSL_C; /* success */ td2->td_frame->tf_edx = 1; /* * If the parent process has the trap bit set (i.e. a debugger had * single stepped the process to the system call), we need to clear * the trap flag from the new frame unless the debugger had set PF_FORK * on the parent. Otherwise, the child will receive a (likely * unexpected) SIGTRAP when it executes the first instruction after * returning to userland. */ if ((p1->p_pfsflags & PF_FORK) == 0) td2->td_frame->tf_eflags &= ~PSL_T; /* * Set registers for trampoline to user mode. Leave space for the * return address on stack. These are the kernel mode register values. */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) pcb2->pcb_cr3 = vtophys(vmspace_pmap(p2->p_vmspace)->pm_pdpt); #else pcb2->pcb_cr3 = vtophys(vmspace_pmap(p2->p_vmspace)->pm_pdir); #endif pcb2->pcb_edi = 0; pcb2->pcb_esi = (int)fork_return; /* fork_trampoline argument */ pcb2->pcb_ebp = 0; pcb2->pcb_esp = (int)td2->td_frame - sizeof(void *); pcb2->pcb_ebx = (int)td2; /* fork_trampoline argument */ pcb2->pcb_eip = (int)fork_trampoline; pcb2->pcb_psl = PSL_KERNEL; /* ints disabled */ /*- * pcb2->pcb_dr*: cloned above. * pcb2->pcb_savefpu: cloned above. * pcb2->pcb_flags: cloned above. * pcb2->pcb_onfault: cloned above (always NULL here?). * pcb2->pcb_gs: cloned above. * pcb2->pcb_ext: cleared below. */ /* * XXX don't copy the i/o pages. this should probably be fixed. */ pcb2->pcb_ext = 0; /* Copy the LDT, if necessary. */ mtx_lock_spin(&dt_lock); if (mdp2->md_ldt != NULL) { if (flags & RFMEM) { mdp2->md_ldt->ldt_refcnt++; } else { mdp2->md_ldt = user_ldt_alloc(mdp2, mdp2->md_ldt->ldt_len); if (mdp2->md_ldt == NULL) panic("could not copy LDT"); } } mtx_unlock_spin(&dt_lock); /* Setup to release spin count in fork_exit(). */ td2->td_md.md_spinlock_count = 1; /* * XXX XEN need to check on PSL_USER is handled */ td2->td_md.md_saved_flags = PSL_KERNEL | PSL_I; /* * Now, cpu_switch() can schedule the new process. * pcb_esp is loaded pointing to the cpu_switch() stack frame * containing the return address when exiting cpu_switch. * This will normally be to fork_trampoline(), which will have * %ebx loaded with the new proc's pointer. fork_trampoline() * will set up a stack to call fork_return(p, frame); to complete * the return to user-mode. */ } /* * Intercept the return address from a freshly forked process that has NOT * been scheduled yet. * * This is needed to make kernel threads stay in kernel mode. */ void cpu_set_fork_handler(td, func, arg) struct thread *td; void (*func)(void *); void *arg; { /* * Note that the trap frame follows the args, so the function * is really called like this: func(arg, frame); */ td->td_pcb->pcb_esi = (int) func; /* function */ td->td_pcb->pcb_ebx = (int) arg; /* first arg */ } void cpu_exit(struct thread *td) { /* * If this process has a custom LDT, release it. Reset pc->pcb_gs * and %gs before we free it in case they refer to an LDT entry. */ mtx_lock_spin(&dt_lock); if (td->td_proc->p_md.md_ldt) { td->td_pcb->pcb_gs = _udatasel; load_gs(_udatasel); user_ldt_free(td); } else mtx_unlock_spin(&dt_lock); } void cpu_thread_exit(struct thread *td) { #ifdef DEV_NPX critical_enter(); if (td == PCPU_GET(fpcurthread)) npxdrop(); critical_exit(); #endif /* Disable any hardware breakpoints. */ if (td->td_pcb->pcb_flags & PCB_DBREGS) { reset_dbregs(); td->td_pcb->pcb_flags &= ~PCB_DBREGS; } } void cpu_thread_clean(struct thread *td) { struct pcb *pcb; pcb = td->td_pcb; if (pcb->pcb_ext != NULL) { /* if (pcb->pcb_ext->ext_refcount-- == 1) ?? */ /* * XXX do we need to move the TSS off the allocated pages * before freeing them? (not done here) */ kmem_free(kernel_arena, (vm_offset_t)pcb->pcb_ext, ctob(IOPAGES + 1)); pcb->pcb_ext = NULL; } } void cpu_thread_swapin(struct thread *td) { } void cpu_thread_swapout(struct thread *td) { } void cpu_thread_alloc(struct thread *td) { struct pcb *pcb; #ifdef CPU_ENABLE_SSE struct xstate_hdr *xhdr; #endif td->td_pcb = pcb = get_pcb_td(td); td->td_frame = (struct trapframe *)((caddr_t)pcb - 16) - 1; pcb->pcb_ext = NULL; pcb->pcb_save = get_pcb_user_save_pcb(pcb); #ifdef CPU_ENABLE_SSE if (use_xsave) { xhdr = (struct xstate_hdr *)(pcb->pcb_save + 1); bzero(xhdr, sizeof(*xhdr)); xhdr->xstate_bv = xsave_mask; } #endif } void cpu_thread_free(struct thread *td) { cpu_thread_clean(td); } void cpu_set_syscall_retval(struct thread *td, int error) { switch (error) { case 0: td->td_frame->tf_eax = td->td_retval[0]; td->td_frame->tf_edx = td->td_retval[1]; td->td_frame->tf_eflags &= ~PSL_C; break; case ERESTART: /* * Reconstruct pc, assuming lcall $X,y is 7 bytes, int * 0x80 is 2 bytes. We saved this in tf_err. */ td->td_frame->tf_eip -= td->td_frame->tf_err; break; case EJUSTRETURN: break; default: if (td->td_proc->p_sysent->sv_errsize) { if (error >= td->td_proc->p_sysent->sv_errsize) error = -1; /* XXX */ else error = td->td_proc->p_sysent->sv_errtbl[error]; } td->td_frame->tf_eax = error; td->td_frame->tf_eflags |= PSL_C; break; } } /* * Initialize machine state (pcb and trap frame) for a new thread about to * upcall. Put enough state in the new thread's PCB to get it to go back * userret(), where we can intercept it again to set the return (upcall) * Address and stack, along with those from upcals that are from other sources * such as those generated in thread_userret() itself. */ void cpu_set_upcall(struct thread *td, struct thread *td0) { struct pcb *pcb2; /* Point the pcb to the top of the stack. */ pcb2 = td->td_pcb; /* * Copy the upcall pcb. This loads kernel regs. * Those not loaded individually below get their default * values here. */ bcopy(td0->td_pcb, pcb2, sizeof(*pcb2)); pcb2->pcb_flags &= ~(PCB_NPXINITDONE | PCB_NPXUSERINITDONE | PCB_KERNNPX); pcb2->pcb_save = get_pcb_user_save_pcb(pcb2); bcopy(get_pcb_user_save_td(td0), pcb2->pcb_save, cpu_max_ext_state_size); /* * Create a new fresh stack for the new thread. */ bcopy(td0->td_frame, td->td_frame, sizeof(struct trapframe)); /* If the current thread has the trap bit set (i.e. a debugger had * single stepped the process to the system call), we need to clear * the trap flag from the new frame. Otherwise, the new thread will * receive a (likely unexpected) SIGTRAP when it executes the first * instruction after returning to userland. */ td->td_frame->tf_eflags &= ~PSL_T; /* * Set registers for trampoline to user mode. Leave space for the * return address on stack. These are the kernel mode register values. */ pcb2->pcb_edi = 0; pcb2->pcb_esi = (int)fork_return; /* trampoline arg */ pcb2->pcb_ebp = 0; pcb2->pcb_esp = (int)td->td_frame - sizeof(void *); /* trampoline arg */ pcb2->pcb_ebx = (int)td; /* trampoline arg */ pcb2->pcb_eip = (int)fork_trampoline; pcb2->pcb_psl &= ~(PSL_I); /* interrupts must be disabled */ pcb2->pcb_gs = rgs(); /* * If we didn't copy the pcb, we'd need to do the following registers: * pcb2->pcb_cr3: cloned above. * pcb2->pcb_dr*: cloned above. * pcb2->pcb_savefpu: cloned above. * pcb2->pcb_flags: cloned above. * pcb2->pcb_onfault: cloned above (always NULL here?). * pcb2->pcb_gs: cloned above. * pcb2->pcb_ext: cleared below. */ pcb2->pcb_ext = NULL; /* Setup to release spin count in fork_exit(). */ td->td_md.md_spinlock_count = 1; td->td_md.md_saved_flags = PSL_KERNEL | PSL_I; } /* * Set that machine state for performing an upcall that has to * be done in thread_userret() so that those upcalls generated * in thread_userret() itself can be done as well. */ void cpu_set_upcall_kse(struct thread *td, void (*entry)(void *), void *arg, stack_t *stack) { /* * Do any extra cleaning that needs to be done. * The thread may have optional components * that are not present in a fresh thread. * This may be a recycled thread so make it look * as though it's newly allocated. */ cpu_thread_clean(td); /* * Set the trap frame to point at the beginning of the uts * function. */ td->td_frame->tf_ebp = 0; td->td_frame->tf_esp = (((int)stack->ss_sp + stack->ss_size - 4) & ~0x0f) - 4; td->td_frame->tf_eip = (int)entry; /* * Pass the address of the mailbox for this kse to the uts * function as a parameter on the stack. */ suword((void *)(td->td_frame->tf_esp + sizeof(void *)), (int)arg); } int cpu_set_user_tls(struct thread *td, void *tls_base) { struct segment_descriptor sd; uint32_t base; /* * Construct a descriptor and store it in the pcb for * the next context switch. Also store it in the gdt * so that the load of tf_fs into %fs will activate it * at return to userland. */ base = (uint32_t)tls_base; sd.sd_lobase = base & 0xffffff; sd.sd_hibase = (base >> 24) & 0xff; sd.sd_lolimit = 0xffff; /* 4GB limit, wraps around */ sd.sd_hilimit = 0xf; sd.sd_type = SDT_MEMRWA; sd.sd_dpl = SEL_UPL; sd.sd_p = 1; sd.sd_xx = 0; sd.sd_def32 = 1; sd.sd_gran = 1; critical_enter(); /* set %gs */ td->td_pcb->pcb_gsd = sd; if (td == curthread) { PCPU_GET(fsgs_gdt)[1] = sd; load_gs(GSEL(GUGS_SEL, SEL_UPL)); } critical_exit(); return (0); } /* * Convert kernel VA to physical address */ vm_paddr_t kvtop(void *addr) { vm_paddr_t pa; pa = pmap_kextract((vm_offset_t)addr); if (pa == 0) panic("kvtop: zero page frame"); return (pa); } #ifdef SMP static void cpu_reset_proxy() { cpuset_t tcrp; cpu_reset_proxy_active = 1; while (cpu_reset_proxy_active == 1) ; /* Wait for other cpu to see that we've started */ CPU_SETOF(cpu_reset_proxyid, &tcrp); stop_cpus(tcrp); printf("cpu_reset_proxy: Stopped CPU %d\n", cpu_reset_proxyid); DELAY(1000000); cpu_reset_real(); } #endif void cpu_reset() { #ifdef XBOX if (arch_i386_is_xbox) { /* Kick the PIC16L, it can reboot the box */ pic16l_reboot(); for (;;); } #endif #ifdef SMP cpuset_t map; u_int cnt; if (smp_started) { map = all_cpus; CPU_CLR(PCPU_GET(cpuid), &map); CPU_NAND(&map, &stopped_cpus); if (!CPU_EMPTY(&map)) { printf("cpu_reset: Stopping other CPUs\n"); stop_cpus(map); } if (PCPU_GET(cpuid) != 0) { cpu_reset_proxyid = PCPU_GET(cpuid); cpustop_restartfunc = cpu_reset_proxy; cpu_reset_proxy_active = 0; printf("cpu_reset: Restarting BSP\n"); /* Restart CPU #0. */ /* XXX: restart_cpus(1 << 0); */ CPU_SETOF(0, &started_cpus); wmb(); cnt = 0; while (cpu_reset_proxy_active == 0 && cnt < 10000000) cnt++; /* Wait for BSP to announce restart */ if (cpu_reset_proxy_active == 0) printf("cpu_reset: Failed to restart BSP\n"); enable_intr(); cpu_reset_proxy_active = 2; while (1); /* NOTREACHED */ } DELAY(1000000); } #endif cpu_reset_real(); /* NOTREACHED */ } static void cpu_reset_real() { struct region_descriptor null_idt; #ifndef PC98 int b; #endif disable_intr(); #ifdef XEN if (smp_processor_id() == 0) HYPERVISOR_shutdown(SHUTDOWN_reboot); else HYPERVISOR_shutdown(SHUTDOWN_poweroff); #endif #ifdef CPU_ELAN if (elan_mmcr != NULL) elan_mmcr->RESCFG = 1; #endif if (cpu == CPU_GEODE1100) { /* Attempt Geode's own reset */ outl(0xcf8, 0x80009044ul); outl(0xcfc, 0xf); } #ifdef PC98 /* * Attempt to do a CPU reset via CPU reset port. */ if ((inb(0x35) & 0xa0) != 0xa0) { outb(0x37, 0x0f); /* SHUT0 = 0. */ outb(0x37, 0x0b); /* SHUT1 = 0. */ } outb(0xf0, 0x00); /* Reset. */ #else #if !defined(BROKEN_KEYBOARD_RESET) /* * Attempt to do a CPU reset via the keyboard controller, * do not turn off GateA20, as any machine that fails * to do the reset here would then end up in no man's land. */ outb(IO_KBD + 4, 0xFE); DELAY(500000); /* wait 0.5 sec to see if that did it */ #endif /* * Attempt to force a reset via the Reset Control register at * I/O port 0xcf9. Bit 2 forces a system reset when it * transitions from 0 to 1. Bit 1 selects the type of reset * to attempt: 0 selects a "soft" reset, and 1 selects a * "hard" reset. We try a "hard" reset. The first write sets * bit 1 to select a "hard" reset and clears bit 2. The * second write forces a 0 -> 1 transition in bit 2 to trigger * a reset. */ outb(0xcf9, 0x2); outb(0xcf9, 0x6); DELAY(500000); /* wait 0.5 sec to see if that did it */ /* * Attempt to force a reset via the Fast A20 and Init register * at I/O port 0x92. Bit 1 serves as an alternate A20 gate. * Bit 0 asserts INIT# when set to 1. We are careful to only * preserve bit 1 while setting bit 0. We also must clear bit * 0 before setting it if it isn't already clear. */ b = inb(0x92); if (b != 0xff) { if ((b & 0x1) != 0) outb(0x92, b & 0xfe); outb(0x92, b | 0x1); DELAY(500000); /* wait 0.5 sec to see if that did it */ } #endif /* PC98 */ printf("No known reset method worked, attempting CPU shutdown\n"); DELAY(1000000); /* wait 1 sec for printf to complete */ /* Wipe the IDT. */ null_idt.rd_limit = 0; null_idt.rd_base = 0; lidt(&null_idt); /* "good night, sweet prince .... " */ breakpoint(); /* NOTREACHED */ while(1); } /* * Get an sf_buf from the freelist. May block if none are available. */ void sf_buf_map(struct sf_buf *sf, int flags) { pt_entry_t opte, *ptep; /* * Update the sf_buf's virtual-to-physical mapping, flushing the * virtual address from the TLB. Since the reference count for * the sf_buf's old mapping was zero, that mapping is not * currently in use. Consequently, there is no need to exchange * the old and new PTEs atomically, even under PAE. */ ptep = vtopte(sf->kva); opte = *ptep; #ifdef XEN PT_SET_MA(sf->kva, xpmap_ptom(VM_PAGE_TO_PHYS(sf->m)) | pgeflag | PG_RW | PG_V | pmap_cache_bits(sf->m->md.pat_mode, 0)); #else *ptep = VM_PAGE_TO_PHYS(sf->m) | pgeflag | PG_RW | PG_V | pmap_cache_bits(sf->m->md.pat_mode, 0); #endif /* * Avoid unnecessary TLB invalidations: If the sf_buf's old * virtual-to-physical mapping was not used, then any processor * that has invalidated the sf_buf's virtual address from its TLB * since the last used mapping need not invalidate again. */ #ifdef SMP if ((opte & (PG_V | PG_A)) == (PG_V | PG_A)) CPU_ZERO(&sf->cpumask); sf_buf_shootdown(sf, flags); #else if ((opte & (PG_V | PG_A)) == (PG_V | PG_A)) pmap_invalidate_page(kernel_pmap, sf->kva); #endif } #ifdef SMP void sf_buf_shootdown(struct sf_buf *sf, int flags) { cpuset_t other_cpus; u_int cpuid; sched_pin(); cpuid = PCPU_GET(cpuid); if (!CPU_ISSET(cpuid, &sf->cpumask)) { CPU_SET(cpuid, &sf->cpumask); invlpg(sf->kva); } if ((flags & SFB_CPUPRIVATE) == 0) { other_cpus = all_cpus; CPU_CLR(cpuid, &other_cpus); CPU_NAND(&other_cpus, &sf->cpumask); if (!CPU_EMPTY(&other_cpus)) { CPU_OR(&sf->cpumask, &other_cpus); smp_masked_invlpg(other_cpus, sf->kva); } } sched_unpin(); } #endif /* * MD part of sf_buf_free(). */ int sf_buf_unmap(struct sf_buf *sf) { #ifdef XEN /* * Xen doesn't like having dangling R/W mappings */ pmap_qremove(sf->kva, 1); return (1); #else return (0); #endif } static void sf_buf_invalidate(struct sf_buf *sf) { vm_page_t m = sf->m; /* * Use pmap_qenter to update the pte for * existing mapping, in particular, the PAT * settings are recalculated. */ pmap_qenter(sf->kva, &m, 1); pmap_invalidate_cache_range(sf->kva, sf->kva + PAGE_SIZE, FALSE); } /* * Invalidate the cache lines that may belong to the page, if * (possibly old) mapping of the page by sf buffer exists. Returns * TRUE when mapping was found and cache invalidated. */ boolean_t sf_buf_invalidate_cache(vm_page_t m) { return (sf_buf_process_page(m, sf_buf_invalidate)); } /* * Software interrupt handler for queued VM system processing. */ void swi_vm(void *dummy) { if (busdma_swi_pending != 0) busdma_swi(); } /* * Tell whether this address is in some physical memory region. * Currently used by the kernel coredump code in order to avoid * dumping the ``ISA memory hole'' which could cause indefinite hangs, * or other unpredictable behaviour. */ int is_physical_memory(vm_paddr_t addr) { #ifdef DEV_ISA /* The ISA ``memory hole''. */ if (addr >= 0xa0000 && addr < 0x100000) return 0; #endif /* * stuff other tests for known memory-mapped devices (PCI?) * here */ return 1; } Index: user/ngie/more-tests/sys/i386/include/param.h =================================================================== --- user/ngie/more-tests/sys/i386/include/param.h (revision 281503) +++ user/ngie/more-tests/sys/i386/include/param.h (revision 281504) @@ -1,160 +1,160 @@ /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)param.h 5.8 (Berkeley) 6/28/91 * $FreeBSD$ */ #ifndef _I386_INCLUDE_PARAM_H_ #define _I386_INCLUDE_PARAM_H_ #include /* * Machine dependent constants for Intel 386. */ #define __HAVE_ACPI #define __HAVE_PIR #define __PCI_REROUTE_INTERRUPT #ifndef MACHINE #define MACHINE "i386" #endif #ifndef MACHINE_ARCH #define MACHINE_ARCH "i386" #endif #define MID_MACHINE MID_I386 #if defined(SMP) || defined(KLD_MODULE) #ifndef MAXCPU #define MAXCPU 32 #endif #else #define MAXCPU 1 #endif /* SMP || KLD_MODULE */ #ifndef MAXMEMDOM #define MAXMEMDOM 1 #endif #define ALIGNBYTES _ALIGNBYTES #define ALIGN(p) _ALIGN(p) /* * ALIGNED_POINTER is a boolean macro that checks whether an address * is valid to fetch data elements of type t from on this architecture. * This does not reflect the optimal alignment, just the possibility * (within reasonable limits). */ #define ALIGNED_POINTER(p, t) 1 /* * CACHE_LINE_SIZE is the compile-time maximum cache line size for an * architecture. It should be used with appropriate caution. */ #define CACHE_LINE_SHIFT 7 #define CACHE_LINE_SIZE (1 << CACHE_LINE_SHIFT) #define PAGE_SHIFT 12 /* LOG2(PAGE_SIZE) */ #define PAGE_SIZE (1<> PAGE_SHIFT) #define ptoa(x) ((x) << PAGE_SHIFT) #define i386_btop(x) ((x) >> PAGE_SHIFT) #define i386_ptob(x) ((x) << PAGE_SHIFT) #define pgtok(x) ((x) * (PAGE_SIZE / 1024)) #endif /* !_I386_INCLUDE_PARAM_H_ */ Index: user/ngie/more-tests/sys/i386/include/pmap.h =================================================================== --- user/ngie/more-tests/sys/i386/include/pmap.h (revision 281503) +++ user/ngie/more-tests/sys/i386/include/pmap.h (revision 281504) @@ -1,473 +1,476 @@ /*- * Copyright (c) 1991 Regents of the University of California. * All rights reserved. * * This code is derived from software contributed to Berkeley by * the Systems Programming Group of the University of Utah Computer * Science Department and William Jolitz of UUNET Technologies Inc. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * Derived from hp300 version by Mike Hibler, this version by William * Jolitz uses a recursive map [a pde points to the page directory] to * map the page tables using the pagetables themselves. This is done to * reduce the impact on kernel virtual memory for lots of sparse address * space, and to reduce the cost of memory to each process. * * from: hp300: @(#)pmap.h 7.2 (Berkeley) 12/16/90 * from: @(#)pmap.h 7.4 (Berkeley) 5/12/91 * $FreeBSD$ */ #ifndef _MACHINE_PMAP_H_ #define _MACHINE_PMAP_H_ /* * Page-directory and page-table entries follow this format, with a few * of the fields not present here and there, depending on a lot of things. */ /* ---- Intel Nomenclature ---- */ #define PG_V 0x001 /* P Valid */ #define PG_RW 0x002 /* R/W Read/Write */ #define PG_U 0x004 /* U/S User/Supervisor */ #define PG_NC_PWT 0x008 /* PWT Write through */ #define PG_NC_PCD 0x010 /* PCD Cache disable */ #define PG_A 0x020 /* A Accessed */ #define PG_M 0x040 /* D Dirty */ #define PG_PS 0x080 /* PS Page size (0=4k,1=4M) */ #define PG_PTE_PAT 0x080 /* PAT PAT index */ #define PG_G 0x100 /* G Global */ #define PG_AVAIL1 0x200 /* / Available for system */ #define PG_AVAIL2 0x400 /* < programmers use */ #define PG_AVAIL3 0x800 /* \ */ #define PG_PDE_PAT 0x1000 /* PAT PAT index */ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) #define PG_NX (1ull<<63) /* No-execute */ #endif /* Our various interpretations of the above */ #define PG_W PG_AVAIL1 /* "Wired" pseudoflag */ #define PG_MANAGED PG_AVAIL2 -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) #define PG_FRAME (0x000ffffffffff000ull) #define PG_PS_FRAME (0x000fffffffe00000ull) #else #define PG_FRAME (~PAGE_MASK) #define PG_PS_FRAME (0xffc00000) #endif #define PG_PROT (PG_RW|PG_U) /* all protection bits . */ #define PG_N (PG_NC_PWT|PG_NC_PCD) /* Non-cacheable */ /* Page level cache control fields used to determine the PAT type */ #define PG_PDE_CACHE (PG_PDE_PAT | PG_NC_PWT | PG_NC_PCD) #define PG_PTE_CACHE (PG_PTE_PAT | PG_NC_PWT | PG_NC_PCD) /* * Promotion to a 2 or 4MB (PDE) page mapping requires that the corresponding * 4KB (PTE) page mappings have identical settings for the following fields: */ #define PG_PTE_PROMOTE (PG_MANAGED | PG_W | PG_G | PG_PTE_PAT | \ PG_M | PG_A | PG_NC_PCD | PG_NC_PWT | PG_U | PG_RW | PG_V) /* * Page Protection Exception bits */ #define PGEX_P 0x01 /* Protection violation vs. not present */ #define PGEX_W 0x02 /* during a Write cycle */ #define PGEX_U 0x04 /* access from User mode (UPL) */ #define PGEX_RSV 0x08 /* reserved PTE field is non-zero */ #define PGEX_I 0x10 /* during an instruction fetch */ /* * Size of Kernel address space. This is the number of page table pages * (4MB each) to use for the kernel. 256 pages == 1 Gigabyte. * This **MUST** be a multiple of 4 (eg: 252, 256, 260, etc). * For PAE, the page table page unit size is 2MB. This means that 512 pages * is 1 Gigabyte. Double everything. It must be a multiple of 8 for PAE. */ #ifndef KVA_PAGES -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) #define KVA_PAGES 512 #else #define KVA_PAGES 256 #endif #endif /* * Pte related macros */ #define VADDR(pdi, pti) ((vm_offset_t)(((pdi)< #include #include #include #include -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) typedef uint64_t pdpt_entry_t; typedef uint64_t pd_entry_t; typedef uint64_t pt_entry_t; #define PTESHIFT (3) #define PDESHIFT (3) #else typedef uint32_t pd_entry_t; typedef uint32_t pt_entry_t; #define PTESHIFT (2) #define PDESHIFT (2) #endif /* * Address of current address space page table maps and directories. */ #ifdef _KERNEL extern pt_entry_t PTmap[]; extern pd_entry_t PTD[]; extern pd_entry_t PTDpde[]; -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) extern pdpt_entry_t *IdlePDPT; #endif extern pd_entry_t *IdlePTD; /* physical address of "Idle" state directory */ /* * Translate a virtual address to the kernel virtual address of its page table * entry (PTE). This can be used recursively. If the address of a PTE as * previously returned by this macro is itself given as the argument, then the * address of the page directory entry (PDE) that maps the PTE will be * returned. * * This macro may be used before pmap_bootstrap() is called. */ #define vtopte(va) (PTmap + i386_btop(va)) /* * Translate a virtual address to its physical address. * * This macro may be used before pmap_bootstrap() is called. */ #define vtophys(va) pmap_kextract((vm_offset_t)(va)) #if defined(XEN) #include #include #include #include extern pt_entry_t pg_nx; #define PG_KERNEL (PG_V | PG_A | PG_RW | PG_M) #define MACH_TO_VM_PAGE(ma) PHYS_TO_VM_PAGE(xpmap_mtop((ma))) #define VM_PAGE_TO_MACH(m) xpmap_ptom(VM_PAGE_TO_PHYS((m))) #define VTOM(va) xpmap_ptom(VTOP(va)) static __inline vm_paddr_t pmap_kextract_ma(vm_offset_t va) { vm_paddr_t ma; if ((ma = PTD[va >> PDRSHIFT]) & PG_PS) { ma = (ma & ~(NBPDR - 1)) | (va & (NBPDR - 1)); } else { ma = (*vtopte(va) & PG_FRAME) | (va & PAGE_MASK); } return ma; } static __inline vm_paddr_t pmap_kextract(vm_offset_t va) { return xpmap_mtop(pmap_kextract_ma(va)); } #define vtomach(va) pmap_kextract_ma(((vm_offset_t) (va))) vm_paddr_t pmap_extract_ma(struct pmap *pmap, vm_offset_t va); void pmap_kenter_ma(vm_offset_t va, vm_paddr_t pa); void pmap_map_readonly(struct pmap *pmap, vm_offset_t va, int len); void pmap_map_readwrite(struct pmap *pmap, vm_offset_t va, int len); static __inline pt_entry_t pte_load_store(pt_entry_t *ptep, pt_entry_t v) { pt_entry_t r; r = *ptep; PT_SET_VA(ptep, v, TRUE); return (r); } static __inline pt_entry_t pte_load_store_ma(pt_entry_t *ptep, pt_entry_t v) { pt_entry_t r; r = *ptep; PT_SET_VA_MA(ptep, v, TRUE); return (r); } #define pte_load_clear(ptep) pte_load_store((ptep), (pt_entry_t)0ULL) #define pte_store(ptep, pte) pte_load_store((ptep), (pt_entry_t)pte) #define pte_store_ma(ptep, pte) pte_load_store_ma((ptep), (pt_entry_t)pte) #define pde_store_ma(ptep, pte) pte_load_store_ma((ptep), (pt_entry_t)pte) #elif !defined(XEN) /* * KPTmap is a linear mapping of the kernel page table. It differs from the * recursive mapping in two ways: (1) it only provides access to kernel page * table pages, and not user page table pages, and (2) it provides access to * a kernel page table page after the corresponding virtual addresses have * been promoted to a 2/4MB page mapping. * * KPTmap is first initialized by locore to support just NPKT page table * pages. Later, it is reinitialized by pmap_bootstrap() to allow for * expansion of the kernel page table. */ extern pt_entry_t *KPTmap; /* * Extract from the kernel page table the physical address that is mapped by * the given virtual address "va". * * This function may be used before pmap_bootstrap() is called. */ static __inline vm_paddr_t pmap_kextract(vm_offset_t va) { vm_paddr_t pa; if ((pa = PTD[va >> PDRSHIFT]) & PG_PS) { pa = (pa & PG_PS_FRAME) | (va & PDRMASK); } else { /* * Beware of a concurrent promotion that changes the PDE at * this point! For example, vtopte() must not be used to * access the PTE because it would use the new PDE. It is, * however, safe to use the old PDE because the page table * page is preserved by the promotion. */ pa = KPTmap[i386_btop(va)]; pa = (pa & PG_FRAME) | (va & PAGE_MASK); } return (pa); } #endif #if !defined(XEN) #define PT_UPDATES_FLUSH() #endif -#if defined(PAE) && !defined(XEN) +#if (defined(PAE) || defined(PAE_TABLES)) && !defined(XEN) #define pde_cmpset(pdep, old, new) atomic_cmpset_64_i586(pdep, old, new) #define pte_load_store(ptep, pte) atomic_swap_64_i586(ptep, pte) #define pte_load_clear(ptep) atomic_swap_64_i586(ptep, 0) #define pte_store(ptep, pte) atomic_store_rel_64_i586(ptep, pte) extern pt_entry_t pg_nx; -#elif !defined(PAE) && !defined(XEN) +#elif !defined(PAE) && !defined(PAE_TABLES) && !defined(XEN) #define pde_cmpset(pdep, old, new) atomic_cmpset_int(pdep, old, new) #define pte_load_store(ptep, pte) atomic_swap_int(ptep, pte) #define pte_load_clear(ptep) atomic_swap_int(ptep, 0) #define pte_store(ptep, pte) do { \ *(u_int *)(ptep) = (u_int)(pte); \ } while (0) #endif /* PAE */ #define pte_clear(ptep) pte_store(ptep, 0) #define pde_store(pdep, pde) pte_store(pdep, pde) #endif /* _KERNEL */ /* * Pmap stuff */ struct pv_entry; struct pv_chunk; struct md_page { TAILQ_HEAD(,pv_entry) pv_list; int pat_mode; }; struct pmap { struct mtx pm_mtx; pd_entry_t *pm_pdir; /* KVA of page directory */ TAILQ_HEAD(,pv_chunk) pm_pvchunk; /* list of mappings in pmap */ cpuset_t pm_active; /* active on cpus */ struct pmap_statistics pm_stats; /* pmap statistics */ LIST_ENTRY(pmap) pm_list; /* List of all pmaps */ -#ifdef PAE - pdpt_entry_t *pm_pdpt; /* KVA of page director pointer +#if defined(PAE) || defined(PAE_TABLES) + pdpt_entry_t *pm_pdpt; /* KVA of page directory pointer table */ #endif struct vm_radix pm_root; /* spare page table pages */ }; typedef struct pmap *pmap_t; #ifdef _KERNEL extern struct pmap kernel_pmap_store; #define kernel_pmap (&kernel_pmap_store) #define PMAP_LOCK(pmap) mtx_lock(&(pmap)->pm_mtx) #define PMAP_LOCK_ASSERT(pmap, type) \ mtx_assert(&(pmap)->pm_mtx, (type)) #define PMAP_LOCK_DESTROY(pmap) mtx_destroy(&(pmap)->pm_mtx) #define PMAP_LOCK_INIT(pmap) mtx_init(&(pmap)->pm_mtx, "pmap", \ NULL, MTX_DEF | MTX_DUPOK) #define PMAP_LOCKED(pmap) mtx_owned(&(pmap)->pm_mtx) #define PMAP_MTX(pmap) (&(pmap)->pm_mtx) #define PMAP_TRYLOCK(pmap) mtx_trylock(&(pmap)->pm_mtx) #define PMAP_UNLOCK(pmap) mtx_unlock(&(pmap)->pm_mtx) #endif /* * For each vm_page_t, there is a list of all currently valid virtual * mappings of that page. An entry is a pv_entry_t, the list is pv_list. */ typedef struct pv_entry { vm_offset_t pv_va; /* virtual address for mapping */ TAILQ_ENTRY(pv_entry) pv_next; } *pv_entry_t; /* * pv_entries are allocated in chunks per-process. This avoids the * need to track per-pmap assignments. */ #define _NPCM 11 #define _NPCPV 336 struct pv_chunk { pmap_t pc_pmap; TAILQ_ENTRY(pv_chunk) pc_list; uint32_t pc_map[_NPCM]; /* bitmap; 1 = free */ TAILQ_ENTRY(pv_chunk) pc_lru; struct pv_entry pc_pventry[_NPCPV]; }; #ifdef _KERNEL extern caddr_t CADDR3; extern pt_entry_t *CMAP3; extern vm_paddr_t phys_avail[]; extern vm_paddr_t dump_avail[]; extern int pseflag; extern int pgeflag; extern char *ptvmmap; /* poor name! */ extern vm_offset_t virtual_avail; extern vm_offset_t virtual_end; #define pmap_page_get_memattr(m) ((vm_memattr_t)(m)->md.pat_mode) #define pmap_page_is_write_mapped(m) (((m)->aflags & PGA_WRITEABLE) != 0) #define pmap_unmapbios(va, sz) pmap_unmapdev((va), (sz)) /* * Only the following functions or macros may be used before pmap_bootstrap() * is called: pmap_kenter(), pmap_kextract(), pmap_kremove(), vtophys(), and * vtopte(). */ void pmap_bootstrap(vm_paddr_t); int pmap_cache_bits(int mode, boolean_t is_pde); int pmap_change_attr(vm_offset_t, vm_size_t, int); void pmap_init_pat(void); void pmap_kenter(vm_offset_t va, vm_paddr_t pa); void *pmap_kenter_temporary(vm_paddr_t pa, int i); void pmap_kremove(vm_offset_t); void *pmap_mapbios(vm_paddr_t, vm_size_t); void *pmap_mapdev(vm_paddr_t, vm_size_t); void *pmap_mapdev_attr(vm_paddr_t, vm_size_t, int); boolean_t pmap_page_is_mapped(vm_page_t m); void pmap_page_set_memattr(vm_page_t m, vm_memattr_t ma); void pmap_unmapdev(vm_offset_t, vm_size_t); pt_entry_t *pmap_pte(pmap_t, vm_offset_t) __pure2; void pmap_invalidate_page(pmap_t, vm_offset_t); void pmap_invalidate_range(pmap_t, vm_offset_t, vm_offset_t); void pmap_invalidate_all(pmap_t); void pmap_invalidate_cache(void); void pmap_invalidate_cache_pages(vm_page_t *pages, int count); void pmap_invalidate_cache_range(vm_offset_t sva, vm_offset_t eva, boolean_t force); #endif /* _KERNEL */ #endif /* !LOCORE */ #endif /* !_MACHINE_PMAP_H_ */ Index: user/ngie/more-tests/sys/i386/include/vmparam.h =================================================================== --- user/ngie/more-tests/sys/i386/include/vmparam.h (revision 281503) +++ user/ngie/more-tests/sys/i386/include/vmparam.h (revision 281504) @@ -1,212 +1,212 @@ /*- * Copyright (c) 1990 The Regents of the University of California. * All rights reserved. * Copyright (c) 1994 John S. Dyson * All rights reserved. * * This code is derived from software contributed to Berkeley by * William Jolitz. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 4. Neither the name of the University nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * from: @(#)vmparam.h 5.9 (Berkeley) 5/12/91 * $FreeBSD$ */ #ifndef _MACHINE_VMPARAM_H_ #define _MACHINE_VMPARAM_H_ 1 /* * Machine dependent constants for 386. */ /* * Virtual memory related constants, all in bytes */ #define MAXTSIZ (128UL*1024*1024) /* max text size */ #ifndef DFLDSIZ #define DFLDSIZ (128UL*1024*1024) /* initial data size limit */ #endif #ifndef MAXDSIZ #define MAXDSIZ (512UL*1024*1024) /* max data size */ #endif #ifndef DFLSSIZ #define DFLSSIZ (8UL*1024*1024) /* initial stack size limit */ #endif #ifndef MAXSSIZ #define MAXSSIZ (64UL*1024*1024) /* max stack size */ #endif #ifndef SGROWSIZ #define SGROWSIZ (128UL*1024) /* amount to grow stack */ #endif /* * Choose between DENSE and SPARSE based on whether lower execution time or * lower kernel address space consumption is desired. Under PAE, kernel * address space is often in short supply. */ #ifdef PAE #define VM_PHYSSEG_SPARSE #else #define VM_PHYSSEG_DENSE #endif /* * The number of PHYSSEG entries must be one greater than the number * of phys_avail entries because the phys_avail entry that spans the * largest physical address that is accessible by ISA DMA is split * into two PHYSSEG entries. */ #define VM_PHYSSEG_MAX 17 /* * Create two free page pools. Since the i386 kernel virtual address * space does not include a mapping onto the machine's entire physical * memory, VM_FREEPOOL_DIRECT is defined as an alias for the default * pool, VM_FREEPOOL_DEFAULT. */ #define VM_NFREEPOOL 2 #define VM_FREEPOOL_CACHE 1 #define VM_FREEPOOL_DEFAULT 0 #define VM_FREEPOOL_DIRECT 0 /* * Create two free page lists: VM_FREELIST_DEFAULT is for physical * pages that are above the largest physical address that is * accessible by ISA DMA and VM_FREELIST_ISADMA is for physical pages * that are below that address. */ #define VM_NFREELIST 2 #define VM_FREELIST_DEFAULT 0 #define VM_FREELIST_ISADMA 1 /* * The largest allocation size is 2MB under PAE and 4MB otherwise. */ #ifdef PAE #define VM_NFREEORDER 10 #else #define VM_NFREEORDER 11 #endif /* * Enable superpage reservations: 1 level. */ #ifndef VM_NRESERVLEVEL #define VM_NRESERVLEVEL 1 #endif /* - * Level 0 reservations consist of 512 pages under PAE and 1024 pages - * otherwise. + * Level 0 reservations consist of 512 pages when PAE pagetables are + * used, and 1024 pages otherwise. */ #ifndef VM_LEVEL_0_ORDER -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) #define VM_LEVEL_0_ORDER 9 #else #define VM_LEVEL_0_ORDER 10 #endif #endif /* * Kernel physical load address. */ #ifndef KERNLOAD #if defined(XEN) && !defined(XEN_PRIVILEGED_GUEST) #define KERNLOAD 0 #else #define KERNLOAD (1 << PDRSHIFT) #endif #endif /* !defined(KERNLOAD) */ /* * Virtual addresses of things. Derived from the page directory and * page table indexes from pmap.h for precision. * Because of the page that is both a PD and PT, it looks a little * messy at times, but hey, we'll do anything to save a page :-) */ #ifdef XEN #define VM_MAX_KERNEL_ADDRESS HYPERVISOR_VIRT_START #else #define VM_MAX_KERNEL_ADDRESS VADDR(KPTDI+NKPDE-1, NPTEPG-1) #endif #define VM_MIN_KERNEL_ADDRESS VADDR(PTDPTDI, PTDPTDI) #define KERNBASE VADDR(KPTDI, 0) #define UPT_MAX_ADDRESS VADDR(PTDPTDI, PTDPTDI) #define UPT_MIN_ADDRESS VADDR(PTDPTDI, 0) #define VM_MAXUSER_ADDRESS VADDR(PTDPTDI, 0) #define SHAREDPAGE (VM_MAXUSER_ADDRESS - PAGE_SIZE) #define USRSTACK SHAREDPAGE #define VM_MAX_ADDRESS VADDR(PTDPTDI, PTDPTDI) #define VM_MIN_ADDRESS ((vm_offset_t)0) /* * How many physical pages per kmem arena virtual page. */ #ifndef VM_KMEM_SIZE_SCALE #define VM_KMEM_SIZE_SCALE (3) #endif /* * Optional floor (in bytes) on the size of the kmem arena. */ #ifndef VM_KMEM_SIZE_MIN #define VM_KMEM_SIZE_MIN (12 * 1024 * 1024) #endif /* * Optional ceiling (in bytes) on the size of the kmem arena: 40% of the * kernel map rounded to the nearest multiple of the superpage size. */ #ifndef VM_KMEM_SIZE_MAX #define VM_KMEM_SIZE_MAX (((((VM_MAX_KERNEL_ADDRESS - \ VM_MIN_KERNEL_ADDRESS) >> (PDRSHIFT - 2)) + 5) / 10) << PDRSHIFT) #endif /* initial pagein size of beginning of executable file */ #ifndef VM_INITIAL_PAGEIN #define VM_INITIAL_PAGEIN 16 #endif #define ZERO_REGION_SIZE (64 * 1024) /* 64KB */ #ifndef VM_MAX_AUTOTUNE_MAXUSERS #define VM_MAX_AUTOTUNE_MAXUSERS 384 #endif #define SFBUF #define SFBUF_MAP #define SFBUF_CPUSET #define SFBUF_PROCESS_PAGE #endif /* _MACHINE_VMPARAM_H_ */ Index: user/ngie/more-tests/sys/netinet6/nd6.c =================================================================== --- user/ngie/more-tests/sys/netinet6/nd6.c (revision 281503) +++ user/ngie/more-tests/sys/netinet6/nd6.c (revision 281504) @@ -1,2410 +1,2409 @@ /*- * Copyright (C) 1995, 1996, 1997, and 1998 WIDE Project. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. Neither the name of the project nor the names of its contributors * may be used to endorse or promote products derived from this software * without specific prior written permission. * * THIS SOFTWARE IS PROVIDED BY THE PROJECT AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE PROJECT OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $KAME: nd6.c,v 1.144 2001/05/24 07:44:00 itojun Exp $ */ #include __FBSDID("$FreeBSD$"); #include "opt_inet.h" #include "opt_inet6.h" #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #define L3_ADDR_SIN6(le) ((struct sockaddr_in6 *) L3_ADDR(le)) #include #include #include #include #include #include #include #include #include #include #include #define ND6_SLOWTIMER_INTERVAL (60 * 60) /* 1 hour */ #define ND6_RECALC_REACHTM_INTERVAL (60 * 120) /* 2 hours */ #define SIN6(s) ((const struct sockaddr_in6 *)(s)) /* timer values */ VNET_DEFINE(int, nd6_prune) = 1; /* walk list every 1 seconds */ VNET_DEFINE(int, nd6_delay) = 5; /* delay first probe time 5 second */ VNET_DEFINE(int, nd6_umaxtries) = 3; /* maximum unicast query */ VNET_DEFINE(int, nd6_mmaxtries) = 3; /* maximum multicast query */ VNET_DEFINE(int, nd6_useloopback) = 1; /* use loopback interface for * local traffic */ VNET_DEFINE(int, nd6_gctimer) = (60 * 60 * 24); /* 1 day: garbage * collection timer */ /* preventing too many loops in ND option parsing */ static VNET_DEFINE(int, nd6_maxndopt) = 10; /* max # of ND options allowed */ VNET_DEFINE(int, nd6_maxnudhint) = 0; /* max # of subsequent upper * layer hints */ static VNET_DEFINE(int, nd6_maxqueuelen) = 1; /* max pkts cached in unresolved * ND entries */ #define V_nd6_maxndopt VNET(nd6_maxndopt) #define V_nd6_maxqueuelen VNET(nd6_maxqueuelen) #ifdef ND6_DEBUG VNET_DEFINE(int, nd6_debug) = 1; #else VNET_DEFINE(int, nd6_debug) = 0; #endif /* for debugging? */ #if 0 static int nd6_inuse, nd6_allocated; #endif VNET_DEFINE(struct nd_drhead, nd_defrouter); VNET_DEFINE(struct nd_prhead, nd_prefix); VNET_DEFINE(int, nd6_recalc_reachtm_interval) = ND6_RECALC_REACHTM_INTERVAL; #define V_nd6_recalc_reachtm_interval VNET(nd6_recalc_reachtm_interval) int (*send_sendso_input_hook)(struct mbuf *, struct ifnet *, int, int); static int nd6_is_new_addr_neighbor(struct sockaddr_in6 *, struct ifnet *); static void nd6_setmtu0(struct ifnet *, struct nd_ifinfo *); static void nd6_slowtimo(void *); static int regen_tmpaddr(struct in6_ifaddr *); static struct llentry *nd6_free(struct llentry *, int); static void nd6_llinfo_timer(void *); static void clear_llinfo_pqueue(struct llentry *); static void nd6_rtrequest(int, struct rtentry *, struct rt_addrinfo *); static int nd6_output_lle(struct ifnet *, struct ifnet *, struct mbuf *, struct sockaddr_in6 *); static int nd6_output_ifp(struct ifnet *, struct ifnet *, struct mbuf *, struct sockaddr_in6 *); static VNET_DEFINE(struct callout, nd6_slowtimo_ch); #define V_nd6_slowtimo_ch VNET(nd6_slowtimo_ch) VNET_DEFINE(struct callout, nd6_timer_ch); void nd6_init(void) { LIST_INIT(&V_nd_prefix); /* initialization of the default router list */ TAILQ_INIT(&V_nd_defrouter); /* start timer */ callout_init(&V_nd6_slowtimo_ch, 0); callout_reset(&V_nd6_slowtimo_ch, ND6_SLOWTIMER_INTERVAL * hz, nd6_slowtimo, curvnet); nd6_dad_init(); } #ifdef VIMAGE void nd6_destroy() { callout_drain(&V_nd6_slowtimo_ch); callout_drain(&V_nd6_timer_ch); } #endif struct nd_ifinfo * nd6_ifattach(struct ifnet *ifp) { struct nd_ifinfo *nd; nd = (struct nd_ifinfo *)malloc(sizeof(*nd), M_IP6NDP, M_WAITOK|M_ZERO); nd->initialized = 1; nd->chlim = IPV6_DEFHLIM; nd->basereachable = REACHABLE_TIME; nd->reachable = ND_COMPUTE_RTIME(nd->basereachable); nd->retrans = RETRANS_TIMER; nd->flags = ND6_IFF_PERFORMNUD; /* A loopback interface always has ND6_IFF_AUTO_LINKLOCAL. * XXXHRS: Clear ND6_IFF_AUTO_LINKLOCAL on an IFT_BRIDGE interface by * default regardless of the V_ip6_auto_linklocal configuration to * give a reasonable default behavior. */ if ((V_ip6_auto_linklocal && ifp->if_type != IFT_BRIDGE) || (ifp->if_flags & IFF_LOOPBACK)) nd->flags |= ND6_IFF_AUTO_LINKLOCAL; /* * A loopback interface does not need to accept RTADV. * XXXHRS: Clear ND6_IFF_ACCEPT_RTADV on an IFT_BRIDGE interface by * default regardless of the V_ip6_accept_rtadv configuration to * prevent the interface from accepting RA messages arrived * on one of the member interfaces with ND6_IFF_ACCEPT_RTADV. */ if (V_ip6_accept_rtadv && !(ifp->if_flags & IFF_LOOPBACK) && (ifp->if_type != IFT_BRIDGE)) nd->flags |= ND6_IFF_ACCEPT_RTADV; if (V_ip6_no_radr && !(ifp->if_flags & IFF_LOOPBACK)) nd->flags |= ND6_IFF_NO_RADR; /* XXX: we cannot call nd6_setmtu since ifp is not fully initialized */ nd6_setmtu0(ifp, nd); return nd; } void nd6_ifdetach(struct nd_ifinfo *nd) { free(nd, M_IP6NDP); } /* * Reset ND level link MTU. This function is called when the physical MTU * changes, which means we might have to adjust the ND level MTU. */ void nd6_setmtu(struct ifnet *ifp) { nd6_setmtu0(ifp, ND_IFINFO(ifp)); } /* XXX todo: do not maintain copy of ifp->if_mtu in ndi->maxmtu */ void nd6_setmtu0(struct ifnet *ifp, struct nd_ifinfo *ndi) { u_int32_t omaxmtu; omaxmtu = ndi->maxmtu; switch (ifp->if_type) { case IFT_ARCNET: ndi->maxmtu = MIN(ARC_PHDS_MAXMTU, ifp->if_mtu); /* RFC2497 */ break; case IFT_FDDI: ndi->maxmtu = MIN(FDDIIPMTU, ifp->if_mtu); /* RFC2467 */ break; case IFT_ISO88025: ndi->maxmtu = MIN(ISO88025_MAX_MTU, ifp->if_mtu); break; default: ndi->maxmtu = ifp->if_mtu; break; } /* * Decreasing the interface MTU under IPV6 minimum MTU may cause * undesirable situation. We thus notify the operator of the change * explicitly. The check for omaxmtu is necessary to restrict the * log to the case of changing the MTU, not initializing it. */ if (omaxmtu >= IPV6_MMTU && ndi->maxmtu < IPV6_MMTU) { log(LOG_NOTICE, "nd6_setmtu0: " "new link MTU on %s (%lu) is too small for IPv6\n", if_name(ifp), (unsigned long)ndi->maxmtu); } if (ndi->maxmtu > V_in6_maxmtu) in6_setmaxmtu(); /* check all interfaces just in case */ } void nd6_option_init(void *opt, int icmp6len, union nd_opts *ndopts) { bzero(ndopts, sizeof(*ndopts)); ndopts->nd_opts_search = (struct nd_opt_hdr *)opt; ndopts->nd_opts_last = (struct nd_opt_hdr *)(((u_char *)opt) + icmp6len); if (icmp6len == 0) { ndopts->nd_opts_done = 1; ndopts->nd_opts_search = NULL; } } /* * Take one ND option. */ struct nd_opt_hdr * nd6_option(union nd_opts *ndopts) { struct nd_opt_hdr *nd_opt; int olen; KASSERT(ndopts != NULL, ("%s: ndopts == NULL", __func__)); KASSERT(ndopts->nd_opts_last != NULL, ("%s: uninitialized ndopts", __func__)); if (ndopts->nd_opts_search == NULL) return NULL; if (ndopts->nd_opts_done) return NULL; nd_opt = ndopts->nd_opts_search; /* make sure nd_opt_len is inside the buffer */ if ((caddr_t)&nd_opt->nd_opt_len >= (caddr_t)ndopts->nd_opts_last) { bzero(ndopts, sizeof(*ndopts)); return NULL; } olen = nd_opt->nd_opt_len << 3; if (olen == 0) { /* * Message validation requires that all included * options have a length that is greater than zero. */ bzero(ndopts, sizeof(*ndopts)); return NULL; } ndopts->nd_opts_search = (struct nd_opt_hdr *)((caddr_t)nd_opt + olen); if (ndopts->nd_opts_search > ndopts->nd_opts_last) { /* option overruns the end of buffer, invalid */ bzero(ndopts, sizeof(*ndopts)); return NULL; } else if (ndopts->nd_opts_search == ndopts->nd_opts_last) { /* reached the end of options chain */ ndopts->nd_opts_done = 1; ndopts->nd_opts_search = NULL; } return nd_opt; } /* * Parse multiple ND options. * This function is much easier to use, for ND routines that do not need * multiple options of the same type. */ int nd6_options(union nd_opts *ndopts) { struct nd_opt_hdr *nd_opt; int i = 0; KASSERT(ndopts != NULL, ("%s: ndopts == NULL", __func__)); KASSERT(ndopts->nd_opts_last != NULL, ("%s: uninitialized ndopts", __func__)); if (ndopts->nd_opts_search == NULL) return 0; while (1) { nd_opt = nd6_option(ndopts); if (nd_opt == NULL && ndopts->nd_opts_last == NULL) { /* * Message validation requires that all included * options have a length that is greater than zero. */ ICMP6STAT_INC(icp6s_nd_badopt); bzero(ndopts, sizeof(*ndopts)); return -1; } if (nd_opt == NULL) goto skip1; switch (nd_opt->nd_opt_type) { case ND_OPT_SOURCE_LINKADDR: case ND_OPT_TARGET_LINKADDR: case ND_OPT_MTU: case ND_OPT_REDIRECTED_HEADER: case ND_OPT_NONCE: if (ndopts->nd_opt_array[nd_opt->nd_opt_type]) { nd6log((LOG_INFO, "duplicated ND6 option found (type=%d)\n", nd_opt->nd_opt_type)); /* XXX bark? */ } else { ndopts->nd_opt_array[nd_opt->nd_opt_type] = nd_opt; } break; case ND_OPT_PREFIX_INFORMATION: if (ndopts->nd_opt_array[nd_opt->nd_opt_type] == 0) { ndopts->nd_opt_array[nd_opt->nd_opt_type] = nd_opt; } ndopts->nd_opts_pi_end = (struct nd_opt_prefix_info *)nd_opt; break; /* What about ND_OPT_ROUTE_INFO? RFC 4191 */ case ND_OPT_RDNSS: /* RFC 6106 */ case ND_OPT_DNSSL: /* RFC 6106 */ /* * Silently ignore options we know and do not care about * in the kernel. */ break; default: /* * Unknown options must be silently ignored, * to accomodate future extension to the protocol. */ nd6log((LOG_DEBUG, "nd6_options: unsupported option %d - " "option ignored\n", nd_opt->nd_opt_type)); } skip1: i++; if (i > V_nd6_maxndopt) { ICMP6STAT_INC(icp6s_nd_toomanyopt); nd6log((LOG_INFO, "too many loop in nd opt\n")); break; } if (ndopts->nd_opts_done) break; } return 0; } /* * ND6 timer routine to handle ND6 entries */ void nd6_llinfo_settimer_locked(struct llentry *ln, long tick) { int canceled; LLE_WLOCK_ASSERT(ln); if (tick < 0) { ln->la_expire = 0; ln->ln_ntick = 0; canceled = callout_stop(&ln->ln_timer_ch); } else { ln->la_expire = time_uptime + tick / hz; LLE_ADDREF(ln); if (tick > INT_MAX) { ln->ln_ntick = tick - INT_MAX; canceled = callout_reset(&ln->ln_timer_ch, INT_MAX, nd6_llinfo_timer, ln); } else { ln->ln_ntick = 0; canceled = callout_reset(&ln->ln_timer_ch, tick, nd6_llinfo_timer, ln); } } if (canceled) LLE_REMREF(ln); } void nd6_llinfo_settimer(struct llentry *ln, long tick) { LLE_WLOCK(ln); nd6_llinfo_settimer_locked(ln, tick); LLE_WUNLOCK(ln); } static void nd6_llinfo_timer(void *arg) { struct llentry *ln; struct in6_addr *dst; struct ifnet *ifp; struct nd_ifinfo *ndi = NULL; KASSERT(arg != NULL, ("%s: arg NULL", __func__)); ln = (struct llentry *)arg; LLE_WLOCK(ln); if (callout_pending(&ln->la_timer)) { /* * Here we are a bit odd here in the treatment of * active/pending. If the pending bit is set, it got * rescheduled before I ran. The active * bit we ignore, since if it was stopped * in ll_tablefree() and was currently running * it would have return 0 so the code would * not have deleted it since the callout could * not be stopped so we want to go through * with the delete here now. If the callout * was restarted, the pending bit will be back on and * we just want to bail since the callout_reset would * return 1 and our reference would have been removed * by nd6_llinfo_settimer_locked above since canceled * would have been 1. */ LLE_WUNLOCK(ln); return; } ifp = ln->lle_tbl->llt_ifp; CURVNET_SET(ifp->if_vnet); if (ln->ln_ntick > 0) { if (ln->ln_ntick > INT_MAX) { ln->ln_ntick -= INT_MAX; nd6_llinfo_settimer_locked(ln, INT_MAX); } else { ln->ln_ntick = 0; nd6_llinfo_settimer_locked(ln, ln->ln_ntick); } goto done; } ndi = ND_IFINFO(ifp); dst = &L3_ADDR_SIN6(ln)->sin6_addr; if (ln->la_flags & LLE_STATIC) { goto done; } if (ln->la_flags & LLE_DELETED) { (void)nd6_free(ln, 0); ln = NULL; goto done; } switch (ln->ln_state) { case ND6_LLINFO_INCOMPLETE: if (ln->la_asked < V_nd6_mmaxtries) { ln->la_asked++; nd6_llinfo_settimer_locked(ln, (long)ndi->retrans * hz / 1000); LLE_WUNLOCK(ln); nd6_ns_output(ifp, NULL, dst, ln, NULL); LLE_WLOCK(ln); } else { struct mbuf *m = ln->la_hold; if (m) { struct mbuf *m0; /* * assuming every packet in la_hold has the * same IP header. Send error after unlock. */ m0 = m->m_nextpkt; m->m_nextpkt = NULL; ln->la_hold = m0; clear_llinfo_pqueue(ln); } EVENTHANDLER_INVOKE(lle_event, ln, LLENTRY_TIMEDOUT); (void)nd6_free(ln, 0); ln = NULL; if (m != NULL) icmp6_error2(m, ICMP6_DST_UNREACH, ICMP6_DST_UNREACH_ADDR, 0, ifp); } break; case ND6_LLINFO_REACHABLE: if (!ND6_LLINFO_PERMANENT(ln)) { ln->ln_state = ND6_LLINFO_STALE; nd6_llinfo_settimer_locked(ln, (long)V_nd6_gctimer * hz); } break; case ND6_LLINFO_STALE: /* Garbage Collection(RFC 2461 5.3) */ if (!ND6_LLINFO_PERMANENT(ln)) { EVENTHANDLER_INVOKE(lle_event, ln, LLENTRY_EXPIRED); (void)nd6_free(ln, 1); ln = NULL; } break; case ND6_LLINFO_DELAY: if (ndi && (ndi->flags & ND6_IFF_PERFORMNUD) != 0) { /* We need NUD */ ln->la_asked = 1; ln->ln_state = ND6_LLINFO_PROBE; nd6_llinfo_settimer_locked(ln, (long)ndi->retrans * hz / 1000); LLE_WUNLOCK(ln); nd6_ns_output(ifp, dst, dst, ln, NULL); LLE_WLOCK(ln); } else { ln->ln_state = ND6_LLINFO_STALE; /* XXX */ nd6_llinfo_settimer_locked(ln, (long)V_nd6_gctimer * hz); } break; case ND6_LLINFO_PROBE: if (ln->la_asked < V_nd6_umaxtries) { ln->la_asked++; nd6_llinfo_settimer_locked(ln, (long)ndi->retrans * hz / 1000); LLE_WUNLOCK(ln); nd6_ns_output(ifp, dst, dst, ln, NULL); LLE_WLOCK(ln); } else { EVENTHANDLER_INVOKE(lle_event, ln, LLENTRY_EXPIRED); (void)nd6_free(ln, 0); ln = NULL; } break; default: panic("%s: paths in a dark night can be confusing: %d", __func__, ln->ln_state); } done: if (ln != NULL) LLE_FREE_LOCKED(ln); CURVNET_RESTORE(); } /* * ND6 timer routine to expire default route list and prefix list */ void nd6_timer(void *arg) { CURVNET_SET((struct vnet *) arg); struct nd_defrouter *dr, *ndr; struct nd_prefix *pr, *npr; struct in6_ifaddr *ia6, *nia6; callout_reset(&V_nd6_timer_ch, V_nd6_prune * hz, nd6_timer, curvnet); /* expire default router list */ TAILQ_FOREACH_SAFE(dr, &V_nd_defrouter, dr_entry, ndr) { if (dr->expire && dr->expire < time_uptime) defrtrlist_del(dr); } /* * expire interface addresses. * in the past the loop was inside prefix expiry processing. * However, from a stricter speci-confrmance standpoint, we should * rather separate address lifetimes and prefix lifetimes. * * XXXRW: in6_ifaddrhead locking. */ addrloop: TAILQ_FOREACH_SAFE(ia6, &V_in6_ifaddrhead, ia_link, nia6) { /* check address lifetime */ if (IFA6_IS_INVALID(ia6)) { int regen = 0; /* * If the expiring address is temporary, try * regenerating a new one. This would be useful when * we suspended a laptop PC, then turned it on after a * period that could invalidate all temporary * addresses. Although we may have to restart the * loop (see below), it must be after purging the * address. Otherwise, we'd see an infinite loop of * regeneration. */ if (V_ip6_use_tempaddr && (ia6->ia6_flags & IN6_IFF_TEMPORARY) != 0) { if (regen_tmpaddr(ia6) == 0) regen = 1; } in6_purgeaddr(&ia6->ia_ifa); if (regen) goto addrloop; /* XXX: see below */ } else if (IFA6_IS_DEPRECATED(ia6)) { int oldflags = ia6->ia6_flags; ia6->ia6_flags |= IN6_IFF_DEPRECATED; /* * If a temporary address has just become deprecated, * regenerate a new one if possible. */ if (V_ip6_use_tempaddr && (ia6->ia6_flags & IN6_IFF_TEMPORARY) != 0 && (oldflags & IN6_IFF_DEPRECATED) == 0) { if (regen_tmpaddr(ia6) == 0) { /* * A new temporary address is * generated. * XXX: this means the address chain * has changed while we are still in * the loop. Although the change * would not cause disaster (because * it's not a deletion, but an * addition,) we'd rather restart the * loop just for safety. Or does this * significantly reduce performance?? */ goto addrloop; } } } else { /* * A new RA might have made a deprecated address * preferred. */ ia6->ia6_flags &= ~IN6_IFF_DEPRECATED; } } /* expire prefix list */ LIST_FOREACH_SAFE(pr, &V_nd_prefix, ndpr_entry, npr) { /* * check prefix lifetime. * since pltime is just for autoconf, pltime processing for * prefix is not necessary. */ if (pr->ndpr_vltime != ND6_INFINITE_LIFETIME && time_uptime - pr->ndpr_lastupdate > pr->ndpr_vltime) { /* * address expiration and prefix expiration are * separate. NEVER perform in6_purgeaddr here. */ prelist_remove(pr); } } CURVNET_RESTORE(); } /* * ia6 - deprecated/invalidated temporary address */ static int regen_tmpaddr(struct in6_ifaddr *ia6) { struct ifaddr *ifa; struct ifnet *ifp; struct in6_ifaddr *public_ifa6 = NULL; ifp = ia6->ia_ifa.ifa_ifp; IF_ADDR_RLOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { struct in6_ifaddr *it6; if (ifa->ifa_addr->sa_family != AF_INET6) continue; it6 = (struct in6_ifaddr *)ifa; /* ignore no autoconf addresses. */ if ((it6->ia6_flags & IN6_IFF_AUTOCONF) == 0) continue; /* ignore autoconf addresses with different prefixes. */ if (it6->ia6_ndpr == NULL || it6->ia6_ndpr != ia6->ia6_ndpr) continue; /* * Now we are looking at an autoconf address with the same * prefix as ours. If the address is temporary and is still * preferred, do not create another one. It would be rare, but * could happen, for example, when we resume a laptop PC after * a long period. */ if ((it6->ia6_flags & IN6_IFF_TEMPORARY) != 0 && !IFA6_IS_DEPRECATED(it6)) { public_ifa6 = NULL; break; } /* * This is a public autoconf address that has the same prefix * as ours. If it is preferred, keep it. We can't break the * loop here, because there may be a still-preferred temporary * address with the prefix. */ if (!IFA6_IS_DEPRECATED(it6)) - public_ifa6 = it6; - - if (public_ifa6 != NULL) - ifa_ref(&public_ifa6->ia_ifa); + public_ifa6 = it6; } + if (public_ifa6 != NULL) + ifa_ref(&public_ifa6->ia_ifa); IF_ADDR_RUNLOCK(ifp); if (public_ifa6 != NULL) { int e; if ((e = in6_tmpifadd(public_ifa6, 0, 0)) != 0) { ifa_free(&public_ifa6->ia_ifa); log(LOG_NOTICE, "regen_tmpaddr: failed to create a new" " tmp addr,errno=%d\n", e); return (-1); } ifa_free(&public_ifa6->ia_ifa); return (0); } return (-1); } /* * Nuke neighbor cache/prefix/default router management table, right before * ifp goes away. */ void nd6_purge(struct ifnet *ifp) { struct nd_defrouter *dr, *ndr; struct nd_prefix *pr, *npr; /* * Nuke default router list entries toward ifp. * We defer removal of default router list entries that is installed * in the routing table, in order to keep additional side effects as * small as possible. */ TAILQ_FOREACH_SAFE(dr, &V_nd_defrouter, dr_entry, ndr) { if (dr->installed) continue; if (dr->ifp == ifp) defrtrlist_del(dr); } TAILQ_FOREACH_SAFE(dr, &V_nd_defrouter, dr_entry, ndr) { if (!dr->installed) continue; if (dr->ifp == ifp) defrtrlist_del(dr); } /* Nuke prefix list entries toward ifp */ LIST_FOREACH_SAFE(pr, &V_nd_prefix, ndpr_entry, npr) { if (pr->ndpr_ifp == ifp) { /* * Because if_detach() does *not* release prefixes * while purging addresses the reference count will * still be above zero. We therefore reset it to * make sure that the prefix really gets purged. */ pr->ndpr_refcnt = 0; /* * Previously, pr->ndpr_addr is removed as well, * but I strongly believe we don't have to do it. * nd6_purge() is only called from in6_ifdetach(), * which removes all the associated interface addresses * by itself. * (jinmei@kame.net 20010129) */ prelist_remove(pr); } } /* cancel default outgoing interface setting */ if (V_nd6_defifindex == ifp->if_index) nd6_setdefaultiface(0); if (ND_IFINFO(ifp)->flags & ND6_IFF_ACCEPT_RTADV) { /* Refresh default router list. */ defrouter_select(); } /* XXXXX * We do not nuke the neighbor cache entries here any more * because the neighbor cache is kept in if_afdata[AF_INET6]. * nd6_purge() is invoked by in6_ifdetach() which is called * from if_detach() where everything gets purged. So let * in6_domifdetach() do the actual L2 table purging work. */ } /* * the caller acquires and releases the lock on the lltbls * Returns the llentry locked */ struct llentry * nd6_lookup(struct in6_addr *addr6, int flags, struct ifnet *ifp) { struct sockaddr_in6 sin6; struct llentry *ln; int llflags; bzero(&sin6, sizeof(sin6)); sin6.sin6_len = sizeof(struct sockaddr_in6); sin6.sin6_family = AF_INET6; sin6.sin6_addr = *addr6; IF_AFDATA_LOCK_ASSERT(ifp); llflags = 0; if (flags & ND6_CREATE) llflags |= LLE_CREATE; if (flags & ND6_EXCLUSIVE) llflags |= LLE_EXCLUSIVE; ln = lla_lookup(LLTABLE6(ifp), llflags, (struct sockaddr *)&sin6); if ((ln != NULL) && (llflags & LLE_CREATE)) ln->ln_state = ND6_LLINFO_NOSTATE; return (ln); } /* * Test whether a given IPv6 address is a neighbor or not, ignoring * the actual neighbor cache. The neighbor cache is ignored in order * to not reenter the routing code from within itself. */ static int nd6_is_new_addr_neighbor(struct sockaddr_in6 *addr, struct ifnet *ifp) { struct nd_prefix *pr; struct ifaddr *dstaddr; /* * A link-local address is always a neighbor. * XXX: a link does not necessarily specify a single interface. */ if (IN6_IS_ADDR_LINKLOCAL(&addr->sin6_addr)) { struct sockaddr_in6 sin6_copy; u_int32_t zone; /* * We need sin6_copy since sa6_recoverscope() may modify the * content (XXX). */ sin6_copy = *addr; if (sa6_recoverscope(&sin6_copy)) return (0); /* XXX: should be impossible */ if (in6_setscope(&sin6_copy.sin6_addr, ifp, &zone)) return (0); if (sin6_copy.sin6_scope_id == zone) return (1); else return (0); } /* * If the address matches one of our addresses, * it should be a neighbor. * If the address matches one of our on-link prefixes, it should be a * neighbor. */ LIST_FOREACH(pr, &V_nd_prefix, ndpr_entry) { if (pr->ndpr_ifp != ifp) continue; if (!(pr->ndpr_stateflags & NDPRF_ONLINK)) { struct rtentry *rt; /* Always use the default FIB here. */ rt = in6_rtalloc1((struct sockaddr *)&pr->ndpr_prefix, 0, 0, RT_DEFAULT_FIB); if (rt == NULL) continue; /* * This is the case where multiple interfaces * have the same prefix, but only one is installed * into the routing table and that prefix entry * is not the one being examined here. In the case * where RADIX_MPATH is enabled, multiple route * entries (of the same rt_key value) will be * installed because the interface addresses all * differ. */ if (!IN6_ARE_ADDR_EQUAL(&pr->ndpr_prefix.sin6_addr, &((struct sockaddr_in6 *)rt_key(rt))->sin6_addr)) { RTFREE_LOCKED(rt); continue; } RTFREE_LOCKED(rt); } if (IN6_ARE_MASKED_ADDR_EQUAL(&pr->ndpr_prefix.sin6_addr, &addr->sin6_addr, &pr->ndpr_mask)) return (1); } /* * If the address is assigned on the node of the other side of * a p2p interface, the address should be a neighbor. */ dstaddr = ifa_ifwithdstaddr((struct sockaddr *)addr, RT_ALL_FIBS); if (dstaddr != NULL) { if (dstaddr->ifa_ifp == ifp) { ifa_free(dstaddr); return (1); } ifa_free(dstaddr); } /* * If the default router list is empty, all addresses are regarded * as on-link, and thus, as a neighbor. */ if (ND_IFINFO(ifp)->flags & ND6_IFF_ACCEPT_RTADV && TAILQ_EMPTY(&V_nd_defrouter) && V_nd6_defifindex == ifp->if_index) { return (1); } return (0); } /* * Detect if a given IPv6 address identifies a neighbor on a given link. * XXX: should take care of the destination of a p2p link? */ int nd6_is_addr_neighbor(struct sockaddr_in6 *addr, struct ifnet *ifp) { struct llentry *lle; int rc = 0; IF_AFDATA_UNLOCK_ASSERT(ifp); if (nd6_is_new_addr_neighbor(addr, ifp)) return (1); /* * Even if the address matches none of our addresses, it might be * in the neighbor cache. */ IF_AFDATA_RLOCK(ifp); if ((lle = nd6_lookup(&addr->sin6_addr, 0, ifp)) != NULL) { LLE_RUNLOCK(lle); rc = 1; } IF_AFDATA_RUNLOCK(ifp); return (rc); } /* * Free an nd6 llinfo entry. * Since the function would cause significant changes in the kernel, DO NOT * make it global, unless you have a strong reason for the change, and are sure * that the change is safe. */ static struct llentry * nd6_free(struct llentry *ln, int gc) { struct llentry *next; struct nd_defrouter *dr; struct ifnet *ifp; LLE_WLOCK_ASSERT(ln); /* * we used to have pfctlinput(PRC_HOSTDEAD) here. * even though it is not harmful, it was not really necessary. */ /* cancel timer */ nd6_llinfo_settimer_locked(ln, -1); ifp = ln->lle_tbl->llt_ifp; if (ND_IFINFO(ifp)->flags & ND6_IFF_ACCEPT_RTADV) { dr = defrouter_lookup(&L3_ADDR_SIN6(ln)->sin6_addr, ifp); if (dr != NULL && dr->expire && ln->ln_state == ND6_LLINFO_STALE && gc) { /* * If the reason for the deletion is just garbage * collection, and the neighbor is an active default * router, do not delete it. Instead, reset the GC * timer using the router's lifetime. * Simply deleting the entry would affect default * router selection, which is not necessarily a good * thing, especially when we're using router preference * values. * XXX: the check for ln_state would be redundant, * but we intentionally keep it just in case. */ if (dr->expire > time_uptime) nd6_llinfo_settimer_locked(ln, (dr->expire - time_uptime) * hz); else nd6_llinfo_settimer_locked(ln, (long)V_nd6_gctimer * hz); next = LIST_NEXT(ln, lle_next); LLE_REMREF(ln); LLE_WUNLOCK(ln); return (next); } if (dr) { /* * Unreachablity of a router might affect the default * router selection and on-link detection of advertised * prefixes. */ /* * Temporarily fake the state to choose a new default * router and to perform on-link determination of * prefixes correctly. * Below the state will be set correctly, * or the entry itself will be deleted. */ ln->ln_state = ND6_LLINFO_INCOMPLETE; } if (ln->ln_router || dr) { /* * We need to unlock to avoid a LOR with rt6_flush() with the * rnh and for the calls to pfxlist_onlink_check() and * defrouter_select() in the block further down for calls * into nd6_lookup(). We still hold a ref. */ LLE_WUNLOCK(ln); /* * rt6_flush must be called whether or not the neighbor * is in the Default Router List. * See a corresponding comment in nd6_na_input(). */ rt6_flush(&L3_ADDR_SIN6(ln)->sin6_addr, ifp); } if (dr) { /* * Since defrouter_select() does not affect the * on-link determination and MIP6 needs the check * before the default router selection, we perform * the check now. */ pfxlist_onlink_check(); /* * Refresh default router list. */ defrouter_select(); } if (ln->ln_router || dr) LLE_WLOCK(ln); } /* * Before deleting the entry, remember the next entry as the * return value. We need this because pfxlist_onlink_check() above * might have freed other entries (particularly the old next entry) as * a side effect (XXX). */ next = LIST_NEXT(ln, lle_next); /* * Save to unlock. We still hold an extra reference and will not * free(9) in llentry_free() if someone else holds one as well. */ LLE_WUNLOCK(ln); IF_AFDATA_LOCK(ifp); LLE_WLOCK(ln); /* Guard against race with other llentry_free(). */ if (ln->la_flags & LLE_LINKED) { LLE_REMREF(ln); llentry_free(ln); } else LLE_FREE_LOCKED(ln); IF_AFDATA_UNLOCK(ifp); return (next); } /* * Upper-layer reachability hint for Neighbor Unreachability Detection. * * XXX cost-effective methods? */ void nd6_nud_hint(struct rtentry *rt, struct in6_addr *dst6, int force) { struct llentry *ln; struct ifnet *ifp; if ((dst6 == NULL) || (rt == NULL)) return; ifp = rt->rt_ifp; IF_AFDATA_RLOCK(ifp); ln = nd6_lookup(dst6, ND6_EXCLUSIVE, NULL); IF_AFDATA_RUNLOCK(ifp); if (ln == NULL) return; if (ln->ln_state < ND6_LLINFO_REACHABLE) goto done; /* * if we get upper-layer reachability confirmation many times, * it is possible we have false information. */ if (!force) { ln->ln_byhint++; if (ln->ln_byhint > V_nd6_maxnudhint) { goto done; } } ln->ln_state = ND6_LLINFO_REACHABLE; if (!ND6_LLINFO_PERMANENT(ln)) { nd6_llinfo_settimer_locked(ln, (long)ND_IFINFO(rt->rt_ifp)->reachable * hz); } done: LLE_WUNLOCK(ln); } /* * Rejuvenate this function for routing operations related * processing. */ void nd6_rtrequest(int req, struct rtentry *rt, struct rt_addrinfo *info) { struct sockaddr_in6 *gateway; struct nd_defrouter *dr; struct ifnet *ifp; gateway = (struct sockaddr_in6 *)rt->rt_gateway; ifp = rt->rt_ifp; switch (req) { case RTM_ADD: break; case RTM_DELETE: if (!ifp) return; /* * Only indirect routes are interesting. */ if ((rt->rt_flags & RTF_GATEWAY) == 0) return; /* * check for default route */ if (IN6_ARE_ADDR_EQUAL(&in6addr_any, &SIN6(rt_key(rt))->sin6_addr)) { dr = defrouter_lookup(&gateway->sin6_addr, ifp); if (dr != NULL) dr->installed = 0; } break; } } int nd6_ioctl(u_long cmd, caddr_t data, struct ifnet *ifp) { struct in6_drlist *drl = (struct in6_drlist *)data; struct in6_oprlist *oprl = (struct in6_oprlist *)data; struct in6_ndireq *ndi = (struct in6_ndireq *)data; struct in6_nbrinfo *nbi = (struct in6_nbrinfo *)data; struct in6_ndifreq *ndif = (struct in6_ndifreq *)data; struct nd_defrouter *dr; struct nd_prefix *pr; int i = 0, error = 0; if (ifp->if_afdata[AF_INET6] == NULL) return (EPFNOSUPPORT); switch (cmd) { case SIOCGDRLST_IN6: /* * obsolete API, use sysctl under net.inet6.icmp6 */ bzero(drl, sizeof(*drl)); TAILQ_FOREACH(dr, &V_nd_defrouter, dr_entry) { if (i >= DRLSTSIZ) break; drl->defrouter[i].rtaddr = dr->rtaddr; in6_clearscope(&drl->defrouter[i].rtaddr); drl->defrouter[i].flags = dr->flags; drl->defrouter[i].rtlifetime = dr->rtlifetime; drl->defrouter[i].expire = dr->expire + (time_second - time_uptime); drl->defrouter[i].if_index = dr->ifp->if_index; i++; } break; case SIOCGPRLST_IN6: /* * obsolete API, use sysctl under net.inet6.icmp6 * * XXX the structure in6_prlist was changed in backward- * incompatible manner. in6_oprlist is used for SIOCGPRLST_IN6, * in6_prlist is used for nd6_sysctl() - fill_prlist(). */ /* * XXX meaning of fields, especialy "raflags", is very * differnet between RA prefix list and RR/static prefix list. * how about separating ioctls into two? */ bzero(oprl, sizeof(*oprl)); LIST_FOREACH(pr, &V_nd_prefix, ndpr_entry) { struct nd_pfxrouter *pfr; int j; if (i >= PRLSTSIZ) break; oprl->prefix[i].prefix = pr->ndpr_prefix.sin6_addr; oprl->prefix[i].raflags = pr->ndpr_raf; oprl->prefix[i].prefixlen = pr->ndpr_plen; oprl->prefix[i].vltime = pr->ndpr_vltime; oprl->prefix[i].pltime = pr->ndpr_pltime; oprl->prefix[i].if_index = pr->ndpr_ifp->if_index; if (pr->ndpr_vltime == ND6_INFINITE_LIFETIME) oprl->prefix[i].expire = 0; else { time_t maxexpire; /* XXX: we assume time_t is signed. */ maxexpire = (-1) & ~((time_t)1 << ((sizeof(maxexpire) * 8) - 1)); if (pr->ndpr_vltime < maxexpire - pr->ndpr_lastupdate) { oprl->prefix[i].expire = pr->ndpr_lastupdate + pr->ndpr_vltime + (time_second - time_uptime); } else oprl->prefix[i].expire = maxexpire; } j = 0; LIST_FOREACH(pfr, &pr->ndpr_advrtrs, pfr_entry) { if (j < DRLSTSIZ) { #define RTRADDR oprl->prefix[i].advrtr[j] RTRADDR = pfr->router->rtaddr; in6_clearscope(&RTRADDR); #undef RTRADDR } j++; } oprl->prefix[i].advrtrs = j; oprl->prefix[i].origin = PR_ORIG_RA; i++; } break; case OSIOCGIFINFO_IN6: #define ND ndi->ndi /* XXX: old ndp(8) assumes a positive value for linkmtu. */ bzero(&ND, sizeof(ND)); ND.linkmtu = IN6_LINKMTU(ifp); ND.maxmtu = ND_IFINFO(ifp)->maxmtu; ND.basereachable = ND_IFINFO(ifp)->basereachable; ND.reachable = ND_IFINFO(ifp)->reachable; ND.retrans = ND_IFINFO(ifp)->retrans; ND.flags = ND_IFINFO(ifp)->flags; ND.recalctm = ND_IFINFO(ifp)->recalctm; ND.chlim = ND_IFINFO(ifp)->chlim; break; case SIOCGIFINFO_IN6: ND = *ND_IFINFO(ifp); break; case SIOCSIFINFO_IN6: /* * used to change host variables from userland. * intented for a use on router to reflect RA configurations. */ /* 0 means 'unspecified' */ if (ND.linkmtu != 0) { if (ND.linkmtu < IPV6_MMTU || ND.linkmtu > IN6_LINKMTU(ifp)) { error = EINVAL; break; } ND_IFINFO(ifp)->linkmtu = ND.linkmtu; } if (ND.basereachable != 0) { int obasereachable = ND_IFINFO(ifp)->basereachable; ND_IFINFO(ifp)->basereachable = ND.basereachable; if (ND.basereachable != obasereachable) ND_IFINFO(ifp)->reachable = ND_COMPUTE_RTIME(ND.basereachable); } if (ND.retrans != 0) ND_IFINFO(ifp)->retrans = ND.retrans; if (ND.chlim != 0) ND_IFINFO(ifp)->chlim = ND.chlim; /* FALLTHROUGH */ case SIOCSIFINFO_FLAGS: { struct ifaddr *ifa; struct in6_ifaddr *ia; if ((ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED) && !(ND.flags & ND6_IFF_IFDISABLED)) { /* ifdisabled 1->0 transision */ /* * If the interface is marked as ND6_IFF_IFDISABLED and * has an link-local address with IN6_IFF_DUPLICATED, * do not clear ND6_IFF_IFDISABLED. * See RFC 4862, Section 5.4.5. */ int duplicated_linklocal = 0; IF_ADDR_RLOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family != AF_INET6) continue; ia = (struct in6_ifaddr *)ifa; if ((ia->ia6_flags & IN6_IFF_DUPLICATED) && IN6_IS_ADDR_LINKLOCAL(IA6_IN6(ia))) { duplicated_linklocal = 1; break; } } IF_ADDR_RUNLOCK(ifp); if (duplicated_linklocal) { ND.flags |= ND6_IFF_IFDISABLED; log(LOG_ERR, "Cannot enable an interface" " with a link-local address marked" " duplicate.\n"); } else { ND_IFINFO(ifp)->flags &= ~ND6_IFF_IFDISABLED; if (ifp->if_flags & IFF_UP) in6_if_up(ifp); } } else if (!(ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED) && (ND.flags & ND6_IFF_IFDISABLED)) { /* ifdisabled 0->1 transision */ /* Mark all IPv6 address as tentative. */ ND_IFINFO(ifp)->flags |= ND6_IFF_IFDISABLED; IF_ADDR_RLOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family != AF_INET6) continue; ia = (struct in6_ifaddr *)ifa; ia->ia6_flags |= IN6_IFF_TENTATIVE; } IF_ADDR_RUNLOCK(ifp); } if (ND.flags & ND6_IFF_AUTO_LINKLOCAL) { if (!(ND_IFINFO(ifp)->flags & ND6_IFF_AUTO_LINKLOCAL)) { /* auto_linklocal 0->1 transision */ /* If no link-local address on ifp, configure */ ND_IFINFO(ifp)->flags |= ND6_IFF_AUTO_LINKLOCAL; in6_ifattach(ifp, NULL); } else if (!(ND.flags & ND6_IFF_IFDISABLED) && ifp->if_flags & IFF_UP) { /* * When the IF already has * ND6_IFF_AUTO_LINKLOCAL, no link-local * address is assigned, and IFF_UP, try to * assign one. */ int haslinklocal = 0; IF_ADDR_RLOCK(ifp); TAILQ_FOREACH(ifa, &ifp->if_addrhead, ifa_link) { if (ifa->ifa_addr->sa_family != AF_INET6) continue; ia = (struct in6_ifaddr *)ifa; if (IN6_IS_ADDR_LINKLOCAL(IA6_IN6(ia))) { haslinklocal = 1; break; } } IF_ADDR_RUNLOCK(ifp); if (!haslinklocal) in6_ifattach(ifp, NULL); } } } ND_IFINFO(ifp)->flags = ND.flags; break; #undef ND case SIOCSNDFLUSH_IN6: /* XXX: the ioctl name is confusing... */ /* sync kernel routing table with the default router list */ defrouter_reset(); defrouter_select(); break; case SIOCSPFXFLUSH_IN6: { /* flush all the prefix advertised by routers */ struct nd_prefix *pr, *next; LIST_FOREACH_SAFE(pr, &V_nd_prefix, ndpr_entry, next) { struct in6_ifaddr *ia, *ia_next; if (IN6_IS_ADDR_LINKLOCAL(&pr->ndpr_prefix.sin6_addr)) continue; /* XXX */ /* do we really have to remove addresses as well? */ /* XXXRW: in6_ifaddrhead locking. */ TAILQ_FOREACH_SAFE(ia, &V_in6_ifaddrhead, ia_link, ia_next) { if ((ia->ia6_flags & IN6_IFF_AUTOCONF) == 0) continue; if (ia->ia6_ndpr == pr) in6_purgeaddr(&ia->ia_ifa); } prelist_remove(pr); } break; } case SIOCSRTRFLUSH_IN6: { /* flush all the default routers */ struct nd_defrouter *dr, *next; defrouter_reset(); TAILQ_FOREACH_SAFE(dr, &V_nd_defrouter, dr_entry, next) { defrtrlist_del(dr); } defrouter_select(); break; } case SIOCGNBRINFO_IN6: { struct llentry *ln; struct in6_addr nb_addr = nbi->addr; /* make local for safety */ if ((error = in6_setscope(&nb_addr, ifp, NULL)) != 0) return (error); IF_AFDATA_RLOCK(ifp); ln = nd6_lookup(&nb_addr, 0, ifp); IF_AFDATA_RUNLOCK(ifp); if (ln == NULL) { error = EINVAL; break; } nbi->state = ln->ln_state; nbi->asked = ln->la_asked; nbi->isrouter = ln->ln_router; if (ln->la_expire == 0) nbi->expire = 0; else nbi->expire = ln->la_expire + (time_second - time_uptime); LLE_RUNLOCK(ln); break; } case SIOCGDEFIFACE_IN6: /* XXX: should be implemented as a sysctl? */ ndif->ifindex = V_nd6_defifindex; break; case SIOCSDEFIFACE_IN6: /* XXX: should be implemented as a sysctl? */ return (nd6_setdefaultiface(ndif->ifindex)); } return (error); } /* * Create neighbor cache entry and cache link-layer address, * on reception of inbound ND6 packets. (RS/RA/NS/redirect) * * type - ICMP6 type * code - type dependent information * * XXXXX * The caller of this function already acquired the ndp * cache table lock because the cache entry is returned. */ struct llentry * nd6_cache_lladdr(struct ifnet *ifp, struct in6_addr *from, char *lladdr, int lladdrlen, int type, int code) { struct llentry *ln = NULL; int is_newentry; int do_update; int olladdr; int llchange; int flags; int newstate = 0; uint16_t router = 0; struct sockaddr_in6 sin6; struct mbuf *chain = NULL; int static_route = 0; IF_AFDATA_UNLOCK_ASSERT(ifp); KASSERT(ifp != NULL, ("%s: ifp == NULL", __func__)); KASSERT(from != NULL, ("%s: from == NULL", __func__)); /* nothing must be updated for unspecified address */ if (IN6_IS_ADDR_UNSPECIFIED(from)) return NULL; /* * Validation about ifp->if_addrlen and lladdrlen must be done in * the caller. * * XXX If the link does not have link-layer adderss, what should * we do? (ifp->if_addrlen == 0) * Spec says nothing in sections for RA, RS and NA. There's small * description on it in NS section (RFC 2461 7.2.3). */ flags = lladdr ? ND6_EXCLUSIVE : 0; IF_AFDATA_RLOCK(ifp); ln = nd6_lookup(from, flags, ifp); IF_AFDATA_RUNLOCK(ifp); if (ln == NULL) { flags |= ND6_EXCLUSIVE; IF_AFDATA_LOCK(ifp); ln = nd6_lookup(from, flags | ND6_CREATE, ifp); IF_AFDATA_UNLOCK(ifp); is_newentry = 1; } else { /* do nothing if static ndp is set */ if (ln->la_flags & LLE_STATIC) { static_route = 1; goto done; } is_newentry = 0; } if (ln == NULL) return (NULL); olladdr = (ln->la_flags & LLE_VALID) ? 1 : 0; if (olladdr && lladdr) { llchange = bcmp(lladdr, &ln->ll_addr, ifp->if_addrlen); } else llchange = 0; /* * newentry olladdr lladdr llchange (*=record) * 0 n n -- (1) * 0 y n -- (2) * 0 n y -- (3) * STALE * 0 y y n (4) * * 0 y y y (5) * STALE * 1 -- n -- (6) NOSTATE(= PASSIVE) * 1 -- y -- (7) * STALE */ if (lladdr) { /* (3-5) and (7) */ /* * Record source link-layer address * XXX is it dependent to ifp->if_type? */ bcopy(lladdr, &ln->ll_addr, ifp->if_addrlen); ln->la_flags |= LLE_VALID; EVENTHANDLER_INVOKE(lle_event, ln, LLENTRY_RESOLVED); } if (!is_newentry) { if ((!olladdr && lladdr != NULL) || /* (3) */ (olladdr && lladdr != NULL && llchange)) { /* (5) */ do_update = 1; newstate = ND6_LLINFO_STALE; } else /* (1-2,4) */ do_update = 0; } else { do_update = 1; if (lladdr == NULL) /* (6) */ newstate = ND6_LLINFO_NOSTATE; else /* (7) */ newstate = ND6_LLINFO_STALE; } if (do_update) { /* * Update the state of the neighbor cache. */ ln->ln_state = newstate; if (ln->ln_state == ND6_LLINFO_STALE) { if (ln->la_hold != NULL) nd6_grab_holdchain(ln, &chain, &sin6); } else if (ln->ln_state == ND6_LLINFO_INCOMPLETE) { /* probe right away */ nd6_llinfo_settimer_locked((void *)ln, 0); } } /* * ICMP6 type dependent behavior. * * NS: clear IsRouter if new entry * RS: clear IsRouter * RA: set IsRouter if there's lladdr * redir: clear IsRouter if new entry * * RA case, (1): * The spec says that we must set IsRouter in the following cases: * - If lladdr exist, set IsRouter. This means (1-5). * - If it is old entry (!newentry), set IsRouter. This means (7). * So, based on the spec, in (1-5) and (7) cases we must set IsRouter. * A quetion arises for (1) case. (1) case has no lladdr in the * neighbor cache, this is similar to (6). * This case is rare but we figured that we MUST NOT set IsRouter. * * newentry olladdr lladdr llchange NS RS RA redir * D R * 0 n n -- (1) c ? s * 0 y n -- (2) c s s * 0 n y -- (3) c s s * 0 y y n (4) c s s * 0 y y y (5) c s s * 1 -- n -- (6) c c c s * 1 -- y -- (7) c c s c s * * (c=clear s=set) */ switch (type & 0xff) { case ND_NEIGHBOR_SOLICIT: /* * New entry must have is_router flag cleared. */ if (is_newentry) /* (6-7) */ ln->ln_router = 0; break; case ND_REDIRECT: /* * If the icmp is a redirect to a better router, always set the * is_router flag. Otherwise, if the entry is newly created, * clear the flag. [RFC 2461, sec 8.3] */ if (code == ND_REDIRECT_ROUTER) ln->ln_router = 1; else if (is_newentry) /* (6-7) */ ln->ln_router = 0; break; case ND_ROUTER_SOLICIT: /* * is_router flag must always be cleared. */ ln->ln_router = 0; break; case ND_ROUTER_ADVERT: /* * Mark an entry with lladdr as a router. */ if ((!is_newentry && (olladdr || lladdr)) || /* (2-5) */ (is_newentry && lladdr)) { /* (7) */ ln->ln_router = 1; } break; } if (ln != NULL) { static_route = (ln->la_flags & LLE_STATIC); router = ln->ln_router; if (flags & ND6_EXCLUSIVE) LLE_WUNLOCK(ln); else LLE_RUNLOCK(ln); if (static_route) ln = NULL; } if (chain != NULL) nd6_flush_holdchain(ifp, ifp, chain, &sin6); /* * When the link-layer address of a router changes, select the * best router again. In particular, when the neighbor entry is newly * created, it might affect the selection policy. * Question: can we restrict the first condition to the "is_newentry" * case? * XXX: when we hear an RA from a new router with the link-layer * address option, defrouter_select() is called twice, since * defrtrlist_update called the function as well. However, I believe * we can compromise the overhead, since it only happens the first * time. * XXX: although defrouter_select() should not have a bad effect * for those are not autoconfigured hosts, we explicitly avoid such * cases for safety. */ if (do_update && router && ND_IFINFO(ifp)->flags & ND6_IFF_ACCEPT_RTADV) { /* * guaranteed recursion */ defrouter_select(); } return (ln); done: if (ln != NULL) { if (flags & ND6_EXCLUSIVE) LLE_WUNLOCK(ln); else LLE_RUNLOCK(ln); if (static_route) ln = NULL; } return (ln); } static void nd6_slowtimo(void *arg) { CURVNET_SET((struct vnet *) arg); struct nd_ifinfo *nd6if; struct ifnet *ifp; callout_reset(&V_nd6_slowtimo_ch, ND6_SLOWTIMER_INTERVAL * hz, nd6_slowtimo, curvnet); IFNET_RLOCK_NOSLEEP(); TAILQ_FOREACH(ifp, &V_ifnet, if_link) { if (ifp->if_afdata[AF_INET6] == NULL) continue; nd6if = ND_IFINFO(ifp); if (nd6if->basereachable && /* already initialized */ (nd6if->recalctm -= ND6_SLOWTIMER_INTERVAL) <= 0) { /* * Since reachable time rarely changes by router * advertisements, we SHOULD insure that a new random * value gets recomputed at least once every few hours. * (RFC 2461, 6.3.4) */ nd6if->recalctm = V_nd6_recalc_reachtm_interval; nd6if->reachable = ND_COMPUTE_RTIME(nd6if->basereachable); } } IFNET_RUNLOCK_NOSLEEP(); CURVNET_RESTORE(); } void nd6_grab_holdchain(struct llentry *ln, struct mbuf **chain, struct sockaddr_in6 *sin6) { LLE_WLOCK_ASSERT(ln); *chain = ln->la_hold; ln->la_hold = NULL; memcpy(sin6, L3_ADDR_SIN6(ln), sizeof(*sin6)); if (ln->ln_state == ND6_LLINFO_STALE) { /* * The first time we send a packet to a * neighbor whose entry is STALE, we have * to change the state to DELAY and a sets * a timer to expire in DELAY_FIRST_PROBE_TIME * seconds to ensure do neighbor unreachability * detection on expiration. * (RFC 2461 7.3.3) */ ln->la_asked = 0; ln->ln_state = ND6_LLINFO_DELAY; nd6_llinfo_settimer_locked(ln, (long)V_nd6_delay * hz); } } static int nd6_output_ifp(struct ifnet *ifp, struct ifnet *origifp, struct mbuf *m, struct sockaddr_in6 *dst) { int error; int ip6len; struct ip6_hdr *ip6; struct m_tag *mtag; #ifdef MAC mac_netinet6_nd6_send(ifp, m); #endif /* * If called from nd6_ns_output() (NS), nd6_na_output() (NA), * icmp6_redirect_output() (REDIRECT) or from rip6_output() (RS, RA * as handled by rtsol and rtadvd), mbufs will be tagged for SeND * to be diverted to user space. When re-injected into the kernel, * send_output() will directly dispatch them to the outgoing interface. */ if (send_sendso_input_hook != NULL) { mtag = m_tag_find(m, PACKET_TAG_ND_OUTGOING, NULL); if (mtag != NULL) { ip6 = mtod(m, struct ip6_hdr *); ip6len = sizeof(struct ip6_hdr) + ntohs(ip6->ip6_plen); /* Use the SEND socket */ error = send_sendso_input_hook(m, ifp, SND_OUT, ip6len); /* -1 == no app on SEND socket */ if (error == 0 || error != -1) return (error); } } m_clrprotoflags(m); /* Avoid confusing lower layers. */ IP_PROBE(send, NULL, NULL, mtod(m, struct ip6_hdr *), ifp, NULL, mtod(m, struct ip6_hdr *)); if ((ifp->if_flags & IFF_LOOPBACK) == 0) origifp = ifp; error = (*ifp->if_output)(origifp, m, (struct sockaddr *)dst, NULL); return (error); } /* * IPv6 packet output - light version. * Checks if destination LLE exists and is in proper state * (e.g no modification required). If not true, fall back to * "heavy" version. */ int nd6_output(struct ifnet *ifp, struct ifnet *origifp, struct mbuf *m, struct sockaddr_in6 *dst, struct rtentry *rt0) { struct llentry *ln = NULL; /* discard the packet if IPv6 operation is disabled on the interface */ if ((ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED)) { m_freem(m); return (ENETDOWN); /* better error? */ } if (IN6_IS_ADDR_MULTICAST(&dst->sin6_addr)) goto sendpkt; if (nd6_need_cache(ifp) == 0) goto sendpkt; IF_AFDATA_RLOCK(ifp); ln = nd6_lookup(&dst->sin6_addr, 0, ifp); IF_AFDATA_RUNLOCK(ifp); /* * Perform fast path for the following cases: * 1) lle state is REACHABLE * 2) lle state is DELAY (NS message sentNS message sent) * * Every other case involves lle modification, so we handle * them separately. */ if (ln == NULL || (ln->ln_state != ND6_LLINFO_REACHABLE && ln->ln_state != ND6_LLINFO_DELAY)) { /* Fall back to slow processing path */ if (ln != NULL) LLE_RUNLOCK(ln); return (nd6_output_lle(ifp, origifp, m, dst)); } sendpkt: if (ln != NULL) LLE_RUNLOCK(ln); return (nd6_output_ifp(ifp, origifp, m, dst)); } /* * Output IPv6 packet - heavy version. * Function assume that either * 1) destination LLE does not exist, is invalid or stale, so * ND6_EXCLUSIVE lock needs to be acquired * 2) destination lle is provided (with ND6_EXCLUSIVE lock), * in that case packets are queued in &chain. * */ static int nd6_output_lle(struct ifnet *ifp, struct ifnet *origifp, struct mbuf *m, struct sockaddr_in6 *dst) { struct llentry *lle = NULL; int flags = 0; KASSERT(m != NULL, ("NULL mbuf, nothing to send")); /* discard the packet if IPv6 operation is disabled on the interface */ if ((ND_IFINFO(ifp)->flags & ND6_IFF_IFDISABLED)) { m_freem(m); return (ENETDOWN); /* better error? */ } if (IN6_IS_ADDR_MULTICAST(&dst->sin6_addr)) goto sendpkt; if (nd6_need_cache(ifp) == 0) goto sendpkt; /* * Address resolution or Neighbor Unreachability Detection * for the next hop. * At this point, the destination of the packet must be a unicast * or an anycast address(i.e. not a multicast). */ if (lle == NULL) { IF_AFDATA_RLOCK(ifp); lle = nd6_lookup(&dst->sin6_addr, ND6_EXCLUSIVE, ifp); IF_AFDATA_RUNLOCK(ifp); if ((lle == NULL) && nd6_is_addr_neighbor(dst, ifp)) { /* * Since nd6_is_addr_neighbor() internally calls nd6_lookup(), * the condition below is not very efficient. But we believe * it is tolerable, because this should be a rare case. */ flags = ND6_CREATE | ND6_EXCLUSIVE; IF_AFDATA_LOCK(ifp); lle = nd6_lookup(&dst->sin6_addr, flags, ifp); IF_AFDATA_UNLOCK(ifp); } } if (lle == NULL) { if ((ifp->if_flags & IFF_POINTOPOINT) == 0 && !(ND_IFINFO(ifp)->flags & ND6_IFF_PERFORMNUD)) { char ip6buf[INET6_ADDRSTRLEN]; log(LOG_DEBUG, "nd6_output: can't allocate llinfo for %s " "(ln=%p)\n", ip6_sprintf(ip6buf, &dst->sin6_addr), lle); m_freem(m); return (ENOBUFS); } goto sendpkt; /* send anyway */ } LLE_WLOCK_ASSERT(lle); /* We don't have to do link-layer address resolution on a p2p link. */ if ((ifp->if_flags & IFF_POINTOPOINT) != 0 && lle->ln_state < ND6_LLINFO_REACHABLE) { lle->ln_state = ND6_LLINFO_STALE; nd6_llinfo_settimer_locked(lle, (long)V_nd6_gctimer * hz); } /* * The first time we send a packet to a neighbor whose entry is * STALE, we have to change the state to DELAY and a sets a timer to * expire in DELAY_FIRST_PROBE_TIME seconds to ensure do * neighbor unreachability detection on expiration. * (RFC 2461 7.3.3) */ if (lle->ln_state == ND6_LLINFO_STALE) { lle->la_asked = 0; lle->ln_state = ND6_LLINFO_DELAY; nd6_llinfo_settimer_locked(lle, (long)V_nd6_delay * hz); } /* * If the neighbor cache entry has a state other than INCOMPLETE * (i.e. its link-layer address is already resolved), just * send the packet. */ if (lle->ln_state > ND6_LLINFO_INCOMPLETE) goto sendpkt; /* * There is a neighbor cache entry, but no ethernet address * response yet. Append this latest packet to the end of the * packet queue in the mbuf, unless the number of the packet * does not exceed nd6_maxqueuelen. When it exceeds nd6_maxqueuelen, * the oldest packet in the queue will be removed. */ if (lle->ln_state == ND6_LLINFO_NOSTATE) lle->ln_state = ND6_LLINFO_INCOMPLETE; if (lle->la_hold != NULL) { struct mbuf *m_hold; int i; i = 0; for (m_hold = lle->la_hold; m_hold; m_hold = m_hold->m_nextpkt){ i++; if (m_hold->m_nextpkt == NULL) { m_hold->m_nextpkt = m; break; } } while (i >= V_nd6_maxqueuelen) { m_hold = lle->la_hold; lle->la_hold = lle->la_hold->m_nextpkt; m_freem(m_hold); i--; } } else { lle->la_hold = m; } /* * If there has been no NS for the neighbor after entering the * INCOMPLETE state, send the first solicitation. */ if (!ND6_LLINFO_PERMANENT(lle) && lle->la_asked == 0) { lle->la_asked++; nd6_llinfo_settimer_locked(lle, (long)ND_IFINFO(ifp)->retrans * hz / 1000); LLE_WUNLOCK(lle); nd6_ns_output(ifp, NULL, &dst->sin6_addr, lle, NULL); } else { /* We did the lookup so we need to do the unlock here. */ LLE_WUNLOCK(lle); } return (0); sendpkt: if (lle != NULL) LLE_WUNLOCK(lle); return (nd6_output_ifp(ifp, origifp, m, dst)); } int nd6_flush_holdchain(struct ifnet *ifp, struct ifnet *origifp, struct mbuf *chain, struct sockaddr_in6 *dst) { struct mbuf *m, *m_head; struct ifnet *outifp; int error = 0; m_head = chain; if ((ifp->if_flags & IFF_LOOPBACK) != 0) outifp = origifp; else outifp = ifp; while (m_head) { m = m_head; m_head = m_head->m_nextpkt; error = nd6_output_ifp(ifp, origifp, m, dst); } /* * XXX * note that intermediate errors are blindly ignored - but this is * the same convention as used with nd6_output when called by * nd6_cache_lladdr */ return (error); } int nd6_need_cache(struct ifnet *ifp) { /* * XXX: we currently do not make neighbor cache on any interface * other than ARCnet, Ethernet, FDDI and GIF. * * RFC2893 says: * - unidirectional tunnels needs no ND */ switch (ifp->if_type) { case IFT_ARCNET: case IFT_ETHER: case IFT_FDDI: case IFT_IEEE1394: #ifdef IFT_L2VLAN case IFT_L2VLAN: #endif #ifdef IFT_IEEE80211 case IFT_IEEE80211: #endif case IFT_INFINIBAND: case IFT_BRIDGE: case IFT_PROPVIRTUAL: return (1); default: return (0); } } /* * Add pernament ND6 link-layer record for given * interface address. * * Very similar to IPv4 arp_ifinit(), but: * 1) IPv6 DAD is performed in different place * 2) It is called by IPv6 protocol stack in contrast to * arp_ifinit() which is typically called in SIOCSIFADDR * driver ioctl handler. * */ int nd6_add_ifa_lle(struct in6_ifaddr *ia) { struct ifnet *ifp; struct llentry *ln; ifp = ia->ia_ifa.ifa_ifp; if (nd6_need_cache(ifp) == 0) return (0); IF_AFDATA_LOCK(ifp); ia->ia_ifa.ifa_rtrequest = nd6_rtrequest; ln = lla_lookup(LLTABLE6(ifp), (LLE_CREATE | LLE_IFADDR | LLE_EXCLUSIVE), (struct sockaddr *)&ia->ia_addr); IF_AFDATA_UNLOCK(ifp); if (ln != NULL) { ln->la_expire = 0; /* for IPv6 this means permanent */ ln->ln_state = ND6_LLINFO_REACHABLE; LLE_WUNLOCK(ln); in6_newaddrmsg(ia, RTM_ADD); return (0); } return (ENOBUFS); } /* * Removes ALL lle records for interface address prefix. * XXXME: That's probably not we really want to do, we need * to remove address record only and keep other records * until we determine if given prefix is really going * to be removed. */ void nd6_rem_ifa_lle(struct in6_ifaddr *ia) { struct sockaddr_in6 mask, addr; struct ifnet *ifp; in6_newaddrmsg(ia, RTM_DELETE); ifp = ia->ia_ifa.ifa_ifp; memcpy(&addr, &ia->ia_addr, sizeof(ia->ia_addr)); memcpy(&mask, &ia->ia_prefixmask, sizeof(ia->ia_prefixmask)); lltable_prefix_free(AF_INET6, (struct sockaddr *)&addr, (struct sockaddr *)&mask, LLE_STATIC); } /* * the callers of this function need to be re-worked to drop * the lle lock, drop here for now */ int nd6_storelladdr(struct ifnet *ifp, struct mbuf *m, const struct sockaddr *dst, u_char *desten, uint32_t *pflags) { struct llentry *ln; if (pflags != NULL) *pflags = 0; IF_AFDATA_UNLOCK_ASSERT(ifp); if (m != NULL && m->m_flags & M_MCAST) { switch (ifp->if_type) { case IFT_ETHER: case IFT_FDDI: #ifdef IFT_L2VLAN case IFT_L2VLAN: #endif #ifdef IFT_IEEE80211 case IFT_IEEE80211: #endif case IFT_BRIDGE: case IFT_ISO88025: ETHER_MAP_IPV6_MULTICAST(&SIN6(dst)->sin6_addr, desten); return (0); default: m_freem(m); return (EAFNOSUPPORT); } } /* * the entry should have been created in nd6_store_lladdr */ IF_AFDATA_RLOCK(ifp); ln = lla_lookup(LLTABLE6(ifp), 0, dst); IF_AFDATA_RUNLOCK(ifp); if ((ln == NULL) || !(ln->la_flags & LLE_VALID)) { if (ln != NULL) LLE_RUNLOCK(ln); /* this could happen, if we could not allocate memory */ m_freem(m); return (1); } bcopy(&ln->ll_addr, desten, ifp->if_addrlen); if (pflags != NULL) *pflags = ln->la_flags; LLE_RUNLOCK(ln); /* * A *small* use after free race exists here */ return (0); } static void clear_llinfo_pqueue(struct llentry *ln) { struct mbuf *m_hold, *m_hold_next; for (m_hold = ln->la_hold; m_hold; m_hold = m_hold_next) { m_hold_next = m_hold->m_nextpkt; m_freem(m_hold); } ln->la_hold = NULL; return; } static int nd6_sysctl_drlist(SYSCTL_HANDLER_ARGS); static int nd6_sysctl_prlist(SYSCTL_HANDLER_ARGS); #ifdef SYSCTL_DECL SYSCTL_DECL(_net_inet6_icmp6); #endif SYSCTL_NODE(_net_inet6_icmp6, ICMPV6CTL_ND6_DRLIST, nd6_drlist, CTLFLAG_RD, nd6_sysctl_drlist, ""); SYSCTL_NODE(_net_inet6_icmp6, ICMPV6CTL_ND6_PRLIST, nd6_prlist, CTLFLAG_RD, nd6_sysctl_prlist, ""); SYSCTL_INT(_net_inet6_icmp6, ICMPV6CTL_ND6_MAXQLEN, nd6_maxqueuelen, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(nd6_maxqueuelen), 1, ""); SYSCTL_INT(_net_inet6_icmp6, OID_AUTO, nd6_gctimer, CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(nd6_gctimer), (60 * 60 * 24), ""); static int nd6_sysctl_drlist(SYSCTL_HANDLER_ARGS) { struct in6_defrouter d; struct nd_defrouter *dr; int error; if (req->newptr) return (EPERM); bzero(&d, sizeof(d)); d.rtaddr.sin6_family = AF_INET6; d.rtaddr.sin6_len = sizeof(d.rtaddr); /* * XXX locking */ TAILQ_FOREACH(dr, &V_nd_defrouter, dr_entry) { d.rtaddr.sin6_addr = dr->rtaddr; error = sa6_recoverscope(&d.rtaddr); if (error != 0) return (error); d.flags = dr->flags; d.rtlifetime = dr->rtlifetime; d.expire = dr->expire + (time_second - time_uptime); d.if_index = dr->ifp->if_index; error = SYSCTL_OUT(req, &d, sizeof(d)); if (error != 0) return (error); } return (0); } static int nd6_sysctl_prlist(SYSCTL_HANDLER_ARGS) { struct in6_prefix p; struct sockaddr_in6 s6; struct nd_prefix *pr; struct nd_pfxrouter *pfr; time_t maxexpire; int error; char ip6buf[INET6_ADDRSTRLEN]; if (req->newptr) return (EPERM); bzero(&p, sizeof(p)); p.origin = PR_ORIG_RA; bzero(&s6, sizeof(s6)); s6.sin6_family = AF_INET6; s6.sin6_len = sizeof(s6); /* * XXX locking */ LIST_FOREACH(pr, &V_nd_prefix, ndpr_entry) { p.prefix = pr->ndpr_prefix; if (sa6_recoverscope(&p.prefix)) { log(LOG_ERR, "scope error in prefix list (%s)\n", ip6_sprintf(ip6buf, &p.prefix.sin6_addr)); /* XXX: press on... */ } p.raflags = pr->ndpr_raf; p.prefixlen = pr->ndpr_plen; p.vltime = pr->ndpr_vltime; p.pltime = pr->ndpr_pltime; p.if_index = pr->ndpr_ifp->if_index; if (pr->ndpr_vltime == ND6_INFINITE_LIFETIME) p.expire = 0; else { /* XXX: we assume time_t is signed. */ maxexpire = (-1) & ~((time_t)1 << ((sizeof(maxexpire) * 8) - 1)); if (pr->ndpr_vltime < maxexpire - pr->ndpr_lastupdate) p.expire = pr->ndpr_lastupdate + pr->ndpr_vltime + (time_second - time_uptime); else p.expire = maxexpire; } p.refcnt = pr->ndpr_refcnt; p.flags = pr->ndpr_stateflags; p.advrtrs = 0; LIST_FOREACH(pfr, &pr->ndpr_advrtrs, pfr_entry) p.advrtrs++; error = SYSCTL_OUT(req, &p, sizeof(p)); if (error != 0) return (error); LIST_FOREACH(pfr, &pr->ndpr_advrtrs, pfr_entry) { s6.sin6_addr = pfr->router->rtaddr; if (sa6_recoverscope(&s6)) log(LOG_ERR, "scope error in prefix list (%s)\n", ip6_sprintf(ip6buf, &pfr->router->rtaddr)); error = SYSCTL_OUT(req, &s6, sizeof(s6)); if (error != 0) return (error); } } return (0); } Index: user/ngie/more-tests/sys/sys/kerneldump.h =================================================================== --- user/ngie/more-tests/sys/sys/kerneldump.h (revision 281503) +++ user/ngie/more-tests/sys/sys/kerneldump.h (revision 281504) @@ -1,129 +1,130 @@ /*- * Copyright (c) 2002 Poul-Henning Kamp * Copyright (c) 2002 Networks Associates Technology, Inc. * All rights reserved. * * This software was developed for the FreeBSD Project by Poul-Henning Kamp * and NAI Labs, the Security Research Division of Network Associates, Inc. * under DARPA/SPAWAR contract N66001-01-C-8035 ("CBOSS"), as part of the * DARPA CHATS research program. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The names of the authors may not be used to endorse or promote * products derived from this software without specific prior written * permission. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _SYS_KERNELDUMP_H #define _SYS_KERNELDUMP_H #include #if BYTE_ORDER == LITTLE_ENDIAN #define dtoh32(x) __bswap32(x) #define dtoh64(x) __bswap64(x) #define htod32(x) __bswap32(x) #define htod64(x) __bswap64(x) #elif BYTE_ORDER == BIG_ENDIAN #define dtoh32(x) (x) #define dtoh64(x) (x) #define htod32(x) (x) #define htod64(x) (x) #endif /* * All uintX_t fields are in dump byte order, which is the same as * network byte order. Use the macros defined above to read or * write the fields. */ struct kerneldumpheader { char magic[20]; #define KERNELDUMPMAGIC "FreeBSD Kernel Dump" #define TEXTDUMPMAGIC "FreeBSD Text Dump" #define KERNELDUMPMAGIC_CLEARED "Cleared Kernel Dump" char architecture[12]; uint32_t version; #define KERNELDUMPVERSION 1 uint32_t architectureversion; +#define KERNELDUMP_AARCH64_VERSION 1 #define KERNELDUMP_AMD64_VERSION 2 #define KERNELDUMP_ARM_VERSION 1 #define KERNELDUMP_I386_VERSION 2 #define KERNELDUMP_MIPS_VERSION 1 #define KERNELDUMP_POWERPC_VERSION 1 #define KERNELDUMP_SPARC64_VERSION 1 #define KERNELDUMP_TEXT_VERSION 1 uint64_t dumplength; /* excl headers */ uint64_t dumptime; uint32_t blocksize; char hostname[64]; char versionstring[192]; char panicstring[192]; uint32_t parity; }; /* * Parity calculation is endian insensitive. */ static __inline u_int32_t kerneldump_parity(struct kerneldumpheader *kdhp) { uint32_t *up, parity; u_int i; up = (uint32_t *)kdhp; parity = 0; for (i = 0; i < sizeof *kdhp; i += sizeof *up) parity ^= *up++; return (parity); } #ifdef _KERNEL struct dump_pa { vm_paddr_t pa_start; vm_paddr_t pa_size; }; void mkdumpheader(struct kerneldumpheader *kdh, char *magic, uint32_t archver, uint64_t dumplen, uint32_t blksz); int dumpsys_generic(struct dumperinfo *); void dumpsys_map_chunk(vm_paddr_t, size_t, void **); typedef int dumpsys_callback_t(struct dump_pa *, int, void *); int dumpsys_foreach_chunk(dumpsys_callback_t, void *); int dumpsys_cb_dumpdata(struct dump_pa *, int, void *); int dumpsys_buf_write(struct dumperinfo *, char *, size_t); int dumpsys_buf_flush(struct dumperinfo *); void dumpsys_gen_pa_init(void); struct dump_pa *dumpsys_gen_pa_next(struct dump_pa *); void dumpsys_gen_wbinv_all(void); void dumpsys_gen_unmap_chunk(vm_paddr_t, size_t, void *); int dumpsys_gen_write_aux_headers(struct dumperinfo *); extern int do_minidump; #endif #endif /* _SYS_KERNELDUMP_H */ Index: user/ngie/more-tests/sys/x86/acpica/acpi_wakeup.c =================================================================== --- user/ngie/more-tests/sys/x86/acpica/acpi_wakeup.c (revision 281503) +++ user/ngie/more-tests/sys/x86/acpica/acpi_wakeup.c (revision 281504) @@ -1,419 +1,419 @@ /*- * Copyright (c) 2001 Takanori Watanabe * Copyright (c) 2001-2012 Mitsuru IWASAKI * Copyright (c) 2003 Peter Wemm * Copyright (c) 2008-2012 Jung-uk Kim * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #if defined(__amd64__) #define DEV_APIC #else #include "opt_apic.h" #endif #ifdef __i386__ #include "opt_npx.h" #endif #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef DEV_APIC #include #include #endif #ifdef SMP #include #include #endif #include #include #include "acpi_wakecode.h" #include "acpi_wakedata.h" /* Make sure the code is less than a page and leave room for the stack. */ CTASSERT(sizeof(wakecode) < PAGE_SIZE - 1024); extern int acpi_resume_beep; extern int acpi_reset_video; #ifdef SMP extern struct susppcb **susppcbs; static cpuset_t suspcpus; #else static struct susppcb **susppcbs; #endif static void *acpi_alloc_wakeup_handler(void); static void acpi_stop_beep(void *); #ifdef SMP static int acpi_wakeup_ap(struct acpi_softc *, int); static void acpi_wakeup_cpus(struct acpi_softc *); #endif #ifdef __amd64__ #define ACPI_PAGETABLES 3 #else #define ACPI_PAGETABLES 0 #endif #define WAKECODE_VADDR(sc) \ ((sc)->acpi_wakeaddr + (ACPI_PAGETABLES * PAGE_SIZE)) #define WAKECODE_PADDR(sc) \ ((sc)->acpi_wakephys + (ACPI_PAGETABLES * PAGE_SIZE)) #define WAKECODE_FIXUP(offset, type, val) do { \ type *addr; \ addr = (type *)(WAKECODE_VADDR(sc) + offset); \ *addr = val; \ } while (0) static void acpi_stop_beep(void *arg) { if (acpi_resume_beep != 0) timer_spkr_release(); } #ifdef SMP static int acpi_wakeup_ap(struct acpi_softc *sc, int cpu) { struct pcb *pcb; int vector = (WAKECODE_PADDR(sc) >> 12) & 0xff; int apic_id = cpu_apic_ids[cpu]; int ms; pcb = &susppcbs[cpu]->sp_pcb; WAKECODE_FIXUP(wakeup_pcb, struct pcb *, pcb); WAKECODE_FIXUP(wakeup_gdt, uint16_t, pcb->pcb_gdt.rd_limit); WAKECODE_FIXUP(wakeup_gdt + 2, uint64_t, pcb->pcb_gdt.rd_base); ipi_startup(apic_id, vector); /* Wait up to 5 seconds for it to resume. */ for (ms = 0; ms < 5000; ms++) { if (!CPU_ISSET(cpu, &suspended_cpus)) return (1); /* return SUCCESS */ DELAY(1000); } return (0); /* return FAILURE */ } #define WARMBOOT_TARGET 0 #define WARMBOOT_OFF (KERNBASE + 0x0467) #define WARMBOOT_SEG (KERNBASE + 0x0469) #define CMOS_REG (0x70) #define CMOS_DATA (0x71) #define BIOS_RESET (0x0f) #define BIOS_WARM (0x0a) static void acpi_wakeup_cpus(struct acpi_softc *sc) { uint32_t mpbioswarmvec; int cpu; u_char mpbiosreason; /* save the current value of the warm-start vector */ mpbioswarmvec = *((uint32_t *)WARMBOOT_OFF); outb(CMOS_REG, BIOS_RESET); mpbiosreason = inb(CMOS_DATA); /* setup a vector to our boot code */ *((volatile u_short *)WARMBOOT_OFF) = WARMBOOT_TARGET; *((volatile u_short *)WARMBOOT_SEG) = WAKECODE_PADDR(sc) >> 4; outb(CMOS_REG, BIOS_RESET); outb(CMOS_DATA, BIOS_WARM); /* 'warm-start' */ /* Wake up each AP. */ for (cpu = 1; cpu < mp_ncpus; cpu++) { if (!CPU_ISSET(cpu, &suspcpus)) continue; if (acpi_wakeup_ap(sc, cpu) == 0) { /* restore the warmstart vector */ *(uint32_t *)WARMBOOT_OFF = mpbioswarmvec; panic("acpi_wakeup: failed to resume AP #%d (PHY #%d)", cpu, cpu_apic_ids[cpu]); } } /* restore the warmstart vector */ *(uint32_t *)WARMBOOT_OFF = mpbioswarmvec; outb(CMOS_REG, BIOS_RESET); outb(CMOS_DATA, mpbiosreason); } #endif int acpi_sleep_machdep(struct acpi_softc *sc, int state) { ACPI_STATUS status; struct pcb *pcb; if (sc->acpi_wakeaddr == 0ul) return (-1); /* couldn't alloc wake memory */ #ifdef SMP suspcpus = all_cpus; CPU_CLR(PCPU_GET(cpuid), &suspcpus); #endif if (acpi_resume_beep != 0) timer_spkr_acquire(); AcpiSetFirmwareWakingVector(WAKECODE_PADDR(sc)); intr_suspend(); pcb = &susppcbs[0]->sp_pcb; if (savectx(pcb)) { #ifdef __amd64__ fpususpend(susppcbs[0]->sp_fpususpend); #elif defined(DEV_NPX) npxsuspend(susppcbs[0]->sp_fpususpend); #endif #ifdef SMP if (!CPU_EMPTY(&suspcpus) && suspend_cpus(suspcpus) == 0) { device_printf(sc->acpi_dev, "Failed to suspend APs\n"); return (0); /* couldn't sleep */ } #endif WAKECODE_FIXUP(resume_beep, uint8_t, (acpi_resume_beep != 0)); WAKECODE_FIXUP(reset_video, uint8_t, (acpi_reset_video != 0)); #ifndef __amd64__ WAKECODE_FIXUP(wakeup_cr4, register_t, pcb->pcb_cr4); #endif WAKECODE_FIXUP(wakeup_pcb, struct pcb *, pcb); WAKECODE_FIXUP(wakeup_gdt, uint16_t, pcb->pcb_gdt.rd_limit); WAKECODE_FIXUP(wakeup_gdt + 2, uint64_t, pcb->pcb_gdt.rd_base); /* Call ACPICA to enter the desired sleep state */ if (state == ACPI_STATE_S4 && sc->acpi_s4bios) status = AcpiEnterSleepStateS4bios(); else status = AcpiEnterSleepState(state); if (ACPI_FAILURE(status)) { device_printf(sc->acpi_dev, "AcpiEnterSleepState failed - %s\n", AcpiFormatException(status)); return (0); /* couldn't sleep */ } for (;;) ia32_pause(); } else { #ifdef __amd64__ fpuresume(susppcbs[0]->sp_fpususpend); #elif defined(DEV_NPX) npxresume(susppcbs[0]->sp_fpususpend); #endif } return (1); /* wakeup successfully */ } int acpi_wakeup_machdep(struct acpi_softc *sc, int state, int sleep_result, int intr_enabled) { if (sleep_result == -1) return (sleep_result); if (!intr_enabled) { /* Wakeup MD procedures in interrupt disabled context */ if (sleep_result == 1) { pmap_init_pat(); initializecpu(); PCPU_SET(switchtime, 0); PCPU_SET(switchticks, ticks); #ifdef DEV_APIC lapic_xapic_mode(); #endif #ifdef SMP if (!CPU_EMPTY(&suspcpus)) acpi_wakeup_cpus(sc); #endif } #ifdef SMP if (!CPU_EMPTY(&suspcpus)) restart_cpus(suspcpus); #endif mca_resume(); #ifdef __amd64__ if (vmm_resume_p != NULL) vmm_resume_p(); #endif intr_resume(/*suspend_cancelled*/false); AcpiSetFirmwareWakingVector(0); } else { /* Wakeup MD procedures in interrupt enabled context */ if (sleep_result == 1 && mem_range_softc.mr_op != NULL && mem_range_softc.mr_op->reinit != NULL) mem_range_softc.mr_op->reinit(&mem_range_softc); } return (sleep_result); } static void * acpi_alloc_wakeup_handler(void) { void *wakeaddr; int i; /* * Specify the region for our wakeup code. We want it in the low 1 MB * region, excluding real mode IVT (0-0x3ff), BDA (0x400-0x4ff), EBDA * (less than 128KB, below 0xa0000, must be excluded by SMAP and DSDT), * and ROM area (0xa0000 and above). The temporary page tables must be * page-aligned. */ wakeaddr = contigmalloc((ACPI_PAGETABLES + 1) * PAGE_SIZE, M_DEVBUF, M_WAITOK, 0x500, 0xa0000, PAGE_SIZE, 0ul); if (wakeaddr == NULL) { printf("%s: can't alloc wake memory\n", __func__); return (NULL); } if (EVENTHANDLER_REGISTER(power_resume, acpi_stop_beep, NULL, EVENTHANDLER_PRI_LAST) == NULL) { printf("%s: can't register event handler\n", __func__); contigfree(wakeaddr, (ACPI_PAGETABLES + 1) * PAGE_SIZE, M_DEVBUF); return (NULL); } susppcbs = malloc(mp_ncpus * sizeof(*susppcbs), M_DEVBUF, M_WAITOK); for (i = 0; i < mp_ncpus; i++) { susppcbs[i] = malloc(sizeof(**susppcbs), M_DEVBUF, M_WAITOK); susppcbs[i]->sp_fpususpend = alloc_fpusave(M_WAITOK); } return (wakeaddr); } void acpi_install_wakeup_handler(struct acpi_softc *sc) { static void *wakeaddr = NULL; #ifdef __amd64__ uint64_t *pt4, *pt3, *pt2; int i; #endif if (wakeaddr != NULL) return; wakeaddr = acpi_alloc_wakeup_handler(); if (wakeaddr == NULL) return; sc->acpi_wakeaddr = (vm_offset_t)wakeaddr; sc->acpi_wakephys = vtophys(wakeaddr); bcopy(wakecode, (void *)WAKECODE_VADDR(sc), sizeof(wakecode)); /* Patch GDT base address, ljmp targets. */ WAKECODE_FIXUP((bootgdtdesc + 2), uint32_t, WAKECODE_PADDR(sc) + bootgdt); WAKECODE_FIXUP((wakeup_sw32 + 2), uint32_t, WAKECODE_PADDR(sc) + wakeup_32); #ifdef __amd64__ WAKECODE_FIXUP((wakeup_sw64 + 1), uint32_t, WAKECODE_PADDR(sc) + wakeup_64); WAKECODE_FIXUP(wakeup_pagetables, uint32_t, sc->acpi_wakephys); #endif /* Save pointers to some global data. */ WAKECODE_FIXUP(wakeup_ret, void *, resumectx); #ifndef __amd64__ -#ifdef PAE +#if defined(PAE) || defined(PAE_TABLES) WAKECODE_FIXUP(wakeup_cr3, register_t, vtophys(kernel_pmap->pm_pdpt)); #else WAKECODE_FIXUP(wakeup_cr3, register_t, vtophys(kernel_pmap->pm_pdir)); #endif #else /* Build temporary page tables below realmode code. */ pt4 = wakeaddr; pt3 = pt4 + (PAGE_SIZE) / sizeof(uint64_t); pt2 = pt3 + (PAGE_SIZE) / sizeof(uint64_t); /* Create the initial 1GB replicated page tables */ for (i = 0; i < 512; i++) { /* * Each slot of the level 4 pages points * to the same level 3 page */ pt4[i] = (uint64_t)(sc->acpi_wakephys + PAGE_SIZE); pt4[i] |= PG_V | PG_RW | PG_U; /* * Each slot of the level 3 pages points * to the same level 2 page */ pt3[i] = (uint64_t)(sc->acpi_wakephys + (2 * PAGE_SIZE)); pt3[i] |= PG_V | PG_RW | PG_U; /* The level 2 page slots are mapped with 2MB pages for 1GB. */ pt2[i] = i * (2 * 1024 * 1024); pt2[i] |= PG_V | PG_RW | PG_PS | PG_U; } #endif if (bootverbose) device_printf(sc->acpi_dev, "wakeup code va %#jx pa %#jx\n", (uintmax_t)sc->acpi_wakeaddr, (uintmax_t)sc->acpi_wakephys); } Index: user/ngie/more-tests/sys =================================================================== --- user/ngie/more-tests/sys (revision 281503) +++ user/ngie/more-tests/sys (revision 281504) Property changes on: user/ngie/more-tests/sys ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head/sys:r281477-281503 Index: user/ngie/more-tests/usr.bin/gzip/gzip.1 =================================================================== --- user/ngie/more-tests/usr.bin/gzip/gzip.1 (revision 281503) +++ user/ngie/more-tests/usr.bin/gzip/gzip.1 (revision 281504) @@ -1,227 +1,228 @@ -.\" $NetBSD: gzip.1,v 1.23 2014/03/18 18:20:45 riastradh Exp $ +.\" $NetBSD: gzip.1,v 1.25 2015/04/06 21:41:17 wiz Exp $ .\" .\" Copyright (c) 1997, 2003, 2004 Matthew R. Green .\" All rights reserved. .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR .\" IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES .\" OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. .\" IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, .\" INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, .\" BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; .\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED .\" AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, .\" OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ -.Dd October 9, 2011 +.Dd April 6, 2015 .Dt GZIP 1 .Os .Sh NAME .Nm gzip .Nd compression/decompression tool using Lempel-Ziv coding (LZ77) .Sh SYNOPSIS .Nm .Op Fl cdfhkLlNnqrtVv .Op Fl S Ar suffix .Ar file .Oo .Ar file Oo ... .Oc .Oc .Nm gunzip .Op Fl cfhkLNqrtVv .Op Fl S Ar suffix .Ar file .Oo .Ar file Oo ... .Oc .Oc .Nm zcat .Op Fl fhV .Ar file .Oo .Ar file Oo ... .Oc .Oc .Sh DESCRIPTION The .Nm program compresses and decompresses files using Lempel-Ziv coding (LZ77). If no .Ar files are specified, .Nm will compress from standard input, or decompress to standard output. When in compression mode, each .Ar file will be replaced with another file with the suffix, set by the .Fl S Ar suffix option, added, if possible. .Pp In decompression mode, each .Ar file will be checked for existence, as will the file with the suffix added. Each .Ar file argument must contain a separate complete archive; when multiple .Ar files are indicated, each is decompressed in turn. .Pp In the case of .Nm gzcat the resulting data is then concatenated in the manner of .Xr cat 1 . .Pp If invoked as .Nm gunzip then the .Fl d option is enabled. If invoked as .Nm zcat or .Nm gzcat then both the .Fl c and .Fl d options are enabled. .Pp This version of .Nm is also capable of decompressing files compressed using -.Xr compress 1 +.Xr compress 1 , +.Xr bzip2 1 , or -.Xr bzip2 1 . +.Xr xz 1 . .Sh OPTIONS The following options are available: .Bl -tag -width XXrXXXrecursiveX .It Fl 1 , -fast .It Fl 2 , 3 , 4 , 5 , 6 , 7 , 8 .It Fl 9 , -best These options change the compression level used, with the .Fl 1 option being the fastest, with less compression, and the .Fl 9 option being the slowest, with optimal compression. The default compression level is 6. .It Fl c , -stdout , -to-stdout This option specifies that output will go to the standard output stream, leaving files intact. .It Fl d , -decompress , -uncompress This option selects decompression rather than compression. .It Fl f , -force This option turns on force mode. This allows files with multiple links, symbolic links to regular files, overwriting of pre-existing files, reading from or writing to a terminal, and when combined with the .Fl c option, allowing non-compressed data to pass through unchanged. .It Fl h , -help This option prints a usage summary and exits. .It Fl k , -keep Keep (don't delete) input files during compression or decompression. .It Fl L , -license This option prints .Nm license. .It Fl l , -list This option displays information about the file's compressed and uncompressed size, ratio, uncompressed name. With the .Fl v option, it also displays the compression method, CRC, date and time embedded in the file. .It Fl N , -name This option causes the stored filename in the input file to be used as the output file. .It Fl n , -no-name This option stops the filename and timestamp from being stored in the output file. .It Fl q , -quiet With this option, no warnings or errors are printed. .It Fl r , -recursive This option is used to .Nm the files in a directory tree individually, using the .Xr fts 3 library. .It Fl S Ar suffix , Fl -suffix Ar suffix This option changes the default suffix from .gz to .Ar suffix . .It Fl t , -test This option will test compressed files for integrity. .It Fl V , -version This option prints the version of the .Nm program. .It Fl v , -verbose This option turns on verbose mode, which prints the compression ratio for each file compressed. .El .Sh ENVIRONMENT If the environment variable .Ev GZIP is set, it is parsed as a white-space separated list of options handled before any options on the command line. Options on the command line will override anything in .Ev GZIP . .Sh SEE ALSO .Xr bzip2 1 , .Xr compress 1 , .Xr xz 1 , .Xr fts 3 , .Xr zlib 3 .Sh HISTORY The .Nm program was originally written by Jean-loup Gailly, licensed under the GNU Public Licence. Matthew R. Green wrote a simple front end for .Nx 1.3 distribution media, based on the freely re-distributable zlib library. It was enhanced to be mostly feature-compatible with the original GNU .Nm program for .Nx 2.0 . .Pp This implementation of .Nm was ported based on the .Nx .Nm , and first appeared in .Fx 7.0 . .Sh AUTHORS .An -nosplit This implementation of .Nm was written by .An Matthew R. Green Aq Mt mrg@eterna.com.au with unpack support written by .An Xin LI Aq Mt delphij@FreeBSD.org . .Sh BUGS According to RFC 1952, the recorded file size is stored in a 32-bit integer, therefore, it can not represent files larger than 4GB. This limitation also applies to .Fl l option of .Nm utility. Index: user/ngie/more-tests/usr.bin/gzip/gzip.c =================================================================== --- user/ngie/more-tests/usr.bin/gzip/gzip.c (revision 281503) +++ user/ngie/more-tests/usr.bin/gzip/gzip.c (revision 281504) @@ -1,2161 +1,2170 @@ -/* $NetBSD: gzip.c,v 1.106 2014/10/18 08:33:30 snj Exp $ */ +/* $NetBSD: gzip.c,v 1.107 2015/01/13 02:37:20 mrg Exp $ */ /*- * Copyright (c) 1997, 1998, 2003, 2004, 2006 Matthew R. Green * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * */ #include #ifndef lint __COPYRIGHT("@(#) Copyright (c) 1997, 1998, 2003, 2004, 2006\ Matthew R. Green. All rights reserved."); __FBSDID("$FreeBSD$"); #endif /* not lint */ /* * gzip.c -- GPL free gzip using zlib. * * RFC 1950 covers the zlib format * RFC 1951 covers the deflate format * RFC 1952 covers the gzip format * * TODO: * - use mmap where possible * - make bzip2/compress -v/-t/-l support work as well as possible */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include /* what type of file are we dealing with */ enum filetype { FT_GZIP, #ifndef NO_BZIP2_SUPPORT FT_BZIP2, #endif #ifndef NO_COMPRESS_SUPPORT FT_Z, #endif #ifndef NO_PACK_SUPPORT FT_PACK, #endif #ifndef NO_XZ_SUPPORT FT_XZ, #endif FT_LAST, FT_UNKNOWN }; #ifndef NO_BZIP2_SUPPORT #include #define BZ2_SUFFIX ".bz2" #define BZIP2_MAGIC "\102\132\150" #endif #ifndef NO_COMPRESS_SUPPORT #define Z_SUFFIX ".Z" #define Z_MAGIC "\037\235" #endif #ifndef NO_PACK_SUPPORT #define PACK_MAGIC "\037\036" #endif #ifndef NO_XZ_SUPPORT #include #define XZ_SUFFIX ".xz" #define XZ_MAGIC "\3757zXZ" #endif #define GZ_SUFFIX ".gz" #define BUFLEN (64 * 1024) #define GZIP_MAGIC0 0x1F #define GZIP_MAGIC1 0x8B #define GZIP_OMAGIC1 0x9E #define GZIP_TIMESTAMP (off_t)4 #define GZIP_ORIGNAME (off_t)10 #define HEAD_CRC 0x02 #define EXTRA_FIELD 0x04 #define ORIG_NAME 0x08 #define COMMENT 0x10 #define OS_CODE 3 /* Unix */ typedef struct { const char *zipped; int ziplen; const char *normal; /* for unzip - must not be longer than zipped */ } suffixes_t; static suffixes_t suffixes[] = { #define SUFFIX(Z, N) {Z, sizeof Z - 1, N} SUFFIX(GZ_SUFFIX, ""), /* Overwritten by -S .xxx */ #ifndef SMALL SUFFIX(GZ_SUFFIX, ""), SUFFIX(".z", ""), SUFFIX("-gz", ""), SUFFIX("-z", ""), SUFFIX("_z", ""), SUFFIX(".taz", ".tar"), SUFFIX(".tgz", ".tar"), #ifndef NO_BZIP2_SUPPORT SUFFIX(BZ2_SUFFIX, ""), SUFFIX(".tbz", ".tar"), SUFFIX(".tbz2", ".tar"), #endif #ifndef NO_COMPRESS_SUPPORT SUFFIX(Z_SUFFIX, ""), #endif #ifndef NO_XZ_SUPPORT SUFFIX(XZ_SUFFIX, ""), #endif SUFFIX(GZ_SUFFIX, ""), /* Overwritten by -S "" */ #endif /* SMALL */ #undef SUFFIX }; #define NUM_SUFFIXES (sizeof suffixes / sizeof suffixes[0]) #define SUFFIX_MAXLEN 30 -static const char gzip_version[] = "FreeBSD gzip 20141022"; +static const char gzip_version[] = "FreeBSD gzip 20150413"; #ifndef SMALL static const char gzip_copyright[] = \ " Copyright (c) 1997, 1998, 2003, 2004, 2006 Matthew R. Green\n" " All rights reserved.\n" "\n" " Redistribution and use in source and binary forms, with or without\n" " modification, are permitted provided that the following conditions\n" " are met:\n" " 1. Redistributions of source code must retain the above copyright\n" " notice, this list of conditions and the following disclaimer.\n" " 2. Redistributions in binary form must reproduce the above copyright\n" " notice, this list of conditions and the following disclaimer in the\n" " documentation and/or other materials provided with the distribution.\n" "\n" " THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR\n" " IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES\n" " OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.\n" " IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,\n" " INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,\n" " BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\n" " LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED\n" " AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\n" " OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY\n" " OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF\n" " SUCH DAMAGE."; #endif static int cflag; /* stdout mode */ static int dflag; /* decompress mode */ static int lflag; /* list mode */ static int numflag = 6; /* gzip -1..-9 value */ #ifndef SMALL static int fflag; /* force mode */ static int kflag; /* don't delete input files */ static int nflag; /* don't save name/timestamp */ static int Nflag; /* don't restore name/timestamp */ static int qflag; /* quiet mode */ static int rflag; /* recursive mode */ static int tflag; /* test */ static int vflag; /* verbose mode */ static const char *remove_file = NULL; /* file to be removed upon SIGINT */ #else #define qflag 0 #define tflag 0 #endif static int exit_value = 0; /* exit value */ static char *infile; /* name of file coming in */ static void maybe_err(const char *fmt, ...) __printflike(1, 2) __dead2; #if !defined(NO_BZIP2_SUPPORT) || !defined(NO_PACK_SUPPORT) || \ !defined(NO_XZ_SUPPORT) static void maybe_errx(const char *fmt, ...) __printflike(1, 2) __dead2; #endif static void maybe_warn(const char *fmt, ...) __printflike(1, 2); static void maybe_warnx(const char *fmt, ...) __printflike(1, 2); static enum filetype file_gettype(u_char *); #ifdef SMALL #define gz_compress(if, of, sz, fn, tm) gz_compress(if, of, sz) #endif static off_t gz_compress(int, int, off_t *, const char *, uint32_t); static off_t gz_uncompress(int, int, char *, size_t, off_t *, const char *); static off_t file_compress(char *, char *, size_t); static off_t file_uncompress(char *, char *, size_t); static void handle_pathname(char *); static void handle_file(char *, struct stat *); static void handle_stdin(void); static void handle_stdout(void); static void print_ratio(off_t, off_t, FILE *); static void print_list(int fd, off_t, const char *, time_t); static void usage(void) __dead2; static void display_version(void) __dead2; #ifndef SMALL static void display_license(void); static void sigint_handler(int); #endif static const suffixes_t *check_suffix(char *, int); static ssize_t read_retry(int, void *, size_t); #ifdef SMALL #define unlink_input(f, sb) unlink(f) #else static off_t cat_fd(unsigned char *, size_t, off_t *, int fd); static void prepend_gzip(char *, int *, char ***); static void handle_dir(char *); static void print_verbage(const char *, const char *, off_t, off_t); static void print_test(const char *, int); static void copymodes(int fd, const struct stat *, const char *file); static int check_outfile(const char *outfile); #endif #ifndef NO_BZIP2_SUPPORT static off_t unbzip2(int, int, char *, size_t, off_t *); #endif #ifndef NO_COMPRESS_SUPPORT static FILE *zdopen(int); static off_t zuncompress(FILE *, FILE *, char *, size_t, off_t *); #endif #ifndef NO_PACK_SUPPORT static off_t unpack(int, int, char *, size_t, off_t *); #endif #ifndef NO_XZ_SUPPORT static off_t unxz(int, int, char *, size_t, off_t *); #endif #ifdef SMALL #define getopt_long(a,b,c,d,e) getopt(a,b,c) #else static const struct option longopts[] = { { "stdout", no_argument, 0, 'c' }, { "to-stdout", no_argument, 0, 'c' }, { "decompress", no_argument, 0, 'd' }, { "uncompress", no_argument, 0, 'd' }, { "force", no_argument, 0, 'f' }, { "help", no_argument, 0, 'h' }, { "keep", no_argument, 0, 'k' }, { "list", no_argument, 0, 'l' }, { "no-name", no_argument, 0, 'n' }, { "name", no_argument, 0, 'N' }, { "quiet", no_argument, 0, 'q' }, { "recursive", no_argument, 0, 'r' }, { "suffix", required_argument, 0, 'S' }, { "test", no_argument, 0, 't' }, { "verbose", no_argument, 0, 'v' }, { "version", no_argument, 0, 'V' }, { "fast", no_argument, 0, '1' }, { "best", no_argument, 0, '9' }, { "ascii", no_argument, 0, 'a' }, { "license", no_argument, 0, 'L' }, { NULL, no_argument, 0, 0 }, }; #endif int main(int argc, char **argv) { const char *progname = getprogname(); #ifndef SMALL char *gzip; int len; #endif int ch; #ifndef SMALL if ((gzip = getenv("GZIP")) != NULL) prepend_gzip(gzip, &argc, &argv); signal(SIGINT, sigint_handler); #endif /* * XXX * handle being called `gunzip', `zcat' and `gzcat' */ if (strcmp(progname, "gunzip") == 0) dflag = 1; else if (strcmp(progname, "zcat") == 0 || strcmp(progname, "gzcat") == 0) dflag = cflag = 1; #ifdef SMALL #define OPT_LIST "123456789cdhlV" #else #define OPT_LIST "123456789acdfhklLNnqrS:tVv" #endif while ((ch = getopt_long(argc, argv, OPT_LIST, longopts, NULL)) != -1) { switch (ch) { case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9': numflag = ch - '0'; break; case 'c': cflag = 1; break; case 'd': dflag = 1; break; case 'l': lflag = 1; dflag = 1; break; case 'V': display_version(); /* NOTREACHED */ #ifndef SMALL case 'a': fprintf(stderr, "%s: option --ascii ignored on this system\n", progname); break; case 'f': fflag = 1; break; case 'k': kflag = 1; break; case 'L': display_license(); /* NOT REACHED */ case 'N': nflag = 0; Nflag = 1; break; case 'n': nflag = 1; Nflag = 0; break; case 'q': qflag = 1; break; case 'r': rflag = 1; break; case 'S': len = strlen(optarg); if (len != 0) { if (len > SUFFIX_MAXLEN) errx(1, "incorrect suffix: '%s': too long", optarg); suffixes[0].zipped = optarg; suffixes[0].ziplen = len; } else { suffixes[NUM_SUFFIXES - 1].zipped = ""; suffixes[NUM_SUFFIXES - 1].ziplen = 0; } break; case 't': cflag = 1; tflag = 1; dflag = 1; break; case 'v': vflag = 1; break; #endif default: usage(); /* NOTREACHED */ } } argv += optind; argc -= optind; if (argc == 0) { if (dflag) /* stdin mode */ handle_stdin(); else /* stdout mode */ handle_stdout(); } else { do { handle_pathname(argv[0]); } while (*++argv); } #ifndef SMALL if (qflag == 0 && lflag && argc > 1) print_list(-1, 0, "(totals)", 0); #endif exit(exit_value); } /* maybe print a warning */ void maybe_warn(const char *fmt, ...) { va_list ap; if (qflag == 0) { va_start(ap, fmt); vwarn(fmt, ap); va_end(ap); } if (exit_value == 0) exit_value = 1; } /* ... without an errno. */ void maybe_warnx(const char *fmt, ...) { va_list ap; if (qflag == 0) { va_start(ap, fmt); vwarnx(fmt, ap); va_end(ap); } if (exit_value == 0) exit_value = 1; } /* maybe print an error */ void maybe_err(const char *fmt, ...) { va_list ap; if (qflag == 0) { va_start(ap, fmt); vwarn(fmt, ap); va_end(ap); } exit(2); } #if !defined(NO_BZIP2_SUPPORT) || !defined(NO_PACK_SUPPORT) || \ !defined(NO_XZ_SUPPORT) /* ... without an errno. */ void maybe_errx(const char *fmt, ...) { va_list ap; if (qflag == 0) { va_start(ap, fmt); vwarnx(fmt, ap); va_end(ap); } exit(2); } #endif #ifndef SMALL /* split up $GZIP and prepend it to the argument list */ static void prepend_gzip(char *gzip, int *argc, char ***argv) { char *s, **nargv, **ac; int nenvarg = 0, i; /* scan how many arguments there are */ for (s = gzip;;) { while (*s == ' ' || *s == '\t') s++; if (*s == 0) goto count_done; nenvarg++; while (*s != ' ' && *s != '\t') if (*s++ == 0) goto count_done; } count_done: /* punt early */ if (nenvarg == 0) return; *argc += nenvarg; ac = *argv; nargv = (char **)malloc((*argc + 1) * sizeof(char *)); if (nargv == NULL) maybe_err("malloc"); /* stash this away */ *argv = nargv; /* copy the program name first */ i = 0; nargv[i++] = *(ac++); /* take a copy of $GZIP and add it to the array */ s = strdup(gzip); if (s == NULL) maybe_err("strdup"); for (;;) { /* Skip whitespaces. */ while (*s == ' ' || *s == '\t') s++; if (*s == 0) goto copy_done; nargv[i++] = s; /* Find the end of this argument. */ while (*s != ' ' && *s != '\t') if (*s++ == 0) /* Argument followed by NUL. */ goto copy_done; /* Terminate by overwriting ' ' or '\t' with NUL. */ *s++ = 0; } copy_done: /* copy the original arguments and a NULL */ while (*ac) nargv[i++] = *(ac++); nargv[i] = NULL; } #endif /* compress input to output. Return bytes read, -1 on error */ static off_t gz_compress(int in, int out, off_t *gsizep, const char *origname, uint32_t mtime) { z_stream z; char *outbufp, *inbufp; off_t in_tot = 0, out_tot = 0; ssize_t in_size; int i, error; uLong crc; #ifdef SMALL static char header[] = { GZIP_MAGIC0, GZIP_MAGIC1, Z_DEFLATED, 0, 0, 0, 0, 0, 0, OS_CODE }; #endif outbufp = malloc(BUFLEN); inbufp = malloc(BUFLEN); if (outbufp == NULL || inbufp == NULL) { maybe_err("malloc failed"); goto out; } memset(&z, 0, sizeof z); z.zalloc = Z_NULL; z.zfree = Z_NULL; z.opaque = 0; #ifdef SMALL memcpy(outbufp, header, sizeof header); i = sizeof header; #else if (nflag != 0) { mtime = 0; origname = ""; } i = snprintf(outbufp, BUFLEN, "%c%c%c%c%c%c%c%c%c%c%s", GZIP_MAGIC0, GZIP_MAGIC1, Z_DEFLATED, *origname ? ORIG_NAME : 0, mtime & 0xff, (mtime >> 8) & 0xff, (mtime >> 16) & 0xff, (mtime >> 24) & 0xff, numflag == 1 ? 4 : numflag == 9 ? 2 : 0, OS_CODE, origname); if (i >= BUFLEN) /* this need PATH_MAX > BUFLEN ... */ maybe_err("snprintf"); if (*origname) i++; #endif z.next_out = (unsigned char *)outbufp + i; z.avail_out = BUFLEN - i; error = deflateInit2(&z, numflag, Z_DEFLATED, (-MAX_WBITS), 8, Z_DEFAULT_STRATEGY); if (error != Z_OK) { maybe_warnx("deflateInit2 failed"); in_tot = -1; goto out; } crc = crc32(0L, Z_NULL, 0); for (;;) { if (z.avail_out == 0) { if (write(out, outbufp, BUFLEN) != BUFLEN) { maybe_warn("write"); out_tot = -1; goto out; } out_tot += BUFLEN; z.next_out = (unsigned char *)outbufp; z.avail_out = BUFLEN; } if (z.avail_in == 0) { in_size = read(in, inbufp, BUFLEN); if (in_size < 0) { maybe_warn("read"); in_tot = -1; goto out; } if (in_size == 0) break; crc = crc32(crc, (const Bytef *)inbufp, (unsigned)in_size); in_tot += in_size; z.next_in = (unsigned char *)inbufp; z.avail_in = in_size; } error = deflate(&z, Z_NO_FLUSH); if (error != Z_OK && error != Z_STREAM_END) { maybe_warnx("deflate failed"); in_tot = -1; goto out; } } /* clean up */ for (;;) { size_t len; ssize_t w; error = deflate(&z, Z_FINISH); if (error != Z_OK && error != Z_STREAM_END) { maybe_warnx("deflate failed"); in_tot = -1; goto out; } len = (char *)z.next_out - outbufp; w = write(out, outbufp, len); if (w == -1 || (size_t)w != len) { maybe_warn("write"); out_tot = -1; goto out; } out_tot += len; z.next_out = (unsigned char *)outbufp; z.avail_out = BUFLEN; if (error == Z_STREAM_END) break; } if (deflateEnd(&z) != Z_OK) { maybe_warnx("deflateEnd failed"); in_tot = -1; goto out; } i = snprintf(outbufp, BUFLEN, "%c%c%c%c%c%c%c%c", (int)crc & 0xff, (int)(crc >> 8) & 0xff, (int)(crc >> 16) & 0xff, (int)(crc >> 24) & 0xff, (int)in_tot & 0xff, (int)(in_tot >> 8) & 0xff, (int)(in_tot >> 16) & 0xff, (int)(in_tot >> 24) & 0xff); if (i != 8) maybe_err("snprintf"); if (write(out, outbufp, i) != i) { maybe_warn("write"); in_tot = -1; } else out_tot += i; out: if (inbufp != NULL) free(inbufp); if (outbufp != NULL) free(outbufp); if (gsizep) *gsizep = out_tot; return in_tot; } /* * uncompress input to output then close the input. return the * uncompressed size written, and put the compressed sized read * into `*gsizep'. */ static off_t gz_uncompress(int in, int out, char *pre, size_t prelen, off_t *gsizep, const char *filename) { z_stream z; char *outbufp, *inbufp; off_t out_tot = -1, in_tot = 0; uint32_t out_sub_tot = 0; enum { GZSTATE_MAGIC0, GZSTATE_MAGIC1, GZSTATE_METHOD, GZSTATE_FLAGS, GZSTATE_SKIPPING, GZSTATE_EXTRA, GZSTATE_EXTRA2, GZSTATE_EXTRA3, GZSTATE_ORIGNAME, GZSTATE_COMMENT, GZSTATE_HEAD_CRC1, GZSTATE_HEAD_CRC2, GZSTATE_INIT, GZSTATE_READ, GZSTATE_CRC, GZSTATE_LEN, } state = GZSTATE_MAGIC0; int flags = 0, skip_count = 0; int error = Z_STREAM_ERROR, done_reading = 0; uLong crc = 0; ssize_t wr; int needmore = 0; #define ADVANCE() { z.next_in++; z.avail_in--; } if ((outbufp = malloc(BUFLEN)) == NULL) { maybe_err("malloc failed"); goto out2; } if ((inbufp = malloc(BUFLEN)) == NULL) { maybe_err("malloc failed"); goto out1; } memset(&z, 0, sizeof z); z.avail_in = prelen; z.next_in = (unsigned char *)pre; z.avail_out = BUFLEN; z.next_out = (unsigned char *)outbufp; z.zalloc = NULL; z.zfree = NULL; z.opaque = 0; in_tot = prelen; out_tot = 0; for (;;) { if ((z.avail_in == 0 || needmore) && done_reading == 0) { ssize_t in_size; if (z.avail_in > 0) { memmove(inbufp, z.next_in, z.avail_in); } z.next_in = (unsigned char *)inbufp; in_size = read(in, z.next_in + z.avail_in, BUFLEN - z.avail_in); if (in_size == -1) { maybe_warn("failed to read stdin"); goto stop_and_fail; } else if (in_size == 0) { done_reading = 1; } z.avail_in += in_size; needmore = 0; in_tot += in_size; } if (z.avail_in == 0) { if (done_reading && state != GZSTATE_MAGIC0) { maybe_warnx("%s: unexpected end of file", filename); goto stop_and_fail; } goto stop; } switch (state) { case GZSTATE_MAGIC0: if (*z.next_in != GZIP_MAGIC0) { if (in_tot > 0) { maybe_warnx("%s: trailing garbage " "ignored", filename); goto stop; } maybe_warnx("input not gziped (MAGIC0)"); goto stop_and_fail; } ADVANCE(); state++; out_sub_tot = 0; crc = crc32(0L, Z_NULL, 0); break; case GZSTATE_MAGIC1: if (*z.next_in != GZIP_MAGIC1 && *z.next_in != GZIP_OMAGIC1) { maybe_warnx("input not gziped (MAGIC1)"); goto stop_and_fail; } ADVANCE(); state++; break; case GZSTATE_METHOD: if (*z.next_in != Z_DEFLATED) { maybe_warnx("unknown compression method"); goto stop_and_fail; } ADVANCE(); state++; break; case GZSTATE_FLAGS: flags = *z.next_in; ADVANCE(); skip_count = 6; state++; break; case GZSTATE_SKIPPING: if (skip_count > 0) { skip_count--; ADVANCE(); } else state++; break; case GZSTATE_EXTRA: if ((flags & EXTRA_FIELD) == 0) { state = GZSTATE_ORIGNAME; break; } skip_count = *z.next_in; ADVANCE(); state++; break; case GZSTATE_EXTRA2: skip_count |= ((*z.next_in) << 8); ADVANCE(); state++; break; case GZSTATE_EXTRA3: if (skip_count > 0) { skip_count--; ADVANCE(); } else state++; break; case GZSTATE_ORIGNAME: if ((flags & ORIG_NAME) == 0) { state++; break; } if (*z.next_in == 0) state++; ADVANCE(); break; case GZSTATE_COMMENT: if ((flags & COMMENT) == 0) { state++; break; } if (*z.next_in == 0) state++; ADVANCE(); break; case GZSTATE_HEAD_CRC1: if (flags & HEAD_CRC) skip_count = 2; else skip_count = 0; state++; break; case GZSTATE_HEAD_CRC2: if (skip_count > 0) { skip_count--; ADVANCE(); } else state++; break; case GZSTATE_INIT: if (inflateInit2(&z, -MAX_WBITS) != Z_OK) { maybe_warnx("failed to inflateInit"); goto stop_and_fail; } state++; break; case GZSTATE_READ: error = inflate(&z, Z_FINISH); switch (error) { /* Z_BUF_ERROR goes with Z_FINISH... */ case Z_BUF_ERROR: if (z.avail_out > 0 && !done_reading) continue; case Z_STREAM_END: case Z_OK: break; case Z_NEED_DICT: maybe_warnx("Z_NEED_DICT error"); goto stop_and_fail; case Z_DATA_ERROR: maybe_warnx("data stream error"); goto stop_and_fail; case Z_STREAM_ERROR: maybe_warnx("internal stream error"); goto stop_and_fail; case Z_MEM_ERROR: maybe_warnx("memory allocation error"); goto stop_and_fail; default: maybe_warn("unknown error from inflate(): %d", error); } wr = BUFLEN - z.avail_out; if (wr != 0) { crc = crc32(crc, (const Bytef *)outbufp, (unsigned)wr); if ( #ifndef SMALL /* don't write anything with -t */ tflag == 0 && #endif write(out, outbufp, wr) != wr) { maybe_warn("error writing to output"); goto stop_and_fail; } out_tot += wr; out_sub_tot += wr; } if (error == Z_STREAM_END) { inflateEnd(&z); state++; } z.next_out = (unsigned char *)outbufp; z.avail_out = BUFLEN; break; case GZSTATE_CRC: { uLong origcrc; if (z.avail_in < 4) { if (!done_reading) { needmore = 1; continue; } maybe_warnx("truncated input"); goto stop_and_fail; } origcrc = ((unsigned)z.next_in[0] & 0xff) | ((unsigned)z.next_in[1] & 0xff) << 8 | ((unsigned)z.next_in[2] & 0xff) << 16 | ((unsigned)z.next_in[3] & 0xff) << 24; if (origcrc != crc) { maybe_warnx("invalid compressed" " data--crc error"); goto stop_and_fail; } } z.avail_in -= 4; z.next_in += 4; if (!z.avail_in && done_reading) { goto stop; } state++; break; case GZSTATE_LEN: { uLong origlen; if (z.avail_in < 4) { if (!done_reading) { needmore = 1; continue; } maybe_warnx("truncated input"); goto stop_and_fail; } origlen = ((unsigned)z.next_in[0] & 0xff) | ((unsigned)z.next_in[1] & 0xff) << 8 | ((unsigned)z.next_in[2] & 0xff) << 16 | ((unsigned)z.next_in[3] & 0xff) << 24; if (origlen != out_sub_tot) { maybe_warnx("invalid compressed" " data--length error"); goto stop_and_fail; } } z.avail_in -= 4; z.next_in += 4; if (error < 0) { maybe_warnx("decompression error"); goto stop_and_fail; } state = GZSTATE_MAGIC0; break; } continue; stop_and_fail: out_tot = -1; stop: break; } if (state > GZSTATE_INIT) inflateEnd(&z); free(inbufp); out1: free(outbufp); out2: if (gsizep) *gsizep = in_tot; return (out_tot); } #ifndef SMALL /* * set the owner, mode, flags & utimes using the given file descriptor. * file is only used in possible warning messages. */ static void copymodes(int fd, const struct stat *sbp, const char *file) { struct timespec times[2]; struct stat sb; /* * If we have no info on the input, give this file some * default values and return.. */ if (sbp == NULL) { mode_t mask = umask(022); (void)fchmod(fd, DEFFILEMODE & ~mask); (void)umask(mask); return; } sb = *sbp; /* if the chown fails, remove set-id bits as-per compress(1) */ if (fchown(fd, sb.st_uid, sb.st_gid) < 0) { if (errno != EPERM) maybe_warn("couldn't fchown: %s", file); sb.st_mode &= ~(S_ISUID|S_ISGID); } /* we only allow set-id and the 9 normal permission bits */ sb.st_mode &= S_ISUID | S_ISGID | S_IRWXU | S_IRWXG | S_IRWXO; if (fchmod(fd, sb.st_mode) < 0) maybe_warn("couldn't fchmod: %s", file); times[0] = sb.st_atim; times[1] = sb.st_mtim; if (futimens(fd, times) < 0) maybe_warn("couldn't futimens: %s", file); /* only try flags if they exist already */ if (sb.st_flags != 0 && fchflags(fd, sb.st_flags) < 0) maybe_warn("couldn't fchflags: %s", file); } #endif /* what sort of file is this? */ static enum filetype file_gettype(u_char *buf) { if (buf[0] == GZIP_MAGIC0 && (buf[1] == GZIP_MAGIC1 || buf[1] == GZIP_OMAGIC1)) return FT_GZIP; else #ifndef NO_BZIP2_SUPPORT if (memcmp(buf, BZIP2_MAGIC, 3) == 0 && buf[3] >= '0' && buf[3] <= '9') return FT_BZIP2; else #endif #ifndef NO_COMPRESS_SUPPORT if (memcmp(buf, Z_MAGIC, 2) == 0) return FT_Z; else #endif #ifndef NO_PACK_SUPPORT if (memcmp(buf, PACK_MAGIC, 2) == 0) return FT_PACK; else #endif #ifndef NO_XZ_SUPPORT if (memcmp(buf, XZ_MAGIC, 4) == 0) /* XXX: We only have 4 bytes */ return FT_XZ; else #endif return FT_UNKNOWN; } #ifndef SMALL /* check the outfile is OK. */ static int check_outfile(const char *outfile) { struct stat sb; int ok = 1; if (lflag == 0 && stat(outfile, &sb) == 0) { if (fflag) unlink(outfile); else if (isatty(STDIN_FILENO)) { char ans[10] = { 'n', '\0' }; /* default */ fprintf(stderr, "%s already exists -- do you wish to " "overwrite (y or n)? " , outfile); (void)fgets(ans, sizeof(ans) - 1, stdin); if (ans[0] != 'y' && ans[0] != 'Y') { fprintf(stderr, "\tnot overwriting\n"); ok = 0; } else unlink(outfile); } else { maybe_warnx("%s already exists -- skipping", outfile); ok = 0; } } return ok; } static void unlink_input(const char *file, const struct stat *sb) { struct stat nsb; if (kflag) return; if (stat(file, &nsb) != 0) /* Must be gone already */ return; if (nsb.st_dev != sb->st_dev || nsb.st_ino != sb->st_ino) /* Definitely a different file */ return; unlink(file); } static void sigint_handler(int signo __unused) { if (remove_file != NULL) unlink(remove_file); _exit(2); } #endif static const suffixes_t * check_suffix(char *file, int xlate) { const suffixes_t *s; int len = strlen(file); char *sp; for (s = suffixes; s != suffixes + NUM_SUFFIXES; s++) { /* if it doesn't fit in "a.suf", don't bother */ if (s->ziplen >= len) continue; sp = file + len - s->ziplen; if (strcmp(s->zipped, sp) != 0) continue; if (xlate) strcpy(sp, s->normal); return s; } return NULL; } /* * compress the given file: create a corresponding .gz file and remove the * original. */ static off_t file_compress(char *file, char *outfile, size_t outsize) { int in; int out; off_t size, insize; #ifndef SMALL struct stat isb, osb; const suffixes_t *suff; #endif in = open(file, O_RDONLY); if (in == -1) { maybe_warn("can't open %s", file); return (-1); } #ifndef SMALL if (fstat(in, &isb) != 0) { maybe_warn("couldn't stat: %s", file); close(in); return (-1); } #endif if (cflag == 0) { #ifndef SMALL if (isb.st_nlink > 1 && fflag == 0) { maybe_warnx("%s has %d other link%s -- skipping", file, isb.st_nlink - 1, (isb.st_nlink - 1) == 1 ? "" : "s"); close(in); return (-1); } if (fflag == 0 && (suff = check_suffix(file, 0)) && suff->zipped[0] != 0) { maybe_warnx("%s already has %s suffix -- unchanged", file, suff->zipped); close(in); return (-1); } #endif /* Add (usually) .gz to filename */ if ((size_t)snprintf(outfile, outsize, "%s%s", file, suffixes[0].zipped) >= outsize) memcpy(outfile + outsize - suffixes[0].ziplen - 1, suffixes[0].zipped, suffixes[0].ziplen + 1); #ifndef SMALL if (check_outfile(outfile) == 0) { close(in); return (-1); } #endif } if (cflag == 0) { out = open(outfile, O_WRONLY | O_CREAT | O_EXCL, 0600); if (out == -1) { maybe_warn("could not create output: %s", outfile); fclose(stdin); return (-1); } #ifndef SMALL remove_file = outfile; #endif } else out = STDOUT_FILENO; insize = gz_compress(in, out, &size, basename(file), (uint32_t)isb.st_mtime); (void)close(in); /* * If there was an error, insize will be -1. * If we compressed to stdout, just return the size. * Otherwise stat the file and check it is the correct size. * We only blow away the file if we can stat the output and it * has the expected size. */ if (cflag != 0) return (insize == -1 ? -1 : size); #ifndef SMALL if (fstat(out, &osb) != 0) { maybe_warn("couldn't stat: %s", outfile); goto bad_outfile; } if (osb.st_size != size) { maybe_warnx("output file: %s wrong size (%ju != %ju), deleting", outfile, (uintmax_t)osb.st_size, (uintmax_t)size); goto bad_outfile; } copymodes(out, &isb, outfile); remove_file = NULL; #endif if (close(out) == -1) maybe_warn("couldn't close output"); /* output is good, ok to delete input */ unlink_input(file, &isb); return (size); #ifndef SMALL bad_outfile: if (close(out) == -1) maybe_warn("couldn't close output"); maybe_warnx("leaving original %s", file); unlink(outfile); return (size); #endif } /* uncompress the given file and remove the original */ static off_t file_uncompress(char *file, char *outfile, size_t outsize) { struct stat isb, osb; off_t size; ssize_t rbytes; unsigned char header1[4]; enum filetype method; int fd, ofd, zfd = -1; #ifndef SMALL ssize_t rv; time_t timestamp = 0; - unsigned char name[PATH_MAX + 1]; + char name[PATH_MAX + 1]; #endif /* gather the old name info */ fd = open(file, O_RDONLY); if (fd < 0) { maybe_warn("can't open %s", file); goto lose; } strlcpy(outfile, file, outsize); if (check_suffix(outfile, 1) == NULL && !(cflag || lflag)) { maybe_warnx("%s: unknown suffix -- ignored", file); goto lose; } rbytes = read(fd, header1, sizeof header1); if (rbytes != sizeof header1) { /* we don't want to fail here. */ #ifndef SMALL if (fflag) goto lose; #endif if (rbytes == -1) maybe_warn("can't read %s", file); else goto unexpected_EOF; goto lose; } method = file_gettype(header1); #ifndef SMALL if (fflag == 0 && method == FT_UNKNOWN) { maybe_warnx("%s: not in gzip format", file); goto lose; } #endif #ifndef SMALL if (method == FT_GZIP && Nflag) { unsigned char ts[4]; /* timestamp */ rv = pread(fd, ts, sizeof ts, GZIP_TIMESTAMP); if (rv >= 0 && rv < (ssize_t)(sizeof ts)) goto unexpected_EOF; if (rv == -1) { if (!fflag) maybe_warn("can't read %s", file); goto lose; } timestamp = ts[3] << 24 | ts[2] << 16 | ts[1] << 8 | ts[0]; if (header1[3] & ORIG_NAME) { rbytes = pread(fd, name, sizeof name, GZIP_ORIGNAME); if (rbytes < 0) { maybe_warn("can't read %s", file); goto lose; } if (name[0] != 0) { + char *dp, *nf; + + /* strip saved directory name */ + nf = strrchr(name, '/'); + if (nf == NULL) + nf = name; + else + nf++; + /* preserve original directory name */ - char *dp = strrchr(file, '/'); + dp = strrchr(file, '/'); if (dp == NULL) dp = file; else dp++; snprintf(outfile, outsize, "%.*s%.*s", (int) (dp - file), - file, (int) rbytes, name); + file, (int) rbytes, nf); } } } #endif lseek(fd, 0, SEEK_SET); if (cflag == 0 || lflag) { if (fstat(fd, &isb) != 0) goto lose; #ifndef SMALL if (isb.st_nlink > 1 && lflag == 0 && fflag == 0) { maybe_warnx("%s has %d other links -- skipping", file, isb.st_nlink - 1); goto lose; } if (nflag == 0 && timestamp) isb.st_mtime = timestamp; if (check_outfile(outfile) == 0) goto lose; #endif } if (cflag == 0 && lflag == 0) { zfd = open(outfile, O_WRONLY|O_CREAT|O_EXCL, 0600); if (zfd == STDOUT_FILENO) { /* We won't close STDOUT_FILENO later... */ zfd = dup(zfd); close(STDOUT_FILENO); } if (zfd == -1) { maybe_warn("can't open %s", outfile); goto lose; } #ifndef SMALL remove_file = outfile; #endif } else zfd = STDOUT_FILENO; switch (method) { #ifndef NO_BZIP2_SUPPORT case FT_BZIP2: /* XXX */ if (lflag) { maybe_warnx("no -l with bzip2 files"); goto lose; } size = unbzip2(fd, zfd, NULL, 0, NULL); break; #endif #ifndef NO_COMPRESS_SUPPORT case FT_Z: { FILE *in, *out; /* XXX */ if (lflag) { maybe_warnx("no -l with Lempel-Ziv files"); goto lose; } if ((in = zdopen(fd)) == NULL) { maybe_warn("zdopen for read: %s", file); goto lose; } out = fdopen(dup(zfd), "w"); if (out == NULL) { maybe_warn("fdopen for write: %s", outfile); fclose(in); goto lose; } size = zuncompress(in, out, NULL, 0, NULL); /* need to fclose() if ferror() is true... */ if (ferror(in) | fclose(in)) { maybe_warn("failed infile fclose"); unlink(outfile); (void)fclose(out); } if (fclose(out) != 0) { maybe_warn("failed outfile fclose"); unlink(outfile); goto lose; } break; } #endif #ifndef NO_PACK_SUPPORT case FT_PACK: if (lflag) { maybe_warnx("no -l with packed files"); goto lose; } size = unpack(fd, zfd, NULL, 0, NULL); break; #endif #ifndef NO_XZ_SUPPORT case FT_XZ: if (lflag) { maybe_warnx("no -l with xz files"); goto lose; } size = unxz(fd, zfd, NULL, 0, NULL); break; #endif #ifndef SMALL case FT_UNKNOWN: if (lflag) { maybe_warnx("no -l for unknown filetypes"); goto lose; } size = cat_fd(NULL, 0, NULL, fd); break; #endif default: if (lflag) { print_list(fd, isb.st_size, outfile, isb.st_mtime); close(fd); return -1; /* XXX */ } size = gz_uncompress(fd, zfd, NULL, 0, NULL, file); break; } if (close(fd) != 0) maybe_warn("couldn't close input"); if (zfd != STDOUT_FILENO && close(zfd) != 0) maybe_warn("couldn't close output"); if (size == -1) { if (cflag == 0) unlink(outfile); maybe_warnx("%s: uncompress failed", file); return -1; } /* if testing, or we uncompressed to stdout, this is all we need */ #ifndef SMALL if (tflag) return size; #endif /* if we are uncompressing to stdin, don't remove the file. */ if (cflag) return size; /* * if we create a file... */ /* * if we can't stat the file don't remove the file. */ ofd = open(outfile, O_RDWR, 0); if (ofd == -1) { maybe_warn("couldn't open (leaving original): %s", outfile); return -1; } if (fstat(ofd, &osb) != 0) { maybe_warn("couldn't stat (leaving original): %s", outfile); close(ofd); return -1; } if (osb.st_size != size) { maybe_warnx("stat gave different size: %ju != %ju (leaving original)", (uintmax_t)size, (uintmax_t)osb.st_size); close(ofd); unlink(outfile); return -1; } #ifndef SMALL copymodes(ofd, &isb, outfile); remove_file = NULL; #endif close(ofd); unlink_input(file, &isb); return size; unexpected_EOF: maybe_warnx("%s: unexpected end of file", file); lose: if (fd != -1) close(fd); if (zfd != -1 && zfd != STDOUT_FILENO) close(fd); return -1; } #ifndef SMALL static off_t cat_fd(unsigned char * prepend, size_t count, off_t *gsizep, int fd) { char buf[BUFLEN]; off_t in_tot; ssize_t w; in_tot = count; w = write(STDOUT_FILENO, prepend, count); if (w == -1 || (size_t)w != count) { maybe_warn("write to stdout"); return -1; } for (;;) { ssize_t rv; rv = read(fd, buf, sizeof buf); if (rv == 0) break; if (rv < 0) { maybe_warn("read from fd %d", fd); break; } if (write(STDOUT_FILENO, buf, rv) != rv) { maybe_warn("write to stdout"); break; } in_tot += rv; } if (gsizep) *gsizep = in_tot; return (in_tot); } #endif static void handle_stdin(void) { unsigned char header1[4]; off_t usize, gsize; enum filetype method; ssize_t bytes_read; #ifndef NO_COMPRESS_SUPPORT FILE *in; #endif #ifndef SMALL if (fflag == 0 && lflag == 0 && isatty(STDIN_FILENO)) { maybe_warnx("standard input is a terminal -- ignoring"); return; } #endif if (lflag) { struct stat isb; /* XXX could read the whole file, etc. */ if (fstat(STDIN_FILENO, &isb) < 0) { maybe_warn("fstat"); return; } print_list(STDIN_FILENO, isb.st_size, "stdout", isb.st_mtime); return; } bytes_read = read_retry(STDIN_FILENO, header1, sizeof header1); if (bytes_read == -1) { maybe_warn("can't read stdin"); return; } else if (bytes_read != sizeof(header1)) { maybe_warnx("(stdin): unexpected end of file"); return; } method = file_gettype(header1); switch (method) { default: #ifndef SMALL if (fflag == 0) { maybe_warnx("unknown compression format"); return; } usize = cat_fd(header1, sizeof header1, &gsize, STDIN_FILENO); break; #endif case FT_GZIP: usize = gz_uncompress(STDIN_FILENO, STDOUT_FILENO, (char *)header1, sizeof header1, &gsize, "(stdin)"); break; #ifndef NO_BZIP2_SUPPORT case FT_BZIP2: usize = unbzip2(STDIN_FILENO, STDOUT_FILENO, (char *)header1, sizeof header1, &gsize); break; #endif #ifndef NO_COMPRESS_SUPPORT case FT_Z: if ((in = zdopen(STDIN_FILENO)) == NULL) { maybe_warnx("zopen of stdin"); return; } usize = zuncompress(in, stdout, (char *)header1, sizeof header1, &gsize); fclose(in); break; #endif #ifndef NO_PACK_SUPPORT case FT_PACK: usize = unpack(STDIN_FILENO, STDOUT_FILENO, (char *)header1, sizeof header1, &gsize); break; #endif #ifndef NO_XZ_SUPPORT case FT_XZ: usize = unxz(STDIN_FILENO, STDOUT_FILENO, (char *)header1, sizeof header1, &gsize); break; #endif } #ifndef SMALL if (vflag && !tflag && usize != -1 && gsize != -1) print_verbage(NULL, NULL, usize, gsize); if (vflag && tflag) print_test("(stdin)", usize != -1); #endif } static void handle_stdout(void) { off_t gsize, usize; struct stat sb; time_t systime; uint32_t mtime; int ret; #ifndef SMALL if (fflag == 0 && isatty(STDOUT_FILENO)) { maybe_warnx("standard output is a terminal -- ignoring"); return; } #endif /* If stdin is a file use its mtime, otherwise use current time */ ret = fstat(STDIN_FILENO, &sb); #ifndef SMALL if (ret < 0) { maybe_warn("Can't stat stdin"); return; } #endif if (S_ISREG(sb.st_mode)) mtime = (uint32_t)sb.st_mtime; else { systime = time(NULL); #ifndef SMALL if (systime == -1) { maybe_warn("time"); return; } #endif mtime = (uint32_t)systime; } usize = gz_compress(STDIN_FILENO, STDOUT_FILENO, &gsize, "", mtime); #ifndef SMALL if (vflag && !tflag && usize != -1 && gsize != -1) print_verbage(NULL, NULL, usize, gsize); #endif } /* do what is asked for, for the path name */ static void handle_pathname(char *path) { char *opath = path, *s = NULL; ssize_t len; int slen; struct stat sb; /* check for stdout/stdin */ if (path[0] == '-' && path[1] == '\0') { if (dflag) handle_stdin(); else handle_stdout(); return; } retry: if (stat(path, &sb) != 0 || (fflag == 0 && cflag == 0 && lstat(path, &sb) != 0)) { /* lets try .gz if we're decompressing */ if (dflag && s == NULL && errno == ENOENT) { len = strlen(path); slen = suffixes[0].ziplen; s = malloc(len + slen + 1); if (s == NULL) maybe_err("malloc"); memcpy(s, path, len); memcpy(s + len, suffixes[0].zipped, slen + 1); path = s; goto retry; } maybe_warn("can't stat: %s", opath); goto out; } if (S_ISDIR(sb.st_mode)) { #ifndef SMALL if (rflag) handle_dir(path); else #endif maybe_warnx("%s is a directory", path); goto out; } if (S_ISREG(sb.st_mode)) handle_file(path, &sb); else maybe_warnx("%s is not a regular file", path); out: if (s) free(s); } /* compress/decompress a file */ static void handle_file(char *file, struct stat *sbp) { off_t usize, gsize; char outfile[PATH_MAX]; infile = file; if (dflag) { usize = file_uncompress(file, outfile, sizeof(outfile)); #ifndef SMALL if (vflag && tflag) print_test(file, usize != -1); #endif if (usize == -1) return; gsize = sbp->st_size; } else { gsize = file_compress(file, outfile, sizeof(outfile)); if (gsize == -1) return; usize = sbp->st_size; } #ifndef SMALL if (vflag && !tflag) print_verbage(file, (cflag) ? NULL : outfile, usize, gsize); #endif } #ifndef SMALL /* this is used with -r to recursively descend directories */ static void handle_dir(char *dir) { char *path_argv[2]; FTS *fts; FTSENT *entry; path_argv[0] = dir; path_argv[1] = 0; fts = fts_open(path_argv, FTS_PHYSICAL | FTS_NOCHDIR, NULL); if (fts == NULL) { warn("couldn't fts_open %s", dir); return; } while ((entry = fts_read(fts))) { switch(entry->fts_info) { case FTS_D: case FTS_DP: continue; case FTS_DNR: case FTS_ERR: case FTS_NS: maybe_warn("%s", entry->fts_path); continue; case FTS_F: handle_file(entry->fts_path, entry->fts_statp); } } (void)fts_close(fts); } #endif /* print a ratio - size reduction as a fraction of uncompressed size */ static void print_ratio(off_t in, off_t out, FILE *where) { int percent10; /* 10 * percent */ off_t diff; char buff[8]; int len; diff = in - out/2; if (diff <= 0) /* * Output is more than double size of input! print -99.9% * Quite possibly we've failed to get the original size. */ percent10 = -999; else { /* * We only need 12 bits of result from the final division, * so reduce the values until a 32bit division will suffice. */ while (in > 0x100000) { diff >>= 1; in >>= 1; } if (in != 0) percent10 = ((u_int)diff * 2000) / (u_int)in - 1000; else percent10 = 0; } len = snprintf(buff, sizeof buff, "%2.2d.", percent10); /* Move the '.' to before the last digit */ buff[len - 1] = buff[len - 2]; buff[len - 2] = '.'; fprintf(where, "%5s%%", buff); } #ifndef SMALL /* print compression statistics, and the new name (if there is one!) */ static void print_verbage(const char *file, const char *nfile, off_t usize, off_t gsize) { if (file) fprintf(stderr, "%s:%s ", file, strlen(file) < 7 ? "\t\t" : "\t"); print_ratio(usize, gsize, stderr); if (nfile) fprintf(stderr, " -- replaced with %s", nfile); fprintf(stderr, "\n"); fflush(stderr); } /* print test results */ static void print_test(const char *file, int ok) { if (exit_value == 0 && ok == 0) exit_value = 1; fprintf(stderr, "%s:%s %s\n", file, strlen(file) < 7 ? "\t\t" : "\t", ok ? "OK" : "NOT OK"); fflush(stderr); } #endif /* print a file's info ala --list */ /* eg: compressed uncompressed ratio uncompressed_name 354841 1679360 78.8% /usr/pkgsrc/distfiles/libglade-2.0.1.tar */ static void print_list(int fd, off_t out, const char *outfile, time_t ts) { static int first = 1; #ifndef SMALL static off_t in_tot, out_tot; uint32_t crc = 0; #endif off_t in = 0, rv; if (first) { #ifndef SMALL if (vflag) printf("method crc date time "); #endif if (qflag == 0) printf(" compressed uncompressed " "ratio uncompressed_name\n"); } first = 0; /* print totals? */ #ifndef SMALL if (fd == -1) { in = in_tot; out = out_tot; } else #endif { /* read the last 4 bytes - this is the uncompressed size */ rv = lseek(fd, (off_t)(-8), SEEK_END); if (rv != -1) { unsigned char buf[8]; uint32_t usize; rv = read(fd, (char *)buf, sizeof(buf)); if (rv == -1) maybe_warn("read of uncompressed size"); else if (rv != sizeof(buf)) maybe_warnx("read of uncompressed size"); else { usize = buf[4] | buf[5] << 8 | buf[6] << 16 | buf[7] << 24; in = (off_t)usize; #ifndef SMALL crc = buf[0] | buf[1] << 8 | buf[2] << 16 | buf[3] << 24; #endif } } } #ifndef SMALL if (vflag && fd == -1) printf(" "); else if (vflag) { char *date = ctime(&ts); /* skip the day, 1/100th second, and year */ date += 4; date[12] = 0; printf("%5s %08x %11s ", "defla"/*XXX*/, crc, date); } in_tot += in; out_tot += out; #else (void)&ts; /* XXX */ #endif printf("%12llu %12llu ", (unsigned long long)out, (unsigned long long)in); print_ratio(in, out, stdout); printf(" %s\n", outfile); } /* display the usage of NetBSD gzip */ static void usage(void) { fprintf(stderr, "%s\n", gzip_version); fprintf(stderr, #ifdef SMALL "usage: %s [-" OPT_LIST "] [ [ ...]]\n", #else "usage: %s [-123456789acdfhklLNnqrtVv] [-S .suffix] [ [ ...]]\n" " -1 --fast fastest (worst) compression\n" " -2 .. -8 set compression level\n" " -9 --best best (slowest) compression\n" " -c --stdout write to stdout, keep original files\n" " --to-stdout\n" " -d --decompress uncompress files\n" " --uncompress\n" " -f --force force overwriting & compress links\n" " -h --help display this help\n" " -k --keep don't delete input files during operation\n" " -l --list list compressed file contents\n" " -N --name save or restore original file name and time stamp\n" " -n --no-name don't save original file name or time stamp\n" " -q --quiet output no warnings\n" " -r --recursive recursively compress files in directories\n" " -S .suf use suffix .suf instead of .gz\n" " --suffix .suf\n" " -t --test test compressed file\n" " -V --version display program version\n" " -v --verbose print extra statistics\n", #endif getprogname()); exit(0); } #ifndef SMALL /* display the license information of FreeBSD gzip */ static void display_license(void) { - fprintf(stderr, "%s (based on NetBSD gzip 20141018)\n", gzip_version); + fprintf(stderr, "%s (based on NetBSD gzip 20150113)\n", gzip_version); fprintf(stderr, "%s\n", gzip_copyright); exit(0); } #endif /* display the version of NetBSD gzip */ static void display_version(void) { fprintf(stderr, "%s\n", gzip_version); exit(0); } #ifndef NO_BZIP2_SUPPORT #include "unbzip2.c" #endif #ifndef NO_COMPRESS_SUPPORT #include "zuncompress.c" #endif #ifndef NO_PACK_SUPPORT #include "unpack.c" #endif #ifndef NO_XZ_SUPPORT #include "unxz.c" #endif static ssize_t read_retry(int fd, void *buf, size_t sz) { char *cp = buf; size_t left = MIN(sz, (size_t) SSIZE_MAX); while (left > 0) { ssize_t ret; ret = read(fd, cp, left); if (ret == -1) { return ret; } else if (ret == 0) { break; /* EOF */ } cp += ret; left -= ret; } return sz - left; } Index: user/ngie/more-tests/usr.bin/ipcs/Makefile =================================================================== --- user/ngie/more-tests/usr.bin/ipcs/Makefile (revision 281503) +++ user/ngie/more-tests/usr.bin/ipcs/Makefile (revision 281504) @@ -1,9 +1,7 @@ # $FreeBSD$ PROG= ipcs SRCS= ipcs.c ipc.c LIBADD= kvm -WARNS?= 1 - .include Index: user/ngie/more-tests/usr.bin/ipcs/ipc.c =================================================================== --- user/ngie/more-tests/usr.bin/ipcs/ipc.c (revision 281503) +++ user/ngie/more-tests/usr.bin/ipcs/ipc.c (revision 281504) @@ -1,206 +1,206 @@ /* * Copyright (c) 1994 SigmaSoft, Th. Lockert * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * The split of ipcs.c into ipcs.c and ipc.c to accommodate the * changes in ipcrm.c was done by Edwin Groothuis */ #include __FBSDID("$FreeBSD$"); #include #include #define _KERNEL #include #include #include #undef _KERNEL #include #include #include #include #include #include #include "ipc.h" int use_sysctl = 1; struct semid_kernel *sema; struct seminfo seminfo; struct msginfo msginfo; struct msqid_kernel *msqids; struct shminfo shminfo; struct shmid_kernel *shmsegs; void kget(int idx, void *addr, size_t size); struct nlist symbols[] = { - {"sema"}, - {"seminfo"}, - {"msginfo"}, - {"msqids"}, - {"shminfo"}, - {"shmsegs"}, - {NULL} + { .n_name = "sema" }, + { .n_name = "seminfo" }, + { .n_name = "msginfo" }, + { .n_name = "msqids" }, + { .n_name = "shminfo" }, + { .n_name = "shmsegs" }, + { .n_name = NULL } }; #define SHMINFO_XVEC X(shmmax, sizeof(u_long)) \ X(shmmin, sizeof(u_long)) \ X(shmmni, sizeof(u_long)) \ X(shmseg, sizeof(u_long)) \ X(shmall, sizeof(u_long)) #define SEMINFO_XVEC X(semmni, sizeof(int)) \ X(semmns, sizeof(int)) \ X(semmnu, sizeof(int)) \ X(semmsl, sizeof(int)) \ X(semopm, sizeof(int)) \ X(semume, sizeof(int)) \ X(semusz, sizeof(int)) \ X(semvmx, sizeof(int)) \ X(semaem, sizeof(int)) #define MSGINFO_XVEC X(msgmax, sizeof(int)) \ X(msgmni, sizeof(int)) \ X(msgmnb, sizeof(int)) \ X(msgtql, sizeof(int)) \ X(msgssz, sizeof(int)) \ X(msgseg, sizeof(int)) #define X(a, b) { "kern.ipc." #a, offsetof(TYPEC, a), (b) }, #define TYPEC struct shminfo -struct scgs_vector shminfo_scgsv[] = { SHMINFO_XVEC { NULL } }; +static struct scgs_vector shminfo_scgsv[] = { SHMINFO_XVEC { .sysctl=NULL } }; #undef TYPEC #define TYPEC struct seminfo -struct scgs_vector seminfo_scgsv[] = { SEMINFO_XVEC { NULL } }; +static struct scgs_vector seminfo_scgsv[] = { SEMINFO_XVEC { .sysctl=NULL } }; #undef TYPEC #define TYPEC struct msginfo -struct scgs_vector msginfo_scgsv[] = { MSGINFO_XVEC { NULL } }; +static struct scgs_vector msginfo_scgsv[] = { MSGINFO_XVEC { .sysctl=NULL } }; #undef TYPEC #undef X kvm_t *kd; void sysctlgatherstruct(void *addr, size_t size, struct scgs_vector *vecarr) { struct scgs_vector *xp; size_t tsiz; int rv; for (xp = vecarr; xp->sysctl != NULL; xp++) { assert(xp->offset <= size); tsiz = xp->size; rv = sysctlbyname(xp->sysctl, (char *)addr + xp->offset, &tsiz, NULL, 0); if (rv == -1) err(1, "sysctlbyname: %s", xp->sysctl); if (tsiz != xp->size) errx(1, "%s size mismatch (expected %zu, got %zu)", xp->sysctl, xp->size, tsiz); } } void kget(int idx, void *addr, size_t size) { const char *symn; /* symbol name */ size_t tsiz; int rv; unsigned long kaddr; const char *sym2sysctl[] = { /* symbol to sysctl name table */ "kern.ipc.sema", "kern.ipc.seminfo", "kern.ipc.msginfo", "kern.ipc.msqids", "kern.ipc.shminfo", "kern.ipc.shmsegs" }; assert((unsigned)idx <= sizeof(sym2sysctl) / sizeof(*sym2sysctl)); if (!use_sysctl) { symn = symbols[idx].n_name; if (*symn == '_') symn++; if (symbols[idx].n_type == 0 || symbols[idx].n_value == 0) errx(1, "symbol %s undefined", symn); /* * For some symbols, the value we retrieve is * actually a pointer; since we want the actual value, * we have to manually dereference it. */ switch (idx) { case X_MSQIDS: tsiz = sizeof(msqids); rv = kvm_read(kd, symbols[idx].n_value, &msqids, tsiz); kaddr = (u_long)msqids; break; case X_SHMSEGS: tsiz = sizeof(shmsegs); rv = kvm_read(kd, symbols[idx].n_value, &shmsegs, tsiz); kaddr = (u_long)shmsegs; break; case X_SEMA: tsiz = sizeof(sema); rv = kvm_read(kd, symbols[idx].n_value, &sema, tsiz); kaddr = (u_long)sema; break; default: rv = tsiz = 0; kaddr = symbols[idx].n_value; break; } if ((unsigned)rv != tsiz) errx(1, "%s: %s", symn, kvm_geterr(kd)); if ((unsigned)kvm_read(kd, kaddr, addr, size) != size) errx(1, "%s: %s", symn, kvm_geterr(kd)); } else { switch (idx) { case X_SHMINFO: sysctlgatherstruct(addr, size, shminfo_scgsv); break; case X_SEMINFO: sysctlgatherstruct(addr, size, seminfo_scgsv); break; case X_MSGINFO: sysctlgatherstruct(addr, size, msginfo_scgsv); break; default: tsiz = size; rv = sysctlbyname(sym2sysctl[idx], addr, &tsiz, NULL, 0); if (rv == -1) err(1, "sysctlbyname: %s", sym2sysctl[idx]); if (tsiz != size) errx(1, "%s size mismatch " "(expected %zu, got %zu)", sym2sysctl[idx], size, tsiz); break; } } } Index: user/ngie/more-tests/usr.bin/ipcs/ipc.h =================================================================== --- user/ngie/more-tests/usr.bin/ipcs/ipc.h (revision 281503) +++ user/ngie/more-tests/usr.bin/ipcs/ipc.h (revision 281504) @@ -1,71 +1,71 @@ /* * Copyright (c) 1994 SigmaSoft, Th. Lockert * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * The split of ipcs.c into ipcs.c and ipc.c to accommodate the * changes in ipcrm.c was done by Edwin Groothuis * * $FreeBSD$ */ /* Part of struct nlist symbols[] */ #define X_SEMA 0 #define X_SEMINFO 1 #define X_MSGINFO 2 #define X_MSQIDS 3 #define X_SHMINFO 4 #define X_SHMSEGS 5 #define SHMINFO 1 #define SHMTOTAL 2 #define MSGINFO 4 #define MSGTOTAL 8 #define SEMINFO 16 #define SEMTOTAL 32 #define IPC_TO_STR(x) (x == 'Q' ? "msq" : (x == 'M' ? "shm" : "sem")) #define IPC_TO_STRING(x) (x == 'Q' ? "message queue" : \ (x == 'M' ? "shared memory segment" : "semaphore")) /* SysCtlGatherStruct structure. */ struct scgs_vector { const char *sysctl; - off_t offset; + size_t offset; size_t size; }; void kget(int idx, void *addr, size_t size); void sysctlgatherstruct(void *addr, size_t size, struct scgs_vector *vec); extern int use_sysctl; extern struct nlist symbols[]; extern kvm_t *kd; extern struct semid_kernel *sema; extern struct seminfo seminfo; extern struct msginfo msginfo; extern struct msqid_kernel *msqids; extern struct shminfo shminfo; extern struct shmid_kernel *shmsegs; Index: user/ngie/more-tests/usr.bin/ipcs/ipcs.c =================================================================== --- user/ngie/more-tests/usr.bin/ipcs/ipcs.c (revision 281503) +++ user/ngie/more-tests/usr.bin/ipcs/ipcs.c (revision 281504) @@ -1,569 +1,570 @@ /* * Copyright (c) 1994 SigmaSoft, Th. Lockert * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * 3. The name of the author may not be used to endorse or promote products * derived from this software without specific prior written permission. * * THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, * INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY * AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL * THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; * OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, * WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR * OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF * ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #define _KERNEL #include #include #include #undef _KERNEL #include #include #include #include #include #include #include #include #include #include #include "ipc.h" char *fmt_perm(u_short); void cvt_time(time_t, char *); void usage(void); uid_t user2uid(char *username); void print_kmsqtotal(struct msginfo msginfo); void print_kmsqheader(int option); void print_kmsqptr(int i, int option, struct msqid_kernel *kmsqptr); void print_kshmtotal(struct shminfo shminfo); void print_kshmheader(int option); void print_kshmptr(int i, int option, struct shmid_kernel *kshmptr); void print_ksemtotal(struct seminfo seminfo); void print_ksemheader(int option); void print_ksemptr(int i, int option, struct semid_kernel *ksemaptr); char * fmt_perm(u_short mode) { static char buffer[100]; buffer[0] = '-'; buffer[1] = '-'; buffer[2] = ((mode & 0400) ? 'r' : '-'); buffer[3] = ((mode & 0200) ? 'w' : '-'); buffer[4] = ((mode & 0100) ? 'a' : '-'); buffer[5] = ((mode & 0040) ? 'r' : '-'); buffer[6] = ((mode & 0020) ? 'w' : '-'); buffer[7] = ((mode & 0010) ? 'a' : '-'); buffer[8] = ((mode & 0004) ? 'r' : '-'); buffer[9] = ((mode & 0002) ? 'w' : '-'); buffer[10] = ((mode & 0001) ? 'a' : '-'); buffer[11] = '\0'; return (&buffer[0]); } void cvt_time(time_t t, char *buf) { struct tm *tm; if (t == 0) { strcpy(buf, "no-entry"); } else { tm = localtime(&t); sprintf(buf, "%2d:%02d:%02d", tm->tm_hour, tm->tm_min, tm->tm_sec); } } #define BIGGEST 1 #define CREATOR 2 #define OUTSTANDING 4 #define PID 8 #define TIME 16 int main(int argc, char *argv[]) { int display = SHMINFO | MSGINFO | SEMINFO; int option = 0; char *core = NULL, *user = NULL, *namelist = NULL; char kvmoferr[_POSIX2_LINE_MAX]; /* Error buf for kvm_openfiles. */ int i; + u_long shmidx; uid_t uid = 0; while ((i = getopt(argc, argv, "MmQqSsabC:cN:optTu:y")) != -1) switch (i) { case 'a': option |= BIGGEST | CREATOR | OUTSTANDING | PID | TIME; break; case 'b': option |= BIGGEST; break; case 'C': core = optarg; break; case 'c': option |= CREATOR; break; case 'M': display = SHMTOTAL; break; case 'm': display = SHMINFO; break; case 'N': namelist = optarg; break; case 'o': option |= OUTSTANDING; break; case 'p': option |= PID; break; case 'Q': display = MSGTOTAL; break; case 'q': display = MSGINFO; break; case 'S': display = SEMTOTAL; break; case 's': display = SEMINFO; break; case 'T': display = SHMTOTAL | MSGTOTAL | SEMTOTAL; break; case 't': option |= TIME; break; case 'u': user = optarg; uid = user2uid(user); break; case 'y': use_sysctl = 0; break; default: usage(); } /* * If paths to the exec file or core file were specified, we * aren't operating on the running kernel, so we can't use * sysctl. */ if (namelist != NULL || core != NULL) use_sysctl = 0; if (!use_sysctl) { kd = kvm_openfiles(namelist, core, NULL, O_RDONLY, kvmoferr); if (kd == NULL) errx(1, "kvm_openfiles: %s", kvmoferr); switch (kvm_nlist(kd, symbols)) { case 0: break; case -1: errx(1, "unable to read kernel symbol table"); default: break; } } kget(X_MSGINFO, &msginfo, sizeof(msginfo)); if ((display & (MSGINFO | MSGTOTAL))) { if (display & MSGTOTAL) print_kmsqtotal(msginfo); if (display & MSGINFO) { struct msqid_kernel *kxmsqids; size_t kxmsqids_len; kxmsqids_len = sizeof(struct msqid_kernel) * msginfo.msgmni; kxmsqids = malloc(kxmsqids_len); kget(X_MSQIDS, kxmsqids, kxmsqids_len); print_kmsqheader(option); for (i = 0; i < msginfo.msgmni; i += 1) { if (kxmsqids[i].u.msg_qbytes != 0) { if (user && uid != kxmsqids[i].u.msg_perm.uid) continue; print_kmsqptr(i, option, &kxmsqids[i]); } } printf("\n"); } } else if (display & (MSGINFO | MSGTOTAL)) { fprintf(stderr, "SVID messages facility " "not configured in the system\n"); } kget(X_SHMINFO, &shminfo, sizeof(shminfo)); if ((display & (SHMINFO | SHMTOTAL))) { if (display & SHMTOTAL) print_kshmtotal(shminfo); if (display & SHMINFO) { struct shmid_kernel *kxshmids; size_t kxshmids_len; kxshmids_len = sizeof(struct shmid_kernel) * shminfo.shmmni; kxshmids = malloc(kxshmids_len); kget(X_SHMSEGS, kxshmids, kxshmids_len); print_kshmheader(option); - for (i = 0; i < shminfo.shmmni; i += 1) { - if (kxshmids[i].u.shm_perm.mode & 0x0800) { + for (shmidx = 0; shmidx < shminfo.shmmni; shmidx += 1) { + if (kxshmids[shmidx].u.shm_perm.mode & 0x0800) { if (user && - uid != kxshmids[i].u.shm_perm.uid) + uid != kxshmids[shmidx].u.shm_perm.uid) continue; - print_kshmptr(i, option, &kxshmids[i]); + print_kshmptr(shmidx, option, &kxshmids[shmidx]); } } printf("\n"); } } else if (display & (SHMINFO | SHMTOTAL)) { fprintf(stderr, "SVID shared memory facility " "not configured in the system\n"); } kget(X_SEMINFO, &seminfo, sizeof(seminfo)); if ((display & (SEMINFO | SEMTOTAL))) { struct semid_kernel *kxsema; size_t kxsema_len; if (display & SEMTOTAL) print_ksemtotal(seminfo); if (display & SEMINFO) { kxsema_len = sizeof(struct semid_kernel) * seminfo.semmni; kxsema = malloc(kxsema_len); kget(X_SEMA, kxsema, kxsema_len); print_ksemheader(option); for (i = 0; i < seminfo.semmni; i += 1) { if ((kxsema[i].u.sem_perm.mode & SEM_ALLOC) != 0) { if (user && uid != kxsema[i].u.sem_perm.uid) continue; print_ksemptr(i, option, &kxsema[i]); } } printf("\n"); } } else if (display & (SEMINFO | SEMTOTAL)) { fprintf(stderr, "SVID semaphores facility " "not configured in the system\n"); } if (!use_sysctl) kvm_close(kd); exit(0); } void -print_kmsqtotal(struct msginfo msginfo) +print_kmsqtotal(struct msginfo local_msginfo) { printf("msginfo:\n"); printf("\tmsgmax: %12d\t(max characters in a message)\n", - msginfo.msgmax); + local_msginfo.msgmax); printf("\tmsgmni: %12d\t(# of message queues)\n", - msginfo.msgmni); + local_msginfo.msgmni); printf("\tmsgmnb: %12d\t(max characters in a message queue)\n", - msginfo.msgmnb); + local_msginfo.msgmnb); printf("\tmsgtql: %12d\t(max # of messages in system)\n", - msginfo.msgtql); + local_msginfo.msgtql); printf("\tmsgssz: %12d\t(size of a message segment)\n", - msginfo.msgssz); + local_msginfo.msgssz); printf("\tmsgseg: %12d\t(# of message segments in system)\n\n", - msginfo.msgseg); + local_msginfo.msgseg); } void print_kmsqheader(int option) { printf("Message Queues:\n"); printf("T %12s %12s %-11s %-8s %-8s", "ID", "KEY", "MODE", "OWNER", "GROUP"); if (option & CREATOR) printf(" %-8s %-8s", "CREATOR", "CGROUP"); if (option & OUTSTANDING) printf(" %20s %20s", "CBYTES", "QNUM"); if (option & BIGGEST) printf(" %20s", "QBYTES"); if (option & PID) printf(" %12s %12s", "LSPID", "LRPID"); if (option & TIME) printf(" %-8s %-8s %-8s", "STIME", "RTIME", "CTIME"); printf("\n"); } void print_kmsqptr(int i, int option, struct msqid_kernel *kmsqptr) { char stime_buf[100], rtime_buf[100], ctime_buf[100]; cvt_time(kmsqptr->u.msg_stime, stime_buf); cvt_time(kmsqptr->u.msg_rtime, rtime_buf); cvt_time(kmsqptr->u.msg_ctime, ctime_buf); printf("q %12d %12d %s %-8s %-8s", IXSEQ_TO_IPCID(i, kmsqptr->u.msg_perm), (int)kmsqptr->u.msg_perm.key, fmt_perm(kmsqptr->u.msg_perm.mode), user_from_uid(kmsqptr->u.msg_perm.uid, 0), group_from_gid(kmsqptr->u.msg_perm.gid, 0)); if (option & CREATOR) printf(" %-8s %-8s", user_from_uid(kmsqptr->u.msg_perm.cuid, 0), group_from_gid(kmsqptr->u.msg_perm.cgid, 0)); if (option & OUTSTANDING) printf(" %12lu %12lu", kmsqptr->u.msg_cbytes, kmsqptr->u.msg_qnum); if (option & BIGGEST) printf(" %20lu", kmsqptr->u.msg_qbytes); if (option & PID) printf(" %12d %12d", kmsqptr->u.msg_lspid, kmsqptr->u.msg_lrpid); if (option & TIME) printf(" %s %s %s", stime_buf, rtime_buf, ctime_buf); printf("\n"); } void -print_kshmtotal(struct shminfo shminfo) +print_kshmtotal(struct shminfo local_shminfo) { printf("shminfo:\n"); printf("\tshmmax: %12lu\t(max shared memory segment size)\n", - shminfo.shmmax); + local_shminfo.shmmax); printf("\tshmmin: %12lu\t(min shared memory segment size)\n", - shminfo.shmmin); + local_shminfo.shmmin); printf("\tshmmni: %12lu\t(max number of shared memory identifiers)\n", - shminfo.shmmni); + local_shminfo.shmmni); printf("\tshmseg: %12lu\t(max shared memory segments per process)\n", - shminfo.shmseg); + local_shminfo.shmseg); printf("\tshmall: %12lu\t(max amount of shared memory in pages)\n\n", - shminfo.shmall); + local_shminfo.shmall); } void print_kshmheader(int option) { printf("Shared Memory:\n"); printf("T %12s %12s %-11s %-8s %-8s", "ID", "KEY", "MODE", "OWNER", "GROUP"); if (option & CREATOR) printf(" %-8s %-8s", "CREATOR", "CGROUP"); if (option & OUTSTANDING) printf(" %12s", "NATTCH"); if (option & BIGGEST) printf(" %12s", "SEGSZ"); if (option & PID) printf(" %12s %12s", "CPID", "LPID"); if (option & TIME) printf(" %-8s %-8s %-8s", "ATIME", "DTIME", "CTIME"); printf("\n"); } void print_kshmptr(int i, int option, struct shmid_kernel *kshmptr) { char atime_buf[100], dtime_buf[100], ctime_buf[100]; cvt_time(kshmptr->u.shm_atime, atime_buf); cvt_time(kshmptr->u.shm_dtime, dtime_buf); cvt_time(kshmptr->u.shm_ctime, ctime_buf); printf("m %12d %12d %s %-8s %-8s", IXSEQ_TO_IPCID(i, kshmptr->u.shm_perm), (int)kshmptr->u.shm_perm.key, fmt_perm(kshmptr->u.shm_perm.mode), user_from_uid(kshmptr->u.shm_perm.uid, 0), group_from_gid(kshmptr->u.shm_perm.gid, 0)); if (option & CREATOR) printf(" %-8s %-8s", user_from_uid(kshmptr->u.shm_perm.cuid, 0), group_from_gid(kshmptr->u.shm_perm.cgid, 0)); if (option & OUTSTANDING) printf(" %12d", kshmptr->u.shm_nattch); if (option & BIGGEST) printf(" %12zu", kshmptr->u.shm_segsz); if (option & PID) printf(" %12d %12d", kshmptr->u.shm_cpid, kshmptr->u.shm_lpid); if (option & TIME) printf(" %s %s %s", atime_buf, dtime_buf, ctime_buf); printf("\n"); } void -print_ksemtotal(struct seminfo seminfo) +print_ksemtotal(struct seminfo local_seminfo) { printf("seminfo:\n"); printf("\tsemmni: %12d\t(# of semaphore identifiers)\n", - seminfo.semmni); + local_seminfo.semmni); printf("\tsemmns: %12d\t(# of semaphores in system)\n", - seminfo.semmns); + local_seminfo.semmns); printf("\tsemmnu: %12d\t(# of undo structures in system)\n", - seminfo.semmnu); + local_seminfo.semmnu); printf("\tsemmsl: %12d\t(max # of semaphores per id)\n", - seminfo.semmsl); + local_seminfo.semmsl); printf("\tsemopm: %12d\t(max # of operations per semop call)\n", - seminfo.semopm); + local_seminfo.semopm); printf("\tsemume: %12d\t(max # of undo entries per process)\n", - seminfo.semume); + local_seminfo.semume); printf("\tsemusz: %12d\t(size in bytes of undo structure)\n", - seminfo.semusz); + local_seminfo.semusz); printf("\tsemvmx: %12d\t(semaphore maximum value)\n", - seminfo.semvmx); + local_seminfo.semvmx); printf("\tsemaem: %12d\t(adjust on exit max value)\n\n", - seminfo.semaem); + local_seminfo.semaem); } void print_ksemheader(int option) { printf("Semaphores:\n"); printf("T %12s %12s %-11s %-8s %-8s", "ID", "KEY", "MODE", "OWNER", "GROUP"); if (option & CREATOR) printf(" %-8s %-8s", "CREATOR", "CGROUP"); if (option & BIGGEST) printf(" %12s", "NSEMS"); if (option & TIME) printf(" %-8s %-8s", "OTIME", "CTIME"); printf("\n"); } void print_ksemptr(int i, int option, struct semid_kernel *ksemaptr) { char ctime_buf[100], otime_buf[100]; cvt_time(ksemaptr->u.sem_otime, otime_buf); cvt_time(ksemaptr->u.sem_ctime, ctime_buf); printf("s %12d %12d %s %-8s %-8s", IXSEQ_TO_IPCID(i, ksemaptr->u.sem_perm), (int)ksemaptr->u.sem_perm.key, fmt_perm(ksemaptr->u.sem_perm.mode), user_from_uid(ksemaptr->u.sem_perm.uid, 0), group_from_gid(ksemaptr->u.sem_perm.gid, 0)); if (option & CREATOR) printf(" %-8s %-8s", user_from_uid(ksemaptr->u.sem_perm.cuid, 0), group_from_gid(ksemaptr->u.sem_perm.cgid, 0)); if (option & BIGGEST) printf(" %12d", ksemaptr->u.sem_nsems); if (option & TIME) printf(" %s %s", otime_buf, ctime_buf); printf("\n"); } uid_t user2uid(char *username) { struct passwd *pwd; uid_t uid; char *r; uid = strtoul(username, &r, 0); if (!*r && r != username) return (uid); if ((pwd = getpwnam(username)) == NULL) errx(1, "getpwnam failed: No such user"); endpwent(); return (pwd->pw_uid); } void usage(void) { fprintf(stderr, "usage: " "ipcs [-abcmopqstyMQST] [-C corefile] [-N namelist] [-u user]\n"); exit(1); } Index: user/ngie/more-tests =================================================================== --- user/ngie/more-tests (revision 281503) +++ user/ngie/more-tests (revision 281504) Property changes on: user/ngie/more-tests ___________________________________________________________________ Modified: svn:ignore ## -8,3 +8,9 ## _.powerpc.* _.sparc64.* _.sun4v.* +GPATH +GRTAGS +GTAGS +ID +cscope.out +?cscope.out Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /head:r281477-281503