Index: head/share/man/man9/efirt.9 =================================================================== --- head/share/man/man9/efirt.9 (revision 342107) +++ head/share/man/man9/efirt.9 (revision 342108) @@ -1,250 +1,263 @@ .\"- .\" SPDX-License-Identifier: BSD-2-Clause-FreeBSD .\" .\" Copyright (c) 2018 Kyle Evans .\" .\" Redistribution and use in source and binary forms, with or without .\" modification, are permitted provided that the following conditions .\" are met: .\" 1. Redistributions of source code must retain the above copyright .\" notice, this list of conditions and the following disclaimer. .\" 2. Redistributions in binary form must reproduce the above copyright .\" notice, this list of conditions and the following disclaimer in the .\" documentation and/or other materials provided with the distribution. .\" .\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND .\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE .\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE .\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE .\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL .\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS .\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) .\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT .\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF .\" SUCH DAMAGE. .\" .\" $FreeBSD$ .\" -.Dd August 12, 2018 +.Dd December 11, 2018 .Dt EFIRT 9 .Os .Sh NAME .Nm efirt , .Nm efi_rt_ok , .Nm efi_get_table , .Nm efi_get_time , .Nm efi_get_time_capabilities , .Nm efi_reset_system , .Nm efi_set_time , .Nm efi_var_get , .Nm efi_var_nextname , .Nm efi_var_set .Nd kernel access to UEFI runtime services .Sh SYNOPSIS .Cd "options EFIRT" .Pp .In sys/efi.h .Ft int .Fn efi_rt_ok "void" .Ft int .Fn efi_get_table "struct uuid *uuid" "void **ptr" .Ft int .Fn efi_get_time "struct efi_tm *tm" .Ft int .Fn efi_get_time_capabilities "struct efi_tmcap *tmcap" .Ft int -.Fn efi_reset_system "void" +.Fn efi_reset_system "enum efi_reset type" .Ft int .Fn efi_set_time "struct efi_tm *tm" .Ft int .Fn efi_var_get "uint16_t *name" "struct uuid *vendor" "uint32_t *attrib" \ "size_t *datasize" "void *data" .Ft int .Fn efi_var_nextname "size_t *namesize" "uint16_t *name" "struct uuid *vendor" .Ft int .Fn efi_var_set "uint16_t *name" "struct uuid *vendor" "uint32_t *attrib" \ "size_t *datasize" "void *data" .Sh DESCRIPTION All of the following calls will return .Dv ENXIO if UEFI runtime services are not available. .Nm is currently only available on amd64 and arm64. .Pp The .Fn efi_rt_ok Returns 0 if UEFI runtime services are present and functional, or .Dv ENXIO if not. .Pp The .Fn efi_get_table function gets a table by uuid from the UEFI system table. Returns 0 if the table was found and populates *ptr with the address. Returns .Dv ENXIO if the configuration table or system table are unset. Returns .Dv ENOENT if the requested table cannot be found. .Pp The .Fn efi_get_time function gets the current time from the RTC, if available. Returns 0 and populates the .Vt struct efi_tm on success. Returns .Dv EINVAL if the .Vt struct efi_tm is .Dv NULL , or .Dv EIO if the time could not be retrieved due to a hardware error. .Pp The .Fn efi_get_time_capabilities function gets the capabilities from the RTC. Returns 0 and populates the .Vt struct efi_tmcap on success. Returns .Dv EINVAL if the .Vt struct efi_tm is .Dv NULL , or .Dv EIO if the time could not be retrieved due to a hardware error. .Pp The .Fn efi_reset_system -function requests a warm reset and reboot of the system. +function requests a reset of the system. +The +.Fa type +argument may be one of the +.Vt enum efi_reset +values: +.Bl -tag -width ".Dv EFI_RESET_SHUTDOWN" +.It Dv EFI_RESET_COLD +Perform a cold reset of the system, and reboot. +.It Dv EFI_RESET_WARM +Perform a warm reset of the system, and reboot. +.It Dv EFI_RESET_SHUTDOWN +Power off the system. +.El .Pp The .Fn efi_set_time function sets the time on the RTC to the time described by the .Vt struct efi_tm passed in. Returns 0 on success, .Dv EINVAL if a time field is out of range, or .Dv EIO if the time could not be set due to a hardware error. .Pp The .Fn efi_var_get function fetches the variable identified by .Fa vendor and .Fa name . Returns 0 and populates .Fa attrib , .Fa datasize , and .Fa data on success. Otherwise, one of the following errors are returned: .Bl -tag -width ".Dv EOVERFLOW" .It Dv ENOENT The variable was not found. .It Dv EOVERFLOW .Fa datasize is not sufficient to hold the variable data. .Fa namesize is updated to reflect the size needed to complete the request. .It Dv EINVAL One of .Fa name , .Fa vendor , or .Fa datasize are NULL. Alternatively, .Fa datasize is large enough to hold the response but .Fa data is NULL. .It Dv EIO The variable could not be retrieved due to a hardware error. .It Dv EDOOFUS The variable could not be retireved due to an authentication failure. .El .Pp The .Fn efi_var_nextname function is used for enumeration of variables. On the initial call to .Fn efi_var_nextname , .Fa name should be an empty string. Subsequent calls should pass in the last .Fa name and .Fa vendor returned until .Dv ENOENT is returned. Returns 0 and populates .Fa namesize , .Fa name , and .Fa vendor with the next variable's data. Otherwise, returns one of the following errors: .Bl -tag -width ".Dv EOVERFLOW" .It Dv ENOENT The next variable was not found. .It Dv EOVERFLOW .Fa datasize is not sufficient to hold the variable data. .Fa namesize is updated to reflect the size needed to complete the request. .It Dv EINVAL One of .Fa name , .Fa vendor , or .Fa datasize are NULL. .It Dv EIO The variable could not be retrieved due to a hardware error. .El .Pp The .Fn efi_var_set function sets the variable described by .Fa name and .Fa vendor . Returns 0 if the variable has been successfully. Otherwise, returns one of the following errors: .Bl -tag -width ".Dv EOVERFLOW" .It Dv EINVAL Either .Fa attrib was an invalid combination of attributes, .Fa datasize exceeds the maximum allowed size, or .Fa name is an empty Unicode stirng. .It Dv EAGAIN Not enough storage is available to hold the variable and its data. .It Dv EIO The variable could not be saved due to a hardware error. .It Dv EROFS The variable in question is read-only or may not be deleted. .It Dv EDOOFUS -The varialbe could not be set due to an authentication failure. +The variable could not be set due to an authentication failure. .It Dv ENOENT The variable trying to be updated or deleted was not found. .El .Sh SEE ALSO .Xr efidev 4 .Sh AUTHORS This manual page was written by .An Kyle Evans Aq Mt kevans@FreeBSD.org . Index: head/sys/contrib/zstd/lib/compress/zstd_compress_internal.h =================================================================== --- head/sys/contrib/zstd/lib/compress/zstd_compress_internal.h (revision 342107) +++ head/sys/contrib/zstd/lib/compress/zstd_compress_internal.h (revision 342108) @@ -1,798 +1,800 @@ /* * Copyright (c) 2016-present, Yann Collet, Facebook, Inc. * All rights reserved. * * This source code is licensed under both the BSD-style license (found in the * LICENSE file in the root directory of this source tree) and the GPLv2 (found * in the COPYING file in the root directory of this source tree). * You may select, at your option, one of the above-listed licenses. */ /* This header contains definitions * that shall **only** be used by modules within lib/compress. */ #ifndef ZSTD_COMPRESS_H #define ZSTD_COMPRESS_H /*-************************************* * Dependencies ***************************************/ #include "zstd_internal.h" #ifdef ZSTD_MULTITHREAD # include "zstdmt_compress.h" #endif #if defined (__cplusplus) extern "C" { #endif /*-************************************* * Constants ***************************************/ #define kSearchStrength 8 #define HASH_READ_SIZE 8 #define ZSTD_DUBT_UNSORTED_MARK 1 /* For btlazy2 strategy, index 1 now means "unsorted". It could be confused for a real successor at index "1", if sorted as larger than its predecessor. It's not a big deal though : candidate will just be sorted again. Additionnally, candidate position 1 will be lost. But candidate 1 cannot hide a large tree of candidates, so it's a minimal loss. The benefit is that ZSTD_DUBT_UNSORTED_MARK cannot be misdhandled after table re-use with a different strategy Constant required by ZSTD_compressBlock_btlazy2() and ZSTD_reduceTable_internal() */ /*-************************************* * Context memory management ***************************************/ typedef enum { ZSTDcs_created=0, ZSTDcs_init, ZSTDcs_ongoing, ZSTDcs_ending } ZSTD_compressionStage_e; typedef enum { zcss_init=0, zcss_load, zcss_flush } ZSTD_cStreamStage; typedef enum { ZSTD_dictDefaultAttach = 0, ZSTD_dictForceAttach = 1, ZSTD_dictForceCopy = -1, } ZSTD_dictAttachPref_e; typedef struct ZSTD_prefixDict_s { const void* dict; size_t dictSize; ZSTD_dictContentType_e dictContentType; } ZSTD_prefixDict; typedef struct { U32 CTable[HUF_CTABLE_SIZE_U32(255)]; HUF_repeat repeatMode; } ZSTD_hufCTables_t; typedef struct { FSE_CTable offcodeCTable[FSE_CTABLE_SIZE_U32(OffFSELog, MaxOff)]; FSE_CTable matchlengthCTable[FSE_CTABLE_SIZE_U32(MLFSELog, MaxML)]; FSE_CTable litlengthCTable[FSE_CTABLE_SIZE_U32(LLFSELog, MaxLL)]; FSE_repeat offcode_repeatMode; FSE_repeat matchlength_repeatMode; FSE_repeat litlength_repeatMode; } ZSTD_fseCTables_t; typedef struct { ZSTD_hufCTables_t huf; ZSTD_fseCTables_t fse; } ZSTD_entropyCTables_t; typedef struct { U32 off; U32 len; } ZSTD_match_t; typedef struct { int price; U32 off; U32 mlen; U32 litlen; U32 rep[ZSTD_REP_NUM]; } ZSTD_optimal_t; typedef enum { zop_dynamic=0, zop_predef } ZSTD_OptPrice_e; typedef struct { /* All tables are allocated inside cctx->workspace by ZSTD_resetCCtx_internal() */ U32* litFreq; /* table of literals statistics, of size 256 */ U32* litLengthFreq; /* table of litLength statistics, of size (MaxLL+1) */ U32* matchLengthFreq; /* table of matchLength statistics, of size (MaxML+1) */ U32* offCodeFreq; /* table of offCode statistics, of size (MaxOff+1) */ ZSTD_match_t* matchTable; /* list of found matches, of size ZSTD_OPT_NUM+1 */ ZSTD_optimal_t* priceTable; /* All positions tracked by optimal parser, of size ZSTD_OPT_NUM+1 */ U32 litSum; /* nb of literals */ U32 litLengthSum; /* nb of litLength codes */ U32 matchLengthSum; /* nb of matchLength codes */ U32 offCodeSum; /* nb of offset codes */ U32 litSumBasePrice; /* to compare to log2(litfreq) */ U32 litLengthSumBasePrice; /* to compare to log2(llfreq) */ U32 matchLengthSumBasePrice;/* to compare to log2(mlfreq) */ U32 offCodeSumBasePrice; /* to compare to log2(offreq) */ ZSTD_OptPrice_e priceType; /* prices can be determined dynamically, or follow a pre-defined cost structure */ const ZSTD_entropyCTables_t* symbolCosts; /* pre-calculated dictionary statistics */ } optState_t; typedef struct { ZSTD_entropyCTables_t entropy; U32 rep[ZSTD_REP_NUM]; } ZSTD_compressedBlockState_t; typedef struct { BYTE const* nextSrc; /* next block here to continue on current prefix */ BYTE const* base; /* All regular indexes relative to this position */ BYTE const* dictBase; /* extDict indexes relative to this position */ U32 dictLimit; /* below that point, need extDict */ U32 lowLimit; /* below that point, no more data */ } ZSTD_window_t; typedef struct ZSTD_matchState_t ZSTD_matchState_t; struct ZSTD_matchState_t { ZSTD_window_t window; /* State for window round buffer management */ U32 loadedDictEnd; /* index of end of dictionary */ U32 nextToUpdate; /* index from which to continue table update */ U32 nextToUpdate3; /* index from which to continue table update */ U32 hashLog3; /* dispatch table : larger == faster, more memory */ U32* hashTable; U32* hashTable3; U32* chainTable; optState_t opt; /* optimal parser state */ const ZSTD_matchState_t *dictMatchState; ZSTD_compressionParameters cParams; }; typedef struct { ZSTD_compressedBlockState_t* prevCBlock; ZSTD_compressedBlockState_t* nextCBlock; ZSTD_matchState_t matchState; } ZSTD_blockState_t; typedef struct { U32 offset; U32 checksum; } ldmEntry_t; typedef struct { ZSTD_window_t window; /* State for the window round buffer management */ ldmEntry_t* hashTable; BYTE* bucketOffsets; /* Next position in bucket to insert entry */ U64 hashPower; /* Used to compute the rolling hash. * Depends on ldmParams.minMatchLength */ } ldmState_t; typedef struct { U32 enableLdm; /* 1 if enable long distance matching */ U32 hashLog; /* Log size of hashTable */ U32 bucketSizeLog; /* Log bucket size for collision resolution, at most 8 */ U32 minMatchLength; /* Minimum match length */ U32 hashEveryLog; /* Log number of entries to skip */ U32 windowLog; /* Window log for the LDM */ } ldmParams_t; typedef struct { U32 offset; U32 litLength; U32 matchLength; } rawSeq; typedef struct { rawSeq* seq; /* The start of the sequences */ size_t pos; /* The position where reading stopped. <= size. */ size_t size; /* The number of sequences. <= capacity. */ size_t capacity; /* The capacity starting from `seq` pointer */ } rawSeqStore_t; struct ZSTD_CCtx_params_s { ZSTD_format_e format; ZSTD_compressionParameters cParams; ZSTD_frameParameters fParams; int compressionLevel; int forceWindow; /* force back-references to respect limit of * 1< 63) ? ZSTD_highbit32(litLength) + LL_deltaCode : LL_Code[litLength]; } /* ZSTD_MLcode() : * note : mlBase = matchLength - MINMATCH; * because it's the format it's stored in seqStore->sequences */ MEM_STATIC U32 ZSTD_MLcode(U32 mlBase) { static const BYTE ML_Code[128] = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 32, 33, 33, 34, 34, 35, 35, 36, 36, 36, 36, 37, 37, 37, 37, 38, 38, 38, 38, 38, 38, 38, 38, 39, 39, 39, 39, 39, 39, 39, 39, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 40, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 41, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42, 42 }; static const U32 ML_deltaCode = 36; return (mlBase > 127) ? ZSTD_highbit32(mlBase) + ML_deltaCode : ML_Code[mlBase]; } /*! ZSTD_storeSeq() : * Store a sequence (literal length, literals, offset code and match length code) into seqStore_t. * `offsetCode` : distance to match + 3 (values 1-3 are repCodes). * `mlBase` : matchLength - MINMATCH */ MEM_STATIC void ZSTD_storeSeq(seqStore_t* seqStorePtr, size_t litLength, const void* literals, U32 offsetCode, size_t mlBase) { #if defined(DEBUGLEVEL) && (DEBUGLEVEL >= 6) static const BYTE* g_start = NULL; if (g_start==NULL) g_start = (const BYTE*)literals; /* note : index only works for compression within a single segment */ { U32 const pos = (U32)((const BYTE*)literals - g_start); DEBUGLOG(6, "Cpos%7u :%3u literals, match%4u bytes at offCode%7u", pos, (U32)litLength, (U32)mlBase+MINMATCH, (U32)offsetCode); } #endif assert((size_t)(seqStorePtr->sequences - seqStorePtr->sequencesStart) < seqStorePtr->maxNbSeq); /* copy Literals */ assert(seqStorePtr->maxNbLit <= 128 KB); assert(seqStorePtr->lit + litLength <= seqStorePtr->litStart + seqStorePtr->maxNbLit); ZSTD_wildcopy(seqStorePtr->lit, literals, litLength); seqStorePtr->lit += litLength; /* literal Length */ if (litLength>0xFFFF) { assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */ seqStorePtr->longLengthID = 1; seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); } seqStorePtr->sequences[0].litLength = (U16)litLength; /* match offset */ seqStorePtr->sequences[0].offset = offsetCode + 1; /* match Length */ if (mlBase>0xFFFF) { assert(seqStorePtr->longLengthID == 0); /* there can only be a single long length */ seqStorePtr->longLengthID = 2; seqStorePtr->longLengthPos = (U32)(seqStorePtr->sequences - seqStorePtr->sequencesStart); } seqStorePtr->sequences[0].matchLength = (U16)mlBase; seqStorePtr->sequences++; } /*-************************************* * Match length counter ***************************************/ static unsigned ZSTD_NbCommonBytes (size_t val) { if (MEM_isLittleEndian()) { if (MEM_64bits()) { # if defined(_MSC_VER) && defined(_WIN64) unsigned long r = 0; _BitScanForward64( &r, (U64)val ); return (unsigned)(r>>3); # elif defined(__GNUC__) && (__GNUC__ >= 4) return (__builtin_ctzll((U64)val) >> 3); # else static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2, 0, 3, 1, 3, 1, 4, 2, 7, 0, 2, 3, 6, 1, 5, 3, 5, 1, 3, 4, 4, 2, 5, 6, 7, 7, 0, 1, 2, 3, 3, 4, 6, 2, 6, 5, 5, 3, 4, 5, 6, 7, 1, 2, 4, 6, 4, 4, 5, 7, 2, 6, 5, 7, 6, 7, 7 }; return DeBruijnBytePos[((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58]; # endif } else { /* 32 bits */ # if defined(_MSC_VER) unsigned long r=0; _BitScanForward( &r, (U32)val ); return (unsigned)(r>>3); # elif defined(__GNUC__) && (__GNUC__ >= 3) return (__builtin_ctz((U32)val) >> 3); # else static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0, 3, 2, 2, 1, 3, 2, 0, 1, 3, 3, 1, 2, 2, 2, 2, 0, 3, 1, 2, 0, 1, 0, 1, 1 }; return DeBruijnBytePos[((U32)((val & -(S32)val) * 0x077CB531U)) >> 27]; # endif } } else { /* Big Endian CPU */ if (MEM_64bits()) { # if defined(_MSC_VER) && defined(_WIN64) unsigned long r = 0; _BitScanReverse64( &r, val ); return (unsigned)(r>>3); # elif defined(__GNUC__) && (__GNUC__ >= 4) return (__builtin_clzll(val) >> 3); # else unsigned r; const unsigned n32 = sizeof(size_t)*4; /* calculate this way due to compiler complaining in 32-bits mode */ if (!(val>>n32)) { r=4; } else { r=0; val>>=n32; } if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; } r += (!val); return r; # endif } else { /* 32 bits */ # if defined(_MSC_VER) unsigned long r = 0; _BitScanReverse( &r, (unsigned long)val ); return (unsigned)(r>>3); # elif defined(__GNUC__) && (__GNUC__ >= 3) return (__builtin_clz((U32)val) >> 3); # else unsigned r; if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; } r += (!val); return r; # endif } } } MEM_STATIC size_t ZSTD_count(const BYTE* pIn, const BYTE* pMatch, const BYTE* const pInLimit) { const BYTE* const pStart = pIn; const BYTE* const pInLoopLimit = pInLimit - (sizeof(size_t)-1); if (pIn < pInLoopLimit) { { size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn); if (diff) return ZSTD_NbCommonBytes(diff); } pIn+=sizeof(size_t); pMatch+=sizeof(size_t); while (pIn < pInLoopLimit) { size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn); if (!diff) { pIn+=sizeof(size_t); pMatch+=sizeof(size_t); continue; } pIn += ZSTD_NbCommonBytes(diff); return (size_t)(pIn - pStart); } } if (MEM_64bits() && (pIn<(pInLimit-3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) { pIn+=4; pMatch+=4; } if ((pIn<(pInLimit-1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) { pIn+=2; pMatch+=2; } if ((pIn> (32-h) ; } MEM_STATIC size_t ZSTD_hash3Ptr(const void* ptr, U32 h) { return ZSTD_hash3(MEM_readLE32(ptr), h); } /* only in zstd_opt.h */ static const U32 prime4bytes = 2654435761U; static U32 ZSTD_hash4(U32 u, U32 h) { return (u * prime4bytes) >> (32-h) ; } static size_t ZSTD_hash4Ptr(const void* ptr, U32 h) { return ZSTD_hash4(MEM_read32(ptr), h); } static const U64 prime5bytes = 889523592379ULL; static size_t ZSTD_hash5(U64 u, U32 h) { return (size_t)(((u << (64-40)) * prime5bytes) >> (64-h)) ; } static size_t ZSTD_hash5Ptr(const void* p, U32 h) { return ZSTD_hash5(MEM_readLE64(p), h); } static const U64 prime6bytes = 227718039650203ULL; static size_t ZSTD_hash6(U64 u, U32 h) { return (size_t)(((u << (64-48)) * prime6bytes) >> (64-h)) ; } static size_t ZSTD_hash6Ptr(const void* p, U32 h) { return ZSTD_hash6(MEM_readLE64(p), h); } static const U64 prime7bytes = 58295818150454627ULL; static size_t ZSTD_hash7(U64 u, U32 h) { return (size_t)(((u << (64-56)) * prime7bytes) >> (64-h)) ; } static size_t ZSTD_hash7Ptr(const void* p, U32 h) { return ZSTD_hash7(MEM_readLE64(p), h); } static const U64 prime8bytes = 0xCF1BBCDCB7A56463ULL; static size_t ZSTD_hash8(U64 u, U32 h) { return (size_t)(((u) * prime8bytes) >> (64-h)) ; } static size_t ZSTD_hash8Ptr(const void* p, U32 h) { return ZSTD_hash8(MEM_readLE64(p), h); } MEM_STATIC size_t ZSTD_hashPtr(const void* p, U32 hBits, U32 mls) { switch(mls) { default: case 4: return ZSTD_hash4Ptr(p, hBits); case 5: return ZSTD_hash5Ptr(p, hBits); case 6: return ZSTD_hash6Ptr(p, hBits); case 7: return ZSTD_hash7Ptr(p, hBits); case 8: return ZSTD_hash8Ptr(p, hBits); } } /*-************************************* * Round buffer management ***************************************/ /* Max current allowed */ #define ZSTD_CURRENT_MAX ((3U << 29) + (1U << ZSTD_WINDOWLOG_MAX)) /* Maximum chunk size before overflow correction needs to be called again */ #define ZSTD_CHUNKSIZE_MAX \ ( ((U32)-1) /* Maximum ending current index */ \ - ZSTD_CURRENT_MAX) /* Maximum beginning lowLimit */ /** * ZSTD_window_clear(): * Clears the window containing the history by simply setting it to empty. */ MEM_STATIC void ZSTD_window_clear(ZSTD_window_t* window) { size_t const endT = (size_t)(window->nextSrc - window->base); U32 const end = (U32)endT; window->lowLimit = end; window->dictLimit = end; } /** * ZSTD_window_hasExtDict(): * Returns non-zero if the window has a non-empty extDict. */ MEM_STATIC U32 ZSTD_window_hasExtDict(ZSTD_window_t const window) { return window.lowLimit < window.dictLimit; } /** * ZSTD_matchState_dictMode(): * Inspects the provided matchState and figures out what dictMode should be * passed to the compressor. */ MEM_STATIC ZSTD_dictMode_e ZSTD_matchState_dictMode(const ZSTD_matchState_t *ms) { return ZSTD_window_hasExtDict(ms->window) ? ZSTD_extDict : ms->dictMatchState != NULL ? ZSTD_dictMatchState : ZSTD_noDict; } /** * ZSTD_window_needOverflowCorrection(): * Returns non-zero if the indices are getting too large and need overflow * protection. */ MEM_STATIC U32 ZSTD_window_needOverflowCorrection(ZSTD_window_t const window, void const* srcEnd) { U32 const current = (U32)((BYTE const*)srcEnd - window.base); return current > ZSTD_CURRENT_MAX; } /** * ZSTD_window_correctOverflow(): * Reduces the indices to protect from index overflow. * Returns the correction made to the indices, which must be applied to every * stored index. * * The least significant cycleLog bits of the indices must remain the same, * which may be 0. Every index up to maxDist in the past must be valid. * NOTE: (maxDist & cycleMask) must be zero. */ MEM_STATIC U32 ZSTD_window_correctOverflow(ZSTD_window_t* window, U32 cycleLog, U32 maxDist, void const* src) { /* preemptive overflow correction: * 1. correction is large enough: * lowLimit > (3<<29) ==> current > 3<<29 + 1< (3<<29 + 1< (3<<29) - (1< (3<<29) - (1<<30) (NOTE: chainLog <= 30) * > 1<<29 * * 2. (ip+ZSTD_CHUNKSIZE_MAX - cctx->base) doesn't overflow: * After correction, current is less than (1<base < 1<<32. * 3. (cctx->lowLimit + 1< 3<<29 + 1<base); U32 const newCurrent = (current & cycleMask) + maxDist; U32 const correction = current - newCurrent; assert((maxDist & cycleMask) == 0); assert(current > newCurrent); /* Loose bound, should be around 1<<29 (see above) */ assert(correction > 1<<28); window->base += correction; window->dictBase += correction; window->lowLimit -= correction; window->dictLimit -= correction; DEBUGLOG(4, "Correction of 0x%x bytes to lowLimit=0x%x", correction, window->lowLimit); return correction; } /** * ZSTD_window_enforceMaxDist(): * Updates lowLimit so that: * (srcEnd - base) - lowLimit == maxDist + loadedDictEnd * * This allows a simple check that index >= lowLimit to see if index is valid. * This must be called before a block compression call, with srcEnd as the block * source end. * * If loadedDictEndPtr is not NULL, we set it to zero once we update lowLimit. * This is because dictionaries are allowed to be referenced as long as the last * byte of the dictionary is in the window, but once they are out of range, * they cannot be referenced. If loadedDictEndPtr is NULL, we use * loadedDictEnd == 0. * * In normal dict mode, the dict is between lowLimit and dictLimit. In * dictMatchState mode, lowLimit and dictLimit are the same, and the dictionary * is below them. forceWindow and dictMatchState are therefore incompatible. */ MEM_STATIC void ZSTD_window_enforceMaxDist(ZSTD_window_t* window, void const* srcEnd, U32 maxDist, U32* loadedDictEndPtr, const ZSTD_matchState_t** dictMatchStatePtr) { U32 const current = (U32)((BYTE const*)srcEnd - window->base); U32 loadedDictEnd = loadedDictEndPtr != NULL ? *loadedDictEndPtr : 0; DEBUGLOG(5, "ZSTD_window_enforceMaxDist: current=%u, maxDist=%u", current, maxDist); if (current > maxDist + loadedDictEnd) { U32 const newLowLimit = current - maxDist; if (window->lowLimit < newLowLimit) window->lowLimit = newLowLimit; if (window->dictLimit < window->lowLimit) { DEBUGLOG(5, "Update dictLimit to match lowLimit, from %u to %u", window->dictLimit, window->lowLimit); window->dictLimit = window->lowLimit; } if (loadedDictEndPtr) *loadedDictEndPtr = 0; if (dictMatchStatePtr) *dictMatchStatePtr = NULL; } } /** * ZSTD_window_update(): * Updates the window by appending [src, src + srcSize) to the window. * If it is not contiguous, the current prefix becomes the extDict, and we * forget about the extDict. Handles overlap of the prefix and extDict. * Returns non-zero if the segment is contiguous. */ MEM_STATIC U32 ZSTD_window_update(ZSTD_window_t* window, void const* src, size_t srcSize) { BYTE const* const ip = (BYTE const*)src; U32 contiguous = 1; DEBUGLOG(5, "ZSTD_window_update"); /* Check if blocks follow each other */ if (src != window->nextSrc) { /* not contiguous */ size_t const distanceFromBase = (size_t)(window->nextSrc - window->base); DEBUGLOG(5, "Non contiguous blocks, new segment starts at %u", window->dictLimit); window->lowLimit = window->dictLimit; assert(distanceFromBase == (size_t)(U32)distanceFromBase); /* should never overflow */ window->dictLimit = (U32)distanceFromBase; window->dictBase = window->base; window->base = ip - distanceFromBase; // ms->nextToUpdate = window->dictLimit; if (window->dictLimit - window->lowLimit < HASH_READ_SIZE) window->lowLimit = window->dictLimit; /* too small extDict */ contiguous = 0; } window->nextSrc = ip + srcSize; /* if input and dictionary overlap : reduce dictionary (area presumed modified by input) */ if ( (ip+srcSize > window->dictBase + window->lowLimit) & (ip < window->dictBase + window->dictLimit)) { ptrdiff_t const highInputIdx = (ip + srcSize) - window->dictBase; U32 const lowLimitMax = (highInputIdx > (ptrdiff_t)window->dictLimit) ? window->dictLimit : (U32)highInputIdx; window->lowLimit = lowLimitMax; DEBUGLOG(5, "Overlapping extDict and input : new lowLimit = %u", window->lowLimit); } return contiguous; } /* debug functions */ +#if 0 MEM_STATIC double ZSTD_fWeight(U32 rawStat) { U32 const fp_accuracy = 8; U32 const fp_multiplier = (1 << fp_accuracy); U32 const stat = rawStat + 1; U32 const hb = ZSTD_highbit32(stat); U32 const BWeight = hb * fp_multiplier; U32 const FWeight = (stat << fp_accuracy) >> hb; U32 const weight = BWeight + FWeight; assert(hb + fp_accuracy < 31); return (double)weight / fp_multiplier; } MEM_STATIC void ZSTD_debugTable(const U32* table, U32 max) { unsigned u, sum; for (u=0, sum=0; u<=max; u++) sum += table[u]; DEBUGLOG(2, "total nb elts: %u", sum); for (u=0; u<=max; u++) { DEBUGLOG(2, "%2u: %5u (%.2f)", u, table[u], ZSTD_fWeight(sum) - ZSTD_fWeight(table[u]) ); } } +#endif #if defined (__cplusplus) } #endif /* ============================================================== * Private declarations * These prototypes shall only be called from within lib/compress * ============================================================== */ /* ZSTD_getCParamsFromCCtxParams() : * cParams are built depending on compressionLevel, src size hints, * LDM and manually set compression parameters. */ ZSTD_compressionParameters ZSTD_getCParamsFromCCtxParams( const ZSTD_CCtx_params* CCtxParams, U64 srcSizeHint, size_t dictSize); /*! ZSTD_initCStream_internal() : * Private use only. Init streaming operation. * expects params to be valid. * must receive dict, or cdict, or none, but not both. * @return : 0, or an error code */ size_t ZSTD_initCStream_internal(ZSTD_CStream* zcs, const void* dict, size_t dictSize, const ZSTD_CDict* cdict, ZSTD_CCtx_params params, unsigned long long pledgedSrcSize); void ZSTD_resetSeqStore(seqStore_t* ssPtr); /*! ZSTD_compressStream_generic() : * Private use only. To be called from zstdmt_compress.c in single-thread mode. */ size_t ZSTD_compressStream_generic(ZSTD_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input, ZSTD_EndDirective const flushMode); /*! ZSTD_getCParamsFromCDict() : * as the name implies */ ZSTD_compressionParameters ZSTD_getCParamsFromCDict(const ZSTD_CDict* cdict); /* ZSTD_compressBegin_advanced_internal() : * Private use only. To be called from zstdmt_compress.c. */ size_t ZSTD_compressBegin_advanced_internal(ZSTD_CCtx* cctx, const void* dict, size_t dictSize, ZSTD_dictContentType_e dictContentType, ZSTD_dictTableLoadMethod_e dtlm, const ZSTD_CDict* cdict, ZSTD_CCtx_params params, unsigned long long pledgedSrcSize); /* ZSTD_compress_advanced_internal() : * Private use only. To be called from zstdmt_compress.c. */ size_t ZSTD_compress_advanced_internal(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize, const void* dict,size_t dictSize, ZSTD_CCtx_params params); /* ZSTD_writeLastEmptyBlock() : * output an empty Block with end-of-frame mark to complete a frame * @return : size of data written into `dst` (== ZSTD_blockHeaderSize (defined in zstd_internal.h)) * or an error code if `dstCapcity` is too small (> hb; U32 const weight = BWeight + FWeight; assert(hb + BITCOST_ACCURACY < 31); return weight; } /* debugging function, @return price in bytes */ +#if 0 MEM_STATIC double ZSTD_fCost(U32 price) { return (double)price / (BITCOST_MULTIPLIER*8); } +#endif static void ZSTD_setBasePrices(optState_t* optPtr, int optLevel) { optPtr->litSumBasePrice = WEIGHT(optPtr->litSum, optLevel); optPtr->litLengthSumBasePrice = WEIGHT(optPtr->litLengthSum, optLevel); optPtr->matchLengthSumBasePrice = WEIGHT(optPtr->matchLengthSum, optLevel); optPtr->offCodeSumBasePrice = WEIGHT(optPtr->offCodeSum, optLevel); } static U32 ZSTD_downscaleStat(U32* table, U32 lastEltIndex, int malus) { U32 s, sum=0; assert(ZSTD_FREQ_DIV+malus > 0 && ZSTD_FREQ_DIV+malus < 31); for (s=0; s<=lastEltIndex; s++) { table[s] = 1 + (table[s] >> (ZSTD_FREQ_DIV+malus)); sum += table[s]; } return sum; } static void ZSTD_rescaleFreqs(optState_t* const optPtr, const BYTE* const src, size_t const srcSize, int optLevel) { optPtr->priceType = zop_dynamic; if (optPtr->litLengthSum == 0) { /* first block : init */ if (srcSize <= 1024) /* heuristic */ optPtr->priceType = zop_predef; assert(optPtr->symbolCosts != NULL); if (optPtr->symbolCosts->huf.repeatMode == HUF_repeat_valid) { /* huffman table presumed generated by dictionary */ optPtr->priceType = zop_dynamic; assert(optPtr->litFreq != NULL); optPtr->litSum = 0; { unsigned lit; for (lit=0; lit<=MaxLit; lit++) { U32 const scaleLog = 11; /* scale to 2K */ U32 const bitCost = HUF_getNbBits(optPtr->symbolCosts->huf.CTable, lit); assert(bitCost <= scaleLog); optPtr->litFreq[lit] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->litSum += optPtr->litFreq[lit]; } } { unsigned ll; FSE_CState_t llstate; FSE_initCState(&llstate, optPtr->symbolCosts->fse.litlengthCTable); optPtr->litLengthSum = 0; for (ll=0; ll<=MaxLL; ll++) { U32 const scaleLog = 10; /* scale to 1K */ U32 const bitCost = FSE_getMaxNbBits(llstate.symbolTT, ll); assert(bitCost < scaleLog); optPtr->litLengthFreq[ll] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->litLengthSum += optPtr->litLengthFreq[ll]; } } { unsigned ml; FSE_CState_t mlstate; FSE_initCState(&mlstate, optPtr->symbolCosts->fse.matchlengthCTable); optPtr->matchLengthSum = 0; for (ml=0; ml<=MaxML; ml++) { U32 const scaleLog = 10; U32 const bitCost = FSE_getMaxNbBits(mlstate.symbolTT, ml); assert(bitCost < scaleLog); optPtr->matchLengthFreq[ml] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->matchLengthSum += optPtr->matchLengthFreq[ml]; } } { unsigned of; FSE_CState_t ofstate; FSE_initCState(&ofstate, optPtr->symbolCosts->fse.offcodeCTable); optPtr->offCodeSum = 0; for (of=0; of<=MaxOff; of++) { U32 const scaleLog = 10; U32 const bitCost = FSE_getMaxNbBits(ofstate.symbolTT, of); assert(bitCost < scaleLog); optPtr->offCodeFreq[of] = bitCost ? 1 << (scaleLog-bitCost) : 1 /*minimum to calculate cost*/; optPtr->offCodeSum += optPtr->offCodeFreq[of]; } } } else { /* not a dictionary */ assert(optPtr->litFreq != NULL); { unsigned lit = MaxLit; HIST_count_simple(optPtr->litFreq, &lit, src, srcSize); /* use raw first block to init statistics */ } optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1); { unsigned ll; for (ll=0; ll<=MaxLL; ll++) optPtr->litLengthFreq[ll] = 1; } optPtr->litLengthSum = MaxLL+1; { unsigned ml; for (ml=0; ml<=MaxML; ml++) optPtr->matchLengthFreq[ml] = 1; } optPtr->matchLengthSum = MaxML+1; { unsigned of; for (of=0; of<=MaxOff; of++) optPtr->offCodeFreq[of] = 1; } optPtr->offCodeSum = MaxOff+1; } } else { /* new block : re-use previous statistics, scaled down */ optPtr->litSum = ZSTD_downscaleStat(optPtr->litFreq, MaxLit, 1); optPtr->litLengthSum = ZSTD_downscaleStat(optPtr->litLengthFreq, MaxLL, 0); optPtr->matchLengthSum = ZSTD_downscaleStat(optPtr->matchLengthFreq, MaxML, 0); optPtr->offCodeSum = ZSTD_downscaleStat(optPtr->offCodeFreq, MaxOff, 0); } ZSTD_setBasePrices(optPtr, optLevel); } /* ZSTD_rawLiteralsCost() : * price of literals (only) in specified segment (which length can be 0). * does not include price of literalLength symbol */ static U32 ZSTD_rawLiteralsCost(const BYTE* const literals, U32 const litLength, const optState_t* const optPtr, int optLevel) { if (litLength == 0) return 0; if (optPtr->priceType == zop_predef) return (litLength*6) * BITCOST_MULTIPLIER; /* 6 bit per literal - no statistic used */ /* dynamic statistics */ { U32 price = litLength * optPtr->litSumBasePrice; U32 u; for (u=0; u < litLength; u++) { assert(WEIGHT(optPtr->litFreq[literals[u]], optLevel) <= optPtr->litSumBasePrice); /* literal cost should never be negative */ price -= WEIGHT(optPtr->litFreq[literals[u]], optLevel); } return price; } } /* ZSTD_litLengthPrice() : * cost of literalLength symbol */ static U32 ZSTD_litLengthPrice(U32 const litLength, const optState_t* const optPtr, int optLevel) { if (optPtr->priceType == zop_predef) return WEIGHT(litLength, optLevel); /* dynamic statistics */ { U32 const llCode = ZSTD_LLcode(litLength); return (LL_bits[llCode] * BITCOST_MULTIPLIER) + (optPtr->litLengthSumBasePrice - WEIGHT(optPtr->litLengthFreq[llCode], optLevel)); } } /* ZSTD_litLengthContribution() : * @return ( cost(litlength) - cost(0) ) * this value can then be added to rawLiteralsCost() * to provide a cost which is directly comparable to a match ending at same position */ static int ZSTD_litLengthContribution(U32 const litLength, const optState_t* const optPtr, int optLevel) { if (optPtr->priceType >= zop_predef) return WEIGHT(litLength, optLevel); /* dynamic statistics */ { U32 const llCode = ZSTD_LLcode(litLength); int const contribution = (LL_bits[llCode] * BITCOST_MULTIPLIER) + WEIGHT(optPtr->litLengthFreq[0], optLevel) /* note: log2litLengthSum cancel out */ - WEIGHT(optPtr->litLengthFreq[llCode], optLevel); #if 1 return contribution; #else return MAX(0, contribution); /* sometimes better, sometimes not ... */ #endif } } /* ZSTD_literalsContribution() : * creates a fake cost for the literals part of a sequence * which can be compared to the ending cost of a match * should a new match start at this position */ static int ZSTD_literalsContribution(const BYTE* const literals, U32 const litLength, const optState_t* const optPtr, int optLevel) { int const contribution = ZSTD_rawLiteralsCost(literals, litLength, optPtr, optLevel) + ZSTD_litLengthContribution(litLength, optPtr, optLevel); return contribution; } /* ZSTD_getMatchPrice() : * Provides the cost of the match part (offset + matchLength) of a sequence * Must be combined with ZSTD_fullLiteralsCost() to get the full cost of a sequence. * optLevel: when <2, favors small offset for decompression speed (improved cache efficiency) */ FORCE_INLINE_TEMPLATE U32 ZSTD_getMatchPrice(U32 const offset, U32 const matchLength, const optState_t* const optPtr, int const optLevel) { U32 price; U32 const offCode = ZSTD_highbit32(offset+1); U32 const mlBase = matchLength - MINMATCH; assert(matchLength >= MINMATCH); if (optPtr->priceType == zop_predef) /* fixed scheme, do not use statistics */ return WEIGHT(mlBase, optLevel) + ((16 + offCode) * BITCOST_MULTIPLIER); /* dynamic statistics */ price = (offCode * BITCOST_MULTIPLIER) + (optPtr->offCodeSumBasePrice - WEIGHT(optPtr->offCodeFreq[offCode], optLevel)); if ((optLevel<2) /*static*/ && offCode >= 20) price += (offCode-19)*2 * BITCOST_MULTIPLIER; /* handicap for long distance offsets, favor decompression speed */ /* match Length */ { U32 const mlCode = ZSTD_MLcode(mlBase); price += (ML_bits[mlCode] * BITCOST_MULTIPLIER) + (optPtr->matchLengthSumBasePrice - WEIGHT(optPtr->matchLengthFreq[mlCode], optLevel)); } price += BITCOST_MULTIPLIER / 5; /* heuristic : make matches a bit more costly to favor less sequences -> faster decompression speed */ DEBUGLOG(8, "ZSTD_getMatchPrice(ml:%u) = %u", matchLength, price); return price; } /* ZSTD_updateStats() : * assumption : literals + litLengtn <= iend */ static void ZSTD_updateStats(optState_t* const optPtr, U32 litLength, const BYTE* literals, U32 offsetCode, U32 matchLength) { /* literals */ { U32 u; for (u=0; u < litLength; u++) optPtr->litFreq[literals[u]] += ZSTD_LITFREQ_ADD; optPtr->litSum += litLength*ZSTD_LITFREQ_ADD; } /* literal Length */ { U32 const llCode = ZSTD_LLcode(litLength); optPtr->litLengthFreq[llCode]++; optPtr->litLengthSum++; } /* match offset code (0-2=>repCode; 3+=>offset+2) */ { U32 const offCode = ZSTD_highbit32(offsetCode+1); assert(offCode <= MaxOff); optPtr->offCodeFreq[offCode]++; optPtr->offCodeSum++; } /* match Length */ { U32 const mlBase = matchLength - MINMATCH; U32 const mlCode = ZSTD_MLcode(mlBase); optPtr->matchLengthFreq[mlCode]++; optPtr->matchLengthSum++; } } /* ZSTD_readMINMATCH() : * function safe only for comparisons * assumption : memPtr must be at least 4 bytes before end of buffer */ MEM_STATIC U32 ZSTD_readMINMATCH(const void* memPtr, U32 length) { switch (length) { default : case 4 : return MEM_read32(memPtr); case 3 : if (MEM_isLittleEndian()) return MEM_read32(memPtr)<<8; else return MEM_read32(memPtr)>>8; } } /* Update hashTable3 up to ip (excluded) Assumption : always within prefix (i.e. not within extDict) */ static U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_matchState_t* ms, const BYTE* const ip) { U32* const hashTable3 = ms->hashTable3; U32 const hashLog3 = ms->hashLog3; const BYTE* const base = ms->window.base; U32 idx = ms->nextToUpdate3; U32 const target = ms->nextToUpdate3 = (U32)(ip - base); size_t const hash3 = ZSTD_hash3Ptr(ip, hashLog3); assert(hashLog3 > 0); while(idx < target) { hashTable3[ZSTD_hash3Ptr(base+idx, hashLog3)] = idx; idx++; } return hashTable3[hash3]; } /*-************************************* * Binary Tree search ***************************************/ /** ZSTD_insertBt1() : add one or multiple positions to tree. * ip : assumed <= iend-8 . * @return : nb of positions added */ static U32 ZSTD_insertBt1( ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iend, U32 const mls, const int extDict) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32* const hashTable = ms->hashTable; U32 const hashLog = cParams->hashLog; size_t const h = ZSTD_hashPtr(ip, hashLog, mls); U32* const bt = ms->chainTable; U32 const btLog = cParams->chainLog - 1; U32 const btMask = (1 << btLog) - 1; U32 matchIndex = hashTable[h]; size_t commonLengthSmaller=0, commonLengthLarger=0; const BYTE* const base = ms->window.base; const BYTE* const dictBase = ms->window.dictBase; const U32 dictLimit = ms->window.dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const BYTE* const prefixStart = base + dictLimit; const BYTE* match; const U32 current = (U32)(ip-base); const U32 btLow = btMask >= current ? 0 : current - btMask; U32* smallerPtr = bt + 2*(current&btMask); U32* largerPtr = smallerPtr + 1; U32 dummy32; /* to be nullified at the end */ U32 const windowLow = ms->window.lowLimit; U32 const matchLow = windowLow ? windowLow : 1; U32 matchEndIdx = current+8+1; size_t bestLength = 8; U32 nbCompares = 1U << cParams->searchLog; #ifdef ZSTD_C_PREDICT U32 predictedSmall = *(bt + 2*((current-1)&btMask) + 0); U32 predictedLarge = *(bt + 2*((current-1)&btMask) + 1); predictedSmall += (predictedSmall>0); predictedLarge += (predictedLarge>0); #endif /* ZSTD_C_PREDICT */ DEBUGLOG(8, "ZSTD_insertBt1 (%u)", current); assert(ip <= iend-8); /* required for h calculation */ hashTable[h] = current; /* Update Hash Table */ while (nbCompares-- && (matchIndex >= matchLow)) { U32* const nextPtr = bt + 2*(matchIndex & btMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ assert(matchIndex < current); #ifdef ZSTD_C_PREDICT /* note : can create issues when hlog small <= 11 */ const U32* predictPtr = bt + 2*((matchIndex-1) & btMask); /* written this way, as bt is a roll buffer */ if (matchIndex == predictedSmall) { /* no need to check length, result known */ *smallerPtr = matchIndex; if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */ smallerPtr = nextPtr+1; /* new "smaller" => larger of match */ matchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */ predictedSmall = predictPtr[1] + (predictPtr[1]>0); continue; } if (matchIndex == predictedLarge) { *largerPtr = matchIndex; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */ largerPtr = nextPtr; matchIndex = nextPtr[0]; predictedLarge = predictPtr[0] + (predictPtr[0]>0); continue; } #endif if (!extDict || (matchIndex+matchLength >= dictLimit)) { assert(matchIndex+matchLength >= dictLimit); /* might be wrong if actually extDict */ match = base + matchIndex; matchLength += ZSTD_count(ip+matchLength, match+matchLength, iend); } else { match = dictBase + matchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iend, dictEnd, prefixStart); if (matchIndex+matchLength >= dictLimit) match = base + matchIndex; /* to prepare for next usage of match[matchLength] */ } if (matchLength > bestLength) { bestLength = matchLength; if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength; } if (ip+matchLength == iend) { /* equal : no way to know if inf or sup */ break; /* drop , to guarantee consistency ; miss a bit of compression, but other solutions can corrupt tree */ } if (match[matchLength] < ip[matchLength]) { /* necessarily within buffer */ /* match is smaller than current */ *smallerPtr = matchIndex; /* update smaller idx */ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop searching */ smallerPtr = nextPtr+1; /* new "candidate" => larger than match, which was smaller than target */ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous and closer to current */ } else { /* match is larger than current */ *largerPtr = matchIndex; commonLengthLarger = matchLength; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop searching */ largerPtr = nextPtr; matchIndex = nextPtr[0]; } } *smallerPtr = *largerPtr = 0; if (bestLength > 384) return MIN(192, (U32)(bestLength - 384)); /* speed optimization */ assert(matchEndIdx > current + 8); return matchEndIdx - (current + 8); } FORCE_INLINE_TEMPLATE void ZSTD_updateTree_internal( ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iend, const U32 mls, const ZSTD_dictMode_e dictMode) { const BYTE* const base = ms->window.base; U32 const target = (U32)(ip - base); U32 idx = ms->nextToUpdate; DEBUGLOG(5, "ZSTD_updateTree_internal, from %u to %u (dictMode:%u)", idx, target, dictMode); while(idx < target) idx += ZSTD_insertBt1(ms, base+idx, iend, mls, dictMode == ZSTD_extDict); ms->nextToUpdate = target; } void ZSTD_updateTree(ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* iend) { ZSTD_updateTree_internal(ms, ip, iend, ms->cParams.searchLength, ZSTD_noDict); } FORCE_INLINE_TEMPLATE U32 ZSTD_insertBtAndGetAllMatches ( ZSTD_matchState_t* ms, const BYTE* const ip, const BYTE* const iLimit, const ZSTD_dictMode_e dictMode, U32 rep[ZSTD_REP_NUM], U32 const ll0, ZSTD_match_t* matches, const U32 lengthToBeat, U32 const mls /* template */) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1); const BYTE* const base = ms->window.base; U32 const current = (U32)(ip-base); U32 const hashLog = cParams->hashLog; U32 const minMatch = (mls==3) ? 3 : 4; U32* const hashTable = ms->hashTable; size_t const h = ZSTD_hashPtr(ip, hashLog, mls); U32 matchIndex = hashTable[h]; U32* const bt = ms->chainTable; U32 const btLog = cParams->chainLog - 1; U32 const btMask= (1U << btLog) - 1; size_t commonLengthSmaller=0, commonLengthLarger=0; const BYTE* const dictBase = ms->window.dictBase; U32 const dictLimit = ms->window.dictLimit; const BYTE* const dictEnd = dictBase + dictLimit; const BYTE* const prefixStart = base + dictLimit; U32 const btLow = btMask >= current ? 0 : current - btMask; U32 const windowLow = ms->window.lowLimit; U32 const matchLow = windowLow ? windowLow : 1; U32* smallerPtr = bt + 2*(current&btMask); U32* largerPtr = bt + 2*(current&btMask) + 1; U32 matchEndIdx = current+8+1; /* farthest referenced position of any match => detects repetitive patterns */ U32 dummy32; /* to be nullified at the end */ U32 mnum = 0; U32 nbCompares = 1U << cParams->searchLog; const ZSTD_matchState_t* dms = dictMode == ZSTD_dictMatchState ? ms->dictMatchState : NULL; const ZSTD_compressionParameters* const dmsCParams = dictMode == ZSTD_dictMatchState ? &dms->cParams : NULL; const BYTE* const dmsBase = dictMode == ZSTD_dictMatchState ? dms->window.base : NULL; const BYTE* const dmsEnd = dictMode == ZSTD_dictMatchState ? dms->window.nextSrc : NULL; U32 const dmsHighLimit = dictMode == ZSTD_dictMatchState ? (U32)(dmsEnd - dmsBase) : 0; U32 const dmsLowLimit = dictMode == ZSTD_dictMatchState ? dms->window.lowLimit : 0; U32 const dmsIndexDelta = dictMode == ZSTD_dictMatchState ? windowLow - dmsHighLimit : 0; U32 const dmsHashLog = dictMode == ZSTD_dictMatchState ? dmsCParams->hashLog : hashLog; U32 const dmsBtLog = dictMode == ZSTD_dictMatchState ? dmsCParams->chainLog - 1 : btLog; U32 const dmsBtMask = dictMode == ZSTD_dictMatchState ? (1U << dmsBtLog) - 1 : 0; U32 const dmsBtLow = dictMode == ZSTD_dictMatchState && dmsBtMask < dmsHighLimit - dmsLowLimit ? dmsHighLimit - dmsBtMask : dmsLowLimit; size_t bestLength = lengthToBeat-1; DEBUGLOG(8, "ZSTD_insertBtAndGetAllMatches: current=%u", current); /* check repCode */ { U32 const lastR = ZSTD_REP_NUM + ll0; U32 repCode; for (repCode = ll0; repCode < lastR; repCode++) { U32 const repOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode]; U32 const repIndex = current - repOffset; U32 repLen = 0; assert(current >= dictLimit); if (repOffset-1 /* intentional overflow, discards 0 and -1 */ < current-dictLimit) { /* equivalent to `current > repIndex >= dictLimit` */ if (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(ip - repOffset, minMatch)) { repLen = (U32)ZSTD_count(ip+minMatch, ip+minMatch-repOffset, iLimit) + minMatch; } } else { /* repIndex < dictLimit || repIndex >= current */ const BYTE* const repMatch = dictMode == ZSTD_dictMatchState ? dmsBase + repIndex - dmsIndexDelta : dictBase + repIndex; assert(current >= windowLow); if ( dictMode == ZSTD_extDict && ( ((repOffset-1) /*intentional overflow*/ < current - windowLow) /* equivalent to `current > repIndex >= windowLow` */ & (((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */) && (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) { repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dictEnd, prefixStart) + minMatch; } if (dictMode == ZSTD_dictMatchState && ( ((repOffset-1) /*intentional overflow*/ < current - (dmsLowLimit + dmsIndexDelta)) /* equivalent to `current > repIndex >= dmsLowLimit` */ & ((U32)((dictLimit-1) - repIndex) >= 3) ) /* intentional overflow : do not test positions overlapping 2 memory segments */ && (ZSTD_readMINMATCH(ip, minMatch) == ZSTD_readMINMATCH(repMatch, minMatch)) ) { repLen = (U32)ZSTD_count_2segments(ip+minMatch, repMatch+minMatch, iLimit, dmsEnd, prefixStart) + minMatch; } } /* save longer solution */ if (repLen > bestLength) { DEBUGLOG(8, "found repCode %u (ll0:%u, offset:%u) of length %u", repCode, ll0, repOffset, repLen); bestLength = repLen; matches[mnum].off = repCode - ll0; matches[mnum].len = (U32)repLen; mnum++; if ( (repLen > sufficient_len) | (ip+repLen == iLimit) ) { /* best possible */ return mnum; } } } } /* HC3 match finder */ if ((mls == 3) /*static*/ && (bestLength < mls)) { U32 const matchIndex3 = ZSTD_insertAndFindFirstIndexHash3(ms, ip); if ((matchIndex3 >= matchLow) & (current - matchIndex3 < (1<<18)) /*heuristic : longer distance likely too expensive*/ ) { size_t mlen; if ((dictMode == ZSTD_noDict) /*static*/ || (dictMode == ZSTD_dictMatchState) /*static*/ || (matchIndex3 >= dictLimit)) { const BYTE* const match = base + matchIndex3; mlen = ZSTD_count(ip, match, iLimit); } else { const BYTE* const match = dictBase + matchIndex3; mlen = ZSTD_count_2segments(ip, match, iLimit, dictEnd, prefixStart); } /* save best solution */ if (mlen >= mls /* == 3 > bestLength */) { DEBUGLOG(8, "found small match with hlog3, of length %u", (U32)mlen); bestLength = mlen; assert(current > matchIndex3); assert(mnum==0); /* no prior solution */ matches[0].off = (current - matchIndex3) + ZSTD_REP_MOVE; matches[0].len = (U32)mlen; mnum = 1; if ( (mlen > sufficient_len) | (ip+mlen == iLimit) ) { /* best possible length */ ms->nextToUpdate = current+1; /* skip insertion */ return 1; } } } /* no dictMatchState lookup: dicts don't have a populated HC3 table */ } hashTable[h] = current; /* Update Hash Table */ while (nbCompares-- && (matchIndex >= matchLow)) { U32* const nextPtr = bt + 2*(matchIndex & btMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ const BYTE* match; assert(current > matchIndex); if ((dictMode == ZSTD_noDict) || (dictMode == ZSTD_dictMatchState) || (matchIndex+matchLength >= dictLimit)) { assert(matchIndex+matchLength >= dictLimit); /* ensure the condition is correct when !extDict */ match = base + matchIndex; matchLength += ZSTD_count(ip+matchLength, match+matchLength, iLimit); } else { match = dictBase + matchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dictEnd, prefixStart); if (matchIndex+matchLength >= dictLimit) match = base + matchIndex; /* prepare for match[matchLength] */ } if (matchLength > bestLength) { DEBUGLOG(8, "found match of length %u at distance %u (offCode=%u)", (U32)matchLength, current - matchIndex, current - matchIndex + ZSTD_REP_MOVE); assert(matchEndIdx > matchIndex); if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength; bestLength = matchLength; matches[mnum].off = (current - matchIndex) + ZSTD_REP_MOVE; matches[mnum].len = (U32)matchLength; mnum++; if ( (matchLength > ZSTD_OPT_NUM) | (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) { if (dictMode == ZSTD_dictMatchState) nbCompares = 0; /* break should also skip searching dms */ break; /* drop, to preserve bt consistency (miss a little bit of compression) */ } } if (match[matchLength] < ip[matchLength]) { /* match smaller than current */ *smallerPtr = matchIndex; /* update smaller idx */ commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ if (matchIndex <= btLow) { smallerPtr=&dummy32; break; } /* beyond tree size, stop the search */ smallerPtr = nextPtr+1; /* new candidate => larger than match, which was smaller than current */ matchIndex = nextPtr[1]; /* new matchIndex, larger than previous, closer to current */ } else { *largerPtr = matchIndex; commonLengthLarger = matchLength; if (matchIndex <= btLow) { largerPtr=&dummy32; break; } /* beyond tree size, stop the search */ largerPtr = nextPtr; matchIndex = nextPtr[0]; } } *smallerPtr = *largerPtr = 0; if (dictMode == ZSTD_dictMatchState && nbCompares) { size_t const dmsH = ZSTD_hashPtr(ip, dmsHashLog, mls); U32 dictMatchIndex = dms->hashTable[dmsH]; const U32* const dmsBt = dms->chainTable; commonLengthSmaller = commonLengthLarger = 0; while (nbCompares-- && (dictMatchIndex > dmsLowLimit)) { const U32* const nextPtr = dmsBt + 2*(dictMatchIndex & dmsBtMask); size_t matchLength = MIN(commonLengthSmaller, commonLengthLarger); /* guaranteed minimum nb of common bytes */ const BYTE* match = dmsBase + dictMatchIndex; matchLength += ZSTD_count_2segments(ip+matchLength, match+matchLength, iLimit, dmsEnd, prefixStart); if (dictMatchIndex+matchLength >= dmsHighLimit) match = base + dictMatchIndex + dmsIndexDelta; /* to prepare for next usage of match[matchLength] */ if (matchLength > bestLength) { matchIndex = dictMatchIndex + dmsIndexDelta; DEBUGLOG(8, "found dms match of length %u at distance %u (offCode=%u)", (U32)matchLength, current - matchIndex, current - matchIndex + ZSTD_REP_MOVE); if (matchLength > matchEndIdx - matchIndex) matchEndIdx = matchIndex + (U32)matchLength; bestLength = matchLength; matches[mnum].off = (current - matchIndex) + ZSTD_REP_MOVE; matches[mnum].len = (U32)matchLength; mnum++; if ( (matchLength > ZSTD_OPT_NUM) | (ip+matchLength == iLimit) /* equal : no way to know if inf or sup */) { break; /* drop, to guarantee consistency (miss a little bit of compression) */ } } if (dictMatchIndex <= dmsBtLow) { break; } /* beyond tree size, stop the search */ if (match[matchLength] < ip[matchLength]) { commonLengthSmaller = matchLength; /* all smaller will now have at least this guaranteed common length */ dictMatchIndex = nextPtr[1]; /* new matchIndex larger than previous (closer to current) */ } else { /* match is larger than current */ commonLengthLarger = matchLength; dictMatchIndex = nextPtr[0]; } } } assert(matchEndIdx > current+8); ms->nextToUpdate = matchEndIdx - 8; /* skip repetitive patterns */ return mnum; } FORCE_INLINE_TEMPLATE U32 ZSTD_BtGetAllMatches ( ZSTD_matchState_t* ms, const BYTE* ip, const BYTE* const iHighLimit, const ZSTD_dictMode_e dictMode, U32 rep[ZSTD_REP_NUM], U32 const ll0, ZSTD_match_t* matches, U32 const lengthToBeat) { const ZSTD_compressionParameters* const cParams = &ms->cParams; U32 const matchLengthSearch = cParams->searchLength; DEBUGLOG(8, "ZSTD_BtGetAllMatches"); if (ip < ms->window.base + ms->nextToUpdate) return 0; /* skipped area */ ZSTD_updateTree_internal(ms, ip, iHighLimit, matchLengthSearch, dictMode); switch(matchLengthSearch) { case 3 : return ZSTD_insertBtAndGetAllMatches(ms, ip, iHighLimit, dictMode, rep, ll0, matches, lengthToBeat, 3); default : case 4 : return ZSTD_insertBtAndGetAllMatches(ms, ip, iHighLimit, dictMode, rep, ll0, matches, lengthToBeat, 4); case 5 : return ZSTD_insertBtAndGetAllMatches(ms, ip, iHighLimit, dictMode, rep, ll0, matches, lengthToBeat, 5); case 7 : case 6 : return ZSTD_insertBtAndGetAllMatches(ms, ip, iHighLimit, dictMode, rep, ll0, matches, lengthToBeat, 6); } } /*-******************************* * Optimal parser *********************************/ typedef struct repcodes_s { U32 rep[3]; } repcodes_t; static repcodes_t ZSTD_updateRep(U32 const rep[3], U32 const offset, U32 const ll0) { repcodes_t newReps; if (offset >= ZSTD_REP_NUM) { /* full offset */ newReps.rep[2] = rep[1]; newReps.rep[1] = rep[0]; newReps.rep[0] = offset - ZSTD_REP_MOVE; } else { /* repcode */ U32 const repCode = offset + ll0; if (repCode > 0) { /* note : if repCode==0, no change */ U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode]; newReps.rep[2] = (repCode >= 2) ? rep[1] : rep[2]; newReps.rep[1] = rep[0]; newReps.rep[0] = currentOffset; } else { /* repCode == 0 */ memcpy(&newReps, rep, sizeof(newReps)); } } return newReps; } static U32 ZSTD_totalLen(ZSTD_optimal_t sol) { return sol.litlen + sol.mlen; } FORCE_INLINE_TEMPLATE size_t ZSTD_compressBlock_opt_generic(ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize, const int optLevel, const ZSTD_dictMode_e dictMode) { optState_t* const optStatePtr = &ms->opt; const BYTE* const istart = (const BYTE*)src; const BYTE* ip = istart; const BYTE* anchor = istart; const BYTE* const iend = istart + srcSize; const BYTE* const ilimit = iend - 8; const BYTE* const base = ms->window.base; const BYTE* const prefixStart = base + ms->window.dictLimit; const ZSTD_compressionParameters* const cParams = &ms->cParams; U32 const sufficient_len = MIN(cParams->targetLength, ZSTD_OPT_NUM -1); U32 const minMatch = (cParams->searchLength == 3) ? 3 : 4; ZSTD_optimal_t* const opt = optStatePtr->priceTable; ZSTD_match_t* const matches = optStatePtr->matchTable; ZSTD_optimal_t lastSequence; /* init */ DEBUGLOG(5, "ZSTD_compressBlock_opt_generic"); assert(optLevel <= 2); ms->nextToUpdate3 = ms->nextToUpdate; ZSTD_rescaleFreqs(optStatePtr, (const BYTE*)src, srcSize, optLevel); ip += (ip==prefixStart); /* Match Loop */ while (ip < ilimit) { U32 cur, last_pos = 0; /* find first match */ { U32 const litlen = (U32)(ip - anchor); U32 const ll0 = !litlen; U32 const nbMatches = ZSTD_BtGetAllMatches(ms, ip, iend, dictMode, rep, ll0, matches, minMatch); if (!nbMatches) { ip++; continue; } /* initialize opt[0] */ { U32 i ; for (i=0; i immediate encoding */ { U32 const maxML = matches[nbMatches-1].len; U32 const maxOffset = matches[nbMatches-1].off; DEBUGLOG(6, "found %u matches of maxLength=%u and maxOffCode=%u at cPos=%u => start new serie", nbMatches, maxML, maxOffset, (U32)(ip-prefixStart)); if (maxML > sufficient_len) { lastSequence.litlen = litlen; lastSequence.mlen = maxML; lastSequence.off = maxOffset; DEBUGLOG(6, "large match (%u>%u), immediate encoding", maxML, sufficient_len); cur = 0; last_pos = ZSTD_totalLen(lastSequence); goto _shortestPath; } } /* set prices for first matches starting position == 0 */ { U32 const literalsPrice = opt[0].price + ZSTD_litLengthPrice(0, optStatePtr, optLevel); U32 pos; U32 matchNb; for (pos = 1; pos < minMatch; pos++) { opt[pos].price = ZSTD_MAX_PRICE; /* mlen, litlen and price will be fixed during forward scanning */ } for (matchNb = 0; matchNb < nbMatches; matchNb++) { U32 const offset = matches[matchNb].off; U32 const end = matches[matchNb].len; repcodes_t const repHistory = ZSTD_updateRep(rep, offset, ll0); for ( ; pos <= end ; pos++ ) { U32 const matchPrice = ZSTD_getMatchPrice(offset, pos, optStatePtr, optLevel); U32 const sequencePrice = literalsPrice + matchPrice; +#if 0 DEBUGLOG(7, "rPos:%u => set initial price : %.2f", pos, ZSTD_fCost(sequencePrice)); +#endif opt[pos].mlen = pos; opt[pos].off = offset; opt[pos].litlen = litlen; opt[pos].price = sequencePrice; ZSTD_STATIC_ASSERT(sizeof(opt[pos].rep) == sizeof(repHistory)); memcpy(opt[pos].rep, &repHistory, sizeof(repHistory)); } } last_pos = pos-1; } } /* check further positions */ for (cur = 1; cur <= last_pos; cur++) { const BYTE* const inr = ip + cur; assert(cur < ZSTD_OPT_NUM); DEBUGLOG(7, "cPos:%zi==rPos:%u", inr-istart, cur) /* Fix current position with one literal if cheaper */ { U32 const litlen = (opt[cur-1].mlen == 0) ? opt[cur-1].litlen + 1 : 1; int const price = opt[cur-1].price + ZSTD_rawLiteralsCost(ip+cur-1, 1, optStatePtr, optLevel) + ZSTD_litLengthPrice(litlen, optStatePtr, optLevel) - ZSTD_litLengthPrice(litlen-1, optStatePtr, optLevel); assert(price < 1000000000); /* overflow check */ if (price <= opt[cur].price) { +#if 0 DEBUGLOG(7, "cPos:%zi==rPos:%u : better price (%.2f<=%.2f) using literal (ll==%u) (hist:%u,%u,%u)", inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price), litlen, opt[cur-1].rep[0], opt[cur-1].rep[1], opt[cur-1].rep[2]); +#endif opt[cur].mlen = 0; opt[cur].off = 0; opt[cur].litlen = litlen; opt[cur].price = price; memcpy(opt[cur].rep, opt[cur-1].rep, sizeof(opt[cur].rep)); } else { +#if 0 DEBUGLOG(7, "cPos:%zi==rPos:%u : literal would cost more (%.2f>%.2f) (hist:%u,%u,%u)", inr-istart, cur, ZSTD_fCost(price), ZSTD_fCost(opt[cur].price), opt[cur].rep[0], opt[cur].rep[1], opt[cur].rep[2]); +#endif } } /* last match must start at a minimum distance of 8 from oend */ if (inr > ilimit) continue; if (cur == last_pos) break; if ( (optLevel==0) /*static_test*/ && (opt[cur+1].price <= opt[cur].price + (BITCOST_MULTIPLIER/2)) ) { DEBUGLOG(7, "move to next rPos:%u : price is <=", cur+1); continue; /* skip unpromising positions; about ~+6% speed, -0.01 ratio */ } { U32 const ll0 = (opt[cur].mlen != 0); U32 const litlen = (opt[cur].mlen == 0) ? opt[cur].litlen : 0; U32 const previousPrice = opt[cur].price; U32 const basePrice = previousPrice + ZSTD_litLengthPrice(0, optStatePtr, optLevel); U32 const nbMatches = ZSTD_BtGetAllMatches(ms, inr, iend, dictMode, opt[cur].rep, ll0, matches, minMatch); U32 matchNb; if (!nbMatches) { DEBUGLOG(7, "rPos:%u : no match found", cur); continue; } { U32 const maxML = matches[nbMatches-1].len; DEBUGLOG(7, "cPos:%zi==rPos:%u, found %u matches, of maxLength=%u", inr-istart, cur, nbMatches, maxML); if ( (maxML > sufficient_len) || (cur + maxML >= ZSTD_OPT_NUM) ) { lastSequence.mlen = maxML; lastSequence.off = matches[nbMatches-1].off; lastSequence.litlen = litlen; cur -= (opt[cur].mlen==0) ? opt[cur].litlen : 0; /* last sequence is actually only literals, fix cur to last match - note : may underflow, in which case, it's first sequence, and it's okay */ last_pos = cur + ZSTD_totalLen(lastSequence); if (cur > ZSTD_OPT_NUM) cur = 0; /* underflow => first match */ goto _shortestPath; } } /* set prices using matches found at position == cur */ for (matchNb = 0; matchNb < nbMatches; matchNb++) { U32 const offset = matches[matchNb].off; repcodes_t const repHistory = ZSTD_updateRep(opt[cur].rep, offset, ll0); U32 const lastML = matches[matchNb].len; U32 const startML = (matchNb>0) ? matches[matchNb-1].len+1 : minMatch; U32 mlen; DEBUGLOG(7, "testing match %u => offCode=%4u, mlen=%2u, llen=%2u", matchNb, matches[matchNb].off, lastML, litlen); for (mlen = lastML; mlen >= startML; mlen--) { /* scan downward */ U32 const pos = cur + mlen; int const price = basePrice + ZSTD_getMatchPrice(offset, mlen, optStatePtr, optLevel); if ((pos > last_pos) || (price < opt[pos].price)) { +#if 0 DEBUGLOG(7, "rPos:%u (ml=%2u) => new better price (%.2f<%.2f)", pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price)); +#endif while (last_pos < pos) { opt[last_pos+1].price = ZSTD_MAX_PRICE; last_pos++; } /* fill empty positions */ opt[pos].mlen = mlen; opt[pos].off = offset; opt[pos].litlen = litlen; opt[pos].price = price; ZSTD_STATIC_ASSERT(sizeof(opt[pos].rep) == sizeof(repHistory)); memcpy(opt[pos].rep, &repHistory, sizeof(repHistory)); } else { +#if 0 DEBUGLOG(7, "rPos:%u (ml=%2u) => new price is worse (%.2f>=%.2f)", pos, mlen, ZSTD_fCost(price), ZSTD_fCost(opt[pos].price)); +#endif if (optLevel==0) break; /* early update abort; gets ~+10% speed for about -0.01 ratio loss */ } } } } } /* for (cur = 1; cur <= last_pos; cur++) */ lastSequence = opt[last_pos]; cur = last_pos > ZSTD_totalLen(lastSequence) ? last_pos - ZSTD_totalLen(lastSequence) : 0; /* single sequence, and it starts before `ip` */ assert(cur < ZSTD_OPT_NUM); /* control overflow*/ _shortestPath: /* cur, last_pos, best_mlen, best_off have to be set */ assert(opt[0].mlen == 0); { U32 const storeEnd = cur + 1; U32 storeStart = storeEnd; U32 seqPos = cur; DEBUGLOG(6, "start reverse traversal (last_pos:%u, cur:%u)", last_pos, cur); (void)last_pos; assert(storeEnd < ZSTD_OPT_NUM); DEBUGLOG(6, "last sequence copied into pos=%u (llen=%u,mlen=%u,ofc=%u)", storeEnd, lastSequence.litlen, lastSequence.mlen, lastSequence.off); opt[storeEnd] = lastSequence; while (seqPos > 0) { U32 const backDist = ZSTD_totalLen(opt[seqPos]); storeStart--; DEBUGLOG(6, "sequence from rPos=%u copied into pos=%u (llen=%u,mlen=%u,ofc=%u)", seqPos, storeStart, opt[seqPos].litlen, opt[seqPos].mlen, opt[seqPos].off); opt[storeStart] = opt[seqPos]; seqPos = (seqPos > backDist) ? seqPos - backDist : 0; } /* save sequences */ DEBUGLOG(6, "sending selected sequences into seqStore") { U32 storePos; for (storePos=storeStart; storePos <= storeEnd; storePos++) { U32 const llen = opt[storePos].litlen; U32 const mlen = opt[storePos].mlen; U32 const offCode = opt[storePos].off; U32 const advance = llen + mlen; DEBUGLOG(6, "considering seq starting at %zi, llen=%u, mlen=%u", anchor - istart, llen, mlen); if (mlen==0) { /* only literals => must be last "sequence", actually starting a new stream of sequences */ assert(storePos == storeEnd); /* must be last sequence */ ip = anchor + llen; /* last "sequence" is a bunch of literals => don't progress anchor */ continue; /* will finish */ } /* repcodes update : like ZSTD_updateRep(), but update in place */ if (offCode >= ZSTD_REP_NUM) { /* full offset */ rep[2] = rep[1]; rep[1] = rep[0]; rep[0] = offCode - ZSTD_REP_MOVE; } else { /* repcode */ U32 const repCode = offCode + (llen==0); if (repCode) { /* note : if repCode==0, no change */ U32 const currentOffset = (repCode==ZSTD_REP_NUM) ? (rep[0] - 1) : rep[repCode]; if (repCode >= 2) rep[2] = rep[1]; rep[1] = rep[0]; rep[0] = currentOffset; } } assert(anchor + llen <= iend); ZSTD_updateStats(optStatePtr, llen, anchor, offCode, mlen); ZSTD_storeSeq(seqStore, llen, anchor, offCode, mlen-MINMATCH); anchor += advance; ip = anchor; } } ZSTD_setBasePrices(optStatePtr, optLevel); } } /* while (ip < ilimit) */ /* Return the last literals size */ return iend - anchor; } size_t ZSTD_compressBlock_btopt( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_compressBlock_btopt"); return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_noDict); } /* used in 2-pass strategy */ static U32 ZSTD_upscaleStat(U32* table, U32 lastEltIndex, int bonus) { U32 s, sum=0; assert(ZSTD_FREQ_DIV+bonus > 0); for (s=0; s<=lastEltIndex; s++) { table[s] <<= ZSTD_FREQ_DIV+bonus; table[s]--; sum += table[s]; } return sum; } /* used in 2-pass strategy */ MEM_STATIC void ZSTD_upscaleStats(optState_t* optPtr) { optPtr->litSum = ZSTD_upscaleStat(optPtr->litFreq, MaxLit, 0); optPtr->litLengthSum = ZSTD_upscaleStat(optPtr->litLengthFreq, MaxLL, 1); optPtr->matchLengthSum = ZSTD_upscaleStat(optPtr->matchLengthFreq, MaxML, 1); optPtr->offCodeSum = ZSTD_upscaleStat(optPtr->offCodeFreq, MaxOff, 1); } size_t ZSTD_compressBlock_btultra( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { DEBUGLOG(5, "ZSTD_compressBlock_btultra (srcSize=%zu)", srcSize); #if 0 /* 2-pass strategy (disabled) * this strategy makes a first pass over first block to collect statistics * and seed next round's statistics with it. * The compression ratio gain is generally small (~0.5% on first block), * the cost is 2x cpu time on first block. */ assert(srcSize <= ZSTD_BLOCKSIZE_MAX); if ( (ms->opt.litLengthSum==0) /* first block */ && (seqStore->sequences == seqStore->sequencesStart) /* no ldm */ && (ms->window.dictLimit == ms->window.lowLimit) ) { /* no dictionary */ U32 tmpRep[ZSTD_REP_NUM]; DEBUGLOG(5, "ZSTD_compressBlock_btultra: first block: collecting statistics"); assert(ms->nextToUpdate >= ms->window.dictLimit && ms->nextToUpdate <= ms->window.dictLimit + 1); memcpy(tmpRep, rep, sizeof(tmpRep)); ZSTD_compressBlock_opt_generic(ms, seqStore, tmpRep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict); /* generate stats into ms->opt*/ ZSTD_resetSeqStore(seqStore); /* invalidate first scan from history */ ms->window.base -= srcSize; ms->window.dictLimit += (U32)srcSize; ms->window.lowLimit = ms->window.dictLimit; ms->nextToUpdate = ms->window.dictLimit; ms->nextToUpdate3 = ms->window.dictLimit; /* re-inforce weight of collected statistics */ ZSTD_upscaleStats(&ms->opt); } #endif return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_noDict); } size_t ZSTD_compressBlock_btopt_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_btultra_dictMatchState( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_dictMatchState); } size_t ZSTD_compressBlock_btopt_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 0 /*optLevel*/, ZSTD_extDict); } size_t ZSTD_compressBlock_btultra_extDict( ZSTD_matchState_t* ms, seqStore_t* seqStore, U32 rep[ZSTD_REP_NUM], const void* src, size_t srcSize) { return ZSTD_compressBlock_opt_generic(ms, seqStore, rep, src, srcSize, 2 /*optLevel*/, ZSTD_extDict); } Index: head/sys/dev/efidev/efirt.c =================================================================== --- head/sys/dev/efidev/efirt.c (revision 342107) +++ head/sys/dev/efidev/efirt.c (revision 342108) @@ -1,544 +1,580 @@ /*- * Copyright (c) 2004 Marcel Moolenaar * Copyright (c) 2001 Doug Rabson * Copyright (c) 2016, 2018 The FreeBSD Foundation * All rights reserved. * * Portions of this software were developed by Konstantin Belousov * under sponsorship from the FreeBSD Foundation. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include +#include #include #include #include #include #include #include #include +#include #include #include #include #include #include #include #include #include #include #include #include #include static struct efi_systbl *efi_systbl; +static eventhandler_tag efi_shutdown_tag; /* * The following pointers point to tables in the EFI runtime service data pages. * Care should be taken to make sure that we've properly entered the EFI runtime * environment (efi_enter()) before dereferencing them. */ static struct efi_cfgtbl *efi_cfgtbl; static struct efi_rt *efi_runtime; static int efi_status2err[25] = { 0, /* EFI_SUCCESS */ ENOEXEC, /* EFI_LOAD_ERROR */ EINVAL, /* EFI_INVALID_PARAMETER */ ENOSYS, /* EFI_UNSUPPORTED */ EMSGSIZE, /* EFI_BAD_BUFFER_SIZE */ EOVERFLOW, /* EFI_BUFFER_TOO_SMALL */ EBUSY, /* EFI_NOT_READY */ EIO, /* EFI_DEVICE_ERROR */ EROFS, /* EFI_WRITE_PROTECTED */ EAGAIN, /* EFI_OUT_OF_RESOURCES */ EIO, /* EFI_VOLUME_CORRUPTED */ ENOSPC, /* EFI_VOLUME_FULL */ ENXIO, /* EFI_NO_MEDIA */ ESTALE, /* EFI_MEDIA_CHANGED */ ENOENT, /* EFI_NOT_FOUND */ EACCES, /* EFI_ACCESS_DENIED */ ETIMEDOUT, /* EFI_NO_RESPONSE */ EADDRNOTAVAIL, /* EFI_NO_MAPPING */ ETIMEDOUT, /* EFI_TIMEOUT */ EDOOFUS, /* EFI_NOT_STARTED */ EALREADY, /* EFI_ALREADY_STARTED */ ECANCELED, /* EFI_ABORTED */ EPROTO, /* EFI_ICMP_ERROR */ EPROTO, /* EFI_TFTP_ERROR */ EPROTO /* EFI_PROTOCOL_ERROR */ }; static int efi_enter(void); static void efi_leave(void); static int efi_status_to_errno(efi_status status) { u_long code; code = status & 0x3ffffffffffffffful; return (code < nitems(efi_status2err) ? efi_status2err[code] : EDOOFUS); } static struct mtx efi_lock; +static SYSCTL_NODE(_hw, OID_AUTO, efi, CTLFLAG_RWTUN, NULL, "EFI"); +static bool efi_poweroff = true; +SYSCTL_BOOL(_hw_efi, OID_AUTO, poweroff, CTLFLAG_RWTUN, &efi_poweroff, 0, + "If true, use EFI runtime services to power off in preference to ACPI"); static bool efi_is_in_map(struct efi_md *map, int ndesc, int descsz, vm_offset_t addr) { struct efi_md *p; int i; for (i = 0, p = map; i < ndesc; i++, p = efi_next_descriptor(p, descsz)) { if ((p->md_attr & EFI_MD_ATTR_RT) == 0) continue; if (addr >= (uintptr_t)p->md_virt && addr < (uintptr_t)p->md_virt + p->md_pages * PAGE_SIZE) return (true); } return (false); } +static void +efi_shutdown_final(void *dummy __unused, int howto) +{ + + /* + * On some systems, ACPI S5 is missing or does not function properly. + * When present, shutdown via EFI Runtime Services instead, unless + * disabled. + */ + if ((howto & RB_POWEROFF) != 0 && efi_poweroff) + (void)efi_reset_system(EFI_RESET_SHUTDOWN); +} + static int efi_init(void) { struct efi_map_header *efihdr; struct efi_md *map; struct efi_rt *rtdm; caddr_t kmdp; size_t efisz; int ndesc, rt_disabled; rt_disabled = 0; TUNABLE_INT_FETCH("efi.rt.disabled", &rt_disabled); if (rt_disabled == 1) return (0); mtx_init(&efi_lock, "efi", NULL, MTX_DEF); if (efi_systbl_phys == 0) { if (bootverbose) printf("EFI systbl not available\n"); return (0); } efi_systbl = (struct efi_systbl *)efi_phys_to_kva(efi_systbl_phys); if (efi_systbl == NULL || efi_systbl->st_hdr.th_sig != EFI_SYSTBL_SIG) { efi_systbl = NULL; if (bootverbose) printf("EFI systbl signature invalid\n"); return (0); } efi_cfgtbl = (efi_systbl->st_cfgtbl == 0) ? NULL : (struct efi_cfgtbl *)efi_systbl->st_cfgtbl; if (efi_cfgtbl == NULL) { if (bootverbose) printf("EFI config table is not present\n"); } kmdp = preload_search_by_type("elf kernel"); if (kmdp == NULL) kmdp = preload_search_by_type("elf64 kernel"); efihdr = (struct efi_map_header *)preload_search_info(kmdp, MODINFO_METADATA | MODINFOMD_EFI_MAP); if (efihdr == NULL) { if (bootverbose) printf("EFI map is not present\n"); return (0); } efisz = (sizeof(struct efi_map_header) + 0xf) & ~0xf; map = (struct efi_md *)((uint8_t *)efihdr + efisz); if (efihdr->descriptor_size == 0) return (ENOMEM); ndesc = efihdr->memory_size / efihdr->descriptor_size; if (!efi_create_1t1_map(map, ndesc, efihdr->descriptor_size)) { if (bootverbose) printf("EFI cannot create runtime map\n"); return (ENOMEM); } efi_runtime = (efi_systbl->st_rt == 0) ? NULL : (struct efi_rt *)efi_systbl->st_rt; if (efi_runtime == NULL) { if (bootverbose) printf("EFI runtime services table is not present\n"); efi_destroy_1t1_map(); return (ENXIO); } #if defined(__aarch64__) || defined(__amd64__) /* * Some UEFI implementations have multiple implementations of the * RS->GetTime function. They switch from one we can only use early * in the boot process to one valid as a RunTime service only when we * call RS->SetVirtualAddressMap. As this is not always the case, e.g. * with an old loader.efi, check if the RS->GetTime function is within * the EFI map, and fail to attach if not. */ rtdm = (struct efi_rt *)efi_phys_to_kva((uintptr_t)efi_runtime); if (rtdm == NULL || !efi_is_in_map(map, ndesc, efihdr->descriptor_size, (vm_offset_t)rtdm->rt_gettime)) { if (bootverbose) printf( "EFI runtime services table has an invalid pointer\n"); efi_runtime = NULL; efi_destroy_1t1_map(); return (ENXIO); } #endif + /* + * We use SHUTDOWN_PRI_LAST - 1 to trigger after IPMI, but before ACPI. + */ + efi_shutdown_tag = EVENTHANDLER_REGISTER(shutdown_final, + efi_shutdown_final, NULL, SHUTDOWN_PRI_LAST - 1); + return (0); } static void efi_uninit(void) { /* Most likely disabled by tunable */ if (efi_runtime == NULL) return; + if (efi_shutdown_tag != NULL) + EVENTHANDLER_DEREGISTER(shutdown_final, efi_shutdown_tag); efi_destroy_1t1_map(); efi_systbl = NULL; efi_cfgtbl = NULL; efi_runtime = NULL; mtx_destroy(&efi_lock); } int efi_rt_ok(void) { if (efi_runtime == NULL) return (ENXIO); return (0); } static int efi_enter(void) { struct thread *td; pmap_t curpmap; if (efi_runtime == NULL) return (ENXIO); td = curthread; curpmap = &td->td_proc->p_vmspace->vm_pmap; PMAP_LOCK(curpmap); mtx_lock(&efi_lock); fpu_kern_enter(td, NULL, FPU_KERN_NOCTX); return (efi_arch_enter()); } static void efi_leave(void) { struct thread *td; pmap_t curpmap; efi_arch_leave(); curpmap = &curproc->p_vmspace->vm_pmap; td = curthread; fpu_kern_leave(td, NULL); mtx_unlock(&efi_lock); PMAP_UNLOCK(curpmap); } int efi_get_table(struct uuid *uuid, void **ptr) { struct efi_cfgtbl *ct; u_long count; if (efi_cfgtbl == NULL || efi_systbl == NULL) return (ENXIO); count = efi_systbl->st_entries; ct = efi_cfgtbl; while (count--) { if (!bcmp(&ct->ct_uuid, uuid, sizeof(*uuid))) { *ptr = (void *)efi_phys_to_kva(ct->ct_data); return (0); } ct++; } return (ENOENT); } static int efi_rt_handle_faults = EFI_RT_HANDLE_FAULTS_DEFAULT; SYSCTL_INT(_machdep, OID_AUTO, efi_rt_handle_faults, CTLFLAG_RWTUN, &efi_rt_handle_faults, 0, "Call EFI RT methods with fault handler wrapper around"); static int efi_rt_arch_call_nofault(struct efirt_callinfo *ec) { switch (ec->ec_argcnt) { case 0: ec->ec_efi_status = ((register_t (*)(void))ec->ec_fptr)(); break; case 1: ec->ec_efi_status = ((register_t (*)(register_t))ec->ec_fptr) (ec->ec_arg1); break; case 2: ec->ec_efi_status = ((register_t (*)(register_t, register_t)) ec->ec_fptr)(ec->ec_arg1, ec->ec_arg2); break; case 3: ec->ec_efi_status = ((register_t (*)(register_t, register_t, register_t))ec->ec_fptr)(ec->ec_arg1, ec->ec_arg2, ec->ec_arg3); break; case 4: ec->ec_efi_status = ((register_t (*)(register_t, register_t, register_t, register_t))ec->ec_fptr)(ec->ec_arg1, ec->ec_arg2, ec->ec_arg3, ec->ec_arg4); break; case 5: ec->ec_efi_status = ((register_t (*)(register_t, register_t, register_t, register_t, register_t))ec->ec_fptr)( ec->ec_arg1, ec->ec_arg2, ec->ec_arg3, ec->ec_arg4, ec->ec_arg5); break; default: panic("efi_rt_arch_call: %d args", (int)ec->ec_argcnt); } return (0); } static int efi_call(struct efirt_callinfo *ecp) { int error; error = efi_enter(); if (error != 0) return (error); error = efi_rt_handle_faults ? efi_rt_arch_call(ecp) : efi_rt_arch_call_nofault(ecp); efi_leave(); if (error == 0) error = efi_status_to_errno(ecp->ec_efi_status); else if (bootverbose) printf("EFI %s call faulted, error %d\n", ecp->ec_name, error); return (error); } #define EFI_RT_METHOD_PA(method) \ ((uintptr_t)((struct efi_rt *)efi_phys_to_kva((uintptr_t) \ efi_runtime))->method) static int efi_get_time_locked(struct efi_tm *tm, struct efi_tmcap *tmcap) { struct efirt_callinfo ec; EFI_TIME_OWNED(); if (efi_runtime == NULL) return (ENXIO); bzero(&ec, sizeof(ec)); ec.ec_name = "rt_gettime"; ec.ec_argcnt = 2; ec.ec_arg1 = (uintptr_t)tm; ec.ec_arg2 = (uintptr_t)tmcap; ec.ec_fptr = EFI_RT_METHOD_PA(rt_gettime); return (efi_call(&ec)); } int efi_get_time(struct efi_tm *tm) { struct efi_tmcap dummy; int error; if (efi_runtime == NULL) return (ENXIO); EFI_TIME_LOCK(); /* * UEFI spec states that the Capabilities argument to GetTime is * optional, but some UEFI implementations choke when passed a NULL * pointer. Pass a dummy efi_tmcap, even though we won't use it, * to workaround such implementations. */ error = efi_get_time_locked(tm, &dummy); EFI_TIME_UNLOCK(); return (error); } int efi_get_time_capabilities(struct efi_tmcap *tmcap) { struct efi_tm dummy; int error; if (efi_runtime == NULL) return (ENXIO); EFI_TIME_LOCK(); error = efi_get_time_locked(&dummy, tmcap); EFI_TIME_UNLOCK(); return (error); } int -efi_reset_system(void) +efi_reset_system(enum efi_reset type) { struct efirt_callinfo ec; + switch (type) { + case EFI_RESET_COLD: + case EFI_RESET_WARM: + case EFI_RESET_SHUTDOWN: + break; + default: + return (EINVAL); + } if (efi_runtime == NULL) return (ENXIO); bzero(&ec, sizeof(ec)); ec.ec_name = "rt_reset"; ec.ec_argcnt = 4; - ec.ec_arg1 = (uintptr_t)EFI_RESET_WARM; + ec.ec_arg1 = (uintptr_t)type; ec.ec_arg2 = (uintptr_t)0; ec.ec_arg3 = (uintptr_t)0; ec.ec_arg4 = (uintptr_t)NULL; ec.ec_fptr = EFI_RT_METHOD_PA(rt_reset); return (efi_call(&ec)); } static int efi_set_time_locked(struct efi_tm *tm) { struct efirt_callinfo ec; EFI_TIME_OWNED(); if (efi_runtime == NULL) return (ENXIO); bzero(&ec, sizeof(ec)); ec.ec_name = "rt_settime"; ec.ec_argcnt = 1; ec.ec_arg1 = (uintptr_t)tm; ec.ec_fptr = EFI_RT_METHOD_PA(rt_settime); return (efi_call(&ec)); } int efi_set_time(struct efi_tm *tm) { int error; if (efi_runtime == NULL) return (ENXIO); EFI_TIME_LOCK(); error = efi_set_time_locked(tm); EFI_TIME_UNLOCK(); return (error); } int efi_var_get(efi_char *name, struct uuid *vendor, uint32_t *attrib, size_t *datasize, void *data) { struct efirt_callinfo ec; if (efi_runtime == NULL) return (ENXIO); bzero(&ec, sizeof(ec)); ec.ec_argcnt = 5; ec.ec_name = "rt_getvar"; ec.ec_arg1 = (uintptr_t)name; ec.ec_arg2 = (uintptr_t)vendor; ec.ec_arg3 = (uintptr_t)attrib; ec.ec_arg4 = (uintptr_t)datasize; ec.ec_arg5 = (uintptr_t)data; ec.ec_fptr = EFI_RT_METHOD_PA(rt_getvar); return (efi_call(&ec)); } int efi_var_nextname(size_t *namesize, efi_char *name, struct uuid *vendor) { struct efirt_callinfo ec; if (efi_runtime == NULL) return (ENXIO); bzero(&ec, sizeof(ec)); ec.ec_argcnt = 3; ec.ec_name = "rt_scanvar"; ec.ec_arg1 = (uintptr_t)namesize; ec.ec_arg2 = (uintptr_t)name; ec.ec_arg3 = (uintptr_t)vendor; ec.ec_fptr = EFI_RT_METHOD_PA(rt_scanvar); return (efi_call(&ec)); } int efi_var_set(efi_char *name, struct uuid *vendor, uint32_t attrib, size_t datasize, void *data) { struct efirt_callinfo ec; if (efi_runtime == NULL) return (ENXIO); bzero(&ec, sizeof(ec)); ec.ec_argcnt = 5; ec.ec_name = "rt_setvar"; ec.ec_arg1 = (uintptr_t)name; ec.ec_arg2 = (uintptr_t)vendor; ec.ec_arg3 = (uintptr_t)attrib; ec.ec_arg4 = (uintptr_t)datasize; ec.ec_arg5 = (uintptr_t)data; ec.ec_fptr = EFI_RT_METHOD_PA(rt_setvar); return (efi_call(&ec)); } static int efirt_modevents(module_t m, int event, void *arg __unused) { switch (event) { case MOD_LOAD: return (efi_init()); case MOD_UNLOAD: efi_uninit(); return (0); case MOD_SHUTDOWN: return (0); default: return (EOPNOTSUPP); } } static moduledata_t efirt_moddata = { .name = "efirt", .evhand = efirt_modevents, .priv = NULL, }; /* After fpuinitstate, before efidev */ DECLARE_MODULE(efirt, efirt_moddata, SI_SUB_DRIVERS, SI_ORDER_SECOND); MODULE_VERSION(efirt, 1); Index: head/sys/dev/ipmi/ipmi.c =================================================================== --- head/sys/dev/ipmi/ipmi.c (revision 342107) +++ head/sys/dev/ipmi/ipmi.c (revision 342108) @@ -1,1104 +1,1104 @@ /*- * SPDX-License-Identifier: BSD-2-Clause-FreeBSD * * Copyright (c) 2006 IronPort Systems Inc. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. */ #include __FBSDID("$FreeBSD$"); #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifdef LOCAL_MODULE #include #include #else #include #include #endif /* * Driver request structures are allocated on the stack via alloca() to * avoid calling malloc(), especially for the watchdog handler. * To avoid too much stack growth, a previously allocated structure can * be reused via IPMI_INIT_DRIVER_REQUEST(), but the caller should ensure * that there is adequate reply/request space in the original allocation. */ #define IPMI_INIT_DRIVER_REQUEST(req, addr, cmd, reqlen, replylen) \ bzero((req), sizeof(struct ipmi_request)); \ ipmi_init_request((req), NULL, 0, (addr), (cmd), (reqlen), (replylen)) #define IPMI_ALLOC_DRIVER_REQUEST(req, addr, cmd, reqlen, replylen) \ (req) = __builtin_alloca(sizeof(struct ipmi_request) + \ (reqlen) + (replylen)); \ IPMI_INIT_DRIVER_REQUEST((req), (addr), (cmd), (reqlen), \ (replylen)) #ifdef IPMB static int ipmi_ipmb_checksum(u_char, int); static int ipmi_ipmb_send_message(device_t, u_char, u_char, u_char, u_char, u_char, int) #endif static d_ioctl_t ipmi_ioctl; static d_poll_t ipmi_poll; static d_open_t ipmi_open; static void ipmi_dtor(void *arg); int ipmi_attached = 0; static int on = 1; static bool wd_in_shutdown = false; static int wd_timer_actions = IPMI_SET_WD_ACTION_POWER_CYCLE; static int wd_shutdown_countdown = 0; /* sec */ static int wd_startup_countdown = 0; /* sec */ static int wd_pretimeout_countdown = 120; /* sec */ static int cycle_wait = 10; /* sec */ static SYSCTL_NODE(_hw, OID_AUTO, ipmi, CTLFLAG_RD, 0, "IPMI driver parameters"); SYSCTL_INT(_hw_ipmi, OID_AUTO, on, CTLFLAG_RWTUN, &on, 0, ""); SYSCTL_INT(_hw_ipmi, OID_AUTO, wd_timer_actions, CTLFLAG_RW, &wd_timer_actions, 0, "IPMI watchdog timer actions (including pre-timeout interrupt)"); SYSCTL_INT(_hw_ipmi, OID_AUTO, wd_shutdown_countdown, CTLFLAG_RW, &wd_shutdown_countdown, 0, "IPMI watchdog countdown for shutdown (seconds)"); SYSCTL_INT(_hw_ipmi, OID_AUTO, wd_startup_countdown, CTLFLAG_RDTUN, &wd_startup_countdown, 0, "IPMI watchdog countdown initialized during startup (seconds)"); SYSCTL_INT(_hw_ipmi, OID_AUTO, wd_pretimeout_countdown, CTLFLAG_RW, &wd_pretimeout_countdown, 0, "IPMI watchdog pre-timeout countdown (seconds)"); SYSCTL_INT(_hw_ipmi, OID_AUTO, cyle_wait, CTLFLAG_RWTUN, &cycle_wait, 0, "IPMI power cycle on reboot delay time (seconds)"); static struct cdevsw ipmi_cdevsw = { .d_version = D_VERSION, .d_open = ipmi_open, .d_ioctl = ipmi_ioctl, .d_poll = ipmi_poll, .d_name = "ipmi", }; static MALLOC_DEFINE(M_IPMI, "ipmi", "ipmi"); static int ipmi_open(struct cdev *cdev, int flags, int fmt, struct thread *td) { struct ipmi_device *dev; struct ipmi_softc *sc; int error; if (!on) return (ENOENT); /* Initialize the per file descriptor data. */ dev = malloc(sizeof(struct ipmi_device), M_IPMI, M_WAITOK | M_ZERO); error = devfs_set_cdevpriv(dev, ipmi_dtor); if (error) { free(dev, M_IPMI); return (error); } sc = cdev->si_drv1; TAILQ_INIT(&dev->ipmi_completed_requests); dev->ipmi_address = IPMI_BMC_SLAVE_ADDR; dev->ipmi_lun = IPMI_BMC_SMS_LUN; dev->ipmi_softc = sc; IPMI_LOCK(sc); sc->ipmi_opened++; IPMI_UNLOCK(sc); return (0); } static int ipmi_poll(struct cdev *cdev, int poll_events, struct thread *td) { struct ipmi_device *dev; struct ipmi_softc *sc; int revents = 0; if (devfs_get_cdevpriv((void **)&dev)) return (0); sc = cdev->si_drv1; IPMI_LOCK(sc); if (poll_events & (POLLIN | POLLRDNORM)) { if (!TAILQ_EMPTY(&dev->ipmi_completed_requests)) revents |= poll_events & (POLLIN | POLLRDNORM); if (dev->ipmi_requests == 0) revents |= POLLERR; } if (revents == 0) { if (poll_events & (POLLIN | POLLRDNORM)) selrecord(td, &dev->ipmi_select); } IPMI_UNLOCK(sc); return (revents); } static void ipmi_purge_completed_requests(struct ipmi_device *dev) { struct ipmi_request *req; while (!TAILQ_EMPTY(&dev->ipmi_completed_requests)) { req = TAILQ_FIRST(&dev->ipmi_completed_requests); TAILQ_REMOVE(&dev->ipmi_completed_requests, req, ir_link); dev->ipmi_requests--; ipmi_free_request(req); } } static void ipmi_dtor(void *arg) { struct ipmi_request *req, *nreq; struct ipmi_device *dev; struct ipmi_softc *sc; dev = arg; sc = dev->ipmi_softc; IPMI_LOCK(sc); if (dev->ipmi_requests) { /* Throw away any pending requests for this device. */ TAILQ_FOREACH_SAFE(req, &sc->ipmi_pending_requests, ir_link, nreq) { if (req->ir_owner == dev) { TAILQ_REMOVE(&sc->ipmi_pending_requests, req, ir_link); dev->ipmi_requests--; ipmi_free_request(req); } } /* Throw away any pending completed requests for this device. */ ipmi_purge_completed_requests(dev); /* * If we still have outstanding requests, they must be stuck * in an interface driver, so wait for those to drain. */ dev->ipmi_closing = 1; while (dev->ipmi_requests > 0) { msleep(&dev->ipmi_requests, &sc->ipmi_requests_lock, PWAIT, "ipmidrain", 0); ipmi_purge_completed_requests(dev); } } sc->ipmi_opened--; IPMI_UNLOCK(sc); /* Cleanup. */ free(dev, M_IPMI); } #ifdef IPMB static int ipmi_ipmb_checksum(u_char *data, int len) { u_char sum = 0; for (; len; len--) { sum += *data++; } return (-sum); } /* XXX: Needs work */ static int ipmi_ipmb_send_message(device_t dev, u_char channel, u_char netfn, u_char command, u_char seq, u_char *data, int data_len) { struct ipmi_softc *sc = device_get_softc(dev); struct ipmi_request *req; u_char slave_addr = 0x52; int error; IPMI_ALLOC_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_SEND_MSG, data_len + 8, 0); req->ir_request[0] = channel; req->ir_request[1] = slave_addr; req->ir_request[2] = IPMI_ADDR(netfn, 0); req->ir_request[3] = ipmi_ipmb_checksum(&req->ir_request[1], 2); req->ir_request[4] = sc->ipmi_address; req->ir_request[5] = IPMI_ADDR(seq, sc->ipmi_lun); req->ir_request[6] = command; bcopy(data, &req->ir_request[7], data_len); temp[data_len + 7] = ipmi_ipmb_checksum(&req->ir_request[4], data_len + 3); ipmi_submit_driver_request(sc, req); error = req->ir_error; return (error); } static int ipmi_handle_attn(struct ipmi_softc *sc) { struct ipmi_request *req; int error; device_printf(sc->ipmi_dev, "BMC has a message\n"); IPMI_ALLOC_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_GET_MSG_FLAGS, 0, 1); ipmi_submit_driver_request(sc, req); if (req->ir_error == 0 && req->ir_compcode == 0) { if (req->ir_reply[0] & IPMI_MSG_BUFFER_FULL) { device_printf(sc->ipmi_dev, "message buffer full"); } if (req->ir_reply[0] & IPMI_WDT_PRE_TIMEOUT) { device_printf(sc->ipmi_dev, "watchdog about to go off"); } if (req->ir_reply[0] & IPMI_MSG_AVAILABLE) { IPMI_ALLOC_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_GET_MSG, 0, 16); device_printf(sc->ipmi_dev, "throw out message "); dump_buf(temp, 16); } } error = req->ir_error; return (error); } #endif #ifdef IPMICTL_SEND_COMMAND_32 #define PTRIN(p) ((void *)(uintptr_t)(p)) #define PTROUT(p) ((uintptr_t)(p)) #endif static int ipmi_ioctl(struct cdev *cdev, u_long cmd, caddr_t data, int flags, struct thread *td) { struct ipmi_softc *sc; struct ipmi_device *dev; struct ipmi_request *kreq; struct ipmi_req *req = (struct ipmi_req *)data; struct ipmi_recv *recv = (struct ipmi_recv *)data; struct ipmi_addr addr; #ifdef IPMICTL_SEND_COMMAND_32 struct ipmi_req32 *req32 = (struct ipmi_req32 *)data; struct ipmi_recv32 *recv32 = (struct ipmi_recv32 *)data; union { struct ipmi_req req; struct ipmi_recv recv; } thunk32; #endif int error, len; error = devfs_get_cdevpriv((void **)&dev); if (error) return (error); sc = cdev->si_drv1; #ifdef IPMICTL_SEND_COMMAND_32 /* Convert 32-bit structures to native. */ switch (cmd) { case IPMICTL_SEND_COMMAND_32: req = &thunk32.req; req->addr = PTRIN(req32->addr); req->addr_len = req32->addr_len; req->msgid = req32->msgid; req->msg.netfn = req32->msg.netfn; req->msg.cmd = req32->msg.cmd; req->msg.data_len = req32->msg.data_len; req->msg.data = PTRIN(req32->msg.data); break; case IPMICTL_RECEIVE_MSG_TRUNC_32: case IPMICTL_RECEIVE_MSG_32: recv = &thunk32.recv; recv->addr = PTRIN(recv32->addr); recv->addr_len = recv32->addr_len; recv->msg.data_len = recv32->msg.data_len; recv->msg.data = PTRIN(recv32->msg.data); break; } #endif switch (cmd) { #ifdef IPMICTL_SEND_COMMAND_32 case IPMICTL_SEND_COMMAND_32: #endif case IPMICTL_SEND_COMMAND: /* * XXX: Need to add proper handling of this. */ error = copyin(req->addr, &addr, sizeof(addr)); if (error) return (error); IPMI_LOCK(sc); /* clear out old stuff in queue of stuff done */ /* XXX: This seems odd. */ while ((kreq = TAILQ_FIRST(&dev->ipmi_completed_requests))) { TAILQ_REMOVE(&dev->ipmi_completed_requests, kreq, ir_link); dev->ipmi_requests--; ipmi_free_request(kreq); } IPMI_UNLOCK(sc); kreq = ipmi_alloc_request(dev, req->msgid, IPMI_ADDR(req->msg.netfn, 0), req->msg.cmd, req->msg.data_len, IPMI_MAX_RX); error = copyin(req->msg.data, kreq->ir_request, req->msg.data_len); if (error) { ipmi_free_request(kreq); return (error); } IPMI_LOCK(sc); dev->ipmi_requests++; error = sc->ipmi_enqueue_request(sc, kreq); IPMI_UNLOCK(sc); if (error) return (error); break; #ifdef IPMICTL_SEND_COMMAND_32 case IPMICTL_RECEIVE_MSG_TRUNC_32: case IPMICTL_RECEIVE_MSG_32: #endif case IPMICTL_RECEIVE_MSG_TRUNC: case IPMICTL_RECEIVE_MSG: error = copyin(recv->addr, &addr, sizeof(addr)); if (error) return (error); IPMI_LOCK(sc); kreq = TAILQ_FIRST(&dev->ipmi_completed_requests); if (kreq == NULL) { IPMI_UNLOCK(sc); return (EAGAIN); } addr.channel = IPMI_BMC_CHANNEL; /* XXX */ recv->recv_type = IPMI_RESPONSE_RECV_TYPE; recv->msgid = kreq->ir_msgid; recv->msg.netfn = IPMI_REPLY_ADDR(kreq->ir_addr) >> 2; recv->msg.cmd = kreq->ir_command; error = kreq->ir_error; if (error) { TAILQ_REMOVE(&dev->ipmi_completed_requests, kreq, ir_link); dev->ipmi_requests--; IPMI_UNLOCK(sc); ipmi_free_request(kreq); return (error); } len = kreq->ir_replylen + 1; if (recv->msg.data_len < len && (cmd == IPMICTL_RECEIVE_MSG #ifdef IPMICTL_RECEIVE_MSG_32 || cmd == IPMICTL_RECEIVE_MSG_32 #endif )) { IPMI_UNLOCK(sc); return (EMSGSIZE); } TAILQ_REMOVE(&dev->ipmi_completed_requests, kreq, ir_link); dev->ipmi_requests--; IPMI_UNLOCK(sc); len = min(recv->msg.data_len, len); recv->msg.data_len = len; error = copyout(&addr, recv->addr,sizeof(addr)); if (error == 0) error = copyout(&kreq->ir_compcode, recv->msg.data, 1); if (error == 0) error = copyout(kreq->ir_reply, recv->msg.data + 1, len - 1); ipmi_free_request(kreq); if (error) return (error); break; case IPMICTL_SET_MY_ADDRESS_CMD: IPMI_LOCK(sc); dev->ipmi_address = *(int*)data; IPMI_UNLOCK(sc); break; case IPMICTL_GET_MY_ADDRESS_CMD: IPMI_LOCK(sc); *(int*)data = dev->ipmi_address; IPMI_UNLOCK(sc); break; case IPMICTL_SET_MY_LUN_CMD: IPMI_LOCK(sc); dev->ipmi_lun = *(int*)data & 0x3; IPMI_UNLOCK(sc); break; case IPMICTL_GET_MY_LUN_CMD: IPMI_LOCK(sc); *(int*)data = dev->ipmi_lun; IPMI_UNLOCK(sc); break; case IPMICTL_SET_GETS_EVENTS_CMD: /* device_printf(sc->ipmi_dev, "IPMICTL_SET_GETS_EVENTS_CMD NA\n"); */ break; case IPMICTL_REGISTER_FOR_CMD: case IPMICTL_UNREGISTER_FOR_CMD: return (EOPNOTSUPP); default: device_printf(sc->ipmi_dev, "Unknown IOCTL %lX\n", cmd); return (ENOIOCTL); } #ifdef IPMICTL_SEND_COMMAND_32 /* Update changed fields in 32-bit structures. */ switch (cmd) { case IPMICTL_RECEIVE_MSG_TRUNC_32: case IPMICTL_RECEIVE_MSG_32: recv32->recv_type = recv->recv_type; recv32->msgid = recv->msgid; recv32->msg.netfn = recv->msg.netfn; recv32->msg.cmd = recv->msg.cmd; recv32->msg.data_len = recv->msg.data_len; break; } #endif return (0); } /* * Request management. */ static __inline void ipmi_init_request(struct ipmi_request *req, struct ipmi_device *dev, long msgid, uint8_t addr, uint8_t command, size_t requestlen, size_t replylen) { req->ir_owner = dev; req->ir_msgid = msgid; req->ir_addr = addr; req->ir_command = command; if (requestlen) { req->ir_request = (char *)&req[1]; req->ir_requestlen = requestlen; } if (replylen) { req->ir_reply = (char *)&req[1] + requestlen; req->ir_replybuflen = replylen; } } /* Allocate a new request with request and reply buffers. */ struct ipmi_request * ipmi_alloc_request(struct ipmi_device *dev, long msgid, uint8_t addr, uint8_t command, size_t requestlen, size_t replylen) { struct ipmi_request *req; req = malloc(sizeof(struct ipmi_request) + requestlen + replylen, M_IPMI, M_WAITOK | M_ZERO); ipmi_init_request(req, dev, msgid, addr, command, requestlen, replylen); return (req); } /* Free a request no longer in use. */ void ipmi_free_request(struct ipmi_request *req) { free(req, M_IPMI); } /* Store a processed request on the appropriate completion queue. */ void ipmi_complete_request(struct ipmi_softc *sc, struct ipmi_request *req) { struct ipmi_device *dev; IPMI_LOCK_ASSERT(sc); /* * Anonymous requests (from inside the driver) always have a * waiter that we awaken. */ if (req->ir_owner == NULL) wakeup(req); else { dev = req->ir_owner; TAILQ_INSERT_TAIL(&dev->ipmi_completed_requests, req, ir_link); selwakeup(&dev->ipmi_select); if (dev->ipmi_closing) wakeup(&dev->ipmi_requests); } } /* Perform an internal driver request. */ int ipmi_submit_driver_request(struct ipmi_softc *sc, struct ipmi_request *req, int timo) { return (sc->ipmi_driver_request(sc, req, timo)); } /* * Helper routine for polled system interfaces that use * ipmi_polled_enqueue_request() to queue requests. This request * waits until there is a pending request and then returns the first * request. If the driver is shutting down, it returns NULL. */ struct ipmi_request * ipmi_dequeue_request(struct ipmi_softc *sc) { struct ipmi_request *req; IPMI_LOCK_ASSERT(sc); while (!sc->ipmi_detaching && TAILQ_EMPTY(&sc->ipmi_pending_requests)) cv_wait(&sc->ipmi_request_added, &sc->ipmi_requests_lock); if (sc->ipmi_detaching) return (NULL); req = TAILQ_FIRST(&sc->ipmi_pending_requests); TAILQ_REMOVE(&sc->ipmi_pending_requests, req, ir_link); return (req); } /* Default implementation of ipmi_enqueue_request() for polled interfaces. */ int ipmi_polled_enqueue_request(struct ipmi_softc *sc, struct ipmi_request *req) { IPMI_LOCK_ASSERT(sc); TAILQ_INSERT_TAIL(&sc->ipmi_pending_requests, req, ir_link); cv_signal(&sc->ipmi_request_added); return (0); } /* * Watchdog event handler. */ static int ipmi_reset_watchdog(struct ipmi_softc *sc) { struct ipmi_request *req; int error; IPMI_ALLOC_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_RESET_WDOG, 0, 0); error = ipmi_submit_driver_request(sc, req, 0); if (error) device_printf(sc->ipmi_dev, "Failed to reset watchdog\n"); return (error); } static int ipmi_set_watchdog(struct ipmi_softc *sc, unsigned int sec) { struct ipmi_request *req; int error; if (sec > 0xffff / 10) return (EINVAL); IPMI_ALLOC_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_SET_WDOG, 6, 0); if (sec) { req->ir_request[0] = IPMI_SET_WD_TIMER_DONT_STOP | IPMI_SET_WD_TIMER_SMS_OS; req->ir_request[1] = (wd_timer_actions & 0xff); req->ir_request[2] = (wd_pretimeout_countdown & 0xff); req->ir_request[3] = 0; /* Timer use */ req->ir_request[4] = (sec * 10) & 0xff; req->ir_request[5] = (sec * 10) >> 8; } else { req->ir_request[0] = IPMI_SET_WD_TIMER_SMS_OS; req->ir_request[1] = 0; req->ir_request[2] = 0; req->ir_request[3] = 0; /* Timer use */ req->ir_request[4] = 0; req->ir_request[5] = 0; } error = ipmi_submit_driver_request(sc, req, 0); if (error) device_printf(sc->ipmi_dev, "Failed to set watchdog\n"); return (error); } static void ipmi_wd_event(void *arg, unsigned int cmd, int *error) { struct ipmi_softc *sc = arg; unsigned int timeout; int e; /* Ignore requests while disabled. */ if (!on) return; /* * To prevent infinite hangs, we don't let anyone pat or change * the watchdog when we're shutting down. (See ipmi_shutdown_event().) * However, we do want to keep patting the watchdog while we are doing * a coredump. */ if (wd_in_shutdown) { if (dumping && sc->ipmi_watchdog_active) ipmi_reset_watchdog(sc); return; } cmd &= WD_INTERVAL; if (cmd > 0 && cmd <= 63) { timeout = ((uint64_t)1 << cmd) / 1000000000; if (timeout == 0) timeout = 1; if (timeout != sc->ipmi_watchdog_active || wd_timer_actions != sc->ipmi_watchdog_actions || wd_pretimeout_countdown != sc->ipmi_watchdog_pretimeout) { e = ipmi_set_watchdog(sc, timeout); if (e == 0) { sc->ipmi_watchdog_active = timeout; sc->ipmi_watchdog_actions = wd_timer_actions; sc->ipmi_watchdog_pretimeout = wd_pretimeout_countdown; } else { (void)ipmi_set_watchdog(sc, 0); sc->ipmi_watchdog_active = 0; sc->ipmi_watchdog_actions = 0; sc->ipmi_watchdog_pretimeout = 0; } } if (sc->ipmi_watchdog_active != 0) { e = ipmi_reset_watchdog(sc); if (e == 0) { *error = 0; } else { (void)ipmi_set_watchdog(sc, 0); sc->ipmi_watchdog_active = 0; sc->ipmi_watchdog_actions = 0; sc->ipmi_watchdog_pretimeout = 0; } } } else if (atomic_readandclear_int(&sc->ipmi_watchdog_active) != 0) { sc->ipmi_watchdog_actions = 0; sc->ipmi_watchdog_pretimeout = 0; e = ipmi_set_watchdog(sc, 0); if (e != 0 && cmd == 0) *error = EOPNOTSUPP; } } static void ipmi_shutdown_event(void *arg, unsigned int cmd, int *error) { struct ipmi_softc *sc = arg; /* Ignore event if disabled. */ if (!on) return; /* * Positive wd_shutdown_countdown value will re-arm watchdog; * Zero value in wd_shutdown_countdown will disable watchdog; * Negative value in wd_shutdown_countdown will keep existing state; * * Revert to using a power cycle to ensure that the watchdog will * do something useful here. Having the watchdog send an NMI * instead is useless during shutdown, and might be ignored if an * NMI already triggered. */ wd_in_shutdown = true; if (wd_shutdown_countdown == 0) { /* disable watchdog */ ipmi_set_watchdog(sc, 0); sc->ipmi_watchdog_active = 0; } else if (wd_shutdown_countdown > 0) { /* set desired action and time, and, reset watchdog */ wd_timer_actions = IPMI_SET_WD_ACTION_POWER_CYCLE; ipmi_set_watchdog(sc, wd_shutdown_countdown); sc->ipmi_watchdog_active = wd_shutdown_countdown; ipmi_reset_watchdog(sc); } } static void ipmi_power_cycle(void *arg, int howto) { struct ipmi_softc *sc = arg; struct ipmi_request *req; /* * Ignore everything except power cycling requests */ if ((howto & RB_POWERCYCLE) == 0) return; device_printf(sc->ipmi_dev, "Power cycling using IPMI\n"); /* * Send a CHASSIS_CONTROL command to the CHASSIS device, subcommand 2 * as described in IPMI v2.0 spec section 28.3. */ IPMI_ALLOC_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_CHASSIS_REQUEST, 0), IPMI_CHASSIS_CONTROL, 1, 0); req->ir_request[0] = IPMI_CC_POWER_CYCLE; ipmi_submit_driver_request(sc, req, MAX_TIMEOUT); if (req->ir_error != 0 || req->ir_compcode != 0) { device_printf(sc->ipmi_dev, "Power cycling via IPMI failed code %#x %#x\n", req->ir_error, req->ir_compcode); return; } /* * BMCs are notoriously slow, give it cyle_wait seconds for the power * down leg of the power cycle. If that fails, fallback to the next * hanlder in the shutdown_final chain and/or the platform failsafe. */ DELAY(cycle_wait * 1000 * 1000); device_printf(sc->ipmi_dev, "Power cycling via IPMI timed out\n"); } static void ipmi_startup(void *arg) { struct ipmi_softc *sc = arg; struct ipmi_request *req; device_t dev; int error, i; config_intrhook_disestablish(&sc->ipmi_ich); dev = sc->ipmi_dev; /* Initialize interface-independent state. */ mtx_init(&sc->ipmi_requests_lock, "ipmi requests", NULL, MTX_DEF); mtx_init(&sc->ipmi_io_lock, "ipmi io", NULL, MTX_DEF); cv_init(&sc->ipmi_request_added, "ipmireq"); TAILQ_INIT(&sc->ipmi_pending_requests); /* Initialize interface-dependent state. */ error = sc->ipmi_startup(sc); if (error) { device_printf(dev, "Failed to initialize interface: %d\n", error); return; } /* Send a GET_DEVICE_ID request. */ IPMI_ALLOC_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_GET_DEVICE_ID, 0, 15); error = ipmi_submit_driver_request(sc, req, MAX_TIMEOUT); if (error == EWOULDBLOCK) { device_printf(dev, "Timed out waiting for GET_DEVICE_ID\n"); return; } else if (error) { device_printf(dev, "Failed GET_DEVICE_ID: %d\n", error); return; } else if (req->ir_compcode != 0) { device_printf(dev, "Bad completion code for GET_DEVICE_ID: %d\n", req->ir_compcode); return; } else if (req->ir_replylen < 5) { device_printf(dev, "Short reply for GET_DEVICE_ID: %d\n", req->ir_replylen); return; } device_printf(dev, "IPMI device rev. %d, firmware rev. %d.%d%d, " "version %d.%d, device support mask %#x\n", req->ir_reply[1] & 0x0f, req->ir_reply[2] & 0x7f, req->ir_reply[3] >> 4, req->ir_reply[3] & 0x0f, req->ir_reply[4] & 0x0f, req->ir_reply[4] >> 4, req->ir_reply[5]); sc->ipmi_dev_support = req->ir_reply[5]; IPMI_INIT_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_CLEAR_FLAGS, 1, 0); ipmi_submit_driver_request(sc, req, 0); /* XXX: Magic numbers */ if (req->ir_compcode == 0xc0) { device_printf(dev, "Clear flags is busy\n"); } if (req->ir_compcode == 0xc1) { device_printf(dev, "Clear flags illegal\n"); } for (i = 0; i < 8; i++) { IPMI_INIT_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_GET_CHANNEL_INFO, 1, 0); req->ir_request[0] = i; ipmi_submit_driver_request(sc, req, 0); if (req->ir_compcode != 0) break; } device_printf(dev, "Number of channels %d\n", i); /* * Probe for watchdog, but only for backends which support * polled driver requests. */ if (sc->ipmi_driver_requests_polled) { IPMI_INIT_DRIVER_REQUEST(req, IPMI_ADDR(IPMI_APP_REQUEST, 0), IPMI_GET_WDOG, 0, 0); ipmi_submit_driver_request(sc, req, 0); if (req->ir_compcode == 0x00) { device_printf(dev, "Attached watchdog\n"); /* register the watchdog event handler */ sc->ipmi_watchdog_tag = EVENTHANDLER_REGISTER( watchdog_list, ipmi_wd_event, sc, 0); sc->ipmi_shutdown_tag = EVENTHANDLER_REGISTER( shutdown_pre_sync, ipmi_shutdown_event, sc, 0); } } sc->ipmi_cdev = make_dev(&ipmi_cdevsw, device_get_unit(dev), UID_ROOT, GID_OPERATOR, 0660, "ipmi%d", device_get_unit(dev)); if (sc->ipmi_cdev == NULL) { device_printf(dev, "Failed to create cdev\n"); return; } sc->ipmi_cdev->si_drv1 = sc; /* * Set initial watchdog state. If desired, set an initial * watchdog on startup. Or, if the watchdog device is * disabled, clear any existing watchdog. */ if (on && wd_startup_countdown > 0) { wd_timer_actions = IPMI_SET_WD_ACTION_POWER_CYCLE; if (ipmi_set_watchdog(sc, wd_startup_countdown) == 0 && ipmi_reset_watchdog(sc) == 0) { sc->ipmi_watchdog_active = wd_startup_countdown; sc->ipmi_watchdog_actions = wd_timer_actions; sc->ipmi_watchdog_pretimeout = wd_pretimeout_countdown; } else (void)ipmi_set_watchdog(sc, 0); ipmi_reset_watchdog(sc); } else if (!on) (void)ipmi_set_watchdog(sc, 0); /* - * Power cycle the system off using IPMI. We use last - 1 since we don't + * Power cycle the system off using IPMI. We use last - 2 since we don't * handle all the other kinds of reboots. We'll let others handle them. * We only try to do this if the BMC supports the Chassis device. */ if (sc->ipmi_dev_support & IPMI_ADS_CHASSIS) { device_printf(dev, "Establishing power cycle handler\n"); sc->ipmi_power_cycle_tag = EVENTHANDLER_REGISTER(shutdown_final, - ipmi_power_cycle, sc, SHUTDOWN_PRI_LAST - 1); + ipmi_power_cycle, sc, SHUTDOWN_PRI_LAST - 2); } } int ipmi_attach(device_t dev) { struct ipmi_softc *sc = device_get_softc(dev); int error; if (sc->ipmi_irq_res != NULL && sc->ipmi_intr != NULL) { error = bus_setup_intr(dev, sc->ipmi_irq_res, INTR_TYPE_MISC, NULL, sc->ipmi_intr, sc, &sc->ipmi_irq); if (error) { device_printf(dev, "can't set up interrupt\n"); return (error); } } bzero(&sc->ipmi_ich, sizeof(struct intr_config_hook)); sc->ipmi_ich.ich_func = ipmi_startup; sc->ipmi_ich.ich_arg = sc; if (config_intrhook_establish(&sc->ipmi_ich) != 0) { device_printf(dev, "can't establish configuration hook\n"); return (ENOMEM); } ipmi_attached = 1; return (0); } int ipmi_detach(device_t dev) { struct ipmi_softc *sc; sc = device_get_softc(dev); /* Fail if there are any open handles. */ IPMI_LOCK(sc); if (sc->ipmi_opened) { IPMI_UNLOCK(sc); return (EBUSY); } IPMI_UNLOCK(sc); if (sc->ipmi_cdev) destroy_dev(sc->ipmi_cdev); /* Detach from watchdog handling and turn off watchdog. */ if (sc->ipmi_shutdown_tag) EVENTHANDLER_DEREGISTER(shutdown_pre_sync, sc->ipmi_shutdown_tag); if (sc->ipmi_watchdog_tag) { EVENTHANDLER_DEREGISTER(watchdog_list, sc->ipmi_watchdog_tag); ipmi_set_watchdog(sc, 0); } /* Detach from shutdown handling for power cycle reboot */ if (sc->ipmi_power_cycle_tag) EVENTHANDLER_DEREGISTER(shutdown_final, sc->ipmi_power_cycle_tag); /* XXX: should use shutdown callout I think. */ /* If the backend uses a kthread, shut it down. */ IPMI_LOCK(sc); sc->ipmi_detaching = 1; if (sc->ipmi_kthread) { cv_broadcast(&sc->ipmi_request_added); msleep(sc->ipmi_kthread, &sc->ipmi_requests_lock, 0, "ipmi_wait", 0); } IPMI_UNLOCK(sc); if (sc->ipmi_irq) bus_teardown_intr(dev, sc->ipmi_irq_res, sc->ipmi_irq); ipmi_release_resources(dev); mtx_destroy(&sc->ipmi_io_lock); mtx_destroy(&sc->ipmi_requests_lock); return (0); } void ipmi_release_resources(device_t dev) { struct ipmi_softc *sc; int i; sc = device_get_softc(dev); if (sc->ipmi_irq) bus_teardown_intr(dev, sc->ipmi_irq_res, sc->ipmi_irq); if (sc->ipmi_irq_res) bus_release_resource(dev, SYS_RES_IRQ, sc->ipmi_irq_rid, sc->ipmi_irq_res); for (i = 0; i < MAX_RES; i++) if (sc->ipmi_io_res[i]) bus_release_resource(dev, sc->ipmi_io_type, sc->ipmi_io_rid + i, sc->ipmi_io_res[i]); } devclass_t ipmi_devclass; /* XXX: Why? */ static void ipmi_unload(void *arg) { device_t * devs; int count; int i; if (ipmi_devclass == NULL) return; if (devclass_get_devices(ipmi_devclass, &devs, &count) != 0) return; for (i = 0; i < count; i++) device_delete_child(device_get_parent(devs[i]), devs[i]); free(devs, M_TEMP); } SYSUNINIT(ipmi_unload, SI_SUB_DRIVERS, SI_ORDER_FIRST, ipmi_unload, NULL); #ifdef IMPI_DEBUG static void dump_buf(u_char *data, int len) { char buf[20]; char line[1024]; char temp[30]; int count = 0; int i=0; printf("Address %p len %d\n", data, len); if (len > 256) len = 256; line[0] = '\000'; for (; len > 0; len--, data++) { sprintf(temp, "%02x ", *data); strcat(line, temp); if (*data >= ' ' && *data <= '~') buf[count] = *data; else if (*data >= 'A' && *data <= 'Z') buf[count] = *data; else buf[count] = '.'; if (++count == 16) { buf[count] = '\000'; count = 0; printf(" %3x %s %s\n", i, line, buf); i+=16; line[0] = '\000'; } } buf[count] = '\000'; for (; count != 16; count++) { strcat(line, " "); } printf(" %3x %s %s\n", i, line, buf); } #endif Index: head/sys/sys/efi.h =================================================================== --- head/sys/sys/efi.h (revision 342107) +++ head/sys/sys/efi.h (revision 342108) @@ -1,197 +1,198 @@ /*- * Copyright (c) 2004 Marcel Moolenaar * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. * * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * $FreeBSD$ */ #ifndef _SYS_EFI_H_ #define _SYS_EFI_H_ #include #include #define EFI_PAGE_SHIFT 12 #define EFI_PAGE_SIZE (1 << EFI_PAGE_SHIFT) #define EFI_PAGE_MASK (EFI_PAGE_SIZE - 1) #define EFI_TABLE_ACPI20 \ {0x8868e871,0xe4f1,0x11d3,0xbc,0x22,{0x00,0x80,0xc7,0x3c,0x88,0x81}} #define EFI_TABLE_SAL \ {0xeb9d2d32,0x2d88,0x11d3,0x9a,0x16,{0x00,0x90,0x27,0x3f,0xc1,0x4d}} enum efi_reset { - EFI_RESET_COLD, - EFI_RESET_WARM + EFI_RESET_COLD = 0, + EFI_RESET_WARM = 1, + EFI_RESET_SHUTDOWN = 2, }; typedef uint16_t efi_char; typedef unsigned long efi_status; struct efi_cfgtbl { struct uuid ct_uuid; uint64_t ct_data; }; struct efi_md { uint32_t md_type; #define EFI_MD_TYPE_NULL 0 #define EFI_MD_TYPE_CODE 1 /* Loader text. */ #define EFI_MD_TYPE_DATA 2 /* Loader data. */ #define EFI_MD_TYPE_BS_CODE 3 /* Boot services text. */ #define EFI_MD_TYPE_BS_DATA 4 /* Boot services data. */ #define EFI_MD_TYPE_RT_CODE 5 /* Runtime services text. */ #define EFI_MD_TYPE_RT_DATA 6 /* Runtime services data. */ #define EFI_MD_TYPE_FREE 7 /* Unused/free memory. */ #define EFI_MD_TYPE_BAD 8 /* Bad memory */ #define EFI_MD_TYPE_RECLAIM 9 /* ACPI reclaimable memory. */ #define EFI_MD_TYPE_FIRMWARE 10 /* ACPI NV memory */ #define EFI_MD_TYPE_IOMEM 11 /* Memory-mapped I/O. */ #define EFI_MD_TYPE_IOPORT 12 /* I/O port space. */ #define EFI_MD_TYPE_PALCODE 13 /* PAL */ #define EFI_MD_TYPE_PERSISTENT 14 /* Persistent memory. */ uint32_t __pad; uint64_t md_phys; void *md_virt; uint64_t md_pages; uint64_t md_attr; #define EFI_MD_ATTR_UC 0x0000000000000001UL #define EFI_MD_ATTR_WC 0x0000000000000002UL #define EFI_MD_ATTR_WT 0x0000000000000004UL #define EFI_MD_ATTR_WB 0x0000000000000008UL #define EFI_MD_ATTR_UCE 0x0000000000000010UL #define EFI_MD_ATTR_WP 0x0000000000001000UL #define EFI_MD_ATTR_RP 0x0000000000002000UL #define EFI_MD_ATTR_XP 0x0000000000004000UL #define EFI_MD_ATTR_NV 0x0000000000008000UL #define EFI_MD_ATTR_MORE_RELIABLE \ 0x0000000000010000UL #define EFI_MD_ATTR_RO 0x0000000000020000UL #define EFI_MD_ATTR_RT 0x8000000000000000UL }; #define efi_next_descriptor(ptr, size) \ ((struct efi_md *)(((uint8_t *)(ptr)) + (size))) struct efi_tm { uint16_t tm_year; /* 1998 - 20XX */ uint8_t tm_mon; /* 1 - 12 */ uint8_t tm_mday; /* 1 - 31 */ uint8_t tm_hour; /* 0 - 23 */ uint8_t tm_min; /* 0 - 59 */ uint8_t tm_sec; /* 0 - 59 */ uint8_t __pad1; uint32_t tm_nsec; /* 0 - 999,999,999 */ int16_t tm_tz; /* -1440 to 1440 or 2047 */ uint8_t tm_dst; uint8_t __pad2; }; struct efi_tmcap { uint32_t tc_res; /* 1e-6 parts per million */ uint32_t tc_prec; /* hertz */ uint8_t tc_stz; /* Set clears sub-second time */ }; struct efi_tblhdr { uint64_t th_sig; uint32_t th_rev; uint32_t th_hdrsz; uint32_t th_crc32; uint32_t __res; }; #ifdef _KERNEL #ifdef EFIABI_ATTR struct efi_rt { struct efi_tblhdr rt_hdr; efi_status (*rt_gettime)(struct efi_tm *, struct efi_tmcap *) EFIABI_ATTR; efi_status (*rt_settime)(struct efi_tm *) EFIABI_ATTR; efi_status (*rt_getwaketime)(uint8_t *, uint8_t *, struct efi_tm *) EFIABI_ATTR; efi_status (*rt_setwaketime)(uint8_t, struct efi_tm *) EFIABI_ATTR; efi_status (*rt_setvirtual)(u_long, u_long, uint32_t, struct efi_md *) EFIABI_ATTR; efi_status (*rt_cvtptr)(u_long, void **) EFIABI_ATTR; efi_status (*rt_getvar)(efi_char *, struct uuid *, uint32_t *, u_long *, void *) EFIABI_ATTR; efi_status (*rt_scanvar)(u_long *, efi_char *, struct uuid *) EFIABI_ATTR; efi_status (*rt_setvar)(efi_char *, struct uuid *, uint32_t, u_long, void *) EFIABI_ATTR; efi_status (*rt_gethicnt)(uint32_t *) EFIABI_ATTR; efi_status (*rt_reset)(enum efi_reset, efi_status, u_long, efi_char *) EFIABI_ATTR; }; #endif struct efi_systbl { struct efi_tblhdr st_hdr; #define EFI_SYSTBL_SIG 0x5453595320494249UL efi_char *st_fwvendor; uint32_t st_fwrev; uint32_t __pad; void *st_cin; void *st_cinif; void *st_cout; void *st_coutif; void *st_cerr; void *st_cerrif; uint64_t st_rt; void *st_bs; u_long st_entries; uint64_t st_cfgtbl; }; extern vm_paddr_t efi_systbl_phys; struct efirt_callinfo; /* Internal MD EFI functions */ int efi_arch_enter(void); void efi_arch_leave(void); vm_offset_t efi_phys_to_kva(vm_paddr_t); int efi_rt_arch_call(struct efirt_callinfo *); bool efi_create_1t1_map(struct efi_md *, int, int); void efi_destroy_1t1_map(void); /* Public MI EFI functions */ int efi_rt_ok(void); int efi_get_table(struct uuid *uuid, void **ptr); int efi_get_time(struct efi_tm *tm); int efi_get_time_capabilities(struct efi_tmcap *tmcap); -int efi_reset_system(void); +int efi_reset_system(enum efi_reset type); int efi_set_time(struct efi_tm *tm); int efi_var_get(uint16_t *name, struct uuid *vendor, uint32_t *attrib, size_t *datasize, void *data); int efi_var_nextname(size_t *namesize, uint16_t *name, struct uuid *vendor); int efi_var_set(uint16_t *name, struct uuid *vendor, uint32_t attrib, size_t datasize, void *data); #endif /* _KERNEL */ #endif /* _SYS_EFI_H_ */